Pith. sign in

REVIEW 5 major objections 5 minor 35 references

LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LinguaMark, a 6,875-pair benchmark in 11 languages, finds closed-source vision-language models generally outperform open-source ones on relevancy and faithfulness, while gender and low-resource languages like Tamil and Urdu show the most…

desk verdict A useful fairness-focused multilingual VQA benchmark, but its headline numbers rest on unvalidated LLM judges and a few internal inconsistencies; fixable and worth refereeing. read the letter →

arxiv 2507.07274 v1 pith:OYRS4KUY submitted 2025-07-09 cs.CV cs.AIcs.CL

Ananya Raval , Aravind Narayanan , Vahid Reza Khazaie , Shaina Raza This is my paper
classification cs.CVcs.AIcs.CL
keywords multilingualvisualquestionansweringlargemultimodalmodelsfairnessevaluationbiasmeasurementlow-resourcelanguagesLLM-as-judgebenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LinguaMark, a benchmark for asking vision-language models questions about images in eleven languages and grading their answers on bias, relevancy, and faithfulness. It is built from 6,875 human-verified image-question pairs spanning five social attributes (gender, age, occupation, ethnicity, and sports) and tested on seven open- and closed-source models. The central finding is that closed-source models generally outperform open ones on answer quality, that gender prompts trigger the most biased responses, and that low-resource languages such as Tamil and Urdu show the weakest results. If the benchmark is sound, it gives researchers a reusable way to audit multilingual fairness in multimodal systems instead of relying on English-only evaluation.

What carries the argument

The load-bearing object is LinguaMark itself: a curated set of 625 images, each paired with a question and reference answer in English and in ten translated languages, giving 6,875 VQA pairs evenly divided across five social attributes. The evaluation mechanism is prompt-based scoring in which GPT-4o-mini judges bias and faithfulness and GPT-4o judges answer relevancy through templated zero-shot prompts; translations were also generated by GPT-4o, then verified by native speakers. The benchmark works by treating language and social attribute as controlled variables, so differences between models can be attributed to how each model handles the same image-question pair in a given language.

What would settle it

Take a random sample of about 200 LinguaMark outputs across the eleven languages and have independent native speakers rate bias, relevancy, and faithfulness using the same prompts. If agreement with the GPT judges is much lower for Tamil and Urdu than for English, or if the model ranking changes, the benchmark's central comparisons would not survive.

Watch

Extended reading notes

Core claim

The paper's claim is that a standardized multilingual visual question answering test can expose fairness and quality gaps that single-language benchmarks miss. Across all languages and attributes, Gemini 2.5 Flash achieves the highest answer relevancy (87.50%) and faithfulness (95.11%), while the GPT-4o family shows the lowest average bias (11.88%) and the open-source Qwen2.5 model leads its category with 86.12% faithfulness and strong cross-lingual generalization. Bias follows a consistent ordering across models: gender is the most biased attribute, followed by age, occupation, ethnicity, and sports. English scores best overall, while Tamil and Urdu show the highest bias and among the lowest relevancy and faithfulness scores.

Load-bearing premise

The whole evaluation assumes GPT-4o-mini and GPT-4o judge bias, faithfulness, and relevance correctly in all eleven languages, with no human check on the judge outputs; if those judges are biased in Tamil or Urdu, every reported score and ranking could shift.

Editorial extensions

If this is right

  • If LinguaMark is adopted, a model's multilingual fairness can be audited along three axes at once, not just accuracy.
  • The consistent bias ordering (gender over age, occupation, ethnicity, sports) implies that interventions targeting gender bias may give the largest fairness payoff.
  • Qwen2.5's strong low-resource performance suggests open-weight models can generalize to languages absent from their reported training lists, making reproducible auditing feasible.
  • The English advantage and Tamil/Urdu deficit imply that training-data language balance, not just model scale, drives multilingual VQA quality.
  • Releasing the benchmark and code lets other groups rerun the comparison as new vision-language models appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the judge models are themselves weakest in Tamil and Urdu, the reported low-resource scores may partly measure the judge rather than the candidate models; testing with native-speaker ratings on a subset would settle this.
  • Beyond the paper: the fixed bias ranking across all seven models hints that the pattern comes from shared pretraining data or benchmark construction rather than model-specific choices; a probing study on synthetic images could separate those causes.
  • Beyond the paper: a practical extension would be to turn LinguaMark into a regression gate for model releases, flagging when a new checkpoint increases gender bias or degrades Tamil and Urdu performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces LinguaMark, a multilingual visual question answering (VQA) benchmark containing 6,875 image-text pairs across 11 languages and five social attributes, and evaluates seven large multimodal models (five open-source and two closed-source) on three metrics: Bias, Answer Relevancy, and Faithfulness. The evaluation protocol uses GPT-4o-mini and GPT-4o as automatic judges. The main findings are that closed-source models, especially Gemini-2.5-flash-preview, achieve the highest relevancy and faithfulness scores, that gender is the most biased social attribute, and that low-resource languages such as Tamil and Urdu show the worst bias and faithfulness performance.

Significance. If the reported results are trustworthy, LinguaMark would be a useful, reusable audit tool for multilingual fairness in LMMs, and the paper's release of benchmark and evaluation code is a concrete contribution. The human-in-the-loop translation verification for 11 languages is also a strength. However, the central quantitative claims are entirely mediated by unvalidated LLM-as-a-judge evaluations, and there are unresolved inconsistencies in the data counts and model naming. Until these load-bearing issues are addressed, the exact numerical conclusions in Tables V and Figures 7-8 cannot be considered established.

major comments (5)
  1. [§III.C, Figure 6] All three metrics are scored by GPT-4o-mini (Bias and Faithfulness) or GPT-4o (Answer Relevancy) with no reported human validation, no inter-annotator agreement, and no per-language judge-quality check. Because the reference answers used for Faithfulness were themselves generated by GPT-4o (§III.A), high faithfulness scores may reflect agreement with a GPT-generated reference rather than with visual ground truth. The headline findings—closed-source supremacy, gender as the most biased attribute, and Tamil/Urdu as worst-performing—are exactly the kind of claims that could be produced by systematic judge bias in low-resource languages. The paper must provide a validation holdout, agreement statistics, or per-language judge reliability evidence before these scores can be interpreted.
  2. [Table III, §III.A] The dataset size is internally inconsistent: 625 unique images times 11 languages equals 6,875 pairs, matching the abstract and §III.A, but Table III reports 'Total annotated instances' as 6,278. If some language or attribute cells are missing or duplicated, the per-language and per-attribute averages in Figures 7 and 8 cannot be interpreted as stated. Please reconcile these numbers and report the actual per-language and per-attribute instance counts.
  3. [§IV.D] The text states that the authors evaluate '7 models (2 open and 5 closed source),' but the paper actually evaluates five open-source models (Aya-Vision, Gemma3, LLaMA-3.2, Phi-4, Qwen2.5) and two closed-source models (GPT-4o and Gemini-2.5-flash-preview). This reversal obscures the open/closed comparison that is a central claim of the paper and must be corrected.
  4. [Figure 6, §III.C] The Bias Evaluation Prompt asks the judge to compare 'two model responses to similar questions differing only in a protected attribute,' but the dataset is composed of single-image VQA pairs with one question and one response. The paper does not describe how such pairs are constructed, whether the compared responses come from the same image, or how attribute differences are isolated. Without this information, the Bias metric may be measuring differences unrelated to the protected attribute.
  5. [Table V, §IV.A] The model naming is inconsistent: Table V lists a 'GPT-4o-mini†' baseline, while §IV.B attributes the lowest bias (11.88%) to 'GPT-4o'; §IV.A refers to 'GPT-4o' being constrained to 150 tokens and also mentions 'Gemini-2.5 Pro' and 'Gemini-2.5-flash-preview.' The paper must state exactly which model versions were evaluated and which judge model was used for each metric, because the current text is not reproducible.
minor comments (5)
  1. [Abstract, §I] The abstract and introduction describe the dataset as 'image-text pairs,' but the benchmark actually consists of image-question-answer triplets; please use consistent terminology.
  2. [Table II] Table II is titled 'Languages supported by the models' but only lists open-source models; the discussion in §IV.D refers to model cards for GPT-4o and Gemini2.5, so the table should either include those rows or explicitly state that their language support is not publicly documented.
  3. [Table IV, §IV.A] The hyperparameter table sets temperature 1.0, top-p 1.0, and top-k 50 for open-source models, while the text says 'Sampling was disabled for all models to ensure deterministic outputs'; please clarify whether sampling was enabled or disabled and how temperature interacts with that statement.
  4. [§IV.D] The sentence 'We have added collected some information and added it to Table II' is ungrammatical and should be rewritten for clarity.
  5. [§V] The limitations section acknowledges that LLM-based annotations and human reviewers may introduce biases, but this acknowledgment does not replace the missing quantitative validation of the judge model; I recommend that the paper report judge reliability statistics in the main text.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation loop: GPT-generated references and GPT-4o-mini self-judging make the headline faithfulness and bias results partially circular.

  1. self definitional [Section III.C (Faithfulness definition) and Section III.A / contributions (reference generation)]
    "Faithfulness ( ↑): Faithfulness is measure to detect how aligned the answer is with the ground truth answer in its respective language, which can measure multilingual fluency. ... a reference answer generated by GPT-4 and validated by native-speaking human annotators to ensure linguistic and cultural fidelity."

    The metric labeled 'faithfulness' is operationally defined as alignment with the 'ground truth answer,' and that ground truth answer was generated by GPT-4—the same model family that supplies the judge (GPT-4o-mini) and the evaluated closed-source systems. The paper elsewhere frames faithfulness as visual factuality ('factually consistent with the provided image description'), but the stated operational definition does not compare against the image; it compares against a GPT-generated reference. Hence the headline claim that closed-source models are more faithful reduces, by construction, to 'answers are more similar to a GPT-4-generated reference as judged by GPT-4o-mini.' This is a self-referential loop rather than an independent check of visual truth.

  2. self definitional [Section III.C (evaluation protocol) and Table V (model results)]
    "We employed three key evaluation metrics, bias, answer relevancy, and faithfulness, all assessed using prompt-based evaluation protocols with GPT-4o-mini as the judge. ... GPT-4o-mini† 11.88 66.51 85.22"

    Table V lists GPT-4o-mini as one of the seven evaluated models, reporting the lowest bias (11.88%) and a high faithfulness (85.22%). Because GPT-4o-mini is also the judge that produces every metric score, its own row is a self-assessment: the bias score is GPT-4o-mini judging whether its own outputs are biased, and the faithfulness score is GPT-4o-mini judging how well its own outputs align with a GPT-4-generated reference. This is a concrete identity between the measurer and the measured, so the paper's claim that closed-source GPT-4o-mini is least biased is not an independent comparison; it is, at least in part, the model grading itself.

full rationale

The paper's central evaluation loop has two concrete circular components. First, the faithfulness metric is defined as agreement with a ground-truth answer that was generated by GPT-4 and then scored by GPT-4o-mini, so the reported 'faithfulness to visual evidence' is really 'agreement with a GPT-family reference answer.' Second, GPT-4o-mini appears both as the judge for all three metrics and as an evaluated model in Table V, meaning its own bias and faithfulness scores are self-judgments. These are not merely unvalidated-judge concerns; they are definitional reductions: the measured quantity is partly the same model family's output compared to itself. This warrants a score of 6 rather than 0-2. The score is not higher because not every conclusion is forced by construction: the answer-relevancy results use GPT-4o as judge rather than GPT-4o-mini, the dataset itself is released and was human-reviewed, and the use of the authors' prior HumaniBench data is ordinary data provenance rather than a circular step. Still, the headline claims about closed-source superiority, gender as the most biased attribute, and low-resource-language failures are all mediated by this self-referential evaluation loop, so the benchmark's headline numbers cannot be treated as independent external evidence without replacing the judge or validating it against human judgments.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters. The paper relies on three domain assumptions: judges are valid across languages, translations preserve meaning, and the source data is unbiased. These are not verified independently in the paper.

assumptions (3)
  • domain assumption GPT-4o-mini and GPT-4o judges provide valid, unbiased measurements of bias, relevance, and faithfulness across all 11 languages.
    All metrics in Section III.C rely on these judges; no human validation of judge outputs is reported.
  • domain assumption GPT-4o-generated translations preserve semantic and cultural equivalence across languages.
    Section III.A: translations generated by GPT-4o and verified by native speakers; verification details are not given.
  • domain assumption The source images and English QA pairs from HumaniBench [15] are appropriate and unbiased.
    Section III.A: curated from prior work; human review claimed but not detailed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation." pith.science (2026). https://pith.science/paper/OYRS4KUY

@misc{pith2026250707274,
  author       = {Pith},
  title        = {Pith review of: LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYRS4KUY}},
  note         = {Machine review of arXiv:2507.07274}
}
read the original abstract

Large Multimodal Models (LMMs) are typically trained on vast corpora of image-text data but are often limited in linguistic coverage, leading to biased and unfair outputs across languages. While prior work has explored multimodal evaluation, less emphasis has been placed on assessing multilingual capabilities. In this work, we introduce LinguaMark, a benchmark designed to evaluate state-of-the-art LMMs on a multilingual Visual Question Answering (VQA) task. Our dataset comprises 6,875 image-text pairs spanning 11 languages and five social attributes. We evaluate models using three key metrics: Bias, Answer Relevancy, and Faithfulness. Our findings reveal that closed-source models generally achieve the highest overall performance. Both closed-source (GPT-4o and Gemini2.5) and open-source models (Gemma3, Qwen2.5) perform competitively across social attributes, and Qwen2.5 demonstrates strong generalization across multiple languages. We release our benchmark and evaluation code to encourage reproducibility and further research.

Figures

Figures reproduced from arXiv: 2507.07274 by the authors.

Figure 1
Figure 1. VQA example showing an image, text pairs. Image [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the LinguaMark evaluation framework. The benchmark uses open-ended VQA prompts grounded in real-world news images and evaluates LMM responses across 11 languages and five social attributes: age, gender, occupation, ethnicity, and sports [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the multi-stage quality assurance pipeline [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Prompt used for evaluation. comprehensive comparison across key evaluation metrics such as bias, answer relevancy, and faithfulness. Closed-source models accessed via proprietary APIs. We employed three key evaluation metrics, bias, answer relevancy, and faithfulness, …
Figure 7
Figure 7. Figure 7: Heat-maps of Bias (lower ↓ is better), Answer Rele￾vancy, and Faithfulness (both higher ↑ is better) across five attributes and seven models. Darker shades indicate better performance for each metric. bias at 11.88%, whereas open-source Gemma3-12B-it shows the highest …
Figure 8
Figure 8. Figure 8: Radar plots across 11 languages for Bias [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Qualitative examples of model responses to open-ended VQA tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 17 canonical work pages

  1. [1]

    The multilingual mind: A survey of multilingual reasoning in language models,

    A. Ghosh, D. Datta, S. Saha, and C. Agarwal, “The multilingual mind: A survey of multilingual reasoning in language models,” arXiv preprint arXiv:2502.09457, 2025

    arXiv 2025

  2. [2]

    All languages matter: Evaluating lmms on culturally diverse 100 languages,

    A. Vayani, D. Dissanayake, H. Watawana, N. Ahsan, N. Sasikumar, O. Thawakar, H. B. Ademtew, Y . Hmaiti, A. Kumar, K. Kuckreja, et al., “All languages matter: Evaluating lmms on culturally diverse 100 languages,” arXiv preprint arXiv:2411.16508 , 2024

    arXiv 2024

  3. [3]

    VQA: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pp. 2425– 2433, 2015

    work page 2015

  4. [4]

    A-okvqa: A benchmark for visual question answering using world knowledge,

    D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, “A-okvqa: A benchmark for visual question answering using world knowledge,” in European conference on computer vision , pp. 146–162, Springer, 2022

    work page 2022

  5. [5]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,

    J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick, “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,” 2016

    work page 2016

  6. [6]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13 , pp. 740–755, Springer, 2014

    2014

  7. [7]

    From recognition to cognition: Visual commonsense reasoning,

    R. Zellers, Y . Bisk, A. Farhadi, and Y . Choi, “From recognition to cognition: Visual commonsense reasoning,” 2019

    work page 2019

  8. [8]

    The state and fate of linguistic diversity and inclusion in the nlp world,

    P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the nlp world,” arXiv preprint arXiv:2004.09095, 2020

    arXiv 2004

Show all 35 references
  1. [9]

    Mm-vet: Evaluating large multimodal models for integrated capabilities,

    W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “Mm-vet: Evaluating large multimodal models for integrated capabilities,” arXiv preprint arXiv:2308.02490 , 2023

    2023 arXiv

  2. [10]

    Alignmmbench: Evaluating chinese multimodal alignment in large vision-language models,

    Y . Wu, W. Yu, Y . Cheng, Y . Wang, X. Zhang, J. Xu, M. Ding, and Y . Dong, “Alignmmbench: Evaluating chinese multimodal alignment in large vision-language models,” arXiv preprint arXiv:2406.09295 , 2024

    2024 arXiv

  3. [11]

    Seed-bench: Benchmarking multimodal large language models,

    B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan, “Seed-bench: Benchmarking multimodal large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299–13308, June 2024

    2024

  4. [12]

    Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models,

    R. J. Das, S. E. Hristov, H. Li, D. I. Dimitrov, I. Koychev, and P. Nakov, “Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models,” arXiv preprint arXiv:2403.10378, 2024

    2024 arXiv

  5. [13]

    Mvl-sib: A massively multilingual vision-language benchmark for cross-modal topical matching,

    F. D. Schmidt, F. Schneider, C. Biemann, and G. Glava ˇs, “Mvl-sib: A massively multilingual vision-language benchmark for cross-modal topical matching,” arXiv preprint arXiv:2502.12852 , 2025

    2025 arXiv

  6. [14]

    Benchmax: A comprehensive multilingual evaluation suite for large language models,

    X. Huang, W. Zhu, H. Hu, C. He, L. Li, S. Huang, and F. Yuan, “Benchmax: A comprehensive multilingual evaluation suite for large language models,” arXiv preprint arXiv:2502.07346 , 2025

    2025 arXiv

  7. [15]

    Humanibench: A human-centric framework for large multimodal models evaluation,

    S. Raza, A. Narayanan, V . R. Khazaie, A. Vayani, M. S. Chettiar, A. Singh, M. Shah, and D. Pandya, “Humanibench: A human-centric framework for large multimodal models evaluation,” 2025

    2025

  8. [16]

    Multimodal large language models: A survey,

    J. Wu, W. Gan, Z. Chen, S. Wan, and S. Y . Philip, “Multimodal large language models: A survey,” in 2023 IEEE International Conference on Big Data (BigData) , pp. 2247–2256, IEEE, 2023

    2023

  9. [17]

    Wenlan: Bridging vision and language by large-scale multi-modal pre-training,

    Y . Huo, M. Zhang, G. Liu, H. Lu, Y . Gao, G. Yang, J. Wen, H. Zhang, B. Xu, W. Zheng, et al. , “Wenlan: Bridging vision and language by large-scale multi-modal pre-training,” arXiv preprint arXiv:2103.06561, 2021

    2021 arXiv

  10. [18]

    Unks everywhere: Adapting multilingual language models to new scripts,

    J. Pfeiffer, I. Vuli ´c, I. Gurevych, and S. Ruder, “Unks everywhere: Adapting multilingual language models to new scripts,” arXiv preprint arXiv:2012.15562, 2020

    2012 arXiv

  11. [19]

    Afriberta: Towards viable multilingual language models for low-resource languages,

    K. Ogueji, “Afriberta: Towards viable multilingual language models for low-resource languages,” Master’s thesis, University of Waterloo, 2022

    2022

  12. [20]

    mgpt: Few-shot learners go multilingual,

    O. Shliazhko, A. Fenogenova, M. Tikhonova, A. Kozlova, V . Mikhailov, and T. Shavrina, “mgpt: Few-shot learners go multilingual,”Transactions of the Association for Computational Linguistics , vol. 12, pp. 58–79, 2024

    2024

  13. [21]

    Xtreme-r: Towards more challenging and nuanced multilingual evaluation,

    S. Ruder, N. Constant, J. Botha, A. Siddhant, O. Firat, J. Fu, P. Liu, J. Hu, D. Garrette, G. Neubig, et al. , “Xtreme-r: Towards more challenging and nuanced multilingual evaluation,” arXiv preprint arXiv:2104.07412, 2021

    2021 arXiv

  14. [22]

    Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation,

    S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila- Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y . Susanto,et al., “Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation,” arXiv preprint arXiv:2412.03304 , 2024

    2024 arXiv

  15. [23]

    Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages,

    J. FitzGerald, C. Hench, C. Peris, S. Mackie, K. Rottmann, A. Sanchez, A. Nash, L. Urbach, V . Kakarala, R. Singh, et al. , “Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages,” arXiv preprint arXiv:2204.08582 , 2022

    2022 arXiv

  16. [24]

    Do all languages cost the same? tokenization in the era of commercial language models,

    O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. R. Mortensen, N. A. Smith, and Y . Tsvetkov, “Do all languages cost the same? tokenization in the era of commercial language models,” arXiv preprint arXiv:2305.13707, 2023

    2023 arXiv

  17. [25]

    Parameter-efficient fine-tuning in large language models: a survey of methodologies,

    L. Wang, S. Chen, L. Jiang, S. Pan, R. Cai, S. Yang, and F. Yang, “Parameter-efficient fine-tuning in large language models: a survey of methodologies,” Artificial Intelligence Review , vol. 58, no. 8, p. 227, 2025

    2025

  18. [26]

    A review on fairness in machine learning,

    D. Pessach and E. Shmueli, “A review on fairness in machine learning,” ACM Computing Surveys (CSUR) , vol. 55, no. 3, pp. 1–44, 2022

    2022

  19. [27]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ere, et al. , “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786 , 2025

    2025 arXiv

  20. [28]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  21. [29]

    Phi-4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al., “Phi-4 technical report,” arXiv preprint arXiv:2412.08905 , 2024

    2024 arXiv

  22. [30]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966, vol. 1, no. 2, p. 3, 2023

    2023 arXiv

  23. [31]

    GPT-4o System Card,

    OpenAI, “GPT-4o System Card,” Aug. 2024. White-paper style system card, version released August 8, 2024. Accessed 2025-04-24

    2024

  24. [32]

    Gemini 2.0 Flash,

    G. Cloud, “Gemini 2.0 Flash,” Apr. 2025. Generative AI on Vertex AI documentation. Last updated 2025-04-23

    2025

  25. [33]

    Aya vision: Expanding the worlds ai can see,

    Cohere For AI Team, “Aya vision: Expanding the worlds ai can see,” Cohere Blog, 2025. Accessed: 2025-03-18

    2025

  26. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  27. [35]

    Fair enough: Develop and assess a fair-compliant dataset for large language model training?,

    S. Raza, S. Ghuge, C. Ding, E. Dolatabadi, and D. Pandya, “Fair enough: Develop and assess a fair-compliant dataset for large language model training?,” Data Intelligence, vol. 6, no. 2, pp. 559–585, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.