Pith. sign in

REVIEW 3 major objections 6 minor 48 references

VLM@school -- Evaluation of AI image understanding on German middle school knowledge

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read German middle-school questions beat the strongest open vision-language models, with best accuracy 43.26%.

desk verdict A genuinely useful German-language VLM benchmark, but the unvalidated LLM judge makes the headline accuracy numbers provisional. read the letter →

arxiv 2506.11604 v2 pith:XARNTTBB submitted 2025-06-13 cs.AI cs.CLcs.CV

René Peinl , Vincent Tischler This is my paper
classification cs.AIcs.CLcs.CV
keywords VisionLanguageModelVLMGermanbenchmarkopen-endedquestionsadversarialmiddleschoolcurriculummultimodalevaluationLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VLM@school, a German-language benchmark of 2,038 open-ended questions tied to 486 images from real middle school curricula across nine subjects. The authors test thirteen open-weight vision-language models and find that even the strongest reaches only 43.26% overall accuracy, while models that dominate English-language benchmarks often rank lower here. The point being established is that school-level questions requiring visual interpretation plus background knowledge are already hard for current VLMs, and that popular English multiple-choice benchmarks overstate real-world multimodal understanding. This matters because it gives the field a non-English, open-ended testbed for measuring progress.

What carries the argument

The benchmark itself is the instrument: 486 images and 2,038 native-German open-ended questions built from grades 8-10 textbooks, with text-only-answerable questions filtered out, some questions printed inside the image, and adversarial items that are intentionally unanswerable. Answers are scored by a panel of three LLM judges, averaging the Llama and Mistral scores while dropping Qwen for being overly critical. This design removes the multiple-choice shortcut and forces models to read visual details and combine them with school-level knowledge, which is what produces the headline numbers.

What would settle it

Take a random sample of about 200 model answers and have German teachers or native speakers score them against the same ground truth; if their scores differ systematically from the averaged LLM-judge scores by more than a few percentage points, the reported accuracies and the below-45% conclusion would need revision.

Watch

Extended reading notes

Core claim

The central claim is that middle-school questions in German, asked with open-ended answers and no text-only shortcut, expose weaknesses that MMMU does not reveal. The best model, Qwen2.5-VL 32B, scores 43.26% overall; the worst, Ovis2 4B, scores 23.53%. Music, mathematics, and adversarial questions that ask about something absent from the image are the hardest, with best scores near 31%, 34%, and 33% respectively. The paper further reports that the InternVL3 model family dominates MMMU but does not translate that dominance to this dataset, while Mistral Small 3.1, Qwen2.5-VL 32B, and Gemma 3 27B outperform it. The authors conclude that benchmark rankings do not reliably predict accuracy on real-world-style visual questions.

Load-bearing premise

The entire accuracy ranking rests on the assumption that the average of two large language models acting as graders correctly scores open-ended German answers, but the paper provides no human validation, no inter-judge agreement statistics, and no confidence intervals.

Editorial extensions

If this is right

  • If the finding holds, beating 45% accuracy on this benchmark becomes a higher bar than MMMU for open-weight models, and strong MMMU scores cannot be assumed to transfer.
  • German-language visual understanding needs separate evaluation; native-speaker question construction avoids the translation artifacts that make cross-lingual comparisons unreliable.
  • The published scores are only as trustworthy as the LLM judges, so future runs should report inter-judge agreement and human spot checks.
  • The adversarial questions isolate a concrete failure mode: models confidently hallucinate near-miss answers instead of saying that the requested object is not in the image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not test is building paired German and English versions of the same questions; if scores stay low in both, the bottleneck is visual-plus-knowledge reasoning rather than the German language.
  • The paper's own observation that a single textual keyword can flip a wrong answer to correct suggests a testable extension: verbalizing image content into text before answering, via chain-of-thought prompting, should substantially raise scores.
  • The decision to discard one of three judges by hand, without a validation set, implies that the absolute accuracy values are less stable than the relative ranking; a human-scored subset would anchor the numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VLM@school, a new German-language benchmark for vision-language models consisting of 2,038 open-ended questions over 486 images from nine middle-school domains. Thirteen open-weight VLMs are evaluated. The central findings are that the best model achieves below 45% overall accuracy, performance is particularly poor on music, math, and adversarially constructed questions, and performance on MMMU does not reliably transfer to this benchmark. The dataset and some raw results are publicly released.

Significance. If the measurement is reliable, the paper makes a useful contribution: a native-speaker German benchmark with open-ended questions, explicit filtering of text-only-solvable questions, and adversarial no-answer items. It addresses a real gap (non-English, school-level multimodal evaluation) and the public release supports reproducibility. The strongest conclusion, that open-weight VLMs score below 45%, is plausible but rests entirely on an unvalidated LLM-as-judge protocol, which is the main risk to the paper's validity.

major comments (3)
  1. [5] The accuracy numbers in Tables 3-5 are entirely derived from an LLM-as-judge protocol that is not validated. The paper states "We finally used the average of Llama and Mistral since we found Qwen to be overly critical," which is a post-hoc exclusion of one judge based on its observed behavior on the same data being scored. No per-judge scores, inter-judge agreement statistics, human validation, or confidence intervals are reported. The best model (QwenVL2.5 32B) scores 43.26% (Table 3), so a systematic leniency or strictness of only about 2 percentage points in the two remaining judges would change the headline claim "even the strongest models achieve less than 45%." The Limitations section (Sec. 8) acknowledges the missing human baseline but does not use it to calibrate the judges. Please provide a human-scored random sample of at least 100-200 answers, report judge-human agreement and judge-judge agreement, and give a bias-corrected estimate with uncertainty bounds.
  2. [5] The "scoring hints" introduced on a case-by-case basis are not a reproducible part of the protocol. The paper claims these hints "improves the correctness of judge scores significantly," but gives no quantitative evidence, no description of how many hints were added, for which questions, or how they were selected. Because the same judge scores feed all downstream results, the hints could overfit the judges to this specific dataset and inflate or deflate accuracy in an unmeasured way. Please document the hinting procedure, provide examples, and report a sensitivity analysis (e.g., accuracy with and without hints on a subsample).
  3. [6] The comparison between MMMU and VLM@school (Fig. 4) and the resulting claim that "results on public benchmarks do not necessarily reflect a model's accuracy" are made without any statistical analysis. The differences among the top models (e.g., QwenVL2.5 32B at 43.26%, Mistral small 3.1 at 42.86%, Llama4 Scout at 40.29%) are not tested for significance, and Table 4 mixes measured and self-reported MMMU scores without clear separation. Please report confidence intervals for the VLM@school accuracies and clearly distinguish measured from self-reported MMMU values before drawing dissociation conclusions.
minor comments (6)
  1. [2] The sentence "Exams-V is probably the closet to our dataset" should read "closest."
  2. [2] The evaluation metrics cited are "ROGUE" and "BERTscore"; the former should be "ROUGE."
  3. [7] In the discussion of physics results, "Hertzsprung -Russel diagram" should be "Hertzsprung-Russell diagram."
  4. [Table 5] The column header "Modell" is German; it should be "Model" for consistency with the rest of the table.
  5. [Table 4] The asterisk footnote for MMMU scores should be explained in the caption or text, since it is not immediately clear which models' scores are self-reported and what that means for the comparison.
  6. [8] The statement "We expect, however, average humans to also not reach >90%" is presented without supporting data and should be explicitly marked as an untested hypothesis, especially since no human baseline is provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical benchmark with no fitted parameters, no equation-level reductions, and no load-bearing self-citation chain.

full rationale

The paper's central claims are empirical measurements: thirteen open-weight VLMs are prompted on a new dataset and their open-ended answers are scored by an LLM judge panel. There is no derivation in which an output is defined in terms of the input, no fitted parameter that is later renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The only self-citations ([12], [19]) are contextual: [19] is cited for prior findings on German factual data and for prior experience with the judge models, but the current benchmark scores are computed independently from the current dataset and do not reduce to those prior results. The evaluation design choices, such as using three LLM judges, dropping the Qwen judge after finding it 'overly critical', and adding case-by-case scoring hints, are potential threats to measurement validity and reliability, especially because no human calibration or inter-judge agreement statistics are provided and the paper itself acknowledges the lack of a human baseline in Section 8. However, these are robustness concerns about the measurement instrument, not circularity: the reported percentages are not equivalent to the judge prompt or to the dataset construction by definition. No equation in the paper is shown to equal itself via a different label, and no load-bearing step is justified only by a self-citation. The honest finding is therefore no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's conclusions rest on the assumption that LLM judges can score open-ended German answers reliably, that the benchmark questions are unambiguous and answerable by humans, and that the text-only filtering guarantees the image is necessary. These are domain assumptions with limited independent validation, since no human baseline is provided.

assumptions (4)
  • domain assumption The three-judge LLM panel (Mistral, Llama, Qwen) produces accurate scores for open-ended answers; averaging Llama and Mistral is a valid aggregation.
    Section 5 states Qwen was 'overly critical' and was dropped; no human validation of judge scores is reported.
  • domain assumption All benchmark questions have unambiguous ground-truth answers that a knowledgeable human would produce.
    Section 8 acknowledges no statistically significant human baseline was collected.
  • domain assumption The filtering step ensures questions cannot be answered from text alone.
    Section 2 mentions LLM-only tests yielding 8.5% correct for the best model, but no systematic verification on the full dataset is shown.
  • domain assumption Model scores are comparable despite some models being quantized to 4-bit AWQ.
    Section 8 compares AWQ vs FP16 on three InternVL3 models, showing small deltas, but for other large models quantization effects are not directly measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLM@school -- Evaluation of AI image understanding on German middle school knowledge." pith.science (2026). https://pith.science/paper/XARNTTBB

@misc{pith2026250611604,
  author       = {Pith},
  title        = {Pith review of: VLM@school -- Evaluation of AI image understanding on German middle school knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XARNTTBB}},
  note         = {Machine review of arXiv:2506.11604}
}
read the original abstract

This paper introduces a novel benchmark dataset designed to evaluate the capabilities of Vision Language Models (VLMs) on tasks that combine visual reasoning with subject-specific background knowledge in the German language. In contrast to widely used English-language benchmarks that often rely on artificially difficult or decontextualized problems, this dataset draws from real middle school curricula across nine domains including mathematics, history, biology, and religion. The benchmark includes over 2,000 open-ended questions grounded in 486 images, ensuring that models must integrate visual interpretation with factual reasoning rather than rely on superficial textual cues. We evaluate thirteen state-of-the-art open-weight VLMs across multiple dimensions, including domain-specific accuracy and performance on adversarial crafted questions. Our findings reveal that even the strongest models achieve less than 45% overall accuracy, with particularly poor performance in music, mathematics, and adversarial settings. Furthermore, the results indicate significant discrepancies between success on popular benchmarks and real-world multimodal understanding. We conclude that middle school-level tasks offer a meaningful and underutilized avenue for stress-testing VLMs, especially in non-English contexts. The dataset and evaluation protocol serve as a rigorous testbed to better understand and improve the visual and linguistic reasoning capabilities of future AI systems.

Figures

Figures reproduced from arXiv: 2506.11604 by the authors.

Figure 1
Figure 1. Example question from biology: “Which limb belongs to a whale? State the number!”. 7.1 Adversarial questions The answers to the adversarial questions were also insightful in the religion domain. The models fell for most of the misleading questions, as expected. Only in very few cases, they correctly answered with “this is not shown in the picture.” The bigger models were better. Qwen 2.5 VL 72B took the lead for rel… view at source ↗
Figure 2
Figure 2. Cropped version of the image “The stoning of the adulteress” [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Model size (log2) vs. overall accuracy for three model families. Comparing the models’ MMMU score against their performance in our dataset (see [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: ), we see that Qwen 2.5 VL shows roughly a linear relation, whereas InternVL3 significantly underperforms on our dataset compared to MMMU. Ovis2 is within the linear trend for 4B and 16B but underperforms for 34B. Gemma3 and Llama3 are slightly above and below the tren…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 30 canonical work pages

  1. [1]

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei -Fei, L.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015)

    work page 2015

  2. [2]

    In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)

    Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an asr corpus based on public domain audio books. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5206–5210. IEEE (2015)

    work page 2015

  3. [3]

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z.: MMLU -Pro: A More Robust and Challenging Multi -Task Language Understanding Benchmark. Adv. Neural Inf. Process. Syst. 37, 95266–95290 (2025)

    work page 2025

  4. [4]

    Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., Fernandez- Llorca, D.: Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation, https://arxiv.org/abs/2502.06559v1, last accessed 2025/05/26

    arXiv 2025

  5. [5]

    Raji, I.D., Bender, E.M., Paullada, A., Denton, E., Hanna, A.: AI and the Everything in the Whole Wide World Benchmark, http://arxiv.org/abs/2111.15366, (2021)

    arXiv 2021

  6. [6]

    van der, Hofstätter, F., Jaffe, O., Brown, S.F., Ward, F.R.: AI Sandbagging: Language Models can Strategically Underperform on Evaluations, http://arxiv.org/abs/2406.07358, (2025)

    Weij, T. van der, Hofstätter, F., Jaffe, O., Brown, S.F., Ward, F.R.: AI Sandbagging: Language Models can Strategically Underperform on Evaluations, http://arxiv.org/abs/2406.07358, (2025). https://doi.org/10.48550/arXiv.2406.07358

    arXiv doi:10.48550/arxiv.2406.07358 2025

  7. [7]

    In: Proceedings of the ACM Conference on Health, Inference, and Learning

    Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Re, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: Proceedings of the ACM Conference on Health, Inference, and Learning. pp. 151 –159. ACM, Toronto Ontario Canada (2020). https://doi.org/10.1145/3368555.3384468

    work page arXiv 2020

  8. [8]

    Ethayarajh, K., Jurafsky, D.: Utility is in the Eye of the User: A Critique of NLP Leaderboards, http://arxiv.org/abs/2009.13888, (2021)

    arXiv 2021

Show all 48 references
  1. [9]

    Reuel-Lamparth, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., Kochenderfer, M.J.: BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. Adv. Neural Inf. Process. Syst. 37, 21763–21813 (2024)

    2024

  2. [10]

    Dehghani, M., Tay, Y., Gritsenko, A.A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., Vinyals, O.: The Benchmark Lottery, http://arxiv.org/abs/2107.07002, (2021)

    2021 arXiv

  3. [11]

    Big Data Soc

    Kang, E.B.: Ground truth tracings (GTT): On the epistemic limits of machine learning. Big Data Soc. 10, 20539517221146122 (2023)

    2023

  4. [12]

    Peinl, R., Wirth, J.: Evaluation of Medium-Sized Language Models in German and English Language. Int. J. Nat. Lang. Comput. 13, 01–18 (2024)

    2024

  5. [13]

    Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., Stanovsky, G.: State of what art? A call for multi -prompt LLM evaluation. Trans. Assoc. Comput. Linguist. 12, 933 –949 (2024)

    2024

  6. [14]

    In: AAAI/ACM Conf

    Rauh, M., Marchal, N., Manzini, A., Hendricks, L.A., Comanescu, R., Akbulut, C., Stepleton, T., Mateos -Garcia, J., Bergman, S., Kay, J.: Gaps in the Safety Evaluation of Generative AI. In: AAAI/ACM Conf. on AI, Ethics, and Society. pp. 1200–1217 (2024)

    2024

  7. [15]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y.: MMMU: A massive multi -discipline multimodal understanding and reasoning benchmark for expert AGI. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9556–...

    2024

  8. [16]

    In: 32nd ACM International Conference on Multimedia

    Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., Chen, K.: VLMEvalKit: An Open -Source ToolKit for Evaluating Large Multi-Modality Models. In: 32nd ACM International Conference on Multimedia. pp. 11198–11201. ACM, Mel...

    2024

  9. [17]

    Announcing ARC-AGI-2 and ARC Prize 2025, https://arcprize.org/blog/announcing -arc- agi-2-and-arc-prize-2025, last accessed 2025/05/26

    2025

  10. [18]

    ArXiv Prepr

    Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C.B.C., Shaaban, M., Ling, J., Shi, S.: Humanity’s last exam. ArXiv Prepr. ArXiv250114249. (2025). 16 René Peinl and Vincent Tischler

    2025

  11. [19]

    Presented at the 21st Int

    Peinl, R., Tischler, V.: Benchmarking Vision Language Models on German Factual Data. Presented at the 21st Int. Conf. on Artificial Intelligence Applications and Innovations , Limassol, Cyprus June 26 (2025)

    2025

  12. [20]

    Gemini 2.5: Our most intelligent AI model, https://blog.google/technology/google - deepmind/gemini-model-thinking-updates-march-2025/, last accessed 2025/05/26

    2025

  13. [21]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, https://ai.meta.com/blog/llama-4-multimodal-intelligence/, last accessed 2025/05/26

    2025

  14. [22]

    Introducing Claude 4, https://www.anthropic.com/news/claude -4, last accessed 2025/05/26

    2025

  15. [23]

    Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., Su, Y., Chen, W., Neubig, G.: MMMU -Pro: A More Robust Multi -discipline Multimodal Understanding Benchmark, http://arxiv.org/abs/2409.02813, (2025)

    2025 arXiv

  16. [24]

    https://doi.org/10.48550/arXiv.2203.10244

    Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning, http://arxiv.org/abs/2203.10244, (2022). https://doi.org/10.48550/arXiv.2203.10244

    2022 doi arXiv

  17. [25]

    In: Leibe, B., Matas, J., Sebe, N., and Welling, M

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A Diagram is Worth a Dozen Images. In: Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 235–251. Springer, Cham (2016)

    2016

  18. [26]

    https://doi.org/10.48550/arXiv.2308.02490

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: MM -Vet: Evaluating Large Multimodal Models for Integrated Capabilities, http://arxiv.org/abs/2308.02490, (2024). https://doi.org/10.48550/arXiv.2308.02490

    2024 doi arXiv

  19. [27]

    Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., Heng, P.-A.: SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems, http://arxiv.org/abs/2503.10627, (2025)

    2025 arXiv

  20. [28]

    In: IEEE/CVF Winter Conf

    Mathew, M., Karatzas, D., Jawahar, C.V.: DocVQA: A dataset for VQA on document images. In: IEEE/CVF Winter Conf. on Applications of Computer Vision. pp. 2200–2209 (2021)

    2021

  21. [29]

    Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., Bai, X.: OCRBench: on the hidden mystery of OCR in large multimodal models. Sci. China Inf. Sci. 67, 220102 (2024). https://doi.org/10.1007/s11432-024-4235-6

    2024 doi

  22. [30]

    https://doi.org/10.48550/arXiv.2503.14607

    Xing, S., Sun, Z., Xie, S., Chen, K., Huang, Y., Wang, Y., Li, J., Song, D., Tu, Z.: Can Large Vision Language Models Read Maps Like a Human?, http://arxiv.org/abs/2503.14607, (2025). https://doi.org/10.48550/arXiv.2503.14607

    2025 doi arXiv

  23. [31]

    Das, R.J., Hristov, S.E., Li, H., Dimitrov, D.I., Koychev, I., Nakov, P.: EXAMS-V: A Multi- Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models, http://arxiv.org/abs/2403.10378, (2024)

    2024 arXiv

  24. [32]

    -G., Xu, Z., Luo, W., Zhang, K., Zhan, D

    Sun, H.-L., Zhou, D.-W., Li, Y., Lu, S., Yi, C., Chen, Q. -G., Xu, Z., Luo, W., Zhang, K., Zhan, D. -C., Ye, H. -J.: Parrot: Multilingual Visual Instruction Tuning, http://arxiv.org/abs/2406.02539, (2024). https://doi.org/10.48550/arXiv.2406.02539

    2024 doi arXiv

  25. [33]

    ArXiv Prepr

    Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X.: M ^ 3 IT: A Large -Scale Dataset towards Multi -Modal Multilingual Instruction Tuning. ArXiv Prepr. ArXiv230604387. (2023)

    2023

  26. [34]

    Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M.O., Roth, S., Vulić, I., Gurevych, I.: xGQA: Cross-Lingual Visual Question Answering, http://arxiv.org/abs/2109.06082, (2022)

    2022 arXiv

  27. [35]

    Park, C., Lee, K., Lim, H., Kim, J., Park, J., Heo, Y.-J., Chang, D.-S., Choo, J.: Translation Deserves Better: Analyzing Translation Artifacts in Cross -lingual Visual Question Answering, http://arxiv.org/abs/2406.02331, (2024)

    2024 arXiv

  28. [36]

    I don’t know

    Deng, Y., Zhao, Y., Li, M., Ng, S. -K., Chua, T.-S.: Don’t Just Say “I don’t know”! Self - aligning Large Language Models for Responding to Unknown Questions with Explanations, http://arxiv.org/abs/2402.15062, (2024)

    2024 arXiv

  29. [37]

    Brahman, F., Kumar, S., Balachandran, V., Dasigi, P., Pyatkin, V., Ravichander, A., Wiegreffe, S., Dziri, N., Chandu, K., Hessel, J.: The art of saying no: Contextual Evaluation of AI image understanding on German knowledge 17 noncompliance in language models. Adv. Neural Inf....

    2024

  30. [38]

    Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Huang, J., Zhang, J., Ning, M., Yuan, L.: MoE-LLaVA: Mixture of Experts for Large Vision-Language Models, (2024)

    2024

  31. [39]

    Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S.: AWQ: Activation -aware Weight Quantization for LLM Compression and Acceleration, (2023)

    2023

  32. [40]

    In: 38 Annual Conference on Neural Information Processing Systems (2024)

    Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., Lu, J.: Q-VLM: Post-training Quantization for Large Vision -Language Models. In: 38 Annual Conference on Neural Information Processing Systems (2024)

    2024

  33. [41]

    Zhu, L., Wang, X., Wang, X.: JudgeLM: Fine -tuned Large Language Models are Scalable Judges, (2023)

    2023

  34. [42]

    Huang, H., Qu, Y., Bu, X., Zhou, H., Liu, J., Yang, M., Xu, B., Zhao, T.: An Empirical Study of LLM -as-a-Judge for LLM Evaluation: Fine -tuned Judge Model is not a General Substitute for GPT-4, http://arxiv.org/abs/2403.02839, (2024)

    2024 arXiv

  35. [43]

    Raina, V., Liusie, A., Gales, M.: Is LLM -as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment, (2024)

    2024

  36. [44]

    Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P.: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, http://arxiv.org/abs/2404.18796, (2024)

    2024 arXiv

  37. [45]

    ArXiv Prepr

    Shi, L., Ma, C., Liang, W., Ma, W., Vosoughi, S.: Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. ArXiv Prepr. ArXiv240607791. (2024)

    2024

  38. [46]

    Alibaba: Qwen3: Think Deeper, Act Faster, https://qwenlm.github.io/blog/qwen3/, last accessed 2025/05/27

    2025

  39. [47]

    Xu, G., Jin, P., Li, H., Song, Y., Sun, L., Yuan, L.: LLaVA -CoT: Let Vision Language Models Reason Step-by-Step, http://arxiv.org/abs/2411.10440, (2025)

    2025 arXiv

  40. [48]

    https://doi.org/10.48550/arXiv.2504.07491

    Team K imi: Kimi -VL Technical Report, http://arxiv.org/abs/2504.07491, (2025). https://doi.org/10.48550/arXiv.2504.07491

    2025 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.