Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Synthetic Document Question Answering in Hungarian

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models underperform on Hungarian document VQA by ~30% versus English, and finetuning on synthetic Hungarian data recovers 7.2% accuracy.

desk verdict Useful Hungarian DocVQA datasets, but the headline finetuning claim is not validated on human-verified data and should not be trusted as-is. read the letter →

arxiv 2505.23008 v1 pith:YY5MUQ7N submitted 2025-05-29 cs.CV cs.AIcs.CL

Jonathan Li , Zoltan Csaki , Nidhi Hiremath , Etash Guha , Fenglu Hong , Edward Ma , Urmish Thakker This is my paper
classification cs.CVcs.AIcs.CL
keywords HungariandocumentVQAsyntheticquestionansweringmultilingualvision-languagemodelOCRtrainingdatadatasetcurationfinetuningLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that modern visual language models suffer a roughly 30% accuracy drop when answering questions about Hungarian documents instead of English ones, and that the gap can be partly closed with synthetic data. It introduces HuDocVQA, a large synthetic question-answer dataset built by having an LLM generate questions from OCR-extracted Hungarian pages, plus HuDocVQA-manual, a 54-example human-verified benchmark, and HuCCPDF, a corpus of 113k Hungarian PDF pages with transcriptions. The paper reports that finetuning Llama 3.2 11B Instruct on a mixture of general visual instruction data, HuDocVQA, and HuCCPDF improves accuracy on the synthetic test set from 0.332 to 0.404. It also claims the synthetic data reaches human-level quality, evidenced by a high correlation between model scores on the synthetic test set and the manual set.

What carries the argument

The load-bearing component is the synthetic data generator: for each Hungarian PDF page, text is extracted twice with pytesseract OCR and PyMuPDF, and Llama 3.3 70B is prompted one-shot with a human-written Hungarian QA pair to produce four new QA pairs per image. Four heuristic filters then remove examples with too-short text, low question-text overlap, wrong language, or duplicate questions. The other essential piece is HuCCPDF, 113k Hungarian pages formatted as OCR-forcing tasks, which is what drives the finetuning gain when added to the SFT mixture.

What would settle it

Take a random sample of about 200 Hungarian PDF pages, have native speakers write QA pairs, and compare model scores on this manual set with scores on the synthetic test set; if the Pearson correlation falls well below the claimed 0.986 or the rank order of models changes, the equivalence claim fails. An independent human audit of the LLM judge's verdicts on a sample of responses would also test the accuracy numbers directly.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that multilingual document VQA remains far behind English DocVQA: four leading VLMs score around 0.5–0.7 on Hungarian sets versus 0.8–0.96 on DocVQA. The central empirical result is that a scalable synthetic data pipeline can produce Hungarian document QA data that, when combined with Hungarian OCR data as a training signal, yields a +7.2% absolute accuracy improvement on the HuDocVQA test set for Llama 3.2 11B Instruct. The paper further claims that the synthetic test set is a faithful proxy for human annotation, based on a 0.986 Pearson correlation between model accuracies on HuDocVQA and HuDocVQA-manual.

Load-bearing premise

The 54-page human-verified set is treated as reliable ground truth for validating both the synthetic test set and the LLM-as-a-judge, despite its very small size.

Editorial extensions

If this is right

  • The same synthetic pipeline can be applied to other mid-resource languages with only a few human-written exemplars.
  • Adding OCR transcription tasks to visual instruction tuning improves document VQA in the target language.
  • Synthetic document VQA test sets can track human-annotated quality closely enough to replace manual benchmarks at scale.
  • Public release of the datasets lets other teams train and evaluate Hungarian document VQA without collecting new data.
  • The persistence of the Hungarian–English gap suggests current VLM multilingual OCR and layout understanding is far from saturated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equivalence claim between synthetic and manual data rests on only 54 manual examples and four model pairs, so the correlation statistic is fragile; a larger manual set could change the conclusion.
  • The LLM-as-a-judge is itself a Hungarian-competent model, so if the judge shares the tested models' weaknesses in Hungarian, part of the measured accuracy gap may be a measurement artifact.
  • The OCR-task boost hints that a dedicated Hungarian OCR model, or more diverse layout data, may improve VQA further than the observed +7.2%.
  • The pipeline filters out low-text pages, so performance on sparse or image-heavy Hungarian documents may be worse than the benchmark suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces three Hungarian document datasets derived from Common Crawl PDFs: HuDocVQA-manual (54 human-verified QA pairs), HuDocVQA (62k synthetic QA pairs generated by Llama 3.3 70B with heuristic filters), and HuCCPDF (113k pages of Hungarian PDF text/transcription). The authors benchmark several VLMs, reporting a roughly 30% accuracy drop on Hungarian document VQA relative to English DocVQA, and show that finetuning Llama 3.2 11B Instruct on a mixture including HuDocVQA and HuCCPDF OCR data improves accuracy on the synthetic HuDocVQA test set by +7.2% (0.332 to 0.404, Table 4), measured with an LLM-as-a-judge (Llama 3.1 405B). They claim the synthetic pipeline achieves human-level quality based on a model-level Pearson correlation of 0.986 across four models on the 54-example manual set.

Significance. If the claims hold, the paper offers a practical, scalable recipe for building document VQA datasets in a lower-resource language, and the released datasets (HuDocVQA, HuCCPDF) would be useful resources for Hungarian OCR and document understanding. The finding that strong VLMs degrade substantially on Hungarian document VQA is a clear, falsifiable observation. The authors also deserve credit for releasing datasets/code, for detailed filtering statistics, and for being transparent about hyperparameters and ablations.

major comments (3)
  1. [Sec. 3.1, Tables 3-4 and Sec. 7] The claim that HuDocVQA matches human-level quality (Sec. 3.1) rests on a Pearson correlation r=0.986 computed from only four model score pairs (n=4) between HuDocVQA-manual and HuDocVQA, and the central finetuning claim in Table 4 (+7.2%, from 0.332 to 0.404) is evaluated exclusively on the synthetic HuDocVQA test set with an LLM-as-a-judge. The 54-example manual set is never used to score the final finetuned models; the only manual-set finetuning result (Table 9) shows degradation from 0.481 to 0.426 on HuDocVQA-manual when training on filtered HuDocVQA. A model-level correlation across four models does not validate that the LLM judge tracks human correctness for finetuned models, and the single available human-verified signal points in the opposite direction of the headline gain. The authors should either evaluate the finetuned models on HuDocVQA-manual, or substantially temper the claim that the synthetic test set is interchangeable with human-annotated data for finetuning evaluation.
  2. [Table 4 and Sec. 10] All finetuning results in Table 4 are single runs without error bars, multiple seeds, or significance testing. Given that the reported differences include small margins (e.g., +1.9% between unfiltered and filtered HuDocVQA in Table 9, and the difference between 'SFT mixture' at 0.303 and baseline at 0.332 is itself a negative result), it is unclear whether the +7.2% gain is stable across training runs or could be within run-to-run variance. At minimum, the paper should report variance across seeds or provide a significance test; without this, the headline finetuning improvement is not fully substantiated.
  3. [Sec. 3.2, Tables 4 and 9, Sec. 7] The LLM-as-a-judge is validated only through selected examples in Figures 3-4 and through an n=4 model-level correlation; no measured agreement rate between Llama 3.1 405B and human judgments on the 54 manual items is reported. Because the +7.2% claim is entirely dependent on this judge, the paper should report judge-human agreement (e.g., Cohen's kappa or accuracy on HuDocVQA-manual) and ideally judge agreement on finetuned model outputs. The current validation is insufficient to rule out that the reported gains reflect the judge preferring outputs that match the synthetic generator's distribution rather than genuine document understanding.
minor comments (5)
  1. [Sec. 3.3] The text says 'In Tab. 4, we ablate the amount of OCR data' but the ablation of OCR data scale actually appears in Table 4 rows 'SFT + 21k OCR' and 'SFT + 105k OCR', which is fine, but the cross-reference in Sec. 12.1 to Table 9 as the filtering ablation is clear; however, the paper should ensure all table references in the main text match the described experiments exactly, as Sec. 3.3's phrasing is slightly confusing.
  2. [Table 5 caption] The percentages in Table 5 are relative to the input dataset to each filter, which is stated, but the presentation would be clearer if the absolute number of QA pairs before and after each filter were also shown, since the cumulative effect (from 93,933 to 62,022) is not directly derivable from the listed percentages.
  3. [Sec. 14] The HuCCPDF collection section states 'we processed 25 at a time on CPU nodes with 200GB memory' but it is unclear whether '25' refers to WARC files or batches; also 'we downloaded 22 million PDFs' and 'filtered out 38,000 Hungarian PDFs' is slightly ambiguous because 'filtered out' could mean 'retained after filtering' as implied by 'resulting in 113,091 pages after filtering'. Please clarify the wording.
  4. [Table 10 caption] The caption says 'Model Merging Ablation' but compares 'Final Checkpoint' vs 'Merged Checkpoint' without defining how many checkpoints are averaged; the body text in Sec. 13 says the last 3 checkpoints, which is only in the appendix. Consider moving this detail into the caption or the main text.
  5. [References [22]] The claim that Hungarian is 'approximately the 17th highest resource language on the internet' is cited, but no further breakdown or justification is given; please provide the exact ranking or a more precise citation to the relevant resource.

Circularity Check

2 steps flagged · score 5.0 of 10

The +7.2% finetuning gain is measured only on synthetic Llama-generated HuDocVQA test data with a Llama-family judge, while the human-verified benchmark is dropped after a 4-model correlation and the one human-verified finetuning result shows the opposite trend.

  1. self definitional [Sec. 7 (LLM-as-a-Judge), with Sec. 2.2.1 and Table 4]
    "All accuracy measurements on all datasets are measured with LLM-as-a-Judge [34], using Llama 3.1 405B Instruct as the judge. ... All accuracy numbers are computed with LLM-as-a-Judge."

    The HuDocVQA test answers were generated by LLaMA 3.3 70B (Sec. 2.2.1), and the finetuning mixture includes HuDocVQA training data from the same generation pipeline. Scoring the finetuned model with Llama 3.1 405B on gold answers produced by a close relative of the same model family means the reported +7.2% gain measures agreement with the generator/judge family rather than with human-verified correctness. The human-verified set is not used to score the finetuned models.

  2. other [Sec. 3.1]
    "the scores have a Pearson coefficient of 0.986 and a p-value of 0.01, suggesting that our synthetic data pipeline achieves comparable quality as human annotation. Given these results, we track accuracy for the HuDocVQA test set only in subsequent experiments."

    A model-level correlation on only four off-the-shelf VLMs is used as the license to drop the human-verified HuDocVQA-manual and treat the synthetic HuDocVQA test set as the sole benchmark. The finetuning data and the test data come from the same LLM generation pipeline, so the subsequent finetuning 'improvement' is partially self-referential. This is not confirmed by human labels: Table 9 shows that filtered HuDocVQA finetuning degrades HuDocVQA-manual accuracy from 0.481 to 0.426, the opposite of the headline trend.

full rationale

The paper is not wholly circular: it releases new datasets, compares against external DocVQA, and the finetuning improvement includes real OCR data from HuCCPDF. However, the central validation chain has a self-referential loop. HuDocVQA's gold answers are generated by LLaMA 3.3 70B, the judge is Llama 3.1 405B Instruct, and the finetuning data includes the same generator's outputs. The human-verified HuDocVQA-manual is used only for a coarse 4-model correlation (r=0.986, p=0.01) and then dropped; the only human-verified finetuning result, Table 9, shows degradation. Thus the headline +7.2% improvement is not anchored to human-verified correctness and may reflect distribution matching to the generator/judge family. This partial circularity warrants a score of 5.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The load-bearing choices are hand-set filtering thresholds, the LLM-as-a-judge assumption, and the very small manual validation set. These choices materially affect every reported accuracy number.

free parameters (5)
  • Text length filter threshold = 60 characters
    Chosen by hand to remove pages with too little text. Affects which pages enter HuDocVQA and thus all reported accuracy numbers.
  • N-gram overlap threshold = 12% overlap at n=4
    Hand-set threshold to filter questions unrelated to document text. Directly changes the composition of HuDocVQA.
  • FastText Hungarian probability threshold = 0.85
    Hand-set cutoff for accepting a PDF as Hungarian. Influences which documents are included in HuCCPDF and HuDocVQA.
  • Plaintext/Markdown word-count mismatch threshold = 50% difference
    Hand-set to discard low-quality or complex-table PDFs. Affects the final page set of HuCCPDF.
  • Generation temperature = 0.7
    Sampling temperature for Llama 3.3 70B QA generation. Affects the diversity and quality of synthetic questions.
assumptions (4)
  • domain assumption LLM-as-a-judge with Llama 3.1 405B is a reliable proxy for human correctness judgments in Hungarian document QA.
    Used for all accuracy measurements. Section 7 justifies this with a few examples, but no large-scale human correlation study is provided.
  • ad hoc to paper The 54-example HuDocVQA-manual set is representative of Hungarian document VQA.
    This small set is used to validate the synthetic test set and the judge. Its representativeness is assumed, not demonstrated.
  • domain assumption fastText reliably identifies Hungarian PDFs at probability >= 0.85.
    This is the primary language filter for all three datasets. No validation on held-out Hungarian documents is reported.
  • domain assumption Common Crawl PDFs passing the filters are representative of Hungarian documents.
    The datasets are built from CC-MAIN-2021-17. The paper assumes this crawl subset covers the variety of Hungarian documents needed for VQA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Document Question Answering in Hungarian." pith.science (2026). https://pith.science/paper/YY5MUQ7N

@misc{pith2026250523008,
  author       = {Pith},
  title        = {Pith review of: Synthetic Document Question Answering in Hungarian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YY5MUQ7N}},
  note         = {Machine review of arXiv:2505.23008}
}
read the original abstract

Modern VLMs have achieved near-saturation accuracy in English document visual question-answering (VQA). However, this task remains challenging in lower resource languages due to a dearth of suitable training and evaluation data. In this paper we present scalable methods for curating such datasets by focusing on Hungarian, approximately the 17th highest resource language on the internet. Specifically, we present HuDocVQA and HuDocVQA-manual, document VQA datasets that modern VLMs significantly underperform on compared to English DocVQA. HuDocVQA-manual is a small manually curated dataset based on Hungarian documents from Common Crawl, while HuDocVQA is a larger synthetically generated VQA data set from the same source. We apply multiple rounds of quality filtering and deduplication to HuDocVQA in order to match human-level quality in this dataset. We also present HuCCPDF, a dataset of 117k pages from Hungarian Common Crawl PDFs along with their transcriptions, which can be used for training a model for Hungarian OCR. To validate the quality of our datasets, we show how finetuning on a mixture of these datasets can improve accuracy on HuDocVQA for Llama 3.2 11B Instruct by +7.2%. Our datasets and code will be released to the public to foster further research in multilingual DocVQA.

Figures

Figures reproduced from arXiv: 2505.23008 by the authors.

Figure 1
Figure 1. An example question-answer pair from HuDocVQA-manual 1. Introduction As large language models (LLMs) quickly reach saturation and human-level performance on a host of traditional NLP tasks, vision-language models (VLMs) are quickly reaching saturation on tradi￾tional multimodal benchmarks as well. One such benchmark is DocVQA [20], where human-level performance is estimated at 94.36% exact-match accuracy and 0.981 A… view at source ↗
Figure 2
Figure 2. A diagram of our synthetic data pipeline [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Examples from HuDocVQA-manual where ANLS fails to correctly measure model response accuracy in [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Examples from HuDocVQA-manual where GPT-4o would rate models more harshly than Llama 3.1 405B [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The Hungarian system prompt we provide to Llama 3.3 70B Instruct for synthetic QA generation. English [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LV-ROVER-MLT: Low-Resource Maltese OCR by Synthetic Fine-Tuning and Multi-Stream Arbitration

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Synthetic-data-trained Tesseract ensemble with ROVER voting and post-processing cuts Maltese OCR CER from 0.0234 to 0.00700 on a competition dev set.

Reference graph

Works this paper leans on

46 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mistral small 3.1.https : / / mistral.ai/news/mistral-small-3-1,

    Mistral AI. Mistral small 3.1.https : / / mistral.ai/news/mistral-small-3-1,

    work page

  2. [2]

    Artifex Software

    Inc. Artifex Software. Pymupdf4llm.https:// 4 pymupdf.readthedocs.io/en/latest/ pymupdf4llm/, 2025. 3

    work page 2025

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1

    arXiv 2025

  4. [4]

    Enriching word vectors with subword information, 2017

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. 3

    work page 2017

  5. [5]

    Maxm: Towards mul- tilingual visual question answering.arXiv preprint arXiv:2209.05401, 2022

    Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. Maxm: Towards mul- tilingual visual question answering.arXiv preprint arXiv:2209.05401, 2022. 2, 1

    arXiv 2022

  6. [6]

    Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. 1

    arXiv 2024

  7. [7]

    The warc format 1.1.https : / / iipc

    International Internet Preservation Consortium. The warc format 1.1.https : / / iipc . github . io / warc - specifications / specifications / warc - format / warc - 1.1/, 2025. 3

    work page 2025

  8. [8]

    Common crawl - open repository of web crawl data.https://commoncrawl

    Common Crawl. Common crawl - open repository of web crawl data.https://commoncrawl. org/, 2025. 1, 3

    work page 2025

Show all 46 references
  1. [9]

    langdetect: Port of google’s language-detection library to python, 2018

    Michal Danilak. langdetect: Port of google’s language-detection library to python, 2018. 3

    2018

  2. [10]

    Exams-v: A multi-discipline multilingual multimodal exam benchmark for eval- uating vision language models, 2024

    Rocktim Jyoti Das, Simeon Emilov Hristov, Hao- nan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multimodal exam benchmark for eval- uating vision language models, 2024. 2

    2024

  3. [11]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017. 1, 3

    2017

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

  5. [13]

    pytesser- act: A python wrapper for google’s tesseract-ocr

    Samuel Hoffstaetter and contributors. pytesser- act: A python wrapper for google’s tesseract-ocr. https : / / github . com / h / pytesseract,

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 2

    2021

  7. [15]

    Llama3-gaja-hindi-8b-v0.1,

    Adithya S Kolavi. Llama3-gaja-hindi-8b-v0.1,

  8. [16]

    Building and better understand- ing vision-language models: insights and future di- rections, 2024

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understand- ing vision-language models: insights and future di- rections, 2024. 4, 2, 6

    2024

  9. [17]

    What matters when building vision-language models?, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024. 4, 2, 6

    2024

  10. [18]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 1

    2024

  11. [19]

    Anwer, Tim Baldwin, Michael Felsberg, and Fahad S

    Muhammad Maaz, Hanoona Rasheed, Abdelrah- man Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. Palo: A polyglot large mul- timodal model for 5b people, 2024. 1

    2024

  12. [20]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021. 1, 3

    2021

  13. [21]

    Jorj X. McKie. and contributors. Pymupdf, 2025. 2, 3

    2025

  14. [22]

    Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.arXiv preprint arXiv:2309.09400, 2023

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.arXiv preprint arXiv:2309.09400, 2023. 1, 3

    2023 arXiv

  15. [23]

    JDocQA: Japanese document question answering dataset for generative language models

    Eri Onami, Shuhei Kurita, Taiki Miyanishi, and Taro Watanabe. JDocQA: Japanese document question answering dataset for generative language models. InProceedings of the 2024 Joint In- ternational Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC...

    2024

  16. [24]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Car- ney, Alex Chow, Alex Kirillov, Alex Nichol, Al...

    2024

  17. [25]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, 8 Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang...

    2025

  18. [26]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 1

    2021

  19. [27]

    multilingual

    Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. Translationese as a language in “multilingual” NMT. InProceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 7737–7746, Online, 2020. Association for Computational Linguistics. 2

    2020

  20. [28]

    Crossmodal-3600: A massively multilingual multimodal evaluation dataset.arXiv preprint arXiv:2205.12522, 2022

    Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset.arXiv preprint arXiv:2205.12522, 2022. 1

    2022 arXiv

  21. [29]

    Nllb-clip – train performant multilingual image retrieval model on a budget,

    Alexander Visheratin. Nllb-clip – train performant multilingual image retrieval model on a budget,

  22. [30]

    Llama3.1-8b-chinese-chat,

    Shenzhi Wang, Yaowei Zheng, Guoyin Wang, Shiji Song, and Gao Huang. Llama3.1-8b-chinese-chat,

  23. [31]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of mul- tiple fine-tuned models improves accuracy wi...

    2022

  24. [32]

    mt5: A massively mul- tilingual pre-trained text-to-text transformer, 2021

    Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively mul- tilingual pre-trained text-to-text transformer, 2021. 1

    2021

  25. [33]

    Pangea: A fully open multilingual multimodal llm for 39 languages, 2024

    Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathya- narayanan Ramamoorthy, and Graham Neubig. Pangea: A fully open multilingual multimodal llm for 39 languages, 2024. 2

    2024

  26. [34]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 1

    2023

  27. [35]

    Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024. 2 9 Synthetic Document Question Answering in Hungarian Supplementary Material

    2024

  28. [36]

    DocVQA DocVQA [20] is the seminal dataset and bench- mark on the Document Visual Question Answer- ing task

    Related Works 5.1. DocVQA DocVQA [20] is the seminal dataset and bench- mark on the Document Visual Question Answer- ing task. Since the release of LLaV A-NeXT [18], most multimodal model releases have bench- marked their OCR abilities on this dataset. At the time of writing, ...

  29. [37]

    Number of synthetic QAs filtered out for each of the filters applied Tab

    Filtering Statistics Filter # QAs filtered text length>60 856 (0.9%) n-gram overlap 7,931 (8.5%) langdetect 2,622 (3.1%) deduplication 20,393 (24.7%) Table 5. Number of synthetic QAs filtered out for each of the filters applied Tab. 5 shows the numbers of question-answer pairs...

  30. [38]

    While ANLS serves as a fine metric for English DocVQA, in early exper- iments with HuDocVQA-manual that it often went 1 against human judgment on model response cor- rectness

    ANLS vs LLM-as-a-Judge All accuracy measurements on all datasets are mea- sured with LLM-as-a-Judge [34], using Llama 3.1 405B Instruct as the judge. While ANLS serves as a fine metric for English DocVQA, in early exper- iments with HuDocVQA-manual that it often went 1 against...

  31. [39]

    6 shows 3 documents from HuDocVQA- manual and their corresponding human-annotated questions and answers

    Dataset Examples Tab. 6 shows 3 documents from HuDocVQA- manual and their corresponding human-annotated questions and answers. These 3 examples were used as few-shot exemplars to Llama 3.3 70B In- struct for generating HuDocVQA

  32. [40]

    Kérdés:

    Synthetic QA Generation Details and Prompt Format When generating questions and answers with Llama 3.3 70B Instruct, we apply the system prompt in Fig. 5. Then, after sampling a human- written input as a (text, question, answer) triple, we format a one-shot prompt as follows: ...

  33. [41]

    Finetuning Details & Hyperparame- ters For all finetuning experiments, we train for 1 epoch and merge the original checkpoint with the last 3 checkpoints. Tab. 7 shows other hyperparameters such as learning rate, batch size, etc. For all train- ing experiments, we utilize Llam...

  34. [42]

    8 outlines all datasets comprising our SFT mixture in Tab

    SFT Mixture Tab. 8 outlines all datasets comprising our SFT mixture in Tab. 4. As The Cauldron [17], Doc- Matix [16], and LAION-COCO-NLLB [29] each comprise several hundreds of thousands of exam- ples, we subsample each dataset to 100k. For LAION-COCO-NLLB, we only subsample c...

  35. [43]

    HuDocVQA Filtering Ablation In Tab

    Ablations 12.1. HuDocVQA Filtering Ablation In Tab. 9 we finetune on HuDocVQA with and without the quality filters in Sec. 2.2.2. Al- though training on both datasets lead to degrada- tion, adding the quality filters leads to an improve- ment of 1.9%

  36. [44]

    10 compares the accuracy of the final check- point versus a merged checkpoint for each training experiment on Sec

    Model Merging Ablation Tab. 10 compares the accuracy of the final check- point versus a merged checkpoint for each training experiment on Sec. 3.2. We apply simple model merging from Wortsman et al. [31] to Llama 3.1 11B Instruct and the last 3 checkpoints of each training run...

    2000

  37. [45]

    HuCCPDF Collection System De- tails Of the 3750 W ARC files in CC 2021-17 we pro- cessed, we processed 25 at a time on CPU nodes with 200GB memory, with each W ARC taking ap- proximately 1.5 hours. In total, we downloaded 22 million PDFs, and filtered out 38,000 Hungarian PDFs...

    2021

  38. [46]

    05 of these regulations

    számú almelléklete tartalmazza.Provisions regarding personal data are contained in sub-appendix no. 05 of these regulations. 5 Peak Learning Rate 1e-6 LR Schedule Cosine Decay LR Warmup Ratio 0.1 Batch Size 32 Epochs 1 Table 7. Finetuning Hyperparameters Dataset # Single- Imag...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.