Pith.

Computation and Language · arXiv:2611.16500 v1 · submitted 12 Nov 2026 · 31 references

EvenVLM: A Text and Multimodal Dataset for the Even Language with Vision-Language Fine-Tuning

Daniil Makovetskii, Ivan Bondarenko

The paper says

“We present EvenVLM, the first text and multimodal (VQA) dataset for Even (Lamut), an endangered Northern Tungusic language spoken by approximately 6,800 people in Siberia.”

more of the abstract

Our dataset comprises 50,521 structured Even-Russian entries extracted from dictionary and phrasebook sources via GPT-4o Mini-based parsing, from which we synthetically generate 107,543 training examples spanning translation, grammar, reading comprehension, and semantic classification. To address tokenization inefficiency on Even's agglutinative morphology, we propose a three-step tokenizer expansion algorithm that adds Even-specific graphemes (eng, oe, o-umlaut), whitelisted vocabulary, and BPE merges, reducing average tokens per word by 27% (4.99 to 3.64). Using LoRA fine-tuning of Qwen3-VL-4B, we adapt the model for text tasks (0.53% trainable parameters, rank 32) and separately for visual question answering (rank 64, frozen vision encoder). Our best configuration achieves 11.4% exact match and a 13.6 weighted score on text benchmarks, 0.1283 chrF for sentence translation, and 65% accuracy on VQA (up from 0% for the base model and text-only fine-tuning). We release our datasets, evaluation benchmarks, and training pipeline to support further research on Even and other low-resource languages.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.

The paper's main value is the dataset and pipeline for Even; the evaluation is honest about its limits, but the missing specification of test set independence is a genuine gap that should be addressed. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 31 references say.

The paper says, at 7.1 Benchmark Suite and Metrics

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The 100-example text evaluation set is described in terms of task composition, but the paper does not specify whether these examples are drawn from dictionary entries that were held out from the 50,521-entry corpus (Section 3.3) used to generate the 107,543 synthetic training examples.

more of objection 1

If the test examples originate from the same dictionary entries or from the same templated transformations, then the reported exact match and weighted score may partially reflect pattern memorization rather than true language generalization. The authors acknowledge small test size (sentence b98) but do not address this potential train-test overlap. Since the central empirical claim for text tasks depends on an independent held-out set, the authors must clarify the source of the test examples and, if there is overlap, demonstrate that the metrics are not inflated by leakage. This is a load-bearing concern for the validity of Table 1.

The paper says, at 4.1 Template-Based Generation

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The training data is synthetically generated from dictionary entries using templated transformations (declension, negation, translation, etc.). However, the text evaluation set (Section 7.1) includes tasks such as grammar and translation that could be generated by the same templates. Even if the specific test tokens differ, if the test examples are constructed from held-out entries, that should be stated.

more of objection 2

If they are not, the paper should explain how the test set was constructed to avoid overlap and why the reported results are not affected by memorization of template patterns.

6 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 6

The abstract states 'reducing average tokens per word by 27% (4.99 to 3.64)' which is clear. However, the improvement is from 4.99 to 3.64, which is a 27% reduction, but the median goes from 5 to 4 (20% reduction). Please verify the median percentage for consistency.

The paper says, at Table 1

Pith says: minor objection 2 of 6

The baseline configuration 'Base + Tokenizer' shows a drop in weighted score from 2.1 to 0.7. The text says tokenizer expansion alone adds noise because new tokens have randomly initialized embeddings. This is plausible, but the drop is large (67% relative). Consider adding a brief explanation of why the drop is so substantial, e.g., that the randomly initialized embeddings disrupt previously working token sequences.

show 4 more

The paper says, at 6.2 Stage 1: Text Fine-Tuning

Pith says: minor objection 3 of 6

The paper states that LoRA rank r=32 fits within the 32GB GPU budget. However, the VQA stage uses r=64 with a frozen vision encoder. The justification for different ranks is given, but it would be helpful to state the peak memory usage for each configuration to support the claim that r=32 was necessary for Stage 1.

The paper says, at 5.1 Expansion Algorithm, Step 2

Pith says: minor objection 4 of 6

The maximum of 300 tokens for whitelist + BPE merging is described. It is not entirely clear whether the 209 whitelist words and 101 VQA answer words are counted against the 300 limit. Since 209+101=310 exceeds 300, presumably not all whitelist words were added, or the limit applies to BPE merges separately. Please clarify the counting methodology.

The paper says, at Figure 2

Pith says: minor objection 5 of 6

The training and evaluation loss curves show eval loss at 1.179 at epoch 6, but the curve appears to be still decreasing slowly. Consider extending training to verify whether the validation loss has truly converged or if additional epochs may yield further improvement.

The paper says, at 7.3 Qualitative Observations

Pith says: minor objection 6 of 6

Table 3 shows a reading comprehension example where the model outputs 'бэй тэгэттэн' instead of the target 'тэгэттэн'. This is classified as 'wrong', but the root 'тэгэттэн' is correct and the extra 'бэй' is a semantic substitution. The weighted Levenshtein score might give partial credit here; the qualitative description could note this nuance.

What would settle it

Request that the authors explicitly state the source of each text evaluation example: whether they come from dictionary entries not used in training data generation, and whether any test example appears in the training set. If overlap exists, recompute metrics after excluding overlapping examples. Alternatively, provide the entry IDs for both training and test sets to allow independent verification.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 53 s · 10,772 tokens read, 3,686 written
  3. Editor wrote the verdict above 24 s · 11,502 tokens read, 2,441 written
  4. Referee major revision: 2 major objections, 6 minor 21 s · 11,081 tokens read, 2,225 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hi [Colleague],

Here's my take on EvenVLM.

What's actually new: this is the first open text and VQA dataset for Even, an endangered Tungusic language. The authors release 50K dictionary entries expanded to 107K synthetic examples, a tokenizer expansion algorithm that reduces tokens per word by 27%, and the full fine-tuning pipeline. That's a concrete resource contribution to a language that previously had no NLP dataset. The work is reproducible (code and data released), and the authors are refreshingly transparent about limitations—small test sets, synthetic data quality, dialect skew, incomplete convergence on VQA.

What it does well: the pipeline design is straightforward and could be adapted to other agglutinative languages with similar resources. The tokenizer expansion is sensible: add missing graphemes, whitelist frequent words, then do targeted BPE merging. The ablation confirms that each component (tokenizer expansion, LoRA) helps, though the absolute gains are modest. The VQA result—65% accuracy on 20 held-out images—shows that 1,908 synthetic examples suffice for basic visual-lexical mapping, which is non-trivial for a language where the base model outputs nothing.

Where the soft spots are: the stress-test concern about text evaluation set independence is real and not addressed. The paper describes 100 text examples across four tasks but never says where these examples came from. If they were drawn from the same dictionary entries that generated the training data, then the reported 11.4% exact match and 0.1283 chrF could partially reflect pattern memorization rather than true generalization. The authors acknowledge small test size (n=100) and synthetic data quality, but missing train-test separation is a separate issue. That said, the numbers are low enough that the overall conclusion—the model learns something from the pipeline—still holds, but the exact magnitude is uncertain. The VQA evaluation is less affected because it uses separate images and annotations.

Who this is for: researchers in low-resource endangered language NLP, especially those working with agglutinative languages that have at least a dictionary. The paper doesn't break new methodological ground (LoRA, BPE, template-based synthesis are standard), but the application and reproducible pipeline are valuable.

Recommendation: the paper deserves a serious referee and should be accepted after the authors clarify how the text test set was constructed and whether it overlaps with the source data. That's a minor revision, not a rejection.

The strongest objection

The paper's text evaluation (Section 7.1) consists of 100 examples spanning reading comprehension, classification, grammar, and translation. However, the paper does not specify whether these examples are drawn from dictionary entries that were held out from the 50,521-entry corpus (Section 3.3) used to generate the 107,543 synthetic training examples (Section 4.1). If the test examples originate from the same dictionary entries or from the same templated transformations, then the reported 11.4% exact match and 13.6 weighted score may partially reflect pattern memorization rather than true language generalization. The authors acknowledge small test size and synthetic data quality (sentences b97, b98) but do not address this potential train-test overlap. Since the central empirical claim for text tasks depends on the assumption of an independent held-out set, this is a load-bearing concern. The VQA evaluation (20 examples, separate images and annotations) is less affected.

The reader's verdict ACCEPT with high confidence acknowledges the honest reporting of limitations. The potential train-test overlap is not explicitly discussed but is a standard concern in dataset creation. The paper's primary contribution is the open dataset and pipeline, not the strong empirical results; thus, even if the text metrics are partially inflated, the resource contribution remains valuable. The VQA evaluation is unaffected because it uses separate images and held-out annotations. Therefore, the verdict need not change, but the concern should be documented for future users of the evaluation benchmarks.

The referee's report

If the empirical claims hold, this work provides the first open NLP and VLM dataset for Even, a language with fewer than 10,000 speakers, and demonstrates a reproducible pipeline (tokenizer expansion + LoRA fine-tuning) that could transfer to other severely low-resource agglutinative languages. The release of data and code is a concrete contribution to linguistic diversity in AI. The paper fills a clear gap, as VLMs for languages with under 10,000 speakers are virtually absent from the literature. The tokenizer adaptation method is simple but effective and documented in sufficient detail to be replicated.

Pith's reading of the paper (a machine's summary, not the authors')

EvenVLM is the first text and multimodal dataset for Even, an endangered Siberian language with about 6,800 speakers. The authors take a 4.4-billion-parameter vision-language model (Qwen3-VL-4B) that starts with zero ability to produce Even and adapt it through a three-part pipeline: they extract structured entries from a dictionary and phrasebook via GPT-4o Mini, expand those into 107,543 synthetic training examples; they modify the tokenizer to handle Even’s agglutinative morphology, cutting average tokens per word by 27%; and they apply two-stage LoRA fine-tuning. The resulting model achieves 65% accuracy on visual question answering (from 0% baseline) and measurable scores on translation, grammar, and classification tasks. The work matters because it shows a reproducible path for bringing modern AI tools to languages that have virtually no prior digital resources.

What carries the argument. Three mechanisms carry the argument. 1) Template-based synthetic data generation: from 50,521 structured Even-Russian dictionary entries, the authors create 107,543 training examples across translation, grammar, reading comprehension, and classification using deterministic linguistic templates, avoiding the cold-start problem that would plague self-training approaches for a language where base models score 0%. 2) Three-step tokenizer expansion: add three missing Cyrillic graphemes (ӈ, ө, ӫ) as single tokens, whitelist 209 high-frequency Even words, and run BPE merging on the full text corpus—adding 303 tokens and reducing average tokens per word from 4.99 to 3.64, a 27% reduction in fragmentation. 3) Two-stage LoRA fine-tuning: Stage 1 adapts the text decoder with rank-32 LoRA (0.53% trainable parameters) on the full synthetic text dataset; Stage 2 applies a higher-rank (64) LoRA to seven projection matrices for VQA, with the vision encoder frozen, overcoming the multimodal cold start that leaves text-only fine-tuning at 0% VQA accuracy.

Core claim. The core claim is that a standard vision-language model can be made functional for an extremely low-resource agglutinating language through a combination of template-based synthetic data generation, specialized tokenizer expansion, and parameter-efficient fine-tuning. The discovery is not that the model performs at human level—it clearly does not, with exact match at 11.4% on text tasks—but that the zero-shot barrier (0% in all metrics) can be crossed with a relatively small and reproducible investment of data and compute. The VQA accuracy of 65% on a 20-example test set, up from 0%, demonstrates that even incomplete convergence (loss falling only from 10.2 to 6.1) yields basic visual-lexical mapping. This shifts the question from 'can these models be applied?' to 'how much more data and validation are needed to make them reliable?'

The premise it rests on. The entire pipeline depends on the assumption that the synthetic training examples generated by template expansion from a single dictionary and phrasebook are sufficiently accurate and representative of the Even language to serve as reliable training signals, yet only one native speaker was consulted for verification.

If the paper is right (Pith's inferences, not the authors')

The synthetic data pipeline can be replicated for other low-resource, agglutinative languages that have a dictionary and phrasebook, offering a deterministic alternative to self-training approaches that require a functional teacher model.

The tokenizer expansion algorithm is language-agnostic and could be applied to any tokenizer serving a language with unrepresented graphemes or high morphological productivity, reducing token fragmentation and improving model efficiency.

The two-stage LoRA strategy demonstrates that multimodal capability for a new language can be acquired with as few as 1,908 VQA examples, suggesting that the data bottleneck for vision-language adaptation in low-resource languages may be less severe than often assumed.

All resources released openly (datasets, benchmarks, pipeline) lower the barrier for future work on Even, and the approach is explicitly designed to extend to Evenki, a related language, enabling comparative studies across Northern Tungusic languages.

Even if the absolute metric values are modest, the paper provides a concrete lower bound and a baseline for future improvements; any subsequent work on Even or similar languages can now compare against these first published numbers.

The reliance on a single native speaker for linguistic validation is a significant vulnerability: the synthetic templates and extracted entries may contain dialectal biases or systematic errors that only multi-speaker checking would reveal; extending the validation pool is a high-impact next step.

The evaluation set sizes (n=20 for VQA, n=100 for text) are acknowledged as too small, yielding 95% confidence intervals roughly 43%–82% for the VQA result; future work should treat the reported numbers as pilot estimates rather than established performances, and re-evaluation with larger sets may revise both the floor and the ceiling.

The template-based generation excludes naturalistic sentence variation; a model trained only on template data may fail on naturally produced Even speech or writing, where word order, idiom, and zero morphology differ from constructed examples. A real-world corpus collection (if feasible) would be a necessary complement.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We present EvenVLM, the first text and multimodal (VQA) dataset for Even (Lamut), an endangered Northern Tungusic language spoken by approximately 6,800 people in Siberia. Our dataset comprises 50,521 structured Even-Russian entries extracted from dictionary and phrasebook sources via GPT-4o Mini-based parsing, from which we synthetically generate 107,543 training examples spanning translation, grammar, reading comprehension, and semantic classification. To address tokenization inefficiency on Even's agglutinative morphology, we propose a three-step tokenizer expansion algorithm that adds Even-specific graphemes (eng, oe, o-umlaut), whitelisted vocabulary, and BPE merges, reducing average tokens per word by 27% (4.99 to 3.64). Using LoRA fine-tuning of Qwen3-VL-4B, we adapt the model for text tasks (0.53% trainable parameters, rank 32) and separately for visual question answering (rank 64, frozen vision encoder). Our best configuration achieves 11.4% exact match and a 13.6 weighted score on text benchmarks, 0.1283 chrF for sentence translation, and 65% accuracy on VQA (up from 0% for the base model and text-only fine-tuning). We release our datasets, evaluation benchmarks, and training pipeline to support further research on Even and other low-resource languages.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.97

“Our best configuration achieves 11.4% exact match and a 13.6 weighted score on text benchmarks, 0.1283 chrF for sentence translation, and 65% accuracy on VQA (up from 0% for the base model and text-only fine-tuning).”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.96 · states limitation 0.33 · human participants 0.09 · controlled comparison 0.90 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.97

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We present EvenVLM, the first text and multimodal (VQA) dataset for Even (Lamut), an endangered Northern Tungusic language spoken by approximately 6,800 people in Siberia. contribution 1.00 0.88 0.08 asserted 1.00
Our dataset comprises 50,521 structured Even-Russian entries extracted from dictionary and phrasebook sources via GPT-4o Mini-based parsing, from which we synthetically generate 107,543 training examples spanning translation, grammar, reading comprehension, and semantic classification. method 0.84 0.96 0.05 asserted 1.00
To address tokenization inefficiency on Even's agglutinative morphology, we propose a three-step tokenizer expansion algorithm that adds Even-specific graphemes (eng, oe, o-umlaut), whitelisted vocabulary, and BPE merges, reducing average tokens per word by 27% (4.99 to 3.64). method 0.91 0.97 0.05 asserted 1.00
Using LoRA fine-tuning of Qwen3-VL-4B, we adapt the model for text tasks (0.53% trainable parameters, rank 32) and separately for visual question answering (rank 64, frozen vision encoder). method 1.00 0.96 0.04 asserted 0.99
Our best configuration achieves 11.4% exact match and a 13.6 weighted score on text benchmarks, 0.1283 chrF for sentence translation, and 65% accuracy on VQA (up from 0% for the base model and text-only fine-tuning). result 1.00 0.97 0.56 asserted 1.00
We release our datasets, evaluation benchmarks, and training pipeline to support further research on Even and other low-resource languages. contribution 0.61 0.86 0.09 asserted 0.98

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.87

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 5,374 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (31)
  1. [1] Lupa. Large Multimodal Models for Low-Resource Languages: A Survey , journal =. 2026 , doi = on Pith
  2. [2] Evens --- 2020--2021 Census Data , year = on Pith
  3. [3] Bai, S. and Cai, Y. and Chen, R. and others , title =. arXiv preprint , year = on Pith
  4. [4] Aishwarya, A. and Lu, J. and Antol, S. and others , title =. Proceedings of the IEEE International Conference on Computer Vision (ICCV) , year = on Pith
  5. [5] Dosovitskiy, A. and Beyer, L. and Kolesnikov, A. and others , title =. Proceedings of ICLR , year = on Pith
  6. [6] Mangrulkar, S. and Gugger, S. and Debut, L. and others , title =. 2022 , howpublished = on Pith
  7. [7] Hu, E. J. and Shen, Y. and Wallis, P. and others , title =. Proceedings of ICLR , year = on Pith
  8. [8] Kanyukova, U. V. and Levin, V. I. , title =. 2025 , pages = on Pith
  9. [9] Sharina, S. I. and Kuzmina, R. P. , title =. 2014 , pages = on Pith
  10. [10] Sennrich, R. and Haddow, B. and Birch, A. , title =. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year = on Pith
  11. [11] Popovi\'. Proceedings of the Tenth Workshop on Statistical Machine Translation , year = on Pith
  12. [12] Liu, H. and Li, C. and Wu, Q. and Lee, Y. J. , title =. Advances in Neural Information Processing Systems (NeurIPS) , year = on Pith
  13. [13] Chen, Z. and Wu, J. and Wang, W. and Su, W. and Chen, G. and others , title =. arXiv preprint , year = on Pith
  14. [14] Karamcheti, S. and Nair, S. and Chen, A. and Kollar, T. and Finn, C. and others , title =. arXiv preprint , year = on Pith
  15. [15] Conneau, A. and Khandelwal, K. and Goyal, N. and Chaudhary, V. and Wenzek, G. and others , title =. Proceedings of ACL , year = on Pith
  16. [16] Xue, L. and Constant, N. and Roberts, A. and Kale, M. and Al-Rfou, R. and others , title =. Proceedings of NAACL , year = on Pith
  17. [17] Lin, X. V. and Mihaylov, T. and Artetxe, M. and Wang, T. and others , title =. Proceedings of EMNLP , year = on Pith
  18. [18] Schwartz, L. and Chen, E. and others , title =. Proceedings of the 1st Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP) , year = on Pith
  19. [19] Mager, M. and Gutierrez-Vasques, X. and Sierra, G. and Meza, I. , title =. Proceedings of COLING , year = on Pith
  20. [20] Joshi, P. and Santy, S. and Budhiraja, A. and Bali, K. and Choudhury, M. , title =. Proceedings of ACL , year = on Pith
show 11 more
  1. [21] Rust, P. and Pfeiffer, J. and Vuli\'. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models , booktitle =. 2021 , url = on Pith
  2. [22] Clark, J. H. and Garrette, D. and Turc, I. and Wieting, J. , title =. Transactions of the Association for Computational Linguistics , volume =. 2022 , pages =. doi:10.1162/tacl_a_00448 , url = on Pithdoi:10.1162/tacl_a_00448
  3. [23] Lin, T.-Y. and Maire, M. and Belongie, S. and Hays, J. and Perona, P. and others , title =. Proceedings of ECCV , year = on Pith
  4. [24] Vedantam, R. and Lawrence Zitnick, C. and Parikh, D. , title =. Proceedings of CVPR , year = on Pith
  5. [25] Banerjee, S. and Lavie, A. , title =. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , year = on Pith
  6. [26] Malchukov, A. L. , title =. 1995 , address = on Pith
  7. [27] Papineni, K. and Roukos, S. and Ward, T. and Zhu, W.-J. , title =. Proc. of ACL , year = on Pith
  8. [28] Lin, C.-Y. , title =. Proc. of ACL Workshop , year = on Pith
  9. [29] Even: a language of. 2025 , url = on Pith
  10. [30] Ranaldi, L. and Ranaldi, F. and Pucci, G. , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) , year = on Pith
  11. [31] Ranaldi, L. and Pucci, G. , title =. Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , year = on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment