{"id":"7bd45caf-77d6-46ba-9132-ffecc1e3c79a","arxiv_id":"2611.16500","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"EvenVLM introduces the first textual and visual question answering dataset for Even, demonstrating that LoRA fine-tuning with a custom tokenizer expansion can effectively adapt a vision-language model to a severely low-resource agglutinative language.","lead":"We present EvenVLM, the first text and multimodal dataset for the critically endangered Even language, with 50,521 dictionary entries synthetically expanded to 107,543 training examples and 1,908 VQA examples. Using LoRA fine-tuning of Qwen3-VL-4B with a three-step tokenizer expansion, we achieve 65% VQA accuracy and reduce token fragmentation by 27%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Text evaluation test set may not be independent of training data, inflating reported metrics.","rationale":"The reader's verdict ACCEPT with high confidence acknowledges the honest reporting of limitations. The potential train-test overlap is not explicitly discussed but is a standard concern in dataset creation. The paper's primary contribution is the open dataset and pipeline, not the strong empirical results; thus, even if the text metrics are partially inflated, the resource contribution remains valuable. The VQA evaluation is unaffected because it uses separate images and held-out annotations. Therefore, the verdict need not change, but the concern should be documented for future users of the evaluation benchmarks.","tokens_in":10772,"tokens_out":3686,"duration_ms":53359,"concrete_test":"Request that the authors explicitly state the source of each text evaluation example: whether they come from dictionary entries not used in training data generation, and whether any test example appears in the training set. If overlap exists, recompute metrics after excluding overlapping examples. Alternatively, provide the entry IDs for both training and test sets to allow independent verification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's text evaluation (Section 7.1) consists of 100 examples spanning reading comprehension, classification, grammar, and translation. However, the paper does not specify whether these examples are drawn from dictionary entries that were held out from the 50,521-entry corpus (Section 3.3) used to generate the 107,543 synthetic training examples (Section 4.1). If the test examples originate from the same dictionary entries or from the same templated transformations, then the reported 11.4% exact match and 13.6 weighted score may partially reflect pattern memorization rather than true language generalization. The authors acknowledge small test size and synthetic data quality (sentences b97, b98) but do not address this potential train-test overlap. Since the central empirical claim for text tasks depends on the assumption of an independent held-out set, this is a load-bearing concern. The VQA evaluation (20 examples, separate images and annotations) is less affected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper presents EvenVLM, a text and multimodal (VQA) dataset for Even (Lamut), an endangered Northern Tungusic language. The authors construct a corpus of 50,521 structured entries from dictionary and phrasebook sources via GPT-4o Mini-based parsing, expand it synthetically to 107,543 training examples, and collect 189 images with 477 bilingual question-answer pairs (expanded to 1,908 VQA examples). They propose a three-step tokenizer expansion algorithm that adds 303 tokens, reducing average tokens per word by 27%. Using LoRA fine-tuning of Qwen3-VL-4B, they report 11.4% exact match / 13.6 weighted score on a 100-example text benchmark, 0.1283 chrF for sentence translation, and 65% accuracy on a 20-example VQA test set (up from 0% baselines). All datasets, benchmarks, and code are released.","tokens_in":11081,"tokens_out":2225,"duration_ms":21058,"significance":"If the empirical claims hold, this work provides the first open NLP and VLM dataset for Even, a language with fewer than 10,000 speakers, and demonstrates a reproducible pipeline (tokenizer expansion + LoRA fine-tuning) that could transfer to other severely low-resource agglutinative languages. The release of data and code is a concrete contribution to linguistic diversity in AI. The paper fills a clear gap, as VLMs for languages with under 10,000 speakers are virtually absent from the literature. The tokenizer adaptation method is simple but effective and documented in sufficient detail to be replicated.","major_comments":[{"comment":"The 100-example text evaluation set is described in terms of task composition, but the paper does not specify whether these examples are drawn from dictionary entries that were held out from the 50,521-entry corpus (Section 3.3) used to generate the 107,543 synthetic training examples. If the test examples originate from the same dictionary entries or from the same templated transformations, then the reported exact match and weighted score may partially reflect pattern memorization rather than true language generalization. The authors acknowledge small test size (sentence b98) but do not address this potential train-test overlap. Since the central empirical claim for text tasks depends on an independent held-out set, the authors must clarify the source of the test examples and, if there is overlap, demonstrate that the metrics are not inflated by leakage. This is a load-bearing concern for the validity of Table 1.","section":"7.1 Benchmark Suite and Metrics","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The training data is synthetically generated from dictionary entries using templated transformations (declension, negation, translation, etc.). However, the text evaluation set (Section 7.1) includes tasks such as grammar and translation that could be generated by the same templates. Even if the specific test tokens differ, if the test examples are constructed from held-out entries, that should be stated. If they are not, the paper should explain how the test set was constructed to avoid overlap and why the reported results are not affected by memorization of template patterns.","section":"4.1 Template-Based Generation","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states 'reducing average tokens per word by 27% (4.99 to 3.64)' which is clear. However, the improvement is from 4.99 to 3.64, which is a 27% reduction, but the median goes from 5 to 4 (20% reduction). Please verify the median percentage for consistency.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The baseline configuration 'Base + Tokenizer' shows a drop in weighted score from 2.1 to 0.7. The text says tokenizer expansion alone adds noise because new tokens have randomly initialized embeddings. This is plausible, but the drop is large (67% relative). Consider adding a brief explanation of why the drop is so substantial, e.g., that the randomly initialized embeddings disrupt previously working token sequences.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that LoRA rank r=32 fits within the 32GB GPU budget. However, the VQA stage uses r=64 with a frozen vision encoder. The justification for different ranks is given, but it would be helpful to state the peak memory usage for each configuration to support the claim that r=32 was necessary for Stage 1.","section":"6.2 Stage 1: Text Fine-Tuning","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The maximum of 300 tokens for whitelist + BPE merging is described. It is not entirely clear whether the 209 whitelist words and 101 VQA answer words are counted against the 300 limit. Since 209+101=310 exceeds 300, presumably not all whitelist words were added, or the limit applies to BPE merges separately. Please clarify the counting methodology.","section":"5.1 Expansion Algorithm, Step 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The training and evaluation loss curves show eval loss at 1.179 at epoch 6, but the curve appears to be still decreasing slowly. Consider extending training to verify whether the validation loss has truly converged or if additional epochs may yield further improvement.","section":"Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 3 shows a reading comprehension example where the model outputs 'бэй тэгэттэн' instead of the target 'тэгэттэн'. This is classified as 'wrong', but the root 'тэгэттэн' is correct and the extra 'бэй' is a semantic substitution. The weighted Levenshtein score might give partial credit here; the qualitative description could note this nuance.","section":"7.3 Qualitative Observations","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hi [Colleague],\n\nHere's my take on EvenVLM.\n\nWhat's actually new: this is the first open text and VQA dataset for Even, an endangered Tungusic language. The authors release 50K dictionary entries expanded to 107K synthetic examples, a tokenizer expansion algorithm that reduces tokens per word by 27%, and the full fine-tuning pipeline. That's a concrete resource contribution to a language that previously had no NLP dataset. The work is reproducible (code and data released), and the authors are refreshingly transparent about limitations—small test sets, synthetic data quality, dialect skew, incomplete convergence on VQA.\n\nWhat it does well: the pipeline design is straightforward and could be adapted to other agglutinative languages with similar resources. The tokenizer expansion is sensible: add missing graphemes, whitelist frequent words, then do targeted BPE merging. The ablation confirms that each component (tokenizer expansion, LoRA) helps, though the absolute gains are modest. The VQA result—65% accuracy on 20 held-out images—shows that 1,908 synthetic examples suffice for basic visual-lexical mapping, which is non-trivial for a language where the base model outputs nothing.\n\nWhere the soft spots are: the stress-test concern about text evaluation set independence is real and not addressed. The paper describes 100 text examples across four tasks but never says where these examples came from. If they were drawn from the same dictionary entries that generated the training data, then the reported 11.4% exact match and 0.1283 chrF could partially reflect pattern memorization rather than true generalization. The authors acknowledge small test size (n=100) and synthetic data quality, but missing train-test separation is a separate issue. That said, the numbers are low enough that the overall conclusion—the model learns something from the pipeline—still holds, but the exact magnitude is uncertain. The VQA evaluation is less affected because it uses separate images and annotations.\n\nWho this is for: researchers in low-resource endangered language NLP, especially those working with agglutinative languages that have at least a dictionary. The paper doesn't break new methodological ground (LoRA, BPE, template-based synthesis are standard), but the application and reproducible pipeline are valuable.\n\nRecommendation: the paper deserves a serious referee and should be accepted after the authors clarify how the text test set was constructed and whether it overlaps with the source data. That's a minor revision, not a rejection.","headline":"The paper's main value is the dataset and pipeline for Even; the evaluation is honest about its limits, but the missing specification of test set independence is a genuine gap that should be addressed.","tokens_in":11502,"tokens_out":2441,"duration_ms":24120},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"EvenVLM creates the first text and multimodal dataset for the endangered Even language, demonstrating that a vision-language model can be adapted to a severely low-resource language using tokenizer expansion and synthetic data.","keywords":["Even language","low-resource NLP","vision-language model","tokenizer adaptation","synthetic data","visual question answering","LoRA fine-tuning","endangered languages"],"falsifier":"Construct a held-out test set of at least 200 Even sentences and 200 VQA examples, validated by three independent native speakers; if the model's exact match accuracy falls below 5% (effectively zero) under these conditions, then the current results are artifacts of the small test set and the template-based dataset does not transfer to natural language use.","tokens_in":14339,"feed_emoji":"📘","tokens_out":1968,"duration_ms":51293,"temperature":0.7,"pith_summary":"EvenVLM is the first text and multimodal dataset for Even, an endangered Siberian language with about 6,800 speakers. The authors take a 4.4-billion-parameter vision-language model (Qwen3-VL-4B) that starts with zero ability to produce Even and adapt it through a three-part pipeline: they extract structured entries from a dictionary and phrasebook via GPT-4o Mini, expand those into 107,543 synthetic training examples; they modify the tokenizer to handle Even’s agglutinative morphology, cutting average tokens per word by 27%; and they apply two-stage LoRA fine-tuning. The resulting model achieves 65% accuracy on visual question answering (from 0% baseline) and measurable scores on translation, grammar, and classification tasks. The work matters because it shows a reproducible path for bringing modern AI tools to languages that have virtually no prior digital resources.","feed_headline":"First VLM dataset for an endangered language hits 65% VQA accuracy","feed_subtitle":"Tokenizer expansion and synthetic data turn a 4.4B model from zero to useful on Even text and visuals.","key_machinery":"Three mechanisms carry the argument. 1) Template-based synthetic data generation: from 50,521 structured Even-Russian dictionary entries, the authors create 107,543 training examples across translation, grammar, reading comprehension, and classification using deterministic linguistic templates, avoiding the cold-start problem that would plague self-training approaches for a language where base models score 0%. 2) Three-step tokenizer expansion: add three missing Cyrillic graphemes (ӈ, ө, ӫ) as single tokens, whitelist 209 high-frequency Even words, and run BPE merging on the full text corpus—adding 303 tokens and reducing average tokens per word from 4.99 to 3.64, a 27% reduction in fragmentation. 3) Two-stage LoRA fine-tuning: Stage 1 adapts the text decoder with rank-32 LoRA (0.53% trainable parameters) on the full synthetic text dataset; Stage 2 applies a higher-rank (64) LoRA to seven projection matrices for VQA, with the vision encoder frozen, overcoming the multimodal cold start that leaves text-only fine-tuning at 0% VQA accuracy.","core_discovery":"The core claim is that a standard vision-language model can be made functional for an extremely low-resource agglutinating language through a combination of template-based synthetic data generation, specialized tokenizer expansion, and parameter-efficient fine-tuning. The discovery is not that the model performs at human level—it clearly does not, with exact match at 11.4% on text tasks—but that the zero-shot barrier (0% in all metrics) can be crossed with a relatively small and reproducible investment of data and compute. The VQA accuracy of 65% on a 20-example test set, up from 0%, demonstrates that even incomplete convergence (loss falling only from 10.2 to 6.1) yields basic visual-lexical mapping. This shifts the question from 'can these models be applied?' to 'how much more data and validation are needed to make them reliable?'","pith_inferences":["The reliance on a single native speaker for linguistic validation is a significant vulnerability: the synthetic templates and extracted entries may contain dialectal biases or systematic errors that only multi-speaker checking would reveal; extending the validation pool is a high-impact next step.","The evaluation set sizes (n=20 for VQA, n=100 for text) are acknowledged as too small, yielding 95% confidence intervals roughly 43%–82% for the VQA result; future work should treat the reported numbers as pilot estimates rather than established performances, and re-evaluation with larger sets may revise both the floor and the ceiling.","The template-based generation excludes naturalistic sentence variation; a model trained only on template data may fail on naturally produced Even speech or writing, where word order, idiom, and zero morphology differ from constructed examples. A real-world corpus collection (if feasible) would be a necessary complement."],"forward_implications":["The synthetic data pipeline can be replicated for other low-resource, agglutinative languages that have a dictionary and phrasebook, offering a deterministic alternative to self-training approaches that require a functional teacher model.","The tokenizer expansion algorithm is language-agnostic and could be applied to any tokenizer serving a language with unrepresented graphemes or high morphological productivity, reducing token fragmentation and improving model efficiency.","The two-stage LoRA strategy demonstrates that multimodal capability for a new language can be acquired with as few as 1,908 VQA examples, suggesting that the data bottleneck for vision-language adaptation in low-resource languages may be less severe than often assumed.","All resources released openly (datasets, benchmarks, pipeline) lower the barrier for future work on Even, and the approach is explicitly designed to extend to Evenki, a related language, enabling comparative studies across Northern Tungusic languages.","Even if the absolute metric values are modest, the paper provides a concrete lower bound and a baseline for future improvements; any subsequent work on Even or similar languages can now compare against these first published numbers."],"supporting_citations":[{"why":"Provides the 256-page Even-Russian dictionary that is the primary source of the 50,521 structured entries from which the synthetic training data is generated.","marker":"Kanyukova and Levin, 2025"},{"why":"Provides the 87-page Russian-Even phrasebook that supplies additional thematic phrases and dialogues for the synthetic data expansion.","marker":"Sharina and Kuzmina, 2014"},{"why":"Supplies the Qwen3-VL-4B base model architecture, including the transformer decoder and ViT vision encoder that the authors adapt.","marker":"Bai et al., 2025"},{"why":"Provides the LoRA method for parameter-efficient fine-tuning, which the authors apply in two stages to adapt the vision-language model to Even.","marker":"Hu et al., 2021"},{"why":"Supplies the GPT-4o Mini model used for parsing the PDF dictionaries and phrasebooks into structured entries in the data extraction pipeline.","marker":"OpenAI, 2024"},{"why":"Provides the survey showing fewer than ten papers apply VLMs to languages with under 10,000 speakers, establishing the gap that EvenVLM addresses.","marker":"Lupaşcu et al., 2026"},{"why":"Supplies the chrF metric (character n-gram F-score) used for evaluating sentence translation, chosen because it operates on character n-grams and is robust to Even's flexible word order.","marker":"Popović, 2015"},{"why":"Provides the linguistic description of the Eastern and Western dialects of Even, used to define the dialect coverage of the collected data.","marker":"Malchukov, 1995"},{"why":"Supplies the BPE subword tokenization technique that the authors adapt and extend for Even-specific tokenizer expansion.","marker":"Sennrich et al., 2016"}],"fun_headline_variants":["Cracking the zero-shot barrier for Even language VLM","65% VQA from zero: first Even vision–language dataset","Tokenizer expansion cuts Even tokens 27%, enables VLM","From 0% to 65%: synthetic data rescues an endangered language VLM","EvenVLM: 11.4% exact match, 65% VQA on first try"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire pipeline depends on the assumption that the synthetic training examples generated by template expansion from a single dictionary and phrasebook are sufficiently accurate and representative of the Even language to serve as reliable training signals, yet only one native speaker was consulted for verification.","fun_headline_variants_meta":{"raw":{"variants":["Cracking the zero-shot barrier for Even language VLM","65% VQA from zero: first Even vision–language dataset","Tokenizer expansion cuts Even tokens 27%, enables VLM","From 0% to 65%: synthetic data rescues an endangered language VLM","EvenVLM: 11.4% exact match, 65% VQA on first try"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"cost":0.000198968,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000198968,"upstream_inference_prompt_cost":0.000111888,"upstream_inference_completions_cost":8.708e-05},"total_tokens":1315,"prompt_tokens":1004,"completion_tokens":311,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":211}},"tokens_in":1004,"tokens_out":311,"cost_source":"openrouter_usage","duration_ms":3185,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":211,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T14:44:23.121966+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Construct a held-out test set of at least 200 Even sentences and 200 VQA examples, validated by three independent native speakers; if the model's exact match accuracy falls below 5% (effectively zero) under these conditions, then the current results are artifacts of the small test set and the template-based dataset does not transfer to natural language use.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 256-page Even-Russian dictionary that is the primary source of the 50,521 structured entries from which the synthetic training data is generated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 87-page Russian-Even phrasebook that supplies additional thematic phrases and dialogues for the synthetic data expansion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the linguistic description of the Eastern and Western dialects of Even, used to define the dialect coverage of the collected data."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}