{"id":"32766cf2-4927-49de-b4bc-20a2986ec93e","arxiv_id":"2506.17715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning open-source LLMs outperforms prompting for POS tagging on medieval Occitan, French, and Spanish, and pooling Romance training data helps the most under-resourced texts.","lead":"This paper tests seven open-source language models on part-of-speech tagging for three medieval Romance languages and seven text datasets. It finds that fine-tuning generally beats prompt-based methods, and pooling training data from related languages helps the least-resourced texts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed cross-lingual transfer benefit is not actually isolated by the experimental design and lacks any uncertainty estimates, so the +0.86 pp average and +5.69 pp Harley gain are not yet supported.","rationale":"The reader's concern about gold annotation quality is legitimate and worth addressing, but it is not the most load-bearing issue for the central claim. If label errors were approximately random across conditions, the comparative conclusions about fine-tuning versus prompting and transfer versus single-dataset training could survive; the missing annotation-quality metrics affect absolute accuracy more directly than these relative comparisons. The more decisive problem is that the paper's main quantitative claim—cross-lingual transfer yielding +0.86 pp on average and +5.69 pp on Harley—rests on an experimental setup that includes the target's own training data in the 'transfer' condition and on single-split point estimates with no uncertainty quantification. This is not a terminological quibble: the authors explicitly recommend pooling data from multiple medieval Romance languages for low-resource varieties (Section 5), so the validity of that recommendation depends on whether the gains come from genuinely cross-lingual signal or simply from increasing the training set with target-language data. The proposed source-only transfer rerun would settle this. I would keep the reader's CONDITIONAL verdict because the released data, code, and broad experiment suite are valuable and the conclusions are plausible, but the transfer claim and the 'few-shot' label (Appendix B provides no tagged examples, only etymological cognates) need to be corrected before the paper's headline findings can be accepted at face value. The fine-tuning-versus-prompting claim also needs scrutiny: the paper's own Table 2 shows few-shot prompting beating fine-tuning on Chauliac (0.8815 vs 0.8413), which strains the word 'consistently' even though the overall pattern favors fine-tuning.","tokens_in":16688,"tokens_out":8066,"duration_ms":89545,"concrete_test":"Re-run the cross-lingual condition in Table 1 Setting (b) in two variants: (1) with the target dataset excluded from training entirely (source-only transfer, evaluating on the full target test split), and (2) with the target included but using 5–10 random seeds/restarts on the same 80/20 split, reporting mean and standard deviation for each per-dataset delta in Table 5. If source-only deltas are near zero or negative for Harley and NAF, the claimed cross-lingual benefit is an artifact of pooling target training data; if 95% confidence intervals for the pooled-data deltas include 0, the +0.86 pp average is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 / Table 1 defines the cross-lingual transfer condition (Setting b) as fine-tuning on 80% of all datasets combined and evaluating on the held-out 20% of a target dataset. Because the target dataset's own training split is included in the combined training set, the +0.86 pp average improvement and the Harley +5.69 pp gain reported in Table 5 do not demonstrate transfer to an unseen target variety; they show only that adding six other datasets to the target's training data improves in-domain test accuracy. This conflates 'more/different training data' with 'cross-lingual transfer.' Additionally, every number in Tables 3–5 comes from a single 80/20 split with no repeated seeds, so even large deltas such as Harley (+5.69) could be split-specific, and small deltas (e.g., qwen NAF +0.09, gemma_9b Cauliaco -0.60) are within run-to-run noise. No variance, confidence intervals, or significance tests are reported anywhere. The headline claim therefore rests on an experimental design that cannot separate transfer from data augmentation, and on point estimates with no measure of stability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a systematic evaluation of POS tagging for seven low-resource medieval Romance datasets (Medieval Occitan, French, and Spanish) using seven open-weight instruction-tuned LLMs. It compares zero-shot versus few-shot prompting and several decoding strategies, then compares single-dataset fine-tuning against multilingual fine-tuning, which it calls cross-lingual transfer. The paper reports that fine-tuning generally outperforms prompting, that multilingual training yields an average improvement of +0.86 percentage points with the largest gain (+5.69) on the low-resource Harley corpus, that model family and pretraining language coverage matter more than parameter count, and it releases two new Medieval Occitan datasets and an open codebase with detailed results.","tokens_in":16897,"tokens_out":5627,"duration_ms":52772,"significance":"If the empirical claims hold, the paper would make a useful contribution to historical NLP: it introduces two new Medieval Occitan datasets totaling 135,667 tokens, runs a broad experiment matrix across languages, models, and decoding strategies, and releases code and detailed results for reproducibility. The systematic comparison of fine-tuning and prompting on genuinely non-standardized medieval texts is valuable, and the error analysis provides concrete practical guidance. However, the central conclusions are currently undermined by a design confound in the transfer experiments, the absence of any uncertainty estimates, and unmeasured annotation quality in the gold data; these issues are load-bearing for the headline claims, though they appear fixable within the scope of a revision.","major_comments":[{"comment":"The cross-lingual transfer condition is not actually a transfer condition as implemented. Table 1 defines Setting (b) as fine-tuning on 80% of all datasets combined and evaluating on the held-out 20% of a target dataset; because the target's own 80% training split is part of that combined training set, the +0.86 pp average and the +5.69 pp Harley gain in Table 5 conflate receiving additional in-domain training data (augmentation) with transfer from other languages. To support RQ3, the target dataset must be held out entirely, or the comparison must be between single-dataset training and training on all other datasets while excluding the target's training split; otherwise the headline 'cross-lingual transfer' claim is not supported by this design.","section":"Section 3.1, Table 1 (Setting b), Table 5"},{"comment":"All accuracy numbers come from a single 80/20 split with no repeated seeds or uncertainty estimates, so the paper has no way to distinguish systematic effects from split noise. For example, qwen_14b NAF changes by +0.09 pp, gemma_9b Cauliaco by -0.60 pp, and mistral_7b Lapidaire by -11.45 pp; without variance or significance tests, even large deltas such as Harley +5.69 could be split-specific. The authors should rerun with multiple seeds or splits and report means and standard deviations (or confidence intervals), and temper claims about small differences accordingly.","section":"Tables 3-5, Section 4.2.2"},{"comment":"The condition labeled 'Few-shot' in Table 7 is not few-shot in the standard sense: the prompt contains no labeled POS-tagged examples, only an instruction and two etymological cognate sets (tercia/tersa/tierce/tercera, sanguina/sanc/sang/sangre). Consequently, the conclusion in Section 4.1.1 that 'providing examples during prompting helps the model understand the POS tagging task' is not supported; this comparison shows the effect of an added descriptive instruction, not of in-context examples. Either rename the condition and re-interpret the results, or include actual tagged examples in the few-shot prompt.","section":"Appendix B, Table 7; Section 4.1.1"},{"comment":"The claim that fine-tuning 'consistently provides more robust performance across language varieties' is contradicted by the Chauliac dataset, where fine-tuning gives 0.8413 against the best few-shot prompting result of 0.8815 (Table 2 vs Table 3). The text acknowledges a 'slight decrease of 0.0402' but the abstract, the recommendations in Section 5, and the conclusion state that fine-tuning consistently outperforms prompting. This qualification must be carried through the paper's central claims, or the claim needs to be restricted to the datasets where it actually holds.","section":"Section 4.2.1, Tables 2 and 3; Section 5"},{"comment":"The gold annotations are treated as ground truth without any quality measurement. Section 3.1 reports that transcriptions come from HTR models and that POS annotation relies on a modern Occitan tagger with manual corrections, existing editions, or prior annotations, but no inter-annotator agreement, annotation guidelines, or error-rate estimates are provided for any of the seven datasets. If the gold labels contain systematic errors, every reported accuracy and all comparative conclusions are affected; the authors should report at least a sample-based quality assessment or an estimate of annotation noise.","section":"Section 3.1"}],"minor_comments":[{"comment":"The definition of accuracy via TP/TN/FP/FN is unusual for per-token multiclass POS tagging and could confuse readers; please clarify that accuracy is simply the proportion of correctly tagged tokens.","section":"Appendix C, Eq. (1)"},{"comment":"The caption says 'percentual points'; this should be 'percentage points'.","section":"Table 5 caption"},{"comment":"The hardware description says RTX 4090 for '7B-12B' models, but the model suite includes 14B and 2B models; please align the hardware description with the actual model sizes used.","section":"Table 1"},{"comment":"The paper does not report fine-tuning hyperparameters (learning rate, number of epochs, LoRA rank or full fine-tuning, sequence length, batch size), which are needed to reproduce Tables 3-5; please provide these details in the appendix.","section":"Tables 3-5; Appendix"},{"comment":"The recommendation 'always prefer fine-tuning over prompting' repeats the unqualified claim contradicted by the Chauliac results; add the caveat identified in the major comments.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the design of the transfer experiments: Setting (b) includes the target dataset's training split, so the paper's central RQ3 claim is currently an augmentation claim rather than a transfer claim. Adding repeated seeds and a sample-based annotation quality audit would substantially strengthen the manuscript. The code and data release is a genuine asset, and the paper fits the scope of an NLP venue that values low-resource and historical-language resources, but it should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper ships two new Medieval Occitan datasets (135k tokens), runs 122 experiments across seven models and seven corpora, and releases code. That is the real value — a usable resource and a solid empirical mapping for a neglected area. The headline claims about fine-tuning and cross-lingual transfer are not yet supported.\n\nWhat it does well: the datasets are a genuine contribution, and the fine-tuning-versus-prompting comparison across many configurations is useful for practitioners. The finding that smaller models with Romance pretraining (Aya-8B, Gemma2-2B) beat larger ones is a practical takeaway. The error analysis and recommendations are concrete.\n\nThe soft spots are in the experimental design and framing. The cross-lingual transfer condition (Setting b) fine-tunes on 80% of all datasets combined and evaluates on the held-out 20% of the target. That includes the target's own training split. So the +0.86 pp average gain and Harley +5.69 are not evidence of transfer to an unseen variety; they show that adding other datasets to the target's training data helps. That is a real comparison, but it is pooling/multitask training, not cross-lingual transfer. The paper should either redesign the condition (train on other languages only, or at least report that ablation) or reframe the claims.\n\nSecond, every number comes from a single 80/20 split. No seeds, no variance, no significance tests. Some deltas are large, but small ones like qwen NAF +0.09 are within noise. A paper that makes practical recommendations needs at least a few seeds and a variance estimate.\n\nThird, the 'few-shot' prompt in Appendix B contains no labeled POS examples — just two etymological twin triples. So it is zero-shot with extra context, not few-shot. The reported few-shot gains are not about in-context learning from tagged examples.\n\nFourth, gold annotation quality is unmeasured. The new Occitan data was transcribed with HTR and annotated with a modern tagger then manually corrected; no inter-annotator agreement is reported. That makes all accuracies provisional.\n\nFinally, the fine-tuning consistency claim is contradicted by their own Chauliac result: fine-tuning (0.8413) is below few-shot with greedy decoding (0.8815). The abstract and conclusion overstate.\n\nOverall, this is a useful resource paper, not a decisive experimental study. It deserves peer review because the datasets and breadth are worth referee time, but it needs major revision: fix the transfer design, add variance, relabel the prompting condition, and temper the claims.","headline":"Useful new datasets and a broad benchmark, but the cross-lingual transfer claim conflates pooling with transfer, and single-run numbers make the conclusions provisional.","tokens_in":17434,"tokens_out":3282,"would_cite":true,"duration_ms":31025,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning open LLMs beats prompting for tagging medieval Romance texts, and pooling related languages helps most on the scarcest corpus.","keywords":["part-of-speech tagging","medieval Romance languages","low-resource NLP","cross-lingual transfer learning","fine-tuning vs prompting","historical text processing","Medieval Occitan","open-source LLMs"],"falsifier":"Re-annotate a random sample of each of the seven corpora with two independent expert annotators, compute inter-annotator agreement, and re-run the main fine-tuning and transfer comparisons on corrected labels; if the +5.69-point Harley transfer gain or the fine-tuning-vs-prompting gaps shrink below noise, the paper's central conclusions fail. Alternatively, a domain-matched pooled-training experiment on an independent medieval language pair would test whether the average +0.86 transfer gain replicates.","tokens_in":16520,"feed_emoji":"📜","tokens_out":6622,"duration_ms":57918,"temperature":0.7,"pith_summary":"This paper asks what actually determines part-of-speech tagging accuracy when large language models are applied to Medieval Occitan, Medieval French, and Medieval Spanish, varieties with erratic spelling, dialectal mixture, and very little annotated data. It reports that fine-tuning an open instruction-tuned model on annotated data consistently outperforms prompting with examples, and that training on several medieval Romance languages together improves accuracy on average by +0.86 percentage points over single-dataset training. The largest gain, +5.69 points, lands on the Harley 7403 corpus, the lowest-resource dataset in the study. The paper also argues that pre-training language coverage and architecture matter more than parameter count, since an 8-billion-parameter model with Romance training beats 14-billion-parameter models. A careful reader would care because these findings give concrete, evidence-based choices for anyone building NLP tools for historical languages.","feed_headline":"Transfer learning lifts rare medieval POS tagging by 5.7 points","feed_subtitle":"A seven-dataset study finds fine-tuning beats prompting and pooled Romance data helps the scarcest corpora.","key_machinery":"The central mechanism is cross-lingual transfer learning through joint fine-tuning: each model is fine-tuned on 80% of all seven datasets combined, then evaluated on the held-out 20% of one target dataset, and compared against single-dataset fine-tuning under the same split. This pooled-training protocol is what lets related Romance varieties share evidence, and is the object that produces the +0.86 average gain and the +5.69-point Harley improvement. A second mechanism is the prompting evaluation on Gemma2-9B with four decoding strategies (greedy, temperature 0.3, temperature 0.9, and top-p 0.95), which establishes the language-specific decoding recommendations. The load-bearing comparability comes from keeping the fine-tuning split and evaluation metrics identical across all seven models and datasets.","core_discovery":"The central claim is that for POS tagging of low-resource medieval Romance languages, fine-tuning is more reliable than prompting, and cross-lingual transfer learning brings a modest but real average gain of +0.86 percentage points over single-dataset fine-tuning. The transfer benefit is sharply heterogeneous: the lowest-resource Occitan corpus, Harley, improves by +5.69 points, Chauliac by +2.61, while Lapidaire drops by -3.79 points, a negative-transfer pattern the paper attributes to limited shared vocabulary. On prompting, few-shot consistently beats zero-shot by an average of +0.0120 accuracy, and the best decoding strategy depends on the language variety: temperature 0.9 for Occitan, greedy decoding for French, and temperature 0.3 for Spanish. Model size alone is not predictive: Aya-8B and even Gemma2-2B outperform larger 14B models, which the paper attributes to better Romance-language representation in pre-training. The study also contributes two newly annotated Medieval Occitan datasets, NAF and Harley, totalling 135,667 tokens.","pith_inferences":["The Harley result suggests a testable rule: transfer benefit grows as target-corpus size shrinks, which could be checked by subsampling the larger datasets and plotting gain against training size.","The negative transfer on Lapidaire hints that domain vocabulary distance can outweigh language-family closeness; a similar experiment with Anglo-Norman or other non-medical genres would clarify when pooling hurts.","Because the paper reports accuracy without annotation-quality metrics, an independent re-annotation of a sample of each corpus would show whether the reported rankings survive gold-label noise."],"forward_implications":["When any annotated medieval Romance data exists, researchers should fine-tune rather than prompt; the accuracy gap on NAF is +9.72 points and on Lanfranco +8.63 points.","For extremely low-resource varieties, pooling data from related medieval Romance languages is the recommended default, with the expectation of gains like the +5.69 points seen on Harley.","Model choice should prioritize Romance-language pre-training over raw size; a 2B model can outperform 14B models on these texts.","If fine-tuning is infeasible, few-shot prompting with language-matched decoding (temperature 0.9 for Occitan, greedy for French, temperature 0.3 for Spanish) is the best fallback.","Negative transfer is possible, as in Lapidaire's -3.79-point drop, so transfer should be validated per domain before adoption."],"supporting_citations":[{"why":"Prior study of prompting strategies, model scale, and language support on Medieval Occitan; the direct baseline this work extends.","marker":"Schöffel et al. (2025)"},{"why":"Supplies the handwriting-recognition model used to transcribe the Harley 7403 manuscript, one of the released datasets.","marker":"Wiedner (2023)"},{"why":"Provides the modern Occitan POS tagger used for the initial annotation of NAF6195 before manual correction.","marker":"Poujade, In progress"},{"why":"Shows that character-level noise injection improves transfer between closely related languages, motivating the cross-lingual transfer setup.","marker":"Aepli & Sennrich (2022)"},{"why":"Raises the question of whether transfer works in realistic divergent-variant scenarios, which the negative-transfer results address.","marker":"Vandenbulcke et al. (2024)"},{"why":"Supplies an existing Medieval French POS-tagging model and corpus that this study's French results can be compared against.","marker":"Camps et al. (2021)"},{"why":"Documents the varieties and data-scarcity characteristics that define the low-resource setting studied here.","marker":"Zampieri et al. (2019)"}],"fun_headline_variants":["Fine-tuning outshines prompting for medieval Romance POS","Cross-lingual transfer: boon for some medieval POS, bane for others","New Occitan datasets fuel POS tagging gains","Medieval POS: model size less key than language fit","Transfer lifts rarest Occitan POS tagging by 5.7"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold POS labels for all seven datasets are accurate enough to serve as ground truth, yet no inter-annotator agreement or annotation-quality metric is reported; if those labels contain systematic errors, every accuracy number and comparative conclusion shifts.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning outshines prompting for medieval Romance POS","Cross-lingual transfer: boon for some medieval POS, bane for others","New Occitan datasets fuel POS tagging gains","Medieval POS: model size less key than language fit","Transfer lifts rarest Occitan POS tagging by 5.7"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1453,"prompt_tokens":935,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":433}},"tokens_in":551,"tokens_out":518,"duration_ms":5180,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:02:06.853115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of each of the seven corpora with two independent expert annotators, compute inter-annotator agreement, and re-run the main fine-tuning and transfer comparisons on corrected labels; if the +5.69-point Harley transfer gain or the fine-tuning-vs-prompting gaps shrink below noise, the paper's central conclusions fail. Alternatively, a domain-matched pooled-training experiment on an independent medieval language pair would test whether the average +0.86 transfer gain replicates.","supporting_citations":[{"cited_title":"Old O ccitan handwriting","cited_arxiv_id":null,"evidence_quote":"Supplies the handwriting-recognition model used to transcribe the Harley 7403 manuscript, one of the released datasets."},{"cited_title":"Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise","cited_arxiv_id":"2109.06772","evidence_quote":"Shows that character-level noise injection improves transfer between closely related languages, motivating the cross-lingual transfer setup."},{"cited_title":"Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?","cited_arxiv_id":"2410.10576","evidence_quote":"Raises the question of whether transfer works in realistic divergent-variant scenarios, which the negative-transfer results address."},{"cited_title":"Corpus and Models for Lemmatisation and POS-tagging of Old French","cited_arxiv_id":"2109.11442","evidence_quote":"Supplies an existing Medieval French POS-tagging model and corpus that this study's French results can be compared against."},{"cited_title":"Natural language processing for similar languages, varieties, and dialects: A survey","cited_arxiv_id":null,"evidence_quote":"Documents the varieties and data-scarcity characteristics that define the low-resource setting studied here."}],"review_version":1}