REVIEW 4 major objections 4 minor 14 references
Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fine-tuned mT5-large hits 88%, beats all prompted LLMs on Czech anaphora resolution.
desk verdict A useful new Czech anaphora benchmark with a plausible main conclusion, but the headline 88% vs 74.5% gap rests on a custom lenient metric and a few unaddressed experiment flaws that should be fixed before the numbers are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a relaxed span-level accuracy metric defined in Section 3.3: a prediction is correct if it contains the syntactic root of the gold antecedent span and is fully contained within that span's token boundaries. This rule is applied uniformly to prompted and fine-tuned outputs, and every headline number in the paper is measured under it. The task itself is standardized by the dataset construction: each example is a passage of at most three sentences with exactly one anaphor-antecedent pair marked by XML tags, which makes the span-extraction objective identical for both paradigms.
What would settle it
Re-run the same models on the same test set with exact span matching or with standard coreference metrics such as MUC, $B^{3}$, and CEAF (using a mention detector to obtain span candidates). If fine-tuned mT5-large no longer beats prompted Mistral Large 2, or the gap drops below a statistically meaningful margin, the paper's central conclusion fails.
Extended reading notes
Core claim
The central claim is that a fine-tuned mT5-large model, trained to output the antecedent span for a marked pronoun in a Czech sentence, achieves 88% accuracy on the test set, outperforming every evaluated prompting approach by at least 13.5 points. The prompted Mistral Large 2 (123B) reaches 74.5% accuracy only in the three-shot question-answering format; the same model in zero-shot scores 69.5%, and other models fall well below. Fine-tuned mT5-base (83.1%) already beats all prompted models, indicating that the gap comes from task-specific training rather than model size. The paper attributes this to the controlled output format of fine-tuned sequence-to-sequence models versus the variable, format-prone outputs of prompted LLMs, and it shows the advantage persists across anaphora types, with grammatical anaphora (91.7% for mT5-large) easier than textual (82.9%).
Load-bearing premise
All reported accuracies, including the headline 88% versus 74.5% gap, are measured with a custom lenient rule that counts a prediction as correct if it contains the root of the true antecedent and stays inside that antecedent's span; if a stricter exact-match or standard coreference metric were applied, the ranking and the size of the gap could change.
Editorial extensions
If this is right
- Fine-tuning a model as small as mT5-base (580M parameters) beats a 123B parameter model prompted in three-shot, so parameter count alone does not determine anaphora resolution quality.
- Prompted LLMs perform best when the task is framed as open-ended question answering; yes/no classification overestimates ability and tag-insertion prompting degrades accuracy and inflates output length.
- Because fine-tuned and prompted models show the same relative difficulty across anaphora types and pronoun categories, the performance variance is unlikely to come from training-data imbalance alone.
- The practical recommendation is that for morphologically rich, lower-resourced languages, building annotated data and fine-tuning compact sequence-to-sequence models remains a competitive strategy.
- A direct next step is extending the fine-tuned model with a mention detector to turn the anaphora resolver into a full coreference resolution pipeline.
Reading between the lines
- The single-anaphor, three-sentence task framing removes most of the ambiguity a real coreference system faces; on full documents with many mentions and no pre-marked anaphor, the fine-tuning advantage may shrink.
- Under exact span matching, all scores would likely drop, and the prompted models' tendency to produce over- or under-specific spans (which the relaxed rule forgives) could bring them closer to the fine-tuned results.
- A testable extension: evaluate the same models on a held-out corpus with different annotation conventions or on non-pronominal anaphora; the paper's conclusions are currently limited to pronominal coreference in Czech.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares prompt-based and fine-tuned approaches to Czech pronominal anaphora resolution using a dataset derived from the Prague Dependency Treebank, with one anaphor-antecedent pair per passage and at most three sentences of context. Prompting experiments cover Yes/No, question-answering, and tagging prompts across six instruction-tuned LLMs; fine-tuning experiments cover mT5-small/base/large and Mistral 0.2 with LoRA. All systems are scored with a relaxed span-level accuracy rule that requires the predicted span to contain the syntactic root of the gold antecedent and to be contained in the gold span. The main reported result is that fine-tuned mT5-large reaches 88% accuracy, outperforming the best prompted model (Mistral Large 2, 74.5% in three-shot question answering), and the paper analyzes error patterns by anaphora type, pronoun category, distance, and subcorpus.
Significance. The comparison is potentially useful for practitioners choosing between fine-tuning compact multilingual models and prompting large LLMs for a morphologically rich, lower-resourced language. The paper contributes a released dataset derived from PDTC, a rule-based baseline, and a relatively broad prompt/model sweep. If the central comparison is robust to the evaluation metric and statistical uncertainty, the result would substantiate the claim that task-specific fine-tuning remains competitive. However, the headline gap rests on a single unvalidated evaluation rule and single-run measurements, so the quantitative conclusions are currently over-stated.
major comments (4)
- [Section 3.3] The relaxed metric is load-bearing but unvalidated. A prediction counts as correct if it contains the syntactic root of the gold antecedent span and is fully contained in that span, so a model that outputs only the root token is scored as fully correct. Every reported accuracy, including the headline 88% versus 74.5% gap in Tables 2 and 5, is computed under this rule. I ask the authors to additionally report exact-span accuracy and, where feasible, standard coreference metrics (MUC, B^3, CEAF) for at least the main systems, and to state whether the ranking and gap persist. The metric is applied symmetrically, so this is a correctness-risk concern rather than an allegation of unfairness; it is directly testable from the released dataset and predictions.
- [Tables 2-5] All experiments are reported as single runs with no confidence intervals, error bars, or significance tests. The abstract and Section 5 use language such as "outperform them significantly," but no statistical procedure supports that wording. I recommend reporting multiple seeds or bootstrap confidence intervals and a paired test (e.g., McNemar) on the test set for the mT5-large versus Mistral Large 2 comparison, so the reader can assess whether the 13.5-point gap is stable.
- [Section 3.2, Table 3] The Yes/No experiment has an always-YES gold distribution: each instance is by construction a valid anaphor-antecedent pair, so the correct answer to the prompt in Figure 2 is always YES. The accuracies in Table 3 therefore measure how often the model complies with the prompt, not anaphora resolution ability; an always-YES baseline would score 100%. The paper should report that baseline, and either reinterpret Table 3 as an instruction-following check or remove it from the resolution comparison.
- [Abstract and Section 5] The claim that fine-tuned mT5-large achieves 88% "while requiring fewer computational resources" is not supported by any measurement. Fine-tuning consumes training compute that prompting does not, so the comparison is ambiguous. I ask the authors to specify whether the claim concerns inference only, and to report a concrete resource proxy (parameters, FLOPs, latency, or energy) for both training and inference.
minor comments (4)
- [Section 3.2] The hyperparameter description is incomplete for reproduction: the number of training epochs for mT5 is not stated, the LoRA rank/alpha for Mistral is not given, and the decoding length constraint is not specified.
- [Section 4.1] The sentence "In each case, the model score in the question-answering experiment was greater than in the other two experiments" is contradicted by Tables 2 and 3, where zero-shot Yes/No scores are consistently higher than zero-shot QA scores; the intended comparison should be clarified.
- [Table 6] In the row "n.pron.indef", the reported average 0.907 does not equal the unweighted mean of the grammatical (0.909) and textual (0.714) accuracies; if the mean is weighted by instance counts, the weighting should be stated in the caption.
- [Section 3.1] The task formulation (one anaphora pair per passage, at most three sentences of context) is a reasonable design choice, but it should be acknowledged as a scope limitation rather than presented as a complete evaluation of coreference resolution, since multi-anaphor discourse and longer-range coreference are excluded.
Circularity Check
Main comparison is an independent train/test evaluation; the only circular step is the Yes/No sub-experiment, where the prompt supplies the gold antecedent, making the task trivially satisfiable and peripheral to the headline claim.
-
self definitional
[Section 3.2, Figure 2 (Yes/No prompt); Section 3.1 (dataset annotation); Section 3.3 (antecedent_subtree definition)]
"Prompt: You are an anaphora resolution system. In the following sentence: "$sentence_ana$" does "$anaphora$" refer to "$antecedent_subtree$" ? Respond only YES or NO. Do not include anything else in your response. ... Each sentence is annotated with a single anaphor-antecedent pair using XML-style tags: <ana></ana> for the anaphor and <ant></ant> for the antecedent."
The Yes/No prompt inserts the gold antecedent span ($antecedent_subtree$) into the question and asks only for confirmation. Since every dataset example contains exactly one anaphor-antecedent pair and the antecedent is already supplied, the correct answer is always YES; no resolution ability is exercised. The high scores in Table 3 (e.g., Aya 23 at 0.992) measure the model's tendency to agree with a leading question, not its capacity to find antecedents. This is self-definitional: the 'correct' output is fixed by the prompt's own input. The step is peripheral, however, because the headline comparison (fine-tuned mT5-large 0.880 vs Mistral Large 2 0.745) comes from the question-answering experiment and the fine-tuning results, not from the Yes/No table.
full rationale
The central claim of the paper is an empirical comparison, not a derivation from first principles: fine-tuned mT5-large and prompted LLMs are evaluated on the same test instances under the same relaxed span-containment metric defined in Section 3.3. That metric is lenient (it counts a bare syntactic root inside the gold span as correct), and the paper reports no exact-span or standard coreference scores, but it is applied symmetrically to both model families and is not equivalent to any fitted parameter or to the training labels. There are no load-bearing self-citations: the cited works on mT5, Mistral, Llama, and PDT are external sources, and the authors do not invoke their own prior results to justify the setup. The only step that reduces to its own input is the Yes/No sub-experiment: the prompt contains the gold antecedent span and asks for a yes/no confirmation, so the correct answer is always YES and the accuracies in Table 3 measure prompt compliance rather than anaphora resolution. This circularity is real but confined to a peripheral table; it does not by itself produce the 88% vs 74.5% gap, which rests on the question-answering and fine-tuning experiments. The score is therefore low.
Assumptions & free parameters
free parameters (3)
- Relaxed metric acceptance criteria (root-token match and span containment)
- Fine-tuning hyperparameters (learning rate, batch size, epochs, LoRA setup) =
LR 2e-5, batch size 1 (mT5), 3-6 epochs
- Dataset task constraints (one anaphora pair per sentence, max 3 sentences of context)
assumptions (4)
- domain assumption PDTC 1.0 coreference annotations are reliable enough to derive training targets.
- ad hoc to paper The relaxed span metric is a valid proxy for anaphora resolution quality.
- domain assumption Held-out test performance generalizes to other Czech texts beyond the three PDT-C subcorpora.
- domain assumption Model outputs were parsed and matched without systematic errors.
Cite this review
Pith. "Pith review of Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution." pith.science (2026). https://pith.science/paper/BUIUJHT7
@misc{pith2026250618091,
author = {Pith},
title = {Pith review of: Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution},
year = {2026},
howpublished = {\url{https://pith.science/paper/BUIUJHT7}},
note = {Machine review of arXiv:2506.18091}
}
read the original abstract
Anaphora resolution plays a critical role in natural language understanding, especially in morphologically rich languages like Czech. This paper presents a comparative evaluation of two modern approaches to anaphora resolution on Czech text: prompt engineering with large language models (LLMs) and fine-tuning compact generative models. Using a dataset derived from the Prague Dependency Treebank, we evaluate several instruction-tuned LLMs, including Mistral Large 2 and Llama 3, using a series of prompt templates. We compare them against fine-tuned variants of the mT5 and Mistral models that we trained specifically for Czech anaphora resolution. Our experiments demonstrate that while prompting yields promising few-shot results (up to 74.5% accuracy), the fine-tuned models, particularly mT5-large, outperform them significantly, achieving up to 88% accuracy while requiring fewer computational resources. We analyze performance across different anaphora types, antecedent distances, and source corpora, highlighting key strengths and trade-offs of each approach.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Agrawal, M., et al.: Large language models are few-shot clinical information ex- tractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 1998–2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Dec 2022).https://doi.org/10.18653/v1/ 2022.emnlp-main.130,https://aclanthology.org...
doi:10.18653/v1/ 2022
-
[2]
Aryabumi, V., et al.: Aya 23: Open weight releases to further multilingual progress (2024),https://arxiv.org/abs/2405.15032
arXiv 2024
-
[3]
Dubey, A., et al.: The llama 3 herd of models (2024),https://arxiv.org/abs/ 2407.21783
arXiv 2024
-
[4]
Hajič, J., et al.: Prague Dependency Treebank – Consolid. 1.0, PDT-C 1.0 (2020), http://hdl.handle.net/11234/1-3185, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics ÚFAL, Faculty of Mathematics and Physics, Charles University
work page 2020
-
[5]
Hajič, J., et al.: Prague dependency treebank 3.5 (2018),http://hdl.handle.net/ 11234/1-2621, LINDAT/CLARIN digital library at the Institute of Formal and AppliedLinguisticsÚFAL,FacultyofMathematicsandPhysics,CharlesUniversity
work page 2018
-
[6]
Hicke, R., Mimno, D.: [Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Liter- ary Coreference Annotation with LLMs. In: Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguisticsfor Cultural Heritage,Social Sciences,Hu- manities and Literature (LaTeCH-CLfL 2024). pp. 270–277. Association for Com- putational Linguistics, St. Julians, Malta ...
work page 2024
-
[7]
In: In- ternational Conference on Learning Representations (2022),https://openreview
Hu, E.J., et al.: LoRA: Low-Rank Adaptation of Large Language Models. In: In- ternational Conference on Learning Representations (2022),https://openreview. net/forum?id=nZeVKeeFYf9
work page 2022
-
[8]
Jiang, A.Q., et al.: Mistral 7b (2023),https://arxiv.org/abs/2310.06825
arXiv 2023
Show all 14 references
-
[9]
Le, N.T., Ritter, A.: Are large language models robust coreference resolvers? (2023),https://arxiv.org/abs/2305.14489
2023 arXiv
-
[10]
Gemma Team, Google Deepmind: Gemma 2: Improving open language models at a practical size (2024),https://arxiv.org/abs/2408.00118
2024 arXiv
-
[11]
Mistral AI Team: Mistral large-instruct-2407 (2024), available athttps:// huggingface.co/mistralai/Mistral-Large-Instruct-2407
2024
-
[12]
Journal of machine learning research21(140), 1–67 (2020)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)
2020
-
[13]
In: Proceedings of the CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies
Straka, M.: UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In: Proceedings of the CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. pp. 197–207. Association for Computational Linguistics (2018)
2018
-
[14]
In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Xue, L., et al.: mT5: A massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 483–498. Association for Computational L...
2021 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.