Pith. sign in

REVIEW 4 major objections 4 minor 14 references

Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Fine-tuned mT5-large hits 88%, beats all prompted LLMs on Czech anaphora resolution.

desk verdict A useful new Czech anaphora benchmark with a plausible main conclusion, but the headline 88% vs 74.5% gap rests on a custom lenient metric and a few unaddressed experiment flaws that should be fixed before the numbers are trusted. read the letter →

arxiv 2506.18091 v1 pith:BUIUJHT7 submitted 2025-06-22 cs.CL

classification cs.CL
keywords anaphoraresolutionCzechpromptengineeringfine-tuningmT5sequence-to-sequencemodelslargelanguagePragueDependencyTreebank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for Czech anaphora resolution, task-specific fine-tuning of compact generative models beats prompt engineering with much larger instruction-tuned LLMs. On a dataset derived from the Prague Dependency Treebank, fine-tuned mT5-large reaches 88% accuracy while the best prompted model, Mistral Large 2 in a three-shot question-answering setup, reaches 74.5%. Even the much smaller fine-tuned mT5-base (83.1%) outperforms every prompted model. The authors also find that both approaches share the same difficulty profile: grammatical anaphora is easier than textual, and demonstrative pronouns are the hardest category. The practical upshot is that investing in annotated data and fine-tuning small models remains competitive for a morphologically rich, lower-resourced language.

What carries the argument

The load-bearing mechanism is a relaxed span-level accuracy metric defined in Section 3.3: a prediction is correct if it contains the syntactic root of the gold antecedent span and is fully contained within that span's token boundaries. This rule is applied uniformly to prompted and fine-tuned outputs, and every headline number in the paper is measured under it. The task itself is standardized by the dataset construction: each example is a passage of at most three sentences with exactly one anaphor-antecedent pair marked by XML tags, which makes the span-extraction objective identical for both paradigms.

What would settle it

Re-run the same models on the same test set with exact span matching or with standard coreference metrics such as MUC, $B^{3}$, and CEAF (using a mention detector to obtain span candidates). If fine-tuned mT5-large no longer beats prompted Mistral Large 2, or the gap drops below a statistically meaningful margin, the paper's central conclusion fails.

Watch

Extended reading notes

Core claim

The central claim is that a fine-tuned mT5-large model, trained to output the antecedent span for a marked pronoun in a Czech sentence, achieves 88% accuracy on the test set, outperforming every evaluated prompting approach by at least 13.5 points. The prompted Mistral Large 2 (123B) reaches 74.5% accuracy only in the three-shot question-answering format; the same model in zero-shot scores 69.5%, and other models fall well below. Fine-tuned mT5-base (83.1%) already beats all prompted models, indicating that the gap comes from task-specific training rather than model size. The paper attributes this to the controlled output format of fine-tuned sequence-to-sequence models versus the variable, format-prone outputs of prompted LLMs, and it shows the advantage persists across anaphora types, with grammatical anaphora (91.7% for mT5-large) easier than textual (82.9%).

Load-bearing premise

All reported accuracies, including the headline 88% versus 74.5% gap, are measured with a custom lenient rule that counts a prediction as correct if it contains the root of the true antecedent and stays inside that antecedent's span; if a stricter exact-match or standard coreference metric were applied, the ranking and the size of the gap could change.

Editorial extensions

If this is right

  • Fine-tuning a model as small as mT5-base (580M parameters) beats a 123B parameter model prompted in three-shot, so parameter count alone does not determine anaphora resolution quality.
  • Prompted LLMs perform best when the task is framed as open-ended question answering; yes/no classification overestimates ability and tag-insertion prompting degrades accuracy and inflates output length.
  • Because fine-tuned and prompted models show the same relative difficulty across anaphora types and pronoun categories, the performance variance is unlikely to come from training-data imbalance alone.
  • The practical recommendation is that for morphologically rich, lower-resourced languages, building annotated data and fine-tuning compact sequence-to-sequence models remains a competitive strategy.
  • A direct next step is extending the fine-tuned model with a mention detector to turn the anaphora resolver into a full coreference resolution pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-anaphor, three-sentence task framing removes most of the ambiguity a real coreference system faces; on full documents with many mentions and no pre-marked anaphor, the fine-tuning advantage may shrink.
  • Under exact span matching, all scores would likely drop, and the prompted models' tendency to produce over- or under-specific spans (which the relaxed rule forgives) could bring them closer to the fine-tuned results.
  • A testable extension: evaluate the same models on a held-out corpus with different annotation conventions or on non-pronominal anaphora; the paper's conclusions are currently limited to pronominal coreference in Czech.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper compares prompt-based and fine-tuned approaches to Czech pronominal anaphora resolution using a dataset derived from the Prague Dependency Treebank, with one anaphor-antecedent pair per passage and at most three sentences of context. Prompting experiments cover Yes/No, question-answering, and tagging prompts across six instruction-tuned LLMs; fine-tuning experiments cover mT5-small/base/large and Mistral 0.2 with LoRA. All systems are scored with a relaxed span-level accuracy rule that requires the predicted span to contain the syntactic root of the gold antecedent and to be contained in the gold span. The main reported result is that fine-tuned mT5-large reaches 88% accuracy, outperforming the best prompted model (Mistral Large 2, 74.5% in three-shot question answering), and the paper analyzes error patterns by anaphora type, pronoun category, distance, and subcorpus.

Significance. The comparison is potentially useful for practitioners choosing between fine-tuning compact multilingual models and prompting large LLMs for a morphologically rich, lower-resourced language. The paper contributes a released dataset derived from PDTC, a rule-based baseline, and a relatively broad prompt/model sweep. If the central comparison is robust to the evaluation metric and statistical uncertainty, the result would substantiate the claim that task-specific fine-tuning remains competitive. However, the headline gap rests on a single unvalidated evaluation rule and single-run measurements, so the quantitative conclusions are currently over-stated.

major comments (4)
  1. [Section 3.3] The relaxed metric is load-bearing but unvalidated. A prediction counts as correct if it contains the syntactic root of the gold antecedent span and is fully contained in that span, so a model that outputs only the root token is scored as fully correct. Every reported accuracy, including the headline 88% versus 74.5% gap in Tables 2 and 5, is computed under this rule. I ask the authors to additionally report exact-span accuracy and, where feasible, standard coreference metrics (MUC, B^3, CEAF) for at least the main systems, and to state whether the ranking and gap persist. The metric is applied symmetrically, so this is a correctness-risk concern rather than an allegation of unfairness; it is directly testable from the released dataset and predictions.
  2. [Tables 2-5] All experiments are reported as single runs with no confidence intervals, error bars, or significance tests. The abstract and Section 5 use language such as "outperform them significantly," but no statistical procedure supports that wording. I recommend reporting multiple seeds or bootstrap confidence intervals and a paired test (e.g., McNemar) on the test set for the mT5-large versus Mistral Large 2 comparison, so the reader can assess whether the 13.5-point gap is stable.
  3. [Section 3.2, Table 3] The Yes/No experiment has an always-YES gold distribution: each instance is by construction a valid anaphor-antecedent pair, so the correct answer to the prompt in Figure 2 is always YES. The accuracies in Table 3 therefore measure how often the model complies with the prompt, not anaphora resolution ability; an always-YES baseline would score 100%. The paper should report that baseline, and either reinterpret Table 3 as an instruction-following check or remove it from the resolution comparison.
  4. [Abstract and Section 5] The claim that fine-tuned mT5-large achieves 88% "while requiring fewer computational resources" is not supported by any measurement. Fine-tuning consumes training compute that prompting does not, so the comparison is ambiguous. I ask the authors to specify whether the claim concerns inference only, and to report a concrete resource proxy (parameters, FLOPs, latency, or energy) for both training and inference.
minor comments (4)
  1. [Section 3.2] The hyperparameter description is incomplete for reproduction: the number of training epochs for mT5 is not stated, the LoRA rank/alpha for Mistral is not given, and the decoding length constraint is not specified.
  2. [Section 4.1] The sentence "In each case, the model score in the question-answering experiment was greater than in the other two experiments" is contradicted by Tables 2 and 3, where zero-shot Yes/No scores are consistently higher than zero-shot QA scores; the intended comparison should be clarified.
  3. [Table 6] In the row "n.pron.indef", the reported average 0.907 does not equal the unweighted mean of the grammatical (0.909) and textual (0.714) accuracies; if the mean is weighted by instance counts, the weighting should be stated in the caption.
  4. [Section 3.1] The task formulation (one anaphora pair per passage, at most three sentences of context) is a reasonable design choice, but it should be acknowledged as a scope limitation rather than presented as a complete evaluation of coreference resolution, since multi-anaphor discourse and longer-range coreference are excluded.

Circularity Check

1 steps flagged · score 2.0 of 10

Main comparison is an independent train/test evaluation; the only circular step is the Yes/No sub-experiment, where the prompt supplies the gold antecedent, making the task trivially satisfiable and peripheral to the headline claim.

  1. self definitional [Section 3.2, Figure 2 (Yes/No prompt); Section 3.1 (dataset annotation); Section 3.3 (antecedent_subtree definition)]
    "Prompt: You are an anaphora resolution system. In the following sentence: "$sentence_ana$" does "$anaphora$" refer to "$antecedent_subtree$" ? Respond only YES or NO. Do not include anything else in your response. ... Each sentence is annotated with a single anaphor-antecedent pair using XML-style tags: <ana></ana> for the anaphor and <ant></ant> for the antecedent."

    The Yes/No prompt inserts the gold antecedent span ($antecedent_subtree$) into the question and asks only for confirmation. Since every dataset example contains exactly one anaphor-antecedent pair and the antecedent is already supplied, the correct answer is always YES; no resolution ability is exercised. The high scores in Table 3 (e.g., Aya 23 at 0.992) measure the model's tendency to agree with a leading question, not its capacity to find antecedents. This is self-definitional: the 'correct' output is fixed by the prompt's own input. The step is peripheral, however, because the headline comparison (fine-tuned mT5-large 0.880 vs Mistral Large 2 0.745) comes from the question-answering experiment and the fine-tuning results, not from the Yes/No table.

full rationale

The central claim of the paper is an empirical comparison, not a derivation from first principles: fine-tuned mT5-large and prompted LLMs are evaluated on the same test instances under the same relaxed span-containment metric defined in Section 3.3. That metric is lenient (it counts a bare syntactic root inside the gold span as correct), and the paper reports no exact-span or standard coreference scores, but it is applied symmetrically to both model families and is not equivalent to any fitted parameter or to the training labels. There are no load-bearing self-citations: the cited works on mT5, Mistral, Llama, and PDT are external sources, and the authors do not invoke their own prior results to justify the setup. The only step that reduces to its own input is the Yes/No sub-experiment: the prompt contains the gold antecedent span and asks for a yes/no confirmation, so the correct answer is always YES and the accuracies in Table 3 measure prompt compliance rather than anaphora resolution. This circularity is real but confined to a peripheral table; it does not by itself produce the 88% vs 74.5% gap, which rests on the question-answering and fine-tuning experiments. The score is therefore low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. The load-bearing choices are the custom evaluation metric, the hand-picked training hyperparameters, and the simplified task framing; these are what the central claim rests on beyond the cited prior work.

free parameters (3)
  • Relaxed metric acceptance criteria (root-token match and span containment)
    Section 3.3 defines correctness for every reported accuracy; it is hand-chosen and lenient, and the paper does not report results under stricter exact-span or standard coreference metrics.
  • Fine-tuning hyperparameters (learning rate, batch size, epochs, LoRA setup) = LR 2e-5, batch size 1 (mT5), 3-6 epochs
    Section 3.2 selects these by hand; results depend on them, and no sweeps or ablations are reported.
  • Dataset task constraints (one anaphora pair per sentence, max 3 sentences of context)
    Section 3.1 imposes these extraction constraints, which shape the task and may favor models trained on the same distribution.
assumptions (4)
  • domain assumption PDTC 1.0 coreference annotations are reliable enough to derive training targets.
    Section 3.1 derives the dataset from PDT-C; no annotation agreement or quality check is reported.
  • ad hoc to paper The relaxed span metric is a valid proxy for anaphora resolution quality.
    Section 3.3 defines the metric; its validity is asserted, not validated against human judgments or standard coreference metrics.
  • domain assumption Held-out test performance generalizes to other Czech texts beyond the three PDT-C subcorpora.
    Section 4.2 extrapolates to 'domain adaptability'; the test set comes from the same corpora as training.
  • domain assumption Model outputs were parsed and matched without systematic errors.
    Section 3.3 scores predicted spans; no mention of handling malformed outputs (e.g., missing brackets in the Tag experiment) is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution." pith.science (2026). https://pith.science/paper/BUIUJHT7

@misc{pith2026250618091,
  author       = {Pith},
  title        = {Pith review of: Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUIUJHT7}},
  note         = {Machine review of arXiv:2506.18091}
}
read the original abstract

Anaphora resolution plays a critical role in natural language understanding, especially in morphologically rich languages like Czech. This paper presents a comparative evaluation of two modern approaches to anaphora resolution on Czech text: prompt engineering with large language models (LLMs) and fine-tuning compact generative models. Using a dataset derived from the Prague Dependency Treebank, we evaluate several instruction-tuned LLMs, including Mistral Large 2 and Llama 3, using a series of prompt templates. We compare them against fine-tuned variants of the mT5 and Mistral models that we trained specifically for Czech anaphora resolution. Our experiments demonstrate that while prompting yields promising few-shot results (up to 74.5% accuracy), the fine-tuned models, particularly mT5-large, outperform them significantly, achieving up to 88% accuracy while requiring fewer computational resources. We analyze performance across different anaphora types, antecedent distances, and source corpora, highlighting key strengths and trade-offs of each approach.

Figures

Figures reproduced from arXiv: 2506.18091 by the authors.

Figure 1
Figure 1. Examples of Grammatical and Textual Anaphora in the dataset. Discourse Treebank (fiction and newspaper articles), the Prague Translation Corpus (English-Czech translations), and the Prague Spoken Corpus (transcripts of spoken Czech). Each sentence is annotated with a single anaphor-antecedent pair using XML￾style tags: <ana></ana> for the anaphor and <ant></ant> for the antecedent. In addition, metadata is provided … view at source ↗
Figure 2
Figure 2. Prompt template for the Yes/No experiment. The dataset is split into training, validation, and test subsets with the content presented in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prompt template for the question-answering experiment. The highlighted sec￾tion shows how the few-shot prompting was implemented. adds the <ant></ant> tags. For Mistral, we used LoRA (Low-Rank Adapta￾tion [7]) for parameter-efficient fine-tuning. The training hyperparameters in￾cluded the learning rate of 2e–5, batch size of 1 for mT5, and a length constraint for decoding outputs. 3.3 Evaluation Metrics The model pr… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Prompt template for the tagging experiment. The highlighted section shows how the few-shot prompting was implemented. the model’s correct understanding of coreference. The metric is suitable for both prompt-based and generative models, which may produce variable surfac…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

  1. [1]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

    Agrawal, M., et al.: Large language models are few-shot clinical information ex- tractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 1998–2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Dec 2022).https://doi.org/10.18653/v1/ 2022.emnlp-main.130,https://aclanthology.org...

  2. [2]

    Aryabumi, V., et al.: Aya 23: Open weight releases to further multilingual progress (2024),https://arxiv.org/abs/2405.15032

  3. [3]

    Dubey, A., et al.: The llama 3 herd of models (2024),https://arxiv.org/abs/ 2407.21783

  4. [4]

    Hajič, J., et al.: Prague Dependency Treebank – Consolid. 1.0, PDT-C 1.0 (2020), http://hdl.handle.net/11234/1-3185, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics ÚFAL, Faculty of Mathematics and Physics, Charles University

  5. [5]

    Hajič, J., et al.: Prague dependency treebank 3.5 (2018),http://hdl.handle.net/ 11234/1-2621, LINDAT/CLARIN digital library at the Institute of Formal and AppliedLinguisticsÚFAL,FacultyofMathematicsandPhysics,CharlesUniversity

  6. [6]

    In: Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguisticsfor Cultural Heritage,Social Sciences,Hu- manities and Literature (LaTeCH-CLfL 2024)

    Hicke, R., Mimno, D.: [Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Liter- ary Coreference Annotation with LLMs. In: Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguisticsfor Cultural Heritage,Social Sciences,Hu- manities and Literature (LaTeCH-CLfL 2024). pp. 270–277. Association for Com- putational Linguistics, St. Julians, Malta ...

  7. [7]

    In: In- ternational Conference on Learning Representations (2022),https://openreview

    Hu, E.J., et al.: LoRA: Low-Rank Adaptation of Large Language Models. In: In- ternational Conference on Learning Representations (2022),https://openreview. net/forum?id=nZeVKeeFYf9

  8. [8]

    Jiang, A.Q., et al.: Mistral 7b (2023),https://arxiv.org/abs/2310.06825

Show all 14 references
  1. [9]

    Le, N.T., Ritter, A.: Are large language models robust coreference resolvers? (2023),https://arxiv.org/abs/2305.14489

  2. [10]

    Gemma Team, Google Deepmind: Gemma 2: Improving open language models at a practical size (2024),https://arxiv.org/abs/2408.00118

  3. [11]

    Mistral AI Team: Mistral large-instruct-2407 (2024), available athttps:// huggingface.co/mistralai/Mistral-Large-Instruct-2407

  4. [12]

    Journal of machine learning research21(140), 1–67 (2020)

    Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)

  5. [13]

    In: Proceedings of the CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies

    Straka, M.: UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In: Proceedings of the CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. pp. 197–207. Association for Computational Linguistics (2018)

  6. [14]

    In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    Xue, L., et al.: mT5: A massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 483–498. Association for Computational L...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.