Pith. sign in

REVIEW 6 major objections 4 minor 30 references

MIRA-Ev:A Benchmark for Granular Evidence Detection and Relational Reasoning in Clinical Exams

T0 review · 6 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MIRA-Ev: a clinical reasoning benchmark that checks whether a model's diagnosis is actually grounded in the right evidence, not just whether the final answer is correct.

desk verdict A genuinely useful clinical reasoning resource, but the paper's empirical and multilingual claims run ahead of the evidence actually shown. read the letter →

arxiv 2607.19201 v1 pith:J7HX5JG6 submitted 2026-07-21 cs.CL cs.AI

Iker De la Iglesia , Johanna Ramirez-Romero , Jose Maria Villa-Gonzalez , Irune Urroz García , Ander Barrena , Aitziber Atutxa This is my paper · ORCID
classification cs.CLcs.AI
keywords clinicalargumentminingMIRexammultilingualNLPevidencesentenceretrievalargumentativecomponentextractionsupport/attackrelationsBasquereasoningtraceevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MIRA-Ev is a new benchmark built from Spanish medical licensing exams that evaluates clinical reasoning by looking at the evidence a model uses, not just the answer it gives. Expert clinicians re-annotated exam cases with fine-grained argumentative structure: spans of text marked as premises or claims, and directed relations of support or attack between them. The dataset is released in parallel Spanish, English, and Basque versions, making it the first clinical argumentation resource in Basque. The paper argues that this resource can reveal failure modes that standard multiple-choice benchmarks miss, such as a model selecting the right diagnosis while relying on irrelevant, absent, or contradictory evidence.

What carries the argument

The central object is the argumentative structure annotated over clinical case vignettes: spans of text typed as Premise or Claim, connected by directed Support or Attack relations. The key mechanism is the deliberate inversion of the standard mapping between candidate options and claims in a subset of cases. Normally, options are claims and the case text provides premises; in inverted cases, an option functions as a premise and the claim is embedded in the case stem. This inversion prevents models from using the positional heuristic 'options are claims' and forces them to infer argumentative role from content, allowing the benchmark to directly probe whether a system understands argumentati

What would settle it

A direct falsifier would be an inter-annotator agreement study on a random sample of the corpus: if expert clinicians disagree on more than a small fraction of span boundaries or relation polarities, or if translated versions fail to align with the native annotations on a per-span basis, the reliability of the benchmark as a gold standard collapses. Similarly, if the Basque version was produced by translation rather than native annotation, and a controlled study shows that models perform differently on translated versus native Basque text, the claim of a 'native' resource would be weakened.

Watch

Extended reading notes

Core claim

The central claim is that clinical NLP evaluation should move beyond answer-only accuracy to inspect the reasoning trace, and that argument mining provides a workable formalism for doing so at scale. The paper introduces a three-tier task hierarchy—evidence sentence retrieval, argumentative component extraction, and relation classification—and shows empirically that current encoder-based systems perform well on retrieving relevant sentences (macroF1 ~73) but collapse on relation classification (strict macroF1 ~7.6). A key finding is that claim detection is inflated by a positional shortcut: because candidate options usually map to claims, models can 'look up' claims from options, and perform

Load-bearing premise

The gold annotations—both the span boundaries and the support/attack relations—must be correct and consistent across the Spanish, English, and Basque versions; the paper does not report inter-annotator agreement or an annotation/projection protocol for the parallel versions.

Editorial extensions

If this is right

  • Benchmarks like MIRA-Ev can detect 'right answer, wrong reasoning'—a failure mode that multiple-choice QA cannot see—and thus offer a more informative signal for clinical model evaluation.
  • The inverted-mapping subset provides a concrete diagnostic for whether a model has learned positional shortcuts; a model that performs well on standard instances but poorly on inverted ones is not reasoning about argumentative roles.
  • The strict-versus-relaxed boundary findings show that span-boundary noise, not semantic misunderstanding, is the main bottleneck in component extraction, suggesting that evaluation metrics and model architectures should focus on boundary robustness.
  • The very low relation classification scores indicate that directed support/attack reasoning in clinical text is a largely unsolved problem, pointing to a clear research target for clinical NLP.
  • The Basque version of the dataset opens a new direction for low-resource clinical reasoning evaluation, allowing progress in a language with essentially no prior annotated reasoning data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's emphasis on the reasoning trace could be extended to evaluate explanation generation tasks: instead of scoring a generated explanation as a whole, one could use MIRA-Ev's annotation schema to check whether each stated premise is grounded in the case and whether the claimed support/attack relations match gold evidence.
  • The inverted-mapping design suggests a general principle for building diagnostic benchmarks: deliberately break syntactic heuristics (position, punctuation) to force deeper semantic processing. The same trick could be applied to other NLP tasks where models exploit positional shortcuts, such as reading comprehension or fact verification.
  • The pattern of over-predicting sentence relevance (high recall, lower precision on the Not-Relevant class) implies that clinical evidence retrieval models might be biased toward treating all case content as relevant; a calibration-aware evaluation could reveal whether this is a dataset effect or a model bias.
  • If the relation classification bottleneck is partly due to cascading span errors, an end-to-end architecture trained jointly on extraction and relation classification might substantially outperform the sequential pipeline, a testable hypothesis the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The manuscript introduces MIRA-Ev, a clinical argument-mining benchmark built from Spanish MIR exam cases, re-annotated by clinicians with span-level premises and claims plus directed support/attack relations. It proposes a three-tier task hierarchy (evidence sentence retrieval, component extraction, relation classification) and parallel Spanish, English, and Basque releases, claiming the first clinical argumentation resource in Basque. The experimental section reports a single EriBERTa encoder pipeline on the Spanish version across the three tiers, with strict and relaxed matching. The authors report Task 1 macro-F1 73.48, Task 2 macro-F1 62.63/68.02 strict/relaxed, Task 3 macro-F1 7.58/10.61, and an aggregate 'Pipeline Overall' of 47.90 repeated across all result tables. The discussion interprets the high claim-detection F1 as a positional-shortcut artifact and attributes low relation F1 to cascading extraction errors.

Significance. If properly validated, MIRA-Ev would address a real gap: MCQA benchmarks cannot verify whether a correct diagnosis is grounded in relevant evidence, and Basque has essentially no clinical argumentation resources. The task hierarchy and strict/relaxed matching protocol in §3.1–§4.1 are clearly defined, and the deliberate preservation of inverted claim/premise mappings is a useful probing device. The promise of a trilingual, token-level clinical argumentation benchmark with expert annotations is significant for multilingual clinical NLP. However, as submitted, the central resource claims are not supported by the reported validation: there is no inter-annotator agreement, no multilingual construction protocol, and no experimental evidence for the LLM and cross-lingual evaluations promised in the introduction. The novelty rests heavily on the Basque and multilingual claims, so the missing validation is load-bearing.

major comments (6)
  1. [§3.3; contributions (i), (v); abstract] The abstract and contributions claim parallel Spanish/English/Basque versions and 'the first clinical argumentation resource in Basque,' but §3.3 consists of a single sentence with no translation/projection protocol. There is no description of human translation, machine translation plus post-editing, span alignment, or clinician validation per language; Table 1 reports only aggregate counts. Because the non-Spanish versions are the basis of the novelty claim, the manuscript must specify how the English and Basque versions were constructed, how sub-sentential spans and directed relations were preserved, and provide per-language validation.
  2. [§1; §5; Tables 2–4] The introduction and contribution (iv) promise benchmarking with 'a range of medical-pretrained and general-purpose generative LLMs' across tiers, languages, and the standard/inverted mapping. The Results section contains only one 'Encoder + Classifier' pipeline, with no LLM results, no per-language results, and no explicit inverted-mapping results. Either add the missing tables or revise the abstract and contributions to state that LLM and multilingual evaluation are future work. As written, the stated empirical scope is unsupported.
  3. [§5, Tables 2–4] The same 'Pipeline Overall' value 47.90 appears in all six rows of Tables 2–4, including both strict and relaxed rows. The explanation that the score is 'shared across all three tables' because it reflects the end-to-end pipeline run does not define the metric; moreover, relaxed Task 2 (68.02) and Task 3 (10.61) differ from their strict counterparts, so an unchanged aggregate is unexpected. Specify the formula (e.g., macro-average of task F1s) and report strict and relaxed aggregates separately, or remove the duplicate column.
  4. [§3.2–§3.3; §4.1] No inter-annotator agreement is reported for the Spanish source, nor for the English and Basque versions. Since MIRA-Ev's value is as gold ground truth for span-level premises/claims and directed relations, the reliability of these labels is central. Please report IAA on at least a held-out subset (span-boundary agreement and relation-level agreement, separately by language), and describe the annotation guidelines, adjudication, and clinician background.
  5. [§6, 'Error Propagation in Relation Classification'; Table 4] The Discussion attributes the low Task 3 F1 (7.58 strict / 10.61 relaxed) to cascading upstream extraction errors, but no experiment isolates this claim. An oracle relation-classification run with gold spans would provide an upper bound; without it, the observed scores could equally reflect relation-model deficiency. Add an oracle/upper-bound experiment or soften the causal interpretation.
  6. [§6, 'Claim Detection: Reliance on Structural Heuristics'] The Discussion asserts that performance 'degraded sharply' on inverted-mapping cases where the claim is embedded in the narrative and options serve as premises, and that models relied on positional heuristics. No table or statistic reports the size of the inverted subset or the standard/inverted results. This is the central evidence for the argumentative-role ambiguity claim; please report subset counts and strict/relaxed F1 separately for standard and inverted mappings.
minor comments (4)
  1. [§3.2] The source data description says prior annotations and clinician explanations were 'stripped' from CasiMedicos, but no details are given on licensing, redistribution rights, or data availability. Since the manuscript claims the resource is released, include a repository URL and license.
  2. [§5] The notation in Tables 2–4 is dense and sometimes unclear (e.g., 'Overall Component Detection' vs 'Pipeline Overall'). Consider adding a legend that explains which columns are macro-averaged and which are per-class.
  3. [§4.2.1] Task 1 is defined as binary sentence classification, but the implementation uses the [SEP] token representation. Clarify how the relevance label for the sentence is obtained from the token-classification head in practice.
  4. [References] A few references are incomplete or informal (e.g., 'Clinical trials' under World Health Organization, 2021). Please verify all citations and expand them where needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's gold annotations are human ground truth independent of the evaluated models, and the authors' self-citations are not load-bearing for the central resource claim.

full rationale

MIRA-Ev is a resource-construction paper: the central claim is that expert clinicians re-annotated MIR exam cases with span-level premises, claims, and support/attack relations, and that the resource is released in Spanish, English, and Basque. The gold labels are human annotations, external to the models being evaluated, so the benchmark is not circular by construction. The three-tier task hierarchy reorganizes existing exam content, but the evaluation targets are independently defined by human judgment rather than derived from the evaluated systems. The only self-citations—EriBERTa (De la Iglesia et al., 2025b) as an encoder baseline and De la Iglesia et al. (2025a) for MCQA critique—support motivation or provide a baseline; neither reduces the benchmark claim to an input of the paper. Section 6 explicitly acknowledges that high claim-detection F1 is largely a structural artifact of options mapping to claims and does not present this as evidence of capability; the inverted-mapping subset is designed to probe that shortcut. The absence of a multilingual construction protocol and inter-annotator agreement in Section 3.3 is a validity limitation and should be addressed, but it is not a circularity: unverified cross-lingual equivalence does not make the benchmark's reasoning trace equivalent to its inputs. Overall, the derivation chain is self-contained against external ground truth, so no circular step is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

All claims rest on human gold labels, the de-annotation of CasiMedicos cases, and the cross-lingual equivalence of span-level annotation; none of these are externally verified (no IAA, no data release). One evaluation threshold (τ=0.5) is hand-set and affects relaxed F1.

free parameters (1)
  • Relaxed matching IoU threshold τ = 0.5
    Hand-chosen in §4.1 for relaxed span matching; reported relaxed F1 scores depend directly on this threshold.
assumptions (5)
  • domain assumption MIRA-Ev gold annotations are accurate and consistent.
    Expert clinicians created the labels, but no inter-annotator agreement or adjudication protocol is reported (§3.3, §4).
  • domain assumption De-annotated MIR cases preserve the original clinical content and options.
    §3.2 states prior annotations/explanations were stripped; no integrity check is described.
  • domain assumption Parallel English/Basque versions preserve argumentative spans and relations.
    §3.3 asserts parallel versions without describing translation/projection or validation; token-level boundaries rarely align across languages.
  • domain assumption EriBERTa is an uncontaminated baseline.
    EriBERTa is the authors' private-data encoder; no overlap analysis with MIR/MIRA-Ev text is provided.
  • standard math Standard definitions of macro-P/R/F1 and Jaccard overlap apply as stated.
    Used in §4.1 without derivation; uncontroversial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIRA-Ev:A Benchmark for Granular Evidence Detection and Relational Reasoning in Clinical Exams." pith.science (2026). https://pith.science/paper/J7HX5JG6

@misc{pith2026260719201,
  author       = {Pith},
  title        = {Pith review of: MIRA-Ev:A Benchmark for Granular Evidence Detection and Relational Reasoning in Clinical Exams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7HX5JG6}},
  note         = {Machine review of arXiv:2607.19201}
}
read the original abstract

Clinical NLP evaluation remains dominated by multiple-choice question answering (MCQA), which scores only final-answer accuracy and cannot detect when a model reaches the correct diagnosis while grounding it in irrelevant, absent, or contradictory evidence. We introduce MIRA-Ev, a clinical argument mining benchmark built on Spanish M\'edico Interno Residente (MIR) licensing-exam cases, re-annotated by expert clinicians with span-level premises, claims, and directed support/attack relations, and released in parallel Spanish (native), English, and Basque versions, the first clinical argumentation resource in Basque. MIRA-Ev organizes evaluation into a three-tier task hierarchy: evidence sentence retrieval, argumentative component extraction, and relation classification.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 1 canonical work pages

  1. [1]

    BMJ , volume =

    Evidence based medicine: what it is and what it isn't , author =. BMJ , volume =. 1996 , publisher =

    1996

  2. [2]

    Artificial Intelligence in Medicine , volume =

    Explanatory argument extraction of correct answers in resident medical exams , author =. Artificial Intelligence in Medicine , volume =. 2024 , publisher =

    2024

  3. [3]

    BMC Medical Informatics and Decision Making , volume =

    A clinical trials corpus annotated with UMLS entities to enhance the access to Evidence-Based Medicine , author =. BMC Medical Informatics and Decision Making , volume =. 2021 , doi =

    2021

  4. [4]

    Nature Communications , volume =

    Benchmarking large language models for biomedical natural language processing applications and recommendations , author =. Nature Communications , volume =. 2025 , publisher =

    2025

  5. [5]

    Findings of the Association for Computational Linguistics: EMNLP 2022 , pages =

    Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings , author =. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages =. 2022 , address =

    2022

  6. [6]

    Artificial Intelligence in Medicine , volume =

    Enhancing evidence-based medicine with natural language argumentative analysis of clinical trials , author =. Artificial Intelligence in Medicine , volume =. 2021 , publisher =

    2021

  7. [7]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review , author =. Findings of the Association for Computational Linguistics: ACL 2025 , pages =. 2025 , address =

    2025

  8. [8]

    and González-Barba, José Ángel, and Chiruzzo, Luis , title =

    Bonet-Jover, A. and González-Barba, José Ángel, and Chiruzzo, Luis , title =. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2026) , year =

    2026

Show all 30 references
  1. [9]

    Journal of Biomedical Informatics , volume =

    TransforMED: End-to-End Transformers for Evidence-Based Medicine and Argument Mining in medical literature , author =. Journal of Biomedical Informatics , volume =. 2021 , publisher =

    2021

  2. [10]

    Proceedings of the 24th Workshop on Biomedical Language Processing , pages =

    Overview of the ArchEHR-QA 2025 Shared Task on Grounded Question Answering from Electronic Health Records , author =. Proceedings of the 24th Workshop on Biomedical Language Processing , pages =. 2025 , address =

    2025

  3. [11]

    Revista de la Sociedad Espa

    Medical Argument Mining: Exploitation of Scarce Data Using NLI Systems , author =. Revista de la Sociedad Espa

  4. [12]

    2025 , address =

    Proceedings of the 12th Argument Mining Workshop , editor =. 2025 , address =

    2025

  5. [13]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , address =

    2024

  6. [14]

    AMIA Annual Symposium Proceedings , volume =

    Clinical text annotation - what factors are associated with the cost of time? , author =. AMIA Annual Symposium Proceedings , volume =. 2018 , organization =

    2018

  7. [15]

    Proceedings of the 3rd Machine Learning for Health Symposium , series =

    LLMs for Patient Summarization: A Comparative Study , author =. Proceedings of the 3rd Machine Learning for Health Symposium , series =

  8. [16]

    arXiv preprint arXiv:2108.09913 , year =

    Analyzing the Granularity and Cost of Annotation in Clinical Sequence Labeling , author =. arXiv preprint arXiv:2108.09913 , year =

    arXiv

  9. [17]

    Clinical trials , year =

  10. [18]

    A clinical trials corpus annotated with

    Campillos-Llanos, Leonardo and Valverde-Mateos, Ana and Capllonch-Carri. A clinical trials corpus annotated with. 2021 , publisher =. doi:10.1186/s12911-021-01395-z , url =

    2021 doi

  11. [19]

    Computational Models of Argument - Proceedings of

    Zakhir-Puig, Sof. Computational Models of Argument - Proceedings of

  12. [20]

    2025 , keywords =

    Iker. 2025 , keywords =

    2025

  13. [21]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation,

    Iker Garc. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation,. 2024 , url =

    2024

  14. [22]

    Anar Yeginbergen and Rodrigo Agerri , title =. Proces. del Leng. Natural , volume =. 2024 , url =

    2024

  15. [23]

    HiTZ@Antidote: Argumentation-driven Explainable Artificial Intelligence for Digital Medicine , booktitle =

    Rodrigo Agerri and I. HiTZ@Antidote: Argumentation-driven Explainable Artificial Intelligence for Digital Medicine , booktitle =. 2023 , url =

    2023

  16. [24]

    Ranking Over Scoring: Towards Reliable and Robust Automated Evaluation of LLM-Generated Medical Explanatory Arguments , booktitle =

    Iker. Ranking Over Scoring: Towards Reliable and Robust Automated Evaluation of LLM-Generated Medical Explanatory Arguments , booktitle =. 2025 , url =

    2025

  17. [25]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Sviridova, Ekaterina and Yeginbergen, Anar and Estarrona, Ainara and Cabrio, Elena and Villata, Serena and Agerri, Rodrigo. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024

    2024

  18. [26]

    Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    CT CL: A Cross-Language Benchmark for Matching Patients to Clinical Trials , author=. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  19. [27]

    Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

    Jeong, Daniel P and Garg, Saurabh and Lipton, Zachary Chase and Oberst, Michael. Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653...

    2024 doi

  20. [28]

    2024 , eprint=

    Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data , author=. 2024 , eprint=

    2024

  21. [29]

    ClinicalBench: Can

    Canyu Chen and Jian Yu and Shan Chen and Che Liu and Zhongwei Wan and Danielle Bitterman and Fei Wang and Kai Shu , year=. ClinicalBench: Can

  22. [30]

    arXiv preprint arXiv:2604.06854 , year=

    To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models , author=. arXiv preprint arXiv:2604.06854 , year=

    arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.