{"id":"6b18bc02-46ad-458f-90f4-253113166008","arxiv_id":"2608.03529","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A systematic comparison of soft inter-rater reliability measures for free-text biomedical annotations finds that NLI-based measures are the most robust compromise.","lead":"This paper tests different ways to measure how well two annotators agree when they write free-text descriptions of biomedical findings. It finds that natural language inference models offer the best balance of accuracy and scalability for this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth validity in Derm1M/REFLACX is the load-bearing assumption: terminology mappings and closed-set labels may not capture true semantic equivalence of free text, so the MAE-based ranking of soft-IRR measures could reflect mapping fidelity rather than accuracy.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the standardized mappings and closed-set labels are assumed to capture the semantic agreement between free-text annotations. I agree, and I do not think the paper's synthetic experiments or its own limitation statement resolve this. The synthetic experiments provide well-controlled evidence that measures differ and that embeddings struggle with close negatives, so the paper has real value; but the real-world validation, which is what makes the conclusion 'NLI-based measures are the best compromise' plausible for actual biomedical annotation tasks, rests on an unvalidated ground truth. If the Derm1M terminology mapping is produced from the captions themselves, or if REFLACX closed-set labels are systematically coarser than the free text, then the MAE values in Table 1 are not measuring accuracy against true semantic agreement. The proposed clinician-judgment test would settle this directly. The reader already issued a CONDITIONAL verdict citing this assumption; my stress-test reinforces that condition rather than moving the verdict. I therefore recommend UNCHANGED: the paper should remain conditionally accepted, with the ground-truth validation (or explicit release of the mappings and code) as the condition.","tokens_in":19384,"tokens_out":5949,"duration_ms":75849,"concrete_test":"Take the 336 Derm1M image-caption pairs and the 109 REFLACX report pairs. Have two board-certified clinicians (dermatologists for Derm1M, radiologists for REFLACX), blinded to the standardized mappings, independently rate the semantic equivalence of each free-text pair (binary or 0–1 scale). Compute the IRR of these human judgments and use it as the gold standard. Recompute the Table 1 MAEs for all measures against this gold standard. If the relative ordering changes—for example, an embedding-based or NLI measure that was penalized by mapping mismatch improves, or a measure that matched the mapping degrades—the reported MAE ranking is an artifact of the ground-truth assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that NLI-based measures are the best compromise for soft IRR—is read off MAE comparisons against ground-truth IRR computed by exact matching on standardized labels. In §5.2, Derm1M treats captions from different web sources as independent annotators and uses the dataset's terminology mapping as ground truth; REFLACX uses each radiologist's closed-set anomaly selections. The paper does not validate that these labels faithfully capture the semantic equivalence of the free-text annotations. If the Derm1M mapping is derived from the captions or from an automatic model, the MAE measures how well each equivalence estimator reproduces that mapping, not true semantic agreement. Similarly, a free-text finding can be semantically equivalent to a closed-set label the annotator did not select, or two different codes can describe the same clinical concept. Because the final ranking (NLI best, embeddings worst) is read directly from these MAE values, the conclusion is only as secure as the ground-truth mapping assumption. This is the same concern the reader identifies as weakest, and it is not resolved by the synthetic experiments, which use clean terminology-based ground truth and may overestimate real-world transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates soft inter-rater reliability (IRR) measures for unstructured biomedical text annotations. It replaces exact label identity in Cohen's κ, Fleiss' κ, and Scott's π with continuous semantic-equivalence estimates derived from set overlap, edit distance, sentence embeddings, mutual-entailment NLI, and LLM-as-a-judge (both log-prob and verbalised variants). The methods are tested on synthetic terminology data (ICD-11, MeSH, MedDRA) with controlled synonym replacement and negative sampling, and on two real-world datasets (Derm1M and REFLACX) where free-text annotations are compared against IRR computed from standardized labels. The paper reports MAE against ground-truth IRR and concludes that the choice of equivalence measure has a significant impact, that embedding-based measures fail on semantically close negatives, and that NLI-based measures are the best compromise for biomedical annotation tasks.","tokens_in":19666,"tokens_out":12198,"duration_ms":146950,"significance":"The question is timely for exploratory biomedical annotation and literature mining, where open-ended text annotations are common but standard IRR tools assume discrete categories. The paper is one of the first to systematically compare NLI-based and LLM-based equivalence estimators for soft IRR, and the synthetic setup with known synonyms and controlled negative types is a useful contribution. The use of open-weights models, the computational cost table, and paired permutation tests strengthen reproducibility. If the empirical ranking is valid, the recommendation to prefer NLI-based measures and avoid embedding-based measures would be practically important. However, the set-level evaluation contains a technical mismatch between the estimator and the ground truth, and the real-world ground-truth assumptions are not validated; both issues directly affect the paper's central empirical claims.","major_comments":[{"comment":"Eq. (13) defines set-level equivalence as 2/(|a|+|b|) times the sum over the optimal matching, i.e. the Dice coefficient of the matched sets. The set-level synthetic ground truth, however, is the Jaccard coefficient of the concept sets before synonym replacement (Section 5.1: 'The ground truth is the Jaccard coefficient between set-pairs before synonym replacement'). These are not interchangeable: for two sets of size 2 with one shared concept, Eq. (13) returns 0.5 while Jaccard returns 1/3. Because Jaccard is a nonlinear function of the Dice coefficient, the per-item aggregation and the chance-agreement term in κ are also affected, not just a constant rescaling. Thus, even a perfect semantic-equivalence oracle would incur a large systematic MAE in the set experiments. This contaminates the set-level MAE values and rankings in Fig. 3(b), Fig. 4 (label-set panels), and the corresponding a","section":"§4.2, Eq. (13); §5.1 set-level generation"},{"comment":"The real-world evaluation treats standardized terminology mappings and closed-set anomaly labels as ground truth for semantic equivalence of free-text annotations, but this assumption is not validated. For Derm1M, captions from different web sources are treated as independent annotators, and the mapping to the dermatology terminology is used without any check of completeness or correctness; if the mapping is noisy or derived from the captions themselves, the reported MAE measures how well each estimator reproduces that mapping rather than true semantic agreement. For REFLACX, free-text findings may be semantically equivalent to closed-set anomaly labels the annotator did not select, or may describe concepts outside the closed set. Because the conclusion that NLI measures are robust and embedding measures should be avoided is read directly from Table 1, this is load-bearing. The authors s","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"The expected chance agreement p_s_e is estimated with only D=3 permutations in the single-label experiments and D=2 in the set experiments. This is a very small Monte Carlo sample, especially with high label uniqueness. No variance or sensitivity analysis for D is reported; please add a stability check or report confidence intervals for p_s_e.","section":"§4.1, Eq. (12)"},{"comment":"The statement that this is 'the first time that IRR has been quantified in this way for unstructured text annotations' is stronger than the related-work discussion supports, since Braylan et al. (2022) already compute agreement for free-text using distance functions. The novelty is better stated as the first systematic comparison of multiple semantic-equivalence measures against ground truth.","section":"§7"},{"comment":"The significance information conveyed by bold type is not visible in the plain-text version of the table. Please ensure the typeset version clearly distinguishes entries that are not significantly different from the best measure.","section":"Table 1"},{"comment":"The text does not explicitly state whether the random-negative baseline and the three hard-negative conditions use the same population sizes, D values, and equivalence-measure settings. Please clarify this in the experimental setup.","section":"Fig. 5 and Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The set-level mismatch between Eq. (13) and the Jaccard ground truth is the most serious technical issue and should be fixed before publication; it is a concrete, correctable error rather than a reason for rejection. The real-world ground-truth concern is also substantial and may require additional annotation or careful re-analysis. If the authors align the set-level ground truth and provide a validation of the free-text/label mappings, the paper's main recommendation could be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the ground-truth-anchored comparison of soft IRR measures for open-ended biomedical text, including NLI and LLM judges; nobody had systematically done that. The synthetic setup is clean—sampled concepts from ICD/MeSH/MedDRA, synonym replacement, controlled negatives—and the error analysis clearly shows embeddings conflate similarity and equivalence, which is a real result. The real-world experiments on Derm1M and REFLACX support the ranking, and the permutation tests are appropriate. I believe the central suggestion, NLI mutual entailment as the best compromise, is plausible and useful.\n\nThe soft spot is the ground truth in the real-world experiments. Derm1M has no annotators, so the authors treat each web source as an annotator. That measures cross-source caption agreement, not inter-rater reliability. The terminology mapping is used as truth, but if the mapping is incomplete or derived from the captions, the MAE tells you how well each measure reproduces that mapping. REFLACX uses closed-set anomaly selections as ground truth for free-text findings; those don't always capture semantic equivalence either. The stress-test note is correct that the final ranking is read off these MAE values. It's not fatal, because the synthetic experiments independently support NLI robustness, but it means the real-world evidence is weaker than the abstract implies. Also, no code or exact prompts are provided. That limits reproducibility of the LLM variants, which are the most expensive and least specified part of the pipeline.\n\nWho is this for: anyone designing annotation studies with open-ended text, and researchers working on semantic equivalence. It deserves a serious referee; I'd send it to review, not desk reject, with the expectation that the authors either release code/prompts and soften the real-world claims, or validate the mapping assumption.","headline":"A solid empirical comparison of soft IRR measures with a convincing NLI recommendation, but the real-world ground truth is a proxy that deserves scrutiny before the ranking is treated as settled.","tokens_in":20124,"tokens_out":1738,"would_cite":true,"duration_ms":20163,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For open-ended biomedical annotations, inter-rater reliability can be recovered by replacing exact string matches with a semantic-equivalence estimate; NLI mutual entailment is the best practical compromise.","keywords":["soft inter-rater reliability","semantic equivalence","natural language inference","biomedical text annotation","embedding similarity","LLM-as-a-judge","Cohen's kappa","free-text annotations"],"falsifier":"Take a corpus of free-text biomedical annotation pairs and have human experts judge semantic equivalence directly; compute soft IRR with the paper's NLI-based measure against human-derived agreement. If NLI-based estimates no longer match human agreement better than embedding or edit-distance baselines—or if the error is dominated by cases where the terminology mapping disagrees with human equivalence judgements—the central recommendation fails.","tokens_in":19316,"feed_emoji":"🩺","tokens_out":4538,"duration_ms":48573,"temperature":0.7,"pith_summary":"The paper tackles a blind spot in biomedical literature mining: when annotators label text with open-ended descriptions instead of predefined codes, standard inter-rater reliability (IRR) collapses because exact string matches are rare. It claims that IRR can be recovered by replacing exact label agreement with a continuous semantic-equivalence estimate, and systematically benchmarks equivalence measures—token overlap, edit distance, sentence embeddings, natural language inference (NLI), and LLM-as-a-judge—against ground-truth agreement computed on standardised terminology labels. In synthetic synonym-replacement experiments and two real-world datasets (Derm1M captions, REFLACX radiology reports), the best deep-learning measures estimate Cohen's kappa to within 0.005-0.065 mean absolute error. The paper's key finding is that the choice of measure changes both the estimate and its failure mode: embeddings are fast but treat similar-but-distinct concepts as equivalent, LLMs are most accurate but too slow for estimating chance agreement, and NLI mutual entailment is the recommended compromise.","feed_headline":"Pick NLI to measure agreement in free-text medical labels","feed_subtitle":"A benchmark against standardised ground truth shows embeddings fail on similar-but-distinct concepts; mutual entailment is the compromise.","key_machinery":"The load-bearing identity is soft Cohen's kappa: kappa_s = (p_s_o - p_s_e)/(1 - p_s_e), where p_s_o is the mean pairwise semantic equivalence and p_s_e is the expected equivalence under a permuted distribution of the second annotator's labels. Equivalence itself is estimated by mutual entailment, P(a≡b) = sqrt(P(a⇒b) * P(b⇒a)), computed by an NLI model or an LLM judge, with embeddings and edit distances serving as alternative estimators. The set-valued extension solves a weighted bipartite assignment with the Hungarian algorithm before averaging the matched equivalences.","core_discovery":"The paper establishes soft IRR for unstructured text annotations: define the observed agreement as the mean semantic equivalence between paired annotations, estimate chance agreement from the expected equivalence between randomly permuted annotation pairs, and plug both into Cohen's kappa. Equivalence is estimated as the probability of mutual entailment—the geometric mean of NLI entailment probabilities in both directions—for NLI and LLM variants, as cosine similarity for embeddings, and as string-based scores for edit and set-overlap measures. For set-valued annotations, pairwise equivalences are first matched with the Hungarian algorithm. Evaluated against exact-match IRR on ICD-11, MeSH a","pith_inferences":["The same equivalence-based IRR machinery could be applied to evaluating human-versus-LLM extraction agreement in systematic-review mining, where the goal is matching underlying information rather than exact codes.","A testable extension: replace terminology-mapping ground truths with direct human judgements of pairwise semantic equivalence of free-text annotations; if NLI-based estimates still track human agreement best, the recommendation holds regardless of mapping noise.","Because embeddings fail on near-synonym negatives, their use in any equivalence-based metric should be gated behind calibration or thresholding before being fed into agreement statistics.","The equivalence estimates are measure-agnostic, so the same soft IRR framework transfers to other agreement statistics such as Krippendorff's alpha, which the paper notes in passing; the ranking of measures would likely transfer as well."],"forward_implications":["Soft IRR with semantic equivalence recovers ground-truth agreement on standardised labels to within 0.005-0.065 MAE on real-world datasets, accurate enough for routine use in annotation studies.","Embedding-based equivalence measures should be avoided for biomedical annotation agreement because they conflate similarity with equivalence and fail on close, non-equivalent concepts.","NLI-based measures (PubMedBERT-NLI, mDeBERTa-v3) are the recommended compromise: decent scalability and stable performance across random, equality, generalisation and lexical negative types.","LLM-based measures are the most accurate but their computational cost makes Monte Carlo estimation of chance agreement impractical at scale.","All soft measures degrade less steeply than exact matching as true agreement rises, so exact matching systematically underestimates reliability when annotators use synonymous wording."],"supporting_citations":[{"why":"Defines the Cohen's kappa statistic that the paper softens and uses as the main evaluation target.","marker":"(Cohen, 1960)"},{"why":"Prior work on soft IRR with distance functions; supplies baseline measures and the additional sigma and KS IRR statistics.","marker":"(Braylan et al., 2022)"},{"why":"Establishes mutual entailment as a signal for semantic equivalence in paraphrase detection, motivating the NLI-based estimator.","marker":"(Nighojkar and Licato, 2021)"},{"why":"Uses mutual entailment for content preservation in paraphrase generation, supporting the equivalence formulation.","marker":"(Babakov et al., 2022)"},{"why":"Provides the Hungarian assignment algorithm used to match set-valued annotations before averaging equivalences.","marker":"(Kuhn, 1955)"},{"why":"Supplies the REFLACX dataset with free-text radiologist annotations and closed-set anomaly labels as ground truth.","marker":"(Bigolin Lanfredi et al., 2022)"},{"why":"Supplies the Derm1M dataset with free-text captions and standardised terminology mappings as ground truth.","marker":"(Yan et al., 2025)"},{"why":"Provides the PubMedBERT-based NLI model used as the specialist NLI equivalence measure.","marker":"(Deka et al., 2023)"},{"why":"Provides the mDeBERTa-v3 NLI model used as the generalist NLI equivalence measure.","marker":"(Laurer et al., 2022)"},{"why":"Provides the MedNLI data used to fine-tune the specialised PubMedBERT NLI model.","marker":"(Shivade, 2019)"}],"fun_headline_variants":["Measure free-text annotation agreement via NLI mutual entailment","NLI compromise for soft IRR in biomedical free-text","Free-text medical labels: Use NLI to measure rater agreement","Mutual entailment gauges agreement in unstructured biomedical text","NLI-based soft IRR for free-text annotations in biomedicine"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The evaluation assumes that the standardised labels used as ground truth—exact concept identities in synthetic data, terminology mappings in Derm1M, and the closed anomaly set in REFLACX—capture exactly the semantic agreement between the free-text annotations.","fun_headline_variants_meta":{"raw":{"variants":["Measure free-text annotation agreement via NLI mutual entailment","NLI compromise for soft IRR in biomedical free-text","Free-text medical labels: Use NLI to measure rater agreement","Mutual entailment gauges agreement in unstructured biomedical text","NLI-based soft IRR for free-text annotations in biomedicine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":1882,"prompt_tokens":619,"completion_tokens":1263,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":363,"completion_tokens_details":{"reasoning_tokens":1179}},"tokens_in":363,"tokens_out":1263,"duration_ms":10314,"temperature":1.0,"reasoning_tokens":1179,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:16:25.617520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus of free-text biomedical annotation pairs and have human experts judge semantic equivalence directly; compute soft IRR with the paper's NLI-based measure against human-derived agreement. If NLI-based estimates no longer match human agreement better than embedding or edit-distance baselines—or if the error is dominated by cases where the terminology mapping disagrees with human equivalence judgements—the central recommendation fails.","supporting_citations":[{"cited_title":"and Chan, Jessica and Duong, Phuong-Anh T","cited_arxiv_id":null,"evidence_quote":"Supplies the REFLACX dataset with free-text radiologist annotations and closed-set anomaly labels as ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mDeBERTa-v3 NLI model used as the generalist NLI equivalence measure."},{"cited_title":"doi:10.13026/C2RS98 , url =","cited_arxiv_id":null,"evidence_quote":"Provides the MedNLI data used to fine-tune the specialised PubMedBERT NLI model."}],"review_version":1}