{"id":"923762c8-ca52-40a7-93a6-dff2f410a368","arxiv_id":"2411.19579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HECSI, a new 16K-post English and Hindi claim-span dataset, was released through an ICPR 2024 competition in which no submitted system clearly beat a simple fine-tuned transformer baseline.","lead":"The authors ran an ICPR 2024 competition on claim-span identification, where systems had to mark which spans of text in social media posts express claims. They released HECSI, a 16K-post English and Hindi dataset, and found that a simple fine-tuned transformer baseline beat most submitted systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No inter-annotator agreement is reported; the benchmark's validity rests on the unmeasured assumption that contracted annotators consistently marked all minimal claim spans.","rationale":"The reader's weakest assumption is exactly the one I find most load-bearing: without measured annotation reliability, the validity of every HECSI label is unestablished, and therefore the competition results in Table 5 are not interpretable as characterizations of current model performance on a stable task. The paper gives no internal evidence for label quality beyond the instruction text and the reputation of the annotation firm; it reports no kappa, no adjudication statistics, and no sample-level consistency checks. This is particularly concerning for span-level tasks, where boundary decisions are known to vary across annotators, and the dataset intentionally includes hard cases such as posts with multiple spans and posts with no span. The reannotation test I propose would directly settle whether the concern lands: high agreement would show that the missing IAA is only a reporting omission, while low agreement would invalidate the benchmark's foundational premise. I agree with the reader that other issues exist, notably the malformed Macro-F1 formula in Section 3 and the unsupported claim that no team significantly beat the baseline, but those are downstream of dataset validity. The conditional-accept verdict is appropriate: the paper should be accepted only if the authors add an agreement study or otherwise demonstrate label reliability, and the metric and significance issues should also be corrected.","tokens_in":8081,"tokens_out":5954,"duration_ms":53716,"concrete_test":"Draw a stratified random sample of 200 posts per language from HECSI, stratifying by no-span, single-span, and multi-span posts. Give two independent annotators the same instructions described in Section 2 without showing them the released labels, and have them mark minimal claim spans. Compute token-level pairwise F1 and Krippendorff's alpha (or Cohen's kappa on post-level claim/no-claim). If pairwise span F1 is below about 0.60 or kappa is below about 0.60, the labels and all derived leaderboard scores are too unstable to support the benchmark claim. If agreement is high, the missing IAA is a reporting gap rather than a validity threat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 describes the HECSI annotation process as performed by 'a team of human workers from a reputed annotation firm' and states that annotators were asked to judge whether a post contains a claim and, if so, mark 'the minimal span(s)' that represent the claims. However, the paper reports no inter-annotator agreement, no adjudication procedure, and no post-hoc quality check. The central claim that HECSI is a valid, publicly available benchmark with human-marked claim spans therefore rests on an unmeasured premise: that the contracted annotators consistently identified every minimal claim span and only those spans. This premise is load-bearing because every leaderboard number in Table 5 is a function of these token-level labels. If annotators disagree about span boundaries or about which posts contain claims at all, then model scores are not comparable across runs, and reported differences (e.g., JU_NLP's M-F1 of 74.8 vs. the organizer baseline of 72.6 in English) may reflect annotation noise rather than model quality. Section 2 itself highlights that HECSI contains posts with multiple claim spans and posts with no claim spans, precisely the cases where annotator disagreement is most likely, so the missing reliability measurement is not a minor omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the ICPR 2024 competition on multilingual Claim Span Identification (CSI). The authors introduce HECSI, a dataset of approximately 8,000 English and 8,000 Hindi social media posts with claim spans marked by human annotators, and describe three tracks: constrained English, constrained Hindi, and unconstrained multilingual. The evaluation treats the task as token-level binary classification and reports Macro-F1 and Jaccard scores. The paper presents the organizer baseline (fine-tuned multilingual BERT) and the solutions of nine participating teams, with JU_NLP achieving the best scores in the English and Hindi constrained tracks and FactFinders and DLRG toping the multilingual track. The conclusion states that the task remains challenging because no team significantly beat the baseline, although JU_NLP's Hindi score substantially exceeded it.","tokens_in":8279,"tokens_out":4757,"duration_ms":41290,"significance":"If the HECSI annotations are reliable, the dataset is a useful public resource for claim-span identification in English and Hindi, a task that is more granular than standard claim/non-claim classification. The competition report documents a diverse set of model solutions and provides a reproducible evaluation protocol based on token-level metrics. The public availability of the dataset and the participation of multiple teams are concrete strengths. The significance is currently tempered, however, by the absence of annotation reliability evidence and by a malformed definition of the primary evaluation metric, both of which need to be addressed before the leaderboard numbers can be fully trusted.","major_comments":[{"comment":"The printed Macro-F1 formula is malformed. As typeset, 'M-F1 = TP / 2·TP+FP+FN+TN / 2·TN+FP+FN' is not a valid mathematical expression and cannot be used to reproduce the leaderboard results. Since M-F1 is the primary ranking metric in Table 5, the formula must be corrected and the averaging procedure must be spelled out clearly, in particular whether the metric is averaged over the positive/negative classes or over individual posts after computing per-post F1.","section":"Section 3, Evaluation"},{"comment":"No inter-annotator agreement, adjudication procedure, or post-hoc quality check is reported for HECSI. The dataset is annotated entirely by a contracted annotation firm, and the claim that HECSI is a valid benchmark presupposes that annotators consistently identified all and only the minimal claim spans. The dataset contains many posts with multiple claim spans and many with no claim span (Table 3), precisely the cases where boundary and relevance decisions are hardest. Without a reliability measurement, the differences between systems in Table 5 (e.g., JU_NLP's M-F1 of 81.7 vs. the baseline's 74.2 in Hindi) could reflect annotation noise. Please report IAA on a sample and describe the adjudication procedure.","section":"Section 2, Dataset annotation"},{"comment":"The conclusion states that 'no team could achieve significantly better results than the baseline,' but no statistical significance testing is reported. In the Hindi track, JU_NLP's M-F1 is 81.7 versus the baseline's 74.2 (Table 5), a 7.5-point gap. To support this claim, the authors should either run significance tests (e.g., bootstrap or approximate randomization over test posts) or soften the wording to remove 'significantly.'","section":"Section 5, Conclusion"}],"minor_comments":[{"comment":"The Multilingual column lists 'Total #posts' as 14,097, but the three rows below it sum to 16,097, and the English and Hindi totals also sum to 16,097. This appears to be a typo and should be corrected.","section":"Table 3"},{"comment":"The captions for Tables 1 through 5 contain the typo 'T able' instead of 'Table'.","section":"Table captions"},{"comment":"The text says the metrics are 'averaged over all the posts in the test set,' but the notation of Macro-F1 is ambiguous: it is not clear whether the macro-averaging is over the two classes or over posts. Please clarify the precise computation, ideally with a corrected formula.","section":"Section 3, Evaluation"},{"comment":"The caption does not explain the meaning of '-' entries; please state that '-' indicates the team did not submit a run for that track.","section":"Table 5"},{"comment":"The text refers to 'the FactFinder team,' but the team name is FactFinders; please use a consistent name.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition overview rather than a full research article, so the lack of a detailed annotation study is understandable, but the missing IAA is a standard requirement for a dataset that underlies all reported results. The malformed metric formula and the unsupported 'significantly better' claim are also fixable. I would ask the authors to provide the annotation reliability data (the annotation firm likely has this information), correct the formula, and either add significance tests or revise the conclusion. These changes are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: HECSI is a real contribution. It is the first Hindi claim-span dataset I know of, and the bilingual English-Hindi resource is genuinely new, with about 16K human-annotated posts, including zero-span and multi-span cases. The organizers made it public, gave clean train/test splits, and ran a sane competition with an organizer baseline. If you work on claim verification or multilingual fact-checking, this is a dataset you will want to know about.\n\nWhat the paper does well is straightforward: it documents the data, the annotation protocol, the tracks, and the leaderboard, and it describes the teams' methods in enough detail to be useful. The statistics in Table 3 largely check out, and the baseline is a reasonable mBERT token classifier. No one is overselling a new method; the value is in the resource.\n\nNow the soft spots, in order of seriousness. First, the printed Macro-F1 formula in Section 3 is malformed. It does not match the standard macro-F1 definition, and since that is the headline metric, this needs a correction. The provided evaluation script was presumably correct, but the paper as written is not self-contained. Second, there is no inter-annotator agreement reported and no adjudication process described. Given that the annotation task involves multiple spans and no-span cases, the reliability of the gold labels is a genuine open question. Every leaderboard number inherits that uncertainty. This is the biggest issue, and it is not a minor omission. Third, the conclusion says no team \"significantly\" beat the baseline, but no significance testing is reported, and earlier the text says JU_NLP \"easily\" beat it. The Hindi gap of 7.5 M-F1 points might well be significant, but the paper does not demonstrate that. Fourth, a small but visible error: Table 3 lists the Multilingual total as 14,097, but the correct sum of 7,999 and 8,098 is 16,097. That looks like a typo, but it should be fixed.\n\nIs it a serious paper? Yes, as a benchmark/competition overview. It is not a methodological advance, but it fills a concrete gap and ships reproducible data. I would recommend sending it to peer review, with the expectation that the authors fix the formula, add agreement numbers or at least a quality-check procedure, run simple significance tests, and correct the table. If those are addressed, HECSI becomes a solid resource for the community.","headline":"A genuinely new Hindi-English claim-span benchmark worth having, but the paper needs metric, agreement, and table fixes before the leaderboard numbers can be taken at face value.","tokens_in":8836,"tokens_out":2876,"would_cite":true,"duration_ms":24702,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents HECSI, a publicly released benchmark of about 16,000 English and Hindi social media posts with human-marked claim spans, and reports the first competition results showing that token-level claim identification remains…","keywords":["claim span identification","HECSI dataset","Hindi-English benchmark","token classification","social media misinformation","competition overview","multilingual NLP"],"falsifier":"Re-annotate a random sample of HECSI posts (say 300 per language) with the same guidelines from two or more independent annotators; if pairwise agreement on exact span boundaries is low (for example, token-level F1 below the 74-81% leaderboard range), the benchmark's labels cannot support the reported model rankings.","tokens_in":7881,"feed_emoji":"🎯","tokens_out":6028,"duration_ms":46309,"temperature":0.7,"pith_summary":"This paper presents HECSI, a publicly available dataset of about 8,000 English and 8,000 Hindi social media posts in which human annotators marked the minimal spans that make claim-like statements. The paper's central aim is to establish HECSI as a benchmark for the claim-span identification (CSI) task, which asks a model to pick out the exact tokens that should be fact-checked, and to characterize how well current models fare on it. Using the competition results it reports, the paper argues that CSI is robustly harder than binary claim classification: fine-tuned multilingual transformers reach their best scores in the constrained tracks, while the unconstrained multilingual track saw the organizer's simple baseline outperform all submitted systems. The dataset includes posts with no claim spans and posts with multiple spans, which the paper identifies as sources of difficulty. The paper concludes that the task and dataset remain open research challenges.","feed_headline":"HECSI benchmark: best claim-spotting models hit 74-82%","feed_subtitle":"Human-marked spans from 16,000 English and Hindi posts show that finding exact claim spans is still unsolved.","key_machinery":"The load-bearing object is the HECSI dataset itself: posts with token-level labels marking each minimal claim span, produced by paid human annotators following the definition of a claim as a statement asserted as true. The evaluation machinery treats CSI as binary token classification, scoring each post with Macro-F1 and Jaccard over tokens and averaging over the test set, following the protocol introduced for the earlier CURT task. The dataset's deliberate inclusion of posts without claim spans and posts with multiple claim spans is what makes the task harder than sentence-level claim classification, and it is the mechanism that the competition results use to separate methods.","core_discovery":"The core claim is that HECSI is a workable testbed for multilingual claim-span identification: it combines the CAVES English anti-vaccine tweets and the CONSTRAINT Hindi hostile posts, re-annotated by a contracted firm with instructions to mark every minimal claim span, and split into train, validation, and test sets for the competition. Under token-level Macro-F1 and Jaccard evaluation, the best constrained-track system (JU_NLP, fine-tuned XLM-RoBERTa and multilingual BERT) scored 74.8 Macro-F1 and 54.5 Jaccard on English and 81.7 and 67.1 on Hindi, beating the organizer's multilingual BERT baseline (72.6/50.3 and 74.2/63.6). In the unconstrained multilingual track, the best submitted Macro-F1 was 59.3 (DLRG) and best Jaccard 41.4 (FactFinders), both below the baseline's 74.2 and 57.4, despite the winning teams adding external claim-span data. The paper reads these results as showing that exact claim-span detection remains difficult, that domain transfer from auxiliary datasets did not help, and that the benchmark is useful precisely because it exposes this gap.","pith_inferences":["The paper does not report inter-annotator agreement, so a natural next step is to measure agreement on a held-out sample; if token-level annotator agreement is near or below the reported leaderboard gaps, part of the ranking could be annotation noise.","HECSI's no-claim and multi-span posts make it possible to decompose errors by post type; such an error analysis could show whether models mostly fail on boundary precision or on whole-span recall.","A testable extension is to use the English HECSI portion to pretrain or adapt a model and then evaluate zero-shot on the Hindi portion, and vice versa, isolating the cross-lingual transfer signal that the multilingual track's data-pooling approach failed to exploit.","Since the English source data is vaccine-focused and the Hindi source is hostile-speech-focused, domain and language are confounded in HECSI; future annotation from matched domains could separate language difficulty from content difficulty."],"forward_implications":["HECSI gives the community a public benchmark with labeled minimal claim spans for English and Hindi, so future models can be compared on identical train, validation, and test splits.","Fine-tuned multilingual encoder models set the current practical bar for CSI; in the constrained tracks they beat instruction-tuned LLMs on token-level Macro-F1.","Throwing external claim-span datasets into multilingual training did not lift performance above a simple baseline, suggesting that more data alone is not a fix for cross-lingual span identification.","Because spans are marked at the minimal level, model outputs can double as explanations of why a post is flagged, supporting fact-checking pipelines that need justifications.","The large gap between English and Hindi constrained performance and the multilingual track indicates that combining languages in one model is currently a bottleneck rather than a benefit."],"supporting_citations":[{"why":"Supplies the CAVES anti-vaccine English tweets that form the English subset of HECSI.","marker":"[15]"},{"why":"Supplies the CONSTRAINT Hindi hostile posts that form the Hindi subset of HECSI.","marker":"[1]"},{"why":"Defines the claim-span identification task and the token-level Macro-F1/Jaccard evaluation protocol the competition adopts.","marker":"[17]"},{"why":"Establishes the prior claim-detection framing that the paper contrasts with the harder token-level span task.","marker":"[5]"},{"why":"Provides the external multilingual claim-span dataset used by top teams in the unconstrained track.","marker":"[13]"},{"why":"Provides the BERT model family used for the organizer baseline and many team submissions.","marker":"[4]"}],"fun_headline_variants":["HECSI: multilingual claim-spotting still stumps AI","Claim-span detection: best models only 74-82% on new HECSI","New HECSI benchmark: claim-span identification is still hard","Best claim-spotting models reach 74-82% on HECSI","ICPR 2024 competition: claim-spotting still fails on exact spans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on the unmeasured assumption that the hired annotators marked every minimal claim span consistently, since no inter-annotator agreement is reported.","fun_headline_variants_meta":{"raw":{"variants":["HECSI: multilingual claim-spotting still stumps AI","Claim-span detection: best models only 74-82% on new HECSI","New HECSI benchmark: claim-span identification is still hard","Best claim-spotting models reach 74-82% on HECSI","ICPR 2024 competition: claim-spotting still fails on exact spans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2831,"prompt_tokens":959,"completion_tokens":1872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1772}},"tokens_in":575,"tokens_out":1872,"duration_ms":12765,"temperature":1.0,"reasoning_tokens":1772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:02:10.851720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of HECSI posts (say 300 per language) with the same guidelines from two or more independent annotators; if pairwise agreement on exact span boundaries is low (for example, token-level F1 below the 74-81% leaderboard range), the benchmark's labels cannot support the reported model rankings.","supporting_citations":[{"cited_title":"In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Supplies the CAVES anti-vaccine English tweets that form the English subset of HECSI."},{"cited_title":"In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Defines the claim-span identification task and the token-level Macro-F1/Jaccard evaluation protocol the competition adopts."},{"cited_title":"LESA: Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content","cited_arxiv_id":"2101.11891","evidence_quote":"Establishes the prior claim-detection framing that the paper contrasts with the harder token-level span task."},{"cited_title":"In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Provides the external multilingual claim-span dataset used by top teams in the unconstrained track."}],"review_version":1}