{"id":"6472a7f7-2166-4b81-b027-55b8a5e9335b","arxiv_id":"2507.00543","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LLMs alone annotate search clarifications unreliably; adding confidence-based selective human review cuts effort 24-45% in simulation, but the evaluation is partly built from the ground truth it predicts.","lead":"This paper tests four large language models on five fine-grained search clarification annotation tasks and finds they agree poorly with human labels. It then proposes a human-in-the-loop workflow, simulated by substituting existing human labels for flagged cases, and reports 24-45% effort reduction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HITL quality is measured after replacing 55-76% of test labels with ground-truth human labels, so the reported 24-45% effort reduction is an upper bound and the automatically accepted subset is never isolated.","rationale":"The paper's qualitative finding—that the evaluated LLMs struggle with subjective, fine-grained annotation tasks—is plausible and supported by the per-model results, and the authors deserve credit for releasing code/data and using human labels independent of LLM outputs. However, the quantitative HITL claim is not supported as stated. The evaluation simulates human review by substituting pre-existing majority-voted human labels on flagged instances; because flagged instances constitute 55-76% of the test set at the reported HER values, the reported agreement metrics are largely tautological. The only way to know whether the HITL pipeline actually preserves quality is to measure the automatically accepted subset, which the paper never does. Threshold selection compounds the problem: optimal thresholds are chosen on a 10% subset using the same substitution, so configurations that flag more instances (or flag the right instances) are rewarded regardless of whether the LLM ensemble would have produced acceptable labels on the accepted set. The uncertainty proxy based on SD of confidence scores is also questionable, but the perfect-substitution design is the more fundamental issue. A concrete re-analysis separating accepted and flagged subsets, plus a noisy-reviewer simulation, would settle whether the 24-45% effort reduction claim survives. Because the reader's REJECT verdict already identifies this evaluation flaw, my assessment does not change the verdict.","tokens_in":21232,"tokens_out":4312,"duration_ms":54916,"concrete_test":"Recompute the Table 2 HITL rows using only the automatically accepted instances (the complement of flagged instances at each reported optimal threshold), and report Kw, Macro F1, and MAE on that subset against the MIMICS-Duo human labels. Also rerun the Final Step substituting, for each flagged instance, the label of a single randomly chosen MIMICS-Duo annotator instead of the majority label, repeated over at least 100 bootstrap draws, and compare the resulting Kw to the reported values. If the accepted-subset Kw falls below the 0.7 threshold, or if the bootstrap Kw drops materially (e.g., by more than 0.05), the claimed 24-45% effort reduction with maintained quality is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—HITL maintains reliability while cutting human effort by 24-45%—rests on the Final Step of Section 4.4: for every flagged instance, the aggregated LLM label is replaced with the pre-existing MIMICS-Duo majority-voted human label before computing agreement. Since HER (Eq. 2) is the percentage of instances not flagged, the reported HITL rows in Table 2 are mixtures of 24-45% LLM-aggregated labels and 55-76% ground-truth labels (e.g., Task 1 has 45% HER, so 55% of the evaluation set is human by construction; Task 2 has 74% human by construction). Comparing this mixture to the human labels guarantees perfect agreement on a majority of test instances, so the weighted kappas of 0.757-0.847 are upper bounds rather than estimates of live HITL quality. The paper never reports quality on the automatically accepted subset alone, which is the only subset where the LLM ensemble actually determines the output. The Discussion's note that the framework 'simulated human-in-the-loop decision-making using majority-voted crowdsourced labels' acknowledges the simulation but does not address its optimism: live reviewers are not guaranteed to reproduce the majority label, and even if they did, the evaluation still cannot separate routing accuracy from label quality. Thresholds are also selected on the 10% subset using the same substitution, so the Pareto selection is biased toward configurations that look good under this construction. The uncertainty rule (SD of verbalised confidence scores, Step 3) is a secondary weakness—confidently opposed models can produce low SD and evade flagging—but the perfect-substitution flaw is sufficient to undermine the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates four LLMs (GPT-4o, Claude 3, Cohere Command R, and Mistral 7B) as annotators for three fine-grained search clarification tasks from MIMICS-Duo (list-wise preference, pair-wise quality, and pair-wise aspect labelling with coverage, diversity, and option-order sub-tasks). It reports that LLMs are poorly calibrated and sensitive to prompts and temperature, and it proposes a human-in-the-loop (HITL) workflow that flags low-confidence or high-variance instances for human review. The paper claims that this workflow reduces human effort by 24-45% while keeping annotation quality comparable to full human annotation, with weighted Cohen's kappas of 0.757-0.847 across tasks. The evaluation, however, is conducted by substituting the pre-existing majority-voted human labels for every flagged test instance, which makes the reported quality metrics partly circular.","tokens_in":21511,"tokens_out":7275,"duration_ms":80485,"significance":"The strengths of the paper are its systematic comparison of multiple proprietary and open-source LLMs, the public release of code and prompts, the inclusion of a temperature and prompt sensitivity analysis, and the use of a real, multi-dimensional annotation dataset. If the HITL claim were supported, the result would be practically useful for reducing annotation cost in nuanced IR tasks, and the task-level analysis of when human oversight is most needed is a helpful qualitative contribution. However, the central quantitative claim rests on an evaluation design that guarantees perfect agreement on a majority of test instances, so the current evidence does not establish the claimed trade-off. The paper is in scope for ICTIR and the data release is valuable, but the headline result needs substantial re-analysis.","major_comments":[{"comment":"The headline HITL evaluation is circular. The manuscript states that for any instance flagged for human review, \"we substitute the aggregated model label with the corresponding pre-existing human label,\" and those pre-existing labels are exactly the majority-voted MIMICS-Duo labels used as ground truth in Table 2. Since HER (Eq. 2) is the percentage of instances not flagged, the HITL rows in Table 2 contain the ground-truth label on 55-76% of the evaluation instances (e.g., Task 1 has 45% HER, so 55% of labels are human by construction; Task 2 has 26% HER, so 74% are human by construction). Agreement on those instances is perfect by construction, so the reported weighted kappas of 0.757-0.847 are upper bounds on live HITL quality rather than estimates of it.","section":"§4.4, Final Step; §4.5, Eq. (2)"},{"comment":"The same substitution is used during threshold selection, so the Pareto front in Figure 3 and the reported optimal thresholds are also selected under the circular construction. For each candidate threshold pair, the flagged labels on the 10% subset are replaced by the human labels before metrics are computed, and the best configuration is chosen by maximizing weighted kappa while minimizing effort. This biases the selection toward configurations that look good under the substitution rule, and the bias propagates to the full-data results. Thresholds should be selected using quality measured only on the automatically accepted subset, or on a separate validation split where flagged labels are not overwritten.","section":"§4.4, Steps 5-6"},{"comment":"The paper never reports quality on the automatically accepted subset alone. The only instances for which the LLM ensemble actually determines the output are those not flagged (24-45% of the test portion), and no macro precision, kappa, or MAE is reported for that subset. Without this isolation, the claim of \"no considerable quality degradation\" is unsupported, because a high overall kappa can be driven entirely by the substituted majority. The authors should report agreement on accepted instances separately and, for the flagged instances, evaluate the routing decision itself, for example by comparing the original LLM label to the human label rather than replacing it.","section":"§5.1-5.3, Table 2"},{"comment":"The acknowledgement that the framework \"simulated human-in-the-loop decision-making using majority-voted crowdsourced labels\" does not address the optimism of the simulation. Live reviewers are not guaranteed to reproduce the majority label, and even if they did, the evaluation still conflates routing accuracy with label quality. The alternative operating point mentioned in the Discussion (if a lower Kw of 0.718 is acceptable, effort reduction increases to 60%) is derived from the same substitution and therefore inherits the same bias.","section":"§6, Discussion"},{"comment":"The reported effort reduction does not account for the full cost of the initial human-annotated subset. HER (Eq. 2) is computed only on the remaining 90% of the data, but the workflow requires fully human-annotating a 10% tuning subset before any thresholds can be selected; in a deployment from scratch, that cost must be included in the comparison against full human annotation. The statement in Section 5.3 that \"even after factoring in the 10% setup cost, the overall reduction in manual effort remained substantial\" is not quantified, and the actual net reduction is lower than 24-45% once the 10% subset is included.","section":"§4.4, Step 1; §5.3"}],"minor_comments":[{"comment":"One panel labels the model \"Mistral B7\" instead of \"Mistral 7B\"; please correct the typo.","section":"Figure 5"},{"comment":"Several references are incomplete; for example, reference [10] lists only an arXiv identifier with no year or venue. Please complete the bibliography.","section":"References"},{"comment":"The table does not indicate which confidence and standard-deviation thresholds correspond to each HITL row; adding threshold values to the table or caption would improve reproducibility.","section":"Table 2"},{"comment":"The standard deviation of verbalized confidence scores across models is treated as a proxy for inter-model disagreement, but the models use different confidence scales; this assumption should be validated or at least discussed more explicitly.","section":"§4.4, Step 3"},{"comment":"The phrase \"considered acceptable based on ICTIR CFP\" is informal for a paper text; consider rephrasing the limitation statement.","section":"§7, Conclusions"}],"recommendation":"major_revision","confidential_remarks":"I am recommending major revision rather than rejection because the authors already have the raw LLM predictions and human labels needed to re-analyze the data without substituting ground truth into the evaluation; the current version, however, does not support the headline claim. If the authors can report quality on the automatically accepted subset alone, redo threshold selection without the substitution, and quantify the setup cost, the result may be salvageable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its empirical comparison of four LLMs on the MIMICS-Duo search clarification annotation tasks, and the authors ship their code, prompts, and annotations, which is real credit. The finding that GPT-4o, Claude 3, Cohere, and Mistral all struggle with subjective ordinal rating tasks, and are sensitive to prompt and temperature variation, is a solid, useful negative result for the IR evaluation community. That part holds up.\n\nThe central HITL claim, however, is not supported by the evaluation as designed. In the Final Step of Section 4.4, every instance flagged for human review is given the pre-existing MIMICS-Duo majority human label before agreement is computed. Since the reported HER values mean 55-76% of the test set falls into that bucket, the weighted kappas in Table 2 are mixtures of genuine LLM-ensemble output and ground-truth labels that agree with themselves by construction. The paper never reports quality on the automatically accepted subset alone, which is the only subset where the LLM ensemble actually determines the output. The Discussion's note that the framework 'simulated human-in-the-loop decision-making' acknowledges the substitution but does not address its optimism: live reviewers are not guaranteed to reproduce the majority label, and even if they did, the evaluation cannot separate routing accuracy from label quality. The thresholds are also fitted on the 10% subset using the same substitution, so the Pareto selection is biased toward configurations that look good under this construction. This is a load-bearing flaw, not a minor one.\n\nSecondary soft spots: the uncertainty rule uses the SD of verbalised confidence scores, which will not flag confidently opposed models; and the paper reports only point estimates with no variance or significance testing. The reliance on a single dataset is acknowledged as a limitation and is a minor issue by itself.\n\nWho should read this? People working on LLM-assisted annotation will benefit from the empirical failure modes documented here, and the release of code and prompts makes the negative results easy to build on. But the headline '24-45% effort reduction with no considerable quality degradation' should be read as an upper bound, not a demonstrated outcome.\n\nMy recommendation: this deserves a serious referee, because the empirical core is valuable and the evaluation flaw is correctable in revision. The authors should report quality on the automatically accepted subset, model human review with noisy annotators or held-out labels, and give interval estimates. As it stands, the HITL efficiency claim needs major revision before it can be taken at face value.","headline":"Useful empirical study of LLM limits on fine-grained annotation, but the headline HITL reliability claim is an upper bound because flagged test instances are replaced with ground-truth human labels.","tokens_in":22107,"tokens_out":1751,"would_cite":false,"duration_ms":21548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-06T21:12:58.600508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}