{"id":"e17ee2fd-4818-40ed-bf2b-3cb220a57fe5","arxiv_id":"2506.08626","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A cascade of LLM judges, fed with search context and behavior, produces multilevel usefulness labels for clicked documents and improves search satisfaction prediction.","lead":"This paper proposes CLUE, an LLM-based pipeline that scores how useful a clicked search result was to the user who clicked it, using the query, document text, and behavior signals like dwell time and click order. The authors report that these usefulness scores match real users' ratings better than third-party relevance labels and that they improve query-level satisfaction prediction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'surpass third-party labeling' claim rests on a small, untested margin on a single dataset; if that difference is noise, the central claim loses its main evidence.","rationale":"The reader's verdict of CONDITIONAL is reasonable, and the identified weakest assumption about guideline generalizability is a real concern. However, the single most load-bearing risk to the paper's central claim is the statistical fragility of the headline 'surpass third-party labeling' result. The abstract promises a decisive advantage, but the evidence is one dataset, one metric, no significance testing, and mixed results on other metrics. If the Δr of 0.029 is not significant, the paper's key differentiator disappears. This concern is distinct from the guideline issue: even if the guidelines generalize perfectly, the superiority over third-party labels could still be unsupported. Conversely, even if the guidelines are partly in-sample, the SIGIR16 comparison is out-of-sample for the guidelines (they were derived from UUST), so a significance test there would be informative. I therefore recommend keeping the CONDITIONAL verdict pending this check.","tokens_in":19889,"tokens_out":7287,"duration_ms":76770,"concrete_test":"Run a paired bootstrap over queries (or sessions) on SIGIR16, resampling queries with replacement and recomputing Pearson r and F1 for CLUE-4o and U_a, to obtain 95% confidence intervals and a p-value for the difference. If the CI for the Δr includes 0, or the p-value is > 0.05, the 'surpass third-party' claim is unsupported and should be downgraded. Additionally, if feasible, collect third-party usefulness labels (U_a) on a subset of KDD19 or UUST and repeat the comparison to test replication.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract—'allowing our approach to surpass third-party labeling methods'—is operationalized in Table 5 on SIGIR16, the only dataset with third-party usefulness annotations (U_a). CLUE-4o achieves Pearson r = 0.383 vs U_a = 0.354 (Δ = 0.029), but no significance test or confidence interval is reported. Other metrics are mixed: F1 is 0.381 vs 0.376 (Δ = 0.006), Cohen's κ is 0.185 vs 0.168 (Δ = 0.017), and MAE is worse for CLUE (0.984 vs 0.921). The comparison is single-dataset and single-metric with a small margin, so the 'surpass' claim may be within noise. Moreover, the GBDT_iid baseline (0.430) exceeds CLUE, and while the paper argues this is an in-sample comparison, it undercuts the 'surpass' narrative. This is load-bearing because RQ1's answer and the abstract's central promise depend on this comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLUE (Cascade LLM-based Usefulness Evaluation), a framework that uses LLMs to generate multilevel usefulness labels for web-search evaluation. CLUE integrates query, document content, user context, and behavioral signals into prompts, decomposes the ordinal usefulness judgment into a cascade of binary selection stages (inspired by ordinal binary decomposition), uses multiple voters to reduce position bias, and optionally fine-tunes an open-source Llama model. The authors introduce a new think-aloud dataset (UUST) from which they extract six usefulness guidelines (helpful, detailed, related, encyclopedic, specific, comprehensive), and they evaluate CLUE on SIGIR16, KDD19, UUST, and an industrial search log (SearchLog24Q3). They compare against third-party annotations (U_a, R_a), a GBDT predictor, and LLM pointwise/pairwise/listwise scoring, and they further show that usefulness labels from CLUE improve query-level satisfaction prediction.","tokens_in":20140,"tokens_out":6230,"duration_ms":68170,"significance":"If the central claims are supported, the paper offers a practical, cheaper, user-centric alternative to relevance annotation, a new think-aloud usefulness dataset, and an open-source implementation. The cascading ordinal decomposition is a sensible adaptation of ordinal-regression ideas to LLM judgment, and the RQ3 result on industrial data (Table 10) is a concrete demonstration that LLM-generated usefulness labels can improve satisfaction prediction. However, the headline claim that the method 'surpasses third-party labeling methods' rests on a single dataset with very small margins and no significance testing, and the usefulness guidelines are extracted from the same UUST data on which they are later ablated, creating an in-sample-leakage risk. The paper's contributions are valuable but require stronger empirical support and more careful framing.","major_comments":[{"comment":"The abstract's claim that the approach 'surpass[es] third-party labeling methods' is supported only by the SIGIR16 comparison between CLUE-4o and U_a. The margins are small (Pearson r 0.383 vs 0.354; F1 0.381 vs 0.376; Cohen's κ 0.185 vs 0.168), MAE is worse for CLUE (0.984 vs 0.921), no significance test or confidence interval is reported, and this is the only dataset with a U_a column. The GBDT_iid baseline (Pearson r 0.430) also exceeds CLUE, so the 'surpass' claim is not robust. Please add significance tests (e.g., bootstrap or paired tests over queries) and report effect sizes, or qualify the claim to describe CLUE as comparable to third-party annotation on this single dataset.","section":"Section 5.1, Table 5"},{"comment":"The six-guideline rubric (helpful, detailed, related, encyclopedic, specific, comprehensive) is extracted from think-aloud comments of the 31 UUST participants, and the guideline ablation is then evaluated on the UUST test set, which is drawn from the same participants and sessions. This is in-sample leakage: the rubric is fitted to the same distribution it is evaluated on. The KDD19 transfer result provides only a second lab-study check without a significance test. To support the claimed generalizability, the guidelines should be derived from a training split of participants/tasks and tested on a held-out split of participants/tasks, or on a third dataset from a different population.","section":"Section 3.1.4 and Section 5.2.4, Figure 4"},{"comment":"The RQ3 comparison confounds label type with input features: U_llm is generated by prompts that include behavior and context features (Table 4), whereas R_llm is explicitly generated without behavior or context features (Table 1), and R'_a is a text-based relevance model. The observed improvement of +U_llm over +R_llm/+R'_a may therefore be due to the additional behavioral/contextual features rather than to the usefulness construct. The authors should add a condition with relevance labels generated using the same behavior/context prompt template, or an ablation in which usefulness is judged without behavior/context, to isolate the effect of the label type.","section":"Section 5.3, Tables 9-10 and Figure 5"}],"minor_comments":[{"comment":"The text says 'the five-voter version of CULE'; 'CULE' should be 'CLUE'.","section":"Section 5.3.2"},{"comment":"The caption contains the duplicated phrase 'where curly braces curly braces indicates the features'.","section":"Figure 2 caption"},{"comment":"The example text says 'The contnet in [2] and [3]'; 'contnet' should be 'content'.","section":"Figure 3"},{"comment":"The row labels for SIGIR16 appear to omit 'FT' on the second row; as printed, both SIGIR16 rows are labeled 'Llama-3', making the table ambiguous.","section":"Table 8"},{"comment":"The x-axis labels 'w/o w' and 'w/ w' are unclear; they presumably mean 'w/o guideline' and 'w/ guideline'.","section":"Figure 4"},{"comment":"The UUST dataset is said to be released after publication; for reproducibility, consider also releasing the anonymized think-aloud transcripts and the exact prompt templates used in the study.","section":"Section 4.4.3 and Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the UUST dataset could be a valuable community resource if released with appropriate anonymization. The main risks are overclaiming based on a single dataset and the in-sample guideline evaluation; both are addressable with additional experiments and more cautious wording. I would not reject, but the current evidence does not support the abstract's 'surpass' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, but read the abstract with a grain of salt. CLUE assembles known pieces—ordinal binary decomposition, LLM scoring, behavior/context prompts—into a coherent pipeline for judging document usefulness, and it ships a new think-aloud dataset (UUST) plus code. The RQ3 result looks real: on a 14K-query industrial log, usefulness labels from CLUE push satisfaction prediction from 0.5309 to 0.5915 F1 (p<0.05) at cutoff 10. That is the strongest evidence in the paper, and likely the most citable.\n\nThe soft spot is exactly where the abstract plants the flag. 'Surpass third-party labeling methods' rests on SIGIR16 alone, where CLUE-4o's Pearson r is 0.383 vs U_a's 0.354—a 0.029 gap with no significance test or confidence interval. F1 and kappa are similarly thin, and MAE is actually worse. So the headline claim is overbroad. I'd also want the guideline derivation handled more carefully: the six adjectives come from UUST think-aloud transcripts, and UUST is then one of the evaluation sets. The KDD19 transfer helps, but the in-sample gain on UUST still leaves a fitting story open. The main comparison tables (5–7) report 'significantly outperforms' without showing tests; only RQ3 gives paired t-tests.\n\nThe GBDT_iid baseline beating CLUE (0.430 vs 0.383 Pearson r) is worth a clear sentence. The authors say it's in-sample, which is fair, but it complicates the 'surpass' narrative more than the paper acknowledges.\n\nOn balance the work is honest and useful. The cascade structure, prompt design, and dataset are real contributions. It's for anyone working on IR evaluation, especially those trying to get user-centric labels without hiring annotators. It needs a referee who pushes on claim calibration, significance, and guideline leakage before it appears.\n\nMy recommendation: send it to peer review, with the expectation of a major revision. It deserves a serious referee, but the abstract and RQ1 answer should not survive unchanged.","headline":"Solid and useful, but the 'surpass third-party labels' claim is a single-dataset, no-significance result that needs scaling back before publication.","tokens_in":20660,"tokens_out":3018,"would_cite":true,"duration_ms":33031,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM usefulness labels, built from click behavior and context, agree with real users better than third-party relevance annotations and improve satisfaction prediction.","keywords":["usefulness evaluation","user satisfaction","LLM relevance judgment","ordinal regression","cascading binary classifiers","search evaluation","click behavior","test collection construction"],"falsifier":"Collect a new dataset of real-user usefulness judgments on queries and tasks not drawn from the UUST design, run CLUE with the published six guidelines, and check whether its agreement with those users (for example, Pearson correlation and F1) still exceeds third-party annotation; if the gap disappears on this held-out population, the reported CLUE advantage is an artifact of the original rubric and task distribution.","tokens_in":1753,"feed_emoji":"🔎","tokens_out":1777,"duration_ms":69351,"temperature":0.7,"pith_summary":"The paper claims that large language models, when shown the searcher's query, task background, and click behavior, can judge how useful a document was to that user more accurately than third-party relevance annotators. The authors build CLUE, a cascading set of LLM binary judges that assigns multilevel usefulness scores by asking, at each level, which remaining clicked documents deserve that tier. They test CLUE against third-party usefulness and relevance labels, a GBDT baseline, and pointwise, pairwise, and listwise LLM prompting, reporting that CLUE outperforms the label baselines and stays competitive with the machine-learning baseline without training data. They then use CLUE's usefulness labels as features in satisfaction-prediction models, reporting gains over behavior-only and relevance-label features on both a lab dataset and industrial search logs. If the claim holds, LLM-produced usefulness labels offer a cheaper, more user-centered substitute for relevance labels in building and evaluating test collections.","feed_headline":"LLM usefulness labels beat relevance annotations for search evaluation","feed_subtitle":"A cascade of LLM judges, fed click behavior and context, outperforms third-party labels and boosts satisfaction prediction.","key_machinery":"The load-bearing mechanism is the cascade of binary LLM classifiers built on the OneVsPrevious ordinal decomposition. Rather than asking the LLM to produce a multilevel label directly, CLUE presents all clicked documents for a query together and asks, at each stage from top level down, which documents deserve that level; documents that pass majority voting among five permuted voters are assigned that level and removed, and the rest fall to lower levels until leftovers are assigned the bottom level. The prompt packs in the query string, task description and background, click order, dwell time, query-level click aggregates, and six usefulness guidelines (helpful, detailed, related, encyclopedic, specific, comprehensive) harvested from think-aloud comments. This design converts usefulness judgment into a sequence of simpler selection tasks where the LLM's ability to hold context and reason matters more than calibrated probability estimates.","core_discovery":"The central discovery the paper argues for is that usefulness, not relevance, is the label that tracks user satisfaction, and that LLMs can produce such labels when the prompt reconstructs the user's situation. Concretely, the paper reports that CLUE with GPT-4o reaches a Pearson correlation of 0.383 with real user usefulness judgments on SIGIR16, beating third-party usefulness annotations (0.354) and relevance annotations (0.320); on UUST it reaches F1 of 0.445 and Pearson correlation of 0.428. On KDD19, a lab dataset with complex tasks, CLUE reaches 0.346 Pearson without any training, which the paper characterizes as comparable to the out-of-distribution GBDT model. The same CLUE labels, when added as features to a satisfaction classifier, outperform relevance-based label features on both KDD19 and SearchLog24Q3. The paper interprets this as evidence that language models can bridge the situational disconnect between third-party annotators and real users.","pith_inferences":["A direct transfer test would be to run CLUE, with the published six guidelines, on a dataset whose usefulness ground truth comes from a different user population than the one that supplied the guidelines; this would measure rubric generality rather than in-sample fit.","The cascade structure could generalize to other graded judgment tasks with short candidate lists, such as rating the helpfulness of answers in community question-answering, where ordinal labels and few candidates are natural.","Vote counts from the five voters could be read as soft confidence scores, not just binary decisions, enabling active learning or threshold calibration for usefulness labeling at scale.","Feature-level ablations, removing behavioral features, context features, and content features one at a time, could reveal which part of the situational information carries the predictive gain; the paper's ablations cover whole components but not individual feature groups."],"forward_implications":["CLUE's usefulness labels can be generated without collecting training data for a new domain, unlike GBDT-based usefulness predictors that the paper finds degrade under distribution shift.","Search evaluation metrics built on CLUE usefulness labels predict user satisfaction better than metrics built on relevance labels, on both a lab dataset and an industrial search log.","The six guidelines derived from the UUST user study transfer to the KDD19 dataset, suggesting a single rubric can guide LLM judges across datasets.","Fine-tuning an open-source LLM as separate binary classifiers improves its usefulness judgments, though the paper reports it does not close the gap with the commercial model.","Because CLUE labels only clicked documents, the framework reduces annotation effort relative to full relevance judgments while still supporting query-level evaluation."],"supporting_citations":[{"why":"Supplies the ground-truth usefulness data and the core premise that relevance correlates weakly with satisfaction; CLUE is designed to outperform its third-party annotations.","marker":"[36]"},{"why":"Defines the GBDT usefulness-prediction baseline whose features and out-of-distribution protocol CLUE compares against.","marker":"[35]"},{"why":"Provides the ordinal regression survey and taxonomy that motivates the OneVsPrevious decomposition used in CLUE.","marker":"[19]"},{"why":"Introduces the Cascade Linear Utility Model whose chained binary classifiers CLUE generalizes by replacing SVMs with LLMs.","marker":"[55]"},{"why":"Provides the DNA prompt template used to structure the CLUE prompt with context, background, and scoring aspects.","marker":"[46]"},{"why":"Supplies the pointwise, pairwise, and listwise LLM utility-judgment baselines and the finding about document-order bias that motivates multi-voting.","marker":"[61]"},{"why":"The closest prior LLM usefulness-judgment method using rubrics, positioned as complementary because CLUE derives guidelines from user think-aloud data.","marker":"[14]"}],"fun_headline_variants":["LLM usefulness labels beat relevance for user satisfaction","Context-aware LLM judges predict search satisfaction better","LLM usefulness judgments surpass relevance in satisfaction models","Cascading LLM judges improve satisfaction prediction","Usefulness labels from LLMs outperform relevance annotations"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The six guidelines derived from think-aloud comments of 31 students on 10 tasks are assumed to describe what any searcher means by useful, so LLMs guided by them will agree with real users beyond this study's setup.","fun_headline_variants_meta":{"raw":{"variants":["LLM usefulness labels beat relevance for user satisfaction","Context-aware LLM judges predict search satisfaction better","LLM usefulness judgments surpass relevance in satisfaction models","Cascading LLM judges improve satisfaction prediction","Usefulness labels from LLMs outperform relevance annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1230,"prompt_tokens":899,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":515,"tokens_out":331,"duration_ms":3547,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:06:40.616287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a new dataset of real-user usefulness judgments on queries and tasks not drawn from the UUST design, run CLUE with the published six guidelines, and check whether its agreement with those users (for example, Pearson correlation and F1) still exceeds third-party annotation; if the gap disappears on this held-out population, the reported CLUE advantage is an artifact of the original rubric and task distribution.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ground-truth usefulness data and the core premise that relevance correlates weakly with satisfaction; CLUE is designed to outperform its third-party annotations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GBDT usefulness-prediction baseline whose features and out-of-distribution protocol CLUE compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Cascade Linear Utility Model whose chained binary classifiers CLUE generalizes by replacing SVMs with LLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior LLM usefulness-judgment method using rubrics, positioned as complementary because CLUE derives guidelines from user think-aloud data."}],"review_version":1}