{"id":"e0adc0cf-16e3-4188-87a3-473f98e1d5f6","arxiv_id":"2412.13268","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Ensembling small open-source LLMs as relevance judges achieves human-correlation scores competitive with GPT-4-based judges on the LLMJudge benchmark.","lead":"This paper reports that combining judgments from several small open-source language models, or from multiple prompts to one model, can produce relevance labels for search that are as accurate as labels from a single large model like GPT-4. The approach, called JudgeBlender, could make automated search evaluation cheaper and more transparent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline kappa advantage is not robust: LLMBlender-MV(Rnd.) beats RelExp by Δκ=0.010 on 25 queries, with post hoc aggregator selection and random tie-breaking; no significance or confidence interval is reported.","rationale":"The reader correctly flags the single 25-query benchmark as a generalization threat, and the paper itself admits in Section 6 that experiments were restricted to one dataset. My stress-test identifies a related but more internal threat: the headline comparison is not shown to be statistically stable even on this benchmark. The differences among JudgeBlender variants and between JudgeBlender and RelExp are small, the winning variant uses random tie-breaking, and no significance testing or confidence intervals are provided. This does not overturn the paper's empirical contribution, because the method is clearly described, the data and code are available, and the reported kappa values are plausible. It does mean the broad conclusion about large models being unnecessary should be read as conditional: it depends on the choice of aggregation variant, on the particular prompt/model panel, and on the benchmark. Since the reader already assigned a conditional verdict, I do not change that verdict; I would keep the paper conditional and require the bootstrap and seed-variance checks before accepting the abstract's generalization as a firm claim.","tokens_in":10071,"tokens_out":5502,"duration_ms":53869,"concrete_test":"Resample the 25 test queries with replacement (bootstrap) for LLMBlender + MV(Rnd.), LLMBlender + MV(Avg.), and RelExp, recomputing Cohen's kappa on each resample, and report 95% bootstrap confidence intervals plus the fraction of resamples where LLMBlender beats RelExp. Separately, rerun LLMBlender + MV(Rnd.) with 100 random seeds on the original data to quantify tie-break variance. If the LLMBlender confidence interval contains RelExp's kappa, or the seed-to-seed spread is comparable to Δκ=0.010, the abstract's claim about large models being unnecessary is not supported by the reported numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's conclusion that 'very large models are often unnecessary' rests on Table 2, where LLMBlender + MV(Rnd.) reaches Cohen's kappa 0.2619 versus RelExp's 0.2519. This comparison is the least secure part of the argument. First, the test collection contains only 25 queries (Table 1), and Cohen's kappa is computed over the pooled qrels without query-level clustering or any reported confidence interval. Second, the winning configuration was selected post hoc from ten aggregator/tie-break combinations defined in Section 4.2 (MV with Avg, Rnd, Max, Min tie-breaking plus AV); the kappa values across these variants span only 0.2502 to 0.2619, and the variants share the same underlying judge scores, so the best value is not an independent observation. Third, the highlighted MV(Rnd.) variant resolves ties randomly, meaning the reported 0.2619 is one realization of a stochastic procedure. With Δκ=0.010 against a GPT-4o baseline and only 25 queries, the headline margin could plausibly reflect sampling noise rather than a genuine advantage of ensembling small models. The paper acknowledges the single-dataset limitation in Section 6, but the missing robustness analysis is more directly load-bearing: without significance testing or confidence intervals, even the within-benchmark claim that JudgeBlender is competitive with large proprietary models is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JudgeBlender, an ensemble framework for automatic relevance assessment with two variants: PromptBlender (one LLM, multiple prompts) and LLMBlender (multiple LLMs, one prompt each). Judgments are aggregated by majority voting with four tie-breaking strategies (random, max, min, average) or by averaging scores. The framework is evaluated on the LLMJudge benchmark derived from TREC DL 2023, reporting Cohen's kappa, Krippendorff's alpha, Kendall's tau, Spearman's rho, system ranking correlation, and per-level inter-annotator agreement. The central claim is that ensembling small open-source 7B-8B models reaches or exceeds the label correlation of a single GPT-4o-based judge, so 'very large models are often unnecessary' for relevance assessment. The experiments are limited to a single test collection with 25 queries.","tokens_in":10329,"tokens_out":2711,"duration_ms":26428,"significance":"If the result is statistically robust, the paper makes a practically useful contribution: it suggests that cost-effective, reproducible, open-weight models can substitute for expensive proprietary judges in relevance labeling. The work uses a public benchmark, reports multiple agreement and ranking metrics, analyzes bias across system types, and provides prompts on GitHub, which are concrete reproducibility strengths. However, the headline kappa advantage over GPT-4o rests on a very small test set and is not supported by significance testing, confidence intervals, or query-level analysis; the significance of the claim is therefore conditional on the robustness analysis that the paper currently lacks.","major_comments":[{"comment":"The central comparison, LLMBlender-MV(Rnd.) with kappa 0.2619 versus RelExp with 0.2519, is based on only 25 test queries (Table 1). No significance test, confidence interval, or query-level clustering is reported; Cohen's kappa is computed over the pooled qrels, which inflates the effective sample size. The paper should provide a bootstrap or permutation-based confidence interval and ideally a query-stratified analysis, because the observed Δκ = 0.010 could easily arise from sampling noise.","section":"§5.1, Table 2"},{"comment":"The best-performing variant is selected post hoc from ten aggregator/tie-break configurations that all share the same underlying judge scores, so the highlighted value is not an independent observation. Moreover, the highlighted MV(Rnd.) variant resolves ties randomly, making the reported 0.2619 one realization of a stochastic procedure. The authors should report the mean and variance over multiple random seeds, or use a deterministic tie-break chosen on the development set, to support the claim that ensembling small models is genuinely competitive with GPT-4o.","section":"§4.2, Table 2"},{"comment":"The text states that 'the best-performing variant of JudgeBlender, LLMBlender-MV(Avg.)' shows consistent agreement, but Table 2 reports the highest kappa for LLMBlender-MV(Rnd.) (0.2619), the highest alpha for LLMBlender-AV (0.4832), and different best values for ranking metrics. This inconsistency makes it unclear which variant is claimed to be best and risks cherry-picking. The authors should define a single selection criterion (e.g., kappa on the dev set) and align the textual claims with the table.","section":"§5.3"}],"minor_comments":[{"comment":"The text contains typos and duplicated words, e.g., 'prposed by Thomas et al.' and 'the prompt used by byMultiCriteria method'; these should be corrected.","section":"§4.5"},{"comment":"The caption labels the fourth subplot as '(b)' instead of '(d)', which is confusing for readers.","section":"Figure 3"},{"comment":"The column header 'PromptBlender - MV(Av g.)' has an odd spacing in the displayed table; the formatting should be fixed.","section":"Table 3"},{"comment":"The model names are used inconsistently: Section 4.3 says 'Meta-Llama-3-8B' and 'Llama-3-8B', while Table 2 reports 'Llama-3-8B-Instruct' for PromptBlender; the exact model variants should be stated consistently.","section":"§4.3"},{"comment":"The limitation section acknowledges the single-dataset scope but does not mention the absence of significance testing; adding this caveat would make the conclusions more appropriately cautious.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and addresses a timely problem. The main weakness is statistical: the headline advantage over GPT-4o is small, on 25 queries, and without uncertainty quantification or a principled selection rule for the aggregation variant. This is fixable by adding significance tests, confidence intervals, and a dev-set-based model selection procedure, so I see it as a major-revision rather than a rejection. I would also encourage the editor to ask the authors to clarify which variant is considered the best, since the current text and Table 2 disagree."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is concrete: a simple majority vote across three 7-8B open models reaches Cohen's kappa 0.2619 on the LLMJudge test set, nudging past the best GPT-4o baseline (RelExp, 0.2519) and clearly beating MultiCriteria (0.1829). The paper also contributes a nice bias analysis showing that the ensemble judges are more even-handed across GPT-, T5-, and other-based systems than the single-model baselines. That bias result is genuinely valuable and arguably more robust than the kappa headline.\n\nThe paper is clearly written, compares against a sensible set of baselines including fine-tuned models, and ships prompts on GitHub. The novelty claim for LLM ensembles specifically for relevance assessment looks correct against the cited literature; Verga et al. is about generation evaluation, not relevance judging.\n\nThe soft spots are real and load-bearing. The test collection is 25 queries, and kappa is computed over pooled qrels with no query-level clustering, no confidence intervals, no significance tests. The winning configuration, MV(Rnd.), resolves ties randomly, so the reported 0.2619 is one stochastic draw. The family of aggregator variants spans only 0.2502 to 0.2619, which brackets the RelExp baseline, and the best variant was selected post hoc from ten options. In other words, the 0.01 kappa advantage is indistinguishable from noise. The paper does acknowledge the single-dataset limitation in the conclusion, but the missing robustness analysis is more damaging than the single dataset per se. A minor internal inconsistency: Section 5.3 treats MV(Avg.) as the best variant while Table 2 gives MV(Rnd.) the highest kappa, which suggests the 'best' label is not computed against a fixed criterion.\n\nWho is this for: IR evaluation researchers and anyone using LLM judges in practice. The method is cheap, open, and reproducible, and the bias analysis should survive the statistical critique. But nobody should cite the kappa number as evidence that large models are unnecessary until it is replicated on more collections with proper significance testing and a pre-specified aggregation rule.\n\nRecommendation: yes, this deserves a serious referee. The question matters, the method is simple enough to verify, and the bias analysis is a real contribution. A referee should push for significance testing, multiple datasets, and a less post hoc selection story.","headline":"A plausible but under-powered result: ensembling small open LLMs lands in the same kappa range as GPT-4o judges, but the headline edge rests on a 0.01 gap over 25 queries with no significance testing.","tokens_in":10906,"tokens_out":1719,"would_cite":true,"duration_ms":18537,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JudgeBlender claims that a panel of small open language models can judge search relevance as reliably as a single large proprietary model.","keywords":["LLM-as-a-Judge","automatic relevance assessment","ensemble learning","LLMBlender","PromptBlender","majority voting","small open-source models","TREC DL 2023"],"falsifier":"Compute per-query Cohen's kappa for LLMBlender with majority voting and random tie-breaking, and for GPT-4o-based RelExp, on the LLMJudge test set: with only 25 queries, dropping a handful of topics could erase the 0.2619 versus 0.2519 margin. The decisive check is to repeat the vote on TREC DL 2019-2021 qrels or a non-web collection; if the open-model ensemble does not match or exceed the single GPT-4o judge there, the claim that large models are unnecessary loses its support.","tokens_in":9847,"feed_emoji":"⚖️","tokens_out":7428,"duration_ms":63809,"temperature":0.7,"pith_summary":"This paper proposes JudgeBlender, an ensemble method that produces search relevance labels by pooling judgments from several small open-weights language models, either by varying the prompt to one model (PromptBlender) or by using several distinct models (LLMBlender). The authors aim to show that such a panel, aggregated by majority or average voting, yields relevance judgments that track human labels as well as a single large commercial judge such as GPT-4o. On the LLMJudge benchmark, LLMBlender with majority voting and random tie-breaking reaches Cohen's kappa 0.2619 against human labels, slightly above the best GPT-4o baseline RelExp (0.2519) and above the LLMJudge-challenge winner MultiCriteria (0.1829). If this holds, evaluation of retrieval systems could become cheaper and more reproducible because large proprietary models are not required.","feed_headline":"Three small open LLMs beat GPT-4o at relevance judging","feed_subtitle":"Blending Mistral, Gemma, and Llama with majority voting lifts Cohen's kappa to 0.2619 on the LLMJudge test set.","key_machinery":"The machinery is a panel of evaluators plus an aggregator. PromptBlender uses one model, Meta-Llama-3-8B, with three distinct prompts: a direct relevance-grading prompt, a criteria-decomposition prompt, and a binary-then-score two-step prompt. LLMBlender uses Mistral-7B, Gemma-7B, and Llama-3-8B, each attached to a different prompt family from the same set. The per-judge scores are pooled by majority voting, with four tie-breaking rules (random, max, min, average), or by average voting. Majority voting is the component that lifts the ensemble above its individual members; the paper reports LLMBlender's best Cohen's kappa under random tie-breaking and its most balanced per-level agreement under average tie-breaking.","core_discovery":"JudgeBlender's central claim is that ensembling matters more than raw model size: a jury of small open models produces automatic relevance judgments whose agreement with human assessors is competitive with, and on the headline metric slightly better than, a single large proprietary model. The paper demonstrates this by comparing label-level agreement (Cohen's kappa and Krippendorff's alpha) and system-ranking agreement (Kendall's tau and Spearman's rho) on the LLMJudge test set, and by showing that both PromptBlender and LLMBlender improve over their individual members. The strongest result is LLMBlender with majority voting and random tie-breaking, at kappa 0.2619, compared with 0.2519 for GPT-4o-based RelExp and 0.1829 for MultiCriteria. The paper also reports that the best JudgeBlender variants are more balanced across all four relevance levels and show less systematic over- or under-estimation for systems grouped by the language model they use.","pith_inferences":["A natural next experiment is to run the same panel on TREC DL 2019-2021 and on non-web test collections; the claim that very large models are unnecessary stands or falls on whether the kappa advantage survives beyond the current 25-query test set.","Because the best LLMBlender result uses random tie-breaking, the practical recipe is unstable at the margin; a deterministic second-stage judge or a learned aggregator could convert the small kappa gain into a more reliable one.","The economic case may be the stronger one: even at parity, replacing GPT-4o with locally run open models removes per-query API costs, data-leakage exposure, and output nondeterminism from the evaluation pipeline."],"forward_implications":["Automatic relevance-label generation for retrieval evaluation can be run locally on open 7-8 billion parameter models instead of through expensive commercial APIs.","Prompt diversity alone is a cheaper form of ensembling: PromptBlender's aggregated judgments beat any of its three individual prompts.","System rankings computed from JudgeBlender judgments closely track official human-based TREC rankings, supporting its use for evaluating shared-task submissions.","The best JudgeBlender variants distribute TREC DL 2023 systems evenly around the diagonal, indicating less model-family bias than single-judge methods.","The optimal configuration depends on the tie-breaking rule and evaluation metric, so JudgeBlender is best treated as a family of methods rather than a single fixed recipe."],"supporting_citations":[{"why":"Supplies the LLMJudge dataset, the challenge baselines, and the human qrels against which all methods are measured.","marker":"[18]"},{"why":"Defines the TREC DL 2023 track, its relevance scale, and the official human judgments used as ground truth.","marker":"[4]"},{"why":"Provides the direct-grading relevance prompt reused as one of the JudgeBlender prompt families and as a baseline.","marker":"[22]"},{"why":"Is the MultiCriteria baseline and contributes the criteria-decomposition prompt used by Mistral and one PromptBlender variant.","marker":"[7]"},{"why":"Supplies the binary-then-score relevance prompt used by Llama-3-8B and one PromptBlender variant, plus the SunMulti GPT-4o baseline.","marker":"[21]"},{"why":"Establishes the original LLM relevance-judgment prompting baselines that JudgeBlender compares against.","marker":"[6]"},{"why":"Provides the fine-tuned GenRE baselines that JudgeBlender is compared with.","marker":"[12]"},{"why":"Provides the system metadata used to group TREC DL 2023 runs by underlying model family in the bias analysis.","marker":"[17]"}],"fun_headline_variants":["Small open LLM juries match GPT-4 in relevance judging","JudgeBlender: small models, big relevance agreement","Ensembling small open LLMs outperforms single GPT-4 for relevance","Jury of small LLMs edges out GPT-4 on relevance judging","Small open LLM ensembles beat GPT-4o at relevance judging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on one benchmark: the LLMJudge test set contains only 25 queries drawn from TREC DL 2023, so the claim that very large models are often unnecessary presumes that one small collection represents relevance judgment across domains, query types, and document scales.","fun_headline_variants_meta":{"raw":{"variants":["Small open LLM juries match GPT-4 in relevance judging","JudgeBlender: small models, big relevance agreement","Ensembling small open LLMs outperforms single GPT-4 for relevance","Jury of small LLMs edges out GPT-4 on relevance judging","Small open LLM ensembles beat GPT-4o at relevance judging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3080,"prompt_tokens":930,"completion_tokens":2150,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2059}},"tokens_in":546,"tokens_out":2150,"duration_ms":15002,"temperature":1.0,"reasoning_tokens":2059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:16:51.864735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute per-query Cohen's kappa for LLMBlender with majority voting and random tie-breaking, and for GPT-4o-based RelExp, on the LLMJudge test set: with only 25 queries, dropping a handful of topics could erase the 0.2619 versus 0.2519 margin. The decisive check is to repeat the vote on TREC DL 2019-2021 qrels or a non-web collection; if the open-model ensemble does not match or exceed the single GPT-4o judge there, the claim that large models are unnecessary loses its support.","supporting_citations":[{"cited_title":"In: Text REtrieval Conference (TREC)","cited_arxiv_id":null,"evidence_quote":"Defines the TREC DL 2023 track, its relevance scale, and the official human judgments used as ground truth."},{"cited_title":"In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the direct-grading relevance prompt reused as one of the JudgeBlender prompt families and as a baseline."},{"cited_title":"Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3","cited_arxiv_id":"2410.14044","evidence_quote":"Is the MultiCriteria baseline and contributes the criteria-decomposition prompt used by Mistral and one PromptBlender variant."},{"cited_title":"In: Proceedings of the 2023 ACM SIGIR Interna- tional Conference on Theory of Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Establishes the original LLM relevance-judgment prompting baselines that JudgeBlender compares against."}],"review_version":1}