{"id":"e64a3701-8c49-4c13-8887-b409a6ea2981","arxiv_id":"2501.17178","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-fidelity, multi-objective search finds cheap open-weight LLM judges that match or outperform prior judge designs on several benchmarks.","lead":"This paper searches over 4,480 ways to configure an LLM judge, including the base model, prompt, output format, and temperature, using a multi-fidelity, multi-objective tuning method. It reports that the resulting open-weight judges match or beat GPT-4-based judges on several benchmarks at a small fraction of the evaluation cost, with the full tuning search costing roughly 2,000 dollars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The search optimizes human agreement on filtered LMSys, but the paper never demonstrates that human agreement correlates with Spearman correlation or PandaLM accuracy across judge configurations; the reported Arena-Hard edge is within bootstrap error. This proxy-validity gap is load-bearing.","rationale":"I read the paper as making a practical, cost-focused contribution: a multi-fidelity, multi-objective search over judge hyperparameters, with open-weight models and a released codebase. The cost model is transparent, and the search is systematic over a large configuration space. What would have to be true for the central claim is that the cheap proxy used for selection (human agreement on filtered LMSys) is a valid predictor of the target metrics on which superiority is claimed. The paper provides aggregate arguments and reliability correlations, but not the configuration-level proxy-target correlation that would directly support this. The reported Arena-Hard difference is within error bars, so the strongest external evidence is the PandaLM result, which itself depends on the untested filtering thresholds. My concern is not that the method is wrong, but that the load-bearing link between the optimization objective and the evaluation targets is unverified. The proposed test is feasible because the 400 final-fidelity configurations are already identified; adding an Arena-Hard and PandaLM evaluation for those configurations costs the same order as the original search. If the test shows low correlation, the verdict should move to UNVERDICTED or REJECT; if high, ACCEPT. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":17351,"tokens_out":6934,"duration_ms":66198,"concrete_test":"Compute the rank correlation across the 400 configurations that reached the final fidelity between human agreement on the 3548 LMSys validation instructions and (a) Spearman correlation with Chatbot Arena ELO on the 20-model Arena-Hard set, and (b) human agreement on the PandaLM test set. If either rank correlation is below roughly 0.5, the multi-fidelity selection is not optimizing the reported target metric; if both are above 0.7, the proxy-validity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that tuning on a cheap proxy yields judges that match or beat GPT-4/Arena-Hard on accuracy. Section 4 replaces Spearman correlation with human agreement as the optimization metric, justified only by aggregate scaling trends (Figs. 1-2) and lower variance (Table 1). Those facts show human agreement can separate model sizes; they do not show it ranks individual configurations by the reported target metrics: Spearman with Chatbot Arena ELO (Table 4) or PandaLM agreement (Table 3). The multi-fidelity loop (Section 5.3) reduces 4480 configurations to 400 using only human agreement, so if the proxy is only weakly correlated with the target at the configuration level, the final judges are not guaranteed to be near-optimal for the target. The only config-level evidence offered, Fig. 13, is split-half reliability of human agreement, not proxy-target validity. Table 4's headline advantage (Ours-medium 0.93 vs. GPT-4 0.90) is within the reported bootstrap error, so the accuracy-outsperforms claim rests heavily on PandaLM; but no analysis shows that the filtered LMSys distribution (Appendix A.2: quality >= 5 plus specificity flag) is predictive of PandaLM. The threshold choices in Appendix A.2 are untested, so the selected judges may be specialized to high-specificity instructions rather than generally better judges.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a systematic, low-cost procedure for tuning the design decisions of zero-shot LLM judges (base model, temperature, prompt template, output format, response averaging). It defines a search space of 4,480 judge configurations, evaluates them on human agreement with the LMSys dataset using a three-stage multi-fidelity selection driven by non-dominated sorting on accuracy and cost, and reports the resulting judges on held-out LMSys test instructions, the PandaLM test set, and Arena-Hard Spearman correlation with Chatbot Arena ELO. The authors claim that the tuned open-weight judges match or outperform GPT-4-based and Arena-Hard judges while costing one to two orders of magnitude less per annotation, and that the whole search costs roughly 1/1000 of a comparable naive grid evaluation. The paper also provides hyperparameter analyses (model scaling, temperature, output format, prompt components, order averaging) and releases code and annotations.","tokens_in":17636,"tokens_out":5127,"duration_ms":48711,"significance":"If the central claims hold, the paper makes a useful contribution: it is one of the first systematic studies of judge design decisions under a controlled search space, and it demonstrates that the search can be made affordable via multi-fidelity multi-objective optimization. The clean validation/test split, the use of bootstrap errors in most tables, the inclusion of external benchmarks (PandaLM, Arena-Hard, Chatbot Arena ELO), and the release of code and a full annotation dataset are concrete strengths. The open-weight nature of the resulting judges is a practical benefit for reproducibility and community leaderboards. The paper also surfaces practically relevant findings (e.g., pair scoring, order averaging, temperature effects) that could guide future judge construction. However, the validity of human agreement on a filtered LMSys set as a proxy for the reported target metrics is not demonstrated at the configuration level, and several accuracy comparisons rest on point estimates without uncertainty quantification; these issues need to be addressed before the headline claims are fully supported.","major_comments":[{"comment":"The optimization target is human agreement on the filtered LMSys validation set, but the paper's headline claims are about Spearman correlation with Chatbot Arena ELO (Table 4) and PandaLM test accuracy (Table 3). The justification in Section 4 for switching from Spearman correlation to human agreement is aggregate: Table 1 and Figs. 1-2 show that human agreement separates model sizes with lower variance. These facts do not establish that human agreement ranks individual judge configurations in the same order as the target metrics. The only configuration-level evidence in the paper, Fig. 13, is split-half reliability of human agreement itself, which says nothing about proxy-target validity. Because the multi-fidelity loop in Section 5.3 prunes 4480 configurations down to 400 using only human agreement, a weak configuration-level correlation between the proxy and the target would make the final judges not necessarily near-optimal for the reported metrics. I request an additional analysis: compute Spearman correlation (or PandaLM agreement) for all (or a random sample of) configurations at the lowest fidelity and report the rank correlation with human agreement on the shared configurations.","section":"Section 4 and Section 5.3"},{"comment":"The LMSys instruction filter (quality score >= 5 plus the 'Specificity' criterion, both assigned by Llama3-8B-instruct) determines the entire distribution over which human agreement is measured and all selection decisions are made. The thresholds are introduced without any sensitivity analysis or validation that they preserve the target property of judge quality. As a result, the selected judges may be specialized to high-specificity, high-quality instructions rather than being generally better judges. The authors should show that the ranking of configurations is stable under reasonable variations of these thresholds, or at least report the results on unfiltered or differently filtered LMSys test splits.","section":"Appendix A.2"},{"comment":"The abstract and conclusion state that the found judges 'outperform existing benchmarks in accuracy,' but the support is weaker than the text suggests. In Table 4, the best result (Ours-medium, 0.93 ± 0.05) overlaps within bootstrap error with GPT-4 (0.90 ± 0.06); the paper's own text acknowledges 'matches or outperforms,' which is the defensible claim. Table 3 lists only point estimates for PandaLM agreement, with no bootstrap errors or significance tests, so the strong claims about outperforming GPT-4 and PandaLM-70B on that dataset are not yet quantified. Please add uncertainty estimates to Table 3 and adjust the abstract/conclusion to reflect the actual significance level of the accuracy gains.","section":"Tables 3 and 4, Abstract"},{"comment":"The paper does not provide a baseline that isolates the contribution of the tuning procedure itself. The final judges are compared against existing judges (Arena-Hard, PandaLM, JudgeLM, GPT-4), but not against, say, a random configuration from the search space or a default prompt with the same base model. Without such a baseline, the reader cannot tell whether the reported test-set gains come from the multi-fidelity search, from the choice of model family, or from the prompt template used in the final configurations. I suggest reporting the test-set performance of several random configurations and of a default (e.g., 'best-model-identifier' with temperature 0 and no added components) configuration on the same three test sets.","section":"Section 5.6 and Section 5.3"}],"minor_comments":[{"comment":"The sentence 'The judges we found outperforms all baselines and slightly underperforms or matches Arena-Hard' is internally contradictory; Table 2 shows Ours-large (0.49) slightly below Arena-Hard (0.50), so the phrasing should be revised to say the judges outperform some baselines and match or slightly underperform Arena-Hard.","section":"Section 5.6 (LMSys paragraph)"},{"comment":"The method is referred to as both 'successful-halving' and 'sucessful-halving'; the standard name is 'successive halving', and the typo should be fixed.","section":"Section 5.3 and Appendix B"},{"comment":"The cost model is stated transparently, but the central cost-efficiency comparisons rely on self-estimated token prices and a fixed 0.6s/annotation average. A brief sensitivity analysis (e.g., 2x runtime or public API prices) would strengthen the robustness of the '1/100' and '1/1000' cost claims, which are a main selling point of the paper.","section":"Appendix B.1"},{"comment":"The caption 'two group of prompts works well' contains a grammar error; it should be 'two groups of prompts work well.'","section":"Section 5.4 (Fig. 7 caption)"},{"comment":"The reference to the LMSys Kaggle competition is listed as 'lin Chiang, W.'; this should be 'Chiang, W.-L.' for consistency with the main citation format.","section":"References"},{"comment":"The limitation discussion is honest and appropriately notes that only flip rate is checked; it would be helpful to also mention that the search itself does not optimize for any bias metric, although this does not affect the validity of the main results.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and the open release of code and annotations is a real plus. The main risk is the proxy-validity gap: the search target (human agreement on filtered LMSys) is not shown to rank configurations consistently with the reported target metrics. This is addressable with additional analysis rather than requiring a rejection. I would also ask the authors to temper the 'outperform in accuracy' phrasing until uncertainty is reported in Table 3 and the Arena-Hard differences are presented as non-significant. The cost comparisons are favorable to the proposed open-weight judges, and while they are transparent, a sensitivity check would make them more persuasive. Overall, the contribution is within the scope of the venue and, after revision, could be a useful reference for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is essentially a large, careful hyperparameter search over LLM judge designs, made affordable by multi-fidelity. The 1/1000 cost claim is a fair reading of their tuning-cost estimate, not a marketing trick. What's new: they search 4,480 configurations varying model, prompt template, output format, order averaging, and temperature; they show human agreement has better signal-to-noise than Spearman correlation for separating judge quality; and they produce open-weight judges that beat fine-tuned PandaLM and roughly match GPT-4-based Arena-Hard on the tested sets. Code and annotation data are released.\n\nWhat it does well: the experimental design is mostly careful—held-out LMSys test, bootstrap errors on most tables, external benchmarks (PandaLM, Arena-Hard), and a limitations section that checks position-bias flip rates. The hyperparameter insights (pair format, temperature sensitivity, order averaging, example helping large models but hurting small ones) are useful and likely to be cited.\n\nThe soft spots are real but manageable. The stress-test is right that the choice of human agreement on the filtered LMSys set as the optimization proxy is not validated at the configuration level against the target metrics. Section 4 justifies the proxy by aggregate scaling and variance, not by showing that ranking configs by human agreement matches ranking by Spearman or PandaLM. Fig. 13 is split-half reliability, not proxy-target validity. So the search may be selecting for something correlated with, but not identical to, the reported accuracy targets. That said, the external results on PandaLM and Arena-Hard provide some reassurance that the selected judges generalize. The abstract's 'outperform… in accuracy' is a bit strong: Table 2 shows they match, not beat, Arena-Hard on LMSys test, and Table 4's advantage over GPT-4 is within bootstrap noise. The strongest accuracy result is PandaLM (Table 3), which has no error bars. The instruction filtering (Llama3-8B quality threshold) is untested for sensitivity. Cost numbers are estimates from their own runtime and token pricing—reasonable but not independently audited.\n\nOverall, this is a solid empirical paper with a genuine practical contribution. The proxy-validity gap is a weakness but doesn't undermine the core finding that you can tune open-weight judges cheaply and get competitive accuracy on held-out sets. The paper is honest about limitations and ships code and data.\n\nWho it's for: people building or using LLM judges, evaluation researchers, and AutoML folks. It deserves serious peer review. I'd accept it with a request to tone down the abstract and ideally add config-level proxy-target correlation, or at least error bars on Table 3. Send it to review.","headline":"A well-executed, cost-effective judge-tuning pipeline with real practical value; the accuracy claims rest more on PandaLM than on Arena-Hard, and the proxy-validity gap is real but not disqualifying.","tokens_in":18157,"tokens_out":3464,"would_cite":true,"duration_ms":31928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Systematic hyperparameter search produces open-weight LLM judges that match or beat proprietary evaluators at a fraction of the cost.","keywords":["LLM-as-a-judge","hyperparameter tuning","multi-fidelity optimization","multi-objective optimization","human agreement","open-weight models","automatic evaluation","prompt search"],"falsifier":"If a fresh batch of human-annotated pairwise battles were collected without the specificity and quality filter, and the paper's selected judges no longer matched human preference or human-ELO rankings while a GPT-4-based judge still did, the claim that agreement on the filtered subset is the right selection signal would be refuted.","tokens_in":17147,"feed_emoji":"⚖️","tokens_out":11533,"duration_ms":97924,"temperature":0.7,"pith_summary":"LLM-as-a-judge systems are usually compared with many design decisions changed at once, so it is hard to know what makes one judge better than another. The paper treats judge design as a search problem: vary the open-weight base model, temperature, prompt structure, output format, and whether both answer orders are averaged, and optimize for agreement with human annotations while tracking cost per annotation. The claim is that this search, made affordable by multi-fidelity pruning and a cheap but discriminating human-agreement metric, finds judges that match or beat existing closed-model and fine-tuned judges on standard test sets while costing roughly $1/100$ to $1/250$ as much per 1,000 annotations, with the entire search itself costing about 1,000 times less than a naive grid. If true, high-quality automatic evaluation no longer requires proprietary APIs or expensive human annotation, and judge performance becomes something the community can cheaply tune and reproduce.","feed_headline":"Open-weight LLM judges match GPT-4 judges at 1/100th the cost","feed_subtitle":"A systematic search over 4,480 judge setups reaches proprietary-evaluator accuracy at a fraction of the price.","key_machinery":"The machinery is a parametrized judge configuration: an open-weight base model, a temperature, a boolean for averaging both answer orders, and a prompt drawn from 80 combinations of five output formats (best-model letter, Likert scale, pairwise scores, preference score, multi-criteria average), three optional pre-output fields (confidence, answer, explanation), JSON versus raw text, and the presence of an example. Each configuration is scored by human agreement, the fraction of pairwise battles where the judge's preference equals the human label, with cost per annotation as the second objective. The search runs all 4,480 configurations on 400 validation battles, keeps the top 1,200 by non-dominated sort, runs those on 1,200 battles, then runs the top 400 on 3,548 battles. Non-dominated sort ranks configurations in Pareto layers so that one survives only if no other configuration is both more accurate and cheaper, which is what lets the search prune early and find cheap-accurate judges without evaluating every configuration on the full set.","core_discovery":"The central claim is that scaling a judge, using a larger base model or more instructions, is necessary but not sufficient, and that the remaining performance gap is closed by tuning a structured set of design decisions that previous work changed all at once. The paper shows that human agreement, the fraction of pairwise battles on which judge and human agree, separates judge configurations more reliably than Spearman correlation against human-ELO ratings given the same annotation budget, so it is used as the selection metric. Running a three-stage multi-fidelity search over 4,480 configurations produces a Pareto front of open-weight judges, and the selected small, medium, and large judges match or beat GPT-4- and Claude-based Arena-Hard judges on Spearman correlation and outperform PandaLM judges on human agreement, at roughly $1/100$ to $1/250$ of the cost per 1,000 annotations. The result is not a single universal prompt but a method for finding cost-accuracy tradeoffs.","pith_inferences":["Editorial inference: because the search already optimizes two objectives, the same pipeline could be rerun with additional objectives such as verbosity bias, self-preference bias, or position-bias flip rate, turning known judge weaknesses into optimization targets instead of post-hoc corrections.","Editorial inference: the paper's prompt-performance transfer across model sizes suggests that the winning prompt choices are likely to transfer to the next generation of open-weight models, although the paper notes that new reasoning capabilities could break this transfer.","Editorial inference: the cost ratio is computed on self-hosted GPU prices; on commercial API providers absolute costs will be higher, but the gap to proprietary closed-model judges should persist because those judges are billed through APIs with their own margins."],"forward_implications":["Judge selection can be treated as a cheap tuning problem: the multi-fidelity search over 4,480 configurations costs roughly $2,000, versus an estimated $2 million for evaluating the same configurations with standard benchmark protocols.","Practitioners get a Pareto front of open-weight judges, so they can choose a small, medium, or large judge to fit a memory or latency budget without sacrificing the search's accuracy gains.","The tuned open-weight judges match or beat GPT-4- and Claude-based Arena-Hard judges on Spearman correlation with human-ELO rankings, at roughly $1/100$ to $1/250$ of the per-annotation cost.","Design choices that consistently help are low temperature, averaging both answer orders, and the pairwise-score output format, while asking for explanations or answers before the preference tends to hurt.","Fine-tuned judge baselines such as PandaLM and JudgeLM are outperformed on the test sets even though the tuned judges are zero-shot, not fine-tuned, models."],"supporting_citations":[{"why":"Supplies the human-preference battle dataset from which the filtered instructions are taken; all validation and test agreement scores are computed on it.","marker":"(lin Chiang et al., 2024)"},{"why":"Supplies the human-ELO leaderboard used as the golden ranking for Spearman-correlation evaluations of judges.","marker":"(Chiang et al., 2024)"},{"why":"Defines the Arena-Hard judge protocol, the primary baseline and test benchmark, and provides the prompt used to filter instructions.","marker":"(Li et al., 2024)"},{"why":"Supplies the PandaLM test set and the fine-tuned PandaLM judges that the tuned open-weight judges are compared against.","marker":"(Wang et al., 2024)"},{"why":"Provides the per-model annotation cost estimates used to compute the millions-dollar cost of naive grid search and the per-annotation comparison.","marker":"(Ni et al., 2024)"},{"why":"Supplies successful halving, the multi-fidelity mechanism that prunes poor judge configurations at early low-budget stages.","marker":"(Karnin et al., 2013)"},{"why":"Supplies non-dominated sorting, the multi-objective ranking used to keep the Pareto-best judges by accuracy and cost.","marker":"(Emmerich & Deutz, 2018)"},{"why":"Supplies the epsilon-net tie-breaking used with non-dominated sort in the multi-fidelity multi-objective selection.","marker":"(Schmucker et al., 2021)"}],"fun_headline_variants":["Systematic tuning yields LLM judges at 1/100th cost","Open-weight judges match GPT-4 at 1/100th cost","Multi-fidelity search finds cost-accuracy sweet spot for judges","Tuning judge design, not just scale, cuts cost 100x","Cheap open judges rival expensive ones via design tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that human agreement on a filtered subset of pairwise battles, where instructions are kept only if a small helper model rates them specific and high-quality, is a faithful proxy for judge quality in general, so judges chosen to maximize agreement on that subset will also rank models correctly on other test sets.","fun_headline_variants_meta":{"raw":{"variants":["Systematic tuning yields LLM judges at 1/100th cost","Open-weight judges match GPT-4 at 1/100th cost","Multi-fidelity search finds cost-accuracy sweet spot for judges","Tuning judge design, not just scale, cuts cost 100x","Cheap open judges rival expensive ones via design tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1610,"prompt_tokens":917,"completion_tokens":693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":603}},"tokens_in":533,"tokens_out":693,"duration_ms":6834,"temperature":1.0,"reasoning_tokens":603,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:57:12.517000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a fresh batch of human-annotated pairwise battles were collected without the specificity and quality filter, and the paper's selected judges no longer matched human preference or human-ELO rankings while a GPT-4-based judge still did, the claim that agreement on the filtered subset is the right selection signal would be refuted.","supporting_citations":[{"cited_title":"Panda LM : An automatic evaluation benchmark for LLM instruction tuning optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the PandaLM test set and the fine-tuned PandaLM judges that the tuned open-weight judges are compared against."},{"cited_title":"Almost optimal exploration in multi-armed bandits","cited_arxiv_id":null,"evidence_quote":"Supplies successful halving, the multi-fidelity mechanism that prunes poor judge configurations at early low-budget stages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies non-dominated sorting, the multi-objective ranking used to keep the Pareto-best judges by accuracy and cost."}],"review_version":1}