{"id":"936b398d-b490-4ebe-a4bb-d209342fc6d6","arxiv_id":"2505.16003","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SLMEval fits a latent strength distribution to human preferences by maximum entropy and uses it to reweight LLM judge scores, reporting stronger correlation with human judgment on two production tasks.","lead":"This paper proposes SLMEval, a method that reweights LLM judge scores using human preference data to better match human opinions on open-ended tasks. The method claims higher correlation with human judgment and lower cost than GPT-4-based evaluators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out split is described between the human annotations used to fit SLMEval's weights p and those used to compute the reported Spearman correlations, so the 0.57 headline may be an in-sample fit rather than predictive alignment.","rationale":"I agree with the reader that this is the load-bearing weakness. The proposed entropy-calibration method is a plausible contribution, and the cost claim is supported by using a local 4-bit SLM; but the empirical superiority over G-Eval rests entirely on the correlation numbers in Table 1/4 and accuracy in Table 2. The table correlations are model-level Spearman ranks over only 10 models, and the paper does not document that the human labels used to set pi are distinct from the human labels used as the reference ranking. This is not a matter of consensus—it is an internal validity question: without a split, the comparison is not an out-of-sample evaluation. I therefore see no reason to change the reader's REJECT. The one concrete repair would be a held-out split or fresh human judgments, which is testable and could salvage the claim.","tokens_in":9234,"tokens_out":4219,"duration_ms":39395,"concrete_test":"Obtain from the authors (or from released code/data) the 360 annotations per task and re-run SLMEval with an explicit split: use 80% of the pairwise judgments to fit p via Eq. 5, then compute win-rate ranking and Spearman rho on the held-out 20%; repeat with 5-fold cross-validation and bootstrap resampling. Also report a two-sided p-value or 95% CI for rho at n=10. If held-out rho is below 0.3 or not significant, the 0.57 headline is not a predictive result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 says 360 Clickworker evaluations are collected per task; Section 5.1 reports Spearman correlations of evaluator rankings with human judgments; Appendix A.5 defines D_human as the human-labeled dataset used in the preference constraints (Eq. 5). The paper never states that the comparisons used to estimate p are disjoint from the comparisons used to compute the correlation, nor does it report any train/test split. If the same 360 annotations are used for both, then p is optimized to satisfy constraints derived from the very labels against which the final ranking is scored, making 0.57 a measure of calibration fit rather than predictive correlation. The n=10 model set also gives a Spearman estimate with wide uncertainty; a value of 0.57 on n=10 is not significant at the two-sided 5% level, and no confidence interval is reported. The central claim therefore rests on an untested independence assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SLMEval, a calibration method for LLM-as-a-Judge evaluation. It estimates a latent strength vector p over the evaluated models by maximizing Shannon entropy subject to relaxed Bradley-Terry-style preference constraints derived from a small set of human pairwise comparisons, then reweights pairwise evaluator scores S_ij by p_i before aggregating win rates into a final ranking. Experiments on two internal production tasks (Peptalk and Recommendation) and on the FairEval public benchmark report that SLMEval improves Spearman correlation over baselines (0.48 and 0.57 on the two production tasks) and reduces API cost by an order of magnitude relative to GPT-4-based calibrated evaluators.","tokens_in":9417,"tokens_out":5094,"duration_ms":43941,"significance":"If the empirical claims hold, SLMEval would be a useful practical contribution: the maximum-entropy formulation is simple and transparent, it requires only a small amount of human preference data, and the use of a 4-bit quantized small model is attractive for cost-sensitive deployment. The comparison against G-Eval, GPTScore, and GPT-4+BPC on a public benchmark is also informative. However, the current validation has a potentially circular evaluation protocol and reports correlations on only ten models without significance testing, so the headline numbers should not be accepted at face value until the protocol is clarified and the results are re-analyzed with an appropriate held-out split.","major_comments":[{"comment":"The paper never states whether the human preference data used to fit p via the constraints in Eq. (5) is disjoint from the human judgment data used to compute the Spearman correlations in Table 1. Section 4.3 describes collecting 360 Clickworker evaluations per task, and Section 5.1 reports correlations against human judgments, but no train/test split or cross-validation is described. If the same 360 annotations are used both to constrain p and to compute the reported ρ, then the values 0.48 and 0.57 measure in-sample calibration fit rather than predictive alignment with human judgment. The authors must specify the split or re-run the evaluation with a held-out set; without this, the central empirical claim is not supported.","section":"§4.3, §5.1, Appendix A.5"},{"comment":"The relaxed constraint pi ≥ P(i>j)(pi + pj) can be infeasible under intransitive human preferences, despite the paper's claim that the relaxation provides robustness to violations of transitivity. For example, if P(A>B)=P(B>C)=P(C>A)=0.6, the constraints imply pA ≥ 1.5 pB ≥ 2.25 pC ≥ 3.375 pA, which has no solution. The paper does not discuss infeasibility, add a feasibility fallback, or report that the empirical D_human satisfied the acyclicity needed for a feasible solution. Since the SLSQP solver would either fail or return an arbitrary point in such cases, this is a load-bearing gap in the method description.","section":"§3.3, Eq. (5), Appendix A.5.1"},{"comment":"All reported correlations are computed over only 10 models. For n=10, a Spearman correlation of 0.57 has a two-sided p-value of roughly 0.085, so the headline result is not statistically significant at the conventional 5% level; no confidence intervals or significance tests are reported. Given that several baselines also have large-magnitude but noisy correlations (e.g., GPTScore at 0.39 on Recommendation), the claim that SLMEval 'achieves strong correlation' needs additional support, such as resampling-based intervals or a larger model set.","section":"Table 1, Table 4"}],"minor_comments":[{"comment":"Equation (5) contains a typo: '∀ ∈ Dhuman' should be '∀(i,j) ∈ Dhuman'.","section":"§3.3, Eq. (5)"},{"comment":"The baseline 'GPT-4 + BPC (k = 3)' is never defined; the acronym BPC and the calibration procedure should be described, since this is the strongest baseline on FairEval.","section":"Table 1, Table 2"},{"comment":"The description of the FairEval evaluation is too terse to reproduce: the paper should state what 'accuracy' means, how MT-Bench serves as the human reference, how many examples are used, and how the cost estimates are computed.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the missing held-out split. I would ask the authors to either document a split or re-run with leave-one-out / cross-validation over the human preference data. If they cannot produce held-out correlations, the empirical claims in the abstract should be withdrawn. The infeasibility of the preference constraints under cycles is a second issue that needs a concrete resolution. These are fixable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core idea is a decent one: estimate latent model strengths p by maximum entropy subject to relaxed Bradley-Terry constraints from a small human preference set, then use those weights to reweight an SLM judge's pairwise scores. It is cheap, single-pass, and aimed at open-ended production tasks where G-Eval really does seem to go negative. Second, the headline numbers are not trustworthy as reported because the paper never says the human annotations used to fit p are separate from the human judgments used to compute the Spearman correlations. Section 4.3 collects 360 evaluations per task; Appendix A.5 defines D_human for the constraints; Section 5.1 reports correlations against those same judgments. If it is the same 360, then the 0.57 on Recommendation measures how well p fits the training labels, not predictive alignment. The paper needs an explicit train/test split or cross-validation before the claim holds.\n\nWhat the paper does well: the entropy-maximization formulation is clean and the implementation is transparent—SLSQP, uniform initialization, epsilon bound, full prompts and complete correlation table in the appendix. The authors are upfront that the scope is narrow. The cost argument is plausible, and the FairEval comparison provides an independent sanity check, though the 58.8% accuracy there is barely above chance and not statistically framed.\n\nSoft spots in proportion: the no-split issue is the big one—load-bearing, not minor. On top of that, the correlation estimates rest on 10 models; a Spearman of 0.57 on n=10 is not significant at the 5% level, and no confidence intervals are given. The baselines include implausibly negative values (GPT-4+BPC at -0.48 on Recommendation) that suggest task-specific prompt or annotation effects, and the paper doesn't analyze them. The cost claim compares local SLM inference with API pricing; it's okay but not apples-to-apples.\n\nBottom line: the method is worth a serious look, but the experimental evidence as written does not support the central claim. The fix is within reach—a re-analysis with held-out human judgments or cross-validation. I'd send this to peer review with a strong request to repair the evaluation protocol, not desk-reject it. The paper speaks to people building cheap, deployable evaluators for subjective tasks; they just need to show the calibration generalizes.","headline":"The entropy-based reweighting idea is neat and cheap, but the headline correlations likely reflect the fitted human data rather than predictive alignment, and the paper needs a held-out split before the numbers can be believed.","tokens_in":9936,"tokens_out":2631,"would_cite":false,"duration_ms":23540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On open-ended tasks, SLMEval claims a maximum-entropy reweighting of raw evaluator scores, fit on a small human-preference set, turns a 4-bit small model into a more human-aligned judge than GPT-4-based calibration, at 5–30x lower cost.","keywords":["LLM-as-a-Judge","entropy maximization","human-aligned evaluation","calibration","small language models","open-ended generation","Spearman correlation","reference-free evaluation"],"falsifier":"Run SLMEval with $p$ fit on one half of the human evaluations and the Spearman correlation computed only on the other half; if the held-out correlation drops toward zero or negative, the reported 0.57 does not represent predictive alignment. Bootstrap the ten-model ranking as well, since with $n=10$ a single model's position can move the correlation substantially.","tokens_in":9050,"feed_emoji":"🎯","tokens_out":8945,"duration_ms":71669,"temperature":0.7,"pith_summary":"The paper claims that the usual fix for LLM-as-a-Judge evaluation—calibrating scores with GPT-4, chain-of-thought prompts, or token-level probabilities—breaks down on subjective, open-ended tasks, where several strong evaluators show weak or negative correlation with human judgments. SLMEval is the proposed alternative: it uses a small amount of human preference data to fit a latent \"strength\" weight for each model under evaluation, choosing the weights by maximum entropy so that they satisfy the observed preferences while staying as unbiased as possible. Those weights are then multiplied into the raw evaluator scores before pairwise comparison. In the paper's results, this lifts Spearman correlation with human judgments to 0.48 on a daily-pep-talk task and 0.57 on a recommendation task, where G-Eval with chain-of-thought scores 0.41 and -0.55, and it does so with a 4-bit quantized small model at 5–30x lower estimated cost than GPT-4-based evaluators. The reason this matters is practical: if it holds, production teams can get human-aligned evaluation of creative or personalized outputs without expensive GPT-4 calls.","feed_headline":"Entropy calibration makes a small model beat GPT-4 judges on taste","feed_subtitle":"SLMEval hits Spearman 0.57 where G-Eval goes negative, at an estimated 5–30x lower cost.","key_machinery":"The load-bearing object is the latent strength vector $p=(p_1,\\ldots,p_n)$, a probability distribution over the models being rated. SLMEval fits $p$ by maximizing the Shannon entropy $H(p)=-\\sum_i p_i \\log p_i$ subject to normalization and to the inequality constraints $p_i \\ge P(i>j)(p_i+p_j)$ for every human-annotated pair $(i,j)$, which relaxes the Bradley-Terry model $P(i>j)=p_i/(p_i+p_j)$ so that noisy and intransitive human data can be tolerated. The fitted weights are applied multiplicatively to the raw evaluator scores, so model $i$ beats model $j$ exactly when $p_i S_{ij} > p_j S_{ji}$, and models are ranked by aggregate calibrated win rate. The entropy objective is what keeps the inferred distribution maximally agnostic about everything except the observed preference constraints.","core_discovery":"On its own terms, the central discovery is that the alignment failure of LLM-based evaluators on open-ended tasks is a calibration problem rather than a capability problem. SLMEval models each evaluated model $i$ with a latent strength $p_i$, estimates the vector $p$ by maximizing Shannon entropy subject to normalization and to relaxed Bradley-Terry-style constraints $p_i \\ge P(i>j)(p_i+p_j)$ built from human preferences, and then redefines the pairwise win as $p_i S_{ij} > p_j S_{ji}$. With a 4-bit quantized LLaMA 3.1 as the scorer, the calibrated win-rate ranking reaches Spearman correlation 0.48 on the pep-talk task and 0.57 on the recommendation task against human judgments, compared with 0.41 and -0.55 for G-Eval with chain-of-thought; on the public FairEval benchmark it reaches 58.8% pairwise accuracy against MT-Bench human labels, approaching GPT-4 + BPC at 62.5%. The same single-pass, small-model design is what yields the 5–30x cost reduction.","pith_inferences":["Editorial extension: the paper never states a train/test split between the 360 human evaluations used to fit $p$ and the human judgments used to compute the Spearman correlations, so the reported 0.57 may partly reflect fitting; a held-out replication would settle it.","Editorial extension: a transfer test—fit $p$ on one task and apply it to another—would show whether per-task human annotation is avoidable or whether each subjective task needs its own calibration set.","Editorial extension: because the calibration only requires pairwise preference inequalities, it could in principle consume any preference signal, including implicit user behavior such as clicks or retention, rather than only curated human annotations.","Editorial extension: with only ten models in the ranking, the correlation estimates carry wide uncertainty; bootstrap confidence intervals would show whether the 0.57 gap over G-Eval is statistically meaningful."],"forward_implications":["A small human-preference calibration set can make reference-free evaluation reliable on subjective tasks, where standard metrics and GPT-4-based calibrators currently fail.","Evaluation costs drop by an estimated 5–30x because the scorer is a 4-bit quantized small model making a single pass, making large-scale or iterative human-aligned evaluation affordable.","Performance on the public FairEval dataset suggests the calibration method transfers beyond the two in-house tasks, outperforming GPTScore and plain G-Eval and approaching GPT-4 + BPC.","The negative correlations of strong baselines on the recommendation task imply that production teams should not trust benchmark-validated evaluators without task-specific calibration."],"supporting_citations":[{"why":"Supplies the maximum-entropy principle used to pick the least-biased strength distribution $p$.","marker":"(Jaynes, 1957)"},{"why":"Provides the paired-comparison model $P(i>j)=p_i/(p_i+p_j)$ that SLMEval relaxes into inequality constraints.","marker":"(Bradley and Terry, 1952)"},{"why":"G-Eval is the GPT-4-based calibrated evaluator that SLMEval must beat, and its weak or negative correlations motivate the proposed method.","marker":"(Liu et al., 2023a)"},{"why":"Source of the per-model weight idea that SLMEval repurposes, replacing token-level probabilities with entropy-fitted strengths.","marker":"(Liu et al., 2023b)"},{"why":"Provides the FairEval dataset, the GPT-4 + BPC baseline, and the position-bias critique that motivates calibration.","marker":"(Wang et al., 2024)"},{"why":"GPTScore serves as a reference-free GPT-4 baseline that SLMEval outperforms on both production and public data.","marker":"(Fu et al., 2024)"},{"why":"Supplies the GPTScorer baseline and the cost-estimation methodology used to compute SLMEval's 5–30x cost reduction.","marker":"(Irugalbandara et al., 2024)"},{"why":"MT-Bench provides the human reference labels for the FairEval accuracy comparison.","marker":"(Zheng et al., 2024)"}],"fun_headline_variants":["Entropy calibration makes LLM judges match human taste on open tasks","SLMEval: small model + entropy beats GPT-4 judges at 5-30x less cost","Entropy maxing recalibrates LLM judges: 0.57 Spearman vs G-Eval's -0.55","Cheap entropy calibration makes a 4-bit LLaMA outrank GPT-4 judges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the human preference data used to fit the strength weights $p$ is not the same data used to measure SLMEval's correlation with human judgments; the paper describes 360 evaluations per task but no hold-out split, so if the same annotations do both jobs, the reported alignment is partly a reflection of the fitting constraints.","fun_headline_variants_meta":{"raw":{"variants":["Entropy calibration makes LLM judges match human taste on open tasks","SLMEval: small model + entropy beats GPT-4 judges at 5-30x less cost","Entropy maxing recalibrates LLM judges: 0.57 Spearman vs G-Eval's -0.55","Cheap entropy calibration makes a 4-bit LLaMA outrank GPT-4 judges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000884,"raw_usage":{"total_tokens":3842,"prompt_tokens":995,"completion_tokens":2847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2746}},"tokens_in":611,"tokens_out":2847,"duration_ms":16510,"temperature":1.0,"reasoning_tokens":2746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:08:05.053240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SLMEval with $p$ fit on one half of the human evaluations and the Spearman correlation computed only on the other half; if the held-out correlation drops toward zero or negative, the reported 0.57 does not represent predictive alignment. Bootstrap the ten-model ranking as well, since with $n=10$ a single model's position can move the correlation substantially.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FairEval dataset, the GPT-4 + BPC baseline, and the position-bias critique that motivates calibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPTScorer baseline and the cost-estimation methodology used to compute SLMEval's 5–30x cost reduction."}],"review_version":1}