{"id":"9d544b02-dc41-419d-a40a-e48549ed53ca","arxiv_id":"2502.04997","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A linear correction learned from 100 labeled examples per task substantially improves LLM-human agreement on evaluation tasks, including for smaller models.","lead":"By learning a simple linear map from an LLM's label choices to human labels, using only a small calibration set, this paper raises agreement with human judgments by 142% on average across 29 tasks. The method needs no model retraining or logit access, making it an easy upgrade for LLM-as-a-judge pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No majority-class baseline: reported gains may be a base-rate artifact, so the claim that the linear mapping aligns LLM judgments is not established.","rationale":"The central claim rests on the empirical improvement in Tables 1, 2, and 5. However, the proposed method with λ→0 is equivalent to a per-category majority remap: W_z,y ≈ P(Y=y|Z=z), and prediction is the modal human label for each LLM label. If the LLM's output is uninformative or concentrated, the method becomes the majority-class predictor. The paper does not report a majority baseline, nor the distribution of LLM labels, nor the learned W. On tasks where non-aligned accuracy is in the low single digits (e.g., SummEval fluency, Medical Safety response type), the post-alignment accuracy (~80%) is consistent with a constant predictor for a highly skewed label distribution. Thus the 142% improvement may not reflect alignment of LLM judgments but rather base-rate correction. This is more load-bearing than the reader's input-independence concern because it questions whether the LLM output contributes at all. The concrete test—adding a majority-class baseline and reporting W—would settle this. If the baseline matches, the paper's contribution is substantially weakened and the framing must change; if the aligned method beats the baseline, the central claim is supported. Therefore the verdict should remain conditional, requiring this baseline analysis.","tokens_in":16622,"tokens_out":13405,"duration_ms":143129,"concrete_test":"Compute a majority-class baseline: for each random split used in Tables 1 and 5, predict the most frequent human label in the training set for all test instances, and report test accuracy. Also report the learned W matrix and the marginal distribution of LLM labels in the calibration set. If the majority baseline equals or exceeds the aligned accuracy on the tasks with large improvements, the method is not exploiting LLM output information and the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The learned mapping W in Eq. (2) with λ=1e-6 is essentially the empirical conditional distribution P(Y|Z): for each LLM label z, the row W_z,: is the proportion of calibration examples with that label that received each human label, and the argmax in Eq. (3) selects the modal human label for that z. Thus the method is a per-category majority remap. If the LLM's output is uninformative or highly concentrated (e.g., always the same label), this reduces exactly to predicting the majority human label in the training set. The paper never reports a majority-class baseline, nor the distribution of LLM labels, so the 142% improvement cannot be attributed to the LLM's judgments carrying signal; on skewed tasks (e.g., SummEval fluency 2.5% -> 80%, Medical Safety response type 5% -> 80%), the observed aligned accuracy is exactly what a constant majority predictor would achieve. Without this baseline, the central claim that the linear mapping aligns LLM judgments is not established: the improvement may be a trivial base-rate effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc alignment method for black-box LLM judges. It learns a linear map W from the one-hot encoded LLM labels to human labels via ridge regression (Eq. 2) on a small calibration set, and at inference time outputs the human label with the largest transformed value (Eq. 3). The method is evaluated on 29 tasks from Judge-Bench plus Feedback-QA, using Claude-3 Sonnet, Mixtral 8x7B, and Llama-3 70B, with 10 random splits. The authors report an average relative improvement of 142% in accuracy over raw LLM labels, show that the approach outperforms in-context learning, and claim that aligned smaller models match larger ones and that the aligned judgments exceed inter-human agreement on four of six multi-annotator tasks.","tokens_in":16786,"tokens_out":6220,"duration_ms":59170,"significance":"The paper offers a simple, logit-free, calibration-set-based approach that can be applied to any LLM without fine-tuning. The mathematical derivation is clear, and the evaluation is repeated with standard deviations. If the gains are real and not a base-rate artifact, the method would be a practically useful post-processing step for deploying LLM judges. The paper's main weakness is that the learned map is essentially a per-category majority remap, and the authors never compare against a majority-class baseline or report LLM label distributions; this omission currently undermines the interpretation of the headline improvement. The paper is valuable as a demonstration of label-bias correction, but the claims need to be scaled to what is actually established.","major_comments":[{"comment":"The learned mapping in Eq. (2) with λ=10^-6 is, for one-hot encoded Z, a per-category majority remap: each row of W is proportional to the empirical distribution of human labels among training examples with that LLM label, and Eq. (3) selects the modal human label for that LLM label. The paper does not report a majority-class baseline (e.g., predicting the overall training majority label for every test item) or the distribution of LLM labels. On heavily skewed tasks such as SummEval Fluency and Medical Safety Response Type, the aligned accuracies (≈80% and ≈70–80%) are close to what a constant majority predictor would achieve, so the 142% relative improvement over raw LLM accuracy may reflect base-rate correction rather than alignment of the LLM's judgments. The authors should add this baseline, report label distributions, and either temper the alignment claim or explicitly frame the contribution as correcting label bias.","section":"§3.1, Eq. (2), Table 1"},{"comment":"The headline '142% average improvement' is a relative gain and is inflated when non-aligned accuracy is near zero, as in the single-digit baselines in Table 1. The paper should also report absolute differences, the distribution of per-task relative improvements, and the proportion of tasks where aligned accuracy exceeds the majority baseline. Reporting the mean of relative improvements across tasks without these context statistics makes the main quantitative claim hard to interpret.","section":"Abstract, §3.1"},{"comment":"The statement that the aligned method 'exceeds inter-human agreement on four out of six tasks' is not backed by an explicit comparison. The six tasks are not enumerated, and it is unclear whether inter-human agreement is measured as accuracy against individual annotators or against a majority vote, and whether the aligned accuracy is computed on the same examples and with the same metric. The authors should list the tasks, show the inter-human agreement for each, and provide confidence intervals or a significance test before making this claim.","section":"§5, Table 1"},{"comment":"The in-context learning comparison reports relative improvements (e.g., 111.74%) without a majority baseline as well. Since the alignment mapping is still a per-category majority remap on top of the ICL outputs, the same base-rate concern applies. Please report the ICL label distributions and a majority predictor on the ICL outputs.","section":"§3.2, Table 3"}],"minor_comments":[{"comment":"'human judments' is a typo for 'human judgments'.","section":"§3.2"},{"comment":"'the each row' should be 'each row'.","section":"Table 1 caption"},{"comment":"'the same graph' should probably be 'the same data'.","section":"Figure 1 caption"},{"comment":"The notation for the j-th component of z^T W_hat is garbled in the rendered text; please use a clear subscript.","section":"§2.4"},{"comment":"The paper does not specify how the method handles LLM labels that never appear in the calibration set; for such rows W would be zero, and the argmax in Eq. (3) would be arbitrary. A short statement on this case would be helpful.","section":"§2.4"},{"comment":"The related work section does not discuss prior work on label-shift or prior-correction methods; a brief connection would help position the contribution.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the method is simple enough to be reproducible, but the missing majority baseline is a substantive gap. I would like to see the baseline and label distribution analysis before a final decision. I have no concerns about attribution or novelty beyond what is stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a straightforward label-remapping method for LLM-as-judge tasks: learn W from one-hot LLM labels to one-hot human labels via ridge regression, then argmax. With their lambda, W is essentially the empirical conditional distribution P(human label | LLM label), so the aligned prediction is the modal human label for each observed LLM label. That's not a flaw per se, but it means the method is a conditional majority remap, not a learned semantic alignment.\n\nWhat's genuinely useful: the evaluation is broad (29 tasks, three models, repeated splits), the transfer experiment is a nice addition, and the low-data results (20 samples) are practical. The method is black-box and trivial to implement, which makes it a convenient baseline for future work. I'd cite it for that.\n\nThe soft spots are real and mostly missing baselines. The stress-test note is right: nowhere do they compare against 'always predict the majority human label' or report the LLM's label distribution. On skewed tasks like SummEval fluency and Medical Safety response type, the aligned accuracy (~80%) is exactly what a constant majority predictor would give, and the non-aligned accuracy is low only because the LLM uses a different label scale. So the headline '142% improvement' can be a base-rate artifact. The paper also calls this 'zero-shot' while training a mapping on 100 labeled examples; that's a misuse of the term. And the inter-human agreement numbers look off (SummEval fluency 4.5%) and need a clear definition.\n\nThe reader's report underplays this: it notes the soft spots but doesn't flag the majority baseline, which is the load-bearing issue. The method may still be useful for correcting systematic response-style bias, but the paper does not establish that the LLM's per-item judgments add signal beyond the label distribution. A revision could fix this with one column in Table 1.\n\nWho's this for? Anyone working with LLM-as-judge who wants a quick bias-correction step. It deserves a serious referee and a major revision; the empirical scope is valuable and the missing baseline is easy to add. I'd send it to review.\n\nRecommendation: engage, but require the majority-class baseline and label distribution analysis before accepting the central claim.","headline":"Simple per-category label remap, useful as a baseline but oversold; the 142% claim needs a majority-class comparison before it means what the authors say.","tokens_in":17329,"tokens_out":3957,"would_cite":true,"duration_ms":41858,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single linear remapping of LLM output labels to human labels raises agreement by 142% on average across 29 judgment tasks.","keywords":["LLM-as-a-judge","human alignment","linear mapping","ridge regression","black-box calibration","response style bias","label remapping","human evaluation"],"falsifier":"Select a task with two clearly distinct item types, learn the mapping on items of the first type, and evaluate on items of the second type when both types have the same human label distribution; if agreement after remapping does not beat the unaligned baseline, the input-independent mapping assumption fails.","tokens_in":16388,"feed_emoji":"🎯","tokens_out":6335,"duration_ms":62866,"temperature":0.7,"pith_summary":"The paper sets out to show that most of the disagreement between an LLM's categorical judgments and human judgments is a systematic label bias, not random noise or a reasoning failure. It proposes learning a single linear mapping from the LLM's output labels to human labels using a small calibration set, then applying that mapping to all future judgments from the same model and task. Across 29 tasks and three black-box models, the authors report that this remapping raises LLM-human agreement by an average of 142 percent, using no model logits, no fine-tuning, and as few as 100 calibration examples. If correct, this means a large part of the 'LLM judge' problem can be fixed by a post-hoc correction rather than by building better judges.","feed_headline":"One linear remap lifts LLM-human judge agreement 142%","feed_subtitle":"Learned from about 100 examples, the correction works without logits and lets small models match large ones.","key_machinery":"The load-bearing object is the transformation matrix $W \\in \\mathbb{R}^{m \\times n}$, where $m$ is the number of LLM label categories and $n$ the number of human label categories. It is fit by ridge regression, $\\hat{W} = (Z^{\\top}Z + \\lambda I)^{-1} Z^{\\top} Y$, on one-hot encoded judgments from a small training set, and applied at inference by $\\varphi(z) = \\arg\\max_{j} (z^{\\top}\\hat{W})_j$. This maps each LLM label to the human label it is most associated with, and it works even when the LLM and human label sets differ in size or meaning. The mapping is what carries the argument: because it needs only the LLM's final categorical output, the method works on black-box models with no access to logits or weights.","core_discovery":"The central discovery is that LLM judges and human judges disagree in a structured, mostly one-directional way: LLMs have their own response styles, such as avoiding negative labels and clustering on highly positive ones, while human annotators spread their labels more evenly. The paper claims that a single input-independent linear transformation $W$, learned by ridge regression from one-hot encodings of LLM labels to one-hot encodings of human labels, corrects this bias and substantially improves agreement with humans on held-out test items. In their experiments this gives a 142% average relative improvement across 29 tasks, exceeds inter-human agreement on four of six multi-annotator tasks, and lets a smaller model such as Mixtral 8x7B Instruct match larger proprietary judges after alignment.","pith_inferences":["A direct testable extension would be to condition the mapping on input features, since tasks where the human meaning of a label changes with content are exactly where the single $W$ should break down.","The result suggests that LLM judgments may already contain a reliable ordering of items, and that rank-based agreement metrics could show even stronger human alignment than the label accuracy reported here.","If LLM response style is the dominant error source, then judge benchmarks should report both raw and calibrated agreement; reporting only raw agreement may systematically undervalue black-box judges that use skewed label scales.","Replacing one-hot labels with soft probability vectors whenever logits are available would likely refine the mapping; the paper notes this possibility but leaves it untested."],"forward_implications":["LLM-based evaluation pipelines can become human-aligned by adding a small labeled calibration set and a post-hoc remapping step, with no retraining and no access to model internals.","Automated judges that currently disagree with humans because of response style can be used in safety-relevant settings such as medical triage only after this kind of alignment, since raw judgments may be systematically skewed.","Smaller, cheaper models can substitute for larger proprietary judges once their labels are remapped, reducing the cost of running evaluations at scale.","A mapping learned on one task transfers to related tasks with the same label scale, so calibration effort can be shared across similar evaluation setups.","In-context examples of human judgments are not a substitute for the alignment step; the structured remapping adds agreement on top of few-shot prompting."],"supporting_citations":[{"why":"Supplies the Judge-Bench tasks, prompts, and human annotations used for all experiments except Feedback-QA.","marker":"Bavaresco et al. (2024)"},{"why":"Ridge regression gives the closed-form solution for the mapping matrix.","marker":"Hoerl and Kennard (1970)"},{"why":"Defines Claude-3 Sonnet, one of the three black-box judges whose outputs are aligned.","marker":"Anthropic (2024)"},{"why":"Defines Mixtral 8x7B Instruct, the smaller model shown to match larger ones after alignment.","marker":"Jiang et al. (2024)"},{"why":"Defines Llama-3 70B Instruct, the third judge and the model with the largest average improvement.","marker":"Dubey et al. (2024)"},{"why":"Provides the WMT-20-EnDe translation-grading task and the multi-annotator human judgments used in Figure 1.","marker":"Freitag et al. (2021)"},{"why":"Provides the Medical-Safety task where alignment lifts agreement from single digits to roughly 80 percent.","marker":"Abercrombie and Rieser (2022)"}],"fun_headline_variants":["One linear map boosts LLM-human judge agreement 142%","Linear remap aligns LLM judges with humans, 142% better","One learned matrix makes LLM judges match human taste","No retraining: linear map aligns LLM judges to humans","Small linear fix makes LLM judges 142% more human"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on the premise that the disagreement between a given LLM and the humans it is aligned to is one consistent label bias, so that a single per-task remapping learned from about 100 examples also applies to every other item in that task.","fun_headline_variants_meta":{"raw":{"variants":["One linear map boosts LLM-human judge agreement 142%","Linear remap aligns LLM judges with humans, 142% better","One learned matrix makes LLM judges match human taste","No retraining: linear map aligns LLM judges to humans","Small linear fix makes LLM judges 142% more human"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3019,"prompt_tokens":901,"completion_tokens":2118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2032}},"tokens_in":517,"tokens_out":2118,"duration_ms":13416,"temperature":1.0,"reasoning_tokens":2032,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:40:01.371636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select a task with two clearly distinct item types, learn the mapping on items of the first type, and evaluate on items of the second type when both types have the same human label distribution; if agreement after remapping does not beat the unaligned baseline, the input-independent mapping assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ridge regression gives the closed-form solution for the mapping matrix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Claude-3 Sonnet, one of the three black-box judges whose outputs are aligned."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Medical-Safety task where alignment lifts agreement from single digits to roughly 80 percent."}],"review_version":1}