{"id":"555feded-0cae-4a62-80be-1c68454de849","arxiv_id":"2606.05122","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Base LLMs show latent judge calibration that Self-Evaluation Elicitation (SEE) surfaces with 160 examples via RL calibration followed by masked distillation.","lead":"The paper finds that base LLMs already predict external judges' multi-attribute scores on their outputs above chance using few-shot prompts. This reframes self-evaluation as elicitation of a latent ability rather than new acquisition via heavy training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Few-shot performance could be driven by prompt format rather than a stable latent quality notion in the base model","rationale":"The reader's weakest assumption directly identifies the same load-bearing point. Full text does not appear to contain the missing prompt ablations needed to rule it out, so the concern remains the primary uncertainty for the 'already there' claim. No stronger internal inconsistency was located.","tokens_in":1685,"tokens_out":282,"duration_ms":16461,"concrete_test":"Re-run the three-benchmark few-shot evaluation using the identical base model and judge but with two altered prompt variants: (1) rephrased instructions that preserve meaning but change sentence structure and (2) shuffled or replaced few-shot examples drawn from the same distribution; measure whether mean absolute error or correlation with judge scores drops by >15% relative to the reported figures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that above-chance few-shot prediction of judge scores demonstrates a pre-existing, transferable internal notion of quality. The abstract and method description provide no ablations on prompt phrasing, example selection, or output format variations. If performance collapses under modest rephrasing of the few-shot template or different example ordering while keeping the same judge scores, the result would be consistent with surface-level pattern matching rather than elicitation of latent calibration.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that base LLMs already possess a latent ability to predict an external judge's multi-attribute quality scores on open-ended responses, which can be surfaced via few-shot prompting well above chance on three benchmarks. It introduces Self-Evaluation Elicitation (SEE), a two-phase procedure (calibration-coupled RL followed by masked distillation) that uses only 160 examples—roughly 31x fewer than a standard RL baseline—to improve held-out calibration while preserving answer quality. The resulting self-evaluation is localized in the model's token distribution and generalizes to judges not seen during training, supporting the interpretation that the model harbors a transferable internal notion of quality rather than acquiring one through targeted training.","tokens_in":1767,"tokens_out":591,"duration_ms":19780,"significance":"If the central results are robust, the work reframes judge-aligned self-evaluation as primarily an elicitation problem, which could substantially lower the data requirements for alignment techniques. The reported localization of the self-evaluation signal and its stability across unseen judges are notable strengths that would, if confirmed, provide evidence for pre-existing quality representations in base models. The minimal-data regime (160 examples) and preservation of answer quality are also positive if the comparisons to the RL baseline hold under controlled conditions.","major_comments":[{"comment":"Abstract and §3 (method overview): The claim that few-shot performance demonstrates a 'stable, transferable notion of quality latent in the base model' is load-bearing for the reframing of the problem as elicitation rather than acquisition. However, the manuscript provides no ablations on prompt phrasing, example ordering, or output format variations while holding judge scores fixed. If performance drops under modest rephrasing of the few-shot template, the result would be consistent with surface pattern matching rather than elicitation of an internal calibration.","section":"Abstract and §3"},{"comment":"§4 (experimental results): The reported gains from SEE over the RL baseline are presented without error bars, dataset splits, or details on how the 160 examples were selected or whether they overlap with the held-out evaluation sets. This makes it difficult to assess whether the 31x data reduction and improved calibration are robust or sensitive to post-hoc choices in example construction.","section":"§4"}],"minor_comments":[{"comment":"The abstract refers to 'three benchmarks' without naming them; the introduction should explicitly list the benchmarks and their characteristics to allow readers to assess domain coverage.","section":"Abstract"},{"comment":"Notation for the multi-attribute quality scores and the masking operation in the distillation phase should be defined more formally (e.g., with equations) rather than described only in prose, to improve reproducibility.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"Thank you for the constructive feedback. We address each major comment below and indicate planned revisions to improve clarity and robustness.","responses":[{"response":"We agree that the absence of explicit ablations on prompt phrasing, ordering, and output format leaves the interpretation open to the alternative of surface-level pattern matching. While the reported generalization to entirely unseen judges offers some counter-evidence, it does not fully substitute for controlled prompt variations. We will therefore add a short ablation subsection reporting performance under two rephrased templates and one altered output format, using the same fixed judge scores.","revision_made":"yes","referee_comment":"[Abstract and §3] Abstract and §3 (method overview): The claim that few-shot performance demonstrates a 'stable, transferable notion of quality latent in the base model' is load-bearing for the reframing of the problem as elicitation rather than acquisition. However, the manuscript provides no ablations on prompt phrasing, example ordering, or output format variations while holding judge scores fixed. If performance drops under modest rephrasing of the few-shot template, the result would be consistent with surface pattern matching rather than elicitation of an internal calibration."},{"response":"The concern is valid; the current presentation omits error bars and precise selection details. The 160 examples were drawn uniformly at random from the training split of each benchmark with zero overlap to the held-out evaluation sets, and the RL baseline was trained on an identically sized random subset for direct comparison. In revision we will (i) report mean and standard deviation over five random seeds, (ii) state the exact random-selection procedure and non-overlap guarantee, and (iii) include a table of the train/validation/test splits used.","revision_made":"yes","referee_comment":"[§4] §4 (experimental results): The reported gains from SEE over the RL baseline are presented without error bars, dataset splits, or details on how the 160 examples were selected or whether they overlap with the held-out evaluation sets. This makes it difficult to assess whether the 31x data reduction and improved calibration are robust or sensitive to post-hoc choices in example construction."}],"tokens_in":1446,"tokens_out":468,"duration_ms":33738,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that the authors observe base LLMs already predict an external judge's multi-attribute scores on open-ended answers above chance using prompted few-shot examples across three benchmarks, and their SEE method (calibration-coupled RL followed by masked distillation) lifts held-out calibration from just 160 examples while leaving answer quality intact.\n\nThey do a reasonable job reframing the issue as elicitation of something already present rather than new acquisition, and the claim that the elicited behavior stays stable across judges never seen in training is a useful angle if the data supports it.\n\nThe soft spot is exactly the one in the stress-test note: nothing rules out that the few-shot gains come from surface pattern matching in the prompt template or example ordering instead of a stable internal quality notion. Without ablations on prompt phrasing, example selection, or output format, and with no metrics, error bars, or dataset details supplied, it's impossible to tell how robust the result is. The abstract alone does not give enough to assess whether the 31x data reduction holds up or depends on post-hoc choices.\n\nThis is for people working on efficient LLM evaluation and alignment who want to cut down on labeled judge data. A reader focused on practical ways to reduce supervision in self-evaluation would get value if the experiments check out.\n\nI would send it to peer review because the core observation, if real, is practically relevant and the minimal-data angle matters, but the referees would need to see the full results, ablations on prompt sensitivity, and reproducibility checks before any stronger claims.","headline":"Base models already predict judge scores above chance with few-shot prompts, and SEE claims to improve it with 31x less data, but no ablations or numbers are visible to test if it's real or prompt-driven.","tokens_in":2233,"tokens_out":402,"would_cite":false,"duration_ms":29866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Base language models already predict how external judges would score their open-ended responses when given few-shot prompts.","keywords":["self-evaluation","LLM calibration","judge alignment","few-shot prompting","reinforcement learning","distillation","base models","open-ended responses"],"falsifier":"A base model given the same few-shot prompt fails to predict judge scores above chance on a fresh benchmark with different response formats, or the SEE cycle fails to raise held-out calibration while leaving answer quality unchanged.","tokens_in":2577,"feed_emoji":"🤖","tokens_out":651,"duration_ms":27771,"temperature":0.7,"pith_summary":"The paper establishes that base LLMs possess a latent ability to forecast an external judge's multi-attribute quality scores on their own outputs, performing above chance across three benchmarks with only a few-shot prompt. This capability does not require targeted training to acquire. The authors present Self-Evaluation Elicitation, a two-phase cycle that first couples reinforcement learning to improve answers while predicting judge scores, then applies masked distillation to refine the predictions alone. Using 160 examples, the method raises held-out calibration while leaving answer quality unchanged and produces predictions that remain stable across judges never encountered in training.","feed_headline":"Base LLMs already predict how judges will score their outputs","feed_subtitle":"Few-shot prompting surfaces this latent ability; a 160-example cycle sharpens calibration across benchmarks without changing answer quality.","key_machinery":"Self-Evaluation Elicitation (SEE), a short cycle consisting of a calibration-coupled reinforcement learning phase that jointly refines answers and judge-score predictions, followed by a masked distillation phase that sharpens only the predictions.","core_discovery":"A base model prompted with few-shot examples can already predict how an external judge will assign multi-attribute quality scores to its open-ended responses, performing well above chance on three benchmarks. Self-Evaluation Elicitation elicits this latent ability through a calibration-coupled reinforcement learning phase followed by masked distillation, achieving better held-out calibration from 160 examples while preserving answer quality. The elicited capability is localized in the model's token distribution and transfers to unseen judges.","pith_inferences":["Minimal-data elicitation may generalize to other latent capabilities that current alignment methods treat as skills to be taught from scratch.","If the quality notion is already inside the base model, targeted interventions could focus on localization rather than broad retraining.","The transfer across unseen judges suggests the notion may connect to general response properties rather than judge-specific preferences."],"forward_implications":["The elicited self-evaluation transfers to judges the model was never trained against.","Calibration on held-out data improves across the three benchmarks tested.","Answer quality stays the same after the elicitation cycle completes.","The ability appears localized inside the model's own token distribution."],"fun_headline_variants":["Base LLMs predict judge scores via few-shot elicitation","Latent calibration to judges emerges in base LLMs","Minimal examples surface self-evaluation in base models","Self-judge prediction present before targeted LLM training","160 examples elicit transferable judge calibration in LLMs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The observed few-shot performance reflects a stable, transferable notion of quality already present in the base model rather than an artifact of the specific prompt format or benchmark construction.","fun_headline_variants_meta":{"raw":{"variants":["Base LLMs predict judge scores via few-shot elicitation","Latent calibration to judges emerges in base LLMs","Minimal examples surface self-evaluation in base models","Self-judge prediction present before targeted LLM training","160 examples elicit transferable judge calibration in LLMs"]},"model":"grok-4.3","cost_usd":0.004808,"raw_usage":{"total_tokens":2276,"prompt_tokens":652,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":48078000,"prompt_tokens_details":{"text_tokens":652,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1553,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":652,"tokens_out":71,"duration_ms":17952,"temperature":1.0,"reasoning_tokens":1553,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T06:28:01.032265+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A base model given the same few-shot prompt fails to predict judge scores above chance on a fresh benchmark with different response formats, or the SEE cycle fails to raise held-out calibration while leaving answer quality unchanged.","supporting_citations":[],"review_version":1}