{"id":"c9367cd0-e58d-40d2-b195-34568ee9ca80","arxiv_id":"2504.16760","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LiLaVe, an XGBoost verifier trained on hidden states of a base LLM, predicts answer correctness with AUC comparable to large LLM-based verifiers, and enables conditional majority voting and self-correction that improve accuracy under fixed generation budgets.","lead":"This paper introduces LiLaVe, a small classifier trained on the hidden states of a language model to judge whether a generated math answer is correct. It reports that LiLaVe matches much larger verifier models while using a fraction of the compute, and uses it to make voting and self-correction strategies cheaper and more accurate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figures 4 and 5 report conditional self-correction and conditional majority voting at thresholds s selected on test labels; the paper acknowledges a validation-set tune is needed but never runs one, so the claimed accuracy/efficiency gains may not be achievable in practice.","rationale":"The paper's core observation—that XGBoost on hidden states yields a usable correctness score—is supported by the AUC numbers and by auxiliary evidence: cross-dataset transfer (Table 4), multiple base models (Table 2), and an explicit oracle gap (Figure 10). The most vulnerable part is the translation of AUC into the two new meta-generation strategies. Both strategies depend on a scalar threshold s, and the reported gains are obtained at thresholds chosen on the test partition. The paper explicitly acknowledges that s must be tuned on a validation set in practice but provides no such experiment, so the reported accuracy/efficiency trade-offs are upper envelopes rather than achievable procedures. This is exactly the reader's weakest assumption, and a validation-split rerun would settle it. If the validation-tuned results reproduce the figures, the conditional strategies stand; if not, the contribution reduces to a standard verifier with standard best-of-n and weighted voting. The ORM-baseline mismatch is a secondary concern for the 'comparably good' claim, but it is less central to the paper's novel strategies and could be addressed by training a Llama-3.1-8B ORM. The XGBoost hyperparameter inconsistency (max depth 5 in the main text versus 10 in Appendix A.2.1) is a reproducibility blemish, not a validity threat. The verdict remains CONDITIONAL pending the validation experiment.","tokens_in":22832,"tokens_out":5016,"duration_ms":49172,"concrete_test":"Split each benchmark's test partition into validation and held-out test halves (or use k-fold cross-validation). Tune s for conditional self-correction and both n and s for conditional majority voting on the validation half only; then evaluate on the held-out test half and plot accuracy against total generated samples. Compare with the test-optimal curves in Figures 4 and 5, and also re-run Table 1 after choosing L,T on a validation split. If validation-tuned conditional strategies no longer beat standard majority voting at matched budgets, the meta-generation claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim that LiLaVe powers conditional self-correction and conditional majority voting rests on the availability of a threshold s (and, secondarily, hidden-state locations L,T) that can be fixed before seeing test labels. Section 4.2 selects L,T by evaluating AUC on the testing partition (Figure 1), while Section 4.4 reports conditional-strategy curves only for thresholds that maximize test accuracy: Figure 4 marks 'optimal thresholds' in orange, and Figure 5 plots an envelope over s. The paper itself notes in Section 4.4 that 'this hyperparameter must be tuned depending on the data' and that 'one would tune the n and s parameters on a validation set', but no validation-set experiment appears anywhere. Because the improvements over standard majority voting in Figures 4 and 5 are the main practical payoff of the method, an oracle-tuned threshold could turn a real advantage into a null result. The AUC comparisons in Table 1 are less affected, but the same test-set selection of L,T could modestly inflate them as well.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LiLaVe, a lightweight verifier trained with XGBoost on hidden states extracted from a base LLM at selected layers and tokens, with the aim of predicting whether a chain-of-thought solution is correct. The verifier is evaluated by AUC on four math benchmarks (GSM8K, GSM-Symbolic, GSM-Symbolic-p2, algebra linear 1d, MATH500) and compared against self-reflection prompting, a log-probability baseline, and two large LLM-based outcome reward models. The authors then use LiLaVe scores in best-of-n, weighted majority voting, conditional self-correction, and conditional majority voting, reporting accuracy gains and compute savings for smaller LLMs. The central claims are that LiLaVe extracts correctness signal comparable to much larger verifiers, and that the proposed conditional meta-generation strategies improve accuracy and efficiency under fixed generation budgets.","tokens_in":23028,"tokens_out":6087,"duration_ms":56114,"significance":"If the claims hold under a properly held-out evaluation protocol, this is a practically valuable result: a CPU-only, small-data classifier on hidden states could replace expensive LLM-based verifiers for math reasoning, and the conditional strategies offer a concrete accuracy-efficiency trade-off. The paper has genuine strengths: the layer/token location study is informative, the temperature analysis is systematic, the evaluation spans multiple benchmarks and base models, and the efficiency measurements give concrete evidence of low overhead. The central weakness is that the main practical claims rely on choices made on the test labels, and the manuscript contains an internal hyperparameter inconsistency. With a validation-based selection protocol, the core idea is likely sound, but the reported gains need to be re-established without test-set peeking.","major_comments":[{"comment":"The hidden-state locations L_LiLaVe and T_LiLaVe are selected after evaluating AUC on the testing partition (Section 4.2, Figure 1). The same test partitions are then used for the AUC comparisons in Table 1 and for the meta-generation experiments in Figures 3–5. This means the reported performance includes a model-selection step performed on the test labels. The final choice of a broad suffix of tokens and five layers may be stable, but the manuscript currently provides no evidence for that stability. Please fix a selection protocol using a validation split (or a priori location sets) and report the corresponding test results.","section":"4.2, Figure 1"},{"comment":"The accuracy and efficiency gains of conditional self-correction and conditional majority voting are reported at thresholds s that maximize test accuracy: Figure 4 marks the 'optimal thresholds' in orange, and Figure 5 plots an envelope over thresholds. The paper itself notes that 'this hyperparameter must be tuned depending on the data' and that 'one would tune the n and s parameters on a validation set,' but no validation-set experiment is run. Since these two strategies are the main practical payoff of the method, the claims are not established without a held-out threshold selection. Please run a validation split, select s (and n) on it, and report test-set accuracy/efficiency; also report the sensitivity of the gains to the chosen threshold.","section":"4.4, Figures 4 and 5"},{"comment":"There is an internal inconsistency in the XGBoost hyperparameters: Section A.2 states 'We set the maximum tree depth to 5,' while Section A.2.1 lists 'max depth=10' among the XGBoost hyperparameters. Since XGBoost is the core verifier model, this inconsistency must be resolved and the exact configuration used for all reported experiments must be stated.","section":"A.2 and A.2.1"},{"comment":"The comparison with ORM baselines may disadvantage them in a way that affects the headline claim of being 'comparably good' as LLM-based verifiers trained on much larger data. ORM-Mistral and ORM-Deepseek are based on Llama 3.1 8B but trained on data generated by Mistral-7B and DeepSeekMath-Instruct 7B, while they are evaluated as verifiers of Llama 3.1 8B outputs. The paper itself cites evidence that verifier performance may not transfer across base LLMs. A domain-matched ORM baseline (e.g., fine-tuned on Llama 3.1 8B generations), or at least a discussion of the expected transfer effect, would make the comparison more convincing.","section":"4.3, Table 1"}],"minor_comments":[{"comment":"Several reported differences are small (e.g., weighted versus standard majority voting on GSM-Symbolic), but no confidence intervals or significance tests are provided. Reporting standard errors or bootstrap intervals would help assess whether the differences are meaningful.","section":"Figures 3–5 and Table 1"},{"comment":"The logprob baseline's suffix length k is chosen on the test set; the paper calls this an idealized upper bound, but it would be clearer to also report results for a fixed k or for k selected on a validation set.","section":"4.3"},{"comment":"There is a typo in Section 4.1: 'obatained' should be 'obtained'; Appendix A.1 has 'comparizon' instead of 'comparison.'","section":"4.1"},{"comment":"The sentence 'For each line and plot The solid lines are medians...' is ungrammatical and should be reworded.","section":"A.2"},{"comment":"The cross-model verification experiment is interesting but is reported on a single base-model pair; stating whether the same trend holds for Llama-generated responses scored by a Phi-based LiLaVe would make the transfer claim more complete.","section":"Appendix A.7"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection issue is fixable and the core idea seems promising; the authors should be able to address it with a validation split and by clarifying the XGBoost configuration. I would not reject the paper, but the practical claims should be reported under a protocol that does not tune on the test labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you work on inference-time verification for small models, read the AUC part of this paper now. The authors train an XGBoost classifier on hidden states from Llama 3.1 8B and get AUC of 0.86–0.93 across GSM8K, GSM-Symbolic, algebra linear 1d, and MATH500, beating self-reflection and logprob confidence and roughly matching two ORMs trained on 250k examples. That is a real result. The probing technique itself is not new—Azaria & Mitchell, Chen et al., and OPENIA are cited—but the application to math outcome verification with a cheap classifier and 5k training samples is probably new enough, and the conditional self-correction and conditional majority voting strategies are a genuine addition.\n\nWhat the paper does well: the temperature transfer experiments, the layer/token location maps, the cross-dataset transfer table, and the efficiency numbers are all useful. The results for Gemma 2 2B and Phi-3.5-mini, with one honest exception, show the idea is not tied to one model. The limitations section is candid about the verifier-oracle gap and about the need to tune thresholds.\n\nSoft spots, in order of importance:\n\n1. The meta-generation gains in Figures 4 and 5 are reported at thresholds selected on the test set. The paper says in Section 4.4 that one would tune n and s on a validation set, but no validation experiment appears. This is the load-bearing weakness. The AUC claims are much less affected; the strategy gains could shrink substantially. The stress-test note is right.\n\n2. The ORM baselines from Xiong et al. are trained on Mistral-7B and DeepSeekMath-Instruct data, not on Llama 3.1 8B outputs. The comparison is informative, but it is not strictly apples-to-apples. Training an ORM on the same base model's outputs would tighten the claim.\n\n3. Minor but real: the main text says XGBoost max depth 5; Appendix A.2.1 says max depth 10. That inconsistency needs fixing. Also, the logprob baseline's suffix length k is chosen on the test set, which the authors admit is an idealized upper bound—fine, but it is the same kind of oracle selection they are criticized for elsewhere.\n\nMy take: the central empirical claim—that a cheap probe on hidden states carries a usable correctness signal—holds up. The strategy-level claims are conditional on a threshold-selection procedure that has not been demonstrated. That is a standard “revise with held-out validation and a same-model ORM baseline” situation, not a desk-reject. I would send it to review, ask for the validation experiment and code release, and see the next version before believing the efficiency-accuracy trade-off curves.","headline":"A lightweight XGBoost probe on a base LLM's hidden states does surprisingly well as a math verifier, but the headline strategy gains lean on thresholds picked on test labels; worth refereeing with a required validation-set experiment.","tokens_in":23598,"tokens_out":2147,"would_cite":true,"duration_ms":18776,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LiLaVe, a lightweight verifier, predicts answer correctness from the base LLM's hidden states and matches much larger verifiers at a fraction of the cost.","keywords":["lightweight latent verifier","hidden states","XGBoost","meta-generation","conditional self-correction","conditional majority voting","mathematical reasoning","LLM verification"],"falsifier":"Use a validation split (no test labels) to pick the threshold $s$ and the token/layer locations $L,T$, then measure test accuracy on the same benchmarks; if conditional self-correction and conditional majority voting no longer beat standard majority voting at equal generation budgets, the central efficiency claim is falsified. A direct check is to compare validation-selected thresholds with the test-optimal thresholds shown in the figures: if the validation choices fall outside the accuracy-improving range, the reported gains will not survive deployment.","tokens_in":22616,"feed_emoji":"🧮","tokens_out":10001,"duration_ms":84827,"temperature":0.7,"pith_summary":"The paper sets out to show that a full-size verifier model is unnecessary: the correctness of a generated math answer can be read directly from the activations of the model that generated it. Its method, LiLaVe, trains a small gradient-boosted classifier on hidden states extracted during chain-of-thought decoding, labelled only by whether the final answer is right, and averages the classifier's per-token scores into a correctness estimate. On GSM8K, GSM-Symbolic, algebra linear 1d, and MATH500, this estimate outranks self-reported confidence and token-log-probability heuristics and matches LLM-based verifiers trained on about fifty times more data. The paper then uses the score as a gate: self-correction and majority voting are invoked only for low-scoring answers, improving both accuracy and sample efficiency for smaller models. If the claims hold, the expensive verifier component of many reasoning pipelines can be replaced by a CPU-only classifier.","feed_headline":"Cheap verifier reads correctness from hidden states","feed_subtitle":"LiLaVe matches large LLM verifiers on math benchmarks using 5k examples and a CPU-only scorer.","key_machinery":"The machinery is LiLaVe's score itself: hidden states from each location in $L \\times T$ are fed, together with the layer and token indices, into gradient-boosted decision trees (the XGBoost implementation), and the per-location predictions are averaged into the final score. Averaging over the last 16 tokens and layers $\\{-1,-2,-4,-8,-16\\}$ is the configuration selected by the location study. This object carries the argument because it turns raw activations into a single correctness probability without running a second LLM; the rest of the paper consists in showing that this probability ranks answers well and can trigger compute-saving meta-generation strategies.","core_discovery":"On the paper's own terms, the central discovery is that hidden states of the base LLM carry a reliable, usable correctness signal. Per-token XGBoost classifiers trained on layers $\\{-1,-2,-4,-8,-16\\}$ and the final 16 token positions give answer-level AUCs of 0.78–0.93 across four math datasets, beating self-reflection and logprob baselines and matching Llama-3.1-8B-based verifiers trained on 250k+ examples while using only 5k scored samples. The paper also claims that this signal generalizes beyond ranking: conditional self-correction and conditional majority voting, which apply extra computation only when the score falls below a threshold, yield accuracy gains at equal or lower generation budgets, and weighted majority voting with LiLaVe scores is consistently the strongest scoring-based strategy among those compared.","pith_inferences":["If the threshold and hidden-state locations are tuned on a held-out validation set rather than the test partition, the practical gains of conditional self-correction and conditional majority voting may be smaller than the figures suggest; the paper does not report this validation-tuned protocol.","The early-token signal suggests an online decoding controller that stops or redirects low-scoring trajectories before full generation; the paper names verifier-conditioned decoding only as future work, so this application is an extension.","The same recipe could be applied to non-mathematical reasoning, such as code correctness or factuality, by training on hidden states with corresponding binary labels; this is a testable extension rather than a claim in the paper.","The asymmetric cross-dataset transfer reported in the appendix implies that training-data coverage drives transfer more than classifier choice does; the paper does not draw that conclusion explicitly."],"forward_implications":["Scoring already-generated responses with LiLaVe runs on CPU and takes about 3.4 seconds for the GSM8K test set, versus about 20 minutes on a GPU for a benchmarked LLM-based verifier, a roughly 350x speedup in verification-side cost.","Gating self-correction on a low LiLaVe score avoids the usual accuracy collapse: unconditional self-correction drops accuracy by 15–30 points, while selective correction improves accuracy on algebra linear 1d and GSM-Symbolic-p2.","Gating majority voting on a low LiLaVe score (conditional majority voting) yields better accuracy than standard majority voting across a wide range of generation budgets, measured as total generated samples.","Weighted majority voting with LiLaVe scores beats both best-of-n and unweighted majority voting on all four benchmarks at every vote count tested.","A verifier trained on 5k scored samples per benchmark reaches AUC comparable to Llama-3.1-8B-based outcome reward models trained on more than 250k samples, and the trained verifier transfers across datasets and to other base models."],"supporting_citations":[{"why":"Introduces GSM8K and the original verifier-as-ranker setup; supplies the benchmark and the baseline that LiLaVe extends.","marker":"Cobbe et al., 2021"},{"why":"Provides chain-of-thought prompting, the generation protocol whose hidden states LiLaVe reads.","marker":"Wei et al., 2022"},{"why":"Defines self-consistency (majority voting), the standard meta-generation strategy that LiLaVe weights or gates.","marker":"Wang et al., 2023"},{"why":"Supplies the XGBoost implementation used as the latent classifier in all LiLaVe experiments.","marker":"Chen & Guestrin, 2016"},{"why":"Provides the two Llama-3.1-8B outcome-reward-model baselines trained on 250k+ examples that LiLaVe is compared against.","marker":"Xiong et al., 2024"},{"why":"Argues that latent structures in LLM reasoning can enable simple verifiers, the premise LiLaVe operationalizes.","marker":"Ye et al., 2024"},{"why":"Supplies the MATH500 evaluation subset and the step-by-step verification framing that motivates verifier research.","marker":"Lightman et al., 2023"},{"why":"Documents the failure of unconditional self-correction, the reason conditional self-correction is needed.","marker":"Huang et al., 2024"},{"why":"Supplies the MATH final-answer extraction code used to produce LiLaVe's correctness labels.","marker":"Gao et al., 2024"},{"why":"Shows expensive LLM verifiers in verifier-guided search, motivating the cost comparison LiLaVe improves on.","marker":"Wu et al., 2025"}],"fun_headline_variants":["Hidden states spill correctness for pennies","Cheap scorer reads LLM's hidden truth","Tiny verifier matches big ones via hidden states","Latent verifier cuts cost, keeps accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the same threshold, and the same hidden-state locations, can be chosen without seeing test answers; in the paper's main plots the thresholds and locations are selected after evaluating on the test partition, so the practical gains depend on those choices transferring to a held-out validation set.","fun_headline_variants_meta":{"raw":{"variants":["Hidden states spill correctness for pennies","Cheap scorer reads LLM's hidden truth","Tiny verifier matches big ones via hidden states","Latent verifier cuts cost, keeps accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000363,"raw_usage":{"total_tokens":1940,"prompt_tokens":909,"completion_tokens":1031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":974}},"tokens_in":525,"tokens_out":1031,"duration_ms":10272,"temperature":1.0,"reasoning_tokens":974,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:56:02.226410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a validation split (no test labels) to pick the threshold $s$ and the token/layer locations $L,T$, then measure test accuracy on the same benchmarks; if conditional self-correction and conditional majority voting no longer beat standard majority voting at equal generation budgets, the central efficiency claim is falsified. A direct check is to compare validation-selected thresholds with the test-optimal thresholds shown in the figures: if the validation choices fall outside the accuracy-improving range, the reported gains will not survive deployment.","supporting_citations":[{"cited_title":"Le, Ed H","cited_arxiv_id":null,"evidence_quote":"Defines self-consistency (majority voting), the standard meta-generation strategy that LiLaVe weights or gates."}],"review_version":1}