{"id":"494fd9e2-67e4-4007-a515-02302a2eac74","arxiv_id":"2501.17994","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Training a small predictor on hidden states from every layer of a frozen LLM improves multiple-choice QA accuracy over direct prompting and calibration baselines, approaching QLoRA fine-tuning on some benchmarks at lower cost.","lead":"The authors train a small neural network on the hidden states of every layer of a frozen large language model, at the last token position, to answer multiple-choice questions. On several benchmarks this improves accuracy over the model's own answers and prior calibration methods, sometimes matching parameter-efficient fine-tuning at a fraction of the compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"For CosmosQA and HellaSwag, the reported test accuracy may come from the same 20% split used for early stopping, which would inflate the headline gains; the paper must clarify and re-evaluate on an untouched test split.","rationale":"The central claim is that a small predictor on all-layer hidden states substantially improves MCQ accuracy. The paper's ablations and cost analysis are useful, and the method is plausible, but the claim depends on unbiased held-out evaluation. The reader's weakest assumption was that custom random splits may not represent official benchmarks. My read sharpens that: for CosmosQA and HellaSwag, the text implies the same 20% split serves both as the early-stopping validation set and as the evaluation set, since no test split is defined. Selecting the epoch with best validation accuracy and then reporting accuracy on that same set introduces positive bias, with no correction applied. This directly affects two of the strongest gains in Table 3 and therefore the central claim that InnerThoughts provides 'substantial gains.' The concrete test of evaluating on the official HellaSwag test after early stopping on a separate validation split would settle whether the inflation is material. The recommended verdict remains conditional: the paper should be accepted only after the evaluation protocol is clarified and the affected numbers are reproduced on a truly held-out split. No methodological fraud is implied; this is a test-design ambiguity that must be resolved before the headline results can be taken at face value.","tokens_in":17031,"tokens_out":11551,"duration_ms":119016,"concrete_test":"For HellaSwag, hold out the official 10,042-example test set and do not touch it until the end. Train the predictor on the same 80% train split as in Section 4.1, early-stop on the 20% validation split, then evaluate the selected checkpoint on the official test. If the reported 87.24 drops by more than the 0.7-point confidence interval, the early-stopping/validation overlap is material and the headline gain overstates real transfer. Repeat for CosmosQA using its official test set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing issue is the evaluation protocol for CosmosQA and HellaSwag. Section 4.1 defines only an '80%-20% train-validation split' for these two datasets, and Table 2 lists no test set for them. The same section states that predictor networks are trained for 50 epochs with early stopping using the validation set. If the Table 3 CosmosQA (80.27) and HellaSwag (87.24) numbers are computed on that 20% validation split, then the model checkpoint was selected on the same examples used to report accuracy. That is not an unbiased held-out estimate: it compares a validation-tuned predictor against Direct and calibration baselines that have no epoch selection, and it can inflate exactly the two headline gains (HellaSwag +5.04, CosmosQA +1.84) that support the central claim. This is a sharper version of the reader's custom-split concern: not only are official test sets replaced, but for two datasets the 'test' may coincide with the early-stopping set. The authors should clarify the exact evaluation split and, if this inference is correct, re-run with an untouched test split.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InnerThoughts, a supervised adapter that predicts multiple-choice answers from the hidden states of all transformer layers at the last token position of a prompt, while keeping the language model frozen. The predictor is a small MLP-Mixer-like network, so training requires only a single forward pass through the LLM. Experiments on six MCQ benchmarks with Llama3 70B compare InnerThoughts with direct prompting, calibration methods, logistic-regression and neural variants, and QLoRA fine-tuning; the authors report substantial accuracy gains (e.g., AQuA 36.47 to 47.24, HellaSwag 82.20 to 87.24) at a fraction of QLoRA's training cost, and they analyze confidence margins and per-layer contributions.","tokens_in":17312,"tokens_out":9748,"duration_ms":93902,"significance":"If the reported gains hold under a clean held-out protocol, the idea is a simple and useful addition to the MCQ toolkit: it converts a frozen LLM into a better MCQ predictor using one forward pass per training example and a small network, with low hardware requirements. The paper's controlled comparisons, bootstrapped confidence intervals, and paired significance tests are strengths, and the cost/gain comparison against QLoRA is informative. However, the evaluation protocol has an unresolved ambiguity for two of the six benchmarks, and the QLoRA baseline is quite weak; both issues need to be addressed before the central claims can be accepted.","major_comments":[{"comment":"Section 4.1 states that for CosmosQA and HellaSwag \"we use a 80%-20% train-validation split of the training set\" and does not describe a test split, yet Table 2 lists test sizes (2,985 for CosmosQA and 10,042 for HellaSwag) and Table 3 reports test accuracies for these datasets. Since Section 4.1 also says predictor networks are trained with early stopping using the validation set, if the Table 3 numbers were computed on the same 20% split used for early stopping, the reported HellaSwag improvement (+5.04) and CosmosQA improvement (+1.84) would not be unbiased held-out estimates. Please clarify the exact evaluation split, and if the reported accuracies were computed on the early-stopping validation split, re-run the evaluation on an untouched test split.","section":"Section 4.1, Table 2, Table 3"},{"comment":"The QLoRA baseline is trained with adapter rank r=1 for three epochs and is described as an upper bound, but Table 4 uses its accuracy to compute gain ratios and the claim that InnerThoughts obtains \"roughly 35% of the gains of QLoRA at 16% of the computational costs.\" A rank-1, three-epoch QLoRA is unlikely to represent the full fine-tuning upper bound, so the efficiency comparison and the statement that InnerThoughts is \"approaching parameter-efficient fine-tuning\" need either a stronger baseline or explicit caveats that the comparison is against a deliberately weak fine-tuning configuration.","section":"Section 4.1, Table 4"},{"comment":"The paper replaces official test sets with custom random splits for AQuA, MedMCQA, CosmosQA, and HellaSwag, and it does not report accuracy on the official test sets. Because the abstract and introduction claim gains on \"hard benchmarks\" without qualification, and because random splits can differ in difficulty from the public test sets, the authors should either use the official test sets or provide evidence (for example, by comparing the Direct baseline on both the custom split and the official set) that the custom splits are representative of the intended benchmarks.","section":"Section 4.1, Table 2"}],"minor_comments":[{"comment":"The legend labels such as \"AQuA (Δ=+0.11)\" are described as gains in accuracy, but they do not match the accuracy gains in Table 4 (+10.75 for AQuA, +5.04 for HellaSwag, etc.); please correct the labels or clarify that they are logit-margin means.","section":"Figure 3"},{"comment":"The claim that the proposed architecture is strictly more expressive than the original LLM predictor assumes that the normalization and activation blocks can be bypassed or set to identity, but the architecture as drawn includes them as fixed components; please clarify whether these components can be removed or configured as identity mappings.","section":"Section 3"},{"comment":"The phrase \"neural network on the last 8192 hidden state\" should be \"neural network on the last hidden state.\"","section":"Section 4.2"},{"comment":"The description of the significance test is incomplete: please state explicitly that the one-sided Wilcoxon signed-rank test is applied to paired per-example correctness indicators, and indicate whether any multiple-comparison correction was applied across methods and datasets.","section":"Section 4.3"},{"comment":"No code or data-processing scripts are provided; releasing the feature-extraction and split-generation code would substantially improve reproducibility, especially because the evaluation uses custom splits.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The split ambiguity is the main correctness risk in this manuscript. It is fixable with a clarification or a re-run on an untouched test split, so I am not recommending rejection. If the authors confirm that the CosmosQA and HellaSwag results were computed on the official validation sets rather than the early-stopping split, and update the text accordingly, I would be willing to accept a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid, useful empirical paper. Training a small MLPMixer-style head on the hidden states of all layers at the last token position improves multiple-choice QA accuracy over direct prompting and logit-calibration baselines on several benchmarks, with costs a fraction of QLoRA. The headline results are probably real, and the paper includes the right kind of controlled evidence: ablations over inputs (logits, last state, last 10 states), bootstrap confidence intervals, and Wilcoxon tests.\n\nThe new bit is the specific combination—all layers, last token, compact trained head—which I don't think appears in the cited calibration/probing literature. The closest are Linear Probe Calibration on logits and SAPLMA on a single inner layer. The layer-influence analysis (Figure 5, and the gradient-based influence in Appendix D) is informative and supports the design choice.\n\nNow the soft spots, in rough order of seriousness.\n\nFirst, the evaluation protocol for AQuA and MedMCQA is non-standard: they split the training set 70-15-15 and report on the random test portion, not the official test set. That's a real weakness—random splits of a training set can be easier or distributionally different—and it should be fixed in revision (or at least justified with a comparison on the official test sets). For CosmosQA and HellaSwag, the stress-test worry that the test numbers might come from the early-stopping validation split does not hold. Table 2 lists the official test sets for those two, and the '80-20 train-validation split' in Section 4.1 refers to splitting the training set; the reported accuracy is on the official test sets. So that particular concern is unfounded.\n\nSecond, the QLoRA baseline is deliberately light: rank 1, 3 epochs, on a 70B model. The paper calls it an upper bound, which is optimistic. The cost-efficiency comparison is against this weak baseline, so the claim 'approaches fine-tuning' is fragile. A stronger fine-tuning baseline would change the framing.\n\nThird, no code or custom splits are provided. That limits reproducibility, though the method is simple enough to reimplement.\n\nThe 'disentangling representations and predictions' framing is more conceptual than literal—they are just training a different head—but that's a marketing overclaim, not a flaw in the experiments.\n\nWho is this for? Anyone working on efficient adaptation of frozen LLMs, calibration, or probing. It is a practical result with a clean method and careful statistics. It deserves a serious referee; the issues are fixable with an honest revision and a bit more evaluation. I'd accept it for review.","headline":"A cheap and well-ablated supervised head over all-layer hidden states buys real multiple-choice QA gains; the evaluation split for two of six datasets is non-standard, but the flagged validation-leak worry does not survive a careful read.","tokens_in":17823,"tokens_out":5528,"would_cite":true,"duration_ms":46493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small predictor network over all hidden layers improves frozen-LLM multiple-choice accuracy, approaching fine-tuning results at far lower cost.","keywords":["InnerThoughts","hidden states","multiple-choice question answering","LLM calibration","frozen language models","predictor network","parameter-efficient adaptation","Llama 3 70B"],"falsifier":"Run the trained InnerThoughts predictor on the official held-out test partitions of AQuA, MedMCQA, CosmosQA, and HellaSwag without retraining on those splits; if the accuracy advantage over direct prompting shrinks to near zero there, the central claim would fail in the conditions the benchmarks are meant to measure.","tokens_in":16871,"feed_emoji":"🧠","tokens_out":7425,"duration_ms":70056,"temperature":0.7,"pith_summary":"This paper tries to establish that a frozen large language model leaves useful answer information in the hidden states of every layer, and that a small trained network can extract it. The proposed method, InnerThoughts, feeds all layers' hidden states at the final token position into a lightweight predictor, replaces the model's own softmax readout, and improves multiple-choice accuracy on several hard benchmarks. The authors report gains such as 47.24% versus 36.47% on AQuA and 87.24% versus 82.20% on HellaSwag, while requiring only one forward pass through the LLM during training. If the claim holds, per-task adaptation of large frozen models becomes much cheaper than parameter-efficient fine-tuning.","feed_headline":"InnerThoughts lifts AQuA accuracy from 36% to 47% on a frozen LLM","feed_subtitle":"A small predictor trained on all layers' last-token states rivals fine-tuning at roughly one-sixth the compute.","key_machinery":"The central object is the predictor module, an MLPMixer-style network that takes an $(L,d)$ tensor of hidden states from the last token position, applies normalization, linear, and activation blocks to reduce both dimensions, flattens the result, and ends in a softmax over answer choices. Because it sits after the transformer blocks, it can be trained with the LLM frozen and with no backward passes through the LLM. The module is strictly more expressive than the original readout, since specific parameter choices recover the original normalization and embedding projection.","core_discovery":"The central claim is that a small predictor network trained on all layers' last-token hidden states can outperform both direct prompting and existing logit-calibration methods on multiple-choice QA, and in some cases approach supervised fine-tuning. The paper reports statistically significant gains over direct prompting on AQuA, CosmosQA, MedMCQA, LogiQA, and HellaSwag, with the largest improvements on datasets where the base LLM's confidence margins are smallest. Training the predictor requires no backward pass through the frozen LLM, so the dominant cost is a single forward pass to extract hidden states. The authors interpret this as disentangling the LLM's representational abilities from its predictive abilities.","pith_inferences":["The headline numbers come from custom random splits of training sets, not the official test sets; evaluating on official held-out splits would show how much of the gain transfers to real benchmark conditions.","InnerThoughts may be learning dataset-specific answer or option-order biases rather than purely better reasoning; shuffling answer order at test time would separate bias correction from genuine task improvement.","The method could plausibly stack with fine-tuning: training the same predictor on a QLoRA-adapted model, or jointly, may produce gains beyond either alone, though the paper only raises this as a possibility."],"forward_implications":["On AQuA, InnerThoughts reaches 47.24% accuracy versus 36.47% for direct prompting, nearly matching QLoRA's 48.50% at about one-fifth of the training cost.","Across the six benchmarks it captures, on average, roughly 35% of QLoRA's accuracy gain at about 16% of the compute cost.","The largest improvements occur on benchmarks where the base LLM's confidence margin between top and second answers is smallest, and applying InnerThoughts also increases those margins.","Because the predictor input is the same set of last-token hidden states, one extraction pass can support multiple predictor heads or downstream tasks without re-running the LLM."],"supporting_citations":[{"why":"Defines linear probe calibration on final logits, the main calibration baseline that InnerThoughts outperforms.","marker":"Abbas et al., 2024"},{"why":"Provides QLoRA parameter-efficient fine-tuning, the accuracy and cost upper bound compared in Tables 3 and 4.","marker":"Dettmers et al., 2023"},{"why":"Supplies the frozen Llama3 70B model whose per-layer hidden states are the method's input.","marker":"Dubey et al., 2024"},{"why":"Supplies the prompt construction guidelines used for every benchmark's question-answer format.","marker":"Robinson and Wingate, 2023"},{"why":"Defines Calibrate Before Use, an unsupervised calibration baseline compared in the results.","marker":"Zhao et al., 2021"},{"why":"Introduces the MLP-Mixer architecture that the predictor module is based on.","marker":"Tolstikhin et al., 2021"},{"why":"Provides the AQuA arithmetic word-problem dataset where the largest reported gain occurs.","marker":"Ling et al., 2017"},{"why":"Provides the HellaSwag commonsense completion dataset where the second-largest reported gain occurs.","marker":"Zellers et al., 2019"}],"fun_headline_variants":["InnerThoughts: frozen LLM + small head rivals fine-tuning","InnerThoughts: all layers' hidden states beat final-only predictions","InnerThoughts: tiny head on all layers, one-sixth the compute","InnerThoughts: 36% to 47% on AQuA with a frozen LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that random 70/15/15 and 80/20 splits of each training set are representative of the official benchmark test sets, so that the measured improvements would also appear in a standard public evaluation.","fun_headline_variants_meta":{"raw":{"variants":["InnerThoughts: frozen LLM + small head rivals fine-tuning","InnerThoughts: all layers' hidden states beat final-only predictions","InnerThoughts: tiny head on all layers, one-sixth the compute","InnerThoughts: 36% to 47% on AQuA with a frozen LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001911,"raw_usage":{"total_tokens":7427,"prompt_tokens":830,"completion_tokens":6597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":6513}},"tokens_in":446,"tokens_out":6597,"duration_ms":45990,"temperature":1.0,"reasoning_tokens":6513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:26:30.113666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained InnerThoughts predictor on the official held-out test partitions of AQuA, MedMCQA, CosmosQA, and HellaSwag without retraining on those splits; if the accuracy advantage over direct prompting shrinks to near zero there, the central claim would fail in the conditions the benchmarks are meant to measure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines linear probe calibration on final logits, the main calibration baseline that InnerThoughts outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides QLoRA parameter-efficient fine-tuning, the accuracy and cost upper bound compared in Tables 3 and 4."},{"cited_title":"and Wingate, D","cited_arxiv_id":null,"evidence_quote":"Supplies the prompt construction guidelines used for every benchmark's question-answer format."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Calibrate Before Use, an unsupervised calibration baseline compared in the results."},{"cited_title":"O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A","cited_arxiv_id":null,"evidence_quote":"Introduces the MLP-Mixer architecture that the predictor module is based on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AQuA arithmetic word-problem dataset where the largest reported gain occurs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HellaSwag commonsense completion dataset where the second-largest reported gain occurs."}],"review_version":1}