{"id":"8667d061-3eeb-48a8-bbcd-5248f4e93f7f","arxiv_id":"2506.00483","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A classifier that decides which hidden states to patch during inference improves 2-hop question answering from 18.45% to 23.63% on MuSiQue, but the evaluation uses the same prompts that trained the classifier.","lead":"A team from Tel Aviv University reports a method that trains a simple classifier to decide which hidden states to copy between layers during inference, lifting LLaMA-2's score on 2-hop questions from 18.45% to 23.63%, closer to Chain-of-Thought prompting's 27.44%. The gain is measured on the same questions used to build the classifier's training labels, so the headline number is in-sample.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample evaluation: the 5.18-point solve-rate gain is measured on the same 1,024 prompts that generated the classifier's labels, and the deployed policy is nearly trivial; no held-out split, no significance test, and no shown random-policy baseline.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the same 1,024 prompts generate both the classifier's training labels and the final solve-rate measurement, with no train/test split. I agree. The paper's own §4.4 shows the deployed policy is nearly constant, and §4.6 references a random-classification baseline that is never shown. These facts together mean the headline number cannot be interpreted as evidence for the method. A held-out split plus trivial-policy and random-policy controls would settle whether the gain is real. The reader's REJECT verdict therefore stands as is; no verdict adjustment is needed.","tokens_in":8058,"tokens_out":7105,"duration_ms":65765,"concrete_test":"Split the 1,024 MuSiQue prompts: use 512 to generate per-position labels and train the SVM, then measure solve rate on the held-out 512 (or a fresh MuSiQue sample). Also run (a) a trivial policy that patches every position except <s>, <unk>, and '.', and (b) a random-patch policy with the same patch rate. If the held-out Auto-Patch gain over baseline is not significantly larger than both controls, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Table 1, §4.4) is that Auto-Patch raises solve rate from 18.45% to 23.63% on MuSiQue 2-hop questions. This number is not a valid estimate of method performance because the evaluation set is the same 1,024 prompts used to create the classifier's training labels. In §3.2, each position is labeled True if patching 'resulted in a higher probability of the correct answer being generated'; §4.3 then measures the final solve rate on those same 1,024 questions after applying the classifier's decisions. No train/test split is described anywhere. The classifier has therefore seen the hidden states of the exact prompts on which solve rate is measured, so the 5.18-point gain may reflect in-sample fitting rather than a generalizable intervention. This concern is amplified by the paper's own deployment analysis in §4.4: the learned policy is nearly constant (patch every position except <s>, <unk>, and '.'), so the contribution of the learned component is indistinguishable from a trivial fixed policy. Section 4.6 claims a random-classification baseline demonstrates the benefit of learned selection, but that comparison is never reported. The headline result therefore rests on an uncontrolled, in-sample comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Auto-Patch, a method that uses an SVM classifier to decide which hidden states of LLaMA-2 should be patched (from layer 15 to layer 8) during inference, with the goal of improving multi-hop question answering. The authors report that on MuSiQue 2-hop questions, Auto-Patch improves the solve rate from 18.45% (baseline) to 23.63 ± 0.7%, narrowing the gap to Chain-of-Thought prompting at 27.44%. The method is positioned as an automation of the PatchScopes framework that removes the need for manual prompt separation. The paper also reports layer-source and layer-distance sweeps on smaller subsets and discusses unsuccessful experiments.","tokens_in":8260,"tokens_out":5166,"duration_ms":50182,"significance":"If the central result were valid, Auto-Patch would be a noteworthy demonstration that a lightweight, classifier-driven hidden-state intervention can improve multi-hop reasoning without prompt engineering or manual decomposition. The paper is clearly written, provides a code link, includes additional layer ablations, and candidly discusses failed experiments and limitations. However, the current evidence does not support the headline claim: the classifier is trained and evaluated on the same 1,024 prompts, the learned policy is nearly constant at inference time, and the claimed random-classification control is never reported. The contribution as stated—learned selective patching—is therefore not established by the experiments.","major_comments":[{"comment":"The central empirical claim is invalidated by in-sample evaluation. In §3.2, each per-position training label is created by checking whether patching increases the probability of the correct answer on the evaluation prompts themselves. In §4.3, the final solve rate is computed on those same 1,024 prompts after applying the classifier's decisions. No train/test split is described anywhere. Consequently, the 23.63% solve rate reported in Table 1 is a measure of fit to the training/evaluation prompts, not an estimate of performance on unseen questions. This is the load-bearing comparison of the paper, and it cannot support the claimed improvement over the 18.45% baseline.","section":"§3.2 and §4.3"},{"comment":"The deployed classifier is nearly constant, so the reported gain cannot be attributed to learned selective patching. Section 4.4 states that \"the model predicts True for almost all positions\" and that only positions corresponding to <s>, <unk>, and \".\" are not patched. This means the learned component is almost equivalent to a fixed patch-everything policy. Section 4.6 claims that a random-classification baseline demonstrates the benefit of the learned classifier, but that comparison is never reported. Without results for a random or always-patch baseline, the claim that the classifier contributes anything beyond a trivial policy is unsupported.","section":"§4.4 and §4.6"},{"comment":"The reported classifier accuracy is not reconciled with the deployment behavior. Table 2 reports accuracy of 0.81 with support 4,983, but the dataset is described as containing 24,912 samples with 23% true labels, and the deployed policy is said to predict True for almost all positions. A classifier that predicts True almost everywhere would not have 0.81 accuracy on a dataset that is 77% false unless the evaluation set is materially different from the deployment distribution. The paper does not explain what data Table 2 is computed on, how SMOTETomek changed the class balance, or why the classifier's decision boundary at deployment differs so sharply from its training behavior. This inconsistency further undermines the claim that the SVM is an accurate selector.","section":"§4.2 and Table 2"},{"comment":"The additional layer-source and layer-distance experiments are run on only 128 prompts and are reported without error bars, confidence intervals, or any indication of a held-out split. In Figures 3 and 4, differences of a few percentage points among conditions are within the sampling error expected for 128 prompts. These experiments therefore do not substantiate the claim that layer 15→8 is an optimal or principled choice, and they inherit the same in-sample labeling issue as the main experiment.","section":"§4.5"}],"minor_comments":[{"comment":"The MuSiQue dataset is cited as Deldjoo et al. [2021], but that reference is a music recommendation paper; the correct citation is Trivedi et al., \"MuSiQue: Multihop Questions via Single-hop Question Composition\" (TACL 2022).","section":"References"},{"comment":"The name of the first author of PatchScopes is misspelled as \"Gandeharioun\" in the Related Work section; the correct spelling is \"Ghandeharioun.\"","section":"Related Work"},{"comment":"The label definition is inconsistent. Section 3.2 says labels are based on whether patching increases the probability of the correct answer, while §4.2 describes the label as 'is_correct_patched', a boolean indicating whether the model generated the correct answer. These are different criteria and the discrepancy should be clarified.","section":"§3.2 and §4.2"},{"comment":"The description of source and target layers is confusing: it says \"For each position i in the 'target layer', we patched the hidden state to the corresponding position i in the 'source layer'\", which appears to reverse the direction stated in §4.2 (from layer 15 to layer 8). Please use consistent terminology throughout.","section":"§3.2"},{"comment":"The y-axes in Figures 3 and 4 are labeled \"Accuracy\" while the main metric elsewhere is \"solve rate\". Please use consistent terminology and define the metric used in the ablation figures.","section":"§4.5"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early project report. The methodological problems are not local: the headline result is an in-sample number, the deployed policy is nearly trivial, and the key control comparison is missing. A revision would require a new evaluation campaign on held-out prompts, a proper random and always-patch baseline, and likely a different conclusion about the role of the classifier. In addition, the incorrect MuSiQue citation suggests the related-work and dataset verification were not carefully checked. I do not see a path to acceptance without fundamentally redoing the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2506.00483. The idea is genuinely new relative to PatchScopes: instead of manually splitting prompts into hops, they train an SVM to decide which hidden states to patch. That framing is the paper's real contribution, and they back it with code and a small set of ablation experiments on layer choice. The limitations section is candid about the classifier's narrow view.\n\nThe problem is that the central number—23.63% versus 18.45% baseline—is not an unbiased estimate. The classifier's training labels come from patching each position and checking whether the gold answer probability goes up, on the same 1,024 MuSiQue prompts that later produce the solve rate table. No train/test split is described. So the reported gain is partly a measure of fit to those exact prompts. That is load-bearing, because the whole paper rests on that number.\n\nThe paper's own deployment analysis makes things worse. The learned policy is almost constant: patch every position except <s>, <unk>, and '.'. So the classifier's contribution is indistinguishable from a trivial fixed policy. Section 4.6 says a random classification baseline demonstrates the benefit, but that comparison is never shown. For a method whose claim is that learned selection matters, this is a missing control, not a minor omission.\n\nMinor stuff: the MuSiQue citation looks wrong—they cite a music-recommendation paper with the same name. The classifier accuracy of 0.81 is reported on the training distribution after SMOTETomek, which isn't informative.\n\nWhat's solid: the additional experiments on layer source and distance, the unsuccessful positional-concatenation experiment, and the honest limitations.\n\nWho is this for? People interested in activation patching as an inference-time intervention. The methodological flaw is fixable: hold out a subset of questions, report a trivial policy baseline, and run a significance test. With that, the result might hold or vanish; as it stands, the headline is an in-sample fit and the paper shouldn't be published as is. But I'd send it to a serious referee rather than desk reject, because the idea and code are worth a careful look under a revised protocol.","headline":"Clever idea, but the headline gain is measured in-sample and the learned policy is nearly trivial; needs a held-out split and a random-policy baseline before the result can be trusted.","tokens_in":8891,"tokens_out":2081,"would_cite":false,"duration_ms":19330,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auto-Patch claims a learned hidden-state transfer lifts multi-hop solve rate from 18.45% to 23.63%.","keywords":["large language models","multi-hop reasoning","model interpretability","PatchScopes","dynamic patching","hidden state intervention","MuSiQue dataset","chain-of-thought prompting"],"falsifier":"Run Auto-Patch on held-out MuSiQue two-hop questions whose hidden states never entered the training labels and compare the solve rate with the 18.45% baseline; if the gain disappears, the Table 1 result reflects training on the test set. A second check replaces the SVM with a fixed rule that patches every position except <s>, <unk>, and '.'; if accuracy stays near 23.63%, the classifier's choices are not what makes the method work.","tokens_in":7784,"feed_emoji":"🧩","tokens_out":10723,"duration_ms":92125,"temperature":0.7,"pith_summary":"Auto-Patch claims that a language model can reason better across two hops if, during inference, a learned classifier copies selected hidden states from an earlier layer into a later layer at the same token position. The method is tested on LLaMA 2 7B with MuSiQue 2-hop questions, where it raises the solve rate from an 18.45% baseline to 23.63% ± 0.7%, compared with 27.44% for chain-of-thought prompting. If the claim holds, it offers a third route to improving multi-step reasoning—neither prompting nor fine-tuning—and shows that interpretability-style interventions can be automated with a lightweight classifier.","feed_headline":"Auto-patching hidden states lifts multi-hop accuracy to 23.63%","feed_subtitle":"A learned classifier decides when to copy internal states, closing half the gap to chain-of-thought prompting without a prompt.","key_machinery":"The central mechanism is hidden-state patching: copying the representation at one layer and position into the same position of another layer during the forward pass. Auto-Patch automates this with an SVM classifier (RBF kernel, trained on hidden states labeled by whether a patch raised the probability of the correct answer) that decides at each position whether to copy layer 8 into layer 15; preprocessing uses SMOTETomek balancing and standardization. The second forward pass then completes generation with the patched states, and the added experiments vary the source layer and the source-to-target distance to map where the transfer helps most.","core_discovery":"On the paper's own account, the discovery is that per-position hidden-state replacements decided by a simple classifier improve multi-hop answering without manual prompt separation. For each token position, an SVM with an RBF kernel inspects the hidden state at layer 15 and decides whether to replace it with the hidden state at the same position in layer 8; when the decision is positive, a second forward pass runs with the patch applied. Across 1,024 MuSiQue two-hop questions, this increases the solve rate from 18.45% to 23.63 ± 0.7%, narrowing the gap to chain-of-thought prompting (27.44%). The authors also observe that the learned policy is nearly uniform: almost every position is patched except the sentence-start, unknown, and period tokens, so the effective intervention is close to a systematic lower-to-higher layer transfer.","pith_inferences":["Because the learned policy is almost constant, patching every non-special token with the same layer-8-to-layer-15 rule may reproduce most of the 23.63% gain; a one-line rule baseline would test whether the classifier's selectivity is the active ingredient.","The paper reports no train/test split between the prompts used to generate classifier labels and the prompts scored in Table 1, so the reported gain should be re-measured on held-out MuSiQue questions before treating it as general method performance.","The bibliography's MuSiQue reference points to a music-recommendation dataset rather than the multi-hop QA benchmark used in the experiments; verifying the actual data source is necessary for reproducibility.","If the layer transfer itself carries the benefit, then a context-aware classifier over neighboring states—the paper's own future direction—should beat the per-position SVM; if it does not, selectivity adds little to the method."],"forward_implications":["Multi-hop question answering can be improved without prompt engineering or fine-tuning, since the intervention is a learned edit applied during the forward pass.","The method's computational cost is roughly one additional forward pass per question, with no parameter updates or external memory.","The gap between plain prompting and chain-of-thought prompting can be halved: 23.63% versus 27.44%, up from an 18.45% baseline.","A lightweight per-position classifier is sufficient to learn where patching helps, suggesting that richer contextual models are not required for the basic effect.","Layer-choice experiments locate the most useful transfer around mid layers (roughly layers 10–12 at a fixed distance of 5), which can guide where future interventions should be aimed."],"supporting_citations":[{"why":"Provides the PatchScopes framework whose layer-to-layer hidden-state patching operation Auto-Patch automates.","marker":"Ghandeharioun et al. [2024]"},{"why":"Supplies the LLaMA 2 7B model that all experiments and the 18.45% baseline use.","marker":"Touvron et al. [2023]"},{"why":"Defines the chain-of-thought prompting baseline whose 27.44% solve rate is the comparison target.","marker":"Wei et al. [2023]"},{"why":"Cited as the source of the 1,024 MuSiQue prompts used for training and evaluation; as printed it points to a music-recommendation dataset, so the data source needs verification.","marker":"Deldjoo et al. [2021]"}],"fun_headline_variants":["Auto-Patch improves multi-hop reasoning by patching hidden states","Hidden-state patching lifts multi-hop solve rate to 23.63%","Auto-Patch: dynamic hidden-state swaps boost multi-hop accuracy","Classifier-guided hidden-state patching improves multi-hop reasoning","Patched hidden states close half the gap to chain-of-thought"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on evaluating the same 1,024 MuSiQue prompts whose per-position patch outcomes were used to create the classifier's training labels, so the reported gain assumes that in-sample improvement equals the method's performance on new questions.","fun_headline_variants_meta":{"raw":{"variants":["Auto-Patch improves multi-hop reasoning by patching hidden states","Hidden-state patching lifts multi-hop solve rate to 23.63%","Auto-Patch: dynamic hidden-state swaps boost multi-hop accuracy","Classifier-guided hidden-state patching improves multi-hop reasoning","Patched hidden states close half the gap to chain-of-thought"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1695,"prompt_tokens":838,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":769}},"tokens_in":454,"tokens_out":857,"duration_ms":6331,"temperature":1.0,"reasoning_tokens":769,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:04:55.818593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Auto-Patch on held-out MuSiQue two-hop questions whose hidden states never entered the training labels and compare the solve rate with the 18.45% baseline; if the gain disappears, the Table 1 result reflects training on the test set. A second check replaces the SVM with a fixed rule that patches every position except <s>, <unk>, and '.'; if accuracy stays near 23.63%, the classifier's choices are not what makes the method work.","supporting_citations":[{"cited_title":"Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the PatchScopes framework whose layer-to-layer hidden-state patching operation Auto-Patch automates."},{"cited_title":"Musique: A large-scale dataset for music recommendation with user-centric information, 2021","cited_arxiv_id":null,"evidence_quote":"Cited as the source of the 1,024 MuSiQue prompts used for training and evaluation; as printed it points to a music-recommendation dataset, so the data source needs verification."}],"review_version":1}