{"id":"fb017b72-d686-46bd-b54d-e5985196a867","arxiv_id":"2412.07282","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HARP improves LLM accuracy by up to 5.16% at about 1.25x inference cost, by running a dropout-perturbed second forward pass only when token-level entropy is high.","lead":"HARP adds an extra, slightly scrambled forward pass to an LLM only when the model is unsure about the next word, lifting accuracy on several benchmarks by up to about five points. It is a simple, training-free recipe that could be layered onto existing models and decoding methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claim depends on a no-KV-cache setup; with standard KV caching, each reframing step recomputes the full prefix, so the reported ~1.25x overhead is not representative.","rationale":"The reader's weakest assumption concerns the mechanism (dropout averaging as variance reduction). That concern is real, but Table 3 partially addresses it: the entropy-gated version outperforms the unconditional extra pass on CsQA and MMLU Pro, so the gate is not purely superfluous. The efficiency claim, by contrast, is directly weakened by the absence of cache-aware evaluation and by the authors' own limitation statement. Since the abstract foregrounds 'minimal additional cost' and the paper claims to be 'twice faster than beam search,' the cost figure must hold in a realistic inference setting. Running without KV cache is a nonstandard condition that artificially amortizes the extra forward pass. I recommend keeping the CONDITIONAL verdict, adding a condition that the authors either measure with KV cache enabled or explicitly scope the efficiency claims to the no-cache setting.","tokens_in":16588,"tokens_out":7164,"duration_ms":74154,"concrete_test":"Run the released HARP code on a long-context dataset (e.g., CNN/DM summarization, average input >512 tokens) with KV cache enabled. Measure wall-clock time and peak VRAM for vanilla greedy, HARP greedy, and beam search (b=3) on the same GPU/quantization. If ratio time(HARP with cache)/time(vanilla with cache) exceeds ~2x, or if HARP becomes slower than beam search, the reported x1.25 overhead does not transfer to the standard deployment setting. Report the fraction of tokens above threshold and average context length to quantify the recomputation cost.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All timing comparisons (Table 6, Figure 2) are run 'without caching (KVCache)' (Section 4.3). In that regime every token already costs O(L) for a length-L context, so one extra forward pass is a constant-factor overhead. In standard inference, KV cache makes vanilla greedy O(1) per token, but HARP's reframing pass (Algorithm 1, Eq. 3) re-embeds the whole sequence and runs f\\emb from scratch, invalidating the cache. Each high-entropy token therefore triggers O(L) extra work plus a VRAM spike. The paper's own Limitations concede that 'embedding dropout may temporarily invalidate the KVCache' and that 'VRAM usage could briefly double.' With frequent uncertainty (e.g., LAMBADA), the actual slowdown over cached greedy could be orders of magnitude larger than x1.25, possibly exceeding beam search. The abstract's 'minimal additional cost' and Section 5's 'twice faster than beam search' claim are thus not established in the setting where KV cache is used. This is load-bearing because HARP's value proposition is accuracy gains at low inference cost.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HARP proposes a training-free, model-agnostic modification to Transformer inference: when the Shannon entropy of the next-token distribution exceeds a threshold theta, the model performs an extra forward pass with dropout-perturbed embeddings and linearly averages the original and reframed logits (Eq. 4). The method is evaluated on five tasks (CsQA, GSM8K, LAMBADA, MMLU Pro, CNN/DM) with three instruction-tuned models under greedy and nucleus sampling, reporting accuracy gains up to +5.16% and an average inference-time overhead of about 1.25x over vanilla decoding, which is claimed to be faster than beam search. The paper also includes analyses of the uncertainty gate, multiple reframing steps, a NEFTune-versus-dropout comparison, a threshold sensitivity study, and an extended evaluation with LM-Evaluation-Harness.","tokens_in":16819,"tokens_out":6916,"duration_ms":68497,"significance":"If the empirical claims hold, HARP would be a simple and useful adaptive-computation method for off-the-shelf LLMs, requiring no retraining and no architectural changes. The paper has genuine strengths: the method is conceptually simple, the code is promised to be public, and Appendix D provides a welcome attempt at a more standard evaluation with multiple seeds. However, the reported gains are not stable across the main and extended evaluations, the hyperparameters were selected on the same benchmarks that are then used to report results, and the efficiency claim is tied to a no-KV-cache setting that is not representative of standard inference. In its current form, the paper supports a proof of concept rather than the abstract's general claim of 'minimal additional cost' and performance improvements up to 5.16%.","major_comments":[{"comment":"The hyperparameters theta=1.0, delta=0.20, and beta=0.5, as well as the choice of dropout over NEFTune noise, were selected using the same five benchmark datasets whose results are then reported as the main findings. This is a form of test-set tuning, so the reported improvements are optimistic and cannot be interpreted as unbiased estimates of generalization. The authors should either use a held-out validation set for hyperparameter selection or present results under a fixed configuration with sensitivity analyses on unseen tasks.","section":"Section 3.4 and Section 4.3, with Appendices A and C"},{"comment":"All timing comparisons in Table 6 and Figure 2 are run \"without caching (KVCache)\". In standard inference with a KV cache, the reframing pass (Eq. 3) re-embeds the whole sequence and recomputes the full prefix, so each high-entropy token incurs O(L) extra work rather than a constant-factor overhead, and VRAM usage can spike. The Limitations section explicitly concedes that \"embedding dropout may temporarily invalidate the KVCache\" and that VRAM usage could double. The abstract's \"minimal additional cost\" and Section 5's \"twice faster than beam search\" claim are therefore not established for the cached setting; the authors should either re-measure with an actual KV-cache implementation or clearly qualify the claims to the no-cache setting.","section":"Algorithm 1, Section 4.3, and Section 8"},{"comment":"For LLaMA-3.1 Instruct 8B with greedy decoding, Table 1 reports HARP's MMLU Pro score as 48.21 and CNN/DM ROUGE-1 as 34.03, while Appendix A's Table 5 reports 71.43 and 48.21 for the same HARP(DROPOUT) setting. These differences of roughly 14 points are too large to be explained by formatting and appear to be an internal inconsistency. The authors must correct the table or explain the discrepancy before the main results can be trusted.","section":"Table 5 versus Table 1"},{"comment":"The extended LM-Evaluation-Harness results in Appendix D differ dramatically from the main results for the same model: LAMBADA vanilla accuracy is 55.52 in Table 9 versus 30.86 in Table 1, MMLU Pro is 33.30 versus 46.42, and CNN/DM ROUGE-1 is 35.77 versus 32.44. The extended gains are also much smaller (e.g., MMLU Pro +0.26, CNN/DM +0.36). The main experiments use unquantified subsets with a single seed, and the paper does not reconcile these differences. Until the evaluation protocol and scoring are clarified and the two sets of numbers are reconciled, the headline improvement of +5.16% cannot be considered robust.","section":"Table 9 versus Table 1"},{"comment":"Because dropout is applied at inference time, HARP introduces stochasticity even under greedy decoding, yet the main results in Table 1 are reported without error bars. Appendix D reports standard deviations for some extended runs and states that \"certain seeds can still negatively impact results,\" which confirms that the method is noisy. The authors should provide variance estimates, significance tests, and the exact subset sizes and seeds used for all main claims.","section":"Section 5 and Appendix D"}],"minor_comments":[{"comment":"The word \"Parallely\" should be \"In parallel\" or \"Parallelly\".","section":"Section 2.3"},{"comment":"The notation \"SHANNON(P) = - |V| X ...\" is typeset in a way that obscures the formula; the summation range and the function name should be formatted more clearly.","section":"Equation (2)"},{"comment":"The formatting of the Mistral CoT row is garbled: \"Ours CoT75.23(+4.59) 46.00(-2.00) 32.95(+0.58)\" needs proper column separation and model labeling, and the -2.00 drop on GSM8K should be discussed in the text.","section":"Table 2"},{"comment":"Both figures contain garbled unicode/rendering artifacts that make them difficult to read; the PDF rendering should be fixed.","section":"Figure 2 and Figure 3"},{"comment":"The paper says a \"subset\" of each dataset is evaluated but never states the subset sizes; this information is essential for assessing the reliability of the main results.","section":"Section 4.3"},{"comment":"Table 8 is labeled \"Relative accuracy,\" but the values appear to be absolute accuracy gains in percentage points; the label should be clarified.","section":"Appendix C"},{"comment":"The phrase \"with a top-k of 5\" after beam search parameters is missing a space before \"with,\" and the beam-search length penalty alpha should be stated more explicitly as a hyperparameter that is fixed across datasets.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the paper is honest about some limitations, but the evaluation as written is not yet trustworthy: the main numbers are inconsistent with the appendix tables, the hyperparameters are tuned on the same benchmarks, and the efficiency claim is tied to a no-KV-cache setting. I would like to see a consistent evaluation with held-out validation or pre-registered hyperparameters, a KV-cache-aware timing measurement, and corrected tables before considering this for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the central result is real but modest: averaging a dropout-perturbed forward pass into the logits when Shannon entropy exceeds a threshold buys a few accuracy points on several benchmarks across three model sizes. Second, the headline efficiency claim—~1.25x overhead and faster than beam search—is measured without KV caching, and the method invalidates the cache on every reframing step, so in standard cached inference the overhead can be far worse than reported.\n\nWhat's new here is genuine: nobody else in the cited literature combines token-level entropy gating with inference-time embedding dropout. NEFTune is training-only, pause tokens need retraining, and SED uses a different uncertainty measure without reframing. The paper is clearly written, the algorithm is simple to implement, and the code is public. The ablation against unconditional extra computation (Table 3) and the multi-step analysis (Table 4) are the right kinds of sanity checks.\n\nNow the soft spots. The efficiency claim is load-bearing, and it is not established for the setting where anyone actually runs inference. All timings are \"without caching (KVCache)\"—fine as a controlled comparison, but the abstract's \"minimal additional cost\" and \"twice faster than beam search\" do not carry over to cached decoding. The reframing pass re-embeds the full prefix, so with a cache each high-entropy token costs O(L) extra work and a VRAM spike; the paper's own Limitations admit this. The accuracy numbers themselves are weaker than the prose: Table 1 shows several negative deltas, yet Section 5 says \"HARP improves the performance of all tasks.\" Hyperparameters θ, δ, β were chosen during preliminary testing on the same five benchmarks, and the main table has no error bars. Appendix D partially addresses this with standard deviations, but only for one model and with mean gains that are small (e.g., 0.27 points on CsQA). The mechanism also remains a bit hand-wavy: why should zeroing 20% of embedding entries and averaging logits improve accuracy? The paper offers cognitive analogies, not a mechanism; it could be variance reduction from a noisy self-ensemble, and the uncertainty gate might be incidental.\n\nWho is this for? Anyone interested in training-free, pluggable inference-time adapters. A serious referee should look at it, but the authors should be asked to (1) redo the efficiency comparison with KV cache enabled, (2) report error bars on the full main table, (3) justify hyperparameter selection without peeking at the test sets, and (4) provide at least one held-out task for θ/δ tuning. The core idea is simple enough that with those fixes it could be a solid short paper.\n\nI'd recommend sending it to peer review—the idea is novel and the paper is honest about its limitations—but the reviewers should treat the efficiency claim with suspicion until it is measured in cached inference.","headline":"HARP is a training-free entropy-gated embedding-dropout inference trick with modest real accuracy gains, but its efficiency claim is tied to a no-KV-cache setup and its hyperparameters are tuned on the test tasks.","tokens_in":17327,"tokens_out":3347,"would_cite":true,"duration_ms":34156,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By letting a language model 'reconsider' when it is uncertain, a training-free tweak raises accuracy by up to 5.16% at modest extra cost.","keywords":["adaptive computation","Shannon entropy","embedding dropout","uncertainty estimation","Transformer inference","training-free","decoding","large language models"],"falsifier":"Replace the Shannon-entropy trigger with a random trigger that fires at the same rate and run HARP on the same benchmarks; if the accuracy gain persists, the entropy gate is not the cause of the improvement.","tokens_in":16346,"feed_emoji":"🧠","tokens_out":7975,"duration_ms":68581,"temperature":0.7,"pith_summary":"The paper claims that off-the-shelf Transformer language models can be made more accurate without any retraining by letting them spend a little extra computation exactly when they are uncertain. The method, HARP, measures the Shannon entropy of the next-token distribution and, for high-entropy steps, runs a second forward pass on embeddings that have been partially dropped out, then averages the two predictions. Across three model families and five benchmarks the authors report consistent gains, the largest being +5.16% on LAMBADA, with an average inference slowdown of about 1.25x compared to greedy decoding—roughly two and a half times faster than beam search. If correct, this makes token-level uncertainty a cheap, model-agnostic switch for adaptive computation.","feed_headline":"Entropy-gated 'reframing' lifts LLM accuracy up to 5.16%","feed_subtitle":"A plug-in inference tweak re-runs uncertain tokens with perturbed embeddings, beating beam search at a fraction of the cost.","key_machinery":"The central mechanism is an entropy-gated second forward pass. Shannon entropy of the softmaxed next-token distribution is the hesitation signal; dropout applied to the embedding matrix at rate δ creates the 'reframed' input; and a convex combination with β=0.5 fuses the two logit distributions. The gate is what makes the added computation selective, and the dropout is what makes the second pass a different perspective rather than a duplicate.","core_discovery":"HARP is a plug-in modification to the Transformer forward pass that operates per generated token. The model first computes its normal logits and converts them to a probability distribution; if the Shannon entropy of that distribution lies above a threshold θ, the token is deemed uncertain and the model 'hesitates'. The embeddings of the input are then passed through dropout with rate δ, a complete second forward pass is executed, and the original and reframed logits are merged with a convex combination β. The authors set θ=1.0, δ=0.20, and β=0.5, and report accuracy improvements of up to +5.16% on five datasets spanning reasoning, multiple-choice, next-word prediction, and summarization, using LLaMA-3.1 8B, Mistral 7B, and Phi-3.5 Mini. They interpret the second pass as a reframing of the input—an inference-time analogue of the human framing effect—and argue that the entropy gate is what directs extra compute to genuinely hard decisions.","pith_inferences":["A control experiment that triggers the second pass on randomly chosen tokens at the same rate as the entropy gate would isolate whether the uncertainty signal itself, rather than the extra averaged forward pass, is responsible for the gains.","The fixed hyperparameters (θ=1.0, δ=0.20) were tuned on a subset of the same benchmarks, so a natural test is whether they transfer to new tasks; an adaptive threshold per example would be a plausible improvement.","The authors note that reframing could be applied to deeper representations or to the key-value cache, which would avoid the cache invalidation and VRAM spike they identify as the main practical limitation."],"forward_implications":["Any off-the-shelf Transformer can be upgraded by swapping in the modified forward pass; no gradient updates, no fine-tuning, and no architectural changes are needed.","Greedy decoding with HARP outperforms beam search on most of the tested tasks while running about 2.5x faster, so it offers a better accuracy-latency trade-off.","HARP composes with nucleus sampling and Chain-of-Thought prompting, producing additional gains on top of those techniques.","The added cost is selective: on average the slowdown is about 1.25x, but the worst-case cost for a sequence of entirely uncertain tokens is nearly a full second forward pass per token."],"supporting_citations":[{"why":"Supplies the Shannon entropy used as the hesitation signal to decide when to reframe.","marker":"Shannon, 1948"},{"why":"Introduces embedding noise (NEFTune) that HARP adapts from training to inference, comparing dropout against uniform noise.","marker":"Jain et al., 2024"},{"why":"Pause tokens are the main performance-focused adaptive-computation baseline that HARP claims to beat without training.","marker":"Goyal et al., 2024"},{"why":"Provides the beam-search decoding baseline and its clarity analysis; HARP is compared against its accuracy and latency.","marker":"Kasai et al., 2024"},{"why":"The Transformer architecture whose forward pass HARP modifies.","marker":"Vaswani et al., 2017"},{"why":"GSM8K, the math-reasoning benchmark where HARP shows gains in greedy and CoT settings.","marker":"Cobbe et al., 2021"},{"why":"LAMBADA, the benchmark with the largest reported gain (+5.16%).","marker":"Paperno et al., 2016"}],"fun_headline_variants":["HARP: Pause and reframe to boost LLM accuracy 5.16%","Entropy-triggered reframing improves LLMs up to 5.16%","Hesitate, reframe, repeat: HARP lifts LLM scores 5.16%","Training-free HARP rechecks hard tokens, gains 5.16%","Plug-in HARP re-runs uncertain tokens for +5.16% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the premise that zeroing out 20% of embedding values and averaging the two resulting predictions is a genuinely informative reframing for high-entropy tokens, rather than a benefit that any small stochastic perturbation could supply at the same gate frequency.","fun_headline_variants_meta":{"raw":{"variants":["HARP: Pause and reframe to boost LLM accuracy 5.16%","Entropy-triggered reframing improves LLMs up to 5.16%","Hesitate, reframe, repeat: HARP lifts LLM scores 5.16%","Training-free HARP rechecks hard tokens, gains 5.16%","Plug-in HARP re-runs uncertain tokens for +5.16% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3499,"prompt_tokens":923,"completion_tokens":2576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2462}},"tokens_in":539,"tokens_out":2576,"duration_ms":54435,"temperature":1.0,"reasoning_tokens":2462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:43.535762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the Shannon-entropy trigger with a random trigger that fires at the same rate and run HARP on the same benchmarks; if the accuracy gain persists, the entropy gate is not the cause of the improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Shannon entropy used as the hesitation signal to decide when to reframe."},{"cited_title":"Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein","cited_arxiv_id":null,"evidence_quote":"Introduces embedding noise (NEFTune) that HARP adapts from training to inference, comparing dropout against uniform noise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the beam-search decoding baseline and its clarity analysis; HARP is compared against its accuracy and latency."}],"review_version":1}