{"id":"834229fe-01de-4693-b215-57f19019de90","arxiv_id":"2608.05872","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A Markov-chain policy over transformer layers, decoded with top-k Viterbi, learns frozen-model routes that improve test accuracy by 5.0% on average.","lead":"MACRO learns a task-specific route through the frozen layers of a large language model, skipping, repeating, or rewinding blocks to improve accuracy. It reports about 5% higher average accuracy than the standard forward pass on reasoning and knowledge benchmarks, with the biggest gains on smaller models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported +5.0% gain may largely reflect an artificially weak baseline configuration, not a general benefit of layer routing; top-1 Viterbi shows gains independently of validation selection.","rationale":"The paper is methodologically solid internally: it reports ablations, statistical significance with cluster bootstrap, route-transfer analyses, and releases code. The top-1 Viterbi control shows that route search itself, not validation selection, produces the gains. However, the external validity of the central claim hinges on the baseline being representative of the model's actual performance. The fixed greedy decoding with a small token budget and no explicit reasoning prompt is known to severely handicap reasoning-distilled models; DeepSeek-R1-Distill-Llama-8B's 20.67% on GSM8K is far below the published ~80-85% range. The pattern across Table 2 and Table A.1—large gains on weak baselines, tiny gain on a strong baseline—is consistent with the hypothesis that MACRO is repairing a suboptimal decoding configuration rather than adding a general capability. This does not make the internal comparisons wrong, but it makes the headline '+5.0% improvement' misleading for practical use. The validation-selection concern is a smaller effect: MACRO(top-1) without validation selection already improves by almost the same amount, so the reader's weakest assumption is not the most load-bearing. I keep the CONDITIONAL verdict because the central claim can be rescued if the authors demonstrate gains under standard inference protocols; otherwise the practical significance is unestablished.","tokens_in":23157,"tokens_out":14361,"duration_ms":140406,"concrete_test":"Run the official lm-evaluation-harness (or the released MACRO code) with each model's native chat template, chain-of-thought prompting, and a token budget of at least 1024 on GSM8K and ASDiv for Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B. If the corrected baseline accuracy approaches published levels and MACRO's routed accuracy cannot beat it by more than 1-2 points, the reported +5.0% average improvement is an artifact of the weak baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MACRO improves test accuracy by +5.0% over the unrouted baseline is measured against a baseline whose decoding configuration is likely far weaker than standard practice. Appendix C fixes generation to greedy decoding with at most 400 thinking tokens and 128 answer tokens, with no mention of a chat template or chain-of-thought prompting. This produces implausibly low baselines, e.g., DeepSeek-R1-Distill-Llama-8B at 20.67% on GSM8K (Table 2), a model that typically scores above 80% under standard inference. The paper's own text notes gains are widest where the baseline is weakest, and on a strong model (Llama-3.2-3B, Table A.1) the gain is only +0.35 points. If the baseline is repaired to a realistic configuration, the headline improvement may shrink or vanish, undermining the practical claim. The validation-sample concern raised by the reader is secondary: MACRO(top-1), which skips validation selection, already captures most of the gain (Table 2, e.g., Qwen3-1.7B top-1 62.77 vs full 63.03), so validation noise is not the main driver.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MACRO, a framework that learns task-specific layer-routing programs for frozen LLMs. Routing is modeled as a context-dependent Markov policy over layer indices, budget phases, displacements, and operator context, updated via an estimation-of-distribution procedure on training feedback and decoded with an exact top-k Viterbi algorithm. The authors report a +5.0% average accuracy improvement over a sequential baseline across six open-weight LLMs and thirteen benchmarks, a +7.2% margin over Dr.LLM, and a 9.4x reduction in route-search time. They also present ablations, a mechanistic interpretability analysis, and route-transfer experiments.","tokens_in":23484,"tokens_out":3756,"duration_ms":38862,"significance":"If the empirical claims hold, MACRO is a well-engineered contribution: it is parameter-free with respect to the LLM (no weight updates), requires no per-instance test-time search, and its top-k Viterbi decoding is argued to be exact over the augmented state space. The paper is commendable for its detailed experimental protocol, including hyperparameter sweeps, statistical significance tests, ablations of each state component, and a public code release. The mechanistic logit-lens analysis is a useful step toward understanding why repeated or revisited layers can help. However, the significance of the headline accuracy gains depends critically on whether the unrouted baseline is a fair, standard inference configuration.","major_comments":[{"comment":"The baseline decoding configuration appears to be non-standard and likely artificially weak. Appendix C specifies deterministic greedy decoding with at most 400 thinking tokens and 128 answer tokens, but it does not mention the chat template, system prompt, few-shot exemplars, or chain-of-thought prompting. Under this configuration, the reported DeepSeek-R1-Distill-Llama-8B baseline on GSM8K is 20.67%, which is far below the model's typical standard-inference performance (often above 80%). If the baseline is set to a non-standard or insufficiently prompted configuration, the headline claim of a +5.0% average improvement over the unrouted baseline is not a meaningful measure of practical gain. The paper's own observation that gains are largest where the baseline is weakest (e.g., Table 2, DeepSeek row) underscores this concern. The authors should either specify and use a standard chat/CoT prompting setup for all methods and baselines, or explicitly scope the claim as an improvement over a constrained inference budget, and report results under both configurations.","section":"Appendix C ('Splits and Decoding') and Table 2"},{"comment":"The statistical significance tests and the per-model average improvements are computed against the same weak baseline. For Qwen3-8B, Qwen3-14B, and Llama-3.2-3B, the gains are small (+0.76, +0.78, +0.35 points respectively), while the large apparent gains on Qwen3-1.7B (+12.68) and DeepSeek-R1-Distill-Llama-8B (+13.04) coincide with baselines that are implausibly low. The paper should report results with a properly configured baseline (including chat template and reasoning prompts) and re-run the significance analysis; otherwise the claim that 'every model improves significantly' is only about a weak reference point.","section":"Section 'Main Results' / Table 2 and Table 3"}],"minor_comments":[{"comment":"The final route is selected among five Viterbi candidates on a 100-example validation split. While MACRO(top-1) largely reproduces the gains, which mitigates the risk, the paper would benefit from an analysis of validation-rank stability across seeds, or a bootstrap estimate of the selection noise, to quantify how much of the final reported accuracy is attributable to selection on a small validation set.","section":"Section 'Final Selection' and Appendix C"},{"comment":"The table reports mean and standard deviation over three seeds, but for the baseline rows no variance is given. It would be helpful to report the seed-level variance of the baseline as well, since the significance tests treat the baseline as deterministic.","section":"Table 2, footnote/legend"},{"comment":"Some of the discovered routes are complex (e.g., Qwen3-1.7B on MS-Arith: 'L0-L2 L4 L3-L7 L7 RJ') and the route notation is dense. A short worked example of how to read a route string (beyond the current caption) would improve reproducibility.","section":"Appendix B (Table B.1) and Figure 2"},{"comment":"The inference-cost analysis reports average overhead (+16% latency, +4.6% memory) but does not discuss the variance across benchmarks; the text says overhead is 'uneven' but only the average is highlighted. Reporting per-benchmark overhead with the same granularity as Figure D.1 would be more informative.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The strongest version of the paper's claim, that MACRO improves accuracy by +5.0% on average, is undermined by the likely non-standard baseline configuration. This is a load-bearing issue that could change the headline result substantially. The authors should be asked to redo the evaluation with a properly configured baseline (standard chat template and reasoning prompts) and report both the original constrained-budget results and the standard-configuration results. If the gains largely disappear under a fair baseline, the paper's contribution should be reframed accordingly. The method itself is interesting and well-described, so I do not recommend rejection outright, but the empirical evidence in its current form is not sufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Markov-chain routing idea is genuinely new, and the engineering around it is careful: the augmented state (layer, budget phase, displacement, operator context), the cross-entropy-style policy update, and the exact top-k Viterbi decoding are all specified cleanly, and the code is public. The ablations, hyperparameter sweeps, and transfer experiments are a cut above what this literature usually ships. I believe the mechanism is real: on models where the baseline is not pathological, MACRO still gives small but consistent gains, typically 0.3–1.5 points, and the statistical tests are sensible.\n\nThe soft spot is the baseline. Appendix C fixes generation to greedy decoding with at most 400 thinking tokens and 128 answer tokens and makes no mention of a chat template or chain-of-thought prompting. The result is a baseline that is far weaker than what these models achieve in standard use: DeepSeek-R1-Distill-Llama-8B at 20.67% on GSM8K is a red flag. The paper's own Table 2 shows gains are widest exactly where the baseline collapses, and on the strongest configurations the gain shrinks to under a point. So the headline +5.0% average is mostly an artifact of a weak comparison point, not a general property of layer routing.\n\nI also have to push back on the stress-test note's claim that top-1 Viterbi captures most of the gain independently of validation selection. On DeepSeek, top-1 is effectively flat (42.94 vs 42.87); on Qwen3-8B and Qwen3-14B, top-1 is slightly below the baseline. The large improvements on those models come only after selecting among five candidates on a 100-example validation split. The validation step is not inert; it is doing real selection work, and with 100 examples that selection is noisy. The reader's concern is thus more central than the stress-test suggests, though the baseline weakness is the bigger problem.\n\nWhat is salvageable: the method itself, the exact decoding trick, and the evidence that on non-degenerate baselines routing can buy a modest, reproducible improvement without weight updates. That is worth a serious referee, but the evaluation needs to be redone with standard prompting/decoding, and the authors should report results against baselines that match each model's recommended inference settings. As written, the practical claim is overstated.\n\nI would send this to review, but I would expect a major revision that confronts the baseline issue head-on. For my own work, I would not cite the quantitative gains, though I might cite the method if I need a reference for Markov-chain layer routing.","headline":"A novel and well-engineered routing method whose headline gains are mostly a byproduct of an artificially weak baseline; the central practical claim needs a re-evaluation under standard inference settings.","tokens_in":23994,"tokens_out":2465,"would_cite":false,"duration_ms":26011,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MACRO claims that task-specific layer rerouting of frozen LLMs—via a Markov-chain policy with top-k Viterbi decoding—improves accuracy by 5.0 percentage points on average over the standard layer order.","keywords":["layer routing","Markov chain","Viterbi decoding","frozen language models","dynamic computation","test-time adaptation","estimation-of-distribution search"],"falsifier":"Re-run the final selection stage with the validation labels randomly permuted, or with two disjoint 100-example validation pools, and compare the selected routes and their test accuracies; if a route selected on shuffled labels still produces the +5% test gain, the improvement is not driven by genuine route quality. A second check is to hold out a second validation set and verify that the validation-best route is also the test-best among the five Viterbi candidates.","tokens_in":23010,"feed_emoji":"🔄","tokens_out":5892,"duration_ms":53637,"temperature":0.7,"pith_summary":"The paper tries to establish that you can make a frozen large language model more accurate on a given task by changing only the order in which its layers run: skipping some, repeating others, and occasionally adding a previous hidden state. Existing dynamic-routing methods need per-instance search, ground-truth labels at test time, or expensive offline label generation. MACRO instead learns one Markov-chain policy per task from training feedback, decodes a small set of high-probability routes with top-k Viterbi, and picks the best on a held-out validation split. If correct, this means accuracy gains are available without any weight updates and with far less route search than prior approaches. The headline numbers are +5.0% average accuracy over the sequential baseline and +7.2% over Dr.LLM, with route-search time cut by 9.4x.","feed_headline":"Frozen LLM layers rerouted for 5-point accuracy gain","feed_subtitle":"Markov-chain routing skips, repeats, and adds layers without touching weights, beating Dr.LLM by 7.2 points.","key_machinery":"The central object is the Markov route generator: a masked, context-dependent transition policy $\\pi_\\theta(a_t | s_t)$ over states $s_t = (\\ell, \\phi, \\delta, o)$ recording the current layer, the computation-budget phase (early, mid, late), the incoming displacement, and the operator context, with three action families: local layer moves within a small radius, add-and-apply of a previous hidden state, and rejoin of the standard suffix. Top-k Viterbi decoding over this augmented state space is the exact decoder that turns the learned transition table into a small set of candidate routing programs without any per-example test-time search.","core_discovery":"MACRO models layer routing as a context-dependent Markov chain whose state records the current layer, the remaining budget phase, the incoming displacement, and whether the hidden state came from a plain layer or an add operation; actions are local layer moves, add-and-apply, and rejoin. It learns a single task-level transition table by sampling routes, scoring them on training subsets, and refitting via weighted maximum likelihood, a structured cross-entropy approach. At deployment it decodes the exact top-k highest-probability valid routes with an augmented Viterbi pass, deduplicates them, and selects one on validation data. The claim is that this route, applied uniformly at test time, improves accuracy over the sequential forward pass on thirteen benchmarks across several open-weight LLMs, with the largest gains on weak and distilled models.","pith_inferences":["If the validation-selection result generalizes, the method's implicit claim is that a large pretrained model often represents the right answer internally but suppresses it near the output; the logit-lens evidence points to routing as exposing an already-computed answer, which would make readout-pathway interventions a promising test-time-compute direction.","A testable extension is input-conditional routing: the paper learns one route per task, but the state space already contains enough structure to condition on a lightweight input embedding, which could push gains on mixed-task streams.","The Markov-table formulation could also apply to other frozen computation graphs, such as vision backbones or diffusion denoisers, where a small per-task transition table over a few stages would be even cheaper to learn.","The 100-example validation selection is the main fragility; a direct comparison with a much larger validation pool or leave-one-benchmark-out selection would show how much of the reported gain is route quality versus selection luck."],"forward_implications":["On six open-weight LLMs across thirteen benchmarks, MACRO reports an average +5.0 percentage-point accuracy gain over the sequential baseline, with the largest gains on the weakest models.","MACRO's route search costs about 1.6 hours per benchmark versus 14.8 hours for Dr.LLM, a 9.4x reduction, while improving accuracy by +7.2 points.","The action space itself is a major contributor: Dr.LLM+Ext., which uses the MACRO action space with Dr.LLM's training procedure, outperforms original Dr.LLM on every tested model.","Both decoding and validation selection matter: MACRO(noVit.) and MACRO(top-1) are individually weaker than full MACRO, and validation selection often picks a non-rank-1 Viterbi candidate.","A single route trained on a pooled set of eight benchmarks transfers out-of-domain, especially to math reasoning tasks, suggesting routes encode reusable computation rather than only benchmark-specific quirks."],"supporting_citations":[{"why":"Dr.LLM is the strongest dynamic-routing baseline; MACRO claims +7.2% accuracy over it and 9.4x lower route-search cost.","marker":"(Heakl et al. 2026)"},{"why":"CoLA performs per-sample MCTS over skipped and looped layers with ground-truth reward, the per-example-search approach MACRO avoids.","marker":"(Li, Li, and Zhou 2025)"},{"why":"Neuroanatomy exhaustively searches a constrained route space; MACRO contrasts its computational prohibitivity.","marker":"(Ng 2026)"},{"why":"Supplies the Markov-chain formalism underlying the route policy and its Viterbi decoding.","marker":"(Norris 1998)"},{"why":"GSM8K is the primary math-reasoning benchmark where MACRO reports its largest single-route gain, from 43.4% to 69.5%.","marker":"(Cobbe et al. 2021)"},{"why":"MATH500 is part of the evaluation suite used to measure MACRO's accuracy improvements.","marker":"(Lightman et al. 2024)"},{"why":"The recurrent-depth behavior of inner thinking transformers inspires the add-and-apply operation in the MACRO action space.","marker":"(Chen et al. 2025)"},{"why":"Mixture-of-recursions provides additional recurrent-depth inspiration for the add-and-apply action.","marker":"(Bae et al. 2026)"}],"fun_headline_variants":["Markov routing reroutes LLM layers for 5% accuracy gain","Weight-free layer routing boosts LLM accuracy 5%","MACRO: 5% accuracy gain from Markov layer routing","Skip, repeat, add: Markov chain reroutes LLM layers","Layer routing via Markov chain beats Dr. LLM by 7.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ranking five candidate routes on a 100-example validation split reliably identifies a route that is genuinely better on the test distribution; if the validation sample is too small or noisy, some reported accuracy gains could be a small-sample selection artifact.","fun_headline_variants_meta":{"raw":{"variants":["Markov routing reroutes LLM layers for 5% accuracy gain","Weight-free layer routing boosts LLM accuracy 5%","MACRO: 5% accuracy gain from Markov layer routing","Skip, repeat, add: Markov chain reroutes LLM layers","Layer routing via Markov chain beats Dr. LLM by 7.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1392,"prompt_tokens":948,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":564,"tokens_out":444,"duration_ms":4824,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:55:46.681019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the final selection stage with the validation labels randomly permuted, or with two disjoint 100-example validation pools, and compare the selected routes and their test accuracies; if a route selected on shuffled labels still produces the +5% test gain, the improvement is not driven by genuine route quality. A second check is to hold out a second validation set and verify that the validation-best route is also the test-best among the five Viterbi candidates.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Dr.LLM is the strongest dynamic-routing baseline; MACRO claims +7.2% accuracy over it and 9.4x lower route-search cost."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Markov-chain formalism underlying the route policy and its Viterbi decoding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The recurrent-depth behavior of inner thinking transformers inspires the add-and-apply operation in the MACRO action space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mixture-of-recursions provides additional recurrent-depth inspiration for the add-and-apply action."}],"review_version":1}