{"id":"acc6954d-7eb8-42e2-a608-73255cb91adc","arxiv_id":"2507.08567","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.","lead":"AbbIE reuses one block of a Transformer several times to process text, and the paper shows that running it longer at test time improves accuracy without retraining. The practical promise is that a model trained cheaply with two internal passes can become smarter at inference by doing more passes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Upward generalization rests on a single 350M run, 16-sample convergence plots, and a Depth baseline that appears to be trained at fixed r=2 rather than the original sampled-iteration protocol; the central comparative claim is under-supported.","rationale":"The reader's weakest assumption identifies the single-seed and small-sample basis for upward generalization; I agree that this is a serious gap. I also think the Depth baseline implementation is at least as load-bearing, because the paper's claim of superiority over alternative iterative methods depends on a comparison with a method that appears to have been trained at a fixed iteration count rather than under the original recurrent-depth protocol. If that is correct, no amount of additional seeds on AbbIE-D would repair the comparative claim without also fixing the baseline. I do not see a fatal flaw in the architecture itself: the method is simple, the within-model improvements at 350M are consistent across tasks, and the perplexity comparisons are plausible. The issues are addressable with more seeds, a faithful baseline, and corrected headline numbers, so a conditional verdict remains appropriate. My concrete test directly checks both the statistical robustness of the 350M upward generalization and the fairness of the Depth comparison.","tokens_in":17792,"tokens_out":18803,"duration_ms":228804,"concrete_test":"Run AbbIE-D 350M with five additional seeds using the exact Table 1 configuration, and require that r=8 ICL accuracy exceed r=2 accuracy by more than the seed spread on HellaSwag, LAMBADA, and ARC-Easy; if the margin is not consistently positive, the upward-generalization claim fails. Independently re-run the Depth baseline under the original recurrent-depth protocol (sampled iteration counts, original input-concatenation/projection and normalization, as in Geiping et al. 2025) and compare at r=8. If Depth also improves with iterations, the uniqueness claim collapses. Additionally, recompute Fig. 4 with a common convergence criterion on at least 100 samples, since the AbbIE-C and AbbIE-D curves currently measure different residuals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing part of the paper is the claim that two training iterations suffice to make AbbIE-D improve at test-time iteration counts up to 4x training, and that this uniquely outperforms alternative iterative methods. That claim has two under-supported pillars.\n\nFirst, the Depth baseline in Fig. 6b and Table 2 collapses for r != 2. That collapse is exactly what one expects from a loop trained at fixed r=2. The original recurrent-depth method (Geiping et al., 2025) is trained over a range of sampled iteration counts, as the paper itself states in Sec. 5; the experimental section never says that Depth received that protocol. If Depth was trained at fixed r=2, the comparison is against a strawman, and the conclusion that AbbIE far outperforms alternative iterative methods is not established.\n\nSecond, the headline upward-generalization result is the 350M configuration, and Sec. 3.2 says only a single 350M training run was performed. Fig. 4's fixed-point evidence uses 16 random samples with one initial state per sample; additionally, the plotted quantities for AbbIE-C and AbbIE-D are different functionals of the Body (for C, ||B(h_k)-h_k||; for D, ||B(h_k)||), so cross-variant convergence conclusions should be treated cautiously. Finally, the abstract's 'up to 12%' improvement is not recoverable from Table 2's 350M rows: the largest relative gains are roughly 7-9%. The upward-generalization claim therefore rests on an unreplicated run, a small convergence sample, and a possibly unfair baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AbbIE, an encoder-only Transformer variant partitioned into Head, Body, and Tail groups, in which the Body stack is applied iteratively in latent space. Two variants are defined: AbbIE-C (direct recursion over the Body) and AbbIE-D (the Body output is added to its input via an extra inter-iteration residual). The authors train with two Body iterations and evaluate at test-time iteration counts of 1, 2, 4, 8, and 32. They claim AbbIE-D matches or beats a standard Transformer at equal token budget, achieves lower perplexity, and “upward generalizes”: performance improves at test-time iteration counts beyond the training count, up to 4× the training iteration count. The experiments compare AbbIE with a standard Transformer baseline (Std) and a recurrent-depth baseline (Depth) at 200M and 350M parameter scales on HellaSwag, LAMBADA, ARC-Easy, and CommonsenseQA.","tokens_in":18098,"tokens_out":5813,"duration_ms":63859,"significance":"The central idea is attractive: if a recurrent Transformer can upward-generalize from only two training iterations without specialized data or projections, it offers a practical, drop-in mechanism for test-time compute scaling that is complementary to parameter scaling. The paper has several strengths: the architecture is simple and clearly described, the training protocol is standard, the authors fix the token budget across comparisons, and they disclose the small number of large-model runs. However, the headline claims are only partially supported by the reported numbers. The abstract's 12% ICL improvement is not recoverable from Table 2, the 5% perplexity improvement is not tabulated, the 350M upward-generalization result rests on a single training run, and the Depth baseline appears to be trained under a protocol that may not match the original method. If the evidence were strengthened through corrected baselines, error bars, and a clearer match between claims and tables, the paper would make a useful contribution.","major_comments":[{"comment":"The Depth baseline training protocol is not specified. Section 5 states that the original recurrent-depth method of Geiping et al. (2025) is trained over a range of randomly sampled iteration counts, but the experimental section never states whether the Depth implementation in this paper received that protocol. The collapse of Depth at every iteration count other than r=2, visible in Fig. 6b and Table 2, is exactly what one expects from a model trained at fixed r=2. Because Depth is the principal iterative baseline, the claim that AbbIE “far outperforms alternative iterative methods” is not established unless Depth was trained with the same sampled-iteration protocol as the original method. Please specify the protocol; if Depth was trained at fixed r=2, re-run it with sampled iterations or clearly state the deviation and justify why the comparison is fair.","section":"§3.1, Fig. 6b, Table 2"},{"comment":"The “up to 12% improvement” claim is not recoverable from the reported numbers. In the 350M rows of Table 2, the relative improvements of AbbIE-D at r=8 over Std are approximately 7.1% on HellaSwag (34.8/32.5), 7.7% on LAMBADA (28.0/26.0), and 6.4% on ARC-Easy (51.3/48.2). Comparing against Depth(r=2) gives similar magnitudes. If the 12% figure comes from a different calculation (for example, against the random baseline or against Depth at a collapsed setting), the derivation must be reported. The same applies to the 5% perplexity improvement, which is mentioned in Fig. 5b and the Discussion but never appears as a tabulated value. The abstract and Discussion should state only the gains that are directly readable from the tables.","section":"Abstract, Table 2, Discussion"},{"comment":"The upward-generalization result rests on very limited statistical evidence. The 350M model, which is the only configuration showing clear gains at r=4 and r=8, was trained with a single seed (Section 3.2). Table 2 reports the “best result” without error bars or confidence intervals, and Fig. 4 uses only 16 random samples with one initial state per sample. Given that the central claim of the paper is that two training iterations suffice for upward generalization at larger scale, the authors should either provide multiple 350M seeds with error bars, or at minimum clearly label the single-run status in every figure and table that supports the scaling claim, and temper the strength of the conclusion accordingly.","section":"§3.2, §4.3, Table 2"},{"comment":"The fixed-point comparison between AbbIE-C and AbbIE-D is based on two different quantities. For AbbIE-C, the inter-iteration distance is ‖B(h_k) − h_k‖, because h_{k+1} = B(h_k). For AbbIE-D, the distance is ‖B(h_k)‖, because h_{k+1} = B(h_k) + h_k. These are different functionals with different fixed-point conditions (B(h*) = h* versus B(h*) = 0), so the observation that “AbbIE-C diverges while AbbIE-D converges” does not directly follow from comparing the two curves. Please plot a common distance measure, such as ‖h_{k+1} − h_k‖ or a normalized state difference, for both variants, and state explicitly which quantity is shown.","section":"§4.1, Fig. 4"},{"comment":"The perplexity improvement claim is not quantified in the text. Figure 5b shows curves that appear separated, and the caption says “roughly 5% better perplexity,” but no numerical Perplexity values or standard deviations are given. Since 200M models were trained with five seeds, the authors should report mean and standard error for Std, Depth, and AbbIE-D at the compute-optimal token budget, and state the actual relative improvement. This is necessary to verify both the direction and the magnitude of the claimed perplexity gain.","section":"§4.2, Fig. 5b"}],"minor_comments":[{"comment":"The caption says “increasing training iterations (1, 2, 4, and 8)”, but the models are trained with two iterations and evaluated at those test-time counts; please rephrase to avoid implying varying training iteration counts.","section":"Fig. 1 caption"},{"comment":"The term “detokenization” is used to mean the transition from token space to concept space inside the model, which is not the standard meaning of detokenization in language modeling; consider a different term or explicitly define the usage.","section":"§2.1"},{"comment":"The notation around path independence is loose: Eq. (1) uses f^∞(x, z0), while Eq. (2) writes f^{(n)}(x_n) and relates it to f^{(n+1)}(x_{n+1}); the role of the initial state z0 and the statement that “we omit z0” should be clarified, since path independence is defined with respect to arbitrary z0.","section":"§2.2, Eq. (1)-(2)"},{"comment":"The learning-rate schedule is described as “Warmup-Stable-Decay”, but the text only describes linear warmup and cosine decay; please specify what the “stable” phase is, or rename the schedule.","section":"§3.2"},{"comment":"There are small typographical issues: “Commensense Question Answering” should be “Commonsense Question Answering”, and the note in the table body is informal; consider moving protocol caveats to the caption.","section":"Table 2"},{"comment":"The sentence “we see a performance significantly higher than the random baseline” uses “significantly” without a statistical test; please replace with “numerically higher” unless a test is performed.","section":"§4.3"},{"comment":"The phrase “which no other general recursive transformer has shown” is absolute; please soften to “to our knowledge” or cite a negative result.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core architectural idea is plausible and potentially useful, but the experimental support for the headline claims is not yet commensurate with the strength of the abstract and Discussion. The most important issues are the likely protocol mismatch for the Depth baseline and the single-seed 350M evidence behind the upward-generalization claim. If the authors can correct the baseline comparison, add uncertainty estimates or additional seeds, and align the stated gains with the tables, the paper could be publishable. If the Depth baseline cannot be re-run, the authors should substantially weaken the comparative claims against iterative methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a simple, plausible idea that is almost certainly oversold in the abstract. The actual contribution is AbbIE-D, a Transformer whose middle block is reused with an extra residual connection, trained with two iterations, and the empirical result that it keeps improving ICL accuracy up to four times that training iteration count. That specific configuration is new relative to Geiping et al. and to earlier looped Transformers.\n\nThe paper does a lot right. The controlled setup is solid: same token budget, same architecture proportions, pre-norm, tied embeddings, and they explicitly compare against a recurrent-depth baseline. The FLOP-efficiency analysis is useful, and the limitation section is honest about the extra cost near the compute-optimal point. There is no circularity or fitted parameters; the path-independence discussion motivates the variant rather than deriving it.\n\nNow the soft spots. The abstract claims 'up to 12% improvement' in ICL and 'up to 5%' perplexity. I cannot reproduce the 12% from Table 2. The largest relative gains over Std on HellaSwag, LAMBADA, or ARC-E are roughly 7-8%. CommonsenseQA is excluded from percentage claims because the baselines sit at random. The 5% perplexity claim appears only as a rough statement in a figure caption, with no numbers in any table. That is a real mismatch between headline and data.\n\nThe bigger issue is the Depth baseline. The paper notes that Geiping et al. train over a range of sampled iteration counts, but the experimental section never says whether the Depth runs here received that protocol. Figure 6b shows Depth collapsing for any r != 2. That is exactly what you'd expect from a model trained at fixed r=2. If that is what they did, the comparison is not against the actual recurrent-depth method, and the claim that AbbIE 'far outperforms alternative iterative methods' is not established. The stress-test note is right on this point.\n\nSmaller but worth saying: the 350M upward-generalization result is from a single training run, with no error bars. Figure 4's fixed-point evidence uses 16 samples, and it measures different functionals for AbbIE-C and AbbIE-D, so the cross-variant convergence comparison is less clean than it looks.\n\nBottom line: directionally plausible, and the core variant is worth trying. But the central comparative claim rests on an under-specified baseline and an unreplicated large-model run. A serious referee could fix this; the paper needs corrected headline numbers, a perplexity table, seed-level variance, and an explicit statement of Depth's training iteration protocol. I'd send it out, but I would not trust the claims until those are addressed.","headline":"A plausible but over-claimed recipe for test-time compute scaling; the residual-injection variant is worth a look, but the headline numbers and the Depth baseline need fixing before the central comparison holds.","tokens_in":18662,"tokens_out":3705,"would_cite":false,"duration_ms":39810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AbbIE-D, a Transformer that reuses its middle block with an extra residual connection, achieves better perplexity than a standard Transformer at the same token budget and keeps improving on zero-shot reasoning tasks when given more…","keywords":["autoregressive block-based iterative encoder","recurrent transformer","test-time compute scaling","upward generalization","fixed-point convergence","latent space reasoning","zero-shot in-context learning","perplexity scaling"],"falsifier":"Train AbbIE-D at 200M and 350M with two iterations, five seeds each, then evaluate at $r=2$, 4, 8, and 32 on the four benchmarks plus held-out tasks; the upward-generalization claim is falsified if mean accuracy at $r=8$ does not exceed mean accuracy at $r=2$, or if inter-iteration distance stops decreasing at the larger iteration counts.","tokens_in":17596,"feed_emoji":"🔄","tokens_out":7233,"duration_ms":70810,"temperature":0.7,"pith_summary":"The paper introduces AbbIE, a recursive variant of the decoder-only Transformer that reuses a middle group of layers, called the Body, over several iterations in latent space. Its central claim is that the AbbIE-D variant, which adds one extra residual connection around the Body and trains with exactly two iterations, reaches lower perplexity than a standard Transformer at the same token budget and remains stable enough that more iterations at test time keep improving zero-shot in-context learning accuracy. This would matter because it gives model developers a way to spend more compute on hard inputs at inference without training on specialized data or changing the architecture's single-pass behavior. The authors frame AbbIE as a complement to scaling parameters and tokens, since it matches a standard Transformer when run once but can be dialed up when extra computation is available.","feed_headline":"A reused block beats baseline perplexity with two training iterations","feed_subtitle":"AbbIE-D reuses its middle layers with an extra residual and keeps improving at four times the trained iteration count.","key_machinery":"The load-bearing object is the recursive Body recurrence. For AbbIE-D it is $h_{k+1} = B(h_k) + h_k$, where $B$ is the stack of Transformer blocks in the Body and $h_0$ is the output of the Head; adding $h_k$ back on each pass strengthens the contribution of the original input relative to the cumulative attention and feed-forward updates. This extra residual makes the iterates converge toward a fixed point in concept space, the latent region between token embedding and unembedding, which is what the paper links to generalization beyond the training iteration count. A standard Transformer is the special case $r=1$, so AbbIE is a direct architectural generalization rather than a separate family.","core_discovery":"The discovery is that upward generalization in iteration count follows from a simple structural change: adding a residual connection around the entire reused Body stack. With training iterations fixed at r=2, the 350M AbbIE-D model's perplexity keeps decreasing up to r=4 and its accuracy on HellaSwag, LAMBADA, ARC-Easy, and CommonsenseQA keeps improving up to r=8, four times the training count; the 200M variant shows weaker effects, suggesting a size threshold. The same training budget that produces this behavior also yields roughly 5% better perplexity than a standard Transformer, up to 12% higher zero-shot accuracy on HellaSwag, and the only above-random CommonsenseQA scores among the compared models. AbbIE-C, which relies only on the Transformer's internal residual stream and omits the extra inter-iteration residual, does not converge and is excluded.","pith_inferences":["If the single-seed 350M result is representative, iteration count could become a per-input adaptive budget, with the model stopping once it reaches its fixed point; this would make inference cost proportional to input difficulty.","The decoupling between perplexity and in-context learning at high iteration counts suggests latent-space iteration may act as a form of implicit reasoning, and one could test this by probing whether intermediate iterates show increasingly structured semantic representations.","The same fixed-point principle could transfer to other autoregressive architectures, such as encoder-decoder or multimodal models, by placing the extra residual around the reused component; this is an untested extension.","Multi-seed and multi-size studies are needed to confirm that the upward-generalization threshold sits between 200M and 350M parameters rather than being a random artifact of one run."],"forward_implications":["At the same token budget, AbbIE-D beats a standard Transformer on perplexity by about 5% while remaining a drop-in replacement, since running it once matches standard perplexity.","Because performance continues to improve at $r=8$ after training at $r=2$, iteration count becomes a test-time compute dial that can be raised for harder inputs without retraining or specialized data.","The FLOP cost of AbbIE relative to a standard Transformer falls as training extends beyond the compute-optimal point, so in the long-training regime common in large-scale runs the overhead approaches that of a standard model.","Zero-shot in-context learning gains persist even after perplexity begins to worsen at higher iteration counts, indicating that the iteration procedure changes task-relevant representations and not just reduces loss.","Recurrent transformers that require many training iterations or random input injection are not necessary for upward generalization; two iterations and the inter-iteration residual suffice at this scale."],"supporting_citations":[{"why":"Supplies the path-independence and fixed-point criterion that connects iteration convergence to upward generalization.","marker":"Anil et al. (2022)"},{"why":"Provides the recurrent Depth baseline and the latent-reasoning approach that AbbIE is compared against.","marker":"Geiping et al. (2025)"},{"why":"Sets the compute-optimal token budget and the FLOP-versus-perplexity methodology used for efficiency analysis.","marker":"Hoffmann et al. (2022)"},{"why":"Defines the Transformer architecture of which AbbIE is presented as a recursive generalization.","marker":"Vaswani et al. (2017)"},{"why":"Motivates reasoning in continuous latent space and frames the contrast with latent reasoning models that need specialized datasets.","marker":"Hao et al. (2024)"},{"why":"Supplies the concept-space and detokenization view of Transformer latent states that motivates iterating in the Body.","marker":"Kaplan et al. (2024)"},{"why":"Represents looped and recurrent Transformers that typically need many iterations and so are not drop-in replacements.","marker":"Yang et al. (2024)"},{"why":"Provides the GPT-2-like architectural proportions used for all compared models.","marker":"Radford et al. (2019)"}],"fun_headline_variants":["Residual connection lets AbbIE train on 2 iterations, test on 8","AbbIE's extra residual unlocks test-time compute scaling","Two iterations train, eight work: AbbIE's residual trick scales compute","Train with 2 iterations, get 12% better zero-shot with AbbIE","AbbIE: trained on 2 iterations, generalizes to 8"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that fixed-point convergence measured on 16 random samples per benchmark and the results of a single 350M training run are enough to establish upward generalization as a stable property of the architecture.","fun_headline_variants_meta":{"raw":{"variants":["Residual connection lets AbbIE train on 2 iterations, test on 8","AbbIE's extra residual unlocks test-time compute scaling","Two iterations train, eight work: AbbIE's residual trick scales compute","Train with 2 iterations, get 12% better zero-shot with AbbIE","AbbIE: trained on 2 iterations, generalizes to 8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3477,"prompt_tokens":918,"completion_tokens":2559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2463}},"tokens_in":534,"tokens_out":2559,"duration_ms":18021,"temperature":1.0,"reasoning_tokens":2463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:16:19.518261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train AbbIE-D at 200M and 350M with two iterations, five seeds each, then evaluate at $r=2$, 4, 8, and 32 on the four benchmarks plus held-out tasks; the upward-generalization claim is falsified if mean accuracy at $r=8$ does not exceed mean accuracy at $r=2$, or if inter-iteration distance stops decreasing at the larger iteration counts.","supporting_citations":[],"review_version":1}