{"id":"4d7d2541-4981-4284-b148-97acf8bb47a2","arxiv_id":"2608.13141","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MergeOver integrates ToMe-style token merging into the recursive SReT transformer post-training, cutting GPU activation memory by about 38% and improving batched throughput while losing 1.47 points of ImageNet top-1 accuracy.","lead":"MergeOver adds token merging to a recursively weight-shared vision transformer without retraining, using an unmerge stack to preserve the architecture's grid structure. It reports modest accuracy loss and platform-dependent speed and memory gains, with the clearest wins at batch size 16.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unmerge duplication feeding SReT's frozen conv pooling is unvalidated: no ablation separates its distortion from merging loss.","rationale":"The reader's weakest_assumption correctly identifies the unmerge-to-pooling interface as the least secure link in the chain. I agree: the paper's own text (Section 3.1) admits that Unmerge duplicates features rather than reversing aggregation, so the conv pooling is fed an approximation of the original grid. The reported accuracy and hardware numbers are consistent with the central feasibility claim, and the paper is transparent about scheduling, CPU-memory measurement, and FLOPs limitations (Section 6). However, the end-to-end result cannot separate the cost of genuine token reduction from the cost of the reconstruction artifact. A missing ablation is the specific gap: no alternative reconstruction is compared, and no distributional check at the pooling input is provided. A secondary, related omission is the treatment of the token-mass tensor through Unmerge and conv pooling; if mass is not updated consistently, the proportional-attention claim in later stages is not fully specified. I do not see an internal numerical inconsistency or a reason to reject; the requested ablation and code inspection would turn CONDITIONAL into a stronger verdict. The paper deserves credit for releasing code and for the honest limitation list, but the central integration mechanism needs this one validation.","tokens_in":19503,"tokens_out":14042,"duration_ms":130216,"concrete_test":"Run an ImageNet-1K validation ablation at rho_shot=0.25 that keeps MergeOver unchanged except for the unmerge reconstruction. Replace the ToMe duplication with (a) zero-filling the vacated source positions and (b) filling them with the mean of the two nearest surviving tokens before the frozen SReT conv pooling. Compare top-1 accuracy and the activation statistics (per-channel mean/variance) at the pooling input against the reported 75.92. If variant (a) or (b) changes accuracy by more than about 1pp, or if the pooling-input statistics shift substantially, the specific duplication scheme is load-bearing and the feasibility claim needs a dedicated validation; if all three reconstructions are within noise, the concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central feasibility claim rests on ToMe's Unmerge operation being consumable by SReT's pretrained convolutional pooling without retraining (Section 3.1, Figures 1 and 3). The Unmerge restores the exact token count by duplicating merged features onto the vacated spatial coordinates, so the tensor entering the inter-stage conv pooling is not the original spatial signal: multiple positions in a pooling receptive field can carry identical vectors. The pretrained conv weights were optimized for spatially distinct activations; duplication changes the local statistics and can double-count features within a 3x3/stride-2 window. The paper reports only end-to-end top-1 accuracy (75.92 vs 77.39, Table 1), which conflates the inherent information loss of merging with any distortion introduced by the reconstruction. No ablation isolates the Unmerge reconstruction: there is no comparison against alternative length-restoration schemes (e.g., interpolation, nearest-neighbor fill, zero-fill), no measurement of the distribution shift at the pooling input, and no test on a second recursive or hierarchical model. Section 6 acknowledges the restriction to SReT-Tiny-Distill, but not this specific gap. If the frozen pooling is sensitive to the duplication artifact, the 1.47pp drop is not a reliable estimate of the cost of token merging in recursive ViTs, and the claimed 'solution' to the spatial-layout constraint (Section 3.1) is only a shape-level fix.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MergeOver, a post-training method that integrates Token Merging (ToMe) into the recursively weight-shared Sliced Recursive Transformer (SReT). To overcome architectural incompatibilities, the method introduces an Unmerge tracking stack that restores the 2D spatial grid before inter-stage convolutional pooling, a constraint-safe merge-rate adjustment that respects the group-divisibility bounds of Sliced Group Self-Attention (Equation 3), and parallel token-mass tracking through SReT's spatial permutations. The authors benchmark several token-reduction schedules on ImageNet-1K and select a stage-wise single-shot configuration (rho_shot = 0.25) that loses 1.47 percentage points of top-1 accuracy while reducing GPU peak activation memory by 37.3% and 38.4% at batch sizes 1 and 16, with GPU throughput increasing by 21.7% at batch size 16. On x86 and ARM CPUs, latency reductions are reported at batch size 16, with only modest or negative effects at batch size 1. The paper includes detailed tables across batch sizes and platforms and makes the source code publicly available.","tokens_in":19751,"tokens_out":6824,"duration_ms":55241,"significance":"If the results hold, MergeOver is a useful demonstration that token merging can be applied post-training to a hierarchical recursive transformer, recovering part of the throughput and memory costs introduced by recursive weight sharing. The core constraint derivation (Equation 3) is simple and correct, and the hardware evaluation is reported transparently across three platforms with an honest discussion of measurement limitations (e.g., FLOPs are not predictive, CPU RSS is not equivalent to activation memory). The authors also explicitly acknowledge that the evaluation is limited to one model and that the schedule comparisons are not matched in total reduction budget. The public release of the source code is a clear strength. The main outstanding risks are the lack of validation of the Unmerge reconstruction and the absence of variance information for the hardware metrics.","major_comments":[{"comment":"The Unmerge stack restores the exact token count by duplicating merged features back onto their original spatial coordinates, but the paper does not validate that SReT's frozen convolutional pooling layers can safely consume this reconstructed signal. Because multiple positions within a pooling receptive field can carry identical vectors, the input distribution to the pretrained convolutions is shifted relative to the spatially distinct activations seen during training. The paper reports only end-to-end accuracy, so the 1.47 percentage-point drop in Table 1 conflates the inherent information loss of token merging with any distortion introduced by the duplication. Please add an ablation that compares the Unmerge reconstruction against alternative length-restoration schemes (e.g., interpolation, nearest-neighbor fill, zero-fill, or truncation to a compatible length), and ideally measure the distribution shift at the pooling input. Without such an analysis, the claimed solution to the spatial-layout constraint is validated only at the shape level, not at the signal level.","section":"Section 3.1, Figures 1 and 3"},{"comment":"All hardware metrics (latency, throughput, PAM, and DeltaRSS) are reported as single median values with no measure of run-to-run variability, such as standard deviation, confidence interval, or range over repeated runs. Several reported effects are small, for example the 2.4% ARM latency reduction at batch size 1 in Table 4, and without error bars it is impossible to judge whether such differences are meaningful or within measurement noise. Given that hardware efficiency is a central claim of the paper, please report variance across repeated trials for at least the headline configurations and batch sizes.","section":"Section 4.2 and Tables 2-7"},{"comment":"The comparison between global and stage-wise schedules is confounded because the schedules differ not only in shape but also in parameterization, depth indexing, and total token-reduction budget. The paper acknowledges this in Section 6, but since 'an analysis of token reduction scheduling strategies' is listed as a contribution, the conclusion that the stage-wise single-shot schedule is preferable would be much stronger with a matched-budget comparison. Please either tune the global schedules so that the total number of removed tokens matches the stage-wise configurations or explicitly restrict the scheduling claim to the particular formulations evaluated, rather than suggesting a general advantage of stage-wise single-shot reduction.","section":"Section 5.1 and Figure 5"}],"minor_comments":[{"comment":"The figure is visually ambiguous: the stage-map sequence lengths (N=584, N=144, N=37) appear to show the post-merge lengths, while the text and Figure 3 emphasize that the original length is restored before pooling. Labeling both the post-merge and post-unmerge lengths at each stage boundary would improve clarity.","section":"Figure 1"},{"comment":"The table contains formatting artifacts, such as '70.95-6.44\"' and '40.97-36.42\"'; these should be typeset consistently as '70.95 (-6.44)' and '40.97 (-36.42)'.","section":"Table 1"},{"comment":"The 'accuracy-preserving criterion' of a maximum 2-percentage-point top-1 accuracy loss is introduced without any justification. A brief rationale for the threshold, or a sensitivity analysis around it, would help the reader interpret the selection of rho_shot = 0.25 as the headline configuration.","section":"Section 5.1"},{"comment":"The limitations paragraph correctly notes that the evaluation is restricted to SReT-Tiny-Distill and that generality cannot be assumed. Given the title's reference to 'Recursive Vision Transformers', it would be helpful to state more explicitly that the empirical demonstration is a single case study, with generality left as future work.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems-integration paper with transparent benchmarking and an honest limitations section, but the novelty is incremental: it applies the existing ToMe algorithm to one recursive transformer with three integration constraints solved. The two main risks are (1) the unvalidated Unmerge reconstruction, which is load-bearing for the spatial-layout claim, and (2) the lack of any variance estimate for the hardware measurements, which weakens the headline efficiency numbers. The scheduling comparison is also confounded, though the authors already acknowledge this. I recommend major revision rather than rejection because the central idea is sound and the identified issues appear addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key things to know: MergeOver is the first post-training integration of ToMe-style token merging into SReT's recursive weight-shared architecture, and it ships as a working implementation with extensive cross-platform measurements. The constraint derivation (Eq. 3) is correct, the stage-wise single-shot schedule is a sensible contribution, and the paper is honest about its own scope. The reported numbers are internally consistent across abstract, tables, and appendix.\n\nWhere it earns credit: the Unmerge stack is a simple, concrete mechanism for restoring a square spatial grid before SReT's conv pooling, and the mass-token tracking across permutations is a real engineering detail. The evaluation covers GPU, x86, and ARM, with batch sizes up to 128, and the paper clearly separates FLOPs estimates from measured hardware performance. The code is publicly available.\n\nSoft spots: the biggest is the one the stress-test note flags—the Unmerge operation duplicates merged features back onto vacated coordinates, and the paper never isolates how much of the 1.47pp accuracy drop comes from that duplication artifact versus inherent merging loss. That is a genuine missing ablation. It is not evidence the result is wrong; the end-to-end accuracy is decent, and on its face the integration works. But without a comparison against a cheap alternative restoration (interpolation, zero-fill), you can't tell if the conv pooling is silently absorbing a distortion that might be larger in other recursive or hierarchical models. This is fixable and should be requested in review.\n\nOther minor issues: no error bars on hardware measurements, the headline configuration is selected using the same validation set used for reporting, and the conclusion's 'negligible' for 1.47pp is not supported by any task-level tolerance. The paper itself acknowledges the single-model restriction and the RSS memory caveats.\n\nBottom line: this is a solid engineering contribution for people working on recursive transformers or post-training compression. It deserves a serious referee—the missing ablation is a real gap but not a fatal one. I'd send it out with a request for an unmerge-isolation experiment and a second model check.","headline":"A competent engineering paper that makes a post-training ToMe/SReT integration work and reports internally consistent numbers, but the missing unmerge isolation ablation and selection details keep it from a clean yes.","tokens_in":20307,"tokens_out":2484,"would_cite":false,"duration_ms":22817,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MergeOver shows that token merging can be added after training to a recursively weight-shared vision transformer, cutting GPU peak activation memory by about 38% on ImageNet-1K with a 1.47-point top-1 accuracy drop.","keywords":["vision transformers","token merging","recursive weight-sharing","post-training compression","hierarchical vision transformer","ImageNet-1K","peak activation memory","edge AI"],"falsifier":"Run an ablation that substitutes exact token reinsertion for ToMe's duplication-based unmerge at the same merge rate and compare ImageNet top-1 accuracy and the cosine similarity of pooled activations; a material accuracy gain or a large divergence in pooled activations would show that the duplication approximation, and not merging itself, is what the frozen pooling layers must absorb.","tokens_in":19270,"feed_emoji":"⚡","tokens_out":6883,"duration_ms":55044,"temperature":0.7,"pith_summary":"The paper tries to show that token merging, a post-training compression trick that fuses similar image patches, can be grafted onto a recursively weight-shared vision transformer without retraining. It does this for the Sliced Recursive Transformer (SReT), whose repeated passes through shared blocks make it cheap in parameters but expensive in attention and activation memory. MergeOver restores the 2D grid before each hierarchical pooling layer, respects group-divisibility constraints inside grouped attention, and keeps token-mass bookkeeping aligned through spatial permutations. On ImageNet-1K the chosen single-shot configuration drops top-1 accuracy by 1.47 percentage points while cutting GPU peak activation memory by 37.3% at batch size 1 and 38.4% at batch size 16, and raising batch-16 GPU throughput by 21.7%. A sympathetic reader would take this as evidence that recursive and token-reduced transformers can be combined post hoc, opening an axis of compression for edge deployment.","feed_headline":"Merging tokens after training cuts a recursive ViT's memory by 38%","feed_subtitle":"Cuts a recursive transformer's activation memory and speeds batch inference with just a 1.47-point accuracy drop.","key_machinery":"The load-bearing object is the Unmerge tracking stack: every ToMe merge pushes a stored inverse operation that later duplicates the merged features back to their original spatial coordinates, restoring the exact sequence length the next convolutional pooling layer expects. Around it sit two bookkeeping devices: a constraint-safe merge-rate adjustment that selects the largest reduction satisfying $(N-r)mod g=0$, $N-r\\ge g$ with $g=\\mathrm{LCM}(g_1,g_2)$, and $r\\le \\lfloor N/2\\rfloor$; and a synchronized token-mass tensor that runs through the same permutation and inverse-permutation as the features so proportional attention stays correct. The stage-wise single-shot schedule applies reduction only at the first block of each stage, holding sequence length fixed through the remaining recursive iterations.","core_discovery":"The central claim is that recursive weight sharing and dynamic token merging are compatible without retraining, provided three structural frictions are removed: merged sequences must be rebuilt to a square 2D layout before convolutional pooling, the number of merged tokens must keep sequence length divisible by the least common multiple of the grouped-attention group sizes, and the per-token mass vector that ToMe uses for proportional attention must be permuted in lockstep with the features. MergeOver resolves these with an Unmerge stack, a constraint-safe merge-rate adjustment, and synchronized mass tracking, then applies reduction once per stage at the first block. On SReT-Tiny-Distill this yields a 1.47-point top-1 accuracy decrease on ImageNet-1K, a 37.3% and 38.4% GPU peak-activation-memory reduction at batch sizes 1 and 16, and a 21.7% GPU throughput increase at batch size 16, with latency reductions on x86 and ARM CPUs at batch size 16.","pith_inferences":["Beyond the paper, the unmerge step is the natural pressure point: since it duplicates merged features instead of restoring originals, replacing it with exact token reinsertion would separate reconstruction error from merging error and likely raise accuracy at a small memory cost.","Because MergeOver is post hoc and leaves weights untouched, it should stack with post-training quantization and kernel-level attention tiling, so the reported memory and latency gains are a lower bound for a full edge pipeline.","The same grid-restoration and divisibility machinery should transfer to non-recursive pyramid transformers whose inter-stage pooling imposes the same square-grid constraint, even though the paper only demonstrates the recursive case.","A testable consequence of the batch-size crossover is that larger sequence lengths amortize the matching and unmerge overhead better, so larger recursive models or higher-resolution inputs should show larger net speedups than the tiny model measured here."],"forward_implications":["Token merging no longer requires retraining or architectural redesign to enter a recursive hierarchical transformer; the post-hoc route is enough for a meaningful memory saving.","Peak activation memory is determined mainly by the initial reduction fraction of each stage, because the largest activations sit near the start of the stage.","On the GPU the method crosses from overhead to speedup as batch size grows: 21.7% slower at batch size 1, 21.7% faster at batch size 16.","FLOPs do not predict hardware throughput for this method, because matching, mass tracking, and unmerge operations fall outside the arithmetic estimate."],"supporting_citations":[{"why":"Supplies the recursive hierarchical baseline architecture (SReT) that MergeOver compresses.","marker":"[7]"},{"why":"Provides the BSM merging mechanism, the unmerge operation, and the token-mass proportional attention that MergeOver adapts.","marker":"[10]"},{"why":"Establishes the convolutional spatial-pyramid blueprint with inter-stage pooling that creates the square-grid constraint.","marker":"[20]"},{"why":"Provides the ImageNet-1K dataset used for the accuracy benchmark.","marker":"[17]"},{"why":"Provides the pretrained SReT-Tiny-Distill weights used as the uncompressed baseline.","marker":"[25]"}],"fun_headline_variants":["Post-training token merge cuts recursive ViT memory by 38%","Token merging after training boosts recursive transformer throughput","MergeOver: No-retrain token merge for recursive ViTs","Recursive ViT gets 38% memory cut via post-training merging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ToMe's unmerge operation, which restores the sequence length by duplicating merged features back into their original spatial positions, gives SReT's frozen convolutional pooling layers a spatial layout they can consume without retraining; if those duplicated features distort local geometry beyond what the frozen weights tolerate, the accuracy result collapses.","fun_headline_variants_meta":{"raw":{"variants":["Post-training token merge cuts recursive ViT memory by 38%","Token merging after training boosts recursive transformer throughput","MergeOver: No-retrain token merge for recursive ViTs","Recursive ViT gets 38% memory cut via post-training merging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2847,"prompt_tokens":1060,"completion_tokens":1787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":1717}},"tokens_in":676,"tokens_out":1787,"duration_ms":11273,"temperature":1.0,"reasoning_tokens":1717,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:50:03.673205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an ablation that substitutes exact token reinsertion for ToMe's duplication-based unmerge at the same merge rate and compare ImageNet top-1 accuracy and the cosine similarity of pooled activations; a material accuracy gain or a large divergence in pooled activations would show that the duplication approximation, and not merging itself, is what the frozen pooling layers must absorb.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pretrained SReT-Tiny-Distill weights used as the uncompressed baseline."}],"review_version":1}