{"id":"8da52ae7-ccf9-4d31-bd31-25526a3eb0b1","arxiv_id":"2608.12756","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reconstruction-guided chunking creates adaptive latent tokens that beat random boundaries on native reconstruction, but downstream readers recover topic information while losing exact lexical details.","lead":"This paper introduces ReconSpan, a way to split text into chunks whose boundaries are chosen by how far a backward decoder can reconstruct them, and it keeps one neural code per chunk. The authors show the resulting latent sequences let language models capture topic but not exact details, exposing a gap between what an autoencoder retains and what other models can access.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's random baseline is not length-distribution-matched, and the selection signal is teacher-forced while the evaluation is free-run reconstruction; the headline advantage may be an artifact.","rationale":"The reader's verdict is CONDITIONAL, and my read keeps that verdict but for a more specific reason. The reader's weakest_assumption focuses on whether backward-decoder reach is a good proxy for downstream access. My concern is more basic: the paper's central positive result, Table 3, may not even establish the native-reconstruction claim cleanly. Algorithm 1 measures teacher-forced reach from a prefix code, while Section 4.4 scores free-run reconstruction of isolated chunk codes; no experiment links the two. Additionally, the random control matches mean chunk length but not the distribution of chunk lengths, although exact-token recovery depends on length. A length-distribution-matched random control would settle this. I do not think the method is false; I think the main evidence is currently confounded. This supports CONDITIONAL acceptance pending the control and the free-run comparison. The paper's independent strengths, including honest limitations, detailed compute accounting, and negative reader results, argue against rejection.","tokens_in":14806,"tokens_out":13048,"duration_ms":130885,"concrete_test":"Recompute Table 3 with two changes in one protocol: (1) select boundaries by free autoregressive reach (decode D(c_t) without teacher forcing, stop at the first argmax error) instead of the teacher-forced scan in Algorithm 1; and (2) use as the random control the same observed multiset of chunk lengths as the selected policy, placed at random document positions (e.g., permute the selected chunk lengths across documents), not merely the same chunk count. If the exact-token advantage over this length-distribution-matched random baseline shrinks or vanishes, the headline gap is not evidence for reconstruction-guided boundary placement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 selects boundaries by teacher-forcing the backward decoder: at each reverse step it is fed the true previous tokens and the stopping rule fires on the first (m-th) incorrect argmax. This measures conditional next-token prediction, not the length of a suffix the code can reconstruct from the code alone, even though Eq. (2) defines the autoencoder as D(c_t) = (x_t,...,x_1). Section 4.4/Table 3 then validates the selected chunks by encoding each chunk in isolation and freely decoding it (Appendix C). The paper never shows that teacher-forced reach predicts free-run reach; free decoding compounds errors, so the selection signal is at best an upper bound on reconstruction reach. The random control in Table 3 matches only per-document chunk count (hence mean length), not the chunk-length distribution induced by the stopping rule. Failure(1) produces 28.1% one-token chunks (Section 4.3); uniformly placed boundaries with the same mean have a different length distribution, and token-level exact recovery is strongly length-dependent. The 0.907 vs 0.618 exact-token gap may therefore be partly an artifact of the length distribution rather than of where boundaries fall. This comparison is the load-bearing support for the central claim that reconstruction fidelity is a viable allocation criterion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReconSpan proposes an adaptive latent tokenization method in which chunk boundaries are placed where a backward decoder, teacher-forced from a prefix code, begins to reconstruct tokens incorrectly. The prefix code at each boundary is retained as a contextual latent token. The paper contributes a new allocation criterion (backward reconstruction reach), shows that one trained autoencoder supports a range of post-training granularities via stopping rules, and presents an experimental characterization of the resulting latent sequences via native reconstruction and separately trained reader models. The headline empirical claim is that at matched average chunk length, reconstruction-guided boundaries preserve more text than random boundaries (Table 3: exact-token recovery 0.907 vs 0.618), and that downstream readers recover semantic topic information more readily than exact lexical details (Table 5). The paper is thorough in reporting limitations, including the short-chunk spike, the reader access gap, and compute costs.","tokens_in":14972,"tokens_out":6297,"duration_ms":67525,"significance":"If the central claim holds, the paper introduces a genuinely new, post-training-controllable signal for latent tokenization that does not require training a boundary predictor or an explicit length target, and it provides a clean way to trade granularity against reconstruction quality. The manuscript is honest and unusually detailed on limitations and compute accounting, and it reports negative results (the code-to-code reader) as well as positive ones, which is a strength. However, the empirical support for the central claim is currently weakened by the design of the random control and by the mismatch between the teacher-forced selection signal and the free-run evaluation metric. These are fixable within the scope of the paper, so the significance of the contribution is real but the evidence needs strengthening.","major_comments":[{"comment":"The random control matches only the per-document chunk count, not the chunk-length distribution. Failure(1) produces 28.1% one-token chunks (Section 6.1), and one-token chunks are far easier to reconstruct exactly than longer chunks. The headline gap in exact-token recovery (0.907 vs 0.618) may therefore be a consequence of the length distribution rather than of where boundaries fall. This concern is sharpened by the fact that the exact-chunk rates are nearly identical (0.797 vs 0.794), so the token-level advantage is driven by which lengths are exact. Please add a random baseline that matches the empirical chunk-length distribution (for example, by randomly permuting the actual chunk boundary positions, or by sampling chunk lengths from the Failure(1) distribution and then placing boundaries accordingly) and report the token-level metrics under that control.","section":"Section 4.4, Table 3"},{"comment":"Boundary selection in Algorithm 1 is teacher-forced: the backward decoder is fed the true previous tokens at each reverse step. The evaluation in Table 3, by contrast, uses free-run reconstruction of each chunk encoded in isolation (Appendix C). The paper never shows that teacher-forced reach predicts free-run reach; free decoding compounds errors, so the selection signal could be exploiting autoencoder idiosyncrasies under teacher forcing rather than genuine reconstruction capacity. Please provide a direct test, for example by measuring the correlation between teacher-forced reach and free-run reach on a sample of positions, or by running the boundary selection with an approximate free-run or sampling-based reach and comparing the resulting Table 3 metrics. The manuscript itself acknowledges this distinction in Appendix C, but it does not close the gap.","section":"Section 3.2, Appendix C, Section 4.4"}],"minor_comments":[{"comment":"The phrase \"length-matched random control\" is misleading because only the mean chunk length (via per-document chunk count) is matched, not the length distribution. The Table 3 caption is accurate; please make the main text consistent.","section":"Section 4.4"},{"comment":"Please report the number of documents and chunks, and provide confidence intervals or a significance test for the exact-token difference against the random baseline, since the claim is load-bearing.","section":"Table 3"},{"comment":"The statement \"O(1 + n)\" sequential model invocations is confusing; it is equivalent to O(n) for n>1. Please simplify or clarify what the constant term refers to.","section":"Section 3.2"},{"comment":"The superscript stars mark significance at 95% confidence, but the caption does not define how multiple comparisons are handled. Given the number of rows, please state whether any correction was applied.","section":"Table 5"},{"comment":"The statement \"Code and checkpoints are not publicly released\" limits reproducibility; at minimum, please provide pseudocode or a reference implementation of the random-control procedure so reviewers can verify the distribution-matching issue.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and transparent, but the central empirical claim currently rests on a random control that is confounded by chunk-length distribution. This is a straightforward fix (a distribution-matched random baseline) and should be feasible within the paper's scope. The second issue, teacher-forced selection vs free-run evaluation, is also addressable with a correlation analysis. I would encourage the editor to seek a revised version that adds these controls before reconsidering the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is the allocation rule: place chunk boundaries where a backward decoder can no longer reconstruct the prefix, and keep the code at that boundary. That's distinct from BLT's forward entropy, SemToken's semantic similarity, or learned routers. The paper is also honest about what its latent codes support: readers get topic but not exact details, which is a useful negative result for the field. The write-up is unusually transparent about compute, architecture choices, and the limitations section is candid.\n\nThe soft spots are mostly in the load-bearing comparison of Table 3. The random baseline matches Failure(1)'s per-document chunk count, hence mean length, but not the chunk-length distribution. Failure(1) makes 28.1% of chunks single tokens, which are trivially exact; random uniform placement won't produce such a spike. So the huge gap in exact-token recovery (0.907 vs 0.618) may be an artifact of where the short chunks go rather than of boundary placement generally. The paper concludes that 'where the boundaries fall, not merely how many there are, decides how much text survives,' but this control doesn't demonstrate that. You need to match the length distribution (e.g., sample random boundaries with the same empirical length distribution) to separate placement from length.\n\nThere's also a teacher-forcing mismatch: boundaries are selected by teacher-forced reach, but evaluated by free-run roundtrip. Free-run compounds errors, so the selection signal is at best an upper bound on reconstruction reach. The paper never checks that the two correlate. That's not fatal if you reframe the claim as 'teacher-forced reach is a useful heuristic,' but the headline result needs that caveat.\n\nMinor issues: no error bars on Tables 1, 3, and 4; no comparison against another adaptive tokenizer (BLT or SemToken) on a common protocol; code and checkpoints are not released, which hurts reproducibility. The limitations section acknowledges the missing comparisons and the entry cost, but not the length-distribution confound.\n\nOverall: the idea is worth taking seriously, and the paper is a solid empirical study of it. But the central claim is not yet proven. I'd send it to a serious referee, with the request to add a length-distribution-matched control and an external evaluation metric, and to release code. Researchers working on latent tokenization and context compression will get value from the framing and the negative reader results, even if the headline numbers need re-examination.","headline":"New allocation rule is real, but the headline comparison against random boundaries is confounded by chunk-length distribution; the paper deserves a serious revision.","tokens_in":15581,"tokens_out":4660,"would_cite":false,"duration_ms":41074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReconSpan chunks text at points where a backward decoder can no longer reconstruct it, preserving more text per latent token than random boundaries.","keywords":["adaptive latent tokenization","reconstruction-guided chunking","backward decoder","contextual prefix codes","latent tokens","tokenizer granularity","direct readout","autoencoder reconstruction"],"falsifier":"Run the same chunking algorithm with a different backward decoder, or with a forward decoder given an oracle starting position, and compare reconstruction-guided boundaries against matched random boundaries using the protocol of Table 3; if the advantage disappears or reverses, backward reconstruction reach is not a general allocation signal.","tokens_in":14508,"feed_emoji":"🧩","tokens_out":7745,"duration_ms":71652,"temperature":0.7,"pith_summary":"The paper proposes treating reconstruction fidelity as the criterion for adaptive latent tokenization: instead of fixed subword units or entropy spikes, text is chunked wherever a backward decoder, starting from a single contextual prefix code, stops being able to reconstruct the recent tokens. Each retained boundary code becomes a latent token, so difficult spans get shorter chunks and easy spans get longer ones. On held-out text, this allocation preserves more content than length-matched random boundaries: at mean chunk length 6.50, exact-token recovery is 0.907 versus 0.618. The same trained autoencoder can be re-granularized after training by loosening the stopping rule, reaching mean chunk lengths from 6.5 to 12.2 input tokens. A reader trained on the latent sequence recovers topic information well but exact details only weakly, so the paper separates what the autoencoder retains from what a downstream model can access.","feed_headline":"Backward decoder reach picks chunks that preserve more text","feed_subtitle":"Matched-length reconstruction-guided splits beat random ones; one model dials granularity after training.","key_machinery":"The load-bearing object is backward-decoder reach: the longest contiguous run of most-recent tokens that a single prefix code can exactly reproduce when decoded in reverse under teacher forcing. The chunking algorithm computes all prefix codes in one encoder pass, repeatedly teacher-forces the decoder from the current endpoint, and cuts a chunk at the first position where the stopping rule fires—Failure($m$), the $m$-th wrong token, or Logit-gap($\\tau$), an accumulated margin. The code at the boundary is the latent token, so chunk length is a direct measurement of how far one code reaches back; this is what makes allocation adaptive and post-training-controllable.","core_discovery":"ReconSpan's central claim is that a chunk boundary is best placed where backward reconstruction stops being reliable. A causal encoder produces a code $c_t = E(x_{1:t})$ at every position; a backward decoder teacher-forced against the true text reconstructs newest-first from the final code until a stopping rule fires, and the longest successfully reconstructed suffix becomes a chunk with $c_t$ as its latent token. The paper reports that this capacity-based boundary rule beats a random rule matched in chunk count and mean length: 0.907 exact-token recovery versus 0.618, with better suffix recovery and lower conditional perplexity. Because the stopping rule is applied only at chunking time, one trained model supplies mean chunk lengths from 6.50 to 12.17 by choosing Failure($m$) or Logit-gap($\\tau$). It also claims that the resulting latent sequences are readable by separately trained language models for coarse semantic information, with task adaptation improving exact-detail access, and that reconstruction fidelity is therefore a viable allocation criterion for adaptive latent tokenization.","pith_inferences":["If backward reconstruction reach tracks local information density, ReconSpan boundaries could serve as an unsupervised proxy for difficulty; a direct test would correlate chunk lengths with surprisal from an independent language model.","The paper leaves open whether boundaries align with syntax or discourse; a comparison of boundary placement against constituent and clause boundaries would settle that question.","The native roundtrip's strong exact-detail performance suggests a better-trained reader, not a different tokenizer, may close the direct-access gap; a reader trained with retrieval-style objectives is a natural next test.","If boundary-selection cost is amortized, the same latent sequence could double as a context-compression interface, trading prefill and KV-cache memory for a one-time reconstruction scan."],"forward_implications":["The same trained autoencoder can produce several granularities, with mean chunk lengths from 6.5 to 12.2 input tokens, by relaxing the stopping rule with no retraining.","Boundary placement, not just boundary count, determines how much text survives: reconstruction-guided boundaries outperform random boundaries matched in count and mean length on exact-token and suffix recovery.","Downstream language models can consume the contextual latent sequence directly; they match raw-text-level topic classification on AG News but lag on exact-detail tasks such as LAMBADA.","Task adaptation on top of generic readout narrows the gap between direct readout and native roundtrip, showing the latent codes carry task-relevant information a generic reader leaves unused.","Reconstruction fidelity is proposed as a viable allocation criterion even though the selected boundaries are not claimed to align with linguistic units."],"supporting_citations":[{"why":"Supplies the Pythia-410M Transformer whose final hidden state serves as the prefix code.","marker":"[Biderman et al., 2023]"},{"why":"Supplies the Mamba2-130M backward decoder whose reconstruction reach sets chunk boundaries.","marker":"[Dao and Gu, 2024]"},{"why":"SONAR is the external single-code autoencoder baseline for reconstruction quality.","marker":"[Duquenne et al., 2023]"},{"why":"The Byte Latent Transformer is the entropy-based variable-patch allocation method ReconSpan is contrasted with.","marker":"[Pagnoni et al., 2025]"},{"why":"MEGABYTE provides the fixed-size patch allocation baseline that motivates adaptive chunking.","marker":"[Yu et al., 2023]"},{"why":"WikiText is the held-out corpus on which native reconstruction under the selected boundaries is measured.","marker":"[Merity et al., 2017]"},{"why":"LAMBADA supplies the exact-lexical readout task showing the gap between retained and accessible information.","marker":"[Paperno et al., 2016]"},{"why":"AG News supplies the coarse-semantic topic task where direct readout approaches raw-text performance.","marker":"[Zhang et al., 2015]"}],"fun_headline_variants":["Reconstruction-guided chunks beat random in text recovery","One autoencoder adapts chunk length via reconstruction","Chunk by backward reach to preserve more text","Reconstruction fidelity picks text-preserving chunks","Backward reconstruction defines optimal chunk boundaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the premise that a backward decoder's reconstruction reach is a good proxy for what a latent token should preserve for downstream models; if chunking merely optimizes for that decoder's own roundtrip behavior, the allocation advantage may not transfer to other readers.","fun_headline_variants_meta":{"raw":{"variants":["Reconstruction-guided chunks beat random in text recovery","One autoencoder adapts chunk length via reconstruction","Chunk by backward reach to preserve more text","Reconstruction fidelity picks text-preserving chunks","Backward reconstruction defines optimal chunk boundaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3393,"prompt_tokens":845,"completion_tokens":2548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2480}},"tokens_in":461,"tokens_out":2548,"duration_ms":19337,"temperature":1.0,"reasoning_tokens":2480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:10.127211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same chunking algorithm with a different backward decoder, or with a forward decoder given an oracle starting position, and compare reconstruction-guided boundaries against matched random boundaries using the protocol of Table 3; if the advantage disappears or reverses, backward reconstruction reach is not a general allocation signal.","supporting_citations":[{"cited_title":"Transformers are","cited_arxiv_id":null,"evidence_quote":"Supplies the Mamba2-130M backward decoder whose reconstruction reach sets chunk boundaries."}],"review_version":1}