Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Continuous chain-of-thought models reason through compressed latent vectors; this paper adds a fast direct-training method, C-MTP, and shows both training regimes collapse to about a third of standard accuracy on realistic long traces.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:22 UTC pith:EV3QCCXY

load-bearing objection C-MTP is a clean, useful direct-supervision method, but the paper's headline claim that continuous CoT collapses on realistic traces rests on self-generated low-quality MATH traces, not the 'strong teacher' the intro promises. the 3 major comments →

arxiv 2607.16972 v1 pith:EV3QCCXY submitted 2026-07-18 cs.AI cs.CL

Training Continuous Chain of Thought Models: A Tale of Two Regimes

Varun Yerram , He He , Eunsol Choi This is my paper
classification cs.AI cs.CL
keywords continuous chain-of-thoughtlatent reasoningmulti-token predictiondirect supervisionindirect supervisionreasoning compressionerror accumulationmath reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether continuous chain-of-thought models — language models that reason through short sequences of dense latent vectors instead of words — hold up on realistic reasoning problems. It introduces C-MTP, a direct-supervision method that compresses spans of reasoning tokens by averaging their embeddings and trains the model to predict exactly the tokens in each span, making training parallel and cheap. On short structured traces, C-MTP matches slower indirect-supervision methods, trains twice as fast, and generalizes better to unseen problem structures. On verbose semi-natural traces, indirect supervision wins because its fixed latent budget compresses filler text. On realistic MATH traces of a few hundred tokens, both regimes collapse to roughly a third of standard chain-of-thought accuracy, leading the paper to conclude that structured and semi-natural benchmarks overstate the practical utility of continuous CoT methods.

Core claim

On the paper's own terms, the contribution is twofold. First, C-MTP — Continuous chain-of-thought via Multi-Token Prediction — defines each latent as the simple average of the embeddings of a span of reasoning tokens, and trains the model to predict exactly the tokens in each span, keeping the objective aligned with pretraining and enabling fully parallel teacher-forced training. Second, comparing this direct-supervision regime against indirect, autoregressive-distillation regimes across three trace difficulties, the paper finds that direct supervision wins on short structured traces (faster, more sample-efficient, better out-of-distribution), indirect supervision wins on verbose semi-natura

What carries the argument

The load-bearing object is the latent z_i, defined as the mean embedding of a span of s consecutive reasoning tokens, with a multi-token prediction head that decodes the s tokens of each span in parallel from a single hidden state. This design collapses the problem of "predict the next latent" into "predict the next s tokens," so training runs one parallel forward pass per trace (teacher forcing), and inference samples s tokens, averages their embeddings to form the next latent, and feeds it back. The compression factor s is the knob that trades latent budget against fidelity: small s preserves arithmetic detail; large s shortens reasoning but amplifies error accumulation at span boundaries.

Load-bearing premise

The headline collapse on realistic traces rests on one representative method per regime (Section 4.5 uses C-MTP for direct and CODI for indirect), trained only on correct model-generated MATH traces at 1–1.5B scale with LoRA; the Limitations section concedes the model-size and math-only scope, and Appendix H notes the reproduced CODI numbers differ from an earlier baseline report — so the result could be specific to these methods, this data, and this scale.

What would settle it

Run the paper's realistic-trace comparison with a fixed-budget direct method (e.g., Gaussian-latent compression with a small, fixed number of steps independent of trace length) at 7B+ scale on MATH: if it holds above roughly 50% of CoT-SFT accuracy, the collapse is a property of the specific methods tested, not of continuous CoT per se. A cheaper check inside the paper's own setup: add self-correction at span boundaries to C-MTP on MATH and see whether accuracy rises above half of CoT-SFT.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the results hold, current continuous-CoT training — whether direct or indirect — does not yet substitute for explicit chain-of-thought on long, realistic reasoning, so the compute savings they promise are not available on such tasks today.
  • On short structured traces (under 100 tokens), direct supervision is a viable drop-in: comparable accuracy to slower indirect methods, roughly 2x faster per training step, and better transfer to held-out problem structures.
  • Direct supervision without teacher distillation works nearly as well (ablations leave accuracy largely unchanged at compression factors 2–3), implying the method's gains come from its pretraining-aligned objective rather than from the teacher.
  • Error cascades are concentrated in precise tokens — arithmetic expressions — rather than natural language; CommonsenseQA results show degradation stays mild as span length grows, while math errors compound.
  • The paper's own conclusion is prescriptive: scaling continuous CoT to realistic settings likely needs hybrid models that adaptively switch between token-based and latent reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves unrun: a re-sampling or self-correction loop at span boundaries during C-MTP inference on MATH — its own error catalog ("3*3=99", "16/3=3") suggests many failures are single-span corruptions that re-decoding alone might fix.
  • The structural-redundancy analysis (over 50% of GSM8k-Aug collapses to 100 templates) implies in-distribution wins on such benchmarks are partly memorization of templates; continuous-CoT evaluation on structurally diverse corpora would likely show smaller apparent gains.
  • Because the collapse is measured at 1–1.5B scale with LoRA on a single dataset, the "two regimes fail" story may not transfer: larger models with stronger arithmetic priors could absorb span-averaged latents better, so the 65% figure should be read as regime-plus-scale, not a universal ceiling.
  • Read constructively, the two regimes fail in complementary ways — direct methods by error accumulation, indirect by losing task-relevant detail in fixed latent budgets — which is exactly the case for switched or gated hybrids; the paper gestures at this, and the evidence here maps out where each regime breaks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes C-MTP, a direct-supervision continuous-CoT method that partitions a reasoning trace into spans of s tokens, defines each latent as the average embedding of the span, and trains the model with a multi-token-prediction objective to predict the span tokens and the final answer. It compares C-MTP against CoLaR (also direct) and against CODI, Sim-CoT, and Coconut (indirect) on three trace regimes: structured GSM8k-Aug, semi-natural GSM8k-Aug-NL, and realistic MATH traces. On structured traces C-MTP is competitive with or better than the indirect methods while being faster and more sample-efficient; on semi-natural traces indirect supervision outperforms direct methods; and on MATH both a representative direct method (C-MTP-2) and a representative indirect method (CODI with K=6,10,20) collapse to roughly 30–35% of CoT-SFT accuracy. The paper concludes that structured and semi-natural evaluations currently overstate the practical effectiveness of continuous-CoT methods.

Significance. If the realistic-trace result is correct, it is an important negative result for the continuous-CoT research program: it would show that current supervision regimes do not scale beyond simplified trace formats, despite positive results on structured benchmarks. The paper has genuine strengths: multi-seed runs, two backbones, fixed wall-clock comparisons, released code/checkpoints, and careful ablations showing that the distillation loss is non-essential and that training configuration matters little. C-MTP itself is simple, reproducible, and clearly outperforms CoLaR on the structured and semi-natural settings. However, the headline generalization rests on a narrow empirical base: one representative per supervision regime on MATH, training traces generated by the weak target model rather than a strong teacher, and a single domain. The appendix actually shows CODI exceeding CoT-SFT on a natural-language reasoning task (CommonsenseQA), so the negative result is not universal across realistic NL traces. The central negative claim is plausible but not yet established at the level claimed in the abstract.

major comments (3)
  1. [§4.5, Table 5; cf. §1 and §4.1] The main negative claim depends on the MATH trace distribution, but the manuscript is internally inconsistent about what that distribution is. §1 promises "realistic CoT traces from a strong teacher," while §4.1 states that for MATH "we generate CoT traces from the target model being trained," and §4.5 filters them by final-answer correctness. For Llama-3.2-1B the base model solves only ~24% of MATH (Table 5, Original row), so the filtered training set is small (a few thousand traces out of the 7.5k MATH train problems) and may be unrepresentative; for Qwen the solve rate is ~54%. Since §4.4 and Appendix E show C-MTP is brittle to arithmetic errors and span-boundary mistakes, the observed 65% drop may be a property of the low-quality supervision distribution rather than of realistic trace length. This needs a controlled manipulation of trace source/quality — e.g., traces from a genuinely
  2. [§4.5, §6, Abstract] The abstract and conclusion generalize to "current continuous CoT methods" from a single representative per supervision regime. The sentence "Since our earlier experiments already identify the best method in each regime" is not a substitute for testing on MATH: superiority on GSM8k-Aug does not imply representativeness for 350-token traces. Neither CoLaR, Sim-CoT, Coconut, nor step-supervised variants are evaluated on MATH. Moreover, Appendix G.7 shows CODI at 77.12 vs CoT-SFT at 74.40 on CommonsenseQA, so the collapse is not universal across real natural-language reasoning traces. At minimum, the claim should be qualified to the two tested methods on MATH, or the missing methods/domains should be added.
  3. [§4.5, §5] The MATH experiments filter correct generations by final-answer correctness only; there is no trace-quality control, no multiple references per problem, and no analysis of the resulting training distribution. CoT-SFT trained on the same filtered traces is a fair token-level control, but continuous methods face the additional difficulty of compressing those traces, and §4.4/Appendix E show that a single arithmetic error is often irrecoverable. The paper never varies trace quality or the number of references per problem, so the conclusion that "both supervision regimes fail on realistic traces" conflates trace realism with trace quality. A trace-quality sweep (e.g., correct-only vs. high-quality teacher traces) is necessary to separate these variables.
minor comments (5)
  1. [§4.5] The text refers to "Table 14" for the MATH results, but the table appears as Table 5 in the main body. The cross-reference should be fixed.
  2. [Abstract and Figure 3] The abstract says "roughly 65% performance drop," while Figure 3 says "~60% drop." The numbers in Table 5 range from about 58% to 76% depending on model and CODI latent count; the text and figure should use consistent, precisely computed figures.
  3. [§4.2.1 and Appendix C] The main text says the fixed wall-clock analysis is in Section G.4, but the timing table is in Appendix C. The pointer should be corrected.
  4. [§3.4 and Appendix A] The architecture description is clear overall, but the notation for the MLP in Appendix A, "g∈R^{d×sd}", is slightly ambiguous; writing it as a map g: R^d → R^{s×d} would be cleaner.
  5. [Table 3] The ablation shows that removing L_KD at s=4 drops GSM8k-Test from 40.70 to 36.60, a larger drop than at s=2 or s=3. The text acknowledges this, but the parenthetical "largely unchanged" in the main text could be more precise by explicitly stating the s=4 exception is 4 points.

Circularity Check

0 steps flagged

No significant circularity: C-MTP's latent definition is constructive, the training targets are ground-truth CoT spans, and the central negative result is an external empirical comparison.

full rationale

I walked the paper's derivation chain. C-MTP defines a latent in Eq. (7) as the average embedding of a span of ground-truth CoT tokens, and Eq. (8) trains the model to predict the tokens composing each span. This is a constructive definition, not a circular derivation: the training signal is ground-truth reasoning text, and performance is measured on held-out benchmarks (GSM8k, GSM-Hard, SVAMP, MultiArith, MATH, CommonsenseQA) rather than read off from the definition. The teacher-distillation loss L_KD is ablated in Tables 3 and 9 and shown to be non-essential, so the method's results do not depend on a self-referential training target. The realistic-trace experiment in §4.5 trains on correct traces generated by the target models themselves, and the paper filters only on final-answer correctness; this is a legitimate experimental-design concern about trace quality and representativeness, but it is not circularity under the stated criteria: no fitted parameter is renamed as a prediction, and the conclusion is an empirical comparison against a CoT-SFT baseline on MATH. The only self-citation I found is Saparov & He (2023), cited as the ProntoQA dataset in the introduction; it is a benchmark attribution and is not load-bearing for any claim. No uniqueness theorem, ansatz-via-citation, or renaming of a known result carries the conclusions. I therefore find no circular step and assign score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central method adds no fitted physical parameters; it depends on model/training choices and on the representativeness of filtered MATH traces. Compression factor s and CODI latent count K are the main hand-set numbers. No new entities are introduced; the MLP projection head g (Section A) is a standard architectural component.

free parameters (4)
  • Compression factor s = 2, 3, 4 (per run)
    Number of tokens per latent span; set by hand in Section 3.4; directly trades accuracy vs # reasoning steps (Tables 2 and 4).
  • CODI latent count K = 6 (structured/semi-natural); 6, 10, 20 (MATH)
    Fixed hyperparameter before training for indirect baselines; performance depends on it (Section 3.2, Table 5).
  • Training hyperparameters (LR, epochs, batch, LoRA rank, warmup) = Table 10
    Chosen per run; in sample-efficiency experiments the best config is selected by validation loss (Section G.3).
  • Wall-clock sample multiplier for C-MTP = ~2x CODI samples
    C-MTP is given twice the samples under fixed wall-clock parity in Table 12; this protocol choice affects the comparison but is stated.
axioms (4)
  • domain assumption Averaging token embeddings of a span produces a latent that retains enough information for reasoning (Eq. 7).
    This is the design premise of C-MTP; not derived from first principles.
  • domain assumption Parallel multi-token prediction over spans is a valid training objective for latent reasoning (Eq. 8).
    Paper assumes MTP alignment with pretraining objective; supported only empirically.
  • ad hoc to paper Correct-only generated MATH traces form a representative SFT distribution (Section 4.5).
    The paper filters wrong answers; if this distribution is unrepresentative, the negative result may not generalize.
  • domain assumption LoRA fine-tuning from pretrained checkpoints is sufficient to learn continuous CoT (Section 4.1, Table 10).
    All results use LoRA; full fine-tuning only for CoT baselines.

pith-pipeline@v1.3.0-alltime-deepseek · 25187 in / 11821 out tokens · 103735 ms · 2026-08-01T19:22:18.313466+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Training Continuous Chain of Thought Models: A Tale of Two Regimes." pith.science (2026). https://pith.science/paper/EV3QCCXY

@misc{pith2026260716972,
  author       = {Pith},
  title        = {Pith review of: Training Continuous Chain of Thought Models: A Tale of Two Regimes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EV3QCCXY}},
  note         = {Machine review of arXiv:2607.16972}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Continuous Chain-of-Thought methods replace verbose reasoning traces with a short sequence of dense latent representations. Earlier continuous CoT methods indirectly supervise the latent representations such that its final state match that of verbose reasoning traces, requiring autoregressive, slow generation during training. We introduce C-MTP, a simpler, faster direct supervision approach that models each latent as an average of the embeddings in the CoT traces to be compressed. Our approach outperforms a prior direct supervision method that approximates the distribution of compressed tokens, and performs competitively to slower indirect supervision approaches in existing evaluation setup with simplified CoT traces (less than 100 tokens). Lastly, we extend the evaluation of Continuous CoT methods to complex tasks with longer reasoning traces ($\ge$ few hundreds reasoning tokens). We find both direct and indirect supervision training methods perform poorly (roughly 65\% performance drop) in this setting, revealing the limitations of current continuous CoT methods. The code and checkpoints are released at https://github.com/Varun221/cmtp_research

Figures

Figures reproduced from arXiv: 2607.16972 by Eunsol Choi, He He, Varun Yerram.

Figure 1
Figure 1. Figure 1: Head-to-Head comparison of standard CoT finetuning with two ContinuousCoT training methods: Direct Supervision (C-MTP, our proposed method) and Indirect Supervision (CODI) on three datasets of different CoT traces. Structured consists of compact mathematical expressions of ∼25 tokens, Semi-Natural contains sentence per-step explanations of ∼62 tokens while Re￾alistic captures traces generated from off-the-… view at source ↗
Figure 2
Figure 2. Figure 2: Fraction of self-consistent expressions by absolute po￾sition in C-MTP-X and CoT traces. We only plot for samples for which CoT arrives at the right answer. Trained on GSM8k-Aug, Eval: GSM8k-Test. 4.3. Training on Semi-Natural CoT: GSM8k-Aug-NL [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: LLaMA-3.2-1B-Instruct Results at a glance. A comparative analysis of weighted average accuracy (%) versus the average number of reasoning steps (log scale) for StandardCoT and two regimes (direct and indirect) of ContinuousCoT across three levels of trace difficulty. On simpler tasks (structured and semi-natural), both regimes perform competitively with Standard CoT. However, on complex, realistic tasks, p… view at source ↗
Figure 5
Figure 5. Figure 5: Sample efficiency comparison on GSM8k-Aug. G.4. Fixed WallClock Time Comparison Our analysis ( [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 5 linked inside Pith

  1. [1]

    - Bella bought 18 packs

    Calculate the total number of chocolates Bella initially had: - Each pack contains 12 chocolates. - Bella bought 18 packs. - Therefore, the total number of chocolates is: 18packs×12chocolates per pack= 216chocolates

  2. [2]

    - Each pack contains 12 chocolates

    Determine the number of chocolates in the packs Bella gave away: - Bella gave away 6 packs. - Each pack contains 12 chocolates. - Therefore, the number of chocolates given away is: 6packs×12chocolates per pack= 72chocolates

  3. [3]

    Calculate the number of chocolates remaining: - Subtract the number of chocolates given away from the initial total: 216chocolates−72chocolates= 144chocolates

  4. [4]

    Gozeten, H

    URL https://openreview.net/forum? id=ph04CRkPdC. Gozeten, H. A., Ildiz, M. E., Zhang, X., Harutyunyan, H., Rawat, A. S., and Oymak, S. Continuous chain of thought enables parallel exploration and reasoning. CoRR, abs/2505.23648, 2025. doi: 10.48550/ARXIV . 2505.23648. URLhttps://doi.org/10.48550/ arXiv.2505.23648. Grattafiori, A., Dubey, A., Jauhri, A., P...

    doi:10.48550/arxiv.2505.23648 2025

  5. [5]

    URL https://openreview.net/forum? id=NjNGlPh8Wh. OpenAI. Openai o1 system card, 2026. URL https: //arxiv.org/abs/2412.16720. Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word prob- lems? In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cot- terell, R., Chakraborty, T...

    Pith/arXiv arXiv 2026

  6. [8]

    URL https: //doi.org/10.48550/arXiv.2509.25239

    doi: 10.48550/ARXIV .2509.25239. URL https: //doi.org/10.48550/arXiv.2509.25239. Xu, X., Yu, T., Chen, X., Wang, H., McAuley, J., and Mitra, S. Thinkrouter: Efficient reasoning via routing thinking between latent and discrete spaces, 2026. URL https://arxiv.org/abs/2602.11683. Yue, Z., Jin, B., Zeng, H., Zhuang, H., Qin, Z., Yoon, J., Shang, L., Han, J., ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.25239 2026

  7. [9]

    URL https: //doi.org/10.48550/arXiv.2602.01148

    doi: 10.48550/ARXIV .2602.01148. URL https: //doi.org/10.48550/arXiv.2602.01148. 12 Training Continuous Chain of Thought Models: A Tale of Two Regimes Table of Contents A.MTP Architecture B.Example of CoT Traces for Three Difficulties C.Timing Analysis for Fixed Wallclock Time Experiments D.Analysis of Problem Structures in GSM8k-Aug E.Examples of Error A...

    doi:10.48550/arxiv.2602.01148

  8. [13]

    Convert the remaining chocolates into dozens: - There are 12 chocolates in a dozen. - To find out how many dozens of chocolates remain, divide the number of chocolates by 12: \frac{144 \text{ chocolates}}{12 \text{ chocolates per dozen}}= 12\text{dozens} Thus, the number of dozens of chocolates Bella has left is\(\boxed{12}\) Table 6.CoT Trace Examples Re...

    2020

  9. [168]

    naacl-main.168/

    URL https://aclanthology.org/2021. naacl-main.168/. Pfau, J., Merrill, W., and Bowman, S. R. Let’s think dot by dot: Hidden computation in transformer language models. CoRR, abs/2404.15758, 2024. doi: 10.48550/ARXIV . 2404.15758. URLhttps://doi.org/10.48550/ arXiv.2404.15758. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.15758 2021

  10. [2023]

    Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B

    URL https://proceedings.mlr.press/ v202/gao23f.html. Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., and Goldstein, T. Scaling up test-time compute with latent reasoning: A recurrent depth approach. CoRR, abs/2502.05171, 2025. doi: 10.48550/ARXIV . 2502.05171. URLhttps://doi.org/10.48550/ arX...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.05171 2025

  11. [2024]

    Goyal, S., Ji, Z., Rawat, A

    URL https://proceedings.mlr.press/ v235/gloeckle24a.html. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V . Think before you speak: Training language models with pause tokens. InThe Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net,

    2024

  12. [2025]

    URL https: //doi.org/10.48550/arXiv.2511.08577

    doi: 10.48550/ARXIV .2511.08577. URL https: //doi.org/10.48550/arXiv.2511.08577. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y ., Callan, J., and Neubig, G. PAL: Program-aided lan- guage models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.),Pro- ceedings of the 40th International Conference on Ma- c...

    Pith/arXiv arXiv doi:10.48550/arxiv.2511.08577

  13. [2026]

    Tang, Y ., Dong, L., Hao, Y ., Dong, Q., Wei, F., and Gu, J

    URL https://openreview.net/forum? id=AQsko3PPUe. Tang, Y ., Dong, L., Hao, Y ., Dong, Q., Wei, F., and Gu, J. Multiplex thinking: Reasoning via token-wise branch- and-merge.CoRR, abs/2601.08808, 2026. doi: 10.48550/ ARXIV .2601.08808. URL https://doi.org/10. 48550/arXiv.2601.08808. Wang, L., Xu, W., Lan, Y ., Hu, Z., Lan, Y ., Lee, R. K.- W., and Lim, E.-...

    doi:10.48550/arxiv.2601.08808 2026