Pith. sign in

REVIEW 4 major objections 4 minor 12 references

LongStraw argues that long-context GRPO can run at 2 million-plus tokens on a fixed GPU budget by capturing the shared prompt once without autograd and replaying one short response at a time under gradient tracking.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LongStraw caches a long prompt without autograd and replays response suffixes with gradients to fit 2M+ token GRPO-like training into fixed GPU budgets, but it never proves the resulting update equals the true RL gradient.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Serious fixed-budget long-context RL systems engineering, but the 'exact GRPO' claim rests on dropping the prompt-side gradient and never checking the approximation. the 4 major comments →

arxiv 2607.14952 v3 pith:3BO5SLY2 submitted 2026-07-16 cs.LG cs.DC

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou , Kieran Liu , Yuhua Zhou , Qian Qiao , Jun Gao , Harry Zhang , Irvine Lu , Nolan Ho
show 12 more authors
Lucian Li Andrew Lei Cleon Cheng Steven Chiang Yihang Zeng Di Zhang Rio Yang Kaijie Chen Andrew Chen Pony Ma Weizhong Zhang Cheng Jin
This is my paper
classification cs.LG cs.DC
keywords long-context reinforcement learningGRPOresponse replayprompt state capturefixed GPU budgetKV cache offloadingcontext parallelismmixture-of-experts training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context reinforcement learning is normally blocked by memory: the shared prompt's entire computation graph stays alive while several responses to it are scored and backpropagated, which is why million-token RL training has needed hundreds of GPUs. LongStraw claims the prompt graph does not need to be live at all. It runs the shared prompt once with gradients disabled, keeps only the minimal state later tokens depend on (compressed attention pages, recurrent state, sparse-indexer keys), then replays one short response at a time under autograd and fires one optimizer step after accumulating the group's gradients. On eight H20 GPUs it reports response-only GRPO steps at exactly 2,097,152 positions for group sizes 2 and 8, plus 4.25M-position prefix reuse across eight optimizer cycles; on 32 H20 GPUs it reports a complete online 2M-token transaction: real sampling, mixed rewards, two 78-layer backwards, distributed gradient finalization, one optimizer step. If correct, this turns long-context RL from a memory-scaling problem into a state-lifetime and ownership problem.

Core claim

LongStraw's central claim is that a GRPO update can be computed from a detached prompt state. The gradient decomposes into a response-token term plus the prompt-side vector–Jacobian product; LongStraw keeps only the first term, storing the prompt state as a read-only stop-gradient quantity and dropping the prompt graph. Because every member of one update shares the same parameters, one captured prefix serves the whole group under this response-only contract. The schedule is: capture the prompt once without autograd, freeze old/reference scores, replay one response at a time under autograd, accumulate gradients, finalize across ranks, step once — bounding the live graph by the response suffix

What carries the argument

The load-bearing mechanism is the stop-gradient prompt boundary: the prompt state is stored as a detached conditioning quantity, turning a multi-million-token prefix into read-only state and shrinking the live autograd graph from prompt-plus-response to response-suffix length. Around it sits a transaction schedule — no-grad capture, frozen old/reference scoring, serial per-member replay, gradient accumulation, one distributed finalization and optimizer step — making group size a time cost rather than a memory cost. Supporting mechanisms are physical page compaction (copying sharded KV and latent state into right-sized allocations so allocator ownership matches page-table ownership), whole-la

Load-bearing premise

LongStraw's schedule rests on the claim that a stop-gradient prompt state carries everything the update needs: it keeps only the response-token gradient and discards the prompt-side term ∂ℓ/∂zP · ∂zP/∂θ (Equation 3, Section 4.1), without showing the result matches the full GRPO gradient or that the dropped term is negligible — and its separate multi-step reuse mode is only approximate, with the paper's own probe showing loss divergence growing to 22.8% by step 4.

What would settle it

Run one short-context GRPO step two ways — full autograd through prompt and response, and LongStraw's detached-prompt replay — and compare the resulting parameter updates; any non-trivial difference means the response-only objective is not the GRPO objective. Since the discarded prompt-side vector–Jacobian product can only grow in importance with context length, a single mismatch at any scale would falsify the exactness claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Group size stops costing memory: at 2,097,152 positions on eight H20s, raising the group from 2 to 8 members added 0.208 GB of peak allocated memory (0.21%) while adding roughly 1,586 seconds of serial replay time.
  • The live training graph is bounded by the response suffix: per-member activation memory scales with response length, not context length, while prompt compute is paid exactly once during no-grad capture.
  • Exact long-context RL is attainable on fixed hardware: the online 2M-token transaction — real sampling, rewards [-1,+1], two 78-layer backwards, distributed gradient finalization, one optimizer step — completed on all 32 ranks.
  • Physical page ownership is a precondition for capacity: copying owned shards into right-sized allocations was required to finish the 2,088,960-token prefix at 58.7 GB per rank, and the same eight H20s then sustained 4.25M-position replay at 83.9 GB per rank.
  • The paper's own probe bounds the multi-step reuse mode: resident-prefix loss diverges from a freshly recaptured prefix as steps accumulate (0.12% at step 1, 22.8% at step 4), so only the per-step recapture contract is claimed exact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The exactness claim rests solely on the response-only term of Equation 3; the paper never bounds or measures the discarded prompt-side vector–Jacobian product, so the decisive experiment is a same-context comparison of response-only versus full-graph updates wherever the full graph fits in memory.
  • Section 9.7's own probe shows resident-prefix reuse diverging from a recaptured prefix from 0.12% (step 1) to 22.8% (step 4); multi-step 4M-token reuse should therefore be read as a capacity demonstration until a staleness correction (periodic recapture or a drift term) is added.
  • The Qwen receipt (Sections 5.7 and 12.1) all-reduces the query gradient but leaves page-owner key/value contributions on eight separate local optimizer instances, so the K/V-projection adapters are not updated coherently across the context-parallel group; the deferred companion audit is a precondition for any quality claim on that path.
  • The online GLM transaction is reported without whole-run wall time or peak memory (Table 5 marks both as not reported); within the fixed-budget claim, the 25,536-event trace inventory is evidence that the run completed, not a memory measurement, so a full allocation trace of the online transaction is the missing quantity for reproduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. LongStraw proposes a fixed-GPU memory-scheduling system for long-context GRPO. It captures the shared prompt once without autograd, retains only architecture-specific conditional state (Qwen GDN/KV pages; GLM MLA/DSA pages on CPU), replays response suffixes under autograd one at a time, accumulates gradients across the group, and performs one distributed optimizer step. The paper reports Qwen runs at 2,097,152 and 4,456,448 positions on eight H20 GPUs and a GLM 'exact-2M online GRPO' transaction on 32 H20 GPUs that includes a vLLM rollout, two response-only 78-layer backwards, global DSA composition, finalize_model_grads, and one optimizer step. The central mechanism is Eq. (3): LongStraw computes only the first term of the gradient decomposition, ∂ℓ/∂θ at a fixed prompt state z̄_P = stopgrad(z_P(θ)), and explicitly omits the prompt-side vector–Jacobian product ∂ℓ/∂z_P · ∂z_P/∂θ. The paper treats this as 'response-only' execution but also describes the runs as 'exact-2M online GRPO' and 'complete million-token RL post-training'.

Significance. The systems insight — bounding the live autograd graph by the response suffix while reusing a shared prompt computation across the GRPO group — is genuinely valuable, and the implementation is substantial: physical page compaction, CP/EP ownership, CPU staging, whole-layer checkpointing, global DSA composition, and a vLLM-to-Megatron topology handoff are all non-trivial. The paper also ships reproducible artifacts (commit-pinned code, trace inventories, allocation logs), which is commendable and makes the execution receipts checkable. However, the significance is undercut by the objective mismatch at the core of the method. The executed update is not the gradient of the GRPO objective defined in Eq. (2): the prompt-side parameter path is detached by construction, and no experiment bounds or measures the omitted term. If reframed as a systems study of a response-only surrogate objective with an explicit error analysis, the contribution could be publishable in a systems venue. As written, the headline claim of 'exact-2M online GRPO' and 'complete million-token RL post-training' is unsupported.

major comments (4)
  1. [§2.2, Eq. (3); §4.1, Eq. (9)] The central claim of exact GRPO is not supported by the math. Eq. (3) correctly decomposes the full gradient into the direct response term and the prompt-side vector–Jacobian product. LongStraw then stores z̄_P = stopgrad(z_P(θ)) and computes only the first term. Since the prompt encoder shares parameters with the response predictor, ∂ℓ/∂z_P · ∂z_P/∂θ is generically nonzero; dropping it changes the objective from the GRPO loss in Eq. (2) to a response-only surrogate. The paper never bounds or measures this error, and no experiment checks gradient equality with full backprop. The phrase 'Changing the graph boundary, not the GRPO objective' (Fig. 2 caption) is therefore contradicted by Eq. (3). A minimal remedy is a short-context gradient-equivalence test (e.g., cosine similarity and norm ratio against full backprop) or an explicit renaming of the contribution as response-only surrogate tr
  2. [§5.7, §12.1, Eqs. (22)–(23)] The Qwen path does not complete a coherent distributed update even for its stated response-only objective. The paper states that the custom backward all-reduces dQ but leaves dK/dV rank-local, and that each of the eight ranks steps its own AdamW instance. For replicated K/V projection LoRA weights, the correct gradient is the sum over page owners (Eq. 23); without this reduction, the eight replicas diverge after the local optimizer calls. The 'companion CP8 adapter-synchronization audit' records the issue but does not repair it. Consequently, the Qwen 2M and 4.25M rows in Table 5 should not be described as completed GRPO training updates; at most they demonstrate per-rank response-gradient materialization and local optimizer application, with a documented unresolved ownership error.
  3. [§2.3, Table 8; §8.2] The validation levels cover execution capacity, forward-operator fidelity, and distributed-update consistency, but none of them verifies gradient fidelity against the full GRPO objective. Table 8's 'Global response forward' row checks only that the forward operator uses the intended prompt-wide attention; it says nothing about whether the backward equals the full GRPO gradient. The only numerical probe of prompt-state sensitivity in §8.2 shows large divergences between recaptured and resident states (loss differences growing to 22.81% and 9.33%, log-prob differences of 3.136 and 3.774). Although that probe concerns cross-step staleness rather than the within-step omitted VJP, it is internal evidence that prompt-state dependence is not negligible and makes the absence of a within-step gradient check particularly damaging.
  4. [§8.3, §7.8, Table 5] The GLM 'exact-2M online GRPO' receipt consists of two response-only backward passes with a detached prompt state, followed by finalize_model_grads and one optimizer step. The run reports no whole-transaction wall time, no valid whole-run peak memory, no comparison with full-graph training, and no task-quality metric; Table 5 marks these fields as n/r. The recorded 'one lower and one upper clipping event' is an execution observation, not evidence that the optimized objective matches Eq. (2). As an execution receipt the run is internally consistent, but it cannot substantiate the headline 'exact-2M online GRPO' or 'complete million-token RL post-training' without either a gradient-equality check or an explicit, validated reformulation of the objective.
minor comments (4)
  1. [§5.4, §8.2, Table 5 footnote] The Qwen scorer drops the first response label, so 8,192 response-input tokens yield at most 8,191 scored targets; the run log does not record the realized count. Please report the actual scored-token count for reproducibility.
  2. [Throughout] The term 'exact' is overloaded: it is used for exact context length (2,097,152 = 2^21), exact partition composition with BF16 numerator reduction (§5.3), and the exact-2M online GRPO workflow. Please disambiguate 'exact' as referring to length, forward operator, or gradient, and state clearly that the GRPO objective is not exactly optimized unless explicitly claimed.
  3. [Fig. 2 caption] The caption 'Changing the graph boundary, not the GRPO objective' is inconsistent with Eq. (3), which shows that omitting the prompt-side vector–Jacobian product changes the gradient. Please rephrase to avoid implying that the omitted term has no effect.
  4. [§8.2] The recaptured-versus-resident probe uses real DAPO text but synthetic rewards and non-model-sampled completions; this is acknowledged, but the text should make clear that the large divergence at steps 4 and 8 is not a training-quality result, only a state-consistency diagnostic.

Circularity Check

1 steps flagged

The central 'exact-2M online GRPO' claim reduces by construction to a self-defined response-only objective: Eq. 3's full GRPO gradient includes a prompt-side vector–Jacobian product that LongStraw explicitly drops and never shows negligible.

specific steps
  1. self definitional [Eq. 3; §2.2 'Conditional Replay as an Efficiency Boundary'; §4.1 'Capture Once, Replay the Suffix'; Abstract]
    "∇θℓ(θ, zP (θ)) = ∂ℓ/∂θ|zP + ∂ℓ/∂zP ∂zP/∂θ .(3) LongStraw stores ¯zP = stopgrad(zP (θ)) and computes the first term. ... Within update k, every group member sees the same θk, so sharing one captured prefix is exact for the stated response-only computation."

    The abstract claims 'exact-2M online GRPO' and 'complete million-token RL post-training', but Eq. 3 defines the full GRPO gradient as the direct response term plus the prompt-side vector–Jacobian product ∂ℓ/∂zP ∂zP/∂θ. The transaction in §4.1 captures the prompt with autograd disabled and computes only the first term, treating z̄P as a constant. The executed update is therefore the gradient of a different, response-only objective; calling it 'exact' is true only by construction because the objective was defined as the part that survives the detached boundary. The paper never bounds or measures the omitted term, and its own recaptured-vs-resident probe shows loss differences growing to 22.81% and log-prob differences to 3.774, indicating the prompt-state path is not negligible. Thus the hea

full rationale

The paper's memory-engineering results are largely self-contained and not circular: physical page compaction, CP8 LSE/output composition, CPU page staging, whole-layer checkpointing, and trace-validated execution events are externally checkable systems facts, and no fitted constant is disguised as a prediction. Self-citations to MinT and OOMB are disclosed and are not used to prove the central gradient claim. The circularity is concentrated in the word 'exact'. LongStraw's own Eq. 3 decomposes the true GRPO gradient into a direct response term and a prompt-side term, then the system keeps only the first term and calls the result 'exact' for 'the stated response-only computation'. Since the stated computation is precisely what the system computes, the exactness is definitional. The broader claim that this is 'exact-2M online GRPO' and 'complete million-token RL post-training' therefore reduces to the self-defined response-only surrogate. The Qwen runs additionally use supplied responses, synthetic rewards, β=0, and old/reference scores from the same runner, and the GLM receipt uses a two-completion G=2 sample without any gradient-equivalence check, so the executed update is not shown to equal the GRPO policy gradient. This is a partial circularity of the central claim, not of the capacity measurements, so the score is 6 rather than higher.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claim rests on treating the prompt state as a stop-gradient boundary, plus standard checkpointing and position-extension assumptions. No new physical entity is introduced. The free parameters are system-configuration choices rather than fitted scientific constants, but the YaRN and response-block settings directly shape the reported capacity envelope.

free parameters (2)
  • YaRN extension factor and max_position_embeddings = 2.000015 / 2097168
    Hand-chosen position-extension configuration used to claim exact-2M GLM context; note 2097168 is not exactly 2097152, and no calibration or ablation is given for the 2M regime.
  • Response block size 2048 = 2048
    Qwen response replay splits suffixes into 2048-token blocks; this value directly controls the live-graph memory bound and is chosen without a sensitivity analysis.
axioms (5)
  • ad hoc to paper The GRPO policy loss is correctly optimized by computing only ∂ℓ/∂θ at fixed prompt state, dropping ∂ℓ/∂z_P ∂z_P/∂θ.
    Invoked in Eq. 3 and §2.2; this is the load-bearing assumption and is never validated against the full policy gradient.
  • standard math The member-normalized GRPO objective in Eq. 2 matches the intended training objective.
    Cites Shao et al. (2024); no derivation needed, but the paper's actual runs use β=0, old=reference, and unclipped ratio, so the objective tested is a special case.
  • domain assumption BF16 numerator reduction in the CP8 LSE merge still yields the intended dense attention operator.
    Stated in §5.3: the merge is partition-correct but not bitwise FP32-equal; no numerical error bound is provided.
  • domain assumption Whole-layer reentrant checkpointing with RNG preservation reproduces the same forward during backward.
    Standard for deterministic checkpointing; the GLM path relies on this for the 78-layer replay, but no per-layer bitwise or tolerance-level equality is verified.
  • domain assumption YaRN position extension preserves model behavior at 2M tokens despite native 1,048,576 training context.
    Used for rollout and training in the exact-2M GLM run; the paper reports no YaRN calibration or quality metric at that length.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget." pith.science (2026). https://pith.science/paper/3BO5SLY2

@misc{pith2026260714952,
  author       = {Pith},
  title        = {Pith review of: LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BO5SLY2}},
  note         = {Machine review of arXiv:2607.14952}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Long-context RL post-training is constrained by the lifetime of state and gradients, not attention cost alone. In GRPO, one multi-million-token prompt must serve old-policy and reference scoring plus multiple policy responses, while conventional autograd keeps the prompt graph and all response graphs live alongside model weights, caches, and distributed communication buffers. We present LongStraw, an objective-aware, architecture-aware system for resident-state virtualization, response replay, and distributed-gradient execution. Its transaction captures the shared prompt without autograd, retains only the architecture-required state on explicitly owned pages, restores that state for each group member, scores old/reference branches without a graph, replays one policy response at a time with autograd, and accumulates the resulting gradients before one distributed finalization and optimizer step. This schedule bounds the live training graph by the response suffix while reusing the expensive prompt computation across the complete GRPO group. We instantiate this design for two incompatible model structures. Qwen3.6-27B combines 48 recurrent GDN layers with 16 full-attention layers; LongStraw keeps the compact recurrent state and physically CP8-sharded KV pages, composes global attention through cross-rank LSE/output merging, and performs blockwise response replay. GLM-5.2 combines a 78-layer MLA/DSA attention stack with a 256-expert, top-8 MoE tail. Its implementation keeps CP-sharded MLA latent pages and DSA indexer-key pages in CPU memory, stages one layer at a time, reconstructs IndexShare-aware global sparse selection over CP32, and dispatches routed response tokens over EP32. The two paths share one transaction contract while specializing the retained state, replay operator, and collective communication to the architecture...

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith

  1. [1]

    all-layer no-grad prompt capture at 128K, 256K, 512K, 1M, and 2.097M; the final 2,097,152-position prefix-only capture took 675.657 s

  2. [2]

    layer-0 2.097M capture, two local backwards, and an optimizer-call test in 738.579 s

  3. [3]

    all-layer CP32 tests at 32K and 64K after IndexShare, CPU page, shared RoPE, and checkpoint fixes

  4. [4]

    Hao Liu, Matei Zaharia, and Pieter Abbeel

    URLhttps://arxiv.org/abs/2504.14960v1. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring Attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889, 2023. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, et al. DoRA: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353, 2024. URLhttps://arxiv.org/abs/2402.09353...

    arXiv 2023

  5. [5]

    the predecessorG= 2rank-complete transaction in 2975.138 s

  6. [6]

    32K/64K online integration canaries; and

  7. [7]

    The sequence is monotonic in execution coverage, not in elapsed time

    the current exact-2M policy rollout and finalizedG= 2update. The sequence is monotonic in execution coverage, not in elapsed time. The CP-local DSA and skipped- finalization boundaries apply only to the predecessor milestones; the current exact-2M transaction observes the global operator and gradient finalization. F Qwen 4.25M Replay within Eight H20s The...

  8. [9]

    a 2.097MG= 1all-78-layer test in 2042.975 s; and

    2042

  9. [2021]

    Qwen Team

    URLhttps://arxiv.org/abs/2104.04473. Qwen Team. Qwen3.6-27B configuration. Hugging Face model configuration, 2026. URLhttps://huggingface.co/Qwen/ Qwen3.6-27B/blob/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9/config.json. Accessed July 16, 2026. Markus N. Rabe and Charles Staats. Self-Attention does not needO(n2)memory.arXiv preprint arXiv:2112.05682, 2021. S...

    Pith/arXiv arXiv 2026

  10. [2022]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

    URLhttps://arxiv.org/abs/2205.05198. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the ACM Symposium on Operating Systems Principles, 2023. Dmitry Lepikhin, HyoukJoong Lee, Yuan...

    Pith/arXiv arXiv 2023

  11. [2023]

    URLhttps://aclanthology.org/2023.emnlp-main.298/

    doi: 10.18653/v1/2023.emnlp-main.298. URLhttps://aclanthology.org/2023.emnlp-main.298/. Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. IndexCache: Acceler- ating sparse attention via cross-layer index reuse.arXiv preprint arXiv:2603.12201, 2026. Lequn Chen, Zihao Ye, Yongji Wu, et al. Punica: Multi-tenant LoRA...

    arXiv 2023

  12. [2025]

    GLM-5-Team

    URLhttps://arxiv.org/abs/2502.21231. GLM-5-Team. GLM-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang, Yonggang Wen, Tianwei Zhang, Xin Jin, and Xuanzhe Liu. LoongTrain: Effi- cient training of ...

    Pith/arXiv arXiv 2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.