Pith. sign in

REVIEW 4 major objections 4 minor 16 references

BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BDH-CQ claims that in-context learning and recurrent latent reasoning combine to reach 29.5% pass@2 on ARC-AGI-1 at $0.00070 per task, establishing a new state of the art in benchmark cost efficiency.

desk verdict Solid behavioral study undermined by an internal cost-accounting contradiction that invalidates the headline Pareto-frontier claim until reconciled. read the letter →

arxiv 2608.09888 v1 pith:CTGMJYBG submitted 2026-08-10 cs.NE cs.AIcs.LGstat.ML

Björn Engdahl , Adrian Kosowski , Jan Chorowski , Zuzanna Stamirowska , Przemys{l}aw Uznański , Junlin Jiang , Rohan Phadke , Remigiusz Kinas
show 1 more author
Richard Zhong
This is my paper · ORCID
classification cs.NEcs.AIcs.LGstat.ML
keywords in-contextlearningrecurrentlatentreasoningARC-AGI-1cost-accuracyfrontiercontinuousthoughtBDHarchitecturevisualskillacquisition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces BDH-CQ, a reasoning system that performs in-context learning through recurrent memory and solves tasks by iterating over a continuous latent workspace instead of verbalizing intermediate steps. The central claim is that this combination lets a 150M-parameter model reach 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed inference cost of $0.00070 per task, a point the paper argues breaks the previously reported cost-accuracy Pareto frontier. The authors also use ConceptARC and controlled ARC-like tasks to map what the model can bind from demonstrations, how consistently it applies an inferred transformation, and where it fails. A sympathetic reader should care because the result suggests that nontrivial abstract reasoning can be achieved without token-by-token narration, and at a small fraction of the cost of verbal-reasoning systems.

What carries the argument

The central mechanism is the separation of contextual memory from the reasoning workspace: demonstrations continuously update a recurrent latent state, and the query is answered by iterative computation in a high-dimensional latent workspace using BDH layers (ReLU-low-rank transformations combined with linear attention). The update equations $S_t = U_\theta(S_{t-1}, D_t)$ and $H_{r+1} = F_\theta(H_r, S_K)$ define the system-level interface that the paper studies; the memory carries what the demonstrations specify, while the workspace carries the ongoing computation for the current query.

What would settle it

Re-measure BDH-CQ's per-task cost on a metered cloud GPU with all overheads (startup, batching, energy, candidate ranking) and compare it with the reported costs of the systems on the leaderboard; if the measured cost at 29.5% pass@2 is not below the cost of every system at equal or higher accuracy, the claimed Pareto-frontier breakthrough is unsupported. A second decisive check would be an independent run that verifies the 118/400 pass@2 score under the stated protocol.

Watch

Extended reading notes

Core claim

BDH-CQ processes each demonstration sequentially, updating a recurrent memory $S_t = U_\theta(S_{t-1}, D_t)$, then encodes the query and iterates a latent workspace $H_{r+1} = F_\theta(H_r, S_K)$ before decoding an answer. The paper's central discovery is that this design acquires unseen visual transformations purely from demonstrations, with no parameter updates and no task-specific identity, and does so efficiently enough to establish a new state of the art in ARC-AGI-1 cost efficiency: 29.5% pass@2 at $0.00070 per task, about 57x cheaper than the comparable leaderboard system GPT 5.6 Luna (Low). The behavioral studies show the model binds dense color mappings, extrapolates boundary propagation and copying across the tested ranges, but shows structured limits on ordering long sequences and deep nesting, with composition success depending on the operation.

Load-bearing premise

The central cost-efficiency claim assumes that BDH-CQ's computed $0.00070 per task (0.85 H200 GPU-seconds at $3/hour) is directly comparable to the hardware estimates and API prices reported for other leaderboard systems, which the paper itself notes may not be commensurable.

Editorial extensions

If this is right

  • ARC-AGI-1 cost efficiency gains a new frontier point: a 150M-parameter system at 29.5% pass@2 and $0.00070 per task, roughly 57x cheaper than GPT 5.6 Luna (Low) at 34.2% and $0.040.
  • Increasing the latent reasoning effort during inference raises pass@2 from 21% (LOW) to 27% (MEDIUM) to 29.5% (HIGH), with cost reductions of 22% and 11% respectively.
  • Demonstrations alone can bind dense task-specific mappings: a fresh color permutation is applied to all 96 held-out outputs at rank one, even with eight simultaneous bindings.
  • The model's consistency gap on ConceptARC (77.92% test-pair pass@2 vs 59.38% strict-task pass@2) shows that correct individual outputs do not always transfer to all test inputs of a task.
  • Ordering eight bars and nesting five containment relations expose distinct bottlenecks: ordering failures break the whole output structure, while nesting failures preserve structure and differ in a single containment decision; adding a matched demonstration largely removes the nesting cliff.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the computed cost is genuinely commensurable with leaderboard hardware and API prices, then ARC-AGI-1's score-cost plane becomes accessible to very small recurrent models, which would shift the competition toward cost-aware benchmarks rather than raw accuracy.
  • The strong in-context binding of dense mappings suggests that the same recurrent memory could be trained to condition on textual demonstrations for language and math tasks, a direction the paper only outlines.
  • The consistency gap between pair and task accuracy implies that pass@2 as usually reported may overstate the rate of rule induction; reporting whole-task solve rates would make capability claims more comparable across systems.
  • The composition results (rotation composes with relocation 72/72, reflection 47/72, color swap 0/72) indicate that compositionality is not a single ability but is mediated by the representation of the operations being composed, which can be tested by varying motif families and operation pairs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces BDH-CQ, a 150M-parameter system that combines in-context learning with recurrent latent reasoning for ARC-style visual tasks. Demonstrations are ingested sequentially into a recurrent memory, and the query is solved by iterating a latent workspace without decoding intermediate tokens. The authors report 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed $0.00070 per task, claim this point breaks the cost-accuracy Pareto frontier, and support the result with ConceptARC profiling, controlled post-freeze generalization experiments, and generated ladders. The paper is unusually transparent about many limitations, including a corrected 240-task rerun, contradictory generated tasks, and the statistical weakness of the effort-tier comparison.

Significance. If the reported cost figure can be reconciled and the system made independently verifiable, the result would be significant: a compact, cheap ARC-AGI-1 system with a high score-to-cost ratio, plus a behavioral methodology that distinguishes isolated correct outputs from consistent rule application. The controlled experiments use deterministic oracles and exact-output metrics, and they deliver falsifiable findings such as the demonstration-coverage effect on nesting and ordering. However, the central cost-efficiency claim is currently undermined by an internal contradiction between §5 and §6.6, and the claimed independent audit is conducted by co-authors. These issues must be resolved before the headline result can be accepted.

major comments (4)
  1. [§5 vs §6.6] Section 5 states that the default 29.5% pass@2 result costs approximately 0.85 H200 GPU-seconds per task, giving $0.00070 at $3/hour. Section 6.6 describes the same full public ARC-AGI-1 evaluation under STANDARD effort, scoring 118/400 pass@2 (exactly 29.5%) at $0.00265246 per task, while MIN effort costs $0.00088399 and scores 111/400. These two cost figures for the same operating point differ by a factor of about 3.8; at $3/hour, $0.00265246 corresponds to 3.18 H200 GPU-seconds per task, not 0.85. Because the Pareto-frontier plot, the 'less than one-tenth of a cent' claim, and the 'approximately 57x cheaper than GPT 5.6 Luna (Low)' comparison are all based on the $0.00070 figure, the manuscript's central cost-efficiency result is not internally consistent. The authors must provide a line-item cost model and reconcile the two figures before the claim can be evaluated.
  2. [§5 (Independent evaluation) and title page] The paper's verification section is labeled 'Independent evaluation' but identifies the auditors as Remigiusz Kinas and Richard Zhong, both of whom are listed as co-authors of this manuscript on the title page. A verification performed by co-authors is not independent in the sense needed to support the claim that the deployed system's 29.5% score was reproduced by outside parties. This matters because the paper contains no publicly accessible audit report or URL, so the manuscript's only external check is an internal one. Please either have the audit performed by genuinely independent researchers who are not authors, or remove the 'independent' wording and describe the procedure as an internal black-box check.
  3. [§3.3, §4.1, §5] Section 3.3 states that dimensions, exact update rules, and implementation details remain proprietary, Section 4.1 states that the complete internal training recipe remains proprietary, and no weights or evaluation code are provided. For an empirical systems claim with a measured cost value, this would be acceptable if every reported number were internally consistent and auditable; the unresolved §5/§6.6 discrepancy shows that this precondition is not met. Please disclose the cost model (GPU-seconds per candidate, number of candidates, batch effects, hardware assumption) and enough of the inference procedure to allow a third party to reproduce the measurement, or explicitly reframe the cost-efficiency claim as a self-reported figure.
  4. [§5, Figure 2] The Pareto-frontier claim assumes that BDH-CQ's computed hardware cost is commensurable with the leaderboard costs of other systems, which the manuscript itself notes may represent hardware estimates or API prices. If other points are API prices that include provider margins or different hardware, an apple-to-apples comparison is not established. Please provide a sensitivity analysis over the $3/hour assumption and, if possible, a comparison using a uniform cost metric (all hardware estimates or all API prices) before claiming a frontier breakthrough.
minor comments (4)
  1. [§6.2, Figure 5] The text says propagation and copying remain correct on 48/48 held-out outputs, but Figure 5's caption says these families use 12 outputs per point; please state the number of plotted points explicitly so the reader can verify the denominator.
  2. [§5] The 'approximately 57x cheaper' and 'approximately 11x cheaper' comparisons to GPT 5.6 Luna (Low) do not show their arithmetic; in particular, the 11x figure appears to assume the 80% price reduction, but the revised Luna price is not stated. Please include the calculation explicitly.
  3. [§6.6] The passage notes that all 75 single-candidate records were correct at rank one, which is important context for pass@2; please also state how many tasks received two candidates and whether the reported cost per task includes the compute for one or two candidate generations.
  4. [Appendix A.3] The corrected 240-task rerun is described in the text, but Table 8 and the surrounding discussion do not annotate which entries come from the corrected set versus the original generator run; please add explicit markers.

Circularity Check

1 steps flagged · score 2.0 of 10

No material circularity: the headline result is an external benchmark measurement with a minor co-author self-citation in the audit chain.

  1. other [Section 5, 'Independent evaluation' paragraph]
    "An independent black-box audit conducted by co-authors from Bielik and New York University reproduced the deployed system’s 29.5% pass@2 score on the public ARC-AGI-1 evaluation set. The auditors evaluated the system under a documented protocol without access to model weights."

    The 'independent' auditors are Remigiusz Kinas and Richard Zhong, who are listed co-authors of this paper, and the supporting citations are their own technical reports. The external-verification claim therefore terminates in a self-citation chain. This is a credibility concern, but it is not the derivation of the measured 29.5% result: that result is a direct evaluation against the public ARC-AGI-1 benchmark, and no fitted constant is renamed as a prediction. Hence this is one minor self-citation rather than a load-bearing circular step.

full rationale

The paper's central claims are empirical measurements on the public ARC-AGI-1 evaluation set, not derivations from fitted constants. Equations (1)-(4) define an architectural interface (memory update, latent iteration, decoding) and no predicted quantity is shown to equal a training input or fitted parameter by construction. The training mixture explicitly excludes evaluation-task demonstration pairs, and the headline 29.5% pass@2 is measured against an external benchmark; the cost figure is a hardware-time estimate at a stated rental rate. The only self-citation with a verification role is the 'independent' audit, whose auditors are listed co-authors; that is a minor self-citation, not the logical source of the headline number. Separately, the paper contains an internal cost-accounting tension: Section 5 reports $0.00070 per task for the 29.5% operating point, while Section 6.6 reports $0.00265246 per task for the same STANDARD setting scoring 118/400. That inconsistency is a correctness and reproducibility issue, but it is not circularity under the requested definition, because the claimed result does not reduce by construction to its own inputs. On the circularity scale, the derivation chain is self-contained and the score is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's claims rely on unverified data hygiene, cost comparability, auditor independence, and generator correctness. These are not backed by released artifacts.

free parameters (2)
  • H200 GPU rental price = $3 per hour
    Used to convert measured 0.85 H200 GPU-seconds into $0.00070 per task (Section 5). Not fitted to data; a different price changes the headline cost and the Pareto claim.
  • pass@2 candidate count = 2
    Design choice following the ARC-AGI leaderboard convention. Section 6.6 notes that pass@2 can overstate performance when candidates are not independently sampled.
assumptions (4)
  • domain assumption ARC-AGI-1 public evaluation tasks are absent from the proprietary training mixture and from checkpoint selection.
    Stated in Sections 4.2 and 5 but no leakage audit is provided. The evaluation set is public, so this is a load-bearing but unverified premise.
  • ad hoc to paper Leaderboard costs for other systems are commensurable with BDH-CQ's computed cost.
    Section 5 acknowledges other costs 'may represent hardware estimates or API prices.' The Pareto-frontier claim depends on this comparability.
  • ad hoc to paper The auditors (Kinas and Zhong) are independent of the system developers.
    The paper lists both as co-authors; treating their evaluation as independent is an assumption that conflicts with the authorship structure.
  • domain assumption The controlled generators produce correct deterministic oracles.
    Appendix A.3 required a corrected 240-task rerun because the original generator omitted a rule; Appendix A.4 reports at least one contradictory test output, so this assumption is only partially satisfied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BDH-CQ: In-Context Learning with Recurrent Latent Reasoning." pith.science (2026). https://pith.science/paper/CTGMJYBG

@misc{pith2026260809888,
  author       = {Pith},
  title        = {Pith review of: BDH-CQ: In-Context Learning with Recurrent Latent Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CTGMJYBG}},
  note         = {Machine review of arXiv:2608.09888}
}
abstract

We introduce BDH-CQ, a reasoning model that combines in-context learning with recurrent latent reasoning. Inputs presented at inference time continuously update the model's recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. We evaluate the model on the public ARC-AGI-1 evaluation set and use controlled ARC-like interventions to study what it learns from demonstrations, how consistently it applies an inferred transformation, and which concepts remain difficult. A 150M-parameter configuration reaches 29.5% pass@2 at a computed inference cost of \$0.0007 per task. This operating point breaks through the previously reported ARC-AGI-1 cost-accuracy Pareto frontier, establishing a new state of the art in benchmark cost efficiency.

Figures

Figures reproduced from arXiv: 2608.09888 by the authors.

Figure 1
Figure 1. An ARC-AGI-1 training task (identifier 0520fde7). Two of three demonstrations and the held-out query [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. ARC-AGI-1 score versus computed cost per task. Data points were collected from the official ARC Prize [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. ConceptARC pass@2 by concept area. Left: semantic-ID test-pair and strict-task accuracy; their gap [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Representative held-out examples from the four controlled generalization families. From top to bottom: [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Controlled generalization curves reporting exact held-out-output accuracy. Ordering and nesting include a [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Representative held-out pairs from an independent motif family. Relocation moves the motif to the gray [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Showing how pass@2 and cost scale with reasoning effort. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages

  1. [7]

    Jolicoeur-Martineau

    A. Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks.arXiv preprint arXiv:2510.04871,

    arXiv

  2. [8]

    Kosowski, P

    A. Kosowski, P. a. Uzna ´nski, J. Chorowski, Z. Stamirowska, and M. Bartoszkiewicz. The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain.arXiv preprint arXiv:2509.26507,

    arXiv

  3. [9]

    Y . Liu. Beyond tokens: A unified framework for latent communication in llm-based multi-agent systems.arXiv preprint arXiv:2606.05711,

    arXiv

  4. [10]

    Pathway Research

    URLhttps://arxiv.org/abs/2112.00114. Pathway Research. Beyond Transformers: A New Architecture for Solving Sudoku.https://pathway.com/ research/beyond-transformers-sudoku-bench,

    arXiv

  5. [11]

    Ramji, T

    K. Ramji, T. Naseem, and R. Fernandez Astudillo. Thinking without words: Efficient latent reasoning with abstract chain-of-thought.arXiv preprint arXiv:2604.22709,

    arXiv

  6. [14]

    G. Wang, C. Liu, C. Wang, et al. Hrm-text: Efficient pretraining beyond scaling.arXiv preprint arXiv:2605.20613,

    work page Pith review arXiv

  7. [15]

    URLhttps://arxiv.org/abs/2505.15778. H. Zhu, S. Hao, Z. Hu, J. Jiao, S. Russell, and Y . Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. InAdvances in Neural Information Processing Systems,

    arXiv

  8. [16]

    14 A Evaluation set analysis A.1 Evaluation sets and surface descriptors The concept-organized experiments above isolate individual transformations and structural demands

    URLhttps://arxiv.org/abs/2510.25741. 14 A Evaluation set analysis A.1 Evaluation sets and surface descriptors The concept-organized experiments above isolate individual transformations and structural demands. We complement them by profiling the public ARC-AGI-1 evaluation set, a generated set calibrated to match its measured distribution, and a generated ...

    arXiv

Show all 16 references
  1. [2016]

    Geiping, S

    J. Geiping, S. McLeish, N. Jain, et al. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171,

    arXiv

  2. [2017]

    G. Wang, J. Li, Y . Sun, et al. Hierarchical reasoning model.arXiv preprint arXiv:2506.21734,

    arXiv

  3. [2019]

    Chollet, M

    F. Chollet, M. Knoop, G. Kamradt, and B. Landers. ARC Prize 2024: Technical Report.arXiv preprint arXiv:2412.04604,

    2024 arXiv

  4. [2020]

    F. Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547,

    1911 arXiv

  5. [2021]

    13 S. Hao, S. Sukhbaatar, D. Su, et al. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    arXiv

  6. [2024]

    Chollet, M

    F. Chollet, M. Knoop, G. Kamradt, B. Landers, and H. Pinkard. ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems.arXiv preprint arXiv:2505.11831,

    arXiv

  7. [2025]

    M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. InProceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    2021

  8. [2026]

    Saunshi, N

    N. Saunshi, N. Dikkala, Z. Li, S. Kumar, and S. J. Reddi. Reasoning with latent thoughts: On the power of looped transformers.arXiv preprint arXiv:2502.17416,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.