Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Bernardo L. Sabatini; Gyuryang Heo; Kazuki Irie; Samuel J. Gershman; Timothy Ngotiaoco

Depth in parallelizable sequence models expands expressivity through successive Lie algebra extensions, and approximation error falls exponentially with depth.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 14:38 UTC pith:LXAXS42A

load-bearing objection Wrong full text was supplied for 2603.05573; the Lie-algebra depth–error claims are uncheckable beyond the abstract. the 2 major comments →

arxiv 2603.05573 v2 pith:LXAXS42A submitted 2026-03-05 cs.LG

Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Gyuryang Heo , Timothy Ngotiaoco , Kazuki Irie , Samuel J. Gershman , Bernardo L. Sabatini This is my paper

classification cs.LG

keywords sequence modelsLie algebraexpressivitydepthTransformersstate-space modelsapproximation errorparallelism

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Parallelizable sequence models such as Transformer variants and structured state-space models gain training speed by limiting what each layer can do. This paper argues that those limits can be described with Lie-algebraic control theory: each added layer of depth corresponds to one more extension in a tower of Lie algebras, so constant-depth models inhabit a fixed, limited expressivity class. Outside that class the models cannot represent the target map exactly, yet the approximation error is shown to shrink exponentially as depth grows. The exponential bound matches the strong practical performance of deeper models and is checked on symbolic word problems and continuous state-tracking tasks. A reader who cares about efficient sequence architectures therefore obtains a precise reason why stacking more parallel layers continues to buy capability rather than merely adding parameters.

Core claim

There is a direct correspondence between the depth of a parallelizable sequence model and a tower of Lie algebra extensions. Constant-depth models therefore occupy a characterized Lie-algebraic expressivity class with sharp bounds; when a target function lies outside that class, the approximation error of the model decays exponentially with depth.

What carries the argument

The depth-to-Lie-algebra-tower correspondence, which classifies the expressivity of constant-depth layers and supplies an analytic approximation-error bound that decreases exponentially in depth.

Load-bearing premise

The Lie-algebraic control abstraction of each parallel sequence layer is faithful enough that the derived exponential depth-error bound still holds for the actual Transformer and state-space architectures used in the experiments.

What would settle it

Measure approximation error versus depth on the paper’s own symbolic-word and continuous state-tracking tasks; if the observed error fails to fall exponentially (or if a constant-depth model systematically exceeds the claimed Lie-algebraic class), the central bound is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Constant-depth parallel models cannot realize maps outside a fixed Lie-algebraic class no matter how wide they are made.
Adding depth is the systematic way to enlarge the representable class, with error guaranteed to drop exponentially.
Architecture design can treat depth as the primary lever for expressivity once the Lie class of a single layer is known.
Empirical gains from deeper Transformers and SSMs on state-tracking tasks are predicted rather than accidental.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the exponential bound is tight, practitioners can choose the minimal depth that meets a target error without exhaustive search.
The same tower construction may apply to other parallel sequence primitives (linear attention, selective SSMs) once their generators are identified.
A natural next measurement is whether the observed error slope matches the analytic rate on longer or higher-dimensional tracking problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Wrong full text was supplied for 2603.05573; the Lie-algebra depth–error claims are uncheckable beyond the abstract.

read the letter

The arXiv id and abstract point to a theory paper on depth in parallelizable sequence models (Transformers/SSMs) via Lie-algebraic control: a depth ↔ tower-of-extensions correspondence, a characterization of constant-depth expressivity, and an analytic claim that approximation error falls exponentially with depth, plus symbolic and continuous state-tracking experiments. That package, if the proofs land, would be a useful organizing result for people who care about why depth still matters when you insist on sequence-level parallelism.

What we actually received as “full manuscript” is a different paper entirely—PRISM (robotics IL→RL refinement with Eureka-style language rewards, pick-and-place in Isaac Lab). No Lie algebras, no sequence-model depth towers, no error bounds. So the load-bearing steps—the modeling map from layers to controlled systems on a Lie algebra, regularity conditions, the derivation of the exponential bound, and the experimental protocols—are invisible. The Reader and stress-test are right on this; it is not a soft spot in the theory, it is a missing manuscript.

From the abstract alone I can only say the framing is coherent and not obviously circular, and that “echoing recent constant-depth expressivity work + analytic depth-error law” is a plausible incremental contribution rather than a paradigm shift. I cannot score soundness, citation pattern, or whether the abstraction is faithful to real Transformers/SSMs. Free parameters and invented entities cannot be audited either.

Who this is for, if the real paper matches the abstract: theory-minded sequence-model people and architecture designers who want analytic reasons for depth under parallelism constraints. On the materials in hand, I would not bring it to reading group, would not cite it, and I cannot treat it as peer-review-ready evidence. If the correct PDF for 2603.05573 appears with the claimed derivations and experiments, that version would deserve a serious referee. Until then, do not spend time on this bundle.

Referee Report

2 major / 0 minor

Summary. The abstract of arXiv:2603.05573 claims a Lie-algebraic control analysis of parallelizable sequence models (Transformer variants and structured SSMs). It asserts a correspondence between model depth and a tower of Lie-algebra extensions, a characterization of the expressivity class of constant-depth models, and an analytic approximation-error bound that decays exponentially with depth, with validation on symbolic word and continuous state-tracking tasks. The body supplied under this paper ID is instead the unrelated PRISM manuscript (arXiv:2603.05574), a robotics pipeline that initializes imitation policies from teleoperated demonstrations and refines them with Eureka-style LLM-generated rewards plus sparse human feedback for pick-and-place personalization. No Lie-algebraic theory, depth-error bounds, or sequence-model experiments appear in the provided text.

Significance. If the abstract claims of 2603.05573 were substantiated by correct derivations and faithful modeling of Transformers/SSMs, the work would be significant for the theory of efficient sequence models: an exponential depth-error law would give a concrete design principle for depth versus parallelism. The PRISM body that was actually supplied is a competent but incremental hybrid IL-RL system; its main empirical result (96.8 % success on a constrained pick-and-place personalization) is useful for robotics practitioners yet does not address the Lie-algebraic claims advertised by the title and abstract under review.

major comments (2)

Manuscript identity failure: the title, abstract and arXiv identifier announce a Lie-algebraic theory of depth in parallelizable sequence models, yet the full text is the PRISM robotics paper (arXiv:2603.05574). No theorem, equation, regularity condition or experiment from the claimed contribution is present, so the central claims (depth ↔ tower of Lie-algebra extensions; constant-depth expressivity class; exponential approximation-error decay) cannot be verified or refereed.
Even if the abstract is taken at face value, the load-bearing faithfulness assumption—that the Lie-algebraic control abstraction of parallelizable layers is tight enough for the derived exponential bound to apply to actual Transformer/SSM architectures and the named validation tasks—remains uninspectable because the modeling map, regularity conditions and proof steps are absent from the supplied document.

Circularity Check

0 steps flagged

No circularity can be established: supplied full text is a different paper (PRISM robotics), so the Lie-algebraic depth–error derivation chain is uninspectable.

full rationale

The claimed paper (arXiv:2603.05573) asserts a correspondence between sequence-model depth and a tower of Lie algebra extensions, a characterization of constant-depth expressivity, and an analytic approximation-error bound that decays exponentially with depth. Circularity analysis requires walking that derivation chain equation-by-equation. The CACHEABLE PAPER SOURCE CONTEXT, however, contains the full manuscript of an unrelated work (PRISM, arXiv:2603.05574) on imitation/RL refinement for robotic manipulation; it contains no Lie algebras, sequence models, depth towers, or error bounds. Only the abstract of 2603.05573 is available. From the abstract alone there is no self-definitional loop, no fitted parameter renamed as prediction, and no load-bearing self-citation uniqueness claim visible. Per the hard rules, circularity may be asserted only when a specific reduction can be quoted from the paper; none can. Therefore the honest finding is score 0 with empty steps: the derivation is simply not present to inspect, and the abstract does not exhibit circularity by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

Abstract-only review: free parameters, axioms, and invented entities cannot be exhaustively extracted from proofs or experiments. Listed items are those implied by the abstract's framing of parallelizable sequence models under a Lie-algebraic control perspective.

axioms (3)

domain assumption Parallelizable sequence layers (Transformer variants / structured SSMs) can be modeled as controlled dynamical systems whose expressivity is governed by Lie-algebraic generation.
Central modeling premise of the abstract; without it the depth–Lie-tower correspondence does not attach to real architectures.
domain assumption Constant-depth models occupy a restricted Lie-algebraic class with nontrivial expressivity bounds outside which approximation is needed.
Stated as echoing recent theory; treated as background for the exponential depth-error claim.
standard math Standard Lie-algebra / control-theoretic constructions (extensions, generated algebras) apply to the discrete layered sequence-model setting.
Implied mathematical toolkit; details not available in abstract.

invented entities (1)

Tower of Lie algebra extensions indexed by sequence-model depth no independent evidence
purpose: Formal correspondence used to explain how depth expands expressivity of parallelizable sequence models and to derive error scaling.
Abstract presents this correspondence as the paper's theoretical formulation; independent evidence outside the paper is not available from the abstract.

pith-pipeline@v1.1.0-grok45 · 12556 in / 2197 out tokens · 30400 ms · 2026-07-15T14:38:53.501306+00:00 · methodology

0 comments

read the original abstract

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 3 linked inside Pith

[1]

Ankile et al

L. Ankile et al. From Imitation to Reﬁnement – Residual RL f or Precise Assembly, December 2024. arXiv:2407.16677 [cs]

Pith/arXiv arXiv 2024
[2]

Billard et al

A.G. Billard et al. In B. Siciliano et al., editors, �� , pages 1995–2014. Springer, Cham, 2016

1995
[3]

Canal et al

G. Canal et al. Personalization Framework for Adaptive Ro botic Feeding Assis- tance. In Arvin Agah et al., editors, �� , pages 22–31, Cham, 2016. Springer International Publishing

2016
[4]

Codevilla et al

F. Codevilla et al. End-to-end driving via conditional im itation learning. In �� , 2018

2018
[5]

Dulac-Arnold et al

D. Dulac-Arnold et al. Challenges of real-world reinforc ement learning. In �� , 2019. identiﬁes sample eﬃciency, safety, reward design constraints

2019
[6]

Haarnoja et al

T. Haarnoja et al. Soft actor-critic: Oﬀ-policy maximum e ntropy deep reinforce- ment learning with a stochastic actor. In �� , 2018

2018
[7]

L. P. Kaelbling et al. Reinforcement learning: A survey. �� , 4:237–285, 1996. earlier classic survey of RL

1996
[8]

Jason Ma et al

Y. Jason Ma et al. Eureka: Human-Level Reward Design via Co ding Large Lan- guage Models, October 2023. arXiv:2310.12931 [cs]

Pith/arXiv arXiv 2023
[9]

Mandlekar et al

A. Mandlekar et al. What matters in learning from oﬄine hum an demonstrations for robot manipulation. �� , 2021

2021
[10]

Maroto-G´ omez et al

M. Maroto-G´ omez et al. An adaptive decision-making sys tem supported on user preference predictions for human–robot interactive commu nication. �� , 33(2):359–403, April 2023

2023
[11]

Nair et al

A. Nair et al. Overcoming exploration in reinforcement l earning with demonstra- tions. �� , 2017

2017
[12]

Nvidia omniverse isaac sim (versio n 5.0.0): Documentation and reference application for physically based robot simul ation, 2025

NVIDIA Corporation. Nvidia omniverse isaac sim (versio n 5.0.0): Documentation and reference application for physically based robot simul ation, 2025

2025
[13]

Rajeswaran et al

A. Rajeswaran et al. Learning complex dexterous manipul ation with deep rein- forcement learning and demonstrations. �� , 2017

2017
[14]

Schulman et al

J. Schulman et al. Proximal Policy Optimization Algorit hms. �� , abs/1707.06347, 2017. arXiv: 1707.06347

Pith/arXiv arXiv 2017
[15]

Shenfeld et al

I. Shenfeld et al. Tgrl: An algorithm for teacher guided r einforcement learning. In �� , volume 202, pages 31077–31093. PMLR, 2023

2023
[16]

Tang et al

C. Tang et al. Deep reinforcement learning for robotics: A survey of real-world successes. �� , 2024. survey emphasising successes in simulation and real-world robotics

2024
[17]

Reconciling Reality through Simulat ion: A Real-to-Sim-to-Real Approach for Robust Manipulation, March 2024

Marcel Torne et al. Reconciling Reality through Simulat ion: A Real-to-Sim-to-Real Approach for Robust Manipulation, March 2024. arXiv:2403. 03949 [cs] version: 1

2024
[18]

Yu et al

C. Yu et al. Icpl: Few-shot in-context preference learni ng via large language models. �� , 2024

2024
[19]

Yuan et al

X. Yuan et al. Policy Decorator: Model-Agnostic Online R eﬁnement for Large Policy Model. �� , 2025:28129– 28164, May 2025

2025
[20]

Yuan et al

Y. Yuan et al. Uni-rlhf: Universal platform and benchmar k suite for reinforcement learning with diverse human feedback. �� , 2024

2024

[1] [1]

Ankile et al

L. Ankile et al. From Imitation to Reﬁnement – Residual RL f or Precise Assembly, December 2024. arXiv:2407.16677 [cs]

Pith/arXiv arXiv 2024

[2] [2]

Billard et al

A.G. Billard et al. In B. Siciliano et al., editors, �� , pages 1995–2014. Springer, Cham, 2016

1995

[3] [3]

Canal et al

G. Canal et al. Personalization Framework for Adaptive Ro botic Feeding Assis- tance. In Arvin Agah et al., editors, �� , pages 22–31, Cham, 2016. Springer International Publishing

2016

[4] [4]

Codevilla et al

F. Codevilla et al. End-to-end driving via conditional im itation learning. In �� , 2018

2018

[5] [5]

Dulac-Arnold et al

D. Dulac-Arnold et al. Challenges of real-world reinforc ement learning. In �� , 2019. identiﬁes sample eﬃciency, safety, reward design constraints

2019

[6] [6]

Haarnoja et al

T. Haarnoja et al. Soft actor-critic: Oﬀ-policy maximum e ntropy deep reinforce- ment learning with a stochastic actor. In �� , 2018

2018

[7] [7]

L. P. Kaelbling et al. Reinforcement learning: A survey. �� , 4:237–285, 1996. earlier classic survey of RL

1996

[8] [8]

Jason Ma et al

Y. Jason Ma et al. Eureka: Human-Level Reward Design via Co ding Large Lan- guage Models, October 2023. arXiv:2310.12931 [cs]

Pith/arXiv arXiv 2023

[9] [9]

Mandlekar et al

A. Mandlekar et al. What matters in learning from oﬄine hum an demonstrations for robot manipulation. �� , 2021

2021

[10] [10]

Maroto-G´ omez et al

M. Maroto-G´ omez et al. An adaptive decision-making sys tem supported on user preference predictions for human–robot interactive commu nication. �� , 33(2):359–403, April 2023

2023

[11] [11]

Nair et al

A. Nair et al. Overcoming exploration in reinforcement l earning with demonstra- tions. �� , 2017

2017

[12] [12]

Nvidia omniverse isaac sim (versio n 5.0.0): Documentation and reference application for physically based robot simul ation, 2025

NVIDIA Corporation. Nvidia omniverse isaac sim (versio n 5.0.0): Documentation and reference application for physically based robot simul ation, 2025

2025

[13] [13]

Rajeswaran et al

A. Rajeswaran et al. Learning complex dexterous manipul ation with deep rein- forcement learning and demonstrations. �� , 2017

2017

[14] [14]

Schulman et al

J. Schulman et al. Proximal Policy Optimization Algorit hms. �� , abs/1707.06347, 2017. arXiv: 1707.06347

Pith/arXiv arXiv 2017

[15] [15]

Shenfeld et al

I. Shenfeld et al. Tgrl: An algorithm for teacher guided r einforcement learning. In �� , volume 202, pages 31077–31093. PMLR, 2023

2023

[16] [16]

Tang et al

C. Tang et al. Deep reinforcement learning for robotics: A survey of real-world successes. �� , 2024. survey emphasising successes in simulation and real-world robotics

2024

[17] [17]

Reconciling Reality through Simulat ion: A Real-to-Sim-to-Real Approach for Robust Manipulation, March 2024

Marcel Torne et al. Reconciling Reality through Simulat ion: A Real-to-Sim-to-Real Approach for Robust Manipulation, March 2024. arXiv:2403. 03949 [cs] version: 1

2024

[18] [18]

Yu et al

C. Yu et al. Icpl: Few-shot in-context preference learni ng via large language models. �� , 2024

2024

[19] [19]

Yuan et al

X. Yuan et al. Policy Decorator: Model-Agnostic Online R eﬁnement for Large Policy Model. �� , 2025:28129– 28164, May 2025

2025

[20] [20]

Yuan et al

Y. Yuan et al. Uni-rlhf: Universal platform and benchmar k suite for reinforcement learning with diverse human feedback. �� , 2024

2024