Pith. sign in

REVIEW 3 major objections 6 minor 29 references

A hard-barriered LLM inference gang can change core membership between tokens without deadlock or silent logit corruption, and on real silicon it Pareto-dominates static core partitions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 15:29 UTC pith:735MKBTG

load-bearing objection A real kernel mechanism for per-token elastic membership of a hard-barriered LLM gang, with silicon bit-exactness and a clean same-binary Pareto result over static partitions. the 3 major comments →

arxiv 2607.04668 v1 pith:735MKBTG submitted 2026-07-06 cs.OS cs.AIcs.PF

Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

Daeyeon Son This is my paper
classification cs.OS cs.AIcs.PF
keywords elastic gangLLM inferencehard-barrier schedulingACK-latched epochwork conservationcore membership changebit-exact logitsOS co-scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On-device LLM decoding is a hard-barriered SIMD job that wants many cores for each token, while ordinary OS processes want those same cores continuously. A naive barrier gang cannot join a preemptive scheduler: a core that leaves mid-token deadlocks the barrier, and a core that joins late can silently corrupt logits. This paper presents an elastic gang for a bare-metal kernel in which the inference gang is a first-class scheduled entity whose membership can change between any two tokens. The enabling protocol never waits on a named core: each token's participants are the intersection of cores the gang requested and cores that acknowledged the current epoch, snapshotted into one generation-tagged latch word. Un-acked cores are simply outside that token and join at most one token later; displaced processes migrate and keep running; cores return when the generation ends. On a real 8-core/16-thread machine, output is bit-exact under verified membership change on 135M and 7B models, and elastic membership beats fair static partitions at intermediate duty cycles while recovering idle cores and converging at saturation.

Core claim

Per-token membership change of a hard-barriered LLM inference gang is both safe and work-conserving: under an ACK-latched epoch protocol that never waits on a named core, greedy tokens and final logits remain byte-identical to fixed-width runs under verified churn, and elastic membership delivers substantially higher general throughput than fair static K-core splits at intermediate inference duty while matching or exceeding inference throughput, recovering stranded cores when idle, and converging at saturation.

What carries the argument

The ACK-latched epoch protocol: RCU/epoch-style membership consent composed with a seqlock-style generation-tagged participant latch. Each token's set is requested ∩ acked-current-epoch, published once into a single tagged word that every barrier and worker reads; stale acks die when the epoch bumps at generation start and end, and no barrier ever waits on a named core.

Load-bearing premise

Decode throughput saturates well below full machine width, so cores lent past that knee cost almost nothing for inference; if the engine scaled to the last core, lending would be a direct capacity trade instead of nearly free.

What would settle it

On the same silicon and engine, measure solo decode throughput versus gang width: if throughput keeps rising substantially past width 8 (or past the paper's measured knee) for the 7B or 135M model, then ceding those cores is no longer nearly free and the Pareto claim against static partitions fails as stated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents the elastic gang, a first-class kernel-schedulable entity in Anima OS (bare-metal x86-64 Rust) that co-schedules a hard-barriered CPU-SIMD LLM inference gang with a general preemptive process population. The core mechanism is an ACK-latched epoch protocol composed with a generation-tagged participant latch: each token's participant set is requested ∩ acked-current-epoch, snapshotted into one tagged word, so no barrier waits on a named core. Un-acked cores are outside the current token and join at most one token later; departures are effected by epoch bumps at generation boundaries. On an AMD Zen 5 (8C/16T) machine, the paper reports bit-exact tokens and final logits under verified membership change on 135M and 7B models (both width-churn and real lending drivers, with negative controls), and a same-binary ablation against fair static K-core partitions showing Pareto dominance on general throughput at intermediate inference duty cycles (1.75×/1.52×/1.28× static-8 at 25/50/75% duty) with full core recovery at idle and convergence at saturation. Return is sub-microsecond; busy-core acquisition costs one scheduling quantum by policy.

Significance. If the results hold, this is a concrete systems contribution at the intersection of OS scheduling and on-device LLM inference: it makes a hard-barriered SIMD gang a first-class, membership-mutable kernel entity without deadlock or silent logit corruption—properties that matter when logits feed a safety gate. Strengths that should be credited include (i) the never-wait-on-a-named-core invariant argument with a single latch-computation site, (ii) bit-exactness under verified churn with negative controls and a separate lending-path experiment, (iii) a same-binary static-mode restriction so the duty-cycle sweep is a policy ablation, (iv) replacement of a migration-contaminated virtual-time metric with a monotone served-quanta counter, and (v) an explicit applicability envelope (§5.8) that scopes the Pareto claim to the measured saturation knee. The work is more than a reallocation of fungible threads; it addresses the barrier-integrity problem that classic gang scheduling and microsecond core-reallocation systems deliberately avoid.

major comments (3)
  1. [Abstract, §1, §5.5] Abstract, title, and §1 frame the contribution as “per-token membership change,” but §5.5 states that mid-token (and mid-generation) departure is unrepresentable: epoch bumps—and therefore departures—occur only at generation boundaries, while joins land at token boundaries as acks arrive. The design is sound and the deadlock of §2.2(1) is avoided by construction, but the headline framing overstates bidirectionality. Please rephrase the abstract/intro (and any claim of “between any two tokens”) to match the actual schedule the protocol can produce: per-token joins, generation-boundary departures. This is load-bearing for what readers will believe the mechanism does.
  2. [§5.2, Table 2, Abstract] Table 2 / §5.2: the abstract and §5.2 claim elastic delivers its general-throughput gains “at equal or better inference throughput.” The paper itself attributes the large inference gap vs static-8 (e.g., 571.7 vs 232.4 fwd/s at duty 100%, where lending is quiescent) to SMT-sibling placement contention, not to the membership protocol, and correctly says headline dominance rests on the general axis. Once that confound is acknowledged, the “equal or better inference” half of the headline is not cleanly supported by a controlled comparison. Either (a) re-run or re-place static-8 so inference cores are not SMT-contended by the 24 tenants and report the matched numbers, or (b) drop/qualify the inference half of the Pareto claim in the abstract and §5.2 so the claim matches the unambiguous general-throughput axis.
  3. [§5.4, Table 4, §5.8, §7] The Pareto claim is correctly scoped in §5.8 to the precondition that decode saturates below machine width (Table 4 knee at width 8 of 16). That precondition is measured on this Zen 5 box for both a DRAM-bound 7B and a cache-resident 135M model, but absolute knee location and the 1.75×/1.52×/1.28× factors are machine- and engine-specific (§7). For a journal audience, please add a short sensitivity discussion: what changes if the knee is at width N−2 vs N/2, and what a deployer must measure before enabling lending. The safety/bit-exactness claim does not depend on the knee and can remain unqualified; the work-conservation multipliers should not be read as portable constants.
minor comments (6)
  1. [§5.1] §5.1 / workload: the 24 tenants are pure-compute AOT spin agents. A sentence on expected sensitivity to I/O-blocking or syscall-heavy general work (migration under blocking, queue dynamics) would help readers judge external validity without requiring new experiments.
  2. [§5.3, Table 3] §5.3 Linux anchor: the unpinned collapse is the robust external claim; the pinned-under-load and 16-thread cells are already flagged as noisy/timeout-prone at small n. Consider moving the fragile cells to an appendix or marking them more prominently so the directional anchor is not over-read.
  3. [§5.6] §5.6: return p50 = 0.22 µs and migration primitive p50 = 0.22 µs are numerically identical; the paper notes this is coincidence. A parenthetical code-path identifier or separate instrumentation note in the table/text would prevent reader suspicion of a copy error.
  4. [Figure 1, §3.1] Figure 1 is described in text but the manuscript dump does not include the rendered figure; ensure the published version shows the token sets {a}, {a,b}, {a,b} and the late-join of core b clearly.
  5. [§1, §2.1] Companion citations [27, 28] supply the engine and safety-gate context. A one-sentence self-contained summary of what the safety gate does with logits would help readers who do not have those drafts.
  6. [Abstract, title page, Listing 1] Typos / polish: “thrown into a preemptive scheduler” appears twice with slight wording drift (abstract vs §1); “July 2026 — draft” on the title page should be updated for camera-ready; ensure GANG_ACK / GANG_EPOCH notation is consistent between Listing 1 and §3.2.

Circularity Check

0 steps flagged

No significant circularity: protocol invariants are by construction and then measured on silicon; companion self-cites supply engine/safety context only.

full rationale

This is an empirical systems paper whose central results (bit-exact tokens/logits under verified per-token membership change on 135M and 7B; Pareto dominance of elastic membership over same-binary static-K partitions at intermediate duty cycles; sub-µs return and quantum-scale busy-core acquisition) are obtained by silicon measurement against an ablation of the shipped binary and a directional external Linux anchor. The ACK-latched epoch protocol and generation-tagged latch are designed so that membership = requested ∩ acked-current-epoch, with invariants I1–I8 argued from construction (epoch bumps at begin and end, single latch word, resync-before-ack, owner CAS, bracket flag on #PF paths); the bit-exactness experiments (width-churn and bitexact-lend) then falsifiably check that the construction holds, including negative controls. Companion citations [27,28] describe the in-kernel engine and the logit-reading safety gate that motivate treating a lost/extra core as a correctness event, but the measured bit-exactness and the work-conservation numbers do not reduce to those papers, to fitted parameters, or to a uniqueness claim. The saturation knee (Table 4) is measured, not assumed as a free parameter that forces the Pareto claim. No self-definitional loop, no fitted-input-called-prediction, and no load-bearing uniqueness imported from the author. Score 1 only for the ordinary presence of same-author context citations that are not required for the derivation of the headline results.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

Load-bearing content is a systems protocol plus silicon measurements, not a fitted physical law. Free parameters are policy knobs (quantum, migration pace, trust clamp) and the observed saturation knee. Axioms are standard concurrency patterns and domain facts about barriered matvecs. Invented entities are the protocol constructs themselves; independent evidence is the bit-exact and throughput measurements on real hardware, not an external collider-style prediction.

free parameters (4)
  • WRR scheduling quantum (~16 ms) = ~16 ms on measured Zen 5 setup
    Busy-core acquisition cost is defined as one quantum by policy (no mid-slice preemption for lending); absolute ms value is machine/scheduler-specific and shapes the amortized-cost claim.
  • Migration pace (≤2 migrations per scheduler pass) = ≤2 per pass
    Hand-chosen service-latency bound that affects how fast displaced tenants drain; not derived from first principles.
  • Trust weight clamp [1,4] for lend-admission quota = [1,4]
    Governance dial range chosen relative to max live trust; Table 5 sweep uses bench override of these weights.
  • Observed decode saturation knee (gang width 8) = width 8 of 16 threads
    Measured on this 8C/16T machine for two models; profitability of ceding cores past the knee depends on this empirical operating point, not a universal constant.
axioms (4)
  • domain assumption A hard-barriered SIMD gang deadlocks if a wait counts a named core that never arrives, and silently corrupts results if a core walks a token it was not latched into.
    Stated as failure modes (1)–(3) in §2.2; drives the never-wait-on-named-core design goal.
  • domain assumption Matvec row results are value-independent of which participant computes them, so work-stealing over a latched mask preserves logits if each row is computed exactly once.
    Invoked in §5.5 as the mechanism-level reason bit-exactness holds under membership change.
  • standard math Seqlock-style generation tags plus RCU/epoch-style consent can publish a consistent membership snapshot without waiting on named cores.
    Composition argued in §3.2–3.3; standard concurrency patterns applied to a new entity.
  • ad hoc to paper With no lendable core designated, the elastic path is dead code and bit-identical to the pre-elastic kernel (G5/I8).
    Regression-gated design goal used to make static mode a restriction of the same binary for fair ablation (§4.2).
invented entities (3)
  • ACK-latched epoch protocol (GANG_EPOCH / GANG_ACK / resync-before-ack) independent evidence
    purpose: Membership consent that never waits on a named core; stale acks die when epoch bumps at generation begin and end.
    Core mechanism of the paper; evidence is silicon bit-exactness and hang-freedom under churn, not independent external theory.
  • Generation-tagged participant latch word independent evidence
    purpose: Single snapshot (gen||mask) so wait-side and worker-side see the same set and straddled reads self-exclude.
    Closes silent-corruption failure mode; verified by bit-exact experiments with negative controls.
  • Elastic gang as first-class kernel-schedulable entity independent evidence
    purpose: Borrow/return cores around generations and change membership per token while co-scheduling general preemptive processes.
    Systems abstraction claimed novel on axes A1–A4; independent handle is the measured duty-cycle Pareto frontier on real hardware.

pith-pipeline@v1.1.0-grok45 · 24247 in / 3866 out tokens · 34246 ms · 2026-07-11T15:29:21.372723+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes." pith.science (2026). https://pith.science/paper/735MKBTG

@misc{pith2026260704668,
  author       = {Pith},
  title        = {Pith review of: Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/735MKBTG}},
  note         = {Machine review of arXiv:2607.04668}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.

Figures

Figures reproduced from arXiv: 2607.04668 by Daeyeon Son.

Figure 1
Figure 1. Figure 1: One generation under the ACK-latched epoch protocol. The participant set of each token is [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 linked inside Pith

  1. [1]

    Scheduling Techniques for Con- current Systems,

    J. K. Ousterhout, “Scheduling Techniques for Con- current Systems,” Proceedings of the 3rd Interna- tional Conference on Distributed Computing Sys- tems (ICDCS), 1982

    1982

  2. [2]

    Gang Scheduling Performance Benefits for Fine-Grain Synchroniza- tion,

    D. G. Feitelson and L. Rudolph, “Gang Scheduling Performance Benefits for Fine-Grain Synchroniza- tion,” Journal of Parallel and Distributed Comput- ing, 1992

    1992

  3. [3]

    Demand- Based Coscheduling of Parallel Jobs on Multipro- grammed Multiprocessors,

    P. G. Sobalvarro and W. E. Weihl, “Demand- Based Coscheduling of Parallel Jobs on Multipro- grammed Multiprocessors,” Job Scheduling Strate- gies for Parallel Processing (IPPS ’95 Workshop), LNCS 949, Springer, 1995

    1995

  4. [4]

    Flexible CoScheduling: Mitigating Load Imbalance and Improving Utilization of Het- erogeneous Resources,

    E. Frachtenberg, D. G. Feitelson, F. Petrini, and J. Fernandez, “Flexible CoScheduling: Mitigating Load Imbalance and Improving Utilization of Het- erogeneous Resources,” IPDPS 2003

    2003

  5. [5]

    Paired Gang Scheduling,

    Y. Wiseman and D. G. Feitelson, “Paired Gang Scheduling,” IEEE Transactions on Parallel and Distributed Systems, vol. 14, 2003

    2003

  6. [6]

    Process Control and Scheduling Issues for Multiprogrammed Shared- Memory Multiprocessors,

    A. Tucker and A. Gupta, “Process Control and Scheduling Issues for Multiprogrammed Shared- Memory Multiprocessors,” Proceedings of the 12th ACM Symposium on Operating Systems Principles (SOSP), 1989

    1989

  7. [7]

    A Dy- namic Processor Allocation Policy for Multipro- grammed Shared-Memory Multiprocessors,

    C. McCann, R. Vaswani, and J. Zahorjan, “A Dy- namic Processor Allocation Policy for Multipro- grammed Shared-Memory Multiprocessors,” ACM Transactions on Computer Systems, vol. 11, no. 2, 1993

    1993

  8. [8]

    Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism,

    T. E. Anderson, B. N. Bershad, E. D. Lazowska, and H. M. Levy, “Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism,” Proceedings of the 13th ACM Sym- posium on Operating Systems Principles (SOSP), 1991

    1991

  9. [9]

    Composing Parallel Software Efficiently with Lithe,

    H.Pan, B.Hindman, andK.Asanović, “Composing Parallel Software Efficiently with Lithe,” Proceed- ings of the ACM SIGPLAN Conference on Pro- gramming Language Design and Implementation (PLDI), 2010

    2010

  10. [10]

    Cal- listo: Co-Scheduling Parallel Runtime Systems,

    T. Harris, M. Maas, and V. J. Marathe, “Cal- listo: Co-Scheduling Parallel Runtime Systems,” Proceedings of the 9th European Conference on Computer Systems (EuroSys), 2014. 13

    2014

  11. [11]

    Greedy Reclama- tion of Unused Bandwidth in Constant-Bandwidth Servers,

    G. Lipari and S. Baruah, “Greedy Reclama- tion of Unused Bandwidth in Constant-Bandwidth Servers,” Proceedings of the 12th Euromicro Con- ference on Real-Time Systems (ECRTS), 2000

    2000

  12. [12]

    Deadline Scheduling in the Linux Kernel,

    J. Lelli, C. Scordino, L. Abeni, and D. Faggioli, “Deadline Scheduling in the Linux Kernel,” Soft- ware: Practice and Experience, vol. 46, no. 6, 2016

    2016

  13. [13]

    Heracles: Improving Resource Efficiency at Scale,

    D. Lo, L. Cheng, R. Govindaraju, P. Ranganathan, and C. Kozyrakis, “Heracles: Improving Resource Efficiency at Scale,” Proceedings of the 42nd In- ternational Symposium on Computer Architecture (ISCA), 2015

    2015

  14. [14]

    PAR- TIES: QoS-Aware Resource Partitioning for Mul- tiple Interactive Services,

    S. Chen, C. Delimitrou, and J. F. Martínez, “PAR- TIES: QoS-Aware Resource Partitioning for Mul- tiple Interactive Services,” Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Sys- tems (ASPLOS), 2019

    2019

  15. [15]

    Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Work- loads,

    A. Ousterhout, J. Fried, J. Behrens, A. Belay, and H. Balakrishnan, “Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Work- loads,” NSDI 2019

    2019

  16. [16]

    Caladan: Mitigating Interference at Microsecond Timescales,

    J. Fried, Z. Ruan, A. Ousterhout, and A. Belay, “Caladan: Mitigating Interference at Microsecond Timescales,” OSDI 2020

    2020

  17. [17]

    IX: A Protected Dataplane Operating System for High Throughput and Low Latency,

    A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion, “IX: A Protected Dataplane Operating System for High Throughput and Low Latency,” OSDI 2014

    2014

  18. [18]

    The CPU Scheduler in VMware vSphere 5.1,

    VMware, Inc., “The CPU Scheduler in VMware vSphere 5.1,” Technical White Paper (Performance Study), 2013

    2013

  19. [19]

    Orca: A Distributed Serving Sys- tem for Transformer-Based Generative Models,

    G.-I. Yu et al., “Orca: A Distributed Serving Sys- tem for Transformer-Based Generative Models,” OSDI 2022

    2022

  20. [20]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,

    A. Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” OSDI 2024. arXiv:2403.02310

    Pith/arXiv arXiv 2024

  21. [21]

    MuxServe: Flexible Spatial- Temporal Multiplexing for Multiple LLM Serving,

    J. Duan et al., “MuxServe: Flexible Spatial- Temporal Multiplexing for Multiple LLM Serving,” ICML 2024. arXiv:2404.02015

    Pith/arXiv arXiv 2024

  22. [22]

    GitHub repository (unpublished), 2026

    coconutOS: a bare-metal Rust microkernel for AI inference. GitHub repository (unpublished), 2026. https://github.com/coconut-os/coconutOS

    2026

  23. [23]

    Efficient Memory Management for Large Language Model Serving with PagedAtten- tion,

    W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAtten- tion,” SOSP 2023

    2023

  24. [24]

    AIOS: LLM Agent Operating Sys- tem,

    K. Mei et al., “AIOS: LLM Agent Operating Sys- tem,” COLM 2025

    2025

  25. [25]

    llama.cpp: LLM inference in C/C++,

    G. Gerganov, “llama.cpp: LLM inference in C/C++,” GitHub repository, 2023

    2023

  26. [26]

    The- seus: An Experiment in Operating System Struc- ture and State Management,

    K. Boos, N. Liyanage, R. Ijaz, and L. Zhong, “The- seus: An Experiment in Operating System Struc- ture and State Management,” OSDI 2020

    2020

  27. [27]

    ProbeLogits: Kernel-Level LLM Infer- ence Primitives for AI-Native Operating Systems,

    D. Son, “ProbeLogits: Kernel-Level LLM Infer- ence Primitives for AI-Native Operating Systems,” arXiv:2604.11943, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Governed MCP: Kernel-Level Tool Gover- nance for AI Agents via Logit-Based Safety Prim- itives,

    D. Son, “Governed MCP: Kernel-Level Tool Gover- nance for AI Agents via Logit-Based Safety Prim- itives,” arXiv:2604.16870, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    Lottery Scheduling: Flexible Proportional-Share Resource Management,

    C. A. Waldspurger and W. E. Weihl, “Lottery Scheduling: Flexible Proportional-Share Resource Management,” OSDI 1994. 14

    1994