Pith. sign in

REVIEW 2 major objections 6 minor 15 references

Mathematically equivalent expert-reduction orders can drive different sparse-MoE trajectories, routes, and continuation texts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 17:54 UTC pith:5Q46D4WT

load-bearing objection Tight freeze-fork isolation shows expert reduction order can really change MoE trajectories; scoped claim is solid, external incidence is not claimed. the 2 major comments →

arxiv 2607.28097 v1 pith:5Q46D4WT submitted 2026-07-30 cs.LG

From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference

Tianyang Zhu This is my paper
classification cs.LG
keywords sparse mixture of expertsfloating-point reductionnumerical reproducibilitydeterministic inferenceruntime conformancepersistent-state conformanceexpert aggregationMoE routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows that how a sparse mixture-of-experts model adds up its expert outputs is not a harmless implementation detail. Even when the selected experts, gates, and inputs are frozen, changing only the floating-point reduction order or the precision of operands and accumulators can produce different hidden states, different later routing choices, and different greedy continuations. In a native DeepSeek-V4-Flash path, exhaustive single-layer order sweeps collapse into a handful of text basins, including opposite workforce-direction stories under one Chinese prompt, and longer horizons reveal more text separations. A BF16-operand / FP32-accumulator contract matches the native reference and stays order-stable on the tested six-expert states, while other contracts do not. Exact reconstruction of the post-mHC state reproduces the rest of a decode forward, and exact reconstruction of the full attention-persistent state at a decode boundary reproduces the branch across later tokens when the next input matches. The practical claim is that operand conversion, accumulator precision, and reduction order belong in the numerical compatibility contract for sparse-MoE runtimes and hardware backends—as controlled causal possibility, not a count of how often production hardware hits each order.

Core claim

With local MoE state held fixed, mathematically equivalent cross-expert reduction orders alone can select different deterministic sparse-MoE executions: distinct post-mHC states, routes, full persistent decode state, and continuation-text basins. Post-mHC is a sufficient intra-token boundary and full persistent state a sufficient cross-token boundary for reproducing a controlled divergent branch when the next input is unchanged; identical emitted tokens therefore need not imply identical autoregressive state.

What carries the argument

Trace-freeze-fork intervention: freeze prefix, experts, gates, weighted terms, shared output, and persistent state at one MoE site, then vary only aggregation semantics across four schemes (P32/C/A/B) that separate operand representation from accumulator precision, plus exact endpoint substitution of post-mHC or full persistent state to test boundary sufficiency.

Load-bearing premise

Findings from one native CPU runtime, one six-expert checkpoint, and exploratory forks and prompts are enough to treat reduction order and precision as a general sparse-MoE compatibility requirement, even without measuring how often real accelerators realize the sensitive orders.

What would settle it

On the same frozen local MoE state, show that all 720/360 same-mode reduction orders produce bitwise-identical post-mHC states, routes, persistent decode state, and greedy tokens as the native path—or that reconstructing the measured post-mHC or full persistent endpoint fails to reproduce the downstream branch.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Sparse-MoE backend qualification should test persistent-state, layer-state, and routing conformance, not only final tokens.
  • Operand conversion, accumulator precision, and merge order must be specified as part of the runtime/hardware contract, not left as interchangeable kernel choices.
  • A hierarchical conformance path can start at decode-boundary persistent-state fingerprints, then bisect by layer post-mHC and operator intermediates.
  • Identical greedy tokens can still hide divergent attention-persistent state that surfaces only after later decode steps.
  • BF16 expert operands with protected FP32 cross-expert accumulation is a scoped native-compatible stabilization contract on the evaluated six-term states.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Distributed or arrival-order expert combines may turn completion order into reduction order unless merge trees or higher-precision accumulators are fixed.
  • Serving stacks that shard experts across devices or fuse kernels without a stated reduction contract may silently disagree on long-horizon behavior under greedy decoding.
  • Tolerance-based cross-backend checks will need canonical persistent-state serialization before bitwise or near-bitwise conformance is even well-defined.
  • Delayed divergence implies regression tests that stop at short prefixes can miss order-induced behavioral splits that only appear deeper in generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper shows that, with local MoE state frozen in native DeepSeek-V4-Flash, mathematically equivalent cross-expert reduction orders alone can change post-mHC states, routes, full persistent decode state, and greedy continuation text. Four schemes (P32/C/A/B) separate operand representation from accumulator precision. Exhaustive single-layer forks map 720 A orders to 10 text basins and 360 B structural classes to 11 basins; under one Chinese prompt the B classes split into opposing hiring/layoffs continuations. Persistent ablations (192 schedules × 4 schemes) find P32/A/B alter all native route trajectories while C preserves routes, tokens, and texts and matches native intermediates bitwise. Exact post-mHC and FP64 full-persistent-state reconstructions reproduce one controlled B branch intra-token and across seven later decode steps. The authors conclude that operand conversion, accumulator precision, and reduction order belong in sparse-MoE numerical compatibility contracts, scoped strictly to controlled causal possibility rather than deployment incidence.

Significance. If the result holds under the stated scope, it is a concrete and practically important contribution to MoE inference systems: reduction order and low-precision combine semantics are not interchangeable kernel details but part of execution semantics near discrete router and argmax boundaries, including delayed divergence that survives identical emitted tokens via persistent attention state. Strengths include exhaustive 6! enumeration, same-mode vs native reference axes, bitwise C checks over 192 trajectories, two endpoint-sufficiency reconstructions with pre-fork identity controls, stratified long-form event-direction extensions with exact token-ID replay, and an unusually careful claim ledger and artifact manifest. The work usefully links floating-point reduction non-associativity to MoE routing and autoregressive state in a way prior nondeterminism and MoE-quantization literature has not isolated.

major comments (2)
  1. [§3.5 Exploratory breadth] §3.5 and Abstract: the 12/24/36-of-50 separation counts are obtained by injecting the maximum-L∞ B branch at each prompt’s captured state, not a random or production-like order. That design answers an existence/horizon question under adversarial branch choice. The abstract sentence is literally correct, but without an adjacent random-order or native-schedule control on the same 50 prompts, readers can easily over-read these fractions as typical breadth. Please state the selection rule in the abstract-level claim and, if feasible, report a paired random-class or identity-order baseline so the exploratory breadth result cannot be mistaken for incidence.
  2. [§3.2.1, §4.2] §3.2.1–§4.2: the representative trace shows MoE → post-mHC → route change → delayed token change, and the two reconstructions establish endpoint sufficiency, but there is still no frozen-route mediation control. The paper correctly refuses to claim routing is the unique mediator; however, the propagation schematic in §4.2 still presents “later layer/route divergence” as a necessary link in the measured sequence. Either add a minimal freeze-top-k (or force-route) control on one branch, or rephrase the schematic so route change is clearly an observed correlate rather than an implied causal step required for the central claim.
minor comments (6)
  1. [§2.4.7 / §3.2] §2.4.7 already discloses exploratory, non-preregistered depth prompts and hard-coded layer 5; consider moving one sentence of that disclosure into the main Results lead-in so basin and event-direction findings are not skimmed as a pre-specified benchmark suite.
  2. [Figure 2, §3.3] Figure 2 panel b and §3.3: the hiring count correction (110→113) is clear in text; ensure the alluvial figure caption states that widths use post-consensus class volumes so the figure cannot disagree with the 202/113/45 table.
  3. [§2.5] §2.5 equations for A/B initialize the accumulator as +0 in BF16 then repeatedly promote to FP32; a one-line note that this matches the probe’s software BF16 container (and is not claiming IEEE BF16 hardware add) would prevent backend readers from over-generalizing the contract.
  4. [§4.5, §6] §4.5 L0–L4 hierarchy is a useful diagnostic proposal but explicitly unvalidated cross-backend; keep it clearly labeled as a procedure motivated by endpoint sufficiency, not as an evaluated contribution in the Conclusion bullet list.
  5. [Appendix B.1, passim] Typos/encoding: several prompt strings and Appendix B.1 continuations appear mojibake-corrupted in the manuscript text (e.g., Chinese prompt and streamed continuations). Replace with correct UTF-8 before production.
  6. [References, Appendix A] References include very recent/adjacent preprints; double-check citation keys and that DeepSeek-V4 / mHC / Colibri identifiers match the public artifacts named in the reproducibility appendix.

Circularity Check

0 steps flagged

No significant circularity: empirical freeze-fork interventions with external native-reference checks, not definitional or fitted predictions.

full rationale

This is a controlled systems/experimental paper, not a first-principles derivation that could close on its inputs. The load-bearing chain is interventional: freeze local MoE operands/gates/prefix, vary only aggregation semantics (P32/C/A/B and permutations), and measure post-mHC, routes, persistent state, tokens, and text against an unmodified native reference and same-mode canonical references. Scheme C’s native compatibility is established by bitwise identity of MoE, post-mHC, next-router, and LM states over 192 trajectories, not by defining C as the native path. The 360 B structural classes follow from an explicit finite-precision symmetry (0⊕xi=xi and xi⊕xj=xj⊕xi under the stated BF16 rule), which is a quotient of the enumeration space rather than a fitted law. Endpoint reconstructions (post-mHC ΔH; FP64 full-persistent ΔS) test causal sufficiency of captured boundaries by additive injection with pre/post bitwise gates; agreement with the measured branch is an empirical outcome, not forced by construction of the labels. Basin volumes and event-direction counts are explicitly permutation/class measures, not posteriors. Citations to DeepSeek/mHC/Colibri supply model and runtime context; none import a uniqueness theorem or ansatz that forbids alternatives. No fitted parameter is relabeled a prediction, and no central claim reduces to self-citation. Ordinary use of the authors’ instrumented runtime is experimental dependence, not circularity under the stated criteria.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 4 invented entities

Load-bearing background is standard floating-point non-associativity, the DeepSeek-V4-Flash/mHC runtime semantics, and the experimental contract that only routed reduction order/precision change. No physical constants or fitted scientific parameters. Invented entities are methodological constructs (schemes, basins, state boundaries), not new ontological objects.

free parameters (3)
  • experiment/request seeds (e.g. 20260722, 20260723) and 64 layer-static schedules = seed 20260722 (persistent/breadth); 20260723 (event-direction sample)
    Choose which permutations and breadth cohort are evaluated; not fitted to maximize a scientific constant, but they define the measured sample.
  • fork layer and depth/breadth prompt set = layer 5; 3 depth prompts + 50 breadth prompts
    Layer 5 and exploratory prompts are discovery choices, hard-coded rather than preregistered or swept as a population sample.
  • generation horizons (8/16/32/64 tokens) = 8 primary; staged 16/32; 64 event-direction
    Horizon choices affect observed separation counts; cumulative breadth is survivor-conditioned by design.
axioms (5)
  • standard math Floating-point addition is non-associative; permutation-equivalent real sums need not be bit-identical after rounded accumulation.
    Background for entire intervention; stated in Introduction and §2.2.
  • domain assumption Native Colibri non-DSpark path with stated BF16 software rounding (bias 0x7fff+lsb, ties-to-even) and FP32 componentwise adds implements the reference MoE combine under test.
    §2.2.3; results are runtime-specific by construction.
  • domain assumption Under observed finite non-NaN/Inf BF16 operands, B has exact first-two-operand symmetry, yielding 360 structural classes from 720 permutations.
    §2.6 structural proof restricted to tested operands and rounding rule.
  • domain assumption Greedy decoding plus top-k routing make small numerical state differences capable of discrete trajectory changes; identical emitted tokens need not imply identical persistent state.
    §2.2.2 decision/persistent boundary framing used to interpret delayed divergence.
  • ad hoc to paper Trace-freeze-fork holds prefix, experts, gates, terms, shared output, and persistent state fixed so only aggregation semantics cause downstream differences.
    Core experimental axiom of §2.4.1; validity of causal attribution rests on this isolation.
invented entities (4)
  • P32/C/A/B aggregation schemes independent evidence
    purpose: Separate expert-operand representation from cross-expert accumulator precision relative to native reference.
    Defined in §2.5 as controlled probe contracts, not claimed universal hardware semantics.
  • Continuation-text basins / structural B classes independent evidence
    purpose: Compress many reduction orders into discrete behavioral equivalence sets under permutation/class measure.
    Operational partitions of experimental outcomes (§3.2), not latent model ontology.
  • post-mHC intra-token boundary and full persistent-state cross-token boundary independent evidence
    purpose: Name state endpoints shown sufficient to reproduce a controlled branch within and across decode steps.
    Supported by reconstruction controls §3.2.2–3.2.3; runtime-specific state bundles.
  • Hierarchical runtime-conformance levels L0–L4 no independent evidence
    purpose: Propose diagnostic order from tokens to operator intermediates for backend qualification.
    Motivated procedure in §4.5; not cross-backend validated in this paper.

pith-pipeline@v1.2.0-daily-grok45 · 24384 in / 3860 out tokens · 73955 ms · 2026-07-31T17:54:04.099167+00:00 · methodology

0 comments
read the original abstract

Mathematically equivalent expert-reduction orders can produce observably different sparse-MoE executions. We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics. Four schemes separate operand representation from accumulator precision. At one layer-5 fork, 720 A-mode orders yield 10 continuation basins; 720 B-mode orders form 360 exact structural classes and 11 basins. Under one Chinese prompt, the B classes split into 202 layoffs, 113 hiring, and 45 other continuations. Maximum-L-infinity B-branch selection separates 12, 24, and 36 of 50 prompts by 8, 16, and 32 tokens. Across 192 persistent trajectories per scheme, P32, A, and B change every native-reference route trajectory, while C preserves routes, token sequences, and texts. A separate 192-trajectory C check matches native MoE, post-mHC, next-router, and LM states bitwise. For one controlled B branch, exact post-mHC endpoint reconstruction reproduces the measured downstream trajectory. At the next decode boundary, exact FP64 reconstruction of the branch's full persistent state yields agreement for 301 downstream post-mHC states, 301 persistent-state checkpoints, 301 routes, predictions, and text over seven steps, given the same naturally generated next input. These controls identify post-mHC as an intra-token boundary and full persistent state as a cross-token continuation boundary. Identical tokens need not imply identical autoregressive state: divergence can survive a token boundary and become visible later. These results make expert operand conversion, accumulator precision, and reduction order part of a numerical compatibility contract for sparse-MoE runtimes and hardware backends. They establish controlled causal possibility, not deployment incidence; C's order invariance is limited to evaluated six-term states and schedules.

Figures

Figures reproduced from arXiv: 2607.28097 by Tianyang Zhu.

Figure 1
Figure 1. Figure 1: Two complementary endpoint controls for one B class. Exact post-mHC reconstruc￾tion matches the branch endpoint and downstream trajectory, establishing an intra-token state boundary. At the following decode boundary, exact FP64 additive reconstruction of the full persistent state matches the branch endpoint; under the same independently generated next in￾put token, 301/301 downstream post-mHC states, 301/3… view at source ↗
Figure 2
Figure 2. Figure 2: Panel a compares normalized continuation-basin rank-frequency profiles for the“why the sheep”fork; point labels retain the exact A-permutation and B-structural-class counts. In panel b, the Chinese-prompt experiment first reduces 720 raw B permutations to 360 structural classes by the first-two-operand symmetry quotient. Alluvial widths begin only at the 360-class measure and continue through six unique ei… view at source ↗
Figure 3
Figure 3. Figure 3: Panel a reports the identity-order scheme shift against three production-native prompt baselines. Panel b reports random-order divergence against both the native reference and each scheme’s identity order over 192 paired trajectories per scheme; lower is better. The sepa￾rately grouped C numerical check has 0/192 mismatches across captured MoE, post-mHC, next￾router, and LM states. Intermediates were not c… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

  1. [1]

    JiayiYuan,HaoLi,XinhengDing,WenyaXie,Yu-JheLi,WentianZhao,KunWan,JingShi, Xia Hu, and Zirui Liu.oUnderstanding and Mitigating Numerical Sources of Nondeter- minism in LLM Inference.parXiv:2506.09501v2, 2025.https://arxiv.org/abs/2506.09501

    arXiv 2025

  2. [2]

    org/abs/2604.15409

    Ranjith Chodavarapu and Lei Xu.oThe Illusion of Equivalence: Systematic FP16 Diver- gence in KV-Cached Autoregressive Inference.parXiv:2604.15409v1, 2026.https://arxiv. org/abs/2604.15409

    Pith/arXiv arXiv 2026

  3. [3]

    oMixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness

    Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. oMixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness. p arXiv:2310.02410v1, 2023.https://arxiv.org/abs/2310.02410

    Pith/arXiv arXiv 2023

  4. [4]

    oValue-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models

    Hancheol Park, Geonho Lee, Tairen Piao, and Tae-Ho Kim. oValue-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models. p arXiv:2606.05688v1, 2026.https://arxiv.org/abs/2606.05688

    Pith/arXiv arXiv 2026

  5. [5]

    pTechnical Report UCB/EECS-2016-121, University of California, Berkeley, 2016

    James Demmel, Willow Ahrens, and Hong Diep Nguyen.oEfficient Reproducible Float- ing Point Summation and BLAS. pTechnical Report UCB/EECS-2016-121, University of California, Berkeley, 2016. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/ EECS-2016-121.html

    2016

  6. [6]

    Neal.oFast Exact Summation Using Small and Large Superaccumulators.p arXiv:1505.05571v1, 2015.https://arxiv.org/abs/1505.05571

    Radford M. Neal.oFast Exact Summation Using Small and Large Superaccumulators.p arXiv:1505.05571v1, 2015.https://arxiv.org/abs/1505.05571

    Pith/arXiv arXiv 2015

  7. [7]

    https://doi.org/10.1016/j.parco.2015.09.001

    Caroline Collange, David Defour, Stef Graillat, and Roman Iakymchuk.oNumerical Re- producibility for the Parallel Reduction on Multi- and Many-Core Architectures.pParallel Computing 49 (2015): 83⚶97. https://doi.org/10.1016/j.parco.2015.09.001

    work page doi:10.1016/j.parco.2015.09.001 2015

  8. [8]

    1027⚶1038

    Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan.oCRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries.pProceedings of the 41st International Conference on Software Engineering, 2019, pp. 1027⚶1038. https://doi. org/10.1109/ICSE.2019.00107

    arXiv 2019

  9. [9]

    oNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compil- ers.pProceedings of ASPLOS 2023, 2023.https://doi.org/10.1145/3575693.3575707

    Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. oNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compil- ers.pProceedings of ASPLOS 2023, 2023.https://doi.org/10.1145/3575693.3575707

    arXiv 2023

  10. [10]

    oDeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale.pProceed- ings of Machine Learning Research 162 (ICML 2022): 18332⚶18346

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. oDeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale.pProceed- ings of Machine Learning Research 162 (ICML 2022): 18332⚶18346. https://proceedings. mlr.press/v162/rajbhandari22a.html

    2022

  11. [11]

    oDeepEP:AnEfficientExpert-ParallelCommunication Library.pSoftware repository, 2025.https://github.com/deepseek-ai/DeepEP

    Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, KuaiYu,JiashiLi,andLiangZhao. oDeepEP:AnEfficientExpert-ParallelCommunication Library.pSoftware repository, 2025.https://github.com/deepseek-ai/DeepEP

    2025

  12. [12]

    DeepSeek-AI et al.oDeepSeek-V4: Towards Highly Efficient Million-Token Context Intel- ligence.parXiv:2606.19348v1, 2026.https://arxiv.org/abs/2606.19348

    arXiv 2026

  13. [13]

    parXiv:2512.24880v2,2026

    Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai,HuazuoGao,JiangChang,LiangZhao,ShangyanZhou,ZheanXu,ZhengyanZhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wen- fengLiang.omHC:Manifold-ConstrainedHyper-Connections. parXiv:2512.24880v2,2026. https://arxiv.org/abs/2512.24880. 31

    Pith/arXiv arXiv 2026

  14. [14]

    oDSpark: Confidence-Scheduled Speculative Decoding with Semi- Autoregressive Generation

    Xin Cheng et al. oDSpark: Confidence-Scheduled Speculative Decoding with Semi- Autoregressive Generation. parXiv:2607.05147v1, 2026. https://arxiv.org/abs/2607. 05147

    Pith/arXiv arXiv 2026

  15. [15]

    Whale Agent Lab.oColibri: Native Inference Runtime.pSoftware repository, 2026.https: //github.com/whale-agent-lab/colibri. 32

    2026