REVIEW 3 major objections 55 references
From Global to Factor-Wise Expert Composition in Discrete Diffusion Models
T0 review · 3 major / 0 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read Simple per-pixel expert routing outperforms global scalar weights when composing discrete diffusion models on spatial reasoning tasks.
desk verdict Solid inference-time idea: hard per-pixel routing of frozen discrete diffusion experts beats scalar SuperDiff/FKC/RNE when specialists are complementary; Col+Occ is the real result, with one calibration caveat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
FactorDiff: a dynamic routing map that assigns each variable (pixel) to the expert with the largest top-two probability margin, so the composed reverse transition is simply the product of the selected experts’ per-variable conditionals; no extra normalization or importance weights are required.
What would settle it
Train or select a pair of complementary specialists whose per-pixel confidence margins anti-correlate with correctness; if FactorDiff then underperforms both the best single expert and a simple global average on exact-match accuracy, the central claim fails.
Extended reading notes
Core claim
Factor-wise composition of discrete diffusion models—implemented as per-pixel confidence routing—consistently outperforms or matches per-sample global scalar weighting (SuperDiff, FKC, RNE, product-of-experts averages) on ARC-AGI, especially when experts are complementary specialists. Color plus occupancy specialists that score 3.3 % and 0.2 % exact alone reach 90.5 % exact under FactorDiff versus 78.4 % for the best sample-level method; full-model pairs remain non-degrading and match the strongest baseline at 95.2 %.
Load-bearing premise
An expert’s local confidence margin is assumed to reliably mark the expert that is actually correct at that pixel, so hard argmax routing recovers the intended specialization.
Editorial extensions
If this is right
- Complementary specialists that are nearly useless alone can be composed into high-accuracy solvers by stitching their locally confident predictions.
- Global product or mixture targets are misspecified whenever expertise varies across factors; factor-wise targets become necessary.
- Per-pixel routing remains safe for strong generalists: it does not degrade below the best sample-level baseline.
- The same routing idea applies unchanged to other hand-designed factorizations (regions, objects) and can be extended to learned routers.
Reading between the lines
- Hard confidence routing may transfer to continuous diffusion or other generative models whose local competence is spatially or functionally uneven.
- When two full models disagree only on architecture, token-level stitching can produce exact solutions that neither model generates alone, suggesting a lightweight ensemble regime without weight merging.
- Automatic discovery of useful factors, rather than hand-designed pixels or specialists, is a natural next test of whether the factor-wise principle scales beyond ARC-style grids.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FactorDiff, a factor-wise composition method for discrete (masked) diffusion models that routes each variable/pixel to one of several frozen experts rather than applying a single global scalar weight or exponent per expert. Composition is defined by a per-position routing map z_v chosen by confidence margin C(p)=p(1)−p(2) (Eqs. 10–12), yielding a product of valid expert conditionals. The method is instantiated with a native 2D masked-diffusion architecture (Conv-IR / Trans-IR) and three complementary training objectives (Full, Occupancy, Color) on a 120-task ARC-AGI-1 subset with RE-ARC augmentation. Empirically, per-pixel max routing matches or beats SuperDiff, FKC, RNE, and PoE averaging; the largest reported gain is Col+Occ (90.5% exact vs best per-sample 78.4%), with Occ+Full at 94.0% vs SuperDiff 93.2% and Full+Full tying FKC at 95.2% (Table 2).
Significance. If the factor-wise gains hold under controlled protocols, the work is a clear and useful contribution to compositional discrete diffusion: it identifies a real misspecification of global scalar product/mixture targets when experts are spatially specialized, and shows that simple hard routing can recover complementary structure that sample-level correctors cannot express. Strengths include a clean factorization of the reverse transition (Eq. 10), complementary specialist objectives, a native 2D backbone that preserves grid topology, and a striking Col+Occ result that would be hard to dismiss if robust. The Full+Full overlap ablation (Section 5.3) further shows stitching of correct regions neither model produces alone. Code is linked. The contribution is primarily empirical and ARC-centric rather than a new composition theorem, but the discrete factor-routing framing is timely and transferable.
major comments (3)
- The headline Col+Occ result in Table 2 (90.5% exact vs best per-sample 78.4%) is the main evidence that factor-wise routing beats global scalar schemes under specialization. Appendix A.5 / Table 6 applies Occ temperature T=0.25 only for this pair (other pairs use T=1.0), while the abstract and §3.2 present routing as simple confidence argmax. This free relative scale is load-bearing for the claim of “simple factor-specific routing” of frozen experts. Please report Col+Occ with all T_m=1.0, a sensitivity sweep over T_Occ, and confirm whether per-sample baselines (FKC/SuperDiff/RNE/PoE) receive the same temperature calibration; without that, the 12-point gap may partly reflect tuned relative confidence rather than routing granularity alone.
- Table 2 and the abstract claim that factor-specific routing “consistently outperforms” global scalar schemes. Full+Full is a tie with FKC at 95.2% exact (and matches SuperDiff/PoE at ~95.1%), so the wording overstates the generalist column. No multi-seed variance, confidence intervals, or task-level significance tests are reported for any column. Given that Occ+Full’s edge over SuperDiff is only +0.8 points, please either (i) add error bars / multi-seed runs and soften “consistently outperforms” to match the table, or (ii) show that the Occ+Full and Col+Occ gaps remain significant under the same protocol.
- §5 and §4.1 evaluate only RE-ARC IID held-outs on 120 tasks with grids ≤10×10, not the official ARC-AGI evaluation set or OOD generators. The composition claim can still stand on this controlled suite, but the paper repeatedly frames results as validation “on the ARC-AGI benchmark.” Please state the evaluation scope prominently in the abstract and results (IID RE-ARC subset, not official ARC), and ideally add at least one OOD or official-split check so that the large Col+Occ gain is not confounded with in-distribution memorization of the 120 generators.
Circularity Check
Empirical methods paper: factor-wise routing is a defined sampling procedure evaluated on held-out ARC data, not a derivation that rewrites fitted inputs as predictions.
full rationale
FactorDiff defines a composed reverse transition by routing each variable to a trained expert conditional (Eq. 10–12) and evaluates that procedure against per-sample baselines on RE-ARC held-out grids. The composed law is valid by construction as a product of trained conditionals; the paper does not claim that this construction recovers a separately specified target density, nor does it fit a parameter on a subset and then report a closely related quantity as a first-principles prediction. Per-expert temperatures in Appendix A.5 / Table 6 are free routing hyperparameters, not outputs of a derivation presented as predictions. Citations to SuperDiff, FKC, and RNE are baselines being compared against, not load-bearing uniqueness or ansatz results that force the reported gains. Continuous factor-composition work is acknowledged as related prior art rather than renamed. No step reduces a claimed prediction to its own inputs by definition or self-citation chain; circularity score is therefore zero.
Assumptions & free parameters
free parameters (4)
- Per-expert routing temperature T_m =
Occ: 0.25; others: 1.0
- Confidence metric C(p)=p(1)-p(2) =
top-2 probability margin
- Denoising steps T =
128
- Task/subset selection (120 tasks, grids ≤10×10) =
120 tasks, max 10×10
assumptions (4)
- domain assumption Masked diffusion reverse transitions factorize over variables/cells (MDLM-style), so a product of per-cell expert conditionals is a valid joint reverse step without extra normalization.
- ad hoc to paper Expert confidence margins correlate with local correctness sufficiently for argmax routing to select the competent specialist.
- domain assumption RE-ARC IID held-out instances from the same 120 training tasks are an adequate testbed for claims about ARC-AGI compositional reasoning.
- standard math Standard continuous-time masked diffusion NELBO / SUBS parameterization and concrete-score composition baselines behave as described in cited SuperDiff/FKC/RNE work.
invented entities (2)
-
FactorDiff routing map z_v
-
Factor-wise product target with per-factor exponents α_{m,f}
Cite this review
Pith. "Pith review of From Global to Factor-Wise Expert Composition in Discrete Diffusion Models." pith.science (2026). https://pith.science/paper/NENAZM56
@misc{pith2026260711758,
author = {Pith},
title = {Pith review of: From Global to Factor-Wise Expert Composition in Discrete Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/NENAZM56}},
note = {Machine review of arXiv:2607.11758}
}
read the original abstract
Discrete diffusion models offer a powerful framework for solving complex reasoning tasks, particularly through compositional generation, which combines multiple pre-trained experts to generalize beyond their individual training data. Recent theoretical corrections introduce time-dependent mixing weights to better align composed diffusion dynamics with the intended target. However, these methods are fundamentally limited by working on a per-sample basis, treating each generated state monolithically and ignoring the potential spatial or functional specializations of different experts. In this work, we address this limitation by proposing FactorDiff - a factor-wise composition framework for diffusion models. We posit that samples can be further decomposed into smaller factors, and propose a sampling process that dynamically routes each factor to the most relevant expert. We instantiate this framework with spatial/pixel-level compositions and validate it on the ARC-AGI benchmark, demonstrating that simple factor-specific routing consistently outperforms complex global scalar weighting schemes on tasks that require logical consistency and spatial disentanglement.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Proceedings of the 38th International Conference on Neural Information Processing Systems , pages=
Simple and effective masked diffusion language models , author=. Proceedings of the 38th International Conference on Neural Information Processing Systems , pages=
-
[2]
The superposition of diffusion models using the it
Skreta, Marta and Atanackovic, Lazar and Bose, Joey and Tong, Alexander and Neklyudov, Kirill , booktitle=. The superposition of diffusion models using the it
-
[3]
Sendera, Marcin and Kim, Minsu and Mittal, Sarthak and Lemos, Pablo and Scimeca, Luca and Rector-Brooks, Jarrid and Akhound-Sadegh, Tara and Tong, Alexander and Bengio, Yoshua , journal =
-
[4]
He, Jiajun and Hern\'andez-Lobato, Jos\'e Miguel and Du, Yuanqi and Vargas, Francisco , booktitle =
-
[5]
Discrete
Hasan, Mohsin and Ohanesian, Viktor and Gazizov, Artem and Bengio, Yoshua and Aspuru-Guzik, Al\'an and Bondesan Roberto and Skreta, Marta and Neklyudov, Kirill , journal =. Discrete
-
[6]
Forty-second International Conference on Machine Learning , year=
Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts , author=. Forty-second International Conference on Machine Learning , year=
-
[7]
Forty-second International Conference on Machine Learning , year =
Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions , author =. Forty-second International Conference on Machine Learning , year =
-
[8]
Jolicoeur-Martineau, Alexia , month = oct, year =. Less is. doi:10.48550/arXiv.2510.04871 , abstract =
Show all 55 references
-
[9]
Tiny Recursive Models on
Roye-Azar, Antonio and Vargas-Naranjo, Santiago and Ghai, Dhruv and Balamurugan, Nithin and Amir, Rayan , journal =. Tiny Recursive Models on
-
[10]
Advances in Neural Information Processing Systems , volume=
Large language diffusion models , author=. Advances in Neural Information Processing Systems , volume=
-
[11]
Nie, Shen and others , journal =
-
[12]
Knoop, Mike and others , year =. The
-
[13]
Compositional-
Mondorf, Philipp and Zhou, Shijia and Riedler, Monica and Plank, Barbara , booktitle =. Compositional-
-
[14]
arXiv preprint arXiv:2512.06104 , year =
ARC-AGI Without Pretraining , author =. arXiv preprint arXiv:2512.06104 , year =
-
[15]
Pourcel, Julien and Colas, Cédric and Oudeyer, Pierre-Yves , month = jun, year =. Self-
-
[16]
arXiv preprint arXiv:2404.07353 , year =
Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation , author =. arXiv preprint arXiv:2404.07353 , year =
-
[17]
arXiv preprint arXiv:1911.01547 , year =
On the Measure of Intelligence , author =. arXiv preprint arXiv:1911.01547 , year =
1911 arXiv
-
[18]
Advances in Neural Information Processing Systems , volume =
Structured Denoising Diffusion Models in Discrete State-Spaces , author =. Advances in Neural Information Processing Systems , volume =
-
[19]
Proceedings of the 41st International Conference on Machine Learning , pages =
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =
2024
-
[20]
The Thirteenth International Conference on Learning Representations , year=
Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning , author=. The Thirteenth International Conference on Learning Representations , year=
-
[21]
Su, Jianlin and Lu, Yu and Pan, Shengfeng and Murtadha, Ahmed and Wen, Bo and Liu, Yunfeng , journal =
-
[22]
Shazeer, Noam , journal =
-
[23]
Tolstikhin, Ilya and Houlsby, Neil and Kolesnikov, Alexander and Beyer, Lucas and Zhai, Xiaohua and Unterthiner, Thomas and Yung, Jessica and Steiner, Andreas and Keysers, Daniel and Uszkoreit, Jakob and others , journal =
-
[24]
Advances in Neural Information Processing Systems , volume =
Attention is All You Need , author =. Advances in Neural Information Processing Systems , volume =
-
[25]
Advances in Neural Information Processing Systems , volume =
Denoising Diffusion Probabilistic Models , author =. Advances in Neural Information Processing Systems , volume =
-
[26]
International Conference on Learning Representations , year =
Score-Based Generative Modeling through Stochastic Differential Equations , author =. International Conference on Learning Representations , year =
-
[27]
arXiv preprint arXiv:2512.07109 , year =
A Neural Affinity Framework for Abstract Reasoning , author =. arXiv preprint arXiv:2512.07109 , year =
-
[28]
, journal =
Park, Jongwoo and Ranasinghe, Kanchana and Jang, Jinhyeok and Mata, Cristina and Jang, Yoo Sung and Ryoo, Michael S. , journal =
-
[29]
2025 , howpublished =
Chollet, Fran. 2025 , howpublished =
2025
-
[30]
2024 , howpublished =
Muon: An Optimizer for Hidden Layers in Neural Networks , author =. 2024 , howpublished =
2024
-
[31]
International conference on machine learning , pages=
Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc , author=. International conference on machine learning , pages=. 2023 , organization=
2023
-
[32]
Neural Computation , volume =
Training Products of Experts by Minimizing Contrastive Divergence , author =. Neural Computation , volume =
-
[33]
International Conference on Learning Representations , year =
Product of Experts for Visual Generation , author =. International Conference on Learning Representations , year =
-
[34]
Advances in Neural Information Processing Systems , volume =
Compositional Visual Generation with Energy Based Models , author =. Advances in Neural Information Processing Systems , volume =
-
[35]
Forty-second International Conference on Machine Learning , year=
Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfaction , author=. Forty-second International Conference on Machine Learning , year=
-
[36]
International Conference on Machine Learning , pages=
Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver , author=. International Conference on Machine Learning , pages=. 2019 , organization=
2019
-
[37]
Advances in neural information processing systems , volume=
Recurrent relational networks , author=. Advances in neural information processing systems , volume=
-
[38]
, title =
Du, Yilun and Mao, Jiayuan and Tenenbaum, Joshua B. , title =. International Conference on Machine Learning (ICML) , year =
-
[39]
2009 , publisher=
Probabilistic graphical models: principles and techniques , author=. 2009 , publisher=
2009
-
[40]
2024 , url=
Yudong Xu and Wenhao Li and Pashootan Vaezipoor and Scott Sanner and Elias Boutros Khalil , journal=. 2024 , url=
2024
-
[41]
Proceedings of the 39th International Conference on Machine Learning , pages=
Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time , author=. Proceedings of the 39th International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[42]
International Conference on Learning Representations , year=
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. International Conference on Learning Representations , year=
-
[43]
2020 , howpublished="
J S Wind , title=. 2020 , howpublished="
2020
-
[44]
Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=
Relational decomposition for program synthesis , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=
-
[45]
International Conference on Complex Networks and Their Applications , pages=
Neural-Guided, Bidirectional Program Search for Abstraction and Reasoning , author=. International Conference on Complex Networks and Their Applications , pages=. 2021 , organization=
2021
-
[46]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
ARC Is a Vision Problem! , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[47]
Transactions on Machine Learning Research , issn=
Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects , author=. Transactions on Machine Learning Research , issn=. 2025 , url=
2025
-
[48]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Graphs, constraints, and search for the abstraction and reasoning corpus , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[49]
Product of Experts with
Daniel Franzen and Jan Disselhoff and David Hartmann , booktitle=. Product of Experts with. 2025 , url=
2025
-
[50]
Conference on Robot Learning , year=
Compositional diffusion-based continuous constraint solvers , author=. Conference on Robot Learning , year=
-
[51]
Set it up
“Set it up”: Functional object arrangement with compositional generative models , author=. The International Journal of Robotics Research , pages=. 2024 , publisher=
2024
-
[52]
Advances in Neural Information Processing Systems , volume=
Adapting diffusion models for improved prompt compliance and controllable image synthesis , author=. Advances in Neural Information Processing Systems , volume=
-
[53]
8th Annual Conference on Robot Learning , year=
Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph , author=. 8th Annual Conference on Robot Learning , year=
-
[54]
Conference on Robot Learning , pages=
Generative skill chaining: Long-horizon skill planning with diffusion models , author=. Conference on Robot Learning , pages=. 2023 , organization=
2023
-
[55]
DiffCollage: Parallel Generation of Large Content with Diffusion Models , year=
Zhang, Qinsheng and Song, Jiaming and Huang, Xun and Chen, Yongxin and Liu, Ming-Yu , booktitle=. DiffCollage: Parallel Generation of Large Content with Diffusion Models , year=
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.