REVIEW 3 major objections 5 minor 34 references
BMOA: Baseline-Mechanism-Outcome Attribution for Compiler-Induced Numerical Deviations
T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Compiler-induced floating-point differences cannot be judged by one pass/fail flag: each difference needs a separate attribution of baseline meaning, supported mechanism, and reference-relative accuracy outcome.
desk verdict The B/M/O separation is a genuine contribution; the headline empirical claim is reference-dependent, but the paper is honest about it and deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ⟨B,M,O⟩ attribution record. B is one of five named baselines—strict floating point, numerical reference, mechanism-local, reproducibility, or across-compiler—that fixes what a difference means. M is a mechanism category such as reassociation, fused multiply-add (FMA) contraction, or compound fast-math; it is marked supported only when a controlled pair D_m(x) exceeds the threshold and code-generation evidence is consistent, otherwise mixed, ambiguous, or unknown. O is the outcome—improvement, degradation, neutral, or unknown—obtained by ordering the errors of candidate and strict configurations against the same higher-precision reference. The mechanism-local compari
What would settle it
Find a kernel-input family where, across all reasonable higher-precision references (different precisions, rounding modes, interval bounds), the sign of the strict-FP deviation always predicts the sign of the reference-error difference (e.g., y_c > y_s always means E_c > E_s). Such a class would collapse baseline and outcome, refuting the paper's claim that none of the three record components follows from the other two; the paper's own 20/162 reference disagreements predict no such class exists.
Extended reading notes
Core claim
BMOA's central claim is that an observed numerical difference between two compiler configurations is not a single fact but a conjunction of three independent facts. The comparison that gives the difference its meaning (the baseline), the compiler behavior supported by controlled evidence (the mechanism), and the numerical consequence measured against a named reference (the outcome) must be recorded separately, because none of the three components follows from the other two. The framework emits a ⟨B,M,O⟩ record per evaluated comparison, supports a mechanism category only when a controlled paired ablation changes the output and assembly evidence is consistent, and preserves mixed, ambiguous, a
Load-bearing premise
Accuracy outcome labels depend on the chosen higher-precision reference (Decimal with 90 digits over the exact represented float32 inputs) correctly representing the intended real arithmetic of the source expression; if a different reference is semantically right for a kernel, every improvement/degradation label can flip.
Editorial extensions
If this is right
- A pass/fail mismatch is not a diagnosis: the same candidate result can carry a strict-FP deviation, a supported mechanism, and an improvement outcome simultaneously.
- Compiler-induced deviation does not imply accuracy loss; the controlled matrix found 16 improvements and 4 degradations, and all three FMA-contraction deviations improved reference accuracy.
- Mechanism attribution must be evidence-bounded: a vector or FMA instruction in the assembly does not establish that it caused the observed difference; the controlled matrix found vector instructions in 102/108 assembly files but no vectorization-enabled output deviations.
- Outcome labels are reference-relative: Decimal and a stable float32 algorithm disagreed on 20/162 instances, always in the same direction, so any accuracy claim must name its reference.
- By retaining inputs, bit patterns, metrics, and provenance, BMOA records give later formal specifications concrete comparison relations and proof obligations to target.
Reading between the lines
- A natural extension is to apply the same B,M,O schema to GPU kernels and parallel reductions, where the reproducibility baseline would separate run-to-run variance from compiler causality—a distinction the current single-CPU, single-seed study cannot probe.
- The 20/162 reference disagreements imply that reporting any single accuracy number without its reference and tolerance is under-specified; a practical takeaway is that accuracy claims should become "improvement relative to X under tolerance T" rather than bare percentages.
- If BMOA records were emitted automatically by testing harnesses, compiler bug reports could carry replayable evidence bundles, letting maintainers distinguish "semantic violation" from "different rounding order but better result" before deciding whether a change is a defect.
- The framework treats strict FP as one comparator among several, not as the truth; a testable consequence is that replacing strict FP with another operational configuration (e.g., a different rounding mode) would change B but should leave the outcome ordering unchanged if the reference is fixed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. BMOA formalizes post-detection diagnosis for compiler-induced floating-point deviations as <B,M,O> records: baseline defines the comparison relation and system boundary, mechanism identifies the compiler behavior category supported by controlled paired ablations, and outcome reports the reference-qualified accuracy result. The evaluation uses strict-FP, mechanism-local, reproducibility, cross-compiler, and higher-precision comparisons over six small kernels, nine configurations, and two Clang distributions on ARM64. The empirical components are a 1,276-record attribution corpus and a 162-instance controlled mechanism matrix. The main empirical findings are that baseline choice changes diagnosis; among 20 distinct nonzero deviations, 16 improve and 4 degrade relative to Decimal; cancellation and large dynamic range dominate the effects; and repeated runs and Apple/Homebrew comparisons are bit-identical. BMOA is explicitly not a proof system; it produces auditable, evidence-bounded records intended as a basis for future formal specifications.
Significance. The paper contributes a genuinely useful separation of concerns: pass/fail mismatch is inadequate as a diagnostic, and distinguishing comparison meaning, evidence-supported mechanism, and reference-relative outcome is valuable. The evaluation is careful in several respects: exact-bit comparisons, serialized inputs reused across configurations, repeated runs, Decimal–NumPy agreement on all 162 instances, identical Apple/Homebrew outcomes, explicit negative controls, and honest statements of scope. The central empirical claim that strict-FP deviation does not imply accuracy loss is, however, reference-conditional. The paper's own §5.4 reports 20 Decimal–stable-reference disagreements, with every disagreement being a Decimal improvement that the stable float32 reference labels as a degradation. Thus the headline 16-improvement/4-degradation split can shift substantially under a different semantically appropriate reference. This is a scoping and strength-of-evidence issue rather than a logical flaw in BMOA itself, and it is fixable with additional analysis and more careful claim qualification.
major comments (3)
- [§5.2, Table 5; §5.4, Table 7] The headline '16 improve, 4 degrade' is computed solely against Decimal. §5.4 reports that Decimal and the stable float32 reference disagree on 20 raw labels, corresponding to ten distinct combinations, and in every disagreement Decimal labels the candidate an improvement while the stable reference labels it a degradation. Since the ten disputed distinct cases are among the 16 Decimal improvements, adopting the stable reference as the intended semantics for those kernels changes the split to 6 improvements and 14 degradations. The paper is transparent about reference sensitivity, but the abstract and §5.2 present the Decimal-relative split as the main evidence that 'strict-FP deviation does not imply accuracy loss.' Please re-run the headline counts under the stable reference and report both, or explicitly restrict the claim to Decimal-relative outcomes and revise the abstract accordingl
- [§4, References, metrics, and scale; §5.4] The 'stable float32 algorithmic reference' is never defined. The paper only says that, where applicable, it changes the finite-precision algorithm; no concrete algorithm is given for the reduction, dot-product, polynomial, reciprocal, threshold, or FMA kernels. Because the outcome label O is reference-qualified and the paper itself demonstrates that the reference choice can flip labels, the omitted definition makes the reported outcome labels and the alternative count proposed above impossible to audit or recompute. Please provide exact algorithms and implementations per kernel, store them as artifacts, and use the same definitions in the re-run requested in the previous comment.
- [§3.3, Eq. (3)] The condition 'E_m(c,x) is consistent with m' in Eq. (3) is not operationalized. The text says assembly evidence corroborates but cannot establish causality, yet it never specifies what pattern counts or code-generation observations are sufficient to mark a mechanism as 'consistent.' Without a precise rule, third parties cannot reproduce S(c,x), and the distinction between 'supported,' 'mixed,' and 'unknown' is not fully auditable. Please define the consistency predicate for each mechanism category used in the evaluation (FMA, vectorization, reassociation, reciprocal math, etc.), or provide a code-level procedure implementing it.
minor comments (5)
- [§3.3, Eq. (3)] The notation E_m(c,x) appears in Eq. (3) but is not defined; define the evidence bundle component and its relation to the evidence table described in §3.5.
- [Table 5] The column headers 'Improve' and 'Degrade' should state explicitly that these are Decimal-relative outcomes, so the reference dependence of the counts is visible at the point of use.
- [Figure 2] State in the caption that all counts are deterministic and based on a single seed and one representative instance per kernel–input family; the text says this, but the figure alone could be misread as a prevalence estimate.
- [§5.4] The absence of positive reproducibility and cross-compiler cases is reported, but it would be clearer to state in the RQ4 conclusion that those categories are validated only as negative controls in this study, not as demonstrated diagnostic categories.
- [§7] Minor formatting issue: the author affiliation markers in the ACM reference format line contain a stray '1,'; fix the metadata formatting.
Circularity Check
No significant circularity: BMOA's labels are fixed-rule classifications against named references, not fitted predictions, and the headline deviations are externally demonstrated with transparent reference sensitivity.
full rationale
BMOA's derivation chain is a diagnostic workflow rather than a fitted model. Outcome labels are assigned by comparing E_c and E_s to a named reference (Section 2.1, Eq. 1; Section 3.3), with fixed thresholds (tau_m = 0, zero-tolerance ordering) rather than parameters tuned to observed outcomes. The headline result—strict-FP deviation does not necessarily imply accuracy loss—is supported by concrete records (e.g., Section 5.3 mixed-magnitude reduction: strict output 61, candidate 992, Decimal reference 1952, so E_c = 960 < E_s = 1891). The paper explicitly qualifies O as reference-relative and reports in Section 5.4 that Decimal and the stable float32 algorithm disagree on 20/162 labels, always in the same direction; this is a transparent scope limitation, not a circular argument. Mechanism labels are based on controlled paired ablations and assembly corroboration, and the component-independence of B, M, and O is an explicit design invariant enforced by the attribution order, not a consequence imported from self-citations or an imported uniqueness theorem. No parameter is fitted to the data to force the improvement/degradation counts; the conclusions could, as the paper acknowledges, differ for other references, kernels, seeds, or platforms without invalidating the framework. No load-bearing self-citations were found. Thus there is no circular step meeting the bar of Eq.-equals-input or fitted-parameter-renamed-as-prediction.
Assumptions & free parameters
free parameters (3)
- tau_m threshold =
0
- outcome tolerance =
0
- Decimal precision =
90 digits
assumptions (4)
- domain assumption IEEE 754 float arithmetic is rounded, order-sensitive, and compiler transformations can alter evaluation order, contraction, or fast-math behavior.
- domain assumption A strict-FP configuration (-fno-fast-math -ffp-contract=off) provides an operational baseline encoding the default evaluation path.
- domain assumption Decimal-90 over exact represented float32 inputs, and NumPy float64, evaluate the intended real-arithmetic expression.
- domain assumption Paired configurations differing by one flag isolate the named mechanism category, with compound interventions permitted to be labeled ambiguous/mixed.
Cite this review
Pith. "Pith review of BMOA: Baseline-Mechanism-Outcome Attribution for Compiler-Induced Numerical Deviations." pith.science (2026). https://pith.science/paper/NXS7MBFY
@misc{pith2026260727270,
author = {Pith},
title = {Pith review of: BMOA: Baseline-Mechanism-Outcome Attribution for Compiler-Induced Numerical Deviations},
year = {2026},
howpublished = {\url{https://pith.science/paper/NXS7MBFY}},
note = {Machine review of arXiv:2607.27270}
}
read the original abstract
Formalizing compiler-aware numerical correctness requires distinguishing what an observed floating-point difference means, what compiler behavior the evidence supports, and what numerical consequence follows. Existing testing workflows often collapse these questions into a pass/fail mismatch. We introduce Baseline--Mechanism--Outcome Attribution (BMOA), a diagnostic framework that separates the comparison relation and system boundary, the evidence-supported compiler mechanism, and the reference-qualified accuracy outcome. BMOA combines operational strict floating-point, transformation-local, reproducibility, cross-compiler, and higher-precision comparisons, while preserving mixed, ambiguous, and unknown attributions when evidence is insufficient. Each record retains inputs, configurations, numerical metrics, and supporting artifacts for audit. We evaluate BMOA on six scientific-computing kernels, deterministic stress-input families, and controlled Clang configurations on ARM64. A 1,276-record attribution corpus and a 162-instance controlled mechanism matrix show that baseline choice changes diagnoses, compiler-induced deviation does not imply accuracy loss, and cancellation and large dynamic range expose the strongest effects within the targeted matrix. BMOA converts raw mismatches into explicit, auditable, evidence-bounded records. Although it is not itself a proof system, these records provide an empirical foundation for future formal specifications and proof obligations for compiler-aware numerical correctness.
Figures
Reference graph
Works this paper leans on
-
[1]
Heiko Becker, Eva Darulova, Magnus O. Myreen, and Zachary Tatlock. 2019. Icing: Supporting Fast-Math Style Optimizations in a Verified Compiler. InComputer Aided Verification (CA V). Springer, Cham, Switzerland, 155–173. doi:10.1007/978- 3-030-25543-5_10
doi:10.1007/978- 2019
-
[2]
Heiko Becker, Pavel Panchekha, Eva Darulova, and Zachary Tatlock. 2018. Com- bining Tools for Optimization and Analysis of Floating-Point Computations. InInternational Symposium on Formal Methods. Springer, Cham, Switzerland, 355–363. doi:10.1007/978-3-319-95582-7_21
-
[3]
Florian Benz, Andreas Hildebrandt, and Sebastian Hack. 2012. A Dynamic Pro- gram Analysis to Find Floating-Point Accuracy Problems. InACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Asso- ciation for Computing Machinery, New York, NY, USA, 453–462. doi:10.1145/ 2254064.2254118
arXiv 2012
-
[4]
Sangeeta Chowdhary and Santosh Nagarakatte. 2021. Parallel Shadow Execution to Accelerate the Debugging of Numerical Errors. InACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). Association for Computing Machinery, New York, NY, USA, 615–626. doi:10.1145/3468264.3468585
arXiv 2021
-
[5]
Sangeeta Chowdhary and Santosh Nagarakatte. 2022. Fast Shadow Execution for Debugging Numerical Errors Using Error Free Transformations.Proceedings of the ACM on Programming Languages6, OOPSLA2 (2022), 1845–1872. doi:10. 1145/3563353
2022
-
[6]
Caroline Collange, David Defour, Stef Graillat, and Roman Iakymchuk. 2015. Numerical Reproducibility for the Parallel Reduction on Multi- and Many-Core Architectures.Parallel Comput.49 (2015), 83–97. doi:10.1016/j.parco.2015.09.001
-
[7]
Clément Courbet. 2021. NSan: A Floating-Point Numerical Sanitizer. InACM SIGPLAN International Conference on Compiler Construction (CC). Association for Computing Machinery, New York, NY, USA, 83–93. doi:10.1145/3446804.3446848
arXiv 2021
-
[8]
Eva Darulova, Anastasiia Izycheva, Fariha Nasir, Fabian Ritter, Heiko Becker, and Robert Bastian. 2018. Daisy: Framework for Analysis and Optimization of Numerical Programs. InTools and Algorithms for the Construction and Analysis of Systems (TACAS). Springer, Cham, Switzerland, 270–287. doi:10.1007/978-3-319- 89960-2_15
Show all 34 references
-
[9]
James Demmel and Hong Diep Nguyen. 2013. Fast Reproducible Floating-Point Summation. InIEEE Symposium on Computer Arithmetic (ARITH). IEEE, Piscat- away, NJ, USA, 163–172. doi:10.1109/ARITH.2013.9
2013 doi
-
[10]
Christophe Denis, Pablo de Oliveira Castro, and Eric Petit. 2016. Verificarlo: Checking Floating Point Accuracy through Monte Carlo Arithmetic. InIEEE Symposium on Computer Arithmetic (ARITH). IEEE, Piscataway, NJ, USA, 55–62. doi:10.1109/ARITH.2016.31
2016 doi
-
[11]
Anthony Di Franco, Hui Guo, and Cindy Rubio-González. 2017. A Comprehensive Study of Real-World Numerical Bug Characteristics. InIEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Piscataway, NJ, USA, 509–519. doi:10.1109/ASE.2017.8115662
2017
-
[12]
Zhoulai Fu, Zhaojun Bai, and Zhendong Su. 2015. Automated Backward Error Analysis for Numerical Code. InACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Association for Computing Machinery, New York, NY, USA, 6...
2015
-
[13]
David Goldberg. 1991. What Every Computer Scientist Should Know About Floating-Point Arithmetic.Comput. Surveys23, 1 (1991), 5–48. doi:10.1145/ 103162.103163
1991
-
[14]
Hui Guo, Ignacio Laguna, and Cindy Rubio-González. 2020. pLiner: Isolating Lines of Floating-Point Code for Compiler-Induced Variability. InInternational Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE, Piscataway, NJ, USA, 1–14. doi:10.1...
2020 arXiv
-
[15]
IEEE. 2019. IEEE Standard for Floating-Point Arithmetic. 84 pages. doi:10.1109/ IEEESTD.2019.8766229
2019
-
[16]
Ignacio Laguna. 2019. FPChecker: Detecting Floating-Point Exceptions in GPU Applications. InIEEE/ACM International Conference on Automated Software Engi- neering (ASE). IEEE, Piscataway, NJ, USA, 1126–1129. doi:10.1109/ASE.2019.00118
2019
-
[17]
Ignacio Laguna. 2020. Varity: Quantifying Floating-Point Variations in HPC Systems Through Randomized Testing. InIEEE International Parallel and Dis- tributed Processing Symposium (IPDPS). IEEE, Piscataway, NJ, USA, 622–633. doi:10.1109/IPDPS47924.2020.00070
2020
-
[18]
Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler Validation via Equiv- alence Modulo Inputs. InACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Association for Computing Machinery, New York, NY, USA, 216–226. doi:10.1145/2594291.2594334
2014
-
[19]
Dolores Miao, Ignacio Laguna, and Cindy Rubio-González. 2023. Expression Isolation of Compiler-Induced Numerical Inconsistencies in Heterogeneous Code. InInternational Conference on High Performance Computing. Springer, Cham, Switzerland, 381–401. doi:10.1007/978-3-031-32041-5_20
2023 doi
-
[20]
Dolores Miao, Ignacio Laguna, and Cindy Rubio-González. 2024. Input Range Generation for Compiler-Induced Numerical Inconsistencies. InACM Interna- tional Conference on Supercomputing (ICS). Association for Computing Machinery, New York, NY, USA, 201–212. doi:10.1145/3650200.3656618
2024
-
[21]
David Monniaux. 2008. The Pitfalls of Verifying Floating-Point Computations. ACM Transactions on Programming Languages and Systems30, 3 (2008), 12:1–12:41. doi:10.1145/1353445.1353446
2008
-
[22]
Andres Nötzli and Fraser Brown. 2016. LifeJacket: Verifying Precise Floating- Point Optimizations in LLVM. InACM SIGPLAN International Workshop on State Of the Art in Program Analysis. Association for Computing Machinery, New York, NY, USA, 24–29. doi:10.1145/2931021.2931024
2016
-
[23]
Wilcox, and Zachary Tatlock
Pavel Panchekha, Alex Sanchez-Stern, James R. Wilcox, and Zachary Tatlock
-
[24]
Bailey, Costin Iancu, and David Hough
Cindy Rubio-González, Cuong Nguyen, Hong Diep Nguyen, James Demmel, William Kahan, Koushik Sen, David H. Bailey, Costin Iancu, and David Hough
-
[25]
Alex Sanchez-Stern, Pavel Panchekha, Sorin Lerner, and Zachary Tatlock. 2018. Finding Root Causes of Floating Point Error. InACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Association for Com- puting Machinery, New York, NY, USA, 256–269. doi:...
2018
-
[26]
Geof Sawaya, Michael Bentley, Ian Briggs, Ganesh Gopalakrishnan, and Dong H. Ahn. 2017. FLiT: Cross-Platform Floating-Point Result-Consistency Tester and Workload. InIEEE International Symposium on Workload Characterization (IISWC). IEEE, Piscataway, NJ, USA, 229–238. doi:10.1...
2017
-
[27]
Baranowski, Ian Briggs, Charles Jacobsen, Zvonimir Rakamarić, and Ganesh Gopalakrishnan
Alexey Solovyev, Marek S. Baranowski, Ian Briggs, Charles Jacobsen, Zvonimir Rakamarić, and Ganesh Gopalakrishnan. 2018. Rigorous Estimation of Floating- Point Round-Off Errors with Symbolic Taylor Expansions.ACM Transactions on Programming Languages and Systems41, 1 (2018), 2...
2018 doi
-
[28]
Jackson Vanover, Xuan Deng, and Cindy Rubio-González. 2020. Discovering Discrepancies in Numerical Libraries. InACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). Association for Computing Machinery, New York, NY, USA, 488–501. doi:10.1145/3395363.3397380
2020
-
[29]
Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Under- standing Bugs in C Compilers. InACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Association for Computing Machin- ery, New York, NY, USA, 283–294. doi:10.1145/1993498.1993532
2011
-
[30]
Xin Yi, Hengbiao Yu, Liqian Chen, Xiaoguang Mao, and Ji Wang. 2024. FPCC: Detecting Floating-Point Errors via Chain Conditions.Proceedings of the ACM on Programming Languages8, OOPSLA2 (2024), 1504–1531. doi:10.1145/3689764
2024 doi
-
[31]
Hengbiao Yu, Xin Yi, Banghu Yin, Fa Li, Zhenbang Chen, and Chun Huang. 2023. Efficient Generation of Floating-Point Inputs for Compiler-Induced Variability. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Piscataway, NJ, USA, 2...
2023
-
[32]
Daming Zou, Yuchen Gu, Yuanfeng Shi, Mingzhe Wang, Yingfei Xiong, and Zhendong Su. 2022. Oracle-Free Repair Synthesis for Floating-Point Programs. Proceedings of the ACM on Programming Languages6, OOPSLA2 (2022), 957–985. doi:10.1145/3563322
2022 doi
-
[2013]
InInter- national Conference for High Performance Computing, Networking, Storage and Analysis (SC)
Precimonious: Tuning Assistant for Floating-Point Precision. InInter- national Conference for High Performance Computing, Networking, Storage and Analysis (SC). Association for Computing Machinery, New York, NY, USA, 1–12. BMOA: Baseline–Mechanism–Outcome Attribution for Compi...
2026
-
[2015]
In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)
Automatically Improving Accuracy for Floating Point Expressions. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Association for Computing Machinery, New York, NY, USA, 1–11. doi:10. 1145/2737924.2737959
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.