REVIEW 6 minor 31 references
Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing
T0 review · 0 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Cross-library fuzzing exposes 72 deep learning API discrepancies
desk verdict XAMT is a solid, thoroughly evaluated extension of the authors' ISSRE work; the single-input validation is the main soft spot, but the confirmed fixes and careful ablations support the 72-case claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the execution-validated API group. Candidate groups are formed by normalizing operation names through curated aliases and mapping parameters to semantic roles such as axis, data, shape, and keepdim, then each cross-library pair is executed on a canonical ordinary input. A pair passes only if both execute successfully and outputs agree after matching NaN and infinity masks and comparing finite elements within absolute tolerance $\tau = 10^{-3}$. Connected components of passing pairs are re-executed as a whole, and only mutually consistent components are kept for fuzzing. Fuzzing then mutates shared logical inputs, including boundary and non-finite values, guided by a scale-normalized disagreement score, with Crash and Inconsistency oracles flagging discrepancies.
What would settle it
Take a random sample of the 676 execution-validated groups and test each on a large set of diverse random inputs within their shared semantic domain; if a substantial fraction produce inconsistent outputs, the canonical-input validation is too weak and the 72-case count may be inflated. Alternatively, search more broadly around the 50 CPU-GPU-applicable triggering inputs; finding even one input that yields a CPU-GPU output difference beyond tolerance would refute the claim that these cases are invisible to backend differential oracles.
Extended reading notes
Core claim
XAMT identifies 72 valid discrepancy cases across seven deep learning libraries: 4 crashes and 68 output inconsistencies, produced by fuzzing 676 execution-validated cross-library API groups that contain 2,563 matched APIs. Of the 72 developer reports, 25 have been confirmed and 23 fixed. A fixed-input backend replay shows that 50 CPU-GPU-applicable cases produce no observable CPU-GPU disagreement, supporting the paper's central argument that these defects are invisible to backend-differential oracles and require cross-framework comparison.
Load-bearing premise
The whole pipeline rests on the hand-curated alias mapping and parameter-role normalization together with the assumption that passing one canonical ordinary input per operation is enough to confirm that two APIs are semantically equivalent; if the mapping is wrong for an operation or the canonical input is too narrow, a reported discrepancy may reflect a bad correspondence rather than a genuine library defect.
Editorial extensions
If this is right
- Cross-framework differential fuzzing can serve as a complement to CPU-GPU oracles for deep learning library testing, covering defects that behave consistently across backends.
- Real library defects are reachable at the API level: 25 confirmed and 23 fixed reports show the identified discrepancies map to actual code changes.
- Boundary and non-finite inputs are a productive source of discrepancies; without them 22 of 72 reference cases are not reached, and 20 of those require such values.
- API matching without a designated reference library expands testing scope: 444 of 676 groups contain APIs from three or more libraries.
- Execution-based validation reduces the density of raw discrepancy candidates from 25.4 to 16.1 per 100 groups, filtering unsuitable correspondences before fuzzing.
Reading between the lines
- Editorial inference: If the alias and role mappings are published as reusable data, the same pipeline could be applied to future library versions or new frameworks with relatively little manual effort, making the matching step semi-automatic.
- Editorial inference: The fact that 59.7% of the 72 cases are not represented in converter-derived correspondence scopes suggests converter registries are an incomplete source of API equivalence, and execution-validated matching may cover a wider space of relations.
- Editorial inference: All 27 MindSpore reports remain pending, so the true false-positive rate for that library is not yet established; re-running against a later MindSpore version could discriminate between slow triage and invalid correspondences.
- Editorial inference: A direct testable extension would be applying XAMT's matching pipeline to additional libraries such as ONNX Runtime or oneDNN; if the alias mapping generalizes, it would support the claim that execution validation, not just the curated mapping, carries the approach.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents XAMT, a cross-framework differential fuzzing approach for testing deep learning library APIs. XAMT first constructs candidate API groups through name normalization, curated alias mappings, and parameter-role normalization; it then validates candidate correspondences by pairwise execution and a group-level consistency check on one canonical ordinary input per operation. Validated groups are fuzzed with variance-guided mutation over ordinary, boundary, and non-finite inputs, and crash and inconsistency oracles flag discrepancies. The evaluation covers PyTorch, TensorFlow, Keras, JAX, MindSpore, PaddlePaddle, and Chainer, producing 676 execution-validated groups containing 2,563 matched APIs, and 72 independently reproduced discrepancy cases, of which 25 have been confirmed by developers and 23 fixed. The paper also reports complementary analyses: none of the 50 CPU-GPU-applicable triggering inputs yields a backend-disagreement signal; FreeFuzz and DeepREL rediscover only 2/15 and 1/11 applicable defects; and only 7/72 cases are fully represented in TensorScope's converter-derived correspondence scope. Component ablations, tolerance sensitivity, and budget sensitivity are also reported. The manuscript is an extended version of an ISSRE paper, with methodological extensions and a broader evaluation.
Significance. If the results hold, XAMT constitutes a meaningful advance in API-level testing of deep learning libraries: it removes the reliance on a designated reference library, uses execution-validated groups as a differential oracle, and detects defects that are invisible to CPU-GPU differential testing. The paper's strengths include an extensive evaluation with a full matching funnel, manual audits of PASS/REJECT/SKIP pairs, a second-input validation audit, component ablations, tolerance and budget sensitivity analyses, and a replication package. The external confirmation of 25 cases (with 23 fixed) provides independent evidence that the detected discrepancies are genuine. The main residual concern, that single-input execution validation may admit semantically non-equivalent API pairs, is explicitly acknowledged in Section VI and mitigated by the second-input audit and manual vetting of retained cases. Overall, the contribution is solid and the empirical support is substantial.
minor comments (6)
- [Section II-B3 / Section VI] The execution validation in Section II-B3 uses one canonical ordinary input per operation, which the paper itself acknowledges in Section VI is insufficient to establish semantic equivalence over the complete domain. The post-hoc second-input audit of 100 groups is reassuring but appears only in Section VI; consider mentioning this audit (or a summary of it) in Section II-B4 or IV-A where the validation is described, so that the robustness evidence is presented at the point where the reader evaluates the matching pipeline.
- [Section IV-A / IV-C] Coverage percentages, discrepancy counts, and component-analysis numbers are reported from single runs. Although random seeds are fixed and CPU execution is deterministic, fuzzing pipelines can still exhibit run-to-run variability; reporting variance (e.g., min-max or standard deviation over a small number of runs) for at least the headline coverage numbers and raw candidate counts would strengthen the quantitative claims.
- [Abstract and full text] There are frequent formatting issues where "XAMT" is concatenated with the following word without a space (e.g., "XAMTconstructs", "XAMTidentifies", "XAMTand"). These should be fixed throughout the manuscript.
- [Table VIII] The "Other" disposition in Table VIII is not defined. Please clarify what this category includes (e.g., duplicate reports, wontfix, invalid, or otherwise closed without confirmation) so readers can interpret the six cases.
- [Section IV-E / Table XI] The column labeled "Raw" in Table XI reports raw discrepancy candidates, but the manuscript does not define how a raw candidate is identified before the reproduction and retention process. A one-sentence definition in Section IV-E1 would make the component analysis easier to interpret.
- [Figure 3] Figure 3 is referenced in Section IV-E5 but is not visible in the provided text. Ensure the figure is included with appropriate axis labels and a caption in the final version.
Circularity Check
No circularity: the matched groups are validated on a canonical input and then tested on a different, broader input domain, with external developer confirmation of the resulting discrepancies.
full rationale
The paper's derivation chain is self-contained and non-circular. XAMT first constructs candidate correspondences from curated API aliases and parameter-role normalization, then filters them by pairwise and group-level execution validation on a single canonical ordinary input per operation. The differential fuzzing stage then explores a much broader input domain (ordinary, boundary, and non-finite values) using a variance-guided score that is explicitly not the final oracle. The final Inconsistency oracle compares outputs among group members on the fuzzed inputs, which are distinct from the canonical validation input. There is no step in which a prediction is defined in terms of the data used to fit it, nor is any parameter fitted so that a target discrepancy appears. The same tolerance τ = 10^-3 appears in both validation and oracle, but the validation requires agreement on the canonical input, while the oracle flags disagreement on new inputs; this is the standard differential-testing structure, not a tautology. The paper also explicitly acknowledges the limitation: 'these checks do not establish semantic equivalence over the complete input domain' (Section IV-A4) and states that when no independent numerical reference is available, it 'report[s] the reproducible behavioral discrepancy without treating majority behavior alone as definitive ground truth' (Section IV-C1). External confirmations are genuinely independent: 25 of 72 reports were confirmed by developers, 23 fixed, and representative fixed cases are checked against NumPy/SciPy references. The citation to the authors' earlier ISSRE paper [14] is used only for scope comparison (Table V) and coverage comparison under the same 12-hour budget, not as the load-bearing justification for the central claim. The comparisons against FreeFuzz, DeepREL, and TensorScope provide further external positioning. No self-definitional step, fitted-input-called-prediction step, or uniqueness-imported-from-authors step is present. The skeptical concern that single-input execution validation may admit non-equivalent pairs is a threats-to-validity issue about filter strength, not circularity: it does not make the fuzzing results equivalent to the matching inputs by construction. The verdict is therefore no significant circularity.
Assumptions & free parameters
free parameters (5)
- output tolerance tau =
1e-3
- improvement threshold delta_score =
0.001
- restart count R =
20
- per-group fuzzing budget =
60 s
- canonical validation inputs =
operation-specific (not enumerated in paper)
assumptions (4)
- domain assumption APIs that share a normalized name and pass execution validation on a canonical ordinary input are behaviorally equivalent over the fuzzed input domain.
- domain assumption Official API specifications, NumPy/SciPy references, and developer confirmations provide reliable ground truth for which library is incorrect in a discrepancy.
- domain assumption Keras with the TensorFlow backend provides a meaningfully independent implementation for differential testing.
- domain assumption Non-finite values (NaN and signed infinities) propagate through the libraries in a way that the oracle's mask-and-sign comparison captures.
Cite this review
Pith. "Pith review of Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing." pith.science (2026). https://pith.science/paper/ZTWVR2VM
@misc{pith2026260811886,
author = {Pith},
title = {Pith review of: Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZTWVR2VM}},
note = {Machine review of arXiv:2608.11886}
}
read the original abstract
Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU--GPU differential oracles and may miss defects that behave consistently across hardware backends. We present Xamt, a cross-framework differential fuzzing approach for deep learning library APIs. Xamt constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs. Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, Xamt constructs 676 execution-validated groups containing 2,563 matched APIs. Among these, Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep learning for self-driving cars: Chances and challenges,
Q. Rao and J. Frtunikj, “Deep learning for self-driving cars: Chances and challenges,” inProceedings of the 1st international workshop on software engineering for AI in autonomous systems, 2018, pp. 35–38
work page 2018
-
[2]
Anomaly detection in autonomous driving: A survey,
D. Bogdoll, M. Nitsche, and J. M. Z ¨ollner, “Anomaly detection in autonomous driving: A survey,” inProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 2022, pp. 4488–4499
2022
-
[3]
Deep learning for healthcare: review, opportunities and challenges,
R. Miotto, F. Wang, S. Wang, X. Jiang, and J. T. Dudley, “Deep learning for healthcare: review, opportunities and challenges,”Briefings in bioinformatics, vol. 19, no. 6, pp. 1236–1246, 2018
2018
-
[4]
Deep learning for smart healthcare—a survey on brain tumor detection from medical imaging,
M. Arabahmadi, R. Farahbakhsh, and J. Rezazadeh, “Deep learning for smart healthcare—a survey on brain tumor detection from medical imaging,”Sensors, vol. 22, no. 5, p. 1960, 2022
work page 1960
-
[5]
Deep learning for finance: deep portfolios,
J. B. Heaton, N. G. Polson, and J. H. Witte, “Deep learning for finance: deep portfolios,”Applied Stochastic Models in Business and Industry, vol. 33, no. 1, pp. 3–12, 2017
work page 2017
-
[6]
Y . Venkateswarlu, K. Baskar, A. Wongchai, V . Gauri Shankar, C. Paolo Martel Carranza, J. L. A. Gonz ´ales, and A. Murali Dharan, “An efficient outlier detection with deep learning-based financial crisis prediction model in big data environment,”Computational Intelligence and Neu- roscience, vol. 2022, 2022
work page 2022
-
[7]
J. Li, B. Zhao, and C. Zhang, “Fuzzing: a survey,”Cybersecurity, vol. 1, no. 1, pp. 1–13, 2018
work page 2018
-
[8]
Docter: documentation-guided fuzzing for testing deep learning api functions,
D. Xie, Y . Li, M. Kim, H. V . Pham, L. Tan, X. Zhang, and M. W. God- frey, “Docter: documentation-guided fuzzing for testing deep learning api functions,” inProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 176–188
work page 2022
Show all 31 references
-
[9]
Audee: Automated testing for deep learning frameworks,
Q. Guo, X. Xie, Y . Li, X. Zhang, Y . Liu, X. Li, and C. Shen, “Audee: Automated testing for deep learning frameworks,” inProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 486–498
2020
-
[10]
Muffin: Testing deep learning libraries via neural architecture fuzzing,
J. Gu, X. Luo, Y . Zhou, and X. Wang, “Muffin: Testing deep learning libraries via neural architecture fuzzing,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 1418– 1430
2022
-
[11]
Deep learning library testing via effective model generation,
Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 788–799
2020
-
[12]
Free lunch for testing: Fuzzing deep-learning libraries from open source,
A. Wei, Y . Deng, C. Yang, and L. Zhang, “Free lunch for testing: Fuzzing deep-learning libraries from open source,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 995–1007
2022
-
[13]
Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,
Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” inProceedings of the 32nd ACM SIGSOFT interna- tional symposium on software testing and analysis, 2023, pp. 423–435
2023
-
[14]
Xamt: Cross- framework api matching for testing deep learning libraries,
B. Duan, R. Dong, N. Dong, D. D. Kim, and G. Yang, “Xamt: Cross- framework api matching for testing deep learning libraries,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE), 2025, pp. 191–202
2025
-
[15]
A synergistic approach to improving symbolic execution using test ranges,
G. Yang, R. Qiu, S. Khurshid, C. S. P ˘as˘areanu, and J. Wen, “A synergistic approach to improving symbolic execution using test ranges,” Innovations in Systems and Software Engineering, vol. 15, pp. 325–342, 2019
2019
-
[16]
Advances in symbolic execution,
G. Yang, A. Filieri, M. Borges, D. Clun, and J. Wen, “Advances in symbolic execution,”Advances in Computers, vol. 113, pp. 225–287, 2019
2019
-
[17]
An algebraic approach for managing inconsistencies in software pro- cesses,
Q. Yang, M. Li, Q. Wang, G. Yang, J. Zhai, J. Li, L. Hou, and Y . Yang, “An algebraic approach for managing inconsistencies in software pro- cesses,” inInternational Conference on Software Process. Springer, 2007, pp. 121–133
2007
-
[18]
Feedback-driven incremental symbolic execution,
Q. Yi and G. Yang, “Feedback-driven incremental symbolic execution,” in2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2022, pp. 505–516
2022
-
[19]
Acid: an api compatibility issue detector for android apps,
T. Mahmud, M. Che, and G. Yang, “Acid: an api compatibility issue detector for android apps,” inProceedings of the ACM/IEEE 44th Inter- national Conference on Software Engineering: Companion Proceedings, 2022, pp. 1–5
2022
-
[20]
Summary-guided incremental symbolic execution,
Q. Yi, J. Wen, and G. Yang, “Summary-guided incremental symbolic execution,” inProceedings of the ACM/IEEE 42nd International Con- ference on Software Engineering: Companion Proceedings, 2020, pp. 310–311
2020
-
[21]
Evaluating and improving neural program- smoothing-based fuzzing,
M. Wu, L. Jiang, J. Xiang, Y . Zhang, G. Yang, H. Ma, S. Nie, S. Wu, H. Cui, and L. Zhang, “Evaluating and improving neural program- smoothing-based fuzzing,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 847–858
2022
-
[22]
{IvySyn}: Automated vulnerability discovery in deep learning frameworks,
N. Christou, D. Jin, V . Atlidakis, B. Ray, and V . P. Kemerlis, “{IvySyn}: Automated vulnerability discovery in deep learning frameworks,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 2383–2400
2023
-
[23]
Metamorphic testing of deep learning compilers,
D. Xiao, Z. Liu, Y . Yuan, Q. Pang, and S. Wang, “Metamorphic testing of deep learning compilers,”Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 6, no. 1, pp. 1–28, 2022. TRANSACTIONS ON SOFTW ARE ENGINEERING 12
2022
-
[24]
Eagle: creating equivalent graphs to test deep learning libraries,
J. Wang, T. Lutellier, S. Qian, H. V . Pham, and L. Tan, “Eagle: creating equivalent graphs to test deep learning libraries,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 798– 810
2022
-
[25]
Fuzzing deep-learning libraries via automated relational api inference,
Y . Deng, C. Yang, A. Wei, and L. Zhang, “Fuzzing deep-learning libraries via automated relational api inference,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 44– 56
2022
-
[26]
Cradle: cross-backend validation to detect and localize bugs in deep learning libraries,
H. V . Pham, T. Lutellier, W. Qi, and L. Tan, “Cradle: cross-backend validation to detect and localize bugs in deep learning libraries,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1027–1038
2019
-
[27]
Nnsmith: Generating diverse and valid test cases for deep learning compilers,
J. Liu, J. Lin, F. Ruffy, C. Tan, J. Li, A. Panda, and L. Zhang, “Nnsmith: Generating diverse and valid test cases for deep learning compilers,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vo...
2023
-
[28]
Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,
Y . Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,”arXiv preprint arXiv:2304.02014, 2023
2023 arXiv
-
[29]
History- guided configuration diversification for compiler test-program genera- tion,
J. Chen, G. Wang, D. Hao, Y . Xiong, H. Zhang, and L. Zhang, “History- guided configuration diversification for compiler test-program genera- tion,” in2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 305–316
2019
-
[30]
Fuzzing with code fragments,
C. Holler, K. Herzig, and A. Zeller, “Fuzzing with code fragments,” in21st USENIX Security Symposium (USENIX Security 12), 2012, pp. 445–458
2012
-
[31]
Differential testing of cross deep learning framework{APIs}: Revealing inconsisten- cies and vulnerabilities,
Z. Deng, G. Meng, K. Chen, T. Liu, L. Xiang, and C. Chen, “Differential testing of cross deep learning framework{APIs}: Revealing inconsisten- cies and vulnerabilities,” in32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 7393–7410
2023
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.