{"id":"ee79838e-7e50-4819-b677-42b174233fca","arxiv_id":"2608.11886","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A cross-framework differential fuzzer matched 2,563 APIs across seven deep learning libraries into 676 validated groups and uncovered 72 functional discrepancies, including 25 confirmed by library developers.","lead":"This paper presents XAMT, a tool that compares the same operation across seven deep learning libraries to find places where the libraries disagree. The tool found 72 real disagreements, 25 of which developers confirmed and 23 of which have already been fixed, showing that cross-library comparison catches bugs that hardware-level checks miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-input execution validation may admit non-equivalent API pairs, potentially inflating the 72-case total; the paper's audits and confirmed fixes mitigate this, so the verdict stands.","rationale":"The reader's weakest-assumption identification matches my own: the correctness of the matching pipeline, especially the use of a single canonical input for execution validation, is the least secure link in the central claim. If non-equivalent APIs pass validation, fuzzing discrepancies between them are not evidence of library defects, and the 72-case count would be overstated. This is a genuine risk, not a manufactured one. However, the paper militates against this risk in several concrete ways: manual audits of PASS/REJECT/SKIP pairs, a post-hoc consistency check on 100 groups with a second input, per-case examination of every retained discrepancy against official specifications and reference implementations, and strong external confirmations (25 confirmed, 23 fixed by developers). These mitigations mean the central mechanism is validated even if some unconfirmed cases later turn out to be false positives. The strongest claim is carefully worded as 'discrepancies' rather than 'bugs,' and the paper transparently separates confirmed from pending cases. Therefore, while I would want the additional all-groups second-input validation and an independent audit of the unconfirmed cases before relying on the exact number 72, the concern does not overturn the ACCEPT verdict. It identifies a direction for strengthening the evidence rather than a fatal flaw.","tokens_in":15934,"tokens_out":7894,"duration_ms":86205,"concrete_test":"Rerun the group-level consistency check on all 676 execution-validated groups using a second, independently designed canonical validation input per operation, with different shapes, dtypes, and axis choices (mirroring the paper's post-hoc audit but on all groups). If any group fails, the single-input validation is insufficient and the 72-case count may include false positives. Separately, have two independent annotators, blinded to the paper's conclusions, classify all 47 unconfirmed discrepancy cases using official API documentation and NumPy/SciPy references with a pre-registered rubric: genuine defect, explainable semantic difference, or out-of-domain input. If more than 30% of the unconfirmed cases are classified as explainable or out-of-domain, the paper should revise its headline claim from 72 discrepancy cases to the confirmed subset or provide the audit results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The execution validation that defines an API group is performed on one canonical ordinary input per operation (Section II-B3). This is a weak filter: two APIs can agree on that single input while differing on other parts of the domain, so a fuzzing discrepancy may reflect an invalid correspondence rather than a genuine library defect. The paper's own funnel shows 1,448 REJECT pairs on the canonical input, and the manual audit covers only 50 PASS, 50 REJECT, and 50 SKIP pairs, while the second-input audit covers 100 groups. More importantly, of the 72 retained discrepancy cases, 47 remain unconfirmed (41 pending, 6 other), including all 27 MindSpore reports and 11 of 12 PaddlePaddle reports (Table VIII). If a substantial fraction of these unconfirmed cases are actually caused by semantically non-equivalent APIs, differing documented defaults, or out-of-domain fuzz inputs, the headline number of 72 independently reproduced discrepancies is inflated. The paper states that each case is examined against official specifications and numerical references, and 25 cases are confirmed with 23 fixed, which provide strong independent support; however, the exact count of 72 rests on the authors' own vetting of the unconfirmed cases, and the single-input validation is the point at which the correspondence could be wrong.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents XAMT, a cross-framework differential fuzzing approach for testing deep learning library APIs. XAMT first constructs candidate API groups through name normalization, curated alias mappings, and parameter-role normalization; it then validates candidate correspondences by pairwise execution and a group-level consistency check on one canonical ordinary input per operation. Validated groups are fuzzed with variance-guided mutation over ordinary, boundary, and non-finite inputs, and crash and inconsistency oracles flag discrepancies. The evaluation covers PyTorch, TensorFlow, Keras, JAX, MindSpore, PaddlePaddle, and Chainer, producing 676 execution-validated groups containing 2,563 matched APIs, and 72 independently reproduced discrepancy cases, of which 25 have been confirmed by developers and 23 fixed. The paper also reports complementary analyses: none of the 50 CPU-GPU-applicable triggering inputs yields a backend-disagreement signal; FreeFuzz and DeepREL rediscover only 2/15 and 1/11 applicable defects; and only 7/72 cases are fully represented in TensorScope's converter-derived correspondence scope. Component ablations, tolerance sensitivity, and budget sensitivity are also reported. The manuscript is an extended version of an ISSRE paper, with methodological extensions and a broader evaluation.","tokens_in":16126,"tokens_out":6403,"duration_ms":57993,"significance":"If the results hold, XAMT constitutes a meaningful advance in API-level testing of deep learning libraries: it removes the reliance on a designated reference library, uses execution-validated groups as a differential oracle, and detects defects that are invisible to CPU-GPU differential testing. The paper's strengths include an extensive evaluation with a full matching funnel, manual audits of PASS/REJECT/SKIP pairs, a second-input validation audit, component ablations, tolerance and budget sensitivity analyses, and a replication package. The external confirmation of 25 cases (with 23 fixed) provides independent evidence that the detected discrepancies are genuine. The main residual concern, that single-input execution validation may admit semantically non-equivalent API pairs, is explicitly acknowledged in Section VI and mitigated by the second-input audit and manual vetting of retained cases. Overall, the contribution is solid and the empirical support is substantial.","major_comments":[],"minor_comments":[{"comment":"The execution validation in Section II-B3 uses one canonical ordinary input per operation, which the paper itself acknowledges in Section VI is insufficient to establish semantic equivalence over the complete domain. The post-hoc second-input audit of 100 groups is reassuring but appears only in Section VI; consider mentioning this audit (or a summary of it) in Section II-B4 or IV-A where the validation is described, so that the robustness evidence is presented at the point where the reader evaluates the matching pipeline.","section":"Section II-B3 / Section VI"},{"comment":"Coverage percentages, discrepancy counts, and component-analysis numbers are reported from single runs. Although random seeds are fixed and CPU execution is deterministic, fuzzing pipelines can still exhibit run-to-run variability; reporting variance (e.g., min-max or standard deviation over a small number of runs) for at least the headline coverage numbers and raw candidate counts would strengthen the quantitative claims.","section":"Section IV-A / IV-C"},{"comment":"There are frequent formatting issues where \"XAMT\" is concatenated with the following word without a space (e.g., \"XAMTconstructs\", \"XAMTidentifies\", \"XAMTand\"). These should be fixed throughout the manuscript.","section":"Abstract and full text"},{"comment":"The \"Other\" disposition in Table VIII is not defined. Please clarify what this category includes (e.g., duplicate reports, wontfix, invalid, or otherwise closed without confirmation) so readers can interpret the six cases.","section":"Table VIII"},{"comment":"The column labeled \"Raw\" in Table XI reports raw discrepancy candidates, but the manuscript does not define how a raw candidate is identified before the reproduction and retention process. A one-sentence definition in Section IV-E1 would make the component analysis easier to interpret.","section":"Section IV-E / Table XI"},{"comment":"Figure 3 is referenced in Section IV-E5 but is not visible in the provided text. Ensure the figure is included with appropriate axis labels and a caption in the final version.","section":"Figure 3"}],"recommendation":"minor_revision","confidential_remarks":"I concur with the reader's overall positive assessment. The stress-test concern about single-input validation is real but does not undermine the core contribution, given the paper's explicit acknowledgement in Section VI, the second-input audit, the manual audits, and the 25 externally confirmed fixes. The remaining issues are local and can be addressed in a minor revision. The paper is within the scope of TSE and makes a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead arXiv:2608.11886. Short version: this is a solid journal extension of the ISSRE XAMT paper, and it deserves a serious referee. The genuinely new material is the removal of the designated-reference library, the explicit alias and parameter-role normalization, execution-validated groups, and boundary/non-finite fuzzing. The evaluation is where the paper earns its keep: 676 groups and 2,563 matched APIs across seven libraries, a complete matching funnel, manual audits of PASS/REJECT/SKIP, component ablations, tolerance and budget sensitivity, and 23 developer-fixed bugs. I also think the complementarity analysis is convincing: 50 CPU-GPU-applicable cases yield no backend disagreement, FreeFuzz and DeepREL mostly fail to rediscover the defects, and TensorScope fully represents only 7 of 72 cases.\n\nThe main soft spot is the one the stress-test note flags: execution validation uses a single canonical ordinary input per operation. That is a weak equivalence filter, and the paper says so itself in the threats section. Two APIs can agree on that input and diverge elsewhere, so some of the 72 cases could be invalid correspondences rather than real defects. The manual audits cover only 50 pairs per category, and a second-input audit covers 100 groups, which helps but is still a sample. It also matters that 47 of the 72 cases are unconfirmed, including all 27 MindSpore reports. I would not call this a load-bearing flaw: the 25 confirmed and 23 fixed bugs are independent external evidence, and the authors examined each case against official specs and numerical references. But I would ask the authors to state more plainly how the unconfirmed cases were vetted, and ideally to publish the per-case evidence.\n\nOne smaller issue: the Keras-is-on-TensorFlow caveat is acknowledged, so it is not a hidden flaw. The coverage numbers are single-run without variance, which limits how much weight to put on the RQ2/RQ5 comparisons; the conclusions there are modest and the sensitivity analysis helps.\n\nThis is not a field-reshaping paper, but it is a well-engineered testing contribution with reproducible empirical results and a replication package. I would bring it to our reading group and I would cite it in my own testing work. Send it to reviewers.","headline":"XAMT is a solid, thoroughly evaluated extension of the authors' ISSRE work; the single-input validation is the main soft spot, but the confirmed fixes and careful ablations support the 72-case claim.","tokens_in":16715,"tokens_out":1697,"would_cite":true,"duration_ms":16515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-library fuzzing exposes 72 deep learning API discrepancies","keywords":["cross-framework differential fuzzing","deep learning library testing","API matching","execution validation","differential oracle","non-finite inputs","variance-guided fuzzing","API alias normalization"],"falsifier":"Take a random sample of the 676 execution-validated groups and test each on a large set of diverse random inputs within their shared semantic domain; if a substantial fraction produce inconsistent outputs, the canonical-input validation is too weak and the 72-case count may be inflated. Alternatively, search more broadly around the 50 CPU-GPU-applicable triggering inputs; finding even one input that yields a CPU-GPU output difference beyond tolerance would refute the claim that these cases are invisible to backend differential oracles.","tokens_in":15685,"feed_emoji":"🐛","tokens_out":6278,"duration_ms":56160,"temperature":0.7,"pith_summary":"This paper tries to establish that cross-framework differential fuzzing can catch defects in deep learning library APIs that intra-library and backend-differential testing miss. The proposed approach, XAMT, builds groups of APIs across seven libraries that are supposed to implement the same operation, validates the groups by executing them on a canonical input, and then fuzzes them with ordinary, boundary, and non-finite inputs. The result is 72 independently reproduced discrepancies, of which 25 have been confirmed by developers and 23 fixed. The paper also claims that these findings complement CPU-GPU differential testing, since none of the 50 CPU-GPU-applicable triggering inputs produces a backend-disagreement signal. If true, this means cross-library comparison is a productive testing direction for deep learning infrastructure.","feed_headline":"Fuzzing 7 deep learning libraries uncovers 72 API discrepancies","feed_subtitle":"Execution-validated API groups flag defects that CPU-GPU differential oracles cannot see.","key_machinery":"The load-bearing object is the execution-validated API group. Candidate groups are formed by normalizing operation names through curated aliases and mapping parameters to semantic roles such as axis, data, shape, and keepdim, then each cross-library pair is executed on a canonical ordinary input. A pair passes only if both execute successfully and outputs agree after matching NaN and infinity masks and comparing finite elements within absolute tolerance $\\tau = 10^{-3}$. Connected components of passing pairs are re-executed as a whole, and only mutually consistent components are kept for fuzzing. Fuzzing then mutates shared logical inputs, including boundary and non-finite values, guided by a scale-normalized disagreement score, with Crash and Inconsistency oracles flagging discrepancies.","core_discovery":"XAMT identifies 72 valid discrepancy cases across seven deep learning libraries: 4 crashes and 68 output inconsistencies, produced by fuzzing 676 execution-validated cross-library API groups that contain 2,563 matched APIs. Of the 72 developer reports, 25 have been confirmed and 23 fixed. A fixed-input backend replay shows that 50 CPU-GPU-applicable cases produce no observable CPU-GPU disagreement, supporting the paper's central argument that these defects are invisible to backend-differential oracles and require cross-framework comparison.","pith_inferences":["Editorial inference: If the alias and role mappings are published as reusable data, the same pipeline could be applied to future library versions or new frameworks with relatively little manual effort, making the matching step semi-automatic.","Editorial inference: The fact that 59.7% of the 72 cases are not represented in converter-derived correspondence scopes suggests converter registries are an incomplete source of API equivalence, and execution-validated matching may cover a wider space of relations.","Editorial inference: All 27 MindSpore reports remain pending, so the true false-positive rate for that library is not yet established; re-running against a later MindSpore version could discriminate between slow triage and invalid correspondences.","Editorial inference: A direct testable extension would be applying XAMT's matching pipeline to additional libraries such as ONNX Runtime or oneDNN; if the alias mapping generalizes, it would support the claim that execution validation, not just the curated mapping, carries the approach."],"forward_implications":["Cross-framework differential fuzzing can serve as a complement to CPU-GPU oracles for deep learning library testing, covering defects that behave consistently across backends.","Real library defects are reachable at the API level: 25 confirmed and 23 fixed reports show the identified discrepancies map to actual code changes.","Boundary and non-finite inputs are a productive source of discrepancies; without them 22 of 72 reference cases are not reached, and 20 of those require such values.","API matching without a designated reference library expands testing scope: 444 of 676 groups contain APIs from three or more libraries.","Execution-based validation reduces the density of raw discrepancy candidates from 25.4 to 16.1 per 100 groups, filtering unsuitable correspondences before fuzzing."],"supporting_citations":[{"why":"The preliminary conference version of this work; supplies the cross-framework API matching idea and the five-library evaluation that this paper extends.","marker":"[14]"},{"why":"FreeFuzz, an API-level fuzzer used as a baseline in RQ4 to test whether existing methods can rediscover the same defects.","marker":"[12]"},{"why":"DeepREL, the relational API inference approach used as a second baseline for method-level rediscovery comparisons.","marker":"[25]"},{"why":"TensorScope, whose converter-derived counterpart relations are audited to measure how much of XAMT's correspondence scope is already covered.","marker":"[31]"},{"why":"CRADLE, which validates across backends; its CPU-GPU oracle is the reference design that XAMT's cross-framework comparison is positioned against.","marker":"[26]"},{"why":"DocTer, a documentation-guided fuzzer for deep learning APIs, cited as prior work in API-level fuzzing that motivates the input-generation and oracle design.","marker":"[8]"}],"fun_headline_variants":["Cross-framework fuzzing finds 72 API bugs in 7 deep learning libraries","72 API discrepancies found by fuzzing across 7 DL libraries","Cross-framework differential fuzzing exposes 72 API bugs","Fuzzing reveals 72 API flaws invisible to CPU-GPU checks","XAMT: 72 cross-library API bugs, 23 already fixed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the hand-curated alias mapping and parameter-role normalization together with the assumption that passing one canonical ordinary input per operation is enough to confirm that two APIs are semantically equivalent; if the mapping is wrong for an operation or the canonical input is too narrow, a reported discrepancy may reflect a bad correspondence rather than a genuine library defect.","fun_headline_variants_meta":{"raw":{"variants":["Cross-framework fuzzing finds 72 API bugs in 7 deep learning libraries","72 API discrepancies found by fuzzing across 7 DL libraries","Cross-framework differential fuzzing exposes 72 API bugs","Fuzzing reveals 72 API flaws invisible to CPU-GPU checks","XAMT: 72 cross-library API bugs, 23 already fixed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2822,"prompt_tokens":850,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":1875}},"tokens_in":466,"tokens_out":1972,"duration_ms":12763,"temperature":1.0,"reasoning_tokens":1875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:23:09.961077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 676 execution-validated groups and test each on a large set of diverse random inputs within their shared semantic domain; if a substantial fraction produce inconsistent outputs, the canonical-input validation is too weak and the 72-case count may be inflated. Alternatively, search more broadly around the 50 CPU-GPU-applicable triggering inputs; finding even one input that yields a CPU-GPU output difference beyond tolerance would refute the claim that these cases are invisible to backend differential oracles.","supporting_citations":[{"cited_title":"Xamt: Cross- framework api matching for testing deep learning libraries,","cited_arxiv_id":null,"evidence_quote":"The preliminary conference version of this work; supplies the cross-framework API matching idea and the five-library evaluation that this paper extends."},{"cited_title":"Free lunch for testing: Fuzzing deep-learning libraries from open source,","cited_arxiv_id":null,"evidence_quote":"FreeFuzz, an API-level fuzzer used as a baseline in RQ4 to test whether existing methods can rediscover the same defects."},{"cited_title":"Fuzzing deep-learning libraries via automated relational api inference,","cited_arxiv_id":null,"evidence_quote":"DeepREL, the relational API inference approach used as a second baseline for method-level rediscovery comparisons."},{"cited_title":"Differential testing of cross deep learning framework{APIs}: Revealing inconsisten- cies and vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"TensorScope, whose converter-derived counterpart relations are audited to measure how much of XAMT's correspondence scope is already covered."},{"cited_title":"Cradle: cross-backend validation to detect and localize bugs in deep learning libraries,","cited_arxiv_id":null,"evidence_quote":"CRADLE, which validates across backends; its CPU-GPU oracle is the reference design that XAMT's cross-framework comparison is positioned against."},{"cited_title":"Docter: documentation-guided fuzzing for testing deep learning api functions,","cited_arxiv_id":null,"evidence_quote":"DocTer, a documentation-guided fuzzer for deep learning APIs, cited as prior work in API-level fuzzing that motivates the input-generation and oracle design."}],"review_version":1}