{"id":"8bafc274-db56-4729-9adb-5848942af178","arxiv_id":"2508.12546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"XAMT matches functionally equivalent APIs across five deep learning frameworks and uses variance-guided differential testing to detect 17 bugs, 12 confirmed, that intra-framework tests miss.","lead":"This paper presents XAMT, a fuzzing tool that matches APIs with similar functions across five deep learning frameworks and tests them with identical inputs to uncover bugs. The approach targets defects that show up only when frameworks are compared, not when a single framework is run on CPU versus GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CPU/GPU invariance of the 17 reported bugs is asserted, not demonstrated; the central novelty claim is under-supported.","rationale":"The 12 developer-confirmed bugs are credible evidence that XAMT finds real issues, and the cross-framework differential design is a plausible complement to backend-based testing. However, the paper's headline claim requires showing that these bugs are invisible to intra-framework CPU/GPU comparison. The text asserts this in one sentence without a protocol or per-bug data. The comparison against FreeFuzz and DeepREL does not fill this gap, since those tools do not use CPU/GPU differential oracles. The reader's weakest assumption about the variance oracle is related but distinct: even if the oracle is reliable, the undetectability claim needs direct evidence. The paper's own acknowledgment of 'acceptable differences' plus 'no false positives' in §IV-D is an internal inconsistency that the authors need to resolve by clarifying how the 17 bugs were counted. My proposed check—running a real CPU/GPU differential oracle on the bug-triggering cases—would decisively test the central claim. Since this is a missing-evidence concern rather than a demonstrated flaw, the conditional verdict remains appropriate; the authors should be asked to provide this evidence. I therefore leave the reader's verdict unchanged.","tokens_in":15583,"tokens_out":6598,"duration_ms":78073,"concrete_test":"Take the 12 confirmed bug-triggering inputs (and, if available, the 3 'acceptable difference' cases) and run each API under a CPU-vs-GPU intra-framework differential oracle, e.g., CRADLE's oracle or a direct CPU/GPU execution compared with both a strict 1e-5 tolerance and a 0.1 output-difference threshold matching XAMT's variance threshold. If any of these cases shows a CPU/GPU output difference above either threshold, that bug is detectable by intra-framework testing, weakening the central undetectability claim. Report per-case CPU and GPU outputs and oracle verdicts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that XAMT detects bugs that manifest identically on CPU and GPU and are therefore invisible to intra-framework differential testing. The only support is the statement in §IV-D: 'We verified that such NaN and Inconsistency bugs produce identical outputs across GPU and CPU.' No per-bug protocol, measurement, or tolerance is reported, and no comparison is made against an actual intra-framework CPU-vs-GPU oracle (e.g., CRADLE-style cross-backend validation). The baselines used (FreeFuzz, DeepREL) are not backend-differential testers, so their failure to detect these bugs does not establish undetectability by backend-differential methods. In addition, §IV-D says three inconsistencies were 'acceptable differences' while also claiming 'no false positives,' making the '17 bugs' count ambiguous; and the API-group equivalence that the variance oracle relies on was validated with only 10 random inputs per group (§IV-A2), so an edge-case divergence between matched but genuinely non-equivalent APIs could be misreported as a bug. The most novel part of the contribution—cross-framework detection of CPU/GPU-invariant bugs—is therefore not yet evidenced by the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XAMT, a cross-framework fuzzing method for deep learning libraries. XAMT matches functionally equivalent APIs across PyTorch, TensorFlow, Keras, Chainer, and JAX using similarity rules over API names, descriptions, and parameter structures, aligns their inputs, and then applies variance-guided differential fuzzing with crash, NaN, and inconsistency oracles. The evaluation reports 238 matched API groups involving 839 APIs and 17 detected bugs, 12 of which were confirmed by developers. The paper's central claim is that XAMT detects bugs that manifest identically on CPU and GPU and are therefore invisible to existing intra-framework, backend-differential testing methods.","tokens_in":15942,"tokens_out":4935,"duration_ms":55239,"significance":"The cross-framework testing idea is timely and, if fully substantiated, would be a useful complement to backend-differential fuzzing. The paper provides a reproducible matching pipeline, concrete case studies with executable inputs (Listings 1–2), and an honest attempt to compare with FreeFuzz and DeepREL. The public code availability and the fact that 12 bugs were developer-confirmed are concrete strengths. However, the central novelty claim — that the detected bugs are CPU/GPU-invariant and hence undetectable by intra-framework methods — is currently supported mainly by a one-sentence assertion, and the inconsistency oracle is partly circular with the fuzzing objective. These gaps require additional experiments before the main contribution can be accepted.","major_comments":[{"comment":"The paper's central novelty claim is that XAMT detects bugs that 'manifest identically on CPU vs GPU' and are 'undetectable by existing intra-framework testing methods.' The only support is the sentence in §IV-D: 'We verified that such NaN and Inconsistency bugs produce identical outputs across GPU and CPU.' No per-bug protocol, no output measurements, no tolerance, and no comparison with an actual CPU-vs-GPU differential oracle is reported. The baselines used in §IV-C (FreeFuzz, DeepREL) are not backend-differential testers, so their failure to find these bugs does not establish undetectability by backend-differential methods. Please add a systematic experiment that runs each reported bug on CPU and GPU for the same framework, reports the outputs and differences, and ideally compares against a CRADLE-style backend-differential oracle. If the invariance claim cannot be substantiated, the","section":"§IV-D, Table V"},{"comment":"The inconsistency oracle and the fuzzing objective are the same quantity: Eq. (7) defines σ²(x) as the variance across APIs, and §II-C2 guides mutation to maximize it via the deviation vector d(x) in Eq. (8); §II-D3 then flags outputs with σ²(x) ≥ 0.1. Detected 'inconsistency' cases are therefore variance maxima by construction. The threshold 0.1 is selected from 'preliminary empirical analyses,' and the variance-improvement threshold (0.001) and reinitialization limit (20) are also manually set. Developer confirmation of 12 cases partially mitigates this, but the reported 7 inconsistency bugs do not form an independent sample. Please report all cases that exceeded the threshold, the distribution of variances, and a sensitivity analysis of the threshold; and separate developer-confirmed bugs from 'acceptable differences' and unconfirmed cases.","section":"§II-C2, §II-D3, §III-B"},{"comment":"Behavioral validation of matched API groups uses only 10 randomly generated input samples per group. This is too weak to establish functional equivalence for 238 groups, especially for the edge cases (signed zeros, denormals, NaN, complex inputs) that later trigger bugs in Listings 1–2. If some matched APIs are not truly equivalent under edge-case inputs, the inconsistency oracle will flag legitimate implementation differences as bugs. Please strengthen the validation protocol (more samples, edge cases, per-group statistical equivalence) or restrict bug claims to developer-confirmed cases, and discuss the impact on the three 'acceptable differences' reported in §IV-D.","section":"§IV-A2"},{"comment":"The baseline bug-detection comparison is performed only on the 8 API cases in PyTorch/TensorFlow where XAMT already found bugs, not on a common random sample of APIs. Moreover, FreeFuzz and DeepREL are not intra-framework backend-differential testers; they use different oracles (e.g., relational APIs and documentation constraints). Therefore Table IV cannot support the claim that these bugs are 'undetectable by intra-framework testing that relies on backend-induced discrepancies.' Please compare on a fair common API set and, if possible, include a backend-differential baseline (e.g., CRADLE-style CPU-vs-GPU comparison) or explicitly restrict the claim to the compared methods.","section":"§IV-C2, Table IV"},{"comment":"The bug counts are ambiguous. The text says 17 detected, 12 confirmed, 3 considered acceptable differences, and 2 received no response, and then states 'Through manual validation, we confirmed that no false positives among these.' If three inconsistencies are acceptable differences, they cannot simultaneously be counted as true bugs; either they are false positives under the bug definition or they should be removed from the 17. Please report exact per-oracle counts after excluding acceptable differences and unconfirmed cases, and make the counting criterion explicit.","section":"§IV-D"}],"minor_comments":[{"comment":"Header typo: 'Chanier' should be 'Chainer.'","section":"Table V"},{"comment":"SIM_Param is defined as SIM_Count + SIM_Type, which can range up to 2. Stage 3 then requires SIM_Param = 2. Please clarify this range explicitly and justify why an additive combination is appropriate.","section":"§II-A3, Eq. (6)"},{"comment":"Minor text issue: 'we use X AMTto generate' should be 'XAMT to generate.' Several other spacing issues around 'X AMT' throughout.","section":"§III-B"},{"comment":"The hyperparameter paragraph lists several thresholds (relative margin 0.3, variance improvement 0.001, variance threshold 0.1, reinitialization limit 20) without a sensitivity analysis. Since the main oracle relies on these thresholds, at least a brief ablation would increase confidence.","section":"§III-B"},{"comment":"The paper filters 'control-related parameters' (Table I) with the stated assumption that removing them preserves functional equivalence, but §V later acknowledges that control parameters such as dtype can affect numerical outcomes. This tension should be discussed in the paper.","section":"§II-A3"},{"comment":"Figure 3 is referenced in §IV-A1 but not included in the text. Please ensure the figure is present in the final version.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and interesting central idea, but the current evidence for the CPU/GPU-invariance claim is mostly asserted rather than demonstrated, and the inconsistency oracle's circularity with the fuzzing objective needs to be addressed. A revision that adds a proper CPU-vs-GPU verification, a threshold sensitivity analysis, and a clearer accounting of confirmed vs. acceptable differences would make the contribution publishable. The scope is a good fit for the journal, and the artifact availability is a positive sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: XAMT is a legitimate extension of cross-framework differential testing, and the confirmed bug set is real value. But the paper's headline claim—that these bugs are invisible to CPU/GPU differential testing—is asserted, not shown.\n\nWhat's new: instead of relying on model converters (TensorScope), XAMT matches APIs from official documentation using name edit distance, Sentence-BERT description similarity, and parameter structure after filtering control-related parameters. That's a sensible, portable matching pipeline. The variance-guided mutation strategy is a reasonable way to amplify divergence, and the evaluation on five frameworks—especially the 238 matched groups and 12 confirmed bugs—is concrete. The argsort signed-zero and complex NaN angle cases are credible, reproducible-looking bugs that developers confirmed.\n\nSoft spots, in order of weight:\n\n1. The CPU/GPU invariance claim. The paper says \"we verified that such NaN and Inconsistency bugs produce identical outputs across GPU and CPU,\" but gives no protocol, no per-bug measurements, no tolerance. And the baselines used—FreeFuzz and DeepREL—are not backend-differential testers, so their failure to find these bugs says nothing about whether CRADLE-style CPU-vs-GPU oracles would catch them. This is the central novelty claim, and it needs real evidence.\n\n2. The oracle is partly circular. The variance threshold σ² ≥ 0.1 is the same quantity the fuzzer is guided to maximize. So every reported inconsistency is a variance maximum by construction. Developer confirmation is independent evidence and helps, but the accounting is sloppy: three cases were \"acceptable differences\" while the paper claims \"no false positives.\" That needs to be reconciled.\n\n3. Matching validation is thin. Ten random inputs per group won't catch edge-case divergences between genuinely non-equivalent APIs, which is exactly what the inconsistency oracle might misreport.\n\n4. Thresholds (0.3 relative margin, 0.001 improvement, 0.1 variance) are tuned on preliminary experiments; a sensitivity analysis would help. Also, \"all code is publicly available\" but there's no link in the text.\n\nNone of this is fatal. The approach is plausible, the bug set is useful, and the limitation section is honest about common-mode bugs. But the paper currently overstates what it demonstrates.\n\nWho it's for: software testing researchers, especially those working on DL library fuzzing. It deserves a serious referee, but the revision needs to supply the artifact, the CPU/GPU protocol, and a cleaner false-positive count. If the authors can show even a few bugs that a real backend-differential oracle misses, the claim will hold.","headline":"XAMT is a useful, well-evaluated extension of cross-framework differential testing, but its central CPU/GPU-invariance claim is asserted, not demonstrated.","tokens_in":16317,"tokens_out":2504,"would_cite":true,"duration_ms":27117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that cross-framework differential testing of equivalent deep learning APIs can detect bugs that manifest identically on CPU and GPU, which intra-framework backend comparisons miss.","keywords":["deep learning testing","differential fuzzing","API matching","cross-framework testing","variance-guided fuzzing","deep learning libraries"],"falsifier":"A concrete counterexample would be a matched API pair that passes the behavioral verification on 10 random inputs but later shows a high-variance output that is a legitimate framework-specific design choice, not a bug—for instance, different deterministic tie-breaking in sorting or different handling of signed zeros without any numerical error. If such cases are common, then the inconsistency oracle would produce false positives that cannot be distinguished from real bugs without manual inspection.","tokens_in":805,"feed_emoji":"🐛","tokens_out":2140,"duration_ms":46189,"temperature":0.7,"pith_summary":"XAMT matches functionally equivalent APIs across PyTorch, TensorFlow, Keras, Chainer, and JAX using similarity rules over names, documentation descriptions, and parameter structures. It then feeds aligned inputs to each matched API and uses variance-guided mutation to amplify output divergence, flagging crashes, NaNs, and inconsistencies. The paper's central claim is that this cross-framework oracle catches bugs that produce identical CPU and GPU outputs, so existing intra-framework testing methods cannot see them. In an evaluation of 839 matched APIs and 238 equivalence groups, XAMT found 17 bugs, 12 confirmed by developers. If this holds, cross-framework comparison becomes a necessary complement to backend-based fuzzing for deep learning library reliability.","feed_headline":"Cross-framework fuzzing finds hidden deep learning bugs","feed_subtitle":"APIs that behave the same on CPU and GPU still diverge across frameworks—XAMT catches those.","key_machinery":"The central mechanism is the variance-guided differential fuzzing loop, built on the output variance $\\sigma^2(x) = \\frac{1}{n}\\sum_i (A_i(x)-\\mu(x))^2$ and the deviation vector $d(x) = [A_1(x)-\\mu(x),\\ldots,A_n(x)-\\mu(x)]$ across $n$ matched APIs. This object is both the oracle signal (inconsistency is high variance under equivalent inputs) and the mutation guide (perturbations aligned with $d(x)$ push toward higher divergence), making it the single device that connects matching, fuzzing, and bug detection.","core_discovery":"The paper establishes that behavioral comparison across, rather than within, deep learning frameworks is a viable and effective bug-detection strategy. It introduces a matching pipeline that combines normalized Levenshtein name similarity, Sentence-BERT description similarity, and structural parameter similarity with control-related parameters filtered out, yielding groups of APIs that are functionally equivalent at the level of tensor inputs and outputs. On these groups, XAMT performs variance-guided differential fuzzing: it computes the element-wise output variance across the APIs, builds a deviation vector, and mutates inputs in the direction that increases divergence, using simulated ann","pith_inferences":["If the variance threshold $\\sigma^2 \\ge 0.1$ is taken as the bug criterion, then any framework pair that passes behavioral verification on 10 random inputs may still hide rare-input inconsistencies, so the matching validation is only as strong as the seed distribution.","The three acknowledged 'acceptable differences' suggest that the inconsistency oracle cannot by itself separate bugs from legitimate implementation choices; a human or reference oracle is still needed to adjudicate high-variance cases.","The paper's implicit bet is that documentation-derived equivalence is a faithful proxy for functional equivalence; a testable extension is to check whether APIs matched with high semantic similarity but low lexical overlap (e.g., `softmax_cross_entropy` vs. `crossentropy_loss`) also fail the same edge-case tests, which would strengthen or weaken the matching rules."],"forward_implications":["If the approach generalizes, future deep learning library testers should treat cross-framework comparison as a standard oracle alongside CPU/GPU differential testing.","Bugs that are silent on both backends—numerical edge cases, sorting-tie inconsistencies, NaN propagation differences—become detectable through this method.","The matching pipeline, because it relies only on documentation and signatures, can be ported to additional frameworks such as PaddlePaddle or MXNet without retraining models.","Variance-guided mutation, shown to outperform random inputs in both coverage and bug count, offers a reusable strategy for other differential testing settings.","Confirmed bugs in TensorFlow and JAX indicate that widely deployed libraries still contain implementation-level inconsistencies that affect real workloads."],"supporting_citations":[{"why":"FreeFuzz is the primary API-level fuzzing baseline that XAMT compares against for coverage and bug detection.","marker":"[35]"},{"why":"DeepREL is the other API-level fuzzing baseline, providing a relational-oracle method that XAMT must outperform.","marker":"[12]"},{"why":"Sentence-BERT supplies the semantic embeddings used in the description-similarity matching rule.","marker":"[29]"},{"why":"Levenshtein distance defines the API name similarity metric that drives the first-stage candidate retrieval.","marker":"[20]"},{"why":"DocTer is cited as a documentation-guided fuzzing approach that XAMT builds on and distinguishes from.","marker":"[38]"}],"fun_headline_variants":["Cross-framework fuzzing finds bugs that backend testing misses","Match DL APIs across frameworks to spot divergence bugs","XAMT: differential test across frameworks to find DL bugs","Find DL bugs by comparing API behavior across frameworks"],"cache_read_input_tokens":18304,"weakest_assumption_plain":"The entire detection pipeline assumes that after filtering control-related parameters, the remaining parameter structures and the 10-sample behavioral check are enough to guarantee that two APIs are truly functionally equivalent, so that any later output divergence is a bug.","fun_headline_variants_meta":{"raw":{"variants":["Cross-framework fuzzing finds bugs that backend testing misses","Match DL APIs across frameworks to spot divergence bugs","XAMT: differential test across frameworks to find DL bugs","Find DL bugs by comparing API behavior across frameworks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1494,"prompt_tokens":736,"completion_tokens":758,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":480,"tokens_out":758,"duration_ms":8770,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:25:11.042055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete counterexample would be a matched API pair that passes the behavioral verification on 10 random inputs but later shows a high-variance output that is a legitimate framework-specific design choice, not a bug—for instance, different deterministic tie-breaking in sorting or different handling of signed zeros without any numerical error. If such cases are common, then the inconsistency oracle would produce false positives that cannot be distinguished from real bugs without manual inspection.","supporting_citations":[{"cited_title":"Free lunch for testing: Fuzzing deep-learning libraries from open source","cited_arxiv_id":null,"evidence_quote":"FreeFuzz is the primary API-level fuzzing baseline that XAMT compares against for coverage and bug detection."},{"cited_title":"Fuzzing deep-learning libraries via automated relational api inference","cited_arxiv_id":null,"evidence_quote":"DeepREL is the other API-level fuzzing baseline, providing a relational-oracle method that XAMT must outperform."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks","cited_arxiv_id":null,"evidence_quote":"Sentence-BERT supplies the semantic embeddings used in the description-similarity matching rule."},{"cited_title":"Binary codes capable of correcting deletions, insertions, and reversals","cited_arxiv_id":null,"evidence_quote":"Levenshtein distance defines the API name similarity metric that drives the first-stage candidate retrieval."},{"cited_title":"Docter: documentation-guided fuzzing for testing deep learning api functions","cited_arxiv_id":null,"evidence_quote":"DocTer is cited as a documentation-guided fuzzing approach that XAMT builds on and distinguishes from."}],"review_version":1}