{"id":"4f04bca3-f85c-444a-a7f8-0320d54e2bac","arxiv_id":"2509.06977","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A configuration-first, three-tier verification framework quantifies cross-backend drift in deep learning, reporting a 72% pass rate across 672 checks and offering deterministic adapters as mitigations.","lead":"This paper proposes a YAML-driven framework that checks whether deep learning models produce consistent outputs across CPU, GPU, and compiled backends, using three levels of verification. It reports that 72% of 672 cross-backend checks pass, with detection models and compiled runtimes most prone to drift, and shows that deterministic pre-NMS sorting can restore agreement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The artifact runner aliases reference and target models due to in-place .to(), so the 672 cross-backend checks may not measure real drift.","rationale":"The reader's weakest assumption focuses on limited sample size and deferred activation surveys, which are valid external-validity concerns. However, a more severe internal-correctness issue exists in the provided implementation: the reference and target models are the same in-place-mutated object. This makes the central empirical claim (the 72% pass rate and tolerance trends) unsupported or even meaningless as a cross-backend measurement. Since the appendix explicitly states that the provided code can regenerate all results, the bug is a load-bearing flaw. I recommend REJECT until the runner is corrected and the experiments re-run, or until the authors provide evidence that the artifact differs from the code actually used. The concern is concrete and testable; if the test shows no aliasing (e.g., the code uses `.to(..., copy=True)` or separate model instances), then the concern would not land and the verdict could revert to CONDITIONAL.","tokens_in":6898,"tokens_out":4793,"duration_ms":53988,"concrete_test":"Run the provided runner with a single YAML config and `--device cuda` (without `--compile`), and observe whether it raises a device mismatch error. Alternatively, insert two debug lines after the two `.to()` calls: `print(model_ref is model_tgt)` and `print(next(model_ref.parameters()).device)`. If it prints `True` and `cuda`, the aliasing bug is confirmed and the published results cannot be reproduced from the artifact as-is.","verdict_should_be":"REJECT","load_bearing_attack":"The sanitized runner in the Appendix (run_once) builds a single `model`, then executes `model_ref = model.to(ref_device)` followed by `model_tgt = model.to(tgt_device)`. In PyTorch, `nn.Module.to()` mutates the module in-place and returns the same object. Therefore `model_ref` and `model_tgt` refer to the same module, both ultimately located on the target device (e.g., CUDA). This means the 'reference' CPU execution is actually performed on the target device, and the comparison either throws a device-mismatch error (CPU input on GPU model) or compares identical executions. The reported pass rates (72.0% aggregate, Table III) and tolerance sensitivity (Table II) are consequently not interpretable as cross-backend drift. This is more fundamental than the sample-size concern: it invalidates the empirical basis for RQ1–RQ3 and, by extension, the central claim that a unified framework systematically quantifies drift.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a configuration-first framework for evaluating cross-backend behavioral drift in deep learning models. The framework decouples experiment specifications via YAML, uses a three-tier verification protocol (tensor closeness, activation alignment, task-level metrics), and reports an empirical study across 19 models, two target backends (GPU eager and torch.compile), and four tolerance settings, totaling 672 cross-backend checks. The central claim is a 72.0% aggregate pass rate, with most discrepancies occurring at tighter tolerances, and the paper additionally demonstrates deterministic adapters (e.g., pre-NMS sorting) that improve agreement. The paper also proposes a failure taxonomy and latency analysis, and provides a sanitized artifact for reproducibility.","tokens_in":7175,"tokens_out":5328,"duration_ms":59821,"significance":"If valid, the framework would be a useful contribution to the reproducibility and deep-learning-systems literature: it provides a practical tool for measuring and mitigating cross-backend drift, and the reported empirical landscape would inform deployment decisions. The paper also makes a concrete methodological proposal—linking tensor-level differences to task-level outcomes—that goes beyond simple allclose checks. However, the significance hinges entirely on the correctness of the empirical results and the artifact. The sanitized runner and the experimental accounting contain inconsistencies that make the reported numbers unexplainable. Because these problems invalidate the core empirical claims, the contribution as submitted cannot be accepted without a complete re-evaluation.","major_comments":[{"comment":"The provided artifact runner constructs a single `model`, then executes `model_ref = model.to(ref_device)` followed by `model_tgt = model.to(tgt_device)`. Since `nn.Module.to()` mutates the module in-place and returns the same object, `model_ref` and `model_tgt` refer to the same module, both ultimately residing on the target device. Consequently, `model_ref(x_ref)` either raises a device-mismatch error (counted as FAIL by the `except` branch in `main`) or, if the target is CPU, compares identical executions. The reported 484 passes (Table III) and the pass rates in Table II cannot be produced by this runner. This invalidates the empirical basis for RQ1–RQ3 and, by extension, the central claim of the paper.","section":"Appendix, run_once (lines 42-45)"},{"comment":"The claim that \"most failures concentrate at 10^-6\" is contradicted by Table II. At atol=1e-6 there are 120 failures out of 168, but the same 120 failures occur at atol=1e-5 and atol=1e-4; only four fewer failures occur at atol=1e-3. The pass rate changes from 71.4% to only 73.8%. The data therefore do not support the statement that \"fine-grained numerical perturbations are the principal driver.\" The monotonic improvement is marginal and no per-model breakdown or error measurement is provided to substantiate the conclusion.","section":"Table II / §V-A"},{"comment":"The experimental accounting is internally inconsistent. With 19 distinct models, 2 target backends, and 4 tolerance settings, the maximum number of cross-backend checks is 19 × 2 × 4 = 152, not 672. To obtain 168 checks per tolerance (as in Table II), one would need 21 model-input combinations per tolerance or additional dimensions not described in the setup. The paper does not specify multiple inputs or repeated runs, so the reported totals (168 per atol, 672 overall) are not derivable from the stated experimental configuration. This further undermines the aggregate pass-rate claim and the quantitative answers to RQ1–RQ3.","section":"Table III / §V-E"}],"minor_comments":[{"comment":"Typo: \"Y AML\" should be \"YAML\" in the bullet list.","section":"§I-C"},{"comment":"The figure caption says \"left shows ... right shows ...\" but the figure itself is not described; please ensure the panels are labeled and the caption is self-contained.","section":"Fig. 7"},{"comment":"The pass-rate heatmap is mentioned but no numerical per-model breakdown is given in the text. The claims about which models/tasks diverge would be easier to evaluate with a table of per-model pass rates or failure counts.","section":"§V-B"},{"comment":"The Scope note explicitly defers a comprehensive activation survey, yet §V-C answers RQ3 (\"where does divergence emerge?\") based on selective probes. The answer should be clearly labeled as preliminary feasibility demonstration, not a definitive localization result.","section":"§III-B"},{"comment":"No error bars, confidence intervals, or repeated-run statistics are reported. Since the framework emphasizes reproducibility, reporting variance across seeds or runs would strengthen the empirical claims.","section":"General"}],"recommendation":"reject","confidential_remarks":"The artifact bug is not a cosmetic or presentation issue; it makes the central quantitative result (72.0% pass rate) unexplainable from the provided code. This is a load-bearing error. The authors could potentially correct the runner and re-run the experiments, but as submitted the empirical core is invalid. The reviewers also noted an arithmetic inconsistency in the experimental counts. If the authors can demonstrate that the artifact was not the one used, or re-run the study with a corrected runner and transparent accounting, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Interesting approach, but the appendixed runner has a load-bearing bug. In run_once, model_ref = model.to(ref_device) then model_tgt = model.to(tgt_device). nn.Module.to() mutates in place, so both names reference the same module, now on the target device. For a CUDA target, the 'reference' run with CPU input throws, or if CPU target, you're comparing eager vs compiled on the same device — not cross-backend drift. So the 72% pass rate and tolerance sweeps in Tables II–III are not evidence for RQ1–RQ3. I had to check the PyTorch semantics twice, but they're unambiguous. The paper says the provided code can reproduce everything; it can't.\n\nThat's a shame, because the design thinking is solid. The YAML-driven config, three-tier verification (tensor, activation, task), and the deterministic adapter idea are reasonable contributions that could help standardize compatibility testing. The failure taxonomy — order-sensitivity in NMS, partial operator coverage — is plausible and matches what I've seen. The case study with pre-NMS sorting is a nice illustration, though it's anecdotal.\n\nOther weaknesses are secondary but real: only two target backends, no full data release, \"first unified framework\" claim overstates the novelty. And the tolerance sensitivity is weak — 71.4% to 73.8% is hardly 'most discrepancies at stricter tolerances.' The paper is honest about the activation survey being deferred, and the threats-to-validity section is reasonable.\n\nWho is this for? People building cross-backend CI pipelines will get useful design ideas from the YAML protocol, but they'll need to redo the empirical work. The paper deserves a serious referee because the framework is worth scrutiny and the bug needs to be caught — but as it stands, the central empirical claim is invalid. Without the reproducibility issue, I'd have said minor revision; with it, this is a reject-and-resubmit with a mandatory code audit.","headline":"The code's .to() bug means the 672 checks never compared CPU vs GPU — the quantitative results don't stand, though the framework idea is worth a conversation.","tokens_in":7575,"tokens_out":3490,"would_cite":false,"duration_ms":36264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Only 72% of deep learning runs pass cross-backend checks","keywords":["cross-backend drift","reproducibility","deep learning","three-tier verification","deterministic adapters","configuration-first","nondeterministic NMS","numerical tolerance"],"falsifier":"Run the same 672-check protocol on a different GPU model or library version; if the aggregate pass rate departs substantially from 72%, or if detection discrepancies persist after deterministic pre-NMS sorting, the central claims about drift magnitude and its primary source would be falsified.","tokens_in":6840,"feed_emoji":"🧪","tokens_out":5021,"duration_ms":52608,"temperature":0.7,"pith_summary":"The paper claims that cross-backend behavioral drift in deep learning—where the same model produces different outputs on CPU, GPU, and compiled runtimes—is measurable, partly systematic, and largely fixable. It introduces a configuration-first framework in which experiments are defined in YAML files and verified by a three-tier protocol comparing tensor closeness, activation alignment, and task-level metrics. Across 672 checks spanning four tolerance thresholds and 19 models, 72.0% passed; most failures concentrate at tighter tolerances, and detection models on compiled backends are the most drift-prone, largely because of nondeterministic post-processing such as NMS tie-breaking. The paper demonstrates that deterministic adapters and selective fallbacks restore agreement without meaningful performance loss. A sympathetic reader would care because consistent behavior across runtimes is a prerequisite for safe deployment in domains like medical imaging and autonomous driving.","feed_headline":"Only 72% of deep learning runs pass cross-backend checks","feed_subtitle":"Three-tier verification shows most failures under strict tolerances, in detection models and compiled runtimes.","key_machinery":"The central object is a configuration-first runner: each experiment is a YAML file specifying the model source (library or repository), preprocessing, backends, and tolerances, decoupling experiment design from code. The three-tier verification protocol carries the argument: Tier-1 tensor closeness via a tolerance inequality, Tier-2 activation alignment via layerwise hooks, and Tier-3 task-level metrics with deterministic adapters such as sorted NMS. The key identity enabling the case study is that compatibility can fail even when tensor differences are within 1e-5, because ordering-sensitive post-processing amplifies small perturbations; the deterministic sort removes that amplification.","core_discovery":"The central claim is that cross-backend drift can be systematically quantified and mitigated under a unified protocol that links tensor-level numerical differences to task-level outcomes. The paper reports that across four tolerance settings (absolute tolerance from 1e-6 to 1e-3) and 672 cross-backend checks, 484 checks (72.0%) pass, with discrepancies concentrated at stricter tolerances. Detection models on compiled backends are especially prone to drift, and the dominant failure mechanism is ordering and tie-breaking in post-processing, exemplified by pre-NMS candidate box ordering. Enforcing a deterministic sort before NMS restores task-level agreement. The paper presents this as the firs","pith_inferences":["If the deterministic-sort fix generalizes, it suggests a broader design principle: cross-backend compatibility should be engineered at the post-processing interface rather than only at the kernel level, since ordering is a discrete amplification point.","The tentative localization of drift to early convolutional layers is based on selective probing; a full activation survey could reveal whether that pattern is architecture-specific.","The 72% figure is tied to the specific backend pairs and library versions tested; the framework's JSONL logs could be used to build a public benchmark of cross-backend drift, though that is a step beyond the paper.","Extending the tolerance sweep beyond 1e-3 would likely push pass rates near 100%, but the practical threshold should depend on how sensitive the task is to output permutations, not just numerical magnitude."],"forward_implications":["If replicated, the 72% pass rate across 672 checks implies that roughly a quarter of cross-backend deployment runs can be expected to diverge under practical tolerances, with the tightest tolerance (1e-6) causing most failures.","Detection models and compiled backends need extra scrutiny; ordering and tie-breaking operations in post-processing are a disproportionate source of drift.","Deterministic adapters, such as sorting candidates before NMS, can restore agreement without sacrificing the latency benefits of compilation.","Selective fallbacks to eager execution or FP32 for unstable kernels improve stability while preserving most performance gains.","The YAML-driven configuration design makes the verification protocol portable to other model families and runtimes, including those the paper lists as future work."],"supporting_citations":[{"why":"Supplies the input-space white-box testing baseline that the paper contrasts with cross-backend compatibility.","marker":"[1]"},{"why":"Establishes task-level metric use in testing, which the three-tier protocol extends.","marker":"[2]"},{"why":"Provides a multi-interface testing perspective that motivates checking across backend interfaces.","marker":"[4]"},{"why":"Defines the compiler transformation context (TVM/XLA/Glow/Inductor) that explains graph-rewrite-induced drift.","marker":"[5]"},{"why":"Gives the reproducibility guidelines that frame the need for configuration-first experimental control.","marker":"[6]"}],"fun_headline_variants":["AI backend drift: only 72% of runs pass cross-backend checks","Nondeterministic post-processing causes most cross-backend AI drift","Deterministic sort fixes cross-backend drift in detection models","First unified framework quantifies and mitigates cross-backend deep learning drift"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The small evaluation sample—19 models and only two target backends, with activation probing applied selectively—is assumed sufficient to generalize the observed 72% pass rate and the localization of drift to post-processing ordering.","fun_headline_variants_meta":{"raw":{"variants":["AI backend drift: only 72% of runs pass cross-backend checks","Nondeterministic post-processing causes most cross-backend AI drift","Deterministic sort fixes cross-backend drift in detection models","First unified framework quantifies and mitigates cross-backend deep learning drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2168,"prompt_tokens":686,"completion_tokens":1482,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1404}},"tokens_in":430,"tokens_out":1482,"duration_ms":12654,"temperature":1.0,"reasoning_tokens":1404,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:57:05.341544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 672-check protocol on a different GPU model or library version; if the aggregate pass rate departs substantially from 72%, or if detection discrepancies persist after deterministic pre-NMS sorting, the central claims about drift magnitude and its primary source would be falsified.","supporting_citations":[{"cited_title":"DeepXplore: Automated Whitebox Testing of Deep Learning Systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the input-space white-box testing baseline that the paper contrasts with cross-backend compatibility."},{"cited_title":"DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,","cited_arxiv_id":null,"evidence_quote":"Establishes task-level metric use in testing, which the three-tier protocol extends."},{"cited_title":"Mist: Automated Neural Network Model Testing via Multiple Interfaces,","cited_arxiv_id":null,"evidence_quote":"Provides a multi-interface testing perspective that motivates checking across backend interfaces."},{"cited_title":"TVM: An Automated End-to-End Optimizing Compiler for Deep Learning,","cited_arxiv_id":null,"evidence_quote":"Defines the compiler transformation context (TVM/XLA/Glow/Inductor) that explains graph-rewrite-induced drift."},{"cited_title":"Improving Repro- ducibility in Machine Learning Research: A Report from the NeurIPS 2019 Reproducibility Program,","cited_arxiv_id":null,"evidence_quote":"Gives the reproducibility guidelines that frame the need for configuration-first experimental control."}],"review_version":1}