Toward Reproducible Cross-Backend Compatibility for Deep Learning: A Configuration-First Framework with Three-Tier Verification

Zehua Li

REVIEW 3 major objections 5 minor 6 references

Toward Reproducible Cross-Backend Compatibility for Deep Learning: A Configuration-First Framework with Three-Tier Verification

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Only 72% of deep learning runs pass cross-backend checks

desk verdict The code's .to() bug means the 672 checks never compared CPU vs GPU — the quantitative results don't stand, though the framework idea is worth a conversation. read the letter →

arxiv 2509.06977 v1 pith:HZ4UHYW5 submitted 2025-08-29 cs.LG cs.AI

Zehua Li This is my paper · ORCID

classification cs.LGcs.AI

keywords cross-backenddriftreproducibilitydeeplearningthree-tierverificationdeterministicadaptersconfiguration-firstnondeterministicNMSnumericaltolerance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that cross-backend behavioral drift in deep learning—where the same model produces different outputs on CPU, GPU, and compiled runtimes—is measurable, partly systematic, and largely fixable. It introduces a configuration-first framework in which experiments are defined in YAML files and verified by a three-tier protocol comparing tensor closeness, activation alignment, and task-level metrics. Across 672 checks spanning four tolerance thresholds and 19 models, 72.0% passed; most failures concentrate at tighter tolerances, and detection models on compiled backends are the most drift-prone, largely because of nondeterministic post-processing such as NMS tie-breaking. The paper demonstrates that deterministic adapters and selective fallbacks restore agreement without meaningful performance loss. A sympathetic reader would care because consistent behavior across runtimes is a prerequisite for safe deployment in domains like medical imaging and autonomous driving.

What carries the argument

The central object is a configuration-first runner: each experiment is a YAML file specifying the model source (library or repository), preprocessing, backends, and tolerances, decoupling experiment design from code. The three-tier verification protocol carries the argument: Tier-1 tensor closeness via a tolerance inequality, Tier-2 activation alignment via layerwise hooks, and Tier-3 task-level metrics with deterministic adapters such as sorted NMS. The key identity enabling the case study is that compatibility can fail even when tensor differences are within 1e-5, because ordering-sensitive post-processing amplifies small perturbations; the deterministic sort removes that amplification.

What would settle it

Run the same 672-check protocol on a different GPU model or library version; if the aggregate pass rate departs substantially from 72%, or if detection discrepancies persist after deterministic pre-NMS sorting, the central claims about drift magnitude and its primary source would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that cross-backend drift can be systematically quantified and mitigated under a unified protocol that links tensor-level numerical differences to task-level outcomes. The paper reports that across four tolerance settings (absolute tolerance from 1e-6 to 1e-3) and 672 cross-backend checks, 484 checks (72.0%) pass, with discrepancies concentrated at stricter tolerances. Detection models on compiled backends are especially prone to drift, and the dominant failure mechanism is ordering and tie-breaking in post-processing, exemplified by pre-NMS candidate box ordering. Enforcing a deterministic sort before NMS restores task-level agreement. The paper presents this as the firs

Load-bearing premise

The small evaluation sample—19 models and only two target backends, with activation probing applied selectively—is assumed sufficient to generalize the observed 72% pass rate and the localization of drift to post-processing ordering.

Editorial extensions

If this is right

If replicated, the 72% pass rate across 672 checks implies that roughly a quarter of cross-backend deployment runs can be expected to diverge under practical tolerances, with the tightest tolerance (1e-6) causing most failures.
Detection models and compiled backends need extra scrutiny; ordering and tie-breaking operations in post-processing are a disproportionate source of drift.
Deterministic adapters, such as sorting candidates before NMS, can restore agreement without sacrificing the latency benefits of compilation.
Selective fallbacks to eager execution or FP32 for unstable kernels improve stability while preserving most performance gains.
The YAML-driven configuration design makes the verification protocol portable to other model families and runtimes, including those the paper lists as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the deterministic-sort fix generalizes, it suggests a broader design principle: cross-backend compatibility should be engineered at the post-processing interface rather than only at the kernel level, since ordering is a discrete amplification point.
The tentative localization of drift to early convolutional layers is based on selective probing; a full activation survey could reveal whether that pattern is architecture-specific.
The 72% figure is tied to the specific backend pairs and library versions tested; the framework's JSONL logs could be used to build a public benchmark of cross-backend drift, though that is a step beyond the paper.
Extending the tolerance sweep beyond 1e-3 would likely push pass rates near 100%, but the practical threshold should depend on how sensitive the task is to output permutations, not just numerical magnitude.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Interesting approach, but the appendixed runner has a load-bearing bug. In run_once, model_ref = model.to(ref_device) then model_tgt = model.to(tgt_device). nn.Module.to() mutates in place, so both names reference the same module, now on the target device. For a CUDA target, the 'reference' run with CPU input throws, or if CPU target, you're comparing eager vs compiled on the same device — not cross-backend drift. So the 72% pass rate and tolerance sweeps in Tables II–III are not evidence for RQ1–RQ3. I had to check the PyTorch semantics twice, but they're unambiguous. The paper says the provided code can reproduce everything; it can't.

That's a shame, because the design thinking is solid. The YAML-driven config, three-tier verification (tensor, activation, task), and the deterministic adapter idea are reasonable contributions that could help standardize compatibility testing. The failure taxonomy — order-sensitivity in NMS, partial operator coverage — is plausible and matches what I've seen. The case study with pre-NMS sorting is a nice illustration, though it's anecdotal.

Other weaknesses are secondary but real: only two target backends, no full data release, "first unified framework" claim overstates the novelty. And the tolerance sensitivity is weak — 71.4% to 73.8% is hardly 'most discrepancies at stricter tolerances.' The paper is honest about the activation survey being deferred, and the threats-to-validity section is reasonable.

Who is this for? People building cross-backend CI pipelines will get useful design ideas from the YAML protocol, but they'll need to redo the empirical work. The paper deserves a serious referee because the framework is worth scrutiny and the bug needs to be caught — but as it stands, the central empirical claim is invalid. Without the reproducibility issue, I'd have said minor revision; with it, this is a reject-and-resubmit with a mandatory code audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a configuration-first framework for evaluating cross-backend behavioral drift in deep learning models. The framework decouples experiment specifications via YAML, uses a three-tier verification protocol (tensor closeness, activation alignment, task-level metrics), and reports an empirical study across 19 models, two target backends (GPU eager and torch.compile), and four tolerance settings, totaling 672 cross-backend checks. The central claim is a 72.0% aggregate pass rate, with most discrepancies occurring at tighter tolerances, and the paper additionally demonstrates deterministic adapters (e.g., pre-NMS sorting) that improve agreement. The paper also proposes a failure taxonomy and latency analysis, and provides a sanitized artifact for reproducibility.

Significance. If valid, the framework would be a useful contribution to the reproducibility and deep-learning-systems literature: it provides a practical tool for measuring and mitigating cross-backend drift, and the reported empirical landscape would inform deployment decisions. The paper also makes a concrete methodological proposal—linking tensor-level differences to task-level outcomes—that goes beyond simple allclose checks. However, the significance hinges entirely on the correctness of the empirical results and the artifact. The sanitized runner and the experimental accounting contain inconsistencies that make the reported numbers unexplainable. Because these problems invalidate the core empirical claims, the contribution as submitted cannot be accepted without a complete re-evaluation.

major comments (3)

[Appendix, run_once (lines 42-45)] The provided artifact runner constructs a single `model`, then executes `model_ref = model.to(ref_device)` followed by `model_tgt = model.to(tgt_device)`. Since `nn.Module.to()` mutates the module in-place and returns the same object, `model_ref` and `model_tgt` refer to the same module, both ultimately residing on the target device. Consequently, `model_ref(x_ref)` either raises a device-mismatch error (counted as FAIL by the `except` branch in `main`) or, if the target is CPU, compares identical executions. The reported 484 passes (Table III) and the pass rates in Table II cannot be produced by this runner. This invalidates the empirical basis for RQ1–RQ3 and, by extension, the central claim of the paper.
[Table II / §V-A] The claim that "most failures concentrate at 10^-6" is contradicted by Table II. At atol=1e-6 there are 120 failures out of 168, but the same 120 failures occur at atol=1e-5 and atol=1e-4; only four fewer failures occur at atol=1e-3. The pass rate changes from 71.4% to only 73.8%. The data therefore do not support the statement that "fine-grained numerical perturbations are the principal driver." The monotonic improvement is marginal and no per-model breakdown or error measurement is provided to substantiate the conclusion.
[Table III / §V-E] The experimental accounting is internally inconsistent. With 19 distinct models, 2 target backends, and 4 tolerance settings, the maximum number of cross-backend checks is 19 × 2 × 4 = 152, not 672. To obtain 168 checks per tolerance (as in Table II), one would need 21 model-input combinations per tolerance or additional dimensions not described in the setup. The paper does not specify multiple inputs or repeated runs, so the reported totals (168 per atol, 672 overall) are not derivable from the stated experimental configuration. This further undermines the aggregate pass-rate claim and the quantitative answers to RQ1–RQ3.

minor comments (5)

[§I-C] Typo: "Y AML" should be "YAML" in the bullet list.
[Fig. 7] The figure caption says "left shows ... right shows ..." but the figure itself is not described; please ensure the panels are labeled and the caption is self-contained.
[§V-B] The pass-rate heatmap is mentioned but no numerical per-model breakdown is given in the text. The claims about which models/tasks diverge would be easier to evaluate with a table of per-model pass rates or failure counts.
[§III-B] The Scope note explicitly defers a comprehensive activation survey, yet §V-C answers RQ3 ("where does divergence emerge?") based on selective probes. The answer should be clearly labeled as preliminary feasibility demonstration, not a definitive localization result.
[General] No error bars, confidence intervals, or repeated-run statistics are reported. Since the framework emphasizes reproducibility, reporting variance across seeds or runs would strengthen the empirical claims.

Circularity Check

1 steps flagged · score 8.0 of 10

Central 672-check result is undermined by in-place .to() aliasing in the reference runner: reference and target are the same module.

self definitional [Appendix, run_once (lines 42-45 and 58)]
"ref_device = torch.device("cpu") model_ref = model.to(ref_device) tgt_device = torch.device(device) model_tgt = model.to(tgt_device) ... y_ref = model_ref(x_ref); y_tgt = model_tgt(x_tgt)"

In PyTorch, nn.Module.to() moves parameters in place and returns self. So after model.to(cpu) and then model.to(cuda), model_ref and model_tgt are the same object, and the original weights now live on the target device. The intended 'reference' CPU computation is not performed on a CPU copy; it either raises a device-mismatch error (CPU input vs CUDA model) or, if device=cpu, executes the same module. Therefore each pass/fail in the 672 checks is produced by this aliasing/error, not by comparing two independent backend executions. The reported 72.0% pass rate is thus determined by construction, not by cross-backend drift.

full rationale

Reviewed all load-bearing steps. There are no fitted parameters renamed as predictions, no self-citation chains, and no imported uniqueness theorem; the deterministic adapter is a post-hoc mitigation, not a circular prediction. The paper honestly scopes RQ3 activation probing (Section III-B) and lists environmental threats (Section VII). However, the sanitized runner in the Appendix—which the paper presents as the artifact that regenerates all tables and figures—builds ref and target from the same model object using in-place .to(). This makes the reference and target computations the same execution (or an immediate device-mismatch error); hence the central empirical result (672 checks, 72.0% pass) is a by-construction artifact rather than a measurement of cross-backend drift. Because the central claim rests on this result, the circularity/self-definition score is high, even though the framework design itself is not circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entities. Its conclusions rest on chosen tolerances, the representativeness of torch.compile, and the sufficiency of the determinism settings.

free parameters (2)

atol tolerance values = [1e-6, 1e-5, 1e-4, 1e-3]
Hand-chosen thresholds that define pass/fail in the tensor closeness check; they are swept as an independent variable, not fitted.
rtol = 1e-5
Fixed relative tolerance in the compatibility criterion (Eq. 1), chosen by the authors.

assumptions (4)

domain assumption torch.compile is representative of compiled backends
The study uses torch.compile as the sole compiled backend and generalizes to 'compiled backends' throughout.
domain assumption Default/pretrained weights and standard preprocessing are used
The paper does not detail which weights are loaded for each model, only that models are from torchvision or repository clones.
standard math torch.testing.assert_close correctly implements the compatibility criterion
The runner uses assert_close to decide pass/fail (appendix utilities).
domain assumption Deterministic flags (seed, cudnn.deterministic, benchmark=False) sufficiently control nondeterminism
The runner calls set_global_seed and disables cudnn benchmark, but residual nondeterminism (e.g., from autotuning) is acknowledged in Section VII.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Toward Reproducible Cross-Backend Compatibility for Deep Learning: A Configuration-First Framework with Three-Tier Verification." pith.science (2026). https://pith.science/paper/HZ4UHYW5

@misc{pith2026250906977,
  author       = {Pith},
  title        = {Pith review of: Toward Reproducible Cross-Backend Compatibility for Deep Learning: A Configuration-First Framework with Three-Tier Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZ4UHYW5}},
  note         = {Machine review of arXiv:2509.06977}
}

read the original abstract

This paper presents a configuration-first framework for evaluating cross-backend compatibility in deep learning systems deployed on CPU, GPU, and compiled runtimes. The framework decouples experiments from code using YAML, supports both library and repository models, and employs a three-tier verification protocol covering tensor-level closeness, activation alignment, and task-level metrics. Through 672 checks across multiple models and tolerance settings, we observe that 72.0% of runs pass, with most discrepancies occurring under stricter thresholds. Our results show that detection models and compiled backends are particularly prone to drift, often due to nondeterministic post-processing. We further demonstrate that deterministic adapters and selective fallbacks can substantially improve agreement without significant performance loss. To our knowledge, this is the first unified framework that systematically quantifies and mitigates cross-backend drift in deep learning, providing a reproducible methodology for dependable deployment across heterogeneous runtimes.

Figures

Figures reproduced from arXiv: 2509.06977 by the authors.

**Figure 1.** Pipeline: YAML → Loader → Preprocess → Exec (ref,tgt) → Verify → Reports. We also provide an at-a-glance summary of our evaluation axes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Overview of RQ1–RQ3: tolerance sweep (RQ1), [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗

**Figure 3.** Pass rate across atol values [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Failure taxonomy distribution. Finding. Detection models show lower agreement on compiled backends; the taxonomy indicates that ordering/tie-breaking in post-processing and partial operator support dominate failures. Segmentation tasks are comparatively stable, likely …

**Figure 6.** Figure 6: Median inference latency by backend. Finding. Compiled backends reduce median latency for several architectures, illustrating an accuracy–performance trade-off [PITH_FULL_IMAGE:figures/full_fig_p003_6.png]

**Figure 7.** Figure 7: Qualitative detection comparison on the same image: [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

[1]

DeepXplore: Automated Whitebox Testing of Deep Learning Systems,

K. Pei, Y . Cao, J. Yang, and S. Jana, “DeepXplore: Automated Whitebox Testing of Deep Learning Systems,” in Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP) , 2017

work page 2017
[2]

DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,

Y . Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,” in Proceedings of the 40th International Conference on Software Engineering (ICSE) , 2018

work page 2018
[3]

TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

N. Odena, C. Olsson, D. Andersen, and I. Goodfellow, “Tensor- Fuzz: Debugging Neural Networks with Coverage-Guided Fuzzing,” arXiv:1807.10875, 2018

work page Pith review arXiv 2018
[4]

Mist: Automated Neural Network Model Testing via Multiple Interfaces,

J. Zhang, X. Zhang, Y . Wei, et al. , “Mist: Automated Neural Network Model Testing via Multiple Interfaces,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2020

work page 2020
[5]

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning,

T. Chen, T. Moreau, Z. Jiang, et al. , “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018. (See also: OpenXLA/XLA; Glow; PyTorch 2.0 Inductor.)

work page 2018
[6]

Improving Repro- ducibility in Machine Learning Research: A Report from the NeurIPS 2019 Reproducibility Program,

J. Pineau, P. Vincent-Lamarre, K. Sinha, et al. , “Improving Repro- ducibility in Machine Learning Research: A Report from the NeurIPS 2019 Reproducibility Program,” Journal of Machine Learning Research , 22(164):1–20, 2021. APPENDIX 1 # Library example (classification) 2 from: library 3 model: resnet18 4 inputs: [assets/cat.jpg] 5 means: [0.485, 0.456, 0...

work page 2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

DeepXplore: Automated Whitebox Testing of Deep Learning Systems,

K. Pei, Y . Cao, J. Yang, and S. Jana, “DeepXplore: Automated Whitebox Testing of Deep Learning Systems,” in Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP) , 2017

work page 2017

[2] [2]

DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,

Y . Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,” in Proceedings of the 40th International Conference on Software Engineering (ICSE) , 2018

work page 2018

[3] [3]

TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

N. Odena, C. Olsson, D. Andersen, and I. Goodfellow, “Tensor- Fuzz: Debugging Neural Networks with Coverage-Guided Fuzzing,” arXiv:1807.10875, 2018

work page Pith review arXiv 2018

[4] [4]

Mist: Automated Neural Network Model Testing via Multiple Interfaces,

J. Zhang, X. Zhang, Y . Wei, et al. , “Mist: Automated Neural Network Model Testing via Multiple Interfaces,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2020

work page 2020

[5] [5]

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning,

T. Chen, T. Moreau, Z. Jiang, et al. , “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning,” in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018. (See also: OpenXLA/XLA; Glow; PyTorch 2.0 Inductor.)

work page 2018

[6] [6]

Improving Repro- ducibility in Machine Learning Research: A Report from the NeurIPS 2019 Reproducibility Program,

J. Pineau, P. Vincent-Lamarre, K. Sinha, et al. , “Improving Repro- ducibility in Machine Learning Research: A Report from the NeurIPS 2019 Reproducibility Program,” Journal of Machine Learning Research , 22(164):1–20, 2021. APPENDIX 1 # Library example (classification) 2 from: library 3 model: resnet18 4 inputs: [assets/cat.jpg] 5 means: [0.485, 0.456, 0...

work page 2019