An End-to-End Differentiable Forward Model for High-Energy Diffraction Microscopy

Hemant Sharma; Mathew Cherukara; Nina Andrejevic; Simon Zhang

An End-to-End Differentiable Forward Model for High-Energy Diffraction Microscopy

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that high-energy diffraction microscopy can be made fully differentiable end to end—across far-field, near-field, and point-focused geometries—without sacrificing pixel-level agreement with the established C reference simu

desk verdict Genuinely useful first differentiable HEDM forward model covering FF/NF/pf, with solid core validation, but the NF pixel-exact claim needs the splatting parameters reported and the abstract's 0.3% residual line corrected. read the letter →

arxiv 2607.28843 v1 pith:NWHBP2EW submitted 2026-07-30 cond-mat.mtrl-sci

Hemant Sharma , Nina Andrejevic , Simon Zhang , Mathew Cherukara This is my paper · ORCID

classification cond-mat.mtrl-sci

keywords high-energydiffractionmicroscopydifferentiableforwardmodelautomaticdifferentiationgrainorientationrefinementdetectorgeometrycalibrationnear-fieldfar-fieldpoint-focusedHEDM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to close a structural gap in high-energy diffraction microscopy (HEDM): the forward models that map grain state and instrument geometry to detector images are fast but not differentiable, so gradient-based joint refinement, physics-informed regularization, and Bayesian uncertainty quantification are unavailable. It presents a differentiable forward model built from standard tensor operations, covering all three HEDM geometries, and validates it against the production C simulators to pixel-level agreement. With exact gradients available, it demonstrates joint recovery of orientation, strain, and position in near-field geometry to about six nanometers, and joint refinement of all per-detector geometry parameters plus per-grain state on a four-panel far-field setup. If the claims hold, HEDM reconstruction moves from sequential, coordinate-descent-style fitting to single gradient flows over the entire experimental state.

What carries the argument

The load-bearing object is the omega quadratic: the rotating-crystal diffraction condition reduced to a quadratic equation in cos(omega), solved analytically, with branch selection handled by masked tensor operations rather than branching. Around it, four substitutions preserve differentiability: masked selection between two fully evaluated paths in place of conditional branching, clamped inverse trigonometric functions, Newton–Schulz projection onto SO(3) instead of SVD, and binary validity masks multiplied into losses. Near-field images are produced by Gaussian splatting with a straight-through pixel estimator so gradients flow through sub-pixel spot positions.

What would settle it

Inspect the test configuration for the near-field comparison and confirm that the splatting sigma and radius are set from an independent physical or numerical criterion; then re-run the 2304-pixel comparison after varying sigma across a small range. If the exact match holds only at the chosen sigma, the pixel-exact claim is parameter-dependent rather than a robust property of the physics port.

Watch

Extended reading notes

Core claim

The central discovery is that every step of the HEDM forward simulation—rotation-matrix construction, reciprocal-lattice strain, the quadratic rotating-crystal diffraction condition, azimuthal angle, and detector projection—can be re-expressed as differentiable tensor primitives without changing the physics. The resulting model reproduces the validated C simulators to within floating-point precision on the vast majority of reflections (162/162 far-field, 2304/2304 near-field pixels, 1088/1096 point-focused), with the only mismatches at a known branch-selection boundary near the azimuthal axis. Because every input is a leaf of the autograd graph, scalar losses on predicted spot positions prop

Load-bearing premise

The near-field 'pixel-exact' agreement depends on the Gaussian-splatting output head reproducing exactly the same pixel values as the C reference, but the paper does not state the splatting sigma and radius used; if those were tuned to force the match, that agreement is not independent evidence.

Editorial extensions

If this is right

HEDM inverse problems can be solved by joint gradient descent over orientation, strain, position, and detector geometry in one flow, replacing sequential fits.
The same gradients enable physics-informed regularisation and Bayesian uncertainty quantification via Hamiltonian Monte Carlo or variational inference.
Detector auto-calibration becomes possible in near-field geometry, where no powder ring exists, recovering tilt angles to sub-microradian precision.
The basin of convergence for orientation refinement is about 10–15 degrees, matching typical production indexer accuracy, so existing indexers can feed this refiner.
The model scales to thousands of grains and can be coupled to differentiable finite-element solvers, closing the experiment–simulation loop for crystal plasticity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The pixel-exact near-field claim may be sensitive to the Gaussian-splatting bandwidth, which the paper does not specify; a fair test would measure whether the exact match survives when the splatting sigma is fixed by an independent point-spread calibration.
The eight unmatched point-focused spots sit at a genuine physical degeneracy; this suggests that a continuous branch mixture rather than a hard choice would make gradients stable across that boundary.
The demonstrated 10–15 degree basin could potentially be widened by soft spot-matching losses, a direction the paper mentions but does not explore.
If these results replicate, HEDM becomes a testbed for differentiable experimental design: because detector geometry is differentiable, one could optimise detector placement or the omega sweep to maximise information gain before the experiment runs.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is the first end-to-end differentiable HEDM forward model covering FF, NF, and pf geometries, and it ships with code, validation scripts, and a real-data demonstration. The core is sound; I'd send it to peer review. But two things need fixing before I'd trust the headline claims: the NF pixel-exact match depends on Gaussian-splatting parameters that aren't reported, and the abstract's "0.3% residual" line doesn't match the body's numbers.

What's new: the PyTorch implementation makes every input (orientation, position, strain, detector geometry) a leaf in the autograd graph, which unlocks gradient-based joint refinement, Bayesian UQ, and physics-informed regularisation. The validation is extensive: FF 162/162 spots with sub-10^-3 px residuals, NF 2304/2304 pixels matched exactly under zero and non-zero tilts, pf 1088/1096 spots, finite-difference gradient checks at 10^-8, a 250-grain noisy cross-code test, and a perturb-and-recover on a real 214-grain alpha-Ti dataset. The multi-panel joint geometry-and-grain refinement is a nice concrete demonstration. They also release the code and reproducibility scripts, which is real.

Soft spots, in proportion. First, the NF pixel-exact claim: section 4.2.5 says Gaussian splatting uses "configurable sigma and radius" but never reports the values, and the C reference's rendering is described only as "ray-traces" in S1.2. If those parameters were chosen to force the match, the agreement is circular. This doesn't sink the paper—the FF and pf agreements don't involve splatting—but it needs a clear statement of the chosen parameters and a justification that they're not tuned. Second, the abstract says residuals match the production fit to 0.3%; the body reports median omega tied and eta/position tighter by 14.7% and 39%. That's an overstatement. Third, the validation reference is the group's own MIDAS C code; the agreement is partly self-consistency. Given the C code's production history that's reasonable, but the paper should say so more explicitly. Minor: the pf match is 99.27%, so "pixel-exact" should be qualified.

Bottom line: the central claim—that a differentiable HEDM forward model can be built without sacrificing agreement with the reference physics—holds up. This is a paper for the synchrotron materials-science audience and for method developers; it deserves a serious referee, with the splatting documentation and abstract fix as required revisions.

Referee Report

3 major / 5 minor

Summary. This paper presents HEDMForwardModel, a PyTorch implementation of an end-to-end differentiable forward model for far-field (FF), near-field (NF), and point-focused (pf) HEDM. The model is validated against the MIDAS C simulators (ForwardSimulationCompressed and simulateNF), reporting pixel-exact agreement for FF and NF and 99.27% for pf, with analytic gradients verified against finite differences. Three optimization demonstrations are given: joint NF orientation–strain–position recovery, a real-data round-trip on a 214-grain α-Ti dataset, and joint multi-panel FF geometry–grain refinement. The software is released as the open-source midas-diffract package.

Significance. If the validation holds, this is a significant methodological contribution: it opens gradient-based joint refinement, regularized reconstruction, and Bayesian inference for HEDM, with a unified framework across all three HEDM geometries. The paper includes extensive cross-code comparisons, gradient-correctness checks, basin-of-convergence sweeps, and a reproducible-script pipeline. However, the NF pixel-exact agreement rests on undocumented Gaussian-splatting parameters, and the validation references are the authors' own C simulators; these limit the independent evidentiary value of the headline claims.

major comments (3)

[§4.2.5, §5.3, §S1.2] The NF 'predict_images' output head uses a Gaussian splatting kernel with 'configurable sigma and radius' (§4.2.5), but the values used for the 2304/2304 pixel-exact comparison (§5.3) are never reported. Supplementary §S1.2 describes the C simulateNF reference only as 'ray-traces the diffracted beam through the detector plane', without specifying the pixel-level point-spread function or discretization rule. If the sigma/radius were chosen to match the C output, the agreement is circular and not an independent validation. Please specify the exact splatting parameters, how they were set, and document the C rendering kernel. Without this, the NF agreement is not reproducible and the strongest quantitative evidence for the pixel-exact claim collapses.
[Abstract; §6.4, Test 1] The abstract's claim that residuals match the production fit 'to 0.3%' is not supported by the body. Section 6.4 reports median ω residuals 'essentially tied' (0.0901° vs 0.0904°, a 0.33% difference), while η residuals are 14.7% tighter and position residuals 39% tighter. The 0.3% figure appears to refer only to the ω residual, not to an overall residual match, and it conflates the perturb-and-recover experiment (100% recovery) with the forward-model agreement test. Similarly, the abstract's '~6 nm precision' for NF-HEDM (Section 6.3, Table 2) is the noise-free result; the noisy case gives 66 nm. Please revise the abstract and contribution 3 to state precisely what is measured and under what noise conditions.
[§3, §5.1, §9] The validation references are the MIDAS C simulators (ForwardSimulationCompressed and simulateNF), which are authored by the same group and, as stated in §9, are documented in the peer-reviewed literature for the first time in this paper. The PyTorch model is a direct port of those codes. The pixel-exact agreement therefore primarily demonstrates that the port is faithful, not that the physics is independently correct. The manuscript should qualify the phrase 'established reference simulators' and clarify that this is a self-consistency check. An independent cross-check against another HEDM code (e.g., ImageD11 or HEXRD spot positions) would materially strengthen the external validity of the central claim.

minor comments (5)

[Abstract; §6.3, Table 2] '~6 nm precision' should be explicitly labeled as the noise-free result; the noisy result is 66 nm. As written, the abstract implies robustness that the body does not support.
[§4.2.5] The straight-through estimator for pixel rounding is mentioned but not described. Please provide the exact gradient rule or a reference, since it is essential for the NF image-space gradients.
[§5.3] The tilt sweep is reported only as 'up to 1° per axis'. Please state the number of tilt configurations, the step size, and the exact comparison criterion (e.g., max pixel difference or exact index match) so the 2322/2322 result can be reproduced.
[§S1.2] The description of simulateNF should include the pixel intensity distribution rule (e.g., bilinear interpolation, as in ForwardSimulationCompressed) and how the detector plane is discretized, to make the NF comparison fully reproducible.
[§6.4, Test 1] The text says 'below the diagonal on 179/214 grains for η and position' and '120/214 for ω'. Please ensure the counts and percentages are internally consistent and clearly tied to the specific residual definitions.

Circularity Check

0 steps flagged · score 2.0 of 10

No demonstrated circularity; NF splatting parameter gap is a reproducibility risk, not a fitted-input reduction.

full rationale

The paper's central quantitative claims are translation checks. The differentiable model is explicitly a line-by-line port: §4.2.2 says 'Both channels are direct ports of CorrectHKLsLatC and CorrectHKLsLatCEpsilon ... faithful correspondence was verified against the C code line by line', and §4.2.3 says 'The implementation ports the quadratic solver from NF HEDM/src/CalcDiffractionSpots.c:87--183'. The Section 5 agreements therefore verify that the PyTorch translation preserves the C arithmetic; agreement is expected if the port is faithful, not a new prediction derived from the agreement target. This is self-consistency, not circularity. The reference simulators are cited to same-group papers (Sharma et al. 2026a/b) and are 'documented here in the peer-reviewed literature for the first time' (Conclusion), so the 'established' label is not fully independent. However, no load-bearing step reduces to an unverified self-citation: the C code is an external artifact with described inputs and physics, and the paper provides reproducibility scripts. The one genuinely suspicious passage is §4.2.5: NF output 'rasterises each valid spot onto a three-dimensional ... detector volume via a Gaussian splatting kernel, with configurable sigma and radius', while §5.3 reports '2304/2304 pixels identical' against simulateNF, whose rendering is only described as 'ray-traces the diffracted beam through the detector plane' (S1.2). No sigma/radius values or C discretization rule are given. If those parameters had been tuned to match simulateNF, the NF match would be circular; but the paper gives no evidence of such tuning, and the comparison script is provided. This is a reproducibility/documentation gap to be weighed as correctness risk, not a demonstrated circular reduction. The inverse-problem demonstrations use the model's own forward output as synthetic observations (§6.3, §6.5), but they are explicitly labeled synthetic and validate optimization, not independent prediction. No fitted parameter is renamed as a prediction. I find no circular step that meets the evidence bar; minor same-group reference dependence justifies a non-zero-but-low score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The model introduces no new physical entities; the scoring depends on standard X-ray diffraction assumptions and a set of numerical parameters (splatting, thresholds, optimizer schedule) that are hand-chosen and, in the NF case, incompletely reported.

free parameters (3)

Gaussian splat sigma and radius (NF predict_images) = not specified
Configurable kernel parameters in §4.2.5; the equivalence to the C reference's pixel rendering in §5.3 is asserted without reporting the values used, so the pixel-exact NF match may depend on this choice.
Adam warm-up per-parameter learning rates (multi-panel demo) = tilts 5e-2, Lsd 5e1, y/zBC 0.1, Eulers 1e-4, positions 1.0, strains 1e-5
Hand-tuned in §6.5 to equalize gradient scales; needed for convergence in the four-panel joint refinement.
epsilon clamp / almostzero thresholds = eps=1e-7, almostzero=1e-12
Numerical substitutions in §4.3 that trade O(eps) bias for finite gradients; chosen by hand but consistent with C conventions.

assumptions (5)

standard math The rotating-crystal diffraction condition -Gx cos(omega) + Gy sin(omega) = |G| sin(theta) with up to two solutions (Eq. 2).
Standard Laue/rotating-crystal condition from crystallography; used in all forward models.
domain assumption The MIDAS fifteen-parameter detector geometry model is the correct description of detector placement.
Adopted from self-cited Sharma et al. 2026a; the paper validates against C code but not against independent detector calibration for FF agreement.
domain assumption Point-source intensity model with CIF relative intensities is sufficient for centroid-based spot predictions.
Acknowledged limitation in §8.4; does not represent partial illumination or extended peak shapes.
ad hoc to paper The Gaussian splatting renderer reproduces the C simulateNF pixel output exactly.
The pixel-exact NF claim (§5.3) relies on this; parameters not reported.
ad hoc to paper The validity-mask loss surface is piecewise smooth and the optimizers do not traverse mask transitions in the tested schedules.
Asserted empirically in §4.3; not proven globally.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An End-to-End Differentiable Forward Model for High-Energy Diffraction Microscopy." pith.science (2026). https://pith.science/paper/NWHBP2EW

@misc{pith2026260728843,
  author       = {Pith},
  title        = {Pith review of: An End-to-End Differentiable Forward Model for High-Energy Diffraction Microscopy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWHBP2EW}},
  note         = {Machine review of arXiv:2607.28843}
}

read the original abstract

High-Energy Diffraction Microscopy (HEDM) recovers crystallographic orientation, strain, and grain position from rotating-crystal X-ray diffraction patterns. Existing forward models in far-field (FF), near-field (NF), and point-focused (pf) HEDM are not differentiable, which forecloses gradient-based joint parameter refinement, physics-informed regularisation, and Bayesian uncertainty quantification. We present the first end-to-end differentiable HEDM forward model covering all three geometries, implemented in PyTorch with pixel-exact agreement against the established MIDAS reference simulators (162/162 FF, 2304/2304 NF including a non-zero detector-tilt sweep, and 1088/1096 pf-HEDM spots matched). Three demonstrations validate the framework: joint orientation-strain-position recovery in NF-HEDM at ~6 nm precision; round-trip refinement on a real 214-grain alpha-Ti FF-HEDM dataset reaching 100% grain recovery from a 1.5 degree initial perturbation with residuals matching the production fit to 0.3%; and joint refinement of all per-detector geometry parameters and per-grain state on a synthetic four-panel FF-HEDM setup, recovering panel rotations about the beam axis to ~10 mu-rad and a global rotation-axis wedge to ~26 mu-rad. The framework is released as the open-source midas-diffract package (pip install midas-diffract).