REVIEW 4 major objections 6 minor 33 references
Training Hybrid Neural Networks with Multimode Optical Nonlinearities Using Digital Twins
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A continuously updated neural twin makes an opaque nonlinear multimode fiber trainable by backpropagation, lifting a 1500-sample Fashion-MNIST task to 80% accuracy and resisting drift.
desk verdict Online surrogate refinement is a useful twist on the digital-twin idea, but the paper never measures the Jacobian alignment that its training scheme depends on, and the accuracy gains are single-run small numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Optical Layer Twin (OLT), a convolutional U-Net that maps the two-dimensional phase pattern written on the spatial light modulator to the two-dimensional intensity pattern measured on the camera. It carries the argument because the backward pass replaces the optical system's unknown Jacobian with the twin's Jacobian, the approximation being $J_{\mathrm{OLT}} \approx J_{\mathrm{OS}}$. Gradients are computed as vector–Jacobian products rather than full Jacobian matrices, since a full matrix for a batch of ten $128\times128$ images would contain roughly $1.7\times10^{10}$ elements and exceed 100 GB of memory. The second essential mechanism is online refinement: at each training step the twin is updated on the experimental input–output pairs collected in the forward pass, so it tracks both the evolving input distribution caused by preprocessor weight updates and slow physical drift, which is what keeps the gradient approximation valid.
What would settle it
On the setup of Fig. 2a, perturb each phase pixel on the spatial light modulator one at a time during online learning, record the camera intensity change, and assemble the resulting sensitivity matrix of the physical system; compare it with the twin's sensitivity matrix on the same inputs. If the two diverge under the input distribution shifts or fiber perturbations where online learning is claimed to help, then the gradient updates to the preprocessor are not the true gradients and the reported accuracy and drift-resilience results would not follow.
Extended reading notes
Core claim
The central claim is that a data-driven neural network can approximate both the forward map and the Jacobian of a nonlinear multimode-fiber optical system closely enough that backpropagating through this twin trains the layers on either side of the physical layer, and that continuously refining the twin during training is what keeps it accurate as the input distribution shifts and the hardware drifts. The authors demonstrate this on a three-layer hybrid network whose middle layer is the fiber: a single convolutional preprocessor, the optical system, and a fully connected classifier. Online learning reaches 80% test accuracy on a 1500-sample Fashion-MNIST subset, compared with 77% for an offline, fixed twin and 75% with no preprocessing, while the twin's output predictions reach a normalized mean absolute error of $1.03\times10^{-2}$ (SNR 96.8). Under deliberately accelerated mechanical drift of the fiber, online learning maintains its advantage, improving final classification accuracy by up to 39% over a fixed twin. The paper also reports that the trained twin predicts optical outputs in about 30 ms on a consumer GPU, whereas a truncated 15-mode numerical simulation of the fiber takes roughly 500 s.
Load-bearing premise
The claim stands on the twin's gradients staying close to the real optical system's gradients throughout training, even as the preprocessor changes what the fiber sees and as the fiber is mechanically perturbed; the paper shows the twin's output images match well, but it does not measure the gradient match directly.
Editorial extensions
If this is right
- Adding a differentiable digital twin of the physical layer lets the standard error-backpropagation algorithm train layers that precede and follow an optical system with no analytic gradient; the demonstrated network trains a convolutional preprocessor before the fiber and a classifier after it.
- Continuously updating the twin with live experimental data (online learning) is what maintains fidelity as the preprocessor changes what the fiber sees; this is the difference between 80% test accuracy and 77% with a fixed twin.
- The same online updating confers resilience to slow physical drift: under mechanically induced fiber perturbation, online learning improves final accuracy by up to 39% relative to a fixed twin.
- The neural twin is fast enough to make training practical: 30 ms per prediction on a consumer GPU versus roughly 500 s for a truncated 15-mode numerical simulation of the same fiber.
- Because the optical layer contributes large nonlinear transformations without any digitally trainable weights, the trained network can run inference with only the small preprocessor and classifier computed digitally; the physical layer itself consumes no digital operations during inference.
Reading between the lines
- If the Jacobian approximation holds beyond this setup, the same twin scheme could in principle stack several physical layers in width and depth, or be applied to other non-differentiable physical systems such as scattering media, waveguides, or mechanical reservoirs; the paper hints at this but does not demonstrate it.
- The online refinement loop is essentially an adaptive model of a drifting plant; an untested extension is whether it can also compensate drift during deployed inference, not just during training, by continuing to collect pairs of inputs and outputs.
- The twin itself is a 70M-parameter U-Net trained on live data, so the total training compute is not obviously reduced; whether the approach saves energy overall depends on how often and how long the twin must be updated, which the paper does not quantify.
- The accuracy comparisons are on a 1500-sample Fashion-MNIST subset; whether the online advantage persists on full-size datasets or more complex tasks is an open empirical question.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes training hybrid neural networks that include a physical nonlinear multimode-fiber layer by replacing the optical system with a learned differentiable surrogate, called the Optical Layer Twin (OLT), during backpropagation. The OLT is a U-Net trained on experimental input-output pairs and is either kept fixed (offline learning) or continuously refined with fresh experimental data during training (online learning). On a 1500-sample Fashion-MNIST subset, the authors report 80% test accuracy with online learning, 77% with offline learning, and 75% without a preprocessing layer; they also report OLT fidelity of normalized MAE 1.03e-2 (SNR 96.8). In a separate set of experiments, a mechanical actuator introduces controlled fiber perturbations, and the online learning scheme is reported to improve final accuracy by up to 39% relative to offline learning. The paper also studies how OLT architecture size affects fidelity and computational cost.
Significance. If the central claims hold, the work is a valuable step toward training physical neural networks with complex nonlinear analog layers: the online-refinement idea addresses input-distribution shift and system drift, and the reported OLT inference latency (30 ms) versus an analytical nonlinear-Schrödinger-equation simulation (~500 s for 15 of 240 modes) is a concrete practical advantage. The experimental demonstration is nontrivial, and the OLT output fidelity is high. However, the paper's central mechanism, that the OLT's Jacobian faithfully approximates the optical system's Jacobian, is asserted rather than demonstrated, and the headline accuracy differences are small and appear to come from single runs without error bars. These issues are load-bearing for the main claim and require additional evidence or experiments.
major comments (4)
- [Appendix Note 2] The training algorithm depends on the approximation J_OLT ≈ J_OS, but the manuscript only validates the OLT in output space: normalized MAE 1.03e-2 and visually similar speckle patterns in Figs. 2c and 2d. For a high-dimensional nonlinear multimode mapping, small output error does not imply small Jacobian error, and the refinement loss L_refine = |y_OS - y_OLT|^2 also constrains only outputs. As a result, the explanation that the preprocessor receives faithful physical gradients is not directly supported; the 80% versus 77% versus 75% differences could in principle arise from the added preprocessing capacity or from a regularizing effect. I recommend adding a direct gradient-alignment check, for example comparing u^T J_OLT v with finite-difference or experimentally measured directional derivatives, or an ablation using a deliberately wrong or frozen OLT during backward passes.
- [Fig. 2b and Fig. 3c] The headline results rest on small margins (80% vs 77% vs 75% test accuracy) and on single training runs. No error bars, repeated seeds, or statistical significance tests are reported anywhere in the manuscript, and Fig. 3c reports the perturbation-rate experiments without indicating the number of independent runs. Given that a 2–3 percentage point gap can arise from initialization or experimental variability, please provide repeated trials with means and standard deviations, and state the number of independent experimental repetitions for the drift experiments.
- [Appendix Note 2, last paragraph] The experimental and training protocols are underspecified for reproducibility. The manuscript does not state how many experimental input-output pairs were used to pretrain the offline OLT, how many refinement steps are taken per batch or per epoch in online learning, the OLT's learning rate and optimizer, or the exact train/test split. The description of the preprocessing block as '6 linear convolutional layers of 1 kernel with 6 × 6 parameters' is ambiguous: it could mean six separate convolution kernels of size 6×6, or six sequential layers each containing one 6×6 kernel. Please clarify these details, since they directly affect the claimed parameter efficiency and the interpretation of the results.
- [Abstract and Fig. 2] The claim of 'state-of-the-art image classification accuracies' is not supported by the presented baselines. An accuracy of 80% on a 1500-sample Fashion-MNIST subset is not state-of-the-art in a general sense, and the only comparisons are the offline and no-preprocessor variants of the same hybrid system. Please include an equivalent fully digital network trained on the same data and, if possible, a random-features baseline, so that the benefit of the physical layer and of the OLT gradient mechanism can be isolated from the effect of simply adding trainable capacity.
minor comments (6)
- [Fig. 2c] Define 'normalized MAE' explicitly and state how SNR is computed from it; as written, the reader cannot verify the reported SNR of 96.8.
- [Guiding Model Training with the Optical Layer Twin] The sentence 'the slight increase in error stems directly from slow drifts in the experimental system over time' is confusing for the no-preprocessor condition, since no drift is intentionally induced in Fig. 2; clarify what drift means in that context.
- [Appendix Note 1] The definition of the nonlinear mode coupling tensor η_p,l,m,n contains a denominator that appears typeset incorrectly (a product of four separate integrals with unclear brackets); please correct the equation.
- [Methods and Results] The manuscript does not include a data or code availability statement. Given the complexity of the experimental setup and the OLT architecture, a reproducibility statement is necessary.
- [References] Reference [13] (Wright et al., Nature 2022) is closely related and should be discussed more explicitly in the introduction to clarify the novelty of the present approach relative to that work.
- [Fig. 3b] Specify whether the MAE values in Fig. 3b are averaged over all test examples or computed for a single representative input, and state the units.
Circularity Check
No circular derivation: the OLT fidelity claim is validated against held-out experimental outputs, and the Jacobian substitution is an explicit modeling assumption; only a minor, non-load-bearing self-citation appears.
full rationale
The central claim is that the Optical Layer Twin (OLT) approximates the experimental multimode-fiber response accurately enough to backpropagate through it. This claim is not circular: the OLT is trained on measured input-output pairs and its fidelity is evaluated on held-out experimental outputs (normalized MAE 1.03e-2, SNR 96.8, Figs. 2c-d). The approximation J_OLT ≈ J_OS stated in Appendix Note 2 is an explicit algorithmic substitution, not a derived identity that presupposes the reported accuracies. The reported classification results are benchmarked on external Fashion-MNIST data, and the online vs offline comparison is an empirical ablation. The only self-citation, ref. [18], is used to justify the optical power level and to reuse Appendix Figure 1; this is a parameter-setting reference to the authors' prior experimental work and does not carry the paper's central argument. The absence of a direct Jacobian-alignment measurement is a robustness/correctness concern, not circularity.
Assumptions & free parameters
free parameters (4)
- OLT architecture hyperparameters (depth, number of filters, kernel size) =
4 down/up blocks; 16 filters; kernels up to 16x16
- Optical power coupled into the multimode fiber =
12.6 mW average, ~10 kW peak
- Actuator perturbation step =
0.12 degrees per epoch
- Digital-layer training hyperparameters =
SGD learning rate 1e-3; 6 conv layers with 1 kernel of 6x6; sigmoid; softmax
assumptions (3)
- domain assumption The Jacobian of the OLT approximates the Jacobian of the physical optical system during training.
- domain assumption The multimode fiber's experimentally measured input-output map is a deterministic, learnable function that a U-Net can approximate with sufficient accuracy.
- domain assumption Experimental drift is slow and smooth enough to be traced by online updates of the OLT at the per-epoch timescale.
Cite this review
Pith. "Pith review of Training Hybrid Neural Networks with Multimode Optical Nonlinearities Using Digital Twins." pith.science (2026). https://pith.science/paper/DBEXHUUT
@misc{pith2026250107991,
author = {Pith},
title = {Pith review of: Training Hybrid Neural Networks with Multimode Optical Nonlinearities Using Digital Twins},
year = {2026},
howpublished = {\url{https://pith.science/paper/DBEXHUUT}},
note = {Machine review of arXiv:2501.07991}
}
read the original abstract
The ability to train ever-larger neural networks brings artificial intelligence to the forefront of scientific and technical discoveries. However, their exponentially increasing size creates a proportionally greater demand for energy and computational hardware. Incorporating complex physical events in networks as fixed, efficient computation modules can address this demand by decreasing the complexity of trainable layers. Here, we utilize ultrashort pulse propagation in multimode fibers, which perform large-scale nonlinear transformations, for this purpose. Training the hybrid architecture is achieved through a neural model that differentiably approximates the optical system. The training algorithm updates the neural simulator and backpropagates the error signal over this proxy to optimize layers preceding the optical one. Our experimental results achieve state-of-the-art image classification accuracies and simulation fidelity. Moreover, the framework demonstrates exceptional resilience to experimental drifts. By integrating low-energy physical systems into neural networks, this approach enables scalable, energy-efficient AI models with significantly reduced computational demands.
Reference graph
Works this paper leans on
-
[1]
Henighan et al., ‘Scaling Laws for Autoregressive Generative Modeling’, Nov
T. Henighan et al., ‘Scaling Laws for Autoregressive Generative Modeling’, Nov. 05, 2020, arXiv: arXiv:2010.14701. doi: 10.48550/arXiv.2010.14701
-
[2]
‘Parameters in notable artificial intelligence systems’, Our World in Data. Accessed: Dec. 09, 2024. [Online]. Available: https://ourworldindata.org/grapher/artificial-intelligence- parameter-count
work page 2024
-
[3]
Zhu et al., ‘Intelligent Computing: The Latest Advances, Challenges, and Future’, Intell
S. Zhu et al., ‘Intelligent Computing: The Latest Advances, Challenges, and Future’, Intell. Comput., vol. 2, p. 0006, Jan. 2023, doi: 10.34133/icomputing.0006
-
[4]
A. S. Luccioni, Y. Jernite, and E. Strubell, ‘Power Hungry Processing: Watts Driving the Cost of AI Deployment?’, in The 2024 ACM Conference on Fairness, Accountability, and Transparency, Jun. 2024, pp. 85–99. doi: 10.1145/3630106.3658542
arXiv 2024
-
[5]
S. Scardapane and D. Wang, ‘Randomness in neural networks: an overview’, WIREs Data Min. Knowl. Discov., vol. 7, no. 2, p. e1200, 2017, doi: 10.1002/widm.1200
-
[6]
F. Liu, X. Huang, Y . Chen, and J. A. K. Suykens, ‘Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond’, IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 7128–7148, Oct. 2022, doi: 10.1109/TPAMI.2021.3097011
arXiv 2022
-
[7]
J. Frankle, D. J. Schwab, and A. S. Morcos, ‘Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs’, Mar. 21, 2021, arXiv: arXiv:2003.00152. doi: 10.48550/arXiv.2003.00152
-
[8]
K. M. Choromanski et al., ‘Rethinking Attention with Performers’, in International Conference on Learning Representations, 2020
work page 2020
Show all 33 references
-
[9]
H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong, ‘Random Feature Attention’, presented at the International Conference on Learning Representations, Oct. 2020. Accessed: Dec. 09, 2024. [Online]. Available: https://openreview.net/forum?id=QtTKTdVrFBB
2020
-
[10]
R. Sato, M. Yamada, and H. Kashima, ‘Random Features Strengthen Graph Neural Networks’, in Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), in Proceedings. , Society for Industrial and Applied Mathematics, 2021, pp. 333–
2021
-
[11]
A. Saade et al., ‘Random projections through multiple optical scattering: Approximating Kernels at the speed of light’, in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Mar. 2016, pp. 6215–6219. doi: 10.1109/ICASSP.2016.7472872
2016
-
[12]
Teğin, M
U. Teğin, M. Yıldırım, İ. Oğuz, C. Moser, and D. Psaltis, ‘Scalable optical learning operator’, Nat. Comput. Sci. 2021 18, vol. 1, no. 8, pp. 542–549, Aug. 2021, doi: 10.1038/s43588-021-00112-0
2021 doi
-
[13]
L. G. Wright et al., ‘Deep physical neural networks trained with backpropagation’, Nature, vol. 601, no. 7894, Art. no. 7894, Jan. 2022, doi: 10.1038/s41586-021-04223-6
2022 doi
-
[14]
Yildirim et al., ‘Nonlinear optical feature generator for machine learning’, APL Photonics, vol
M. Yildirim et al., ‘Nonlinear optical feature generator for machine learning’, APL Photonics, vol. 8, no. 10, p. 106104, Oct. 2023, doi: 10.1063/5.0158611
2023 doi
-
[15]
L. G. Wright, D. N. Christodoulides, and F. W. Wise, ‘Controllable spatiotemporal nonlinear effects in multimode fibres’, Nat. Photonics, vol. 9, no. 5, pp. 306–310, May 2015, doi: 10.1038/nphoton.2015.61
2015 doi
- [16]
-
[17]
R. Shao, G. Zhang, G. Zhang, X. Gong, and X. Gong, ‘Generalized robust training scheme using genetic algorithm for optical neural networks with imprecise components’, Photonics Res., vol. 10, no. 8, pp. 1868–1876, Aug. 2022, doi: 10.1364/PRJ.449570
2022 doi
-
[18]
Oguz et al., ‘Programming nonlinear propagation for efficient optical learning machines’, Adv
I. Oguz et al., ‘Programming nonlinear propagation for efficient optical learning machines’, Adv. Photonics, vol. 6, no. 1, p. 016002, Jan. 2024, doi: 10.1117/1.AP.6.1.016002
2024 doi
-
[19]
Oguz et al., ‘Forward–forward training of an optical neural network’, Opt
I. Oguz et al., ‘Forward–forward training of an optical neural network’, Opt. Lett., vol. 48, no. 20, pp. 5249–5252, Oct. 2023, doi: 10.1364/OL.496884
2023 doi
-
[20]
Wang et al., ‘Digital Twin of Optical Networks: A Review of Recent Advances and Future Trends’, J
D. Wang et al., ‘Digital Twin of Optical Networks: A Review of Recent Advances and Future Trends’, J. Light. Technol., vol. 42, no. 12, pp. 4233–4259, Jun. 2024, doi: 10.1109/JLT.2024.3401419
2024
-
[21]
Bernard and Y
O. Bernard and Y . Bellouard, ‘On the use of a digital twin to enhance femtosecond laser inscription of arbitrary phase patterns’, J. Phys. Photonics, vol. 3, no. 3, p. 035003, May 2021, doi: 10.1088/2515-7647/abf743
2021 doi
-
[22]
Borhani, E
N. Borhani, E. Kakkava, C. Moser, and D. Psaltis, ‘Learning to see through multimode fibers’, Optica, vol. 5, no. 8, p. 960, Aug. 2018, doi: 10.1364/optica.5.000960
2018 doi
-
[23]
Rahmani et al., ‘Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media’, Nat
B. Rahmani et al., ‘Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media’, Nat. Mach. Intell., vol. 2, no. 7, pp. 403–410, Jul. 2020, doi: 10.1038/s42256-020-0199-9
2020 doi
-
[24]
Antolini et al., ‘Combined HW/SW Drift and Variability Mitigation for PCM-Based Analog In-Memory Computing for Neural Network Applications’, IEEE J
A. Antolini et al., ‘Combined HW/SW Drift and Variability Mitigation for PCM-Based Analog In-Memory Computing for Neural Network Applications’, IEEE J. Emerg. Sel. Top. Circuits Syst., vol. 13, no. 1, pp. 395–407, Mar. 2023, doi: 10.1109/JETCAS.2023.3241750
2023
-
[25]
Hasan, C
M. Hasan, C. Nicholls, K. Pitre, B. Spokoinyi, and T. Hall, ‘Delay drift compensation of an optoelectronic oscillator over a large temperature range through continuous tuning’, Commun. Eng., vol. 3, no. 1, pp. 1–8, Nov. 2024, doi: 10.1038/s44172-024-00301-5
2024 doi
-
[26]
H. Xiao, K. Rasul, and R. Vollgraf, ‘Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms’, ArXiv170807747 Cs Stat, Sep. 2017, Accessed: Apr. 30, 2022. [Online]. Available: http://arxiv.org/abs/1708.07747
2017 arXiv
-
[27]
Kazakov, J
E. Kazakov, J. Gao, P. Anisimov, and V. Zemlyakov, ‘Parallelization of the Generalized Multimode Nonlinear Schrödinger Equation Solver: A Performance Analysis’, in Parallel Computational Technologies, L. Sokolinsky and M. Zymbler, Eds., Cham: Springer Nature Switzerland, 2023,...
2023 doi
-
[28]
J. P. Koplow, D. A. V. Kliner, and L. Goldberg, ‘Single-mode operation of a coiled multimode fiber amplifier’, Opt. Lett., vol. 25, no. 7, pp. 442–444, Apr. 2000, doi: 10.1364/OL.25.000442
-
[29]
Ronneberger, P
O. Ronneberger, P . Fischer, and T. Brox, ‘U-net: Convolutional networks for biomedical image segmentation’, in Medical Image Computing and Computer-Assisted Intervention– MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18,...
2015
-
[30]
R. W. Boyd, Nonlinear Optics, Third Edition, 3rd ed. USA: Academic Press, Inc., 2008
2008
-
[31]
Agrawal, Nonlinear Fiber Optics
G. Agrawal, Nonlinear Fiber Optics. Elsevier Inc., 2012. doi: 10.1016/C2011-0-00045-5
2012 doi
-
[32]
Mafi, ‘Pulse Propagation in a Short Nonlinear Graded-Index Multimode Optical Fiber’, J
A. Mafi, ‘Pulse Propagation in a Short Nonlinear Graded-Index Multimode Optical Fiber’, J. Light. Technol. Vol 30 Issue 17 Pp 2803-2811, vol. 30, no. 17, pp. 2803–2811, Sep. 2012
2012
-
[341]
doi: 10.1137/1.9781611976700.38
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.