pith. sign in

arxiv: 1907.01528 · v1 · pith:CDWKRDZWnew · submitted 2019-07-02 · ⚛️ physics.optics · eess.IV

Deep Learned Optical Multiplexing for Multi-Focal Plane Microscopy

Yi Fei Cheng , Ziad Sabry , Megan Strachan , Skyler Cornell , Jake Chanenson , Eva-Maria S. Collins , Vidya Ganapati This is my paper

Pith reviewed 2026-05-25 10:41 UTC · model grok-4.3

classification ⚛️ physics.optics eess.IV
keywords deep learningLED array microscopemulti-focal plane microscopyoptical multiplexingdigital refocusinglive imagingplanarianneural network post-processing
0
0 comments X

The pith

A jointly optimized LED pattern and neural network extracts five focal planes from one image.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to capture information from multiple focal planes by illuminating the sample with one carefully chosen LED pattern and then feeding the resulting single image into a trained neural network that separates the planes. Training occurs on image stacks from fixed planarians, after which the same pattern and network are used on live, moving samples. This removes both mechanical scanning and the need to acquire many sequential images, raising the speed of multi-plane live microscopy to the rate of a single camera exposure. A sympathetic reader would care because the method promises to make it practical to watch dynamic processes across depth without trading off temporal resolution.

Core claim

By jointly optimizing a single LED illumination pattern with the parameters of a post-processing deep neural network using training sets of LED image stacks from fixed planarians, the authors establish that multiple focal planes can be multiplexed into one image and recovered by inputting that image into the trained network, enabling live imaging at five focal planes.

What carries the argument

The single multiplexed LED illumination pattern jointly optimized with the post-processing deep neural network that encodes and decodes focal-plane information.

If this is right

  • Live multi-focal imaging becomes possible at the temporal resolution of a single camera frame.
  • Mechanical movement of the objective is eliminated for acquiring images at different axial depths.
  • Five focal planes are recovered in live D. japonica planarians after training exclusively on fixed samples.
  • Digital refocusing requires only one exposure instead of a full LED stack.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single-exposure approach could lower total light dose and thereby reduce phototoxicity during extended live observations.
  • The same joint-optimization procedure might be tested on other transparent organisms whose scattering properties resemble those of planarians.
  • Increasing the number of target planes or adapting the pattern for different microscope objectives would be direct next experiments.

Load-bearing premise

The illumination pattern and network weights optimized on fixed-sample LED stacks will successfully separate focal planes when applied to live, moving samples whose optical properties may differ.

What would settle it

Acquire one multiplexed image of a live moving planarian with the optimized pattern and verify that the network produces five distinct sharp images at the target focal planes with no visible mixing or defocus.

Figures

Figures reproduced from arXiv: 1907.01528 by Eva-Maria S. Collins, Jake Chanenson, Megan Strachan, Skyler Cornell, Vidya Ganapati, Yi Fei Cheng, Ziad Sabry.

Figure 1
Figure 1. Figure 1: In our optical setup, we replace a conventional micro [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: shows an example of the results of the shift-add algorithm [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Diagram of training and evaluation of our deep learning framework. (a) In the training step, 69 images are collected for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: This figure illustrates an evaluation example, a field [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparisons between the actual digitally refocused [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Approximately 300,000 image patches of 16 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Our deep learning approach allows us to obtain a 5-plane focal stack with single-shot imaging. This allows for live [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
read the original abstract

To obtain microscope images at multiple focal planes, the distance between the objective and sample can be mechanically adjusted. Images are acquired sequentially at each axial distance. Digital refocusing with a light-emitting diode (LED) array microscope allows elimination of this mechanical movement. In an LED array microscope, the light source of a conventional widefield microscope is replaced with a 2-dimensional LED matrix. A stack of images is acquired from the LED array microscope by sequentially illuminating each LED and capturing an image. Previous work has shown that we can achieve digital refocusing by post-processing this LED image stack. Though mechanical scanning is eliminated, digital refocusing with an LED array microscope has low temporal resolution due to the acquisition of multiple images. In this work, we propose a new paradigm for multi-focal plane microscopy for live imaging, utilizing an LED array microscope and deep learning. In our deep learning approach, we look for a single LED illumination pattern that allows the information from multiple focal planes to be multiplexed into a single image. We jointly optimize this LED illumination pattern with the parameters of a post-processing deep neural network, using a training set of LED image stacks from fixed, not live, Dugesia japonica planarians. Once training is complete, we obtain multiple focal planes by inputting a single multiplexed LED image into the trained post-processing deep neural network. We demonstrate live imaging of a D. japonica planarian at 5 focal planes with our method.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes jointly optimizing a single LED illumination pattern and a post-processing deep neural network on fixed Dugesia japonica planarian LED stacks so that a single multiplexed image can be decoded into five focal planes. After training on fixed samples, the method is applied to live planarians to demonstrate multi-plane imaging without mechanical scanning or sequential LED acquisition.

Significance. If the transfer from fixed-sample training to live specimens holds with quantifiable fidelity, the approach would enable higher temporal resolution multi-focal imaging in LED-array microscopes. The joint optimization of illumination and network is a concrete technical contribution, but the current evidence consists only of a qualitative live demonstration without metrics or baselines.

major comments (2)
  1. [Abstract] Abstract and live-imaging results: the central claim that the fixed-sample-trained illumination pattern and network successfully separate five focal planes on live, moving planarians is supported only by qualitative images; no PSNR, SSIM, focal-plane error, or comparison against sequential LED-stack refocusing is reported for the live data, leaving the generalization performance unquantified.
  2. [Methods] Methods and results sections: the training distribution is restricted to fixed samples, yet the manuscript provides no ablation or hold-out test that quantifies degradation when the same pattern and weights are applied to specimens whose scattering or motion statistics differ from the training set.
minor comments (2)
  1. Figure captions should explicitly state whether displayed live images are raw multiplexed inputs or network outputs at each focal plane.
  2. The manuscript would benefit from a brief statement of the network architecture (layer count, loss function) and the precise optimization objective used for the joint illumination/network training.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each major comment below and indicate where revisions have been made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract and live-imaging results: the central claim that the fixed-sample-trained illumination pattern and network successfully separate five focal planes on live, moving planarians is supported only by qualitative images; no PSNR, SSIM, focal-plane error, or comparison against sequential LED-stack refocusing is reported for the live data, leaving the generalization performance unquantified.

    Authors: We agree that quantitative metrics on live data would strengthen the claims. However, acquiring ground-truth focal planes for moving live specimens via mechanical scanning or sequential LED stacks is not feasible, as it would introduce motion artifacts and alter the experimental conditions the method aims to avoid. The live results are presented as a qualitative demonstration. In the revised manuscript we have added explicit discussion of this limitation in the Results and Discussion sections. revision: partial

  2. Referee: [Methods] Methods and results sections: the training distribution is restricted to fixed samples, yet the manuscript provides no ablation or hold-out test that quantifies degradation when the same pattern and weights are applied to specimens whose scattering or motion statistics differ from the training set.

    Authors: We acknowledge the absence of an explicit ablation or hold-out quantification for live samples. Paired ground-truth stacks for live specimens do not exist in our dataset, precluding direct numerical comparison of degradation. We have revised the Methods and Results sections to clarify the fixed-sample training distribution, note the domain shift to live imaging, and discuss potential effects of differing scattering and motion statistics. revision: partial

Circularity Check

0 steps flagged

No circularity; purely empirical data-driven optimization with no self-referential derivations

full rationale

The paper presents a standard supervised deep-learning pipeline: an LED illumination pattern and post-processing network are jointly trained on fixed-sample LED stacks, then applied at inference time to produce multi-focal outputs. No equations, uniqueness theorems, or fitted parameters are redefined as predictions; the live-sample demonstration is an empirical transfer test rather than a mathematical reduction. The derivation chain is therefore self-contained and externally falsifiable via imaging experiments.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the central claim rests on empirical training and generalization from fixed to live samples.

pith-pipeline@v0.9.0 · 5814 in / 927 out tokens · 26056 ms · 2026-05-25T10:41:31.538135+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 3 internal anchors

  1. [1]

    Light Field Microscopy,

    M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light Field Microscopy,” in ACM SIGGRAPH 2006 Papers , ser. SIGGRAPH ’06. New York, NY , USA: ACM, 2006, pp. 924–934

    work page 2006

  2. [2]

    Simultaneous multiplane imaging with a distorted diffraction grating,

    P. M. Blanchard and A. H. Greenaway, “Simultaneous multiplane imaging with a distorted diffraction grating,” Applied Optics , vol. 38, no. 32, pp. 6692–6699, Nov. 1999

    work page 1999

  3. [3]

    Multifocus microscopy with precise color multi-phase diffractive optics applied in functional neuronal imaging,

    S. Abrahamsson, R. Ilic, J. Wisniewski, B. Mehl, L. Yu, L. Chen, M. Davanco, L. Oudjedi, J.-B. Fiche, B. Hajj, X. Jin, J. Pulupa, C. Cho, M. Mir, M. E. Beheiry, X. Darzacq, M. Nollmann, M. Dahan, C. Wu, T. Lionnet, J. A. Liddle, and C. I. Bargmann, “Multifocus microscopy with precise color multi-phase diffractive optics applied in functional neuronal imag...

    work page 2016

  4. [4]

    Computational multifocal microscopy,

    K. He, Z. Wang, X. Huang, X. Wang, S. Yoo, P. Ruiz, I. Gdor, A. Selewa, N. J. Ferrier, N. Scherer, M. Hereld, A. K. Katsaggelos, and O. Cossairt, “Computational multifocal microscopy,” Biomedical Optics Express, vol. 9, no. 12, pp. 6477–6496, Dec. 2018. 7 Fig. 7: Our deep learning approach allows us to obtain a 5-plane focal stack with single-shot imaging...

    work page 2018

  5. [5]

    Popescu, Quantitative phase imaging of cells and tissues , ser

    G. Popescu, Quantitative phase imaging of cells and tissues , ser. McGraw-Hill Biophotonics. New York: McGraw-Hill, 2011, oCLC: ocn659763966

    work page 2011

  6. [6]

    Microscopy refocusing and dark- field imaging by using a simple LED array,

    G. Zheng, C. Kolner, and C. Yang, “Microscopy refocusing and dark- field imaging by using a simple LED array,” Optics Letters , vol. 36, no. 20, pp. 3987–3989, Oct. 2011

    work page 2011

  7. [7]

    3d differential phase-contrast mi- croscopy with computational illumination using an LED array,

    L. Tian, J. Wang, and L. Waller, “3d differential phase-contrast mi- croscopy with computational illumination using an LED array,” Optics Letters, vol. 39, no. 5, p. 1326, Mar. 2014

    work page 2014

  8. [8]

    Wide-field, high-resolution Fourier ptychographic microscopy,

    G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nature photonics, vol. 7, no. 9, pp. 739–745, Sep. 2013

    work page 2013

  9. [9]

    Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,

    L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,” Biomedical Optics Express , vol. 5, no. 7, p. 2376, Jul. 2014

    work page 2014

  10. [10]

    Com- putational illumination for high-speed in vitro Fourier ptychographic microscopy,

    L. Tian, Z. Liu, L.-H. Yeh, M. Chen, J. Zhong, and L. Waller, “Com- putational illumination for high-speed in vitro Fourier ptychographic microscopy,” Optica, vol. 2, no. 10, p. 904, Oct. 2015

    work page 2015

  11. [11]

    Optimal physical preprocessing for example-based super-resolution,

    A. Robey and V . Ganapati, “Optimal physical preprocessing for example-based super-resolution,” Optics Express , vol. 26, no. 24, pp. 31 333–31 350, Nov. 2018

    work page 2018

  12. [12]

    Illumination pattern design with deep learning for single-shot Fourier ptychographic microscopy,

    Y . F. Cheng, M. Strachan, Z. Weiss, M. Deb, D. Carone, and V . Ganapati, “Illumination pattern design with deep learning for single-shot Fourier ptychographic microscopy,” Optics Express, vol. 27, no. 2, pp. 644–656, Jan. 2019

    work page 2019

  13. [13]

    Data-Driven Design for Fourier Ptychographic Microscopy

    M. Kellman, E. Bostan, M. Chen, and L. Waller, “Data-Driven Design for Fourier Ptychographic Microscopy,” arXiv:1904.04175 [cs, eess] , Apr. 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  14. [14]

    Ptychnet: CNN based fourier ptychography,

    A. Kappeler, S. Ghosh, J. Holloway, O. Cossairt, and A. Katsaggelos, “Ptychnet: CNN based fourier ptychography,” in 2017 IEEE Interna- tional Conference on Image Processing (ICIP) , Sep. 2017, pp. 1712– 1716

    work page 2017

  15. [15]

    Deep learning approach for Fourier ptychography microscopy,

    T. Nguyen, Y . Xue, Y . Li, L. Tian, and G. Nehmetallah, “Deep learning approach for Fourier ptychography microscopy,” Optics Express, vol. 26, no. 20, pp. 26 470–26 484, Oct. 2018

    work page 2018

  16. [16]

    Phase retrieval for Fourier Ptychography under varying amount of measurements

    L. Boominathan, M. Maniparambil, H. Gupta, R. Baburajan, and K. Mi- tra, “Phase retrieval for Fourier Ptychography under varying amount of measurements,” arXiv:1805.03593 [cs] , May 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  17. [17]

    Reliable deep-learning-based phase imaging with uncertainty quantification,

    Y . Xue, S. Cheng, Y . Li, and L. Tian, “Reliable deep-learning-based phase imaging with uncertainty quantification,” Optica, vol. 6, no. 5, pp. 618–629, May 2019

    work page 2019

  18. [18]

    Fourier ptychographic microscopy reconstruction with multiscale deep residual network,

    J. Zhang, T. Xu, Z. Shen, Y . Qiao, and Y . Zhang, “Fourier ptychographic microscopy reconstruction with multiscale deep residual network,” Op- tics Express, vol. 27, no. 6, p. 8612, Mar. 2019

    work page 2019

  19. [19]

    Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections

    X.-J. Mao, C. Shen, and Y .-B. Yang, “Image Restoration Us- ing Convolutional Auto-encoders with Symmetric Skip Connections,” arXiv:1606.08921 [cs] , Jun. 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  20. [20]

    Planarian cholinesterase: molecular and functional characterization of an evolutionarily ancient enzyme to study organophosphorus pesticide toxicity,

    D. Hagstrom, S. Zhang, A. Ho, E. S. Tsai, Z. Radi, A. Jahromi, K. J. Kaj, Y . He, P. Taylor, and E.-M. S. Collins, “Planarian cholinesterase: molecular and functional characterization of an evolutionarily ancient enzyme to study organophosphorus pesticide toxicity,” Archives of Toxicology, vol. 92, no. 3, pp. 1161–1176, Mar. 2018

    work page 2018

  21. [21]

    Smed- dynA-1 is a planarian nervous system specific dynamin 1 homolog required for normal locomotion,

    J. A. Talbot, K. W. Currie, B. J. Pearson, and E.-M. S. Collins, “Smed- dynA-1 is a planarian nervous system specific dynamin 1 homolog required for normal locomotion,” Biology Open, vol. 3, no. 7, pp. 627– 634, Jul. 2014

    work page 2014

  22. [22]

    Noise2Noise: Learning Image Restoration without Clean Data,

    J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2Noise: Learning Image Restoration without Clean Data,” in International Conference on Machine Learning , Jul. 2018, pp. 2965–2974

    work page 2018

  23. [23]

    Non-parametric Entropy Estimation Toolbox,

    G. V . Steeg, “Non-parametric Entropy Estimation Toolbox,” Oct. 2014. [Online]. Available: https://github.com/gregversteeg/NPEET 8

    work page 2014

  24. [24]

    Aperture scanning Fourier ptychographic microscopy,

    X. Ou, J. Chung, R. Horstmeyer, and C. Yang, “Aperture scanning Fourier ptychographic microscopy,” Biomedical Optics Express , vol. 7, no. 8, pp. 3140–3150, Aug. 2016

    work page 2016

  25. [25]

    Deconvolution Microscopy,

    J.-B. Sibarita, “Deconvolution Microscopy,” in Microscopy Techniques, ser. Advances in Biochemical Engineering, J. Rietdorf, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 201–243

    work page 2005

  26. [26]

    Diffraction tomography with Fourier ptychography,

    R. Horstmeyer, J. Chung, X. Ou, G. Zheng, and C. Yang, “Diffraction tomography with Fourier ptychography,” Optica, vol. 3, no. 8, pp. 827– 835, Aug. 2016

    work page 2016

  27. [27]

    High-throughput intensity diffraction tomography with a computational microscope,

    R. Ling, W. Tahir, H.-Y . Lin, H. Lee, and L. Tian, “High-throughput intensity diffraction tomography with a computational microscope,” Biomedical Optics Express , vol. 9, no. 5, p. 2130, May 2018

    work page 2018

  28. [28]

    High-speed in vitro intensity diffraction tomography,

    J. Li, A. Matlock, Y . Li, Q. Chen, C. Zuo, and L. Tian, “High-speed in vitro intensity diffraction tomography,” arXiv:1904.06004 [physics] , Apr. 2019

    work page arXiv 1904

  29. [29]

    Separation of three-dimensional scattering effects in tilt-series Fourier ptychography,

    P. Li, D. J. Batey, T. B. Edo, and J. M. Rodenburg, “Separation of three-dimensional scattering effects in tilt-series Fourier ptychography,” Ultramicroscopy, vol. 158, pp. 1–7, Nov. 2015

    work page 2015

  30. [30]

    3d intensity and phase imaging from light field measurements in an LED array microscope,

    L. Tian and L. Waller, “3d intensity and phase imaging from light field measurements in an LED array microscope,” Optica, vol. 2, no. 2, pp. 104–111, Feb. 2015

    work page 2015