pith. sign in

arxiv: 2508.01981 · v7 · submitted 2025-08-04 · ⚛️ physics.optics · eess.IV

Deep Feature-specific Imaging

Yizhou Lu , Andreas Velten This is my paper

Pith reviewed 2026-05-19 01:32 UTC · model grok-4.3

classification ⚛️ physics.optics eess.IV
keywords feature-specific imagingdeep neural networksPoisson noisecomputational imagingmeasurement masksclassification accuracyphoton-limited imagingoptical-electronic co-design
0
0 comments X

The pith

DeepFSI optimizes optical masks with a neural network under Poisson noise to raise classification accuracy over PCA-based feature-specific imaging.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DeepFSI, an end-to-end optical-electronic framework that unfreezes PCA-derived measurement masks so a deep neural network can optimize them directly. Conventional PCA-based feature-specific imaging preserves variance under additive Gaussian noise assumptions, yet modern photon-counting sensors are dominated by Poisson noise that makes those masks suboptimal for task performance. By computing gradients through a simulated Poisson-plus-additive noise model, the network learns masks that deliver higher classification accuracy and more reliable transfer from simulation to physical hardware across photon budgets. This matters for any low-light application where sensor noise follows photon arrival statistics rather than constant variance.

Core claim

DeepFSI unfreezes PCA-derived masks and enables a deep neural network to learn globally optimal measurement masks by computing gradients directly under realistic Poisson and additive noise conditions. Simulations and hardware experiments demonstrate that this yields improved classification accuracy and stronger transfer robustness compared to PCA-based FSI across varying photon budgets, particularly in Poisson-noise-dominant environments, while also maintaining performance under additive Gaussian noise and showing greater robustness to design choices.

What carries the argument

End-to-end gradient optimization of unfrozen PCA-derived measurement masks inside a deep network trained under a simulated Poisson-plus-additive noise model

If this is right

  • Classification accuracy rises across a range of photon budgets when masks are task-optimized rather than variance-optimized.
  • Transfer from simulation to physical hardware becomes more reliable under Poisson-dominant noise.
  • The same framework continues to work when the dominant noise is additive Gaussian instead of Poisson.
  • Performance becomes less sensitive to exact choices of network architecture or initialization.
  • The approach extends the usefulness of feature-specific imaging into photon-limited regimes where conventional methods lose their edge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Task-driven mask optimization could replace variance-driven methods in other computational imaging pipelines once the noise statistics are known.
  • Real-time adaptation of masks based on measured photon flux might become feasible if the network can be updated on the fly.
  • Combining the learned masks with task-specific loss functions beyond classification could improve performance in detection or reconstruction problems.

Load-bearing premise

That masks found by gradient descent in a simulated noise model will remain near-optimal when deployed on real photon-counting hardware without large simulation-to-reality gaps.

What would settle it

Hardware trials with a real photon-counting sensor at low photon budgets that show DeepFSI masks yielding classification accuracy no higher than PCA masks would falsify the performance gain.

Figures

Figures reproduced from arXiv: 2508.01981 by Andreas Velten, Yizhou Lu.

Figure 1
Figure 1. Figure 1: Single-pixel camera configuration. 𝒙: field of view, 𝑴: masks of coding, 𝒚: counted photon numbers at the sensor, 𝑴−1 : reconstruction operator, 𝒙˜: reconstructed object. (a) Only one pixel (white) is scanned in each measurement, and the measured data requires no reconstruction. (b) The sum of all white pixels is measured in each measurement and it requires a decoding step to reconstruct the field of view.… view at source ↗
Figure 2
Figure 2. Figure 2: (a) the general configuration of scanner-classifier networks. (b) the node-level architecture of our scanner (green diamonds)-classifier (blue circles) network. Noise is implemented after the sensing matrix 𝑴. Different from other coding schemes, DeepFSI has a trainable scanner where 𝑴 is not fixed and can be optimized by the gradient of the classifier. To further prove the necessity of our hardware optimi… view at source ↗
Figure 3
Figure 3. Figure 3: OViT Overview. The process begins by dividing the input image into fixed-size [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experimental configuration of the single-pixel camera: The light path is [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Raw image (left) VS SPC-observed image [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A DMD mask with decimal values. Black regions are OFF-state and white [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Classification rates on simulated data. RS: Raster scan. HB: Hadamard basis. II: Impulse imaging. TH: Low Frequency Truncated Hadamard basis. FSI: PCA basis. DFSI: DeepFSI. CG: control group. Feature-specific methods are marked by ⃝ from traditional ones marked by □. The CG employs the optimized masks under the AGN and trains the software classifier under PN. Its overall performance is still worse than Dee… view at source ↗
Figure 8
Figure 8. Figure 8: Classification rates on simulated data regarding compression ratios and light [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Classification rates of different optical coding with ViT on CIFAR10 under [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Classification rates of different optical coding on MNIST in the hardware [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
read the original abstract

Modern photon-counting sensors are increasingly dominated by Poisson noise, yet conventional feature-specific imaging (FSI), based on principal component analysis (PCA), is optimized for additive Gaussian noise and variance preservation rather than task-specific objectives, leading to suboptimal performance and a loss of its advantages under Poisson noise. To address this, we introduce DeepFSI, what we believe to be a novel end-to-end optical-electronic framework. DeepFSI "unfreezes" PCA-derived masks, enabling a deep neural network to learn globally optimal measurement masks by computing gradients directly under realistic Poisson and additive noise conditions. Simulations and hardware experiments demonstrate that DeepFSI achieves improved classification accuracy and stronger transfer robustness compared to PCAbased FSI across varying photon budgets, particularly in Poisson-noise-dominant environments. DeepFSI also exhibits enhanced robustness to design choices and performs well under additive Gaussian noise, representing a significant advance for noise-robust computational imaging in photon-limited applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces DeepFSI, an end-to-end framework that unfreezes PCA-derived measurement masks and optimizes them via a deep neural network whose gradients are computed under a simulated Poisson-plus-additive noise model. The central claim is that the resulting masks yield higher classification accuracy and greater transfer robustness than conventional PCA-based FSI across photon budgets, with particular gains in Poisson-noise-dominant regimes, as demonstrated in both simulations and hardware experiments.

Significance. If the hardware-transfer results hold, the work offers a concrete route to task-specific optical front-ends that outperform variance-preserving PCA under realistic photon-counting noise. The explicit inclusion of hardware validation and the demonstration of robustness to design choices are strengths that could influence practical low-light imaging systems.

major comments (2)
  1. [§4.2] §4.2 (Noise Model): The transfer-robustness claim in §5.3 rests on the assumption that the simulated Poisson-plus-additive noise faithfully reproduces the hardware sensor statistics. No quantitative validation—such as measured versus modeled variance-mean curves or noise histograms at the tested photon budgets—is presented, leaving open the possibility that observed hardware gains arise from unmodeled factors.
  2. [§5.1] §5.1 (Classification Accuracy): The reported accuracy improvements over PCA-FSI are stated without error bars, number of trials, or statistical significance tests. This omission makes it difficult to judge whether the gains are reliable or merely within the variability of the experimental setup.
minor comments (2)
  1. [Abstract] The abstract and §3.1 would benefit from a concise statement of the network architecture, loss function, and training hyperparameters to support reproducibility.
  2. [§5] Figure captions in §5 should explicitly distinguish simulation curves from hardware data points to avoid reader confusion.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for their thoughtful review and valuable suggestions. Below, we provide point-by-point responses to the major comments and indicate the revisions we will make to the manuscript.

read point-by-point responses

  1. Referee: [§4.2] §4.2 (Noise Model): The transfer-robustness claim in §5.3 rests on the assumption that the simulated Poisson-plus-additive noise faithfully reproduces the hardware sensor statistics. No quantitative validation—such as measured versus modeled variance-mean curves or noise histograms at the tested photon budgets—is presented, leaving open the possibility that observed hardware gains arise from unmodeled factors.

    Authors: We agree that quantitative validation of the noise model would strengthen the transfer-robustness claims. In the revised manuscript, we will include variance-mean curves comparing measured hardware noise statistics to the simulated Poisson-plus-additive model at the relevant photon budgets, along with noise histograms to demonstrate model fidelity. This will confirm that the observed gains arise from the optimized masks. revision: yes

  2. Referee: [§5.1] §5.1 (Classification Accuracy): The reported accuracy improvements over PCA-FSI are stated without error bars, number of trials, or statistical significance tests. This omission makes it difficult to judge whether the gains are reliable or merely within the variability of the experimental setup.

    Authors: We acknowledge the need for statistical rigor. In the revised manuscript, we will report error bars as standard deviations over multiple independent trials (with the number of trials explicitly stated) and include statistical significance tests (e.g., paired t-tests) to establish the reliability of the accuracy improvements. revision: yes

Circularity Check

0 steps flagged

No significant circularity; optimization under external noise model is independent of claimed results

full rationale

The paper presents DeepFSI as an end-to-end optimization of measurement masks via gradients computed under a simulated Poisson-plus-additive noise model, with performance then validated through separate simulations and hardware experiments. No equations or steps reduce the claimed accuracy improvements or transfer robustness to a fitted parameter renamed as a prediction, a self-referential definition, or a load-bearing self-citation chain. The central derivation relies on standard gradient-based learning under an explicitly stated external noise model rather than on any result that is constructed from the target outcomes themselves. Hardware validation is presented as an independent check rather than an input to the derivation.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the premise that gradient-based optimization through a differentiable Poisson noise model will discover superior masks and that simulation-to-hardware transfer will hold.

free parameters (1)
  • Neural network parameters
    Weights and biases of the deep network are fitted during training to produce the measurement masks.
axioms (1)
  • domain assumption The forward imaging process including Poisson noise is differentiable and accurately modeled for gradient computation.
    Invoked when the paper states that gradients are computed directly under realistic Poisson and additive noise conditions.

pith-pipeline@v0.9.0 · 5678 in / 1048 out tokens · 47104 ms · 2026-05-19T01:32:03.446551+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 1 internal anchor

  1. [1]

    High flux passive imaging with single-photon sensors,

    A. Ingle, A. Velten, and M. Gupta, “High flux passive imaging with single-photon sensors,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,(2019), pp. 6760–6769

    work page 2019

  2. [2]

    Multi-target spectrally resolved fluorescence lifetime imaging microscopy,

    T. Niehörster, A. Löschberger, I. Gregor,et al., “Multi-target spectrally resolved fluorescence lifetime imaging microscopy,” Nat. methods13, 257–262 (2016)

    work page 2016

  3. [3]

    Spad imagers for super resolution localization microscopy enable analysis of fast fluorophore blinking,

    I. M. Antolovic, S. Burri, C. Bruschini,et al., “Spad imagers for super resolution localization microscopy enable analysis of fast fluorophore blinking,” Sci. reports7, 44108 (2017)

    work page 2017

  4. [4]

    Quantum-inspired computational imaging,

    Y. Altmann, S. McLaughlin, M. J. Padgett,et al., “Quantum-inspired computational imaging,” Science361, eaat2298 (2018)

    work page 2018

  5. [5]

    When does computational imaging improve performance?

    O. Cossairt, M. Gupta, and S. K. Nayar, “When does computational imaging improve performance?” IEEE transactions on image processing22, 447–458 (2012)

    work page 2012

  6. [6]

    Compressed sensing performance bounds under poisson noise,

    M. Raginsky, R. M. Willett, Z. T. Harmany, and R. F. Marcia, “Compressed sensing performance bounds under poisson noise,” IEEE Trans. on Signal Process.58, 3990–4002 (2010)

    work page 2010

  7. [7]

    Harwit and N

    M. Harwit and N. J. A. Sloane,Hadamard Transform Optics(Academic Press, 1979)

    work page 1979

  8. [8]

    Feature-specific imaging,

    M. Neifeld and P. Shankar, “Feature-specific imaging,” Appl. optics42, 3379–89 (2003)

    work page 2003

  9. [9]

    Multispectral principal component imaging,

    H. S. Pal and M. A. Neifeld, “Multispectral principal component imaging,” Opt Express11, 2118–25 (2003 Sep 8)

    work page 2003

  10. [10]

    Optimizing measurements for feature-specific compressive sensing,

    A. Mahalanobis and M. Neifeld, “Optimizing measurements for feature-specific compressive sensing,” Appl. Opt.53, 6108–6118 (2014)

    work page 2014

  11. [11]

    Adaptive feature-specific imaging: a face recognition example,

    P. K. Baheti and M. A. Neifeld, “Adaptive feature-specific imaging: a face recognition example,” Appl. optics47, B21–B31 (2008)

    work page 2008

  12. [12]

    Optimal transformations for optical multiplex measurements in the presence of photon noise,

    A. Wuttig, “Optimal transformations for optical multiplex measurements in the presence of photon noise,” Appl. Opt. 44, 2710–2719 (2005)

    work page 2005

  13. [13]

    Can we beat hadamard multiplexing? data driven design and analysis for computational imaging systems,

    K. Mitra, O. Cossairt, and A. Veeraraghavan, “Can we beat hadamard multiplexing? data driven design and analysis for computational imaging systems,” in2014 IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–9

    work page 2014

  14. [14]

    Optimal multiplexed sensing: bounds, conditions and a graph theory link,

    N. Ratner, Y. Y. Schechner, and F. Goldberg, “Optimal multiplexed sensing: bounds, conditions and a graph theory link,” Opt. express15, 17072–17092 (2007)

    work page 2007

  15. [15]

    Matrix optimization for poisson compressed sensing,

    M. Mordechay and Y. Y. Schechner, “Matrix optimization for poisson compressed sensing,” in2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP),(IEEE, 2014), pp. 684–688

    work page 2014

  16. [16]

    Dirtypixels: Towardsend-to-endimageprocessingandperception,

    S.Diamond,V.Sitzmann,F.Julca-Aguilar,etal.,“Dirtypixels: Towardsend-to-endimageprocessingandperception,” ACM Trans. on Graph. (TOG)40, 1–15 (2021)

    work page 2021

  17. [17]

    Deep compressed imaging via optimized pattern scanning,

    K. Zhang, J. Hu, and W. Yang, “Deep compressed imaging via optimized pattern scanning,” Photonics research9, B57–B70 (2021)

    work page 2021

  18. [18]

    Middle output regularized end-to-end optimization for computational imaging,

    R. Jacome, P. Gomez, and H. Arguello, “Middle output regularized end-to-end optimization for computational imaging,” Optica10, 1421–1431 (2023)

    work page 2023

  19. [19]

    Physics vs. learned priors: Rethinking camera and algorithm design for task-specific imaging,

    T. Klinghoffer, S. Somasundaram, K. Tiwary, and R. Raskar, “Physics vs. learned priors: Rethinking camera and algorithm design for task-specific imaging,” in2022 IEEE International Conference on Computational Photography (ICCP),(IEEE, 2022), pp. 1–12

    work page 2022

  20. [20]

    Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification,

    J. Chang, V. Sitzmann, X. Dun,et al., “Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification,” Sci. reports8, 1–10 (2018)

    work page 2018

  21. [21]

    Learning privacy-preserving optics for human pose estimation,

    C. Hinojosa, J. C. Niebles, and H. Arguello, “Learning privacy-preserving optics for human pose estimation,” in Proceedings of the IEEE/CVF international conference on computer vision,(2021), pp. 2573–2582

    work page 2021

  22. [22]

    Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging,

    X. Dun, H. Ikoma, G. Wetzstein,et al., “Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging,” Optica7, 913–922 (2020)

    work page 2020

  23. [23]

    Deep optics for single-shot high-dynamic-range imaging,

    C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for single-shot high-dynamic-range imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,(2020), pp. 1375–1385

    work page 2020

  24. [24]

    Deep optics for monocular depth estimation and 3d object detection,

    J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” inProceedings of the IEEE/CVF International Conference on Computer Vision,(2019), pp. 10193–10202

    work page 2019

  25. [25]

    Neural auto-exposure for high-dynamic range object detection,

    E. Onzon, F. Mannan, and F. Heide, “Neural auto-exposure for high-dynamic range object detection,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition,(2021), pp. 7710–7720

    work page 2021

  26. [26]

    Hybrid training of optical neural networks,

    J. Spall, X. Guo, and A. I. Lvovsky, “Hybrid training of optical neural networks,” Optica9, 803–811 (2022)

    work page 2022

  27. [27]

    Single-pixel imaging 12 years on: a review,

    G. M. Gibson, S. D. Johnson, and M. J. Padgett, “Single-pixel imaging 12 years on: a review,” Opt. express28, 28190–28208 (2020)

    work page 2020

  28. [28]

    Deep camera obscura: an image restoration pipeline for pinhole photography,

    J. D. Rego, H. Chen, S. Li,et al., “Deep camera obscura: an image restoration pipeline for pinhole photography,” Opt. Express30, 27214–27235 (2022)

    work page 2022

  29. [29]

    Single-pixel imaging via compressive sampling,

    M. F. Duarte, M. A. Davenport, D. Takhar,et al., “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag.25, 83–91 (2008)

    work page 2008

  30. [30]

    An optical neural network using less than 1 photon per multiplication,

    T. Wang, S.-Y. Ma, L. G. Wright,et al., “An optical neural network using less than 1 photon per multiplication,” Nat. Commun.13, 123 (2022)

    work page 2022

  31. [31]

    Compressed domain image classification using a dynamic-rate neural network,

    Y. Xu, W. Liu, and K. F. Kelly, “Compressed domain image classification using a dynamic-rate neural network,” IEEE Access8, 217711–217722 (2020)

    work page 2020

  32. [32]

    Differentiable compound optics and processing pipeline optimization for end-to-end camera design,

    E. Tseng, A. Mosleh, F. Mannan,et al., “Differentiable compound optics and processing pipeline optimization for end-to-end camera design,” ACM Trans. on Graph. (TOG)40, 1–19 (2021)

    work page 2021

  33. [33]

    Computational photography: Epsilon to coded photography,

    R. Raskar, “Computational photography: Epsilon to coded photography,” inEmerging Trends in Visual Computing: LIX Fall Colloquium, ETVC 2008, Palaiseau, France, November 18-20, 2008. Revised Invited Papers,(Springer, 2009), pp. 238–253

    work page 2008

  34. [34]

    A Review Paper: Noise Models in Digital Image Processing

    A. K. Boyat and B. K. Joshi, “A review paper: Noise models in digital image processing,” ArXivabs/1505.03489 (2015)

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  35. [35]

    Coded aperture imaging: novel approaches to high-energy high-resolution laboratory imaging,

    M. Selwood, “Coded aperture imaging: novel approaches to high-energy high-resolution laboratory imaging,” Ph.D. thesis, University of York (2022)

    work page 2022

  36. [36]

    Gradient-based learning applied to document recognition,

    Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE86, 2278–2324 (1998)

    work page 1998

  37. [37]

    Single pixel imaging at high pixel resolutions,

    R. Stojek, A. Pastuszczak, P. Wróbel, and R. Kotyński, “Single pixel imaging at high pixel resolutions,” Opt. Express 30, 22730–22745 (2022)

    work page 2022