Pith. sign in

REVIEW 2 major objections 8 minor 2 cited by

Computationally Efficient Information-Driven Optical Design with Interchanging Optimization

T0 review · 2 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Information-driven optical design can be made up to 6x cheaper by alternating density fitting and optical updates.

desk verdict A useful empirical extension of IDEAL with credible efficiency gains, but the unquantified ∂ψ/∂θ approximation keeps the design-quality claims conditional. read the letter →

arxiv 2507.07789 v2 pith:BQY6EXPZ submitted 2025-07-10 eess.IV cs.CEcs.CVcs.ITmath.ITphysics.optics

Eric Markley , Henry Pinkard , Leyla Kabuli , Nalini Singh , Laura Waller This is my paper
classification eess.IVcs.CEcs.CVcs.ITmath.ITphysics.optics
keywords computationalimaginginformation-drivenopticaldesignmutualinformationalternatingoptimizationdensityestimationdiffractiveopticslenslesssnapshot3Dmicroscopy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Information-driven optical design aims to choose a lens, phase mask, or other optical element by maximizing the mutual information between scenes and measurements, without training a reconstruction network. This paper argues that the standard way of performing that optimization, which backpropagates through the density model used to estimate entropy, carries high memory and runtime costs and restricts the choice of density models to simple, differentiable ones. The proposed method, IDEAL-IO, alternates between fitting the density model to current measurements with gradients frozen and updating the optical parameters with that model held fixed. This removes the need to differentiate through the fitting process, cutting runtime and memory usage by up to 6x while permitting expressive models such as PixelCNN. The paper validates the approach on snapshot 3D microscopy, diffractive optical elements, and lensless imaging, and reports that the cheaper optimization matches or improves design quality.

What carries the argument

The objective is the mutual information $\hat{I}(X;Y) = \hat{H}(Y) - H(Y|X)$, where $H(Y|X)$ is computed analytically from the noise model and $\hat{H}(Y)$ is a cross-entropy estimate on held-out noisy measurements using a fitted density model $p_\psi$. The carrying mechanism is the alternating scheme: fit $\psi$ to measurements generated with the current optical parameters, then update the optical parameters using the fixed density model and the known noise law. The validity of the scheme rests on omitting the term $\partial p/\partial\psi \cdot \partial\psi/\partial\theta$ from the optical gradient, which the paper justifies by an empirical claim that the fitted distribution parameters change slowly with respect to the imaging-system parameters. Patch-based entropy estimation keeps the memory footprint manageable and is the setting in which the 6x runtime and memory reductions are measured.

What would settle it

Compute the full gradient including $\partial\psi/\partial\theta$ whenever the density model fitting is differentiable, and compare the resulting optical update trajectories with the IDEAL-IO approximation on the same tasks and initialization; if the final designs or the mutual-information values diverge substantially, particularly on non-Gaussian benchmarks such as MNIST, the small-gradient assumption is violated.

Watch

Extended reading notes

Core claim

The central claim is that the gradient of the fitted density-model parameters with respect to the optical parameters, $\partial\psi/\partial\theta$, is small enough to omit from the mutual-information gradient, and that omitting it converts the costly monolithic backpropagation of IDEAL into two cheap alternating steps. Concretely, the optical update in Equation 8 drops the term $\frac{\partial p}{\partial\psi}\frac{\partial\psi}{\partial\theta}$ while the density model is refit outside the differentiation path as in Equation 7. The paper reports that this reduces runtime and GPU memory by up to 6x compared with original IDEAL, and it enables density models that are not differentiable or not practical to differentiate, such as PixelCNN. On lensless imaging with highly non-Gaussian measurements, the PixelCNN-guided design achieves substantially higher test mutual information than a Gaussian-guided design, indicating that the decoupling does more than save compute: it changes which designs are reachable.

Load-bearing premise

The method assumes that the fitted density-model parameters change only slowly as the optical parameters change, so ignoring the gradient of those parameters in the optical update does not bias the optimization; the paper states this as an empirical observation but does not quantify or bound the error.

Editorial extensions

If this is right

  • Scaling improves substantially: for the diffractive-optics task, IDEAL-IO required about 7.9 GB of GPU memory where IDEAL required 44.2 GB, and its runtime per optimization step grew roughly 0.007 seconds per 100 additional patch pixels versus roughly 0.098 seconds for IDEAL.
  • Expressive density models become usable, and they change designs: on MNIST lensless imaging, the PixelCNN-optimized design reached 0.473 bits/pixel of test mutual information versus 0.208 for the Gaussian-optimized design.
  • Mutual-information maximization can replace end-to-end training as a practical objective: in snapshot 3D microscopy, IDEAL-IO converged in about 25 minutes without training a decoder, about 4x faster than the roughly 2-hour end-to-end baseline, while producing a similar lenslet design.
  • The method does not sacrifice design quality on the tested tasks; for the diffractive element, both IDEAL and IDEAL-IO converged to Fresnel-like zone-plate designs without explicit structural priors.
  • Because the fitting and optimization stages are decoupled, the same framework can be applied to other density model families and other differentiable optical forward models without re-deriving end-to-end gradients.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the smallness of $\partial\psi/\partial\theta$ is likely to depend on how far the optimization is from convergence and on the density model family; if it becomes nonnegligible early in training, the effective objective could differ from true mutual information in ways that the reported loss curves would only partially reveal.
  • Editorial inference: the decoupling principle could extend beyond mutual information to other information-theoretic objectives, such as task-specific or class-conditional information, by alternating any differentiable information estimator with any nondifferentiable fitting procedure.
  • Editorial inference: because IDEAL-IO is cheap enough to run many initializations, it could serve as a fast warm-start for end-to-end design, first exploring the optical parameter space with information maximization and then fine-tuning jointly with a decoder.
  • Editorial inference: the patch-size trade-off the paper identifies suggests a natural testable extension toward multiscale or hierarchical density models, which could capture larger spatial correlations without the memory cost of full-frame estimation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper proposes IDEAL-IO, an alternating optimization scheme for information-driven optical design. Instead of differentiating through the density-model fitting procedure as in the original IDEAL, IDEAL-IO alternates between fitting a density model to measurements from the current optical system and updating the optical parameters while holding the fitted density model fixed. The central approximation is that the term ∂ψ/∂θ in the total derivative of the mutual-information estimate is small, so the gradient update in Eq. (8) ignores it. The authors report up to 6× reductions in runtime and memory, the ability to use expressive density models such as PixelCNN, and validation on snapshot 3D microscopy, diffractive optical elements, and lensless imaging.

Significance. If the claims hold, IDEAL-IO makes information-theoretic optical design substantially more practical by removing the differentiability requirement on density estimation. The paper's strengths include measured memory and runtime scaling (Fig. 3b–c, Table 1), an independent reconstruction-fidelity comparison in Sec. 4.1 using NMSE from a separately trained decoder, and the observation that the DOE optimization converges to a known Fresnel zone plate. These elements provide credible evidence for the efficiency contribution. However, the central gradient approximation is not quantified, and the lensless 'superior design' claim in Sec. 4.3 is evaluated with the same PixelCNN mutual-information quantity that guides the optimization, which is circular. The efficiency result is sound, but the design-quality claim needs additional support before the paper's full message can be accepted.

major comments (2)
  1. [Section 3, Eq. (8)] The central approximation ∂ψ/∂θ≈0 is stated as an empirical observation but never quantified. More importantly, the correct sufficient condition for the dropped term to vanish is not smallness of ∂ψ/∂θ but first-order optimality of ψ with respect to the loss being differentiated: if ψ_{t+1} exactly minimized the cross-entropy on the same measurements used in the gradient, the term ∂p/∂ψ ∂ψ/∂θ would vanish by the envelope theorem. The paper's actual schedule violates this condition: the PixelCNN is refit only every K=50 optical updates (Sec. 4.3), and the entropy estimate uses held-out measurements (Eqs. 3–4), so ψ is not at the optimum of the evaluated loss. The authors should provide a direct measurement of the dropped gradient term or of ∂L/∂ψ over training, or demonstrate that final designs are insensitive to K and to the approximation. Without this, the claim that IDEAL-IO maintains or improves design quality while maximizing true mutual information is unsupported.
  2. [Section 4.3, Fig. 4c] The lensless imaging evaluation uses PixelCNN mutual information on held-out test data as the design-quality metric, which is exactly the quantity that the PixelCNN-guided optimization maximizes. The finding that PixelCNN-optimized designs achieve higher PixelCNN MI is therefore expected and does not independently establish that the designs are 'superior.' The manuscript needs an external evaluation metric, such as reconstruction NMSE with a separately trained decoder (as done in Sec. 4.1) or a downstream task accuracy, to support the abstract's claim that expressive models 'guide optimization toward superior designs.' Without such an independent metric, the lensless result only demonstrates that the optimizer succeeds at maximizing its own surrogate.
minor comments (8)
  1. [Abstract and Sec. 4.2] The abstract claims 'up to 6×' reduction, but the reported results are approximately 4× faster than the E2E baseline, roughly 3× faster than IDEAL, and about 5.6× lower peak memory; please specify which quantity yields the 6× factor or adjust the claim.
  2. [Section 3, Eqs. (3)–(4) and (7)–(8)] It is unclear whether ψ is fit on the same measurements used for the entropy estimate in Eq. (8); if the training and evaluation sets differ, the envelope-theorem reasoning is unavailable and the bias from the dropped term should be discussed explicitly.
  3. [Section 4.2] The text states that IDEAL-IO's training curve appears noisier due to its smaller test set, but it also states that both methods used 6480 patches; please clarify the test-set sizes used for the loss curves.
  4. [Section 4.1, Fig. 2c] The figure label says 'Training decreases error and increases information,' but the plot shows MI versus NMSE at three checkpoints, not training curves; please relabel or replot as functions of training step.
  5. [Section 4.1] The word 'fluorescense' should be 'fluorescence.'
  6. [References] References [7] and [9] appear to describe the same work, one listed as an arXiv preprint and one as anonymous under review; please reconcile these to avoid duplicate self-citation.
  7. [Section 4.3] Final MI values for CIFAR10 and MNIST are reported without standard deviations or the number of random seeds; please add error bars or at least state the number of independent runs.
  8. [Supplement] Several derivations and model details are deferred to the Supplement (Secs. 2, 7, 9), but the supplement is not included with the manuscript; please ensure it is available to reviewers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the IDEAL-IO update is an explicitly stated approximation of the IDEAL gradient, and the main validations are external to the fitted MI objective.

full rationale

The derivation chain is self-contained and does not reduce to its inputs. Equation 8 is obtained from Equation 6 by explicitly dropping the ∂p/∂ψ ∂ψ/∂θ term under a stated empirical smallness assumption, and the density-model refit in Equation 7 is defined separately from the optical-parameter update. This is an approximation with a potentially nonzero bias, not a fitted parameter renamed as a prediction. The Sec. 4.3 lensless-imaging comparison uses PixelCNN MI both as the optimization objective and as the evaluation metric, so the claim that the PixelCNN-guided design is 'better' is internally consistent with the objective rather than independently established; however, the MI values are computed on held-out test patches, and the paper also provides external anchors: NMSE from a separately trained decoder in Sec. 4.1, and convergence to a known Fresnel-zone-plate design in Sec. 4.2. Self-citations to the authors' IDEAL papers [7,9] and to [17] provide background and expected-design context, but the load-bearing gradient equations and runtime/memory comparisons are derived and measured in this paper. The main fragility is the unquantified assertion in Sec. 3 that ∂ψ/∂θ is small, and the K=50 refit schedule in Sec. 4.3, which means ψ may not be at the optimum of the evaluated loss and the dropped term may be nonzero; this is a correctness risk, not a circularity, because Equation 8 is not equivalent to Equation 6 by construction. Overall circularity score: 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The method rests on a standard information-theoretic decomposition, the accuracy of the fitted density model, and two unquantified domain assumptions: the smallness of ∂ψ/∂θ and the validity of patch-based entropy estimation. Hyperparameters such as patch size and refit frequency are chosen by hand.

free parameters (3)
  • Density model parameters (Gaussian covariance or PixelCNN weights) = Refit each step (no single value)
    The central claim depends on the quality of the fitted pψ; these parameters are fitted to measurements at each step or every K steps.
  • Patch size = e.g., 36x36, 16x16
    Chosen by hand; patch-based entropy estimation is a central approximation and a key hyperparameter.
  • PixelCNN refit frequency K = 50
    Chosen to reduce runtime; affects the accuracy of the MI estimate during optimization.
assumptions (4)
  • standard math Mutual information decomposes as I(X;Y) = H(Y) - H(Y|X), with H(Y|X) computable analytically for Poisson and Gaussian noise.
    Invoked in Section 2, Eq. 2, and relies on the known noise model.
  • standard math The cross-entropy of a held-out set upper bounds the true entropy H(Y).
    Used in Section 2, Eqs. 3-4, to estimate H(Y) from a fitted density model.
  • ad hoc to paper The term ∂ψ/∂θ is small enough to ignore in the gradient.
    Section 3, first paragraph, asserts this as an empirical observation without quantification.
  • domain assumption Patch-based measurements sufficiently represent the full measurement distribution for MI estimation.
    Section 3 states 'we operate on measurement patches'; the Discussion acknowledges the trade-off between patch size and model fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Computationally Efficient Information-Driven Optical Design with Interchanging Optimization." pith.science (2026). https://pith.science/paper/BQY6EXPZ

@misc{pith2026250707789,
  author       = {Pith},
  title        = {Pith review of: Computationally Efficient Information-Driven Optical Design with Interchanging Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQY6EXPZ}},
  note         = {Machine review of arXiv:2507.07789}
}
read the original abstract

Recent work has demonstrated that imaging systems can be evaluated through the information content of their measurements alone, enabling application-agnostic optical design that avoids computational decoding challenges. Information-Driven Encoder Analysis Learning (IDEAL) was proposed to automate this process through gradient-based optimization. In this work, we study IDEAL across diverse imaging systems and find that it suffers from high memory usage, long runtimes, and a potentially mismatched objective function due to end-to-end differentiability requirements. We introduce IDEAL with Interchanging Optimization (IDEAL-IO), a method that decouples density estimation from optical parameter optimization by alternating between fitting models to current measurements and updating optical parameters using fixed models for information estimation. This approach reduces runtime and memory usage by up to 6x while enabling more expressive density models that guide optimization toward superior designs. We validate our method on diffractive optics, lensless imaging, and snapshot 3D microscopy applications, establishing information-theoretic optimization as a practical, scalable strategy for real-world imaging system design.

Figures

Figures reproduced from arXiv: 2507.07789 by the authors.

Figure 1
Figure 1. Comparison of Information-Driven Encoder Analysis Learning (IDEAL) and IDEAL with Interchanging Optimization (IDEAL-IO) optimization frameworks. (a) IDEAL jointly optimizes a differentiable density model and imaging system to maximize mutual information (MI) between noiseless and noisy measurements, I(X; Y). This coupling requires the entire pipeline to be differentiable with respect to the optical parameters, resul… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Scaling behavior and learned designs for diffractive optical element (DOE) opti￾mization with IDEAL and IDEAL-IO. (a) Optimization of a DOE for single-plane imaging using IDEAL (blue) and IDEAL-IO (ours, purple). Training loss curves are shown for both methods. (b) Runtime per optimization step vs. number of pixels in a patch used for mutual information estimation. IDEAL runtime increases rapidly with patch size; ID… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: IDEAL-IO improves lensless imaging design by using expressive mutual information estimators. (a) IDEAL-IO is used to optimize the PSF of a lenslet array by maximizing mutual information (MI) between scenes and noisy measurements. We compare two density models used in t…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. End-to-End Optimization of Incoherent Imaging for Classification Under Detector-Limited Readout

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    End-to-end phase-mask optimization improves classification under constrained detector readout by increasing class separability but yields no benefit under full readout, where a conventional lens approaches the mutual-...

  2. End-to-end meta-imagers: Information-theoretic objectives and generalized focusing optima

    physics.optics 2026-06 conditional novelty 6.0 of 10

    For intensity-only detectors, optimal incoherent transfer matrices for Shannon and Fisher objectives are permutation matrices, so each source must focus onto a distinct detector.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Computational imaging

    Ayush Bhandari, Achuta Kadambi, and Ramesh Raskar. Computational imaging. The MIT Press, Cambridge, Massachusetts, 2022

    work page 2022

  2. [2]

    Zin Lin, Charles Roques-Carmes, Raphaël Pestourie, Marin Solja ˇci´c, Arka Majumdar, and Steven G. Johnson. End-to-End Nanophotonic Inverse Design for Imaging and Polarimetry. 2020. arXiv: 2006.09145. 9

    work page Pith review arXiv 2020

  3. [3]

    End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging

    Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics , 37(4), 2018. Publisher: Association for Computing Machinery

    work page 2018

  4. [4]

    End-to-end complex lens design with differentiable ray tracing

    Qilin Sun, Congli Wang, Fu Qiang, Dun Xiong, and Heidrich Wolfgang. End-to-end complex lens design with differentiable ray tracing. ACM Transactions on Graphics (TOG), 40(4), 2021

    work page 2021

  5. [5]

    FourierNets enable the design of highly non-local optical encoders for computational imaging

    Diptodip Deb, Zhenfei Jiao, Alex B. Chen, Misha B. Ahrens, Kaspar Podgorski, and Srinivas C. Turaga. Programmable 3D snapshot microscopy with Fourier convolutional networks. i:1–27, 2021. arXiv: 2104.10611

    work page Pith review arXiv 2021

  6. [6]

    Image Quality Is Not All You Want: Task-Driven Lens Design for Image Classification, 2023

    Xinge Yang, Qiang Fu, Yunfeng Nie, and Wolfgang Heidrich. Image Quality Is Not All You Want: Task-Driven Lens Design for Image Classification, 2023. arXiv:2305.17185

    arXiv 2023

  7. [7]

    Information- driven design of imaging systems

    Henry Pinkard, Leyla Kabuli, Eric Markley, Tiffany Chien, Jiantao Jiao, and Laura Waller. Information- driven design of imaging systems. 2024

    work page 2024

  8. [8]

    Memory-efficient Learning for Large-scale Computational Imaging

    Michael Kellman, Kevin Zhang, Eric Markley, Jon Tamir, Emrah Bostan, Michael Lustig, and Laura Waller. Memory-Efficient Learning for Large-Scale Computational Imaging. IEEE Transactions on Computational Imaging, 6:1403–1414, March 2020. arXiv: 2003.05551

    work page Pith review arXiv 2020

Show all 20 references
  1. [9]

    Information-driven design of imaging systems

    Anonymous. Information-driven design of imaging systems. Under review at NeurIPS 2025, 2025

    2025

  2. [10]

    Conditional image generation with PixelCNN decoders

    Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with PixelCNN decoders. 2016. arXiv:1606.05328

    2016 arXiv

  3. [11]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  4. [12]

    Fourier diffuserscope: Single-shot 3d fourier light field microscopy with a diffuser.Optics Express, June 2020

    Fanglin Linda Liu, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller. Fourier diffuserscope: Single-shot 3d fourier light field microscopy with a diffuser.Optics Express, June 2020. arXiv: 2006.16343 Publisher: OSA

    2020 arXiv

  5. [13]

    FISTA-Net: Learning a Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging

    Jinxi Xiang, Yonggui Dong, and Yunjie Yang. FISTA-Net: Learning a Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging. IEEE Transactions on Medical Imaging, 40(5):1329–1339, 2021

    2021

  6. [14]

    Physics-based learned design for fourier diffuserscope

    Eric Markley, Fanglin Linda Liu, Michael Kellman, Nick Antipa, and Laura Waller. Physics-based learned design for fourier diffuserscope. In OSA Imaging and Applied Optics Congress 2021 (3D, COSI, DH, ISA, pcAOP), OSA Technical Digest, page CM6B.3. Optica Publishing Group, 2021

    2021

  7. [15]

    Hamming Embedding and Weak Geometric Consis- tency for Large Scale Image Search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Hamming Embedding and Weak Geometric Consis- tency for Large Scale Image Search. In Proceedings of the 10th European Conference on Computer Vision: Part I, ECCV ’08, pages 304–317. Springer-Verlag, 2008

    2008

  8. [16]

    Introduction to Fourier optics

    Joseph W Goodman. Introduction to Fourier optics. Roberts and Company, 2005

    2005

  9. [17]

    Designing lensless imaging systems to maximize information capture

    Leyla Kabuli, Henry Pinkard, Eric Markley, Clara S Hung, and Laura Waller. Designing lensless imaging systems to maximize information capture. In review, 2025

    2025

  10. [18]

    High-quality lensless imaging with a random multi-focal lenslet phase mask

    Leyla Kabuli, Gina Wu, and Laura Waller. High-quality lensless imaging with a random multi-focal lenslet phase mask. In Optica Imaging Congress (3D, COSI, DH, FLatOptics, IS, pcAOP), Technical Digest Series, Optica Publishing Group, page paper CW3B.2, 2023

    2023

  11. [19]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009

    2009

  12. [20]

    The mnist database of handwritten digit images for machine learning research

    Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012. 10

    2012

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.