pith. sign in

arxiv: 2606.24096 · v1 · pith:EGOP6HEHnew · submitted 2026-06-23 · 💻 cs.CV · cs.AI

Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation

Reeshad Khan , John Gauch This is my paper

Pith reviewed 2026-06-26 01:15 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords sensor co-designcolour filter arrayautonomous drivingsemantic segmentationdifferentiable imagingpoint spread functionadverse weather
0
0 comments X

The pith

Learning camera colour-filter-array weights improves autonomous driving segmentation accuracy more than optics redesign.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper uses a differentiable RAW-to-task pipeline to test which sensor parameters most affect dense prediction performance in driving scenes. It shows that optimizing the spectral weights of the colour filter array yields clear mIoU gains on both KITTI-360 and ACDC while redesigning the point-spread function produces a net loss, a result the authors tie to the data-processing inequality. The gains remain stable across fog, night, rain and snow conditions and do not depend on the choice of downstream model. A simple takeaway follows: tune the standard 2x2 CFA pattern and retain an identity PSF.

Core claim

Using a differentiable RAW-to-task pipeline, learning the spectral CFA weights improves mIoU by 0.017 on KITTI-360 and 0.023 on ACDC over a fixed camera; point-spread-function co-design instead lowers mIoU by 0.020 on KITTI-360, consistent with the data-processing inequality that limits task information recoverable by any downstream model.

What carries the argument

The differentiable RAW-to-task pipeline that jointly optimizes sensor parameters (CFA weights, PSF, noise) directly for the segmentation loss.

If this is right

  • CFA weight learning produces consistent mIoU gains on KITTI-360 and ACDC.
  • PSF co-design reduces performance on KITTI-360.
  • Noise co-optimization yields only marginal benefit.
  • CFA patterns larger than 2x2 lower accuracy because they remain confined to rank-three sRGB input.
  • The resulting gains are model-agnostic and hold under fog, night, rain and snow.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hardware designers could embed tunable spectral filters rather than fixed Bayer patterns for perception tasks.
  • The information bound implies that scaling model size alone cannot compensate for a poorly chosen sensor.
  • The same co-design method could be tested on other dense-prediction tasks such as depth estimation or object detection.

Load-bearing premise

The differentiable simulation accurately captures real sensor physics and data flow so that measured mIoU differences arise from the optimized parameters rather than modeling artifacts.

What would settle it

Deploying a physical camera whose 2x2 CFA weights match the learned pattern and measuring its mIoU on real driving sequences against a standard Bayer camera would confirm or refute the reported gains.

read the original abstract

Robust perception underpins autonomous driving, and most recent progress comes from scaling the model-larger backbones, foundation models, and cooperative multi-agent fusion. We pursue a complementary, upstream question: what should the camera itself measure? Using a differentiable RAW-to-task pipeline, we decompose which sensor degrees of freedom benefit dense prediction. Learning the spectral colour-filter-array (CFA) weights is the dominant lever, improving mIoU by +0.017 (KITTI-360) and +0.023 (ACDC) over a fixed camera. In contrast, point-spread-function (optics) co-design is net-negative (-0.020 mIoU on KITTI-360) - a consequence of the data-processing inequality, which also bounds the task information that any downstream model, however large or cooperative, can recover. Noise co-optimisation is marginal, and counter to intuition enlarging the CFA tile beyond 2x2 consistently hurts, as the filters are confined to the rank three sRGB input. Because the intervention is at the sensor, the gains are model-agnostic; we validate robustness on ACDC's fog, night, rain, and snow, and conclude with a simple recipe: learn the 2x2 CFA weights and keep an identity PSF.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces a differentiable RAW-to-task pipeline to co-optimize camera sensor parameters (CFA spectral weights, PSF, noise) for dense semantic segmentation in autonomous driving. It reports that learning 2x2 CFA weights yields mIoU gains of +0.017 on KITTI-360 and +0.023 on ACDC over a fixed Bayer camera, while PSF co-design is net-negative (-0.020 mIoU on KITTI-360), noise optimization is marginal, and larger CFA tiles hurt performance; the gains are attributed to the data-processing inequality and claimed to be model-agnostic, with a final recipe of learned 2x2 CFA plus identity PSF, validated on ACDC adverse conditions.

Significance. If the forward model is faithful, the work provides an empirical decomposition of sensor degrees of freedom showing that CFA spectral optimization is the dominant lever for task performance, complementary to model scaling, and bounded by information-theoretic limits. It supplies concrete quantitative deltas, a negative result on PSF, and robustness checks across weather conditions, which are strengths of the empirical approach.

major comments (1)
  1. [Abstract and pipeline description] The central claim that mIoU differences can be decomposed across sensor DOFs (CFA dominant, PSF harmful) and attributed to the data-processing inequality rests on the differentiable pipeline faithfully modeling real sensor physics (scene radiance → spectral filtering → PSF convolution → noise → RAW → task network). No validation of spectral responses, diffraction modeling, or demosaicing against physical hardware is described, raising the risk that observed deltas (+0.017/+0.023 vs -0.020) arise from simulation artifacts rather than true sensor-task interactions. This assumption is load-bearing for the abstract's headline results and the recommendation to learn 2x2 CFA weights.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on pipeline fidelity. We address the concern directly below and outline planned revisions to clarify the simulation scope of the work.

read point-by-point responses

  1. Referee: [Abstract and pipeline description] The central claim that mIoU differences can be decomposed across sensor DOFs (CFA dominant, PSF harmful) and attributed to the data-processing inequality rests on the differentiable pipeline faithfully modeling real sensor physics (scene radiance → spectral filtering → PSF convolution → noise → RAW → task network). No validation of spectral responses, diffraction modeling, or demosaicing against physical hardware is described, raising the risk that observed deltas (+0.017/+0.023 vs -0.020) arise from simulation artifacts rather than true sensor-task interactions. This assumption is load-bearing for the abstract's headline results and the recommendation to learn 2x2 CFA weights.

    Authors: We agree that no hardware validation of spectral responses, diffraction, or demosaicing is described. The work is a controlled simulation study that holds the forward model fixed while varying only the optimized sensor parameters; all reported deltas are therefore relative differences within this consistent simulator rather than claims of absolute physical accuracy. The data-processing inequality argument is applied strictly inside the model. To address the concern we will (1) revise the abstract to state that results are obtained in simulation, (2) add an explicit limitations section discussing the gap between the differentiable pipeline and real sensor physics, and (3) qualify the final recipe as a simulation-derived recommendation pending hardware calibration. These changes bound the claims without altering the empirical decomposition of sensor DOFs. revision: yes

Circularity Check

0 steps flagged

Empirical optimization results do not reduce to self-referential definitions or fitted inputs

full rationale

The paper reports mIoU gains from optimizing CFA weights (+0.017/+0.023) and PSF penalties (-0.020) via a differentiable RAW-to-task pipeline on KITTI-360 and ACDC. These quantities are presented as experimental outcomes of gradient-based search rather than quantities that equal their inputs by the paper's own equations. The data-processing inequality is invoked as a standard information-theoretic bound, not a self-citation. No self-definitional steps, fitted-input predictions, or ansatz smuggling appear in the provided abstract or claims; the central decomposition rests on external dataset benchmarks and is therefore self-contained.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim depends on learned CFA weights as the primary free parameter and on the domain assumption that the differentiable pipeline accurately captures sensor-to-task information flow; no new entities are postulated.

free parameters (1)
  • CFA spectral weights
    The 2x2 color filter weights are optimized end-to-end to maximize downstream mIoU; these are the dominant fitted parameters driving the reported gains.
axioms (1)
  • domain assumption The RAW-to-task differentiable pipeline accurately models real sensor physics and information flow.
    Invoked when the abstract states the pipeline is used to decompose which sensor degrees of freedom benefit the task.

pith-pipeline@v0.9.1-grok · 5759 in / 1312 out tokens · 28542 ms · 2026-06-26T01:15:26.626587+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    In: ICCV

    Baek, S.H., Ikoma, H., Jeon, D.S., Li, Y., Heidrich, W., Wetzstein, G., Kim, M.H.: Single-shot hyperspectral-depth imaging with learned diffractive optics. In: ICCV. pp. 2651–2660 (2021) 7

    2021

  2. [2]

    In: ICCV

    Buckler, M., Jayasuriya, S., Sampson, A.: Reconfiguring the imaging pipeline for computer vision. In: ICCV. pp. 975–984 (2017) 8

    2017

  3. [3]

    In: NeurIPS (2016) 8

    Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: NeurIPS (2016) 8

    2016

  4. [4]

    In: ICCV

    Chang, J., Wetzstein, G.: Deep optics for monocular depth estimation and 3D object detection. In: ICCV. pp. 10193–10202 (2019) 7

    2019

  5. [5]

    In: CVPR

    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR. pp. 3291–3300 (2018) 8

    2018

  6. [6]

    In: ECCV (2018) 8

    Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018) 8

    2018

  7. [7]

    In: CVPR (2022) 8

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR (2022) 8

    2022

  8. [8]

    In: CVPR (2016) 8

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016) 8

    2016

  9. [9]

    In: CVPR

    Côté, G., Mannan, F., Thibault, S., Heide, F.: The differentiable lens: Compound lens search over glass surfaces and materials for object detection. In: CVPR. pp. 20803–20812 (2023) 7

    2023

  10. [10]

    arXiv preprint arXiv:2502.04719 (2025), https://arxiv.org/abs/2502.04719 1, 7

    Dai, J., Chen, L., Yang, X., Hu, Y., Gu, J., Xue, T.: Tolerance-aware deep optics. arXiv preprint arXiv:2502.04719 (2025), https://arxiv.org/abs/2502.04719 1, 7

    work page arXiv 2025

  11. [11]

    ACM TOG 40(3) (2021) 8

    Diamond, S., Sitzmann, V., Julca-Aguilar, F., Boyd, S., Wetzstein, G., Heide, F.: Dirty pixels: Towards end-to-end image processing and perception. ACM TOG 40(3) (2021) 8

    2021

  12. [12]

    In: CVPR (2012) 8

    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the KITTI vision benchmark suite. In: CVPR (2012) 8

    2012

  13. [13]

    ACM TOG 35(6) (2016) 8

    Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM TOG 35(6) (2016) 8

    2016

  14. [14]

    Henz, B., Gastal, E.S.L., Oliveira, M.M.: Deep joint design of color filter arrays and demosaicing. Comput. Graph. Forum 37(2), 389–399 (2018) 8

    2018

  15. [15]

    arXiv preprint arXiv:2211.11257 (2024), https://arxiv.org/ abs/2211.11257 1, 8

    Jiang, Q., Shi, H., Gao, S., Zhang, J., Yang, K., Sun, L., Ni, H., Wang, K.: Com-putational imaging for machine perception: Transferring semantic segmentation beyond aberrations. arXiv preprint arXiv:2211.11257 (2024), https://arxiv.org/ abs/2211.11257 1, 8

    work page arXiv 2024

  16. [16]

    IEEE TPAMI (2022) 1, 8

    Liao, Y., Xie, J., Geiger, A.: KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. IEEE TPAMI (2022) 1, 8

    2022

  17. [17]

    In: CVPR (2015) 8 16 F

    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015) 8 16 F. Author et al

    2015

  18. [18]

    In: CVPR

    Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: CVPR. pp. 1375–1385 (2020) 7

    2020

  19. [19]

    In: ICCV (2017) 8

    Neuhold, G., Ollmann, T., Rota Bulò, S., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: ICCV (2017) 8

    2017

  20. [20]

    arXiv preprint arXiv:2412.14603 (2024) 1, 7

    Ren, Z., Zhou, J., Zhang, W., Yan, J., Chen, B., Feng, H., Chen, S.: Successive op-timization of optics and post-processing with differentiable coherent PSF operator and field information. arXiv preprint arXiv:2412.14603 (2024) 1, 7

    work page arXiv 2024

  21. [21]

    IJCV 126(9) (2018) 8

    Sakaridis, C., Dai, D., Van Gool, L.: Semantic foggy scene understanding with synthetic data. IJCV 126(9) (2018) 8

    2018

  22. [22]

    In: ICCV (2019) 8

    Sakaridis, C., Dai, D., Van Gool, L.: Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In: ICCV (2019) 8

    2019

  23. [23]

    In: ICCV (2021) 1, 8

    Sakaridis, C., Dai, D., Van Gool, L.: ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding. In: ICCV (2021) 1, 8

    2021

  24. [24]

    Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

    Shah, I.A., Li, J., Delaney, E., Ward, E., Glavin, M., Jones, E., Deegan, B.: Learnable quantum efficiency filters for urban hyperspectral segmentation. arXiv preprint arXiv:2603.26528 (2026) 2, 6, 8, 13

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    arXiv preprint arXiv:2508.19905 (2025) 2, 6, 8, 13

    Shah, I.A., Li, J., George, R., Brophy, T., Ward, E., Glavin, M., Jones, E., Deegan, B.: Hyperspectral sensors and autonomous driving: Technologies, limitations, and opportunities. arXiv preprint arXiv:2508.19905 (2025) 2, 6, 8, 13

    work page arXiv 2025

  26. [26]

    ACM TOG 37(4) (2018) 7

    Sitzmann, V., Diamond, S., Peng, Y., Dun, X., Boyd, S., Heidrich, W., Heide, F., Wetzstein, G.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM TOG 37(4) (2018) 7

    2018

  27. [27]

    In: ICCV (2021) 8

    Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for seman-tic segmentation. In: ICCV (2021) 8

    2021

  28. [28]

    In: NeurIPS (2021) 1, 8

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: Simple and efficient design for semantic segmentation with transformers. In: NeurIPS (2021) 1, 8

    2021

  29. [29]

    Nature Communications 15(1), 6572 (2024)

    Yang, X., Fu, Q., Heidrich, W.: Curriculum learning for ab initio deep learned refractive optics. Nature Communications 15(1), 6572 (2024). https://doi.org/ 10.1038/s41467-024-50835-7 1, 3, 7

    work page doi:10.1038/s41467-024-50835-7 2024

  30. [30]

    arXiv preprint arXiv:2305.17185 (2023) 1, 6, 8, 12

    Yang, X., Fu, Q., Nie, Y., Heidrich, W.: Image quality is not all you want: Task-driven lens design for image classification. arXiv preprint arXiv:2305.17185 (2023) 1, 6, 8, 12

    work page arXiv 2023

  31. [31]

    In: CVPR (2020) 8

    Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: BDD100K: A diverse driving dataset for heterogeneous multitask learning. In: CVPR (2020) 8

    2020

  32. [32]

    In: CVPR (2017) 8

    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017) 8

    2017