Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Beyond Calibration: Physically Informed Learning for Raw-to-Raw Mapping

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 2.7K-parameter network that simulates a color checker under measured light outperforms million-parameter camera-mapping models.

desk verdict Novel lightweight physically-informed raw-to-raw architecture with honest scope, but the SotA comparison is too uncontrolled to support the headline claim. read the letter →

arxiv 2506.08650 v2 pith:D3MQ3CJB submitted 2025-06-10 cs.CV

Peter Grönquist , Stepan Tulyakov , Dengxin Dai This is my paper
classification cs.CV
keywords raw-to-rawmappingilluminationestimationspectralpowerdistributionneuralphysicalmodelcolorconstancymulti-cameraconsistencyunpairedtrainingimagesignalprocessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the hard problem of mapping one camera's raw colors to another camera's can be solved with physics instead of a big black-box network. Its Neural Physical Model (NPM) estimates the scene's illumination, simulates how a standard color checker would look to each camera under that illumination, and then solves for a 3x3 color transform that maps source raw values to target raw values. Because the transform is recomputed for every illumination, the mapping adapts to changing light, something fixed calibration matrices cannot do. The claimed payoff is large: on the NUS-8 dataset (Nikon to Canon) the method reaches a CIEDE2000 error of 1.76 versus 2.53 for Rawformer and 5.95 for a semi-supervised approach, using only 2.7K parameters. The method also supports training without paired images when an auxiliary illumination sensor is available, which would let manufacturers add new cameras without recapturing data.

What carries the argument

The load-bearing object is the Neural Physical Model (NPM): a network that simulates raw ColorChecker images under a given illumination SPD through the physical image formation equation $\hat{I}_x = S R_x L$. It carries the argument because the illumination-adaptive $3\times3$ raw-to-raw transform is estimated as a least-squares fit from these simulated checker images alone, so the network's only job is to make the simulation accurate enough that the fitted transform transfers to real images. The NPM is trained with a simulation loss comparing simulated to measured checker raw values, optionally a matching loss in XYZ space, and can be initialized from physical calibration measurements.

What would settle it

Capture a scene with two distinct illuminants (e.g., daylight from a window plus a warm lamp) with a ColorChecker visible under each light, run the NPM raw-to-raw mapping, and measure CIEDE2000 between transformed source and target patches per illuminant; if both cannot be aligned simultaneously, the single-SPD premise is falsified.

Watch

Extended reading notes

Core claim

The central claim is that an illumination-adaptive raw-to-raw transformation can be derived from a small physically-parameterized simulation rather than learned as a black-box image translation. The NPM takes an estimated spectral power distribution of the illumination and generates synthetic raw ColorChecker images for both source and target cameras using the image formation model $\hat{I}_x = S R_x L$, where $S$ is a learnable camera spectral sensitivity matrix, $R_x$ a learnable diagonal reflectance matrix for checker patch $x$, and $L$ the estimated SPD. From these simulated checker images the model fits a $3\times3$ linear transform by least squares, normalizing by the green channel of the Neutral 8 patch. That transform is then applied to the entire raw source image. The paper reports that this inexpensive, physically grounded procedure outperforms recent learning-based methods on the NUS-8 dataset and improves over an illumination-agnostic baseline on the BeyondRGB dataset, with the best results obtained when illumination comes from a spectrometer rather than color constancy from RGB.

Load-bearing premise

That every scene is lit by one uniform light whose full spectrum can be captured by a single estimate; in mixed-light scenes, the one estimated spectrum cannot describe the image and the resulting single 3x3 transform cannot align the colors.

Editorial extensions

If this is right

  • Raw-to-raw conversion becomes cheap enough for on-device use: 2.7K parameters instead of 26.1M, with a 3x3 matrix application per image.
  • Adding a new camera to a product line does not require re-capturing paired data: with an auxiliary illumination sensor, unpaired training suffices.
  • Performance scales with illumination measurement quality: moving from grey-world to C5 to a 36-channel spectrometer steadily lowers color error, so better sensors directly improve mapping.
  • The simulation approach is not tied to the 3x3 transform; the paper states it can be combined with root-polynomial or other color correction transformations, and potentially local white-balance-based pixel-local transforms.
  • Multi-camera hybrid zoom and ISP compatibility can be standardized even when cameras have different fields of view and are not active simultaneously.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the network separates camera sensitivity from illumination, the same framework could be run in reverse to estimate camera spectral sensitivities from checker images under known illuminants, giving manufacturers a calibration path that does not require a spectrometer.
  • The single-global-SPD assumption is the real limit: mixed-illumination scenes fail visibly in the paper's own examples, so a natural extension is per-pixel or local white-balance maps instead of one SPD per scene.
  • The results suggest illumination estimation, not network capacity, is the bottleneck; a reasonable testable prediction is that sharing NPM's learned sensitivities across datasets would generalize if illumination is measured accurately.
  • The lab-only training data claim could be tested by training NPM purely on synthetic checker images generated from measured sensitivities and reflectances, then evaluating on real scenes without any paired real captures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces the Neural Physical Model (NPM), a lightweight physically-informed network that estimates the scene illumination (from an RGB image, multispectral sensor, or spectrometer), simulates ColorChecker images of the source and target cameras under that illumination via the linear image formation model, and then solves for an illumination-adaptive 3x3 raw-to-raw transform from the simulated checker values. The method supports paired and unpaired training, can be initialized with calibration data, and is evaluated on the NUS-8 and BeyondRGB datasets plus an internal Huawei Pura 70 dataset. The central claim is that NPM outperforms recent state-of-the-art raw-to-raw translation methods while being orders of magnitude smaller (2.7K vs. 26.1M parameters).

Significance. If substantiated, the NPM is a practical and physically interpretable alternative to heavyweight unpaired translation networks for multi-camera color consistency. The explicit modeling of illumination and the ability to train with unpaired data when an auxiliary sensor is available are genuine contributions, and the parameter count reduction is striking. The paper also makes a useful methodological point about the chromaticity diversity of raw-to-raw benchmarks. However, the strongest quantitative evidence for the headline claim—the comparison on NUS-8—relies on numbers taken from prior papers without a common protocol, which materially weakens the significance unless the comparison is put on a sound footing.

major comments (3)
  1. [§4.2, Table 2]
  2. [§4, Evaluation Metrics]
  3. [§3.1 and Supplementary Examples 4,5,8]
minor comments (4)
  1. [Eq. (5) and Eq. (6)]
  2. [§4.1, Table 1]
  3. [§4, Datasets]
  4. [§3.1, Figure 4]

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: test-time raw-to-raw transforms are computed from held-out illumination estimates, not fitted to test targets.

full rationale

The paper's derivation chain is self-contained. The NPM (Eq. 4) simulates ColorChecker raw values from the estimated SPD L and learned sensitivity and reflectance matrices S and R; the illumination-adaptive 3x3 transform is obtained by least-squares from these simulated checkers (Sec. 3.1). The simulation loss (Eq. 5) and matching loss (Eq. 6) supervise S, R, and M on the training split only, and the benchmarks (Tables 1-2) are computed on held-out test scenes: the NUS-8 split is explicitly given as 192/35/35 and the BeyondRGB split is taken from [12]. The evaluation metric (CIEDE2000 on ColorChecker patches) is a standard objective for this task; the matching loss is a similar XYZ-space proxy, but the reported test numbers are not produced by applying the learned transform to the same data used to fit it. The admitted reliance on reported numbers for Semi-Supervised [1] and Rawformer [22] is an experimental-protocol weakness, not a circular reduction, and the supplementary admission of failure under multi-illumination scenes is an empirical limitation rather than a definitional equivalence. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation appears in the derivation. Therefore no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the standard linear image-formation model, a uniform-illumination assumption, the sufficiency of a 3x3 transform on the 24-patch target, and the ability to recover an SPD from a color-constancy white point via a learned linear matrix. The model parameters (S, R, M) are all fitted to the checker data, so the physical 'initialization' is a starting point, not an independently measured quantity for the public datasets.

free parameters (4)
  • Camera spectral sensitivity matrices S^s, S^t = Not reported; learned per dataset (except Pura 70 initialization)
    N x 3 learnable matrices that map SPD to raw RGB. Fitted to minimize the simulation loss on real ColorChecker images; optionally initialized from measured sensitivities for the Huawei Pura 70 dataset.
  • ColorChecker reflectance matrices R_x = Initialized from BabelColor Macbeth reflectances; fine-tuned
    24 learnable diagonal matrices, one per patch, representing spectral reflectance. They are initialized with measured values and then updated during training.
  • Spectral recovery matrix M = Learned; not reported
    Linear map from color-constancy white point (3 channels, or multi-spectral channels) to an N-bin SPD. Fitted as part of the illumination estimation module; the recovered SPD is not independently verified.
  • Loss weight w and training hyperparameters = w not specified numerically; lr=0.01, 100 epochs, batch size 4
    Minor hyperparameters that affect the learned models but are not central to the method's claim.
assumptions (4)
  • standard math Camera response is a linear integral over wavelength of sensitivity, reflectance, and illumination SPD (Eq. 1).
    The physical image formation model underpins Eq. 4 and the entire approach; it is a standard assumption in color science.
  • domain assumption Scene illumination is uniform and fully specified by one SPD L(lambda).
    Eq. 1 uses a single L(lambda) for all pixels; the paper's own supplement shows failures in multi-illumination scenes.
  • domain assumption A 3x3 linear transform computed from 24 checker patches is sufficient for raw-to-raw mapping.
    The paper adopts the linear transform because it 'has been shown to perform well compared to more complex transformations [19]'. It is used in all experiments and is not evaluated against full-image scene content.
  • ad hoc to paper White point from color constancy can be converted to an SPD through a single learned linear matrix M.
    The mapping from 3-channel white point to an N-bin SPD is severely underdetermined; the paper posits a linear recovery matrix and learns it jointly, but never validates the recovered SPD against a spectrometer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Calibration: Physically Informed Learning for Raw-to-Raw Mapping." pith.science (2026). https://pith.science/paper/D3MQ3CJB

@misc{pith2026250608650,
  author       = {Pith},
  title        = {Pith review of: Beyond Calibration: Physically Informed Learning for Raw-to-Raw Mapping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3MQ3CJB}},
  note         = {Machine review of arXiv:2506.08650}
}
read the original abstract

Achieving consistent color reproduction across multiple cameras is essential for seamless image fusion and Image Processing Pipeline (ISP) compatibility in modern devices, but it is a challenging task due to variations in sensors and optics. Existing raw-to-raw conversion methods face limitations such as poor adaptability to changing illumination, high computational costs, or impractical requirements such as simultaneous camera operation and overlapping fields-of-view. We introduce the Neural Physical Model (NPM), a lightweight, physically-informed approach that simulates raw images under specified illumination to estimate transformations between devices. The NPM effectively adapts to varying illumination conditions, can be initialized with physical measurements, and supports training with or without paired data. Experiments on public datasets like NUS and BeyondRGB demonstrate that NPM outperforms recent state-of-the-art methods, providing robust chromatic consistency across different sensors and optical systems.

Figures

Figures reproduced from arXiv: 2506.08650 by the authors.

Figure 1
Figure 1. Hybrid zoom with our physically informed raw-to-raw method. The top row shows the raw RGB-space images by different camera sensors. The bottom row displays images transformed using our proposed physically informed raw-to-raw method to match the main camera’s color space, followed by a simplified color post-processing pipeline. Our method effectively reduces color discrepancies across images captured by the wide, mai… view at source ↗
Figure 3
Figure 3. The Illumination Estimation module estimates the spectral power distribution (SPD) of the scene illumination using either an RGB source camera, or an auxiliary multi-spectral camera or spectrometer. For RGB and multi-spectral cam￾eras, a color constancy method is first applied to the images to determine the illumination’s white point, which is then converted to an SPD using a learnable spectral recovery matrix shown… view at source ↗
Figure 5
Figure 5. xyY distribution of the NUS-8 and BeyondRGB dataset [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 32 canonical work pages

  1. [19]

    Raw- to-raw: Mapping between image sensor color responses

    Rang Nguyen, Dilip K Prasad, and Michael S Brown. Raw- to-raw: Mapping between image sensor color responses. In CVPR, pages 3398–3405, 2014. 2, 4, 6, 7

    work page 2014

  2. [1]

    Semi-supervised raw-to-raw mapping

    Mahmoud Afifi and Abdullah Abuolaim. Semi-supervised raw-to-raw mapping. 2021. 2, 3, 4, 5, 6, 7

    work page 2021

  3. [2]

    Barron, Chloe LeGendre, Yun- Ta Tsai, and Francois Bleibel

    Mahmoud Afifi, Jonathan T. Barron, Chloe LeGendre, Yun- Ta Tsai, and Francois Bleibel. Cross-camera convolutional color constancy. InICCV, 2021. 3, 6, 7

    work page 2021

  4. [3]

    Robust photometric alignment for asymmetric camera system

    N Anirudth, B H Pawan Prasad, Anurag Jain, and Venkat Peddigari. Robust photometric alignment for asymmetric camera system. In2018 IEEE International Conference on Consumer Electronics (ICCE), pages 1–4, 2018. 3

    work page 2018

  5. [4]

    Improvements to gamut mapping colour constancy algorithms

    Kobus Barnard. Improvements to gamut mapping colour constancy algorithms. InECCV, pages 390–403. Springer,

    work page

  6. [5]

    Jonathan T. Barron. Convolutional color constancy. InICCV, pages 379–387, December 2015. 3

    work page 2015

  7. [6]

    Color constancy with spatio-spectral statistics.TPAMI, 34(8):1509–1519, 2011

    Ayan Chakrabarti, Keigo Hirakawa, and Todd Zickler. Color constancy with spatio-spectral statistics.TPAMI, 34(8):1509–1519, 2011. 3

    work page 2011

  8. [7]

    Cheng, D

    D. Cheng, D. K. Prasad, and M. S. Brown. Illuminant es- timation for color constancy: why spatial-domain methods work and the role of the color distribution.Journal of the Optical Society of America A, 31(5):1049–1058, 2014. 2, 3, 5

    work page 2014

Show all 33 references
  1. [8]

    Colour: A python colour science pack- age.https://www.colour-science.org/, 2022

    Colour Developers. Colour: A python colour science pack- age.https://www.colour-science.org/, 2022. 6

    2022

  2. [9]

    Finlayson, Han Gong, and Robert B

    Graham D. Finlayson, Han Gong, and Robert B. Fisher. Color homography color correction, 2016. 4

    2016

  3. [10]

    Finlayson, Michal Mackiewicz, and Anya Hurl- bert

    Graham D. Finlayson, Michal Mackiewicz, and Anya Hurl- bert. Color correction using root-polynomial regression. IEEE Transactions on Image Processing, 24(5):1460–1470,

  4. [11]

    Color constancy

    David Forsyth. Color constancy. In Bernd J ¨ahne, Horst Haussecker, and Peter Geissler, editors,Handbook of Com- puter Vision and Applications, pages 567–589. Academic Press, 2002. 3

    2002

  5. [12]

    Beyond rgb: A real world dataset for multispectral imaging in mobile devices

    Ortal Glatt, Yotam Ater, Woo-Shik Kim, Shira Werman, Oded Berby, Yael Zini, Shay Zelinger, Sangyoon Lee, Hee- jin Choi, and Evgeny Soloveichik. Beyond rgb: A real world dataset for multispectral imaging in mobile devices. InWACV, pages 4344–4354, January 2024. 2, 5, 6

    2024

  6. [13]

    Fc4: Fully convolutional color constancy with confidence-weighted pooling

    Yuanming Hu, Baoyuan Wang, and Stephen Lin. Fc4: Fully convolutional color constancy with confidence-weighted pooling. InCVPR, pages 4085–4094, July 2017. 3

    2017

  7. [14]

    Ensuring color consistency across multiple cameras

    Adrian Ilie and Greg Welch. Ensuring color consistency across multiple cameras. InICCV, volume 2, pages 1268–

  8. [15]

    Automatic color calibration for large camera arrays

    Neel Joshi, Bennett Wilburn, Vaibhav Vaish, Marc Levoy Levoy, and Mark Horowitz. Automatic color calibration for large camera arrays. 2005. Technical Report. 2

    2005

  9. [16]

    Improving color reproduction accuracy on cameras

    Hakki Can Karaimer and Michael S Brown. Improving color reproduction accuracy on cameras. InCVPR, pages 6440– 6449, 2018. 5

    2018

  10. [17]

    Single pixel spectral color constancy.IJCV, 132(2):287–299,

    Samu Koskinen, Erman Acar, and Joni-Kristian K¨am¨ar¨ainen. Single pixel spectral color constancy.IJCV, 132(2):287–299,

  11. [18]

    Photometric alignment for surround view camera system

    Yucheng Liu and Buyue Zhang. Photometric alignment for surround view camera system. In2014 IEEE International Conference on Image Processing (ICIP), pages 1827–1831,

  12. [20]

    Contrastive learning for unpaired image-to-image translation

    Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. InComputer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part IX 16, pages 319–345. Springer, 2020. 2, 3

    2020

  13. [21]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, ...

    2019

  14. [22]

    Rawformer: Unpaired raw-to-raw translation for learnable camera isps

    Georgy Perevozchikov, Nancy Mehta, Mahmoud Afifi, and Radu Timofte. Rawformer: Unpaired raw-to-raw translation for learnable camera isps. 2024. 2, 3, 4, 6, 7

    2024

  15. [23]

    Dataset and pipeline for multi-view light-field video

    Neus Sabater, Guillaume Boisson, Benoit Vandame, Paul Kerbiriou, Frederic Babon, Matthieu Hog, Remy Gendrot, Tristan Langlois, Olivier Bureller, Arno Schubert, et al. Dataset and pipeline for multi-view light-field video. In CVPRW, pages 30–40, 2017. 2

    2017

  16. [24]

    The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations

    Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application, 30(1):21–30, 2005. 5

    2005

  17. [25]

    Spectral sensitiv- ity estimation without a camera

    Grigory Solomatov and Derya Akkaynak. Spectral sensitiv- ity estimation without a camera. InICCP, pages 1–12. IEEE,

  18. [26]

    Weakly-supervised stitching net- work for real-world panoramic image generation

    Dae-Young Song, Geonsoo Lee, HeeKyung Lee, Gi-Mun Um, and Donghyeon Cho. Weakly-supervised stitching net- work for real-world panoramic image generation. InECCV, pages 54–71. Springer, 2022. 2, 3

    2022

  19. [27]

    Measure- ment and estimation of spectral sensitivity functions for mo- bile phone cameras.Sensors, 21(15):4985, 2021

    Shoji Tominaga, Shogo Nishi, and Ryo Ohtera. Measure- ment and estimation of spectral sensitivity functions for mo- bile phone cameras.Sensors, 21(15):4985, 2021. 2

    2021

  20. [28]

    High performance imaging using large camera arrays

    Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Tal- vala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy. High performance imaging using large camera arrays. InSIGGRAPH, pages 765–776. 2005. 2

    2005

  21. [29]

    Dual-camera smooth zoom on mobile phones.CoRR, 2024

    Renlong Wu, Zhilu Zhang, Yu Yang, and Wangmeng Zuo. Dual-camera smooth zoom on mobile phones.CoRR, 2024. 1

    2024

  22. [30]

    Cascading convolutional color con- stancy

    Huanglin Yu, Ke Chen, Kaiqi Wang, Yanlin Qian, Zhaoxi- ang Zhang, and Kui Jia. Cascading convolutional color con- stancy. InProceedings of the AAAI Conference on Artifi- cial Intelligence, volume 34, pages 10694–10701, February

  23. [31]

    Unpaired image- to-image translation using adversarial consistency loss

    Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image- to-image translation using adversarial consistency loss. In ECCV, pages 800–815. Springer, 2020. 2, 3

    2020

  24. [32]

    Unpaired image-to-image translation using cycle- consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. InICCV, pages 2223–2232,

  25. [33]

    Supplementary Material 6.1. Examples For visual qualitative evaluation, we present eight test ex- amples comparing images captured directly from the latest available flagship phones and compare them with the out- put of our physically inspired network. The outputs from the wid...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.