{"id":"3bb9daae-a69a-42d6-99ee-8491f2856e76","arxiv_id":"1908.06126","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 24-scene laboratory dataset with 18 illumination scenarios per scene, spectral ground truth for lights and surfaces, and pixel masks for benchmarking colour constancy and segmentation.","lead":"This paper presents a new laboratory dataset for colour research: 24 scenes, each captured under 18 different multi-light illumination setups, together with spectral measurements of lights, surfaces, and a camera, plus pixel-level masks of uniformly coloured regions. A generalist might read it because the dataset targets a known unsolved problem, estimating the colours of multiple light sources, and could serve as a shared benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spectral ground truth is not independently validated: the only consistency check is one 2HAL image, and agreement is shown after fitting a colour homography to the same data.","rationale":"The reader's weakest assumption was the accuracy of the digitized camera spectral sensitivities. That is part of the concern, but the more general and more load-bearing issue is that the paper's only validation of the complete spectral pipeline is not independent: it uses one illumination scenario, reports no quantitative error, and applies a homography fitted to the same data before declaring consistency. Because the homography can compensate for systematic errors in the camera sensitivities, illuminant spectra, or reflectance spectra, the visual agreement in Fig. 5b cannot be taken as evidence that the spectral measurements are correct. The dataset itself is still likely useful as a resource, and the concerns are addressable with additional validation, so the reader's CONDITIONAL verdict remains appropriate; no change to the verdict is needed. My only partial disagreement is that I would locate the primary weakness not just in the digitization step but in the circular validation protocol that was intended to support the ground-truth claim.","tokens_in":5740,"tokens_out":3880,"duration_ms":42666,"concrete_test":"Hold-out spectral validation: fit the colour homography (or any linear correction) on half of the colour-chart patches from the 2HAL image, then predict the remaining half and all patches in at least one LED-mixture image without refitting; report per-patch chromaticity error (e.g., mean angular error) before and after correction. If held-out errors are comparable to the pre-correction scatter in Fig. 5a, the homography is absorbing real spectral error and the ground-truth claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dataset supplies spectral ground truth adequate for benchmarking multiple-illuminant colour constancy. The load-bearing step is the consistency of the measured spectra (illuminant and reflectance) with the camera-specific image data. Section 6 validates this on a single colour-chart image under 2HAL only, and the protocol has two weaknesses. First, the predicted and observed chromaticities visibly disagree before any correction (Fig. 5a), and the paper attributes the mismatch to 'linear miscalculations in camera spectral sensitivity and lighting setup' without quantifying it. Second, the reported agreement is obtained by fitting a colour homography to the very same patch chromaticities (Fig. 5b); because any homography can absorb linear errors in the camera sensitivities, illuminant spectra, or reflectance spectra, the post-fit visual correspondence is not an independent check of the spectral ground truth. No error metric, no held-out patches, and no check under any of the 17 LED-mixture illuminations are reported. Since the camera responses were not measured by the authors but digitized from a figure in [24], the absolute accuracy of the predicted chromaticities is therefore unverified. This does not invalidate the dataset as a resource, but it does mean the 'spectral ground truth' component has not been demonstrated to the accuracy implied by the benchmark claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Multiple Light Source (MLS) dataset: 24 multi-object scenes, each recorded under 18 illumination configurations involving halogen lamps, a desk lamp, and controlled red/green/blue LED contributions. The release includes full-resolution linear raw images and demosaiced color images, quarter-resolution previews, pixel-wise annotation masks of uniformly coloured surfaces, measured illuminant emission spectra, measured object reflectance spectra, and camera spectral sensitivities digitized from a published figure. The authors position the dataset as a complete benchmark for multiple-illuminant colour constancy, spectral colour modelling, and colour-based segmentation, and they report in Section 6 a consistency check between spectral predictions and observed chromaticities for one colour-chart image under 2HAL illumination.","tokens_in":5980,"tokens_out":2953,"duration_ms":29748,"significance":"If the spectral ground truth is accurate at the implied level, the dataset fills a genuine gap: most existing laboratory datasets do not simultaneously provide multiple light sources, measured illuminant spectra, object reflectance spectra, and linear raw camera images. The public release of 24 scenes with three image formats, pixel-wise masks, and a detailed spectral measurement protocol is a valuable resource for the colour-constancy community. The authors are also transparent about their main limitation, noting that they did not measure the camera responses themselves. However, the consistency demonstration in Section 6 does not currently establish the accuracy needed to call the spectral data 'ground truth' for benchmarking; the confidence in the central claim would be materially improved by a stronger independent validation.","major_comments":[{"comment":"The consistency check is performed on a single colour-chart image under the 2HAL illumination condition only. The initial agreement between measured and observed chromaticities is visibly poor for several patches, and no quantitative error metric is reported. With 18 illumination configurations in the dataset, a one-image, one-illuminant demonstration does not support the benchmark accuracy claimed for the full dataset.","section":"Section 6, Fig. 5a"},{"comment":"The improved agreement shown in Fig. 5b is obtained by fitting a planar colour homography to the same patch chromaticities that are subsequently compared. Because a homography can absorb linear discrepancies in camera sensitivity, illuminant spectra, or reflectance spectra, the post-fit visual correspondence is not an independent validation of the measured spectral data. A validation using held-out patches or, better, predicting chromaticities for a different illumination scenario from the separately measured LED spectra would be needed to support the ground-truth claim.","section":"Section 6, Fig. 5b"},{"comment":"The camera spectral sensitivities are digitized from a figure in Ref. [24] rather than measured by the authors, and Section 6 attributes the remaining mismatch to 'linear miscalculations in camera spectral sensitivity and lighting setup'. This makes the absolute accuracy of the predicted chromaticities unquantified. Please provide an uncertainty estimate for the digitized camera curves, or explicitly qualify the claim that the spectral data constitute camera-specific ground truth, since the predictions depend on this input.","section":"Section 2, camera spectral responses"}],"minor_comments":[{"comment":"The word 'usefull' should be 'useful'.","section":"Section 1"},{"comment":"The lens model 'Canon EF 2470 mm' is likely meant to be 'Canon EF 24-70 mm'.","section":"Section 2"},{"comment":"The sentence 'the colour chart was glance, not matte' should read 'glossy, not matte'.","section":"Section 6"},{"comment":"The marker colours are difficult to distinguish in the figure; enlarging the plots or adding text labels would improve readability.","section":"Figure 5"},{"comment":"The word 'chomaticities' is misspelled in several places; it should be 'chromaticities'.","section":"Sections 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about the camera-sensitivity limitation, and the dataset is likely to be used by the community regardless of the consistency check. The main risk is overclaiming 'spectral ground truth' without an independent validation. I do not see grounds for rejection, because the dataset construction is sensible and the resource is valuable, but the validation in Section 6 needs to be strengthened or the claims moderated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MLS dataset is a real addition to the colour constancy toolbox: 24 scenes under 18 illumination conditions, with measured illuminant and reflectance spectra from a calibrated spectrometer, pixel-wise surface masks, and linear raw images. The combination of multi-light scenes with simultaneous spectral measurements of lights, surfaces, and camera sensitivities is new among the cited laboratory datasets, and the release on GitHub makes it immediately usable. The experimental setup section is detailed and the measurement protocol for the spectra sounds careful.\n\nThe soft spot is exactly what the stress-test flags. The camera spectral sensitivities are not measured by the authors but digitized with Web Plot Digitizer from a figure in Baek et al. That alone would be acceptable if the downstream validation were independent, but it is not. Section 6 validates the spectra on a single colour-chart image under 2HAL, and the agreement that looks good in Fig. 5b is obtained after fitting a colour homography to the very same chromaticity coordinates used for the comparison. Any 3x3 colour transform can absorb linear errors in the camera sensitivity, illuminant, or reflectance spectra, so the post-fit visual correspondence tells you little about absolute accuracy. The paper is transparent about the mismatch before the homography, but it never reports a quantitative error, never holds out patches, and never checks any of the 17 LED-mixture illuminations. That leaves the central \"spectral ground truth\" claim under-supported.\n\nThis is not a fatal flaw for a dataset resource. The measured illuminant and reflectance spectra are likely fine on their own; the weakness is in the claimed link between those spectra and the image pixels. Users who need accurate absolute chromaticities can recalibrate the camera sensitivity themselves. But the paper's current benchmarking claim implies more accuracy than is demonstrated. There is also a smaller unaddressed issue: the pixel masks were built with help from an automatic segmentation algorithm, yet no validation of mask quality is reported, which matters for segmentation benchmarking and for any per-pixel spectral ground truth.\n\nWho should read it: anyone working on multi-illuminant colour constancy or colour-based segmentation who wants a public, well-documented dataset. It deserves peer review. The right outcome is acceptance after revision where the authors either measure the camera sensitivity directly or provide a held-out validation using several images and all illumination types, with a quantitative error metric.\n\nMy recommendation: engage with this paper, but insist the validation match the benchmark claim.","headline":"Genuinely useful multi-illuminant dataset, but its spectral ground truth is only weakly validated; worth publishing after the authors tighten the accuracy story.","tokens_in":6478,"tokens_out":1727,"would_cite":true,"duration_ms":20047,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents a collection of 24 scenes, each recorded under 18 multi-light illumination scenarios, with spectral measurements of camera, lights, and object surfaces plus pixel-by-pixel annotation masks.","keywords":["colour constancy","multiple illuminants","spectral ground truth","reflectance spectra","dataset","pixel-wise annotation","camera spectral sensitivity","benchmark"],"falsifier":"Take the same camera model, measure its spectral response directly with a monochromator, and recompute the chromaticities of the colour-chart patches under the two-halogen illumination from the supplied spectra; if the remeasured response moves predicted chromaticities outside the residual scatter shown in the paper's Figure 5, then the digitized camera response is the weak link in the ground-truth claim.","tokens_in":5557,"feed_emoji":"💡","tokens_out":6951,"duration_ms":64525,"temperature":0.7,"pith_summary":"This paper presents a laboratory dataset built for colour research: 24 scenes, each photographed under 18 lighting configurations that combine halogen, desk-lamp, and RGB-LED illumination. Alongside every image the authors publish the emission spectra of the sources, reflectance spectra of 122 surfaces, the camera's spectral sensitivities, and pixel-by-pixel masks of uniformly coloured regions. The aim is to give computational colour constancy and colour-based segmentation algorithms a benchmark with simultaneous spectral ground truth, which existing laboratory and uncontrolled datasets lack. If the spectral measurements are accurate, algorithm outputs can be compared against known per-scene illuminant and surface colours rather than against a single global white-point estimate.","feed_headline":"24 scenes, 18 lighting setups, full spectral ground truth","feed_subtitle":"Multiple-illuminant colour constancy gets a benchmark with known light spectra, surface reflectance, per-pixel masks.","key_machinery":"The carrying object is the dataset's calibration-and-annotation pipeline: a softbox stage with fixed halogen sources, a dimmable desk lamp, and a computer-controlled RGB LED strip; a camera recording linear RAW images; a miniature spectrometer with a reflectance standard for measuring source and surface spectra; and masks produced by an automatic linear-segmentation algorithm, manually merged into uniform-colour regions. The link between spectra and pixels is the Lambertian image-formation model, and the paper's consistency check uses the colour-homography relation to align observed and predicted chromaticities. The dataset itself is the mechanism: it is what makes the claimed ground truth available to other algorithms.","core_discovery":"The central claim is that this dataset supplies simultaneous spectral ground truth for a multi-light colour constancy benchmark. For each of 24 object scenes, the same scene is recorded under 18 illumination scenarios, and for each scenario the authors provide linear raw and colour images, the measured emission spectra of all active sources, the measured reflectance spectra of the objects, and manually refined pixel-wise masks that tie image regions to those reflectance spectra. The paper argues that this combination is the missing piece among published laboratory datasets, which offer ground truth for one or two illuminants but not the complete spectral description needed to evaluate multi-source illumination estimation, spectral colour models, and segmentation at once.","pith_inferences":["The fixed geometry across the 18 illumination states creates a natural self-supervised setup: train a network to predict reflectance from several images of the same scene under different lights, using the measured spectra as weak supervision.","If the digitized camera response is the main source of residual error, users can still use the dataset to estimate a colour homography between spectral predictions and images, turning the benchmark into a calibration procedure.","The varying distance and position of the light sources, not just their colour, could support estimating light-source geometry from a single image, a task the paper lists as motivation but does not evaluate.","Adding synthetic renders with the same measured spectra would give dense per-pixel ground truth, complementing the sparse surface masks of the real data."],"forward_implications":["Algorithms that estimate multiple illuminants can be scored against the measured spectra of the active sources in each of the 18 configurations, instead of a single scene-level white point.","Colour-based segmentation can be evaluated with the supplied pixel-wise masks, each region tied to a measured reflectance spectrum.","Low-parametric spectral models of colour transformation and multiple-reflection effects can be tested on linear images because camera sensitivities, source spectra, and surface reflectances are all provided.","Holding scene geometry fixed while varying LED colour and intensity isolates the effect of illumination on observed colours, a controlled comparison the paper positions as missing from earlier laboratory sets."],"supporting_citations":[{"why":"Supplies the camera spectral sensitivity curves that connect measured spectra to predicted image colours.","marker":"[24]"},{"why":"Provides the plot-digitizing tool used to extract those sensitivity curves from the published figure.","marker":"[25]"},{"why":"Produces the initial automatically segmented regions that were manually merged into the pixel-wise annotation masks.","marker":"[26]"},{"why":"Gives the colour-homography relation used in the consistency check between spectrally predicted and observed chromaticities.","marker":"[27]"}],"fun_headline_variants":["24 scenes, 18 lights, full spectra for colour research","Multi-light dataset: 24 scenes, 18 illuminants, full spectra","24 scenes, 18 illuminations, spectral ground truth","Benchmark for multi-illuminant colour with full spectral truth","Spectral ground truth for multi-light colour: 24 scenes, 18 setups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ground-truth chain rests on the camera's spectral sensitivity curves, which the authors took from an earlier paper's figure rather than measuring themselves; if those curves are inaccurate, the predicted colours will not match the images.","fun_headline_variants_meta":{"raw":{"variants":["24 scenes, 18 lights, full spectra for colour research","Multi-light dataset: 24 scenes, 18 illuminants, full spectra","24 scenes, 18 illuminations, spectral ground truth","Benchmark for multi-illuminant colour with full spectral truth","Spectral ground truth for multi-light colour: 24 scenes, 18 setups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3329,"prompt_tokens":779,"completion_tokens":2550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":2457}},"tokens_in":395,"tokens_out":2550,"duration_ms":16032,"temperature":1.0,"reasoning_tokens":2457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:54:48.859724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same camera model, measure its spectral response directly with a monochromator, and recompute the chromaticities of the colour-chart patches under the two-halogen illumination from the supplied spectra; if the remeasured response moves predicted chromaticities outside the residual scatter shown in the paper's Figure 5, then the digitized camera response is the weak link in the ground-truth claim.","supporting_citations":[{"cited_title":"Compact single-shot hyperspectral imaging using a prism,","cited_arxiv_id":null,"evidence_quote":"Supplies the camera spectral sensitivity curves that connect measured spectra to predicted image colours."},{"cited_title":"Webplotdigitizer","cited_arxiv_id":null,"evidence_quote":"Provides the plot-digitizing tool used to extract those sensitivity curves from the published figure."},{"cited_title":"Linear colour segmentation revisited,","cited_arxiv_id":null,"evidence_quote":"Produces the initial automatically segmented regions that were manually merged into the pixel-wise annotation masks."},{"cited_title":"Color homography: theory and applications,","cited_arxiv_id":null,"evidence_quote":"Gives the colour-homography relation used in the consistency check between spectrally predicted and observed chromaticities."}],"review_version":1}