{"id":"f4147226-603e-4991-aaa0-56cee323e5e0","arxiv_id":"2411.18025","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Authors built a pixel-aligned RGB-NIR stereo and LiDAR dataset from a mobile robot and two fusion methods that improve depth, detection, and structure-from-motion under hard lighting.","lead":"A robot with prism-based cameras that capture pixel-aligned color and near-infrared images plus LiDAR was used to build an 80,000-frame dataset across day, night, indoor, and outdoor scenes. Learned RGB-NIR fusion improves depth estimation, object detection, and 3D reconstruction from pretrained models without retraining them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pixel-aligned RGB-NIR claim rests on unmeasured sensor alignment; no spatial registration error is reported, so the dataset's core property and fusion gains attributed to it are unverified.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the hardware's pixel-aligned property is asserted but never measured. I agree this is the most fundamental issue because it underpins the dataset's unique value and the interpretation of all downstream improvements. Other concerns—such as pseudo-labeled detection ground truth and the choice of RMSE in the correlation-volume ablation—are secondary: the pseudo-labels are disclosed in the supplemental and affect only the object-detection evaluation, while the ablation choice is supported by a majority of metrics in Table 16 (Fusion-NIR is better on RMSE, δ1, δ2, δ3, though worse on MAE). The alignment concern, by contrast, could invalidate the central premise. The paper otherwise has real strengths: a substantial real dataset with LiDAR ground truth, a synthetic augmentation pipeline, and evaluations across three tasks. The depth evaluation uses sparse LiDAR points rather than the densified pseudo-depth, which is a point in its favor. The absence of an alignment measurement is a missing piece of evidence, not a demonstrated falsity, so the appropriate verdict remains CONDITIONAL as the reader concluded. No verdict change is needed; the paper should add the alignment verification before the pixel-aligned claim is taken at face value.","tokens_in":38954,"tokens_out":5453,"duration_ms":50833,"concrete_test":"Capture a checkerboard or dot-grid target that is visible in both RGB and NIR (e.g., illuminated by the 850 nm active light) with the JAI FS-1600D-10GE. Detect corresponding corners or centroids across the full field of view at several depths, compute per-point Euclidean distances between RGB and NIR detections, and report mean, RMS, and 95th percentile alignment error in pixels. If the error is subpixel (<0.5 px), the pixel-aligned claim is supported; if it exceeds 1–2 px, the claim needs qualification and the downstream results should be re-evaluated with a registered or corrected input.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the assertion in Section 3 (Calibration) that 'calibration between RGB and NIR images is unnecessary' for the JAI FS-1600D-10GE dichroic-prism camera. The paper never reports a measured spatial alignment error between the RGB and NIR sensors. The supplemental (Section 1.1.4) details stereo-pose and LiDAR calibration but contains no RGB-NIR alignment verification. This premise enters directly into the methods: image fusion (Eq. 3) combines I_V and I_NIR pixel-wise, and feature fusion (Eq. 7) builds correlation volumes under the assumption of exact spatial correspondence. If residual misregistration exists due to prism tolerances, lens distortion, or sensor mounting, the 'pixel-aligned' property—and the claimed advantage over pixel-misaligned datasets such as CS-Stereo—is not established. The paper's title, dataset contribution, and bypass of the registration chicken-and-egg problem all rest on this unquantified hardware assumption. The reported depth/detection gains could still be real, but their attribution to pixel-level alignment is unsupported without an alignment-error measurement or a sensitivity analysis to small misalignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a robotic imaging system with two prism-based RGB-NIR cameras, active NIR illumination, and LiDAR, plus a dataset of synchronized pixel-aligned RGB-NIR stereo images and LiDAR depth for indoor/outdoor day/night scenes. It also proposes two fusion methods: an image-level fusion that lets pretrained RGB models consume RGB-NIR data without finetuning, and a feature-level fusion that extends RAFT-Stereo with attentional RGB-NIR feature fusion and alternating correlation volumes. Experiments report improvements over RGB-only, NIR-only, and several RGB-NIR fusion baselines on stereo depth estimation, object detection, and structure-from-motion.","tokens_in":39214,"tokens_out":4039,"duration_ms":38838,"significance":"If the main claims hold, the dataset and hardware design would be a useful community resource: the paper is among few to provide stereo RGB and NIR pairs with LiDAR ground truth, and the image-fusion method has practical value because it enables off-the-shelf RGB-pretrained models to exploit NIR without retraining. The paper is also stronger than a pure dataset paper because it includes controlled comparisons against single-modality baselines and several established fusion methods, and it states limitations explicitly. However, the central 'pixel-aligned' property is asserted from the camera datasheet rather than measured, and several ablation-based architecture choices are supported by only one metric while other metrics in the supplement point differently. These issues need to be resolved before the specific claims about alignment-driven gains and the proposed fusion designs can be accepted.","major_comments":[{"comment":"The 'pixel-aligned' property is load-bearing for the dataset, the title, and the fusion equations (Eq. 3 and Eq. 7), but it is never verified quantitatively. Section 3 states that 'calibration between RGB and NIR images is unnecessary' and reports only stereo-pose and LiDAR calibration; the supplemental calibration section likewise contains no measured RGB-NIR registration error. Please report a measured alignment error (e.g., residual reprojection error of a checkerboard or dot pattern visible in both bands, or the epipolar error between RGB and NIR images) and, ideally, a sensitivity analysis showing that the reported downstream gains persist under small residual misregistration. Without this, the claimed advantage over pixel-misaligned datasets such as CS-Stereo is not established.","section":"Section 3 (Calibration)"},{"comment":"The architecture choice of alternating Fusion-NIR correlation volumes is justified by Table 3, which reports only RMSE. However, Supplemental Table 16 shows that on MAE the 'Fusion-RGB-NIR' variant is better than the selected 'Fusion-NIR' variant (2.5238 m vs. 2.5886 m), and Supplemental Table 15 shows that simple feature multiplication yields better RMSE and MAE than the proposed attentional feature fusion. Because these ablations are used to justify the core feature-fusion design, the paper must report all evaluation metrics (MAE, RMSE, d1-d3) for every ablation and either justify the chosen metric or explain why the alternative is preferred despite worse quantitative results. As written, the claim in the Table 3 caption that the selected variant gives 'the highest depth-estimation accuracy' is not supported by the full set of reported metrics.","section":"Section 4.2 and Table 3; Supplement Tables 15 and 16"},{"comment":"The real-world object-detection evaluation relies on pseudo-labels, but the manuscript gives no information about how these pseudo-labels were generated, whether they were manually verified, or what their accuracy is. The detection mAP numbers are also inconsistent between the main paper and the supplement: Table 1(b) reports Ours = 0.809, while Supplemental Table 11 reports Ours = 0.828 for the same task. Please clarify the labeling protocol, provide a verification procedure or error estimate for the pseudo-labels, and reconcile the two mAP values.","section":"Supplemental Section 4.2.1 and Table 11 / Table 1"},{"comment":"It is not stated whether the test split was used to select among the correlation-volume variants in Table 3 and among the feature-fusion variants in Supplemental Table 15. If these architecture choices were made using the same test data on which final results are reported, the reported gains are potentially optimistic. Please specify the validation procedure (e.g., a held-out split used for model selection, or a statement that the test set was used only once) and, if no validation set was used, discuss the risk of selection on the test set.","section":"Section 5.2 / Table 2"}],"minor_comments":[{"comment":"There is a typo: 'where where RGB imaging is inadequate' should have only one 'where'.","section":"Section 1, first paragraph"},{"comment":"The main text says temporal synchronization is maintained to 'less than 1 microsecond', while the supplemental Section 1.1.4 says the cameras were aligned 'within 100 microseconds' and packets with differences exceeding 1 ms were discarded. Please reconcile these numbers.","section":"Section 3 (Calibration)"},{"comment":"The sentence 'inspired by the cross-spectral time-of-flight imaging method [5]' is unexplained; please state what design element is borrowed from that method.","section":"Section 4.2"},{"comment":"The caption groups 'different RGB-NIR stereo depth estimation model [64]' with methods that were fine-tuned on the proposed dataset, but the text says only methods [16,51] were retrained. Please clarify whether CS-Stereo [64] was fine-tuned or used with its original checkpoint, since this affects fairness of the comparison.","section":"Table 2"},{"comment":"The dense disparity labels are produced by a BPNet-based reconstruction with an occlusion threshold; the paper would benefit from a short validation of this pseudo-ground-truth against raw LiDAR, e.g., the median error of the densified depth at held-out LiDAR points.","section":"Supplemental Section 2.1.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset and hardware description are the most original contributions; the two fusion methods are incremental but reasonable. The main risk is that the paper's headline claims are tied to unmeasured alignment and to ablation decisions that look different under other metrics. If the authors can provide a real RGB-NIR alignment measurement, a unified metric reporting, and a clearer evaluation protocol, the paper could become acceptable. I would also encourage the editor to verify that the dataset will actually be released with the claimed annotations and calibration files."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is the real contribution, and it looks worth having. The fusion methods are competent but the headline gains are moderate, and a few reporting choices need tightening. I would send this to review, not desk-reject it.\n\nWhat is actually new: a prism-based pixel-aligned RGB-NIR stereo rig with LiDAR ground truth, mounted on a mobile robot, captured continuously across day/night and indoor/outdoor conditions, plus a synthetic RGB-NIR extension. That combination is not in the prior datasets they compare against. The image fusion method is a modest, learned extension of HSV blending, but it is useful: it lets frozen RGB-pretrained models consume NIR data without fine-tuning, and the detection and SfM results support that claim. The feature fusion on RAFT-Stereo is a sensible modification, and the controlled comparisons against single-modality and several multispectral baselines are the right kind of evidence.\n\nSoft spots, in rough order of importance. First, the paper never reports a measured RGB-NIR alignment error. The JAI prism camera is marketed as pixel-aligned, and that is a reasonable starting point, but the title, the dataset's core property, and the claimed advantage over pixel-misaligned datasets all rest on it. A reprojection or residual-disparity measurement on a few checkerboard or scene frames would settle this. Without it, the attribution of gains to pixel-level alignment is not fully supported. Second, the object detection ground truth for the real dataset is described only as \"pseudo labels\" in the supplement. That could be noisy, and there is no detail on how they were created or validated. Third, the main text highlights the correlation-volume ablation via RMSE, while the supplement shows the chosen variant is not best on MAE. They do report both numbers, so it is not hidden, but the wording \"highest accuracy\" overstates the case. Fourth, for a dataset paper, there is no link to the dataset or code anywhere in the text despite claims that it will be shared. That needs to be fixed before publication.\n\nNone of these are load-bearing flaws in the sense that the central argument collapses. The central argument—that pixel-aligned RGB-NIR helps downstream tasks in challenging lighting—is plausible and backed by consistent, if moderate, gains. The paper is clearly written and engages fairly with the prior literature.\n\nWho is this for? Anyone working on multispectral robotic perception, RGB-NIR depth estimation, or day/night vision. It deserves a serious referee, with the expectation of major revision to add the alignment measurement, document the pseudo-labels, include the dataset link, and soften the metric claims.","headline":"A genuinely useful pixel-aligned RGB-NIR stereo + LiDAR dataset for robotics, with reasonable fusion methods; the unmeasured alignment claim and a few reporting choices need attention, but this deserves a serious referee.","tokens_in":39762,"tokens_out":1763,"would_cite":true,"duration_ms":19327,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pixel-aligned RGB and near-infrared stereo data cuts depth error by a quarter and lifts detection in dark scenes.","keywords":["RGB-NIR imaging","pixel-aligned multispectral","stereo depth estimation","image fusion","feature fusion","robot vision","multispectral dataset","near-infrared"],"falsifier":"Photograph a high-contrast checkerboard illuminated by both visible light and 850 nm NIR light with the same camera, and measure the per-corner offset between the RGB and NIR captures; if the median offset is even one pixel, the dataset's pixel-aligned premise and the attribution of the depth and detection gains to alignment would need to be revisited.","tokens_in":38746,"feed_emoji":"🤖","tokens_out":7089,"duration_ms":59467,"temperature":0.7,"pith_summary":"This paper argues that pixel-level alignment between RGB and near-infrared (NIR) stereo images is what lets robot vision systems exploit both spectra, and it builds a capture rig and dataset to demonstrate the claim. The authors introduce two fusion routes: an image-fusion method that encodes RGB and NIR into a single three-channel image usable by any pretrained RGB model without fine-tuning, and a feature-fusion method that fine-tunes a pretrained stereo network on fused RGB-NIR features. On their collected data the feature-fusion stereo estimator reaches 6.747 m depth RMSE versus 8.943 m for RGB-only input, and the learned fused image raises YOLO detection mAP from 0.756 to 0.828. The paper also releases a real-world dataset of 43 scenes and 80,000 RGB-NIR stereo frames with LiDAR ground truth, plus a synthetic RGB-NIR dataset with dense disparity labels.","feed_headline":"Pixel-aligned RGB-NIR fusion cuts stereo depth error","feed_subtitle":"One learned fusion lets pretrained RGB models use near-infrared in the dark, boosting detection and 3D reconstruction.","key_machinery":"The load-bearing hardware is a pair of JAI FS-1600D-10GE cameras: a dichroic prism inside each one reflects NIR onto one CMOS sensor and lets RGB pass to another, so the two spectral images share one viewpoint and one lens. The central computational operator is a learned pair of spatially-varying weights $\\alpha$ and $\\beta$ that blend the HSV brightness channel with the NIR image before converting back to RGB, producing a three-channel fused image for pretrained models. The feature-level variant adds an attentional feature-fusion module and alternates between fused-feature and NIR correlation volumes inside the GRU updates of RAFT-Stereo, which the paper finds is the most accurate way to use both spectra.","core_discovery":"The central discovery is that a single dichroic-prism camera can produce RGB and NIR frames that are pixel-aligned, so no registration or color-transform step is needed before fusion. With that alignment in place, the paper shows two ways to combine the spectra: fuse them at the image level with learned spatially-varying weights, which lets pretrained RGB-only models consume NIR data directly, or fuse them at the feature level inside a recurrent stereo network, which yields the largest accuracy gain. The reported numbers are a stereo depth RMSE of 6.747 m for feature fusion versus 8.943 m for RGB-only RAFT-Stereo, and detection mAP of 0.828 for the fused image versus 0.756 for RGB-only YOLO. These results support the paper's claim that pixel-aligned RGB-NIR data, rather than post-hoc registration, is what improves downstream vision in low-light and high-contrast scenes.","pith_inferences":["Pith inference (not a paper claim): the image-fusion module contains no geometry, so it may transfer to other prism-based RGB-NIR cameras without retraining; that transferability is a natural follow-up.","Pith inference (not a paper claim): if residual RGB-NIR misregistration is actually subpixel, the fusion gains may partly come from the spectral content itself; an artificial-misalignment experiment would separate the two.","Pith inference (not a paper claim): the dataset records separate exposure times for RGB and NIR sensors, so a future method could make fusion exposure-aware, which the current fixed-weight formulation does not exploit."],"forward_implications":["Pretrained RGB-only stereo, detection, and structure-from-motion pipelines can exploit NIR information with zero fine-tuning when fed the learned fused image.","Fine-tuned feature fusion beats image fusion and all compared cross-spectral baselines, with the largest gains appearing in nighttime scenes.","The 80,000-frame real dataset plus LiDAR-derived dense depth gives other methods a directly comparable RGB-NIR stereo training and evaluation target.","The alternating fused-NIR correlation volume result indicates that NIR features carry cues RGB features miss, especially under active NIR illumination."],"supporting_citations":[{"why":"Base pretrained stereo depth network used for both image-fusion inference and feature-fusion finetuning; also the RGB-only baseline.","marker":"[36]"},{"why":"Object detector that consumes the fused three-channel image without finetuning.","marker":"[45]"},{"why":"Structure-from-motion pipeline whose feature matching and reconstruction are evaluated on RGB, NIR, and fused images.","marker":"[46]"},{"why":"Fusion module that combines RGB and NIR feature maps in both proposed methods.","marker":"[12]"},{"why":"Baseline blending of brightness channel and NIR; the learned spatially-varying weights extend it.","marker":"[14]"},{"why":"Source of rendered RGB stereo and disparity used to synthesize the RGB-NIR stereo training set.","marker":"[39]"},{"why":"Supplies the RGB-to-NIR reflectance mapping used to generate synthetic NIR images.","marker":"[15]"},{"why":"Cross-spectral stereo method that must convert or register spectra, a comparison baseline showing the benefit of pixel alignment.","marker":"[64]"}],"fun_headline_variants":["Pixel-aligned RGB-NIR fusion lowers depth error by 25%","No-registration RGB-NIR fusion improves robot vision in dark","Aligned RGB-NIR lets pretrained models see in the dark","Feature fusion of aligned RGB-NIR beats RGB-only stereo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything downstream rests on the claim that the JAI dichroic-prism cameras deliver true pixel-level RGB-NIR alignment, but the paper reports no measured registration error between the two spectral images.","fun_headline_variants_meta":{"raw":{"variants":["Pixel-aligned RGB-NIR fusion lowers depth error by 25%","No-registration RGB-NIR fusion improves robot vision in dark","Aligned RGB-NIR lets pretrained models see in the dark","Feature fusion of aligned RGB-NIR beats RGB-only stereo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001147,"raw_usage":{"total_tokens":4746,"prompt_tokens":921,"completion_tokens":3825,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":3752}},"tokens_in":537,"tokens_out":3825,"duration_ms":26826,"temperature":1.0,"reasoning_tokens":3752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:06.090168+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Photograph a high-contrast checkerboard illuminated by both visible light and 850 nm NIR light with the same camera, and measure the per-corner offset between the RGB and NIR captures; if the median offset is even one pixel, the dataset's pixel-aligned premise and the attribution of the depth and detection gains to alignment would need to be revisited.","supporting_citations":[{"cited_title":"Color outdoor image enhancement by v-nir fusion and weighted luminance","cited_arxiv_id":null,"evidence_quote":"Supplies the RGB-to-NIR reflectance mapping used to generate synthetic NIR images."},{"cited_title":"Bilateral propagation network for depth completion","cited_arxiv_id":null,"evidence_quote":"Base pretrained stereo depth network used for both image-fusion inference and feature-fusion finetuning; also the RGB-only baseline."},{"cited_title":"Gated2depth: Real-time dense lidar from gated images","cited_arxiv_id":null,"evidence_quote":"Fusion module that combines RGB and NIR feature maps in both proposed methods."},{"cited_title":"Deep multispectral semantic scene understanding of forested environments using multimodal fusion","cited_arxiv_id":null,"evidence_quote":"Source of rendered RGB stereo and disparity used to synthesize the RGB-NIR stereo training set."}],"review_version":1}