{"id":"3ef26f8a-ffc5-4a65-9874-577c4a8f4236","arxiv_id":"2412.06192","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PoLaRIS is a new multi-modal maritime dataset with object detection labels, point-wise LiDAR and radar annotations, and tracking IDs, claimed to be the first such dataset.","lead":"This paper presents PoLaRIS, a maritime dataset that combines RGB, thermal, LiDAR, and radar images with labeled ships and buoys, including objects as small as 10 by 10 pixels, and tracking IDs for moving hazards. It is designed to give autonomous boats a benchmark for detecting and tracking obstacles in narrow canals and ports.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 2 in §III.B.1 projects left-image bounding-box corners into the TIR camera without scene depth; for a rig with nonzero translation this is not a valid projection, so TIR labels are not justified by the stated formula and no quality metric is provided.","rationale":"The reader's verdict is already conditional, and this stress-test finds the same broad risk but identifies a sharper mechanism: Eq. 2 is not a valid 2D-to-2D transfer when the TIR-left transform has translation. This is an internal mathematical issue, not a matter of consensus. It does not automatically falsify the dataset because the authors state that all TIR boxes are manually checked, and the LiDAR/Radar labels are separately filtered and verified. However, the paper gives no quantitative annotation-quality study, so the benchmark's reliability rests entirely on undocumented human correction. The evaluation leakage noted by the reader is real but secondary: benchmark numbers do not affect whether the dataset exists or is first. Dataset availability and code absence are practical concerns, not correctness concerns. Therefore the appropriate verdict remains conditional: accept if the public release includes calibration and a label-quality evaluation; otherwise the multi-modal claim is under-supported. This is an UNCHANGED verdict relative to the reader.","tokens_in":10171,"tokens_out":5834,"duration_ms":65128,"concrete_test":"Use the released calibration to project the same 3D points with Eq. 2 and with the depth-correct formula for objects at 5, 10, and 50 m; if bounding-box corner positions differ by more than a few pixels, Eq. 2 is invalid as stated. Then independently manual-annotate a 100-frame subset of TIR images and compare IoU with the released TIR boxes, stratified by LiDAR-derived depth. If IoU is high at all depths, manual correction rescued the labels; if errors increase at short range, the multi-modal annotation quality is compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's central claim is that PoLaRIS is the first maritime dataset with reliable multi-modal annotations (RGB, TIR, LiDAR, Radar), so the geometric validity of the TIR labels is load-bearing. Eq. 2 states bb_t = K_t * T_t^l * K_l^{-1} * bb_l. Since K_l^{-1} * bb_l is the ray direction normalized to z=1, applying a Euclidean transform containing translation and then K_t projects the point as though every object lies at depth 1 m. The correct transfer requires the per-object depth Z: bb_t = K_t * (R * (Z * K_l^{-1} * bb_l) + t), which Eq. 2 omits. Unless the TIR-left baseline is zero, projected boxes will be displaced by an amount that grows with baseline and inverse depth. The paper says manual post-processing corrects reference labels, but it reports no quantitative validation of TIR label quality. The same dependence on unvalidated extrinsic calibration and temporal synchronization affects the LiDAR and Radar point labels, which are filtered by projecting into RGB bounding boxes. Thus the core claim of reliable multi-modal annotations is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PoLaRIS, a maritime object detection and tracking dataset built on the Pohang Canal dataset. It provides left/right RGB, TIR, LiDAR, and Radar annotations for five sequences, including tracking IDs for dynamic objects and annotations for small objects down to 10×10 pixels. A semi-automatic pipeline initializes RGB labels with YOLOv8 and manual refinement, transfers labels to TIR via extrinsics, filters LiDAR points projected into boxes, and assigns Radar clusters by overlap with LiDAR labels. Benchmarks with YOLOv8/YOLOv10/RT-DETR and SORT/ByteTrack/OC-SORT/Hybrid-SORT are reported for RGB and TIR. The paper claims to be the first maritime dataset with multi-modal annotations across these four modalities.","tokens_in":10460,"tokens_out":6632,"duration_ms":66889,"significance":"If the annotation quality is established, PoLaRIS fills a genuine gap: no existing maritime dataset provides synchronized RGB, TIR, LiDAR, and Radar labels with tracking IDs and small-object coverage. The resource is potentially valuable for USV perception, and the benchmark numbers for RGB/TIR detection and tracking are useful. Strengths include the public dataset, use of an existing raw-data collection (Pohang Canal), a semi-automatic pipeline with human verification, and evaluation with several SOTA detectors and trackers. The construction is independent of the benchmark results, so there is no circularity in the evaluation. However, the paper currently lacks quantitative validation of the transferred and point-wise labels, so the significance depends on additional evidence.","major_comments":[{"comment":"Equation (2) is not a valid perspective transfer unless the TIR-left baseline is zero. Since bb_l is the homogeneous pixel coordinate [u v 1]^T, K_l^{-1} bb_l is the ray direction at depth 1, and applying T_t^l (which contains translation) projects that point as though every object lies at depth 1 m. The correct transfer needs per-object depth Z: bb_t = K_t (R (Z K_l^{-1} bb_l) + t). The manual post-processing mentioned in the text may compensate, but no quantitative measure of TIR label quality (e.g., alignment error, correction rate, or inter-annotator agreement) is reported. Because TIR annotations are a core contribution, this must be addressed.","section":"§III.B.1, Eq. (2)"},{"comment":"The Radar annotation procedure relies on DBSCAN clustering and assigns labels to clusters that overlap with LiDAR labels, but the paper does not report the DBSCAN parameters (eps, min_samples), the overlap criterion, or any validation of the resulting point labels. Given the wide horizontal beamwidth of marine Radar, cluster-level overlap can merge distinct objects or include clutter. The paper should report parameters and a quantitative assessment (e.g., precision/recall against manually verified Radar points or a manual verification study).","section":"§III.B.3"},{"comment":"For the night sequence, ground truth is produced by manually annotating restored RGB images produced by GSAD. The text does not specify whether these labels are applied to the raw or restored images; if applied to the raw night RGB images, the ground truth implicitly assumes that restoration preserves object boundaries and positions. No analysis of this label-transfer error is provided. The paper should either annotate the raw images directly or quantify the difference between annotations on restored and raw images, since night RGB is one of the benchmark modalities.","section":"§III.A.1"},{"comment":"Despite the claim of a multi-modal benchmark, Section IV evaluates only RGB and TIR detection and tracking. There are no benchmark experiments using the LiDAR point-wise labels or the Radar point-wise labels, so the utility of those annotations for algorithm evaluation is not demonstrated. Adding at least basic LiDAR/Radar baseline experiments, or explicitly stating that these modalities are provided without benchmark results, would strengthen the paper.","section":"§IV"},{"comment":"The paper reports only approximate totals ('approximately 360,000 images' and 'around 190,000 labels') and a qualitative bar chart. For a benchmark dataset, exact per-sequence and per-modality label counts, class distributions, object-size distributions, and the exact train/validation/test splits are needed for reproducibility. These numbers are currently missing.","section":"§III, Fig. 2"}],"minor_comments":[{"comment":"Reference [2] is called the 'USVland dataset' in the text, but Table I and the reference title say 'USVInland'; please unify the name.","section":"§II"},{"comment":"The sentence 'To briefly explain the two transformations' is followed by only one equation; the text should be rephrased to match the content.","section":"§III.B.1"},{"comment":"The caption says the vertical axis represents sequences, but the axes are not labeled in the figure, making it hard to read the exact label counts.","section":"Fig. 2 caption"},{"comment":"The sentence 'we propose a PoLaRIS presented a multi-modal 3D object’s bounding box' is ungrammatical and should be rewritten.","section":"§II"},{"comment":"For TIR rows, the Pre-train column uses '-' for COCO; the text explains this, but a footnote in the table would improve readability.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The dataset is potentially valuable and within the journal's scope. The main risk is annotation quality; the concern about Eq. (2) is valid and should be addressed with a corrected formula and quantitative validation. The paper would also benefit from exact dataset statistics and LiDAR/Radar baseline experiments. No citation or novelty concerns beyond what is stated; the authors properly acknowledge the Pohang Canal dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real dataset contribution—first maritime set I know of with RGB, TIR, LiDAR, and Radar all carrying object-level labels and tracking IDs, including 10x10-pixel targets. The comparison table is useful and the semi-automatic pipeline is sensible. The benchmarks with YOLOv8/v10, RT-DETR, SORT-family trackers are straight evaluations, not fitting, so no circularity problem.\n\nThe soft spots are real but manageable. The biggest is Eq. 2. As written, bb_t = K_t * T_t^l * K_l^{-1} * bb_l treats the left-camera ray as a point at depth 1, then applies a Euclidean transform with translation. That is not a valid projection for objects at arbitrary depth unless the TIR-left baseline is zero. The paper says manual post-processing fixes the converted boxes, and that may well be true, but there is no quantitative check—no inter-annotator agreement, no calibration reprojection error, no count of how much manual correction was needed. Since the dataset's selling point is reliable cross-modal annotations, this is load-bearing. If manual correction is extensive, the formula is just a crutch and the labels might still be fine; if not, the TIR labels are systematically off for near objects. The same unquantified calibration/timing uncertainty applies to LiDAR and Radar points, which are filtered by projecting into RGB boxes.\n\nSecond, the evaluation split draws training images randomly from the same sequences as test (5% per region). Adjacent frames in a canal sequence are near-duplicates, so those mAP/MOTA numbers are optimistic and shouldn't be compared across papers. A chronological split, or held-out sequences, would be more honest. No code is released, which makes the DBSCAN parameters and IoU thresholds impossible to reproduce.\n\nWhat's good: the scale-aware annotation effort, the inclusion of night with restored RGB, the explicit tracking IDs for dynamic objects, and the honest comparison table. The paper is clearly written and the authors know the existing maritime datasets.\n\nWho this is for: anyone building or evaluating multi-sensor USV perception systems. It deserves peer review, but acceptance should be conditional on: (1) public release of data and code, (2) quantitative label-quality metrics, (3) fixing or re-deriving Eq. 2 or at least reporting the TIR projection error, and (4) redoing the benchmark split to avoid temporal leakage. If those are addressed, this could be a standard benchmark; as is, treat the annotations with caution.","headline":"A genuinely new multi-modal maritime dataset, but the TIR label-generation formula is geometrically unsound as stated and label quality is unquantified; worth reviewing with conditions.","tokens_in":10948,"tokens_out":2191,"would_cite":true,"duration_ms":22395,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"New dataset tags ships and buoys in RGB, thermal, LiDAR, and radar.","keywords":["maritime object detection","multi-modal dataset","small object detection","object tracking","thermal infrared","LiDAR point cloud","radar point cloud","unmanned surface vehicle"],"falsifier":"Take a scene with a calibration target visible to all four sensors, project its corners from the left camera into the TIR image and LiDAR/radar coordinates using the published pipeline, and measure the offset against manually labeled ground truth; if the offset is comparable to or larger than the smallest annotated objects (10 pixels) or causes a substantial fraction of LiDAR/radar points to fall on non-object surfaces, the cross-modal labels at the claimed scale are not reliable.","tokens_in":10027,"feed_emoji":"🚢","tokens_out":6530,"duration_ms":57209,"temperature":0.7,"pith_summary":"This paper introduces PoLaRIS, a maritime obstacle dataset recorded in Pohang Canal, and claims it is the first to pair RGB and thermal camera bounding boxes with point-wise LiDAR and radar annotations and tracking IDs for dynamic obstacles. The authors argue that existing maritime datasets lack depth information and tracking labels, especially for small distant objects, which limits work on collision avoidance for unmanned surface vehicles. The dataset provides roughly 190,000 labels across five day and night sequences, including objects as small as 10 by 10 pixels. Benchmark runs with standard detectors and trackers on RGB and TIR imagery show strong daytime RGB performance and reveal that open-vocabulary detectors struggle with the small objects.","feed_headline":"New dataset tags ships and buoys in RGB, thermal, LiDAR, and radar","feed_subtitle":"PoLaRIS tracks obstacles as small as 10 by 10 pixels in day and night scenes, with point-wise depth from LiDAR and radar.","key_machinery":"The load-bearing mechanism is the semi-automatic multi-modal annotation pipeline. Left RGB bounding boxes are produced by detector initialization plus manual cleanup, then propagated to the TIR camera through the projective transformation in Equation 2 and to LiDAR by filtering for points whose projections fall inside the RGB box. Radar labels come from converting labeled LiDAR points into radar coordinates, projecting both into bird's-eye view, and applying DBSCAN clustering so radar clusters overlapping the LiDAR labels inherit the label. This pipeline turns one manual 2D annotation into synchronized thermal, LiDAR, and radar labels across sensors.","core_discovery":"The central claim is that PoLaRIS is the first maritime object detection and tracking dataset to provide simultaneous annotations in RGB, thermal infrared, LiDAR, and radar, including per-point depth labels and tracking ground truth for ships and buoys. Labeling begins on left RGB frames: initial boxes from a detector are manually refined, then projected into the TIR image with the extrinsic calibration transformation $bb_t = K_t T^t_l K_l^{-1} bb_l$, while LiDAR points are kept only when they project inside the RGB bounding box. Radar points are clustered with DBSCAN and labeled by overlap with the LiDAR annotations in bird's-eye view. Evaluations with YOLOv8, YOLOv10, RT-DETR, SORT, ByteTrack, OC-SORT, and Hybrid-SORT on day and night, RGB and TIR data are presented as evidence that the dataset supports detection and tracking benchmarking.","pith_inferences":["Because radar labels are derived only from clusters overlapping LiDAR points inside RGB boxes, the dataset likely under-labels distant radar returns beyond LiDAR range; users testing radar-only long-range detection should treat the radar annotations as bounded by LiDAR coverage.","If the extrinsic calibration between cameras drifts, the projected TIR boxes and LiDAR filtering inherit that error; a simple check would be comparing projected TIR boxes against the manually corrected versions the authors describe, since the paper does not report this agreement.","The same semi-automatic pipeline could be reused to add multi-modal labels to other single-modality maritime datasets, provided the sensors can be calibrated and synchronized.","The paper reports overall tracking scores but does not break them down by object size; analyzing MOTA and identity switches for only the smallest objects would test whether the dataset actually supports the claimed small-object tracking contribution."],"forward_implications":["Detectors and trackers can now be evaluated on identical maritime scenes across visible, thermal, LiDAR, and radar modalities, enabling direct cross-sensor fusion studies.","The 10-by-10-pixel annotations provide a small-object benchmark; in the reported tests, open-vocabulary detectors score below 2.5 mAP without fine-tuning, showing where generalization currently fails.","Tracking IDs for dynamic objects make it possible to measure identity preservation (MOTA, IDF1) on small maritime obstacles rather than only detection accuracy.","Night-sequence labels produced with low-light restoration and TIR projection allow quantitative comparison of RGB and thermal performance under poor visibility."],"supporting_citations":[{"why":"Supplies the raw Pohang Canal sequences (six multimodal sessions) that PoLaRIS re-annotates.","marker":"[1]"},{"why":"Generates the initial large-object bounding boxes that annotators refine, and serves as the detector for tracking benchmarks.","marker":"[15]"},{"why":"Restores low-light night images so that manual annotation can be done on visible RGB frames.","marker":"[17]"},{"why":"Converts 16-bit TIR images to 8-bit for visualization and manual correction of projected labels.","marker":"[18]"},{"why":"Clusters radar returns so clusters overlapping labeled LiDAR points receive point-wise labels.","marker":"[20]"},{"why":"Provides the ByteTrack baseline used to validate the tracking ground truth on the single-dynamic-object night sequence.","marker":"[33]"},{"why":"Provides the OC-SORT baseline that achieves the best tracking accuracy on the multi-dynamic-object day sequence.","marker":"[34]"}],"fun_headline_variants":["First maritime dataset with RGB, thermal, LiDAR, and radar annotations","PoLaRIS: multimodal dataset for detecting tiny maritime obstacles","Maritime hazard dataset tracks ships and buoys in four sensors","PoLaRIS dataset: first to combine four sensors for maritime tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cross-modal labels are only as correct as the extrinsic calibration and temporal synchronization among the RGB, TIR, LiDAR, and radar sensors, and the paper does not quantify how calibration error propagates into the projected boxes, LiDAR filtering, or radar clusters.","fun_headline_variants_meta":{"raw":{"variants":["First maritime dataset with RGB, thermal, LiDAR, and radar annotations","PoLaRIS: multimodal dataset for detecting tiny maritime obstacles","Maritime hazard dataset tracks ships and buoys in four sensors","PoLaRIS dataset: first to combine four sensors for maritime tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000794,"raw_usage":{"total_tokens":3482,"prompt_tokens":917,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2492}},"tokens_in":533,"tokens_out":2565,"duration_ms":18241,"temperature":1.0,"reasoning_tokens":2492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:54:15.115350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a scene with a calibration target visible to all four sensors, project its corners from the left camera into the TIR image and LiDAR/radar coordinates using the published pipeline, and measure the offset against manually labeled ground truth; if the offset is comparable to or larger than the smallest annotated objects (10 pixels) or causes a substantial fraction of LiDAR/radar points to fall on non-object surfaces, the cross-modal labels at the claimed scale are not reliable.","supporting_citations":[{"cited_title":"Pohang canal dataset: A multimodal mar- itime dataset for autonomous navigation in restricted waters,","cited_arxiv_id":null,"evidence_quote":"Supplies the raw Pohang Canal sequences (six multimodal sessions) that PoLaRIS re-annotates."},{"cited_title":"Fieldscale: Locality-aware field-based adaptive rescaling for thermal infrared image,","cited_arxiv_id":null,"evidence_quote":"Converts 16-bit TIR images to 8-bit for visualization and manual correction of projected labels."},{"cited_title":"Observation-centric sort: Rethinking sort for robust multi-object tracking,","cited_arxiv_id":null,"evidence_quote":"Provides the OC-SORT baseline that achieves the best tracking accuracy on the multi-dynamic-object day sequence."}],"review_version":1}