Pith. sign in

REVIEW 4 major objections 4 minor 17 references

A Multi-Sensor Dataset for Monitoring the Operational Environment of Rail Vehicles

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new multi-sensor railway dataset packs 7,052,055 annotations, available on request.

desk verdict A competent dataset paper whose headline number checks out; the real limitations are access and missing quantitative quality metrics, not internal arithmetic. read the letter →

arxiv 2608.04704 v1 pith:CC4RAJEE submitted 2026-08-05 cs.CV cs.RO

Claudio Diotallevi , Rodrigo Gudiño , Zaharia Pachalieva , Philipp Neumaier , Patrick Naumann , Erik Bochinski , Volker Eiselein , Martin Köppel This is my paper · ORCID
classification cs.CVcs.RO
keywords railwayperceptionmulti-sensordatasetgradeofautomationlidarannotationradarobjectdetectionenvironmentmonitoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Railway automation at every Grade of Automation—from GoA2 with a driver still on board to fully driverless GoA4—requires perception systems that detect and classify hazards in real time, and those systems need large, accurately annotated datasets. This paper reports the completion of a new multi-sensor dataset built for that purpose: over seven million annotations across 21 object classes, recorded on two rail vehicles with front-mounted RGB, infrared, lidar, and radar sensors under a variety of operational scenarios. The annotation strategy is to mark objects in the 3D lidar point clouds first and then project those labels into the camera and radar images, which gives the dataset a common multi-sensor ground truth. The authors state the dataset is now available on request from the rail infrastructure operator, positioning it as a resource for developing environment monitoring for GoA2–GoA4 systems.

What carries the argument

The mechanism that carries the dataset is the 3D-first annotation and projection pipeline. Annotators draw 3D bounding boxes, polygons, and polylines in lidar point clouds; projection functions then transfer those annotations into 2D bounding boxes on RGB camera images, 2D polygons, 2D boxes on IR images, and lidar points mapped into radar images. This makes the lidar point cloud the single source of geometric truth, and the resulting labels are exported as JSON files following a rail-specific annotation schema that builds on an open labeling standard.

What would settle it

Open the requested dataset and compute the reprojection error: take a sample of lidar-annotated 3D boxes, project them into the camera images, and compare with manually drawn 2D boxes on those images; if the average intersection-over-union falls below a pre-specified threshold (for example, 0.5) or systematic offsets of several pixels appear, the claimed multi-sensor alignment is falsified.

Watch

Extended reading notes

Core claim

The central claim is the existence and practical availability of a railway perception dataset with 7,052,055 annotations across 21 classes, produced from 88.2 minutes (5,292 seconds) of annotated sensor data: 1,981 seconds in 69 sequences from a track maintenance vehicle and 3,311 seconds in 194 sequences from a commuter train. The sensor configurations include six RGB cameras, three IR cameras, six lidars, and a radar on the maintenance vehicle, and three RGB cameras, one IR camera, six lidars, and four radars on the commuter train. All objects were first annotated by experts in the 3D lidar point clouds, then projected onto the 2D camera and radar frames using projection functions, with manual checking and rework afterward. The annotation set is dominated by railway-specific classes such as catenary poles, signals, signal poles, switches, tracks, and trains, in addition to general classes like persons, road vehicles, and bicycles. The authors present this as a resource for GoA2–GoA4 automation, infrastructure monitoring, and environment observation.

Load-bearing premise

The dataset's usefulness rests on the assumption that all the sensors on each vehicle are precisely aligned and synchronized, so that the lidar-made 3D labels land on the correct pixels in the camera images and radar frames.

Editorial extensions

If this is right

  • The dataset can be used as training and validation data for perception models that support partially automated (GoA2) through fully automated (GoA4) railway operation.
  • Because the annotations live first in 3D lidar space and are projected to all other sensors, the same object has aligned labels in RGB, IR, and radar, enabling multi-sensor fusion and cross-modal learning.
  • The class distribution, with railway infrastructure elements such as catenary poles, signals, switches, and tracks represented in the millions, covers object categories that automotive datasets largely ignore.
  • As an available-on-request resource, the dataset gives industry teams a common benchmark for comparing environment-monitoring algorithms without repeating the expensive data collection.
  • The 88.2 minutes of annotated data provide a real-world complement to synthetic railway datasets, supporting validation of simulation-trained models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not report a quantitative evaluation of projection accuracy; measuring how far lidar-derived 3D boxes deviate from hand-drawn 2D boxes would turn the qualitative statement of alignment into a number users can trust.
  • The heavy class imbalance—over a million catenary-pole annotations versus a few thousand bicycles or wheelchairs—means naive training on the full set will favor frequent classes; benchmark designers would need to define balanced evaluation splits.
  • Because the dataset contains only 88.2 minutes of annotated sequences, conclusions about generalization across seasons, weather, and geographical regions would need to be established by additional collections or domain adaptation.
  • If the projection pipeline is as reliable as claimed, a camera-only perception system could be trained in part using lidar-derived labels, effectively transferring 3D geometry into 2D detectors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes a multi-sensor dataset for railway environment perception, recorded from two rail vehicles (a GAF track maintenance vehicle and a BR472 commuter train) equipped with multiple RGB cameras, IR cameras, lidars, radar, and position/acceleration sensors. The authors report 7,052,055 annotations across 21 object classes, 88.2 minutes of annotated sensor data, and describe an annotation methodology that starts with 3D bounding boxes in lidar point clouds and projects them to 2D boxes, polygons, IR images, and radar images. The paper includes a comparison to existing railway datasets, a per-class annotation breakdown, and a description of quality-control steps. The dataset is available upon request from DB InfraGO AG.

Significance. If the dataset is made accessible and its quality can be independently verified, it would be a valuable complement to existing railway perception datasets such as OSDaR23 and RailGoerl24, offering a larger annotation count and multi-sensor coverage. The paper is a concise description of a substantial annotation effort, and the internal arithmetic of Table 2 is consistent (the sum of the class totals equals the stated overall total of 7,052,055). The use of the RailLabel/ASAM OpenLABEL schema is also a strength. However, the current manuscript does not provide sufficient quantitative evidence for the claimed annotation quality, does not report per-sensor or per-annotation-type statistics needed for comparability, and offers no public or verifiable access to the data, all of which are central for a dataset paper.

major comments (4)
  1. [Section 6, Table 2] The reported total of 7,052,055 annotations aggregates across different annotation types (2D boxes, polygons, polylines, 3D boxes, and projected radar points) and across fifteen or more sensors, but the paper does not provide any per-annotation-type, per-sensor, or per-sequence frame count. Consequently, the headline figure is not interpretable or comparable to existing datasets such as OSDaR23, which report per-sensor frame counts and annotation breakdowns. Please provide a detailed breakdown of annotation counts by type, by sensor, and by sequence, at least in a supplementary table.
  2. [Sections 4 and 5] The paper claims 'high-quality annotations' (Abstract) and describes a projection pipeline from 3D lidar boxes to 2D boxes, polygons, IR images, and radar images, but provides no quantitative validation. There are no calibration reprojection errors, no synchronization offsets, no IoU or other agreement metrics between projected and manually corrected annotations, and no inter-annotator agreement is reported. Section 5 mentions that 'about 5% of the data was reviewed' but does not report the outcome of that review, such as an error rate or correction rate. Because the dataset's central value rests on annotation quality, this omission is load-bearing and should be addressed with concrete numbers.
  3. [Sections 3 and 6] The dataset is only available 'upon request' via email, with no download mechanism, persistent identifier, license, or terms of use described. This means reviewers and potential users cannot independently verify any of the reported quantities, including sensor configuration, annotation counts, or quality. Please provide at least a representative sample subset for review, a formal data access agreement, and a persistent identifier, or clearly state why this is not possible.
  4. [Section 4] The annotation process is described as starting with 3D bounding boxes in lidar point clouds, followed by projection and manual reworking, but the paper does not specify which of the 21 object classes are annotated in 3D versus 2D only, nor how the attributes mentioned in Section 4 are encoded. This information is essential for users to understand the dataset's applicability and for fair comparison with other datasets. Please include a per-class annotation-type matrix.
minor comments (4)
  1. [Figures 5 and 6] The captions and in-text references use 'Disribution' instead of 'Distribution'.
  2. [Table 2] Adding a total row for the GAF, BR472, and overall columns would facilitate verification of the stated 7,052,055 total.
  3. [Section 6] Please report the number of annotated frames per sensor (or at least per sequence) in addition to the total annotated duration, since frame counts are the metric used in the comparison table in Section 2.
  4. [Section 2, Table 1] The comparison table does not list the annotation types or sensor frame rates for the proposed dataset; adding these columns would clarify how the dataset relates to the existing ones.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a descriptive dataset report with no fitted parameters, predictions, or self-cited derivation chain.

full rationale

This is a dataset description paper, not a derivation or prediction paper. Its central claim is that a multi-sensor railway dataset contains 7,052,055 annotations across 21 classes, and summing the per-class counts in Table 2 (4,081,482 for GAF plus 2,970,573 for BR472) reproduces that total, so the headline count is internally consistent rather than circular. The annotation pipeline (Section 4) is described operationally: 3D lidar annotations are projected into camera and radar frames and then manually checked and reworked; this is an engineering workflow, not an equation whose output is defined by its input. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity, and no uniqueness theorem or load-bearing self-citation is invoked to force a choice. Some limitations exist that are separate from circularity: the paper does not report calibration reprojection errors, per-sensor frame counts, or quantitative validation of projected annotations, and the dataset is only available on request. Those concerns relate to completeness and external verification, not to circular reasoning. The comparisons to existing datasets (Table 1) are descriptive, and the methodology (Section 5) is a stated quality-control sequence with no claim of deriving annotations from the total counts. Accordingly, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper makes no theoretical derivation, so the ledger contains only the data-quality assumptions required for the dataset to be useful.

assumptions (3)
  • domain assumption Sensor calibration and synchronization are accurate across all modalities.
    The projection of 3D lidar annotations to 2D cameras and radar relies on precise calibration, which is asserted but not quantified in Section 4.
  • domain assumption Annotation quality is high as claimed.
    The QC process in Section 5 is described, but no quantitative metrics like inter-annotator agreement are given.
  • domain assumption The two recording vehicles provide representative railway environments.
    Data comes from selected routes in Hamburg and Berlin; the paper does not discuss coverage of other infrastructure types or weather conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multi-Sensor Dataset for Monitoring the Operational Environment of Rail Vehicles." pith.science (2026). https://pith.science/paper/CC4RAJEE

@misc{pith2026260804704,
  author       = {Pith},
  title        = {Pith review of: A Multi-Sensor Dataset for Monitoring the Operational Environment of Rail Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CC4RAJEE}},
  note         = {Machine review of arXiv:2608.04704}
}
read the original abstract

Reliable environment monitoring is essential for the safe and efficient operation of automated railway systems, covering all Grades of Automation (GoA), from partially automated (GoA2) to fully automated operation (GoA4). Artificial Intelligence (AI) plays a central role in enabling these systems to detect, classify, and react to potential hazards in real time. The development of such AI-based perception systems requires large volumes of accurately annotated data for training and validation. Within the Digitale Schiene Deutschland (DSD) program, DB InfraGO AG and understandAI GmbH have developed a comprehensive multi- sensor dataset tailored to the needs of railway environment perception. This dataset contains over 7 million high-quality annotations of both railway-specific and general perception objects, captured under varying operational scenarios. The finalized dataset can now be requested at the DB InfraGO AG and serve as a valuable resource for advancing AI-driven environment monitoring in the railway domain.

Figures

Figures reproduced from arXiv: 2608.04704 by the authors.

Figure 1
Figure 1. Track maintenance vehicle (GAF). was carefully planned. Furthermore, the generation pipeline allows an easy extensions for new scenarios and the integration of other annotations. In addition to these open datasets, there are others that have been used in research but are not publicly available, such as the RAILO dataset, which comprises 4,651 manually annotated single RGB-camera frames from six different scenes [12]… view at source ↗
Figure 2
Figure 2. Commuter train (BR472). 3.2. Commuter vehicle (Hamburg) The second test vehicle ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. 2D-Bounding Boxes of persons in the annotated camera image [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: 3D-bounding boxes of workers, projected into the camera image. tion about the objects via attributes. Since 2021, UAI and DB InfraGO AG have jointly created annotations for 21 object classes. The utilized annotation methodology included both automatic software tools an…
Figure 5
Figure 5. Figure 5: Disribution of annotations across classes for GAF data [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Disribution of annotations across classes for commuter train data. 6. Multi-sensor dataset. The created dataset comprises a total of 88.2 min (5292 s) of annotated sensor data. This includes 1981 seconds and 69 sequences of real sensor data from the GAF vehicle and 331…
Figure 7
Figure 7. Figure 7: Train annotated with a polygon and tracks annotated with polylines in the 2D image [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Train and tracks annotated with 3D bounding boxes in the 3D point clouds. The pie charts in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

  1. [1]

    Digitale Schiene Deutschland, https://digitale-schiene-deutschland.de/de, 2025

    work page 2025

  2. [2]

    Data Factory, https://digitale-schiene-deutschland.de/en/projects/DataFactory, 2025

    work page 2025

  3. [3]

    UnderstandAI, https://understand.ai/, 2025

    work page 2025

  4. [4]

    1221-1229

    Zendel O, Murschitz M, Zeilinger M, Steininger D, and Abbasi S, Beleznai C, RailSem19: A Dataset for Semantic Rail Scene Understanding, CVPRW, 2019, pp. 1221-1229

    work page 2019

  5. [5]

    Harb J, R ´eb´ena N, Chosidow R, Roblin G, Potarusov R, Hajiri H, FRSign: A Large-Scale Traffic Light Dataset for Autonomous Trains, CoRR, 2002.05665, 2020

    arXiv 2002

  6. [6]

    5041-5054

    Toprak T, Belenlioglu B, Aydın B, Guzelis C, Selver MA, Conditional Weighted Ensemble of Trans- ferred Models for Camera Based Onboard Pedestrian Detection in Railway Driver Support Systems, IEEE TVT, 2020, pp. 5041-5054

    work page 2020

  7. [7]

    6455–6463

    Li X, Peng X, Rail Detection: An Efficient Row-Based Network and a New Benchmark, Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 6455–6463

    work page 2022

  8. [8]

    Zouaoui A, Mahtani A, Hadded MA, Ambellouis S, Boonaert J, Wannous H, RailSet: A Unique Dataset for Railway Anomaly Detection, 2022, Journal of Rail and Rapid Transit, pp. 1-6

    work page 2022

Show all 17 references
  1. [9]

    Leibner P, Hampel F, Schindler C, GERALD: A novel dataset for the detection of German mainline railway signals, Journal of Rail and Rapid Transit, 2023

    2023

  2. [10]

    Tagiew R, Klasek P, Tilly R, K ¨oppel M, Denzler P, Neumaier P, Klockau T, Boekhoff M, Schwalbe K, OSDaR23: Open sensor data for rail 2023, 2023 8th International Conference on Robotics and Automa- tion Engineering (ICRAE), 2023

    2023

  3. [11]

    Tagiew R, Wunderlich I, Sastuba M, Seitz S, RailGoerl24: G ¨orlitz Rail Test Center CV Dataset 2024, https://arxiv.org/abs/2504.00204, 2025

    2024 arXiv

  4. [12]

    D’Amico, F

    D’Amico G, Nesti F, Rossolini G, Marinoni M, Sabina S, Buttazzo G. D’Amico, F. Nesti, G. Rossolini, M. Marinoni, S. Sabina and G. Buttazzo, SynDRA: Synthetic Dataset for Railway Applications, 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), Tucson, A...

    2025

  5. [13]

    , Ziegler M., Jacobs A., Eiselein V ., Neumaier P

    Denzler P. , Ziegler M., Jacobs A., Eiselein V ., Neumaier P. and K ¨oppel M., Multi-Sensor Data Anno- tation Using Sequence-based Active Learning, 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 258-263

    2022

  6. [14]

    , Mhasawade V ., K ¨oppel M., Neumaier P

    Ziegler M. , Mhasawade V ., K ¨oppel M., Neumaier P. and Eiselein V ., ”A Comprehensive Framework for Evaluating Vision-Based on-Board Rail Track Detection,” 2023 IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA, 2023, pp. 1-8

    2023

  7. [15]

    Hauswald, C.; Skibinski, S.; Reiniger, F.; Euler, T.; Isaac, G.; Irvathraya, A.: Sensors4Rail: Ein Erfol- gsprojekt ist zu Ende, EI 09/2023

    2023

  8. [16]

    RailLabel, https://github.com/DSD-DBS/raillabel, 2025

    2025

  9. [17]

    ASAM OpenLABEL, https://www.asam.net/standards/detail/openlabel/, 2025

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.