FisheyeMODNet: Moving Object detection on Surround-view Cameras for Autonomous Driving

Ganesh Sistu; Hazem Rashed; Ian Clancy; Letizia Mariotti; Lucie Yahiaoui; Marie Yahiaoui; Senthil Yogamani; Varun Ravi Kumar

REVIEW 4 major objections 5 minor 18 references

FisheyeMODNet: Moving Object detection on Surround-view Cameras for Autonomous Driving

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A lightweight two-stream CNN trained on fisheye surround-view images detects moving objects at embedded speed, lifting moving-object IoU from the 10% of a rectilinear-trained model to about 40%.

desk verdict A useful first benchmark for fisheye moving-object detection, but every accuracy number sits on an unvalidated semi-automatic annotation pipeline and the KITTI comparison is confounded. read the letter →

arxiv 1908.11789 v1 pith:Q2A6WXXB submitted 2019-08-30 cs.CV eess.IV

Marie Yahiaoui , Hazem Rashed , Letizia Mariotti , Ganesh Sistu , Ian Clancy , Lucie Yahiaoui , Varun Ravi Kumar , Senthil Yogamani This is my paper · ORCID

classification cs.CVeess.IV

keywords movingobjectdetectionfisheyecamerassurround-viewautonomousdrivingsemanticsegmentationtwo-streamCNNembeddedsystemsLiDARannotation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that moving object detection—deciding which visible objects around a vehicle are actually in motion—can be done directly on 360-degree fisheye surround-view images by a small CNN fast enough for embedded automotive hardware. It introduces a new dataset of fisheye parking-scene frames with moving-object masks and a two-stream network that takes two consecutive frames and learns motion without optical-flow preprocessing. The same architecture trained on ordinary rectilinear images reaches only 10% IoU (Intersection over Union, the standard overlap score) on fisheye test frames, while training on fisheye data raises moving-object IoU to about 40% and mean IoU to 69.5%, at 15 fps on a 1-teraflops embedded system. If the results hold, surround-view fisheye systems can support near-field collision-risk estimation in real time, and rectilinear-trained moving-object detection models cannot simply be ported to fisheye cameras.

What carries the argument

The load-bearing mechanism is a two-stream encoder-decoder with shared weights: two consecutive fisheye frames pass through the same lightweight encoder built from point-wise group convolutions and channel shuffling, and a three-deconvolution-layer decoder outputs a binary moving/static mask. Because the input is the frame pair itself, motion is encoded directly in the two streams and no optical-flow preprocessing or camera-specific flow distortion correction is needed. The ground-truth generation pipeline is the other supporting mechanism: LiDAR points falling inside existing object bounding boxes are classified as moving or static, projected to the image via calibration, and wrapped into convex-hull polygons; weighted cross-entropy handles the extreme class imbalance, with about 0.54% moving pixels per frame.

What would settle it

Have human annotators redraw ground-truth moving-object masks on a randomly chosen subset of the test frames and compare them to the semi-automated convex-hull labels; if the agreement is low (say, label IoU below roughly 0.8) or if most disagreements involve static pedestrians labelled as moving, then the exact 40% versus 10% IoU comparison and the augmentation gains are not trustworthy as reported.

Watch

Extended reading notes

Core claim

The central discovery is that fisheye-specific training, not generic motion cues, is what unlocks moving-object detection on surround-view cameras. Feeding two temporally sequential fisheye images into a weight-shared two-stream encoder lets the network separate ego-motion from true object motion implicitly, and this model trained on fisheye data reaches 39.8% moving-object IoU and 69.5% mIoU, whereas the same two-stream design trained on rectilinear automotive data collapses to 10% IoU when evaluated on fisheye frames. Weight sharing between the two encoders costs almost nothing (69.5% mIoU, 39.6% IoU), and augmenting with annotation-free static-only scenes nudges IoU to 42%. The authors frame this as evidence that non-linear fisheye projection geometry changes the motion pattern enough that domain-specific data is required.

Load-bearing premise

The load-bearing premise is that the semi-automated LiDAR-derived convex-hull masks are reliable ground truth; no manual verification or noise analysis is reported, and if those masks are systematically wrong, the reported IoU numbers and the claimed advantage of fisheye training collapse.

Editorial extensions

If this is right

Surround-view fisheye camera systems can run moving-object detection on embedded hardware at 15 fps without a separate optical-flow computation stage.
Moving-object detection models trained on rectilinear camera data should not be expected to transfer to fisheye cameras; fisheye-specific training data is needed.
Sharing encoder weights across the two time steps costs less than a point of IoU (39.8 to 39.6), so deployment can reuse the previous frame's features from one inference to the next.
Adding annotation-free static-only scenes raises IoU from roughly 40% to 42%, offering a cheap way to grow the training set.
The public release of the fisheye moving-object dataset gives the community a common benchmark for comparing moving-object detection methods on near-field surround-view cameras.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the rectilinear-versus-fisheye gap likely mixes projection geometry with dataset domain shift (road scenes vs. parking scenes), since the two training sets differ in both; a controlled experiment would fine-tune the rectilinear model on a small fisheye set to isolate the cause.
Beyond the paper: the convex-hull labels will under- or over-segment objects whose LiDAR points are sparse or occluded, so the reported numbers should be checked against manually drawn masks on a sample before the exact gains are taken at face value.
Beyond the paper: because the two encoders share weights, a deployable system could cache the previous frame's encoded features and run inference at higher than 15 fps with only one encoder pass per new frame, an extension the paper leaves unexplored.
Beyond the paper: although the network is trained on parking scenes, the same two-frame design may transfer to other fisheye settings such as surveillance; the paper's own observation that static pedestrians are the main false-positive source hints that appearance cues leak into the motion decision, so scene-agnostic generalization should be tested explicitly.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

FisheyeMODNet is worth knowing about mainly because it introduces the first public fisheye MOD dataset (5139 frames, parking scenes, 0.54% moving pixels) and reports a concrete transfer gap: a rectilinear MODNet drops to 10% IoU on fisheye, while their fisheye-trained model gets about 40% IoU and 69.5% mIoU at 15 fps on a 1 TOPS embedded system. That transfer gap is the meaningful new empirical result, and it is plausible. The architecture itself is ShuffleSeg with a two-stream shared encoder from MODNet; that is incremental, not a new mechanism, and the paper does not oversell it.

What the paper does well: it identifies a real gap in the literature — public datasets for surround-view fisheye MOD — and it honestly compares training-domain choices and augmentation/static-scene variants. The 42% IoU after balanced augmentation is a sensible baseline number for later work. They also document the main failure mode (static pedestrians detected as moving), which is the kind of detail that helps future authors.

The soft spots are exactly where the stress test lands. First and most important, the ground truth comes from a semi-automatic LiDAR pipeline (parse 2D boxes, select points, classify moving/static, project, convex hull) with no reported manual verification or sensitivity analysis. Because moving pixels are only 0.54% of a frame, a small false-positive rate among static pixels will dominate MOD IoU. Every number in Table 1 inherits whatever error that pipeline has. That is a load-bearing unvalidated assumption, not a minor nit. Second, the KITTI-vs-fisheye comparison (row 1 vs 2) is confounded: 1,300 KITTI samples vs 3,638 fisheye samples, and different scene types (parking vs driving). So the 10 vs 40 difference does not isolate camera geometry. A same-size, same-domain comparison would be needed to support the claim that rectilinear models fail specifically because of fisheye distortion. Third, smaller but real: the abstract says 40% IoU while Table 1 reports 39.8, and there are no error bars or statistical tests; given the tiny moving-pixel fraction, variance across scenes is likely meaningful. Finally, data and code are not released at publication time; the paper promises an "improved version" under WoodScape, so reproducibility is currently limited.

None of this kills the paper. The dataset contribution and the transfer-gap finding are worth taking seriously, but the numerical claims should be treated as provisional until the annotation pipeline is validated or the masks are released for inspection. The citation pattern is standard for this group; pointing to WoodScape as the dataset release venue is expected.

Who this is for: people working on surround-view fisheye perception, parking/urban MOD, or efficient embedded segmentation. It deserves a serious referee, but the referee should demand an annotation-quality study and a deconfounded transfer experiment before final acceptance. My recommendation: send it to review with heavy revision required, not a desk reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes FisheyeMODNet, a lightweight two-stream CNN for moving object detection (MOD) in surround-view fisheye images for autonomous driving. The authors collect a 5139-frame fisheye dataset from parking scenes, generate binary moving-object masks through a semi-automated LiDAR-based pipeline, and train a two-stream encoder with shared weights to output per-pixel moving/static labels. The central empirical claims are that a rectilinear-KITTI-trained MODNet generalizes poorly to fisheye images (10% MOD IoU), that training on the new fisheye dataset raises MOD IoU to 39.8-40%, that static-object augmentation improves it to 42%, and that the network runs at 15 fps on a 1 TOPS automotive embedded system.

Significance. If the results are reliable, the paper makes a useful contribution: it is an early attempt at MOD for 360-degree fisheye surround-view cameras, it provides a new dataset with MOD annotations, and it demonstrates a lightweight architecture with a realistic embedded-speed measurement. The design choice of feeding two sequential images directly instead of precomputed optical flow is sensible for fisheye cameras, where optical flow is distorted and requires calibration-dependent preprocessing. The weight-sharing scheme that reuses the previous encoder is a practical efficiency idea. However, the significance is conditional on the validity of the ground-truth masks and on the fairness of the rectilinear-versus-fisheye comparison; both are currently unverified. The paper also does not report variance, error bars, or statistical tests, so the small reported differences between configurations (39.6, 39.8, 42) may not be meaningful. The dataset itself is not yet publicly released, only promised as an 'improved version,' which limits immediate reproducibility.

major comments (4)

[Section 3, Semi-automated annotation procedure] The ground-truth generation pipeline is the load-bearing assumption of the entire evaluation, yet no validation is reported. The masks are produced by parsing existing 2D bounding boxes, selecting LiDAR points inside those boxes, classifying the points as moving or static, projecting them onto the fisheye image, and taking a convex hull. There is no manual verification, no inter-annotator agreement, and no sensitivity analysis of the LiDAR moving/static classifier or the ego-motion compensation. Because moving pixels constitute only 0.54% of a frame on average, even a small false-positive rate among static pixels can dominate the reported MOD IoU. The paper should validate the annotation pipeline against a manually labeled subset and report precision/recall of the generated masks, or at minimum provide a qualitative error analysis.
[Table 1, rows 1-2] The central claim that rectilinear models do not generalize to fisheye images is confounded by dataset size and scene domain. Row 1 trains on 1300 KITTI frames, row 2 trains on 3638 fisheye parking frames; the 10% versus 39.8% IoU gap could be caused by the amount of training data, the scene distribution, or object classes rather than by the camera model. A cleaner control would train the same architecture on an equal number of fisheye frames, or fine-tune the KITTI model on a matched fisheye subset. Without such a control, the conclusion 'models learnt on rectilinear cameras do not generalize well to fisheye cameras' is not established by these numbers.
[Section 4, Results and Table 1] No error bars, standard deviations, or statistical tests are reported for any of the IoU values. Given that the moving-object class is only 0.54% of the pixels, the mIoU of 69.5 is dominated by the static class, and the informative quantity is the MOD IoU, whose values (39.6, 39.8, 42) differ by only a few points. The paper should report results over multiple training runs or test splits to show that the observed improvements from augmentation are not within run-to-run noise.
[Abstract and Table 1] The abstract states 'accuracy of 40% IoU and 69.5% mIoU,' but Table 1 reports MOD IoU of 39.8 for the corresponding row and 42 for the augmented row. The headline number should match the table exactly, and the paper should clarify which configuration the abstract refers to. This inconsistency matters because the 40% IoU is the paper's most prominent quantitative claim.

minor comments (5)

[Section 4, Experimental Setup] The text says the fisheye network was trained on '3k fisheye images,' while Section 3 reports 3638 training frames; please make the numbers consistent.
[Section 4, Proposed Model] The weighted cross-entropy loss is mentioned twice, but the class weights are never specified. Reporting the exact weights would improve reproducibility.
[Section 4, Results] The 'static objects scene augmentation' row increases the training set from 3638 to 5849 samples, but the augmentation procedure is described only qualitatively. Please specify how static scenes were selected and whether the moving-object annotations were unchanged.
[Section 4, Results] The embedded platform is described only as '1 teraflops automotive embedded system.' Please name the hardware and state whether the 15 fps measurement includes both encoder passes or only the reused-encoder inference path.
[Section 3, Dataset Statistics] The paper promises to release an 'improved version' of the dataset but does not state when or under what license, and the current version is not available. A statement on data availability would strengthen the reproducibility claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the accuracy claims are held-out empirical measurements, not derivations from the inputs.

full rationale

The paper's load-bearing numbers (40% IoU, 69.5% mIoU, Table 1) are obtained by training the proposed two-stream CNN on the authors' fisheye dataset and evaluating on a held-out test split; nothing is fitted to the test set and then reported as a prediction. The Section 3 semi-automated LiDAR-to-convex-hull annotation pipeline is a validity risk because no manual verification or sensitivity analysis is reported, and the KITTI-vs-fisheye comparison in Table 1 is confounded by dataset size and scene domain, but these are experimental-validity concerns, not instances of a conclusion reducing to its inputs by construction. Self-citations (MODNet [14], WoodScape [18], ShuffleSeg [5]) supply architecture and data provenance; they are not invoked as unverified uniqueness theorems or as the logical ground for the empirical result. No fitted parameter is renamed as a prediction, and no self-definitional equation appears, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the semi-automatically generated ground truth and on the choice of a two-frame input representation. There are no new physical entities and no mathematical derivation, so the only nontrivial assumptions are empirical data-generation and training assumptions.

free parameters (2)

Weighted cross-entropy class weights
Hand-chosen weights compensate for the 0.54% moving-pixel ratio; the exact values are not reported, and the reported IoU depends on them.
Training frame count and static augmentation = 3638 and 5849 frames
The main result uses 3638 fisheye frames; adding 2211 static-only frames raises MOD IoU from 39.8 to 42. This is an experimental design choice, not a fitted constant.

assumptions (4)

domain assumption Pre-existing object bounding boxes and LiDAR points inside them correctly identify each object's motion state.
Section 3 annotation pipeline relies on this for ground truth; no manual quality check is reported.
domain assumption Camera-LiDAR calibration and projection are accurate enough to create pixel-level convex hull masks.
Section 3 projects LiDAR points to image coordinates using camera calibration; errors would directly corrupt training labels.
domain assumption Two temporally adjacent frames contain sufficient motion information for the network to separate ego-motion from object motion.
Section 4 uses two sequential images as the only motion cue; no optical flow or geometric compensation is provided.
domain assumption ImageNet pretrained weights provide a useful initialization for fisheye input.
Section 4 states the encoder is initialized with ImageNet weights, which is standard but unverified for this distortion-heavy domain.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FisheyeMODNet: Moving Object detection on Surround-view Cameras for Autonomous Driving." pith.science (2026). https://pith.science/paper/Q2A6WXXB

@misc{pith2026190811789,
  author       = {Pith},
  title        = {Pith review of: FisheyeMODNet: Moving Object detection on Surround-view Cameras for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2A6WXXB}},
  note         = {Machine review of arXiv:1908.11789}
}

read the original abstract

Moving Object Detection (MOD) is an important task for achieving robust autonomous driving. An autonomous vehicle has to estimate collision risk with other interacting objects in the environment and calculate an optional trajectory. Collision risk is typically higher for moving objects than static ones due to the need to estimate the future states and poses of the objects for decision making. This is particularly important for near-range objects around the vehicle which are typically detected by a fisheye surround-view system that captures a 360{\deg} view of the scene. In this work, we propose a CNN architecture for moving object detection using fisheye images that were captured in autonomous driving environment. As motion geometry is highly non-linear and unique for fisheye cameras, we will make an improved version of the current dataset public to encourage further research. To target embedded deployment, we design a lightweight encoder sharing weights across sequential images. The proposed network runs at 15 fps on a 1 teraflops automotive embedded system at accuracy of 40% IoU and 69.5% mIoU.

Figures

Figures reproduced from arXiv: 1908.11789 by the authors.

**Figure 2.** Network Architecture adapted from ShuffleSeg base network. Two sequential images encoding the motion infor [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

Briot, P

A. Briot, P. Viswanath, and S. Yogamani. Analysis of efﬁ- cient cnn design techniques for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018. 1

work page 2018
[2]

Chennupati, G

S. Chennupati, G. Sistu, S. Yogamani, and S. A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion Workshops, 2019. 1

work page 2019
[3]

J. C. Clarke and A. Zisserman. Detection and tracking of in- dependent motion. Image and Vision Computing, 14(8):565– 572, 1996. 2

work page 1996
[4]

S. Dey, V . Reilly, I. Saleemi, and M. Shah. Detection of independently moving objects in nonplanar scenes via multi- frame monocular epipolar constraint. In Proceedings of the European Conference on Computer Vision, pages 860–873,

work page
[5]

ShuffleSeg: Real-time Semantic Segmentation Network

M. Gamal, M. Siam, and M. Abdel-Razek. Shufﬂeseg: Real-time semantic segmentation network. arXiv preprint arXiv:1803.03816, 2018. 2, 3

work page Pith review arXiv 2018
[6]

Heimberger, J

M. Heimberger, J. Horgan, C. Hughes, J. McDonald, and S. Yogamani. Computer vision in automated parking sys- tems: Design, implementation and challenges. Image and Vision Computing, 68:88–101, 2017. 2

work page 2017
[7]

Horgan, C

J. Horgan, C. Hughes, J. McDonald, and S. Yogamani. Vision-based driver assistance systems: Survey, taxonomy and advances. In 2015 IEEE 18th International Confer- ence on Intelligent Transportation Systems , pages 2032–

work page 2015
[8]

S. D. Jain, B. Xiong, and K. Grauman. Fusionseg: Learn- ing to combine motion and appearance for fully automatic segmention of generic objects in videos. arXiv preprint arXiv:1701.05384, 2017. 1, 3

arXiv 2017

Show all 18 references

[9]

Klappstein, F

J. Klappstein, F. Stein, and U. Franke. Monocular motion de- tection using spatial constraints in a uniﬁed manner. In Pro- ceedings of the IEEE Intelligent Vehicles Symposium, pages 261–267, 2006. 2

2006
[10]

Rashed, A

H. Rashed, A. El Sallab, S. Yogamani, and M. ElHelw. Mo- tion and depth augmented semantic segmentation for au- tonomous navigation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition Work- shops, pages 0–0, 2019. 1

2019
[11]

M. Siam, S. Eikerdawy, M. Gamal, M. Abdel-Razek, M. Jagersand, and H. Zhang. Real-time segmentation with appearance, motion and geometry. In 2018 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 5793–5800, Oct 2018. 3

2018
[12]

M. Siam, S. Elkerdawy, M. Jagersand, and S. Yogamani. Deep semantic segmentation for automated driving: Taxon- omy, roadmap and challenges. In 2017 IEEE 20th Inter- national Conference on Intelligent Transportation Systems (ITSC), pages 1–8. IEEE, 2017. 1

2017
[13]

M. Siam, M. Gamal, M. Abdel-Razek, S. Yogamani, and M. Jagersand. Rtseg: Real-time semantic segmentation com- parative study. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 1603–1607. IEEE, 2018. 1

2018
[14]

M. Siam, H. Mahgoub, M. Zahran, S. Yogamani, M. Jager- sand, and A. El-Sallab. Modnet: Motion and appearance based moving object detection network for autonomous driv- ing. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) , pages 2859–2864. IEEE,

2018
[15]

Sistu, I

G. Sistu, I. Leang, S. Chennupati, S. Milz, S. Yogamani, and S. Rawashdeh. Neurall: Towards a uniﬁed model for visual perception in automated driving. arXiv preprint arXiv:1902.03589, 2019. 1

1902 arXiv
[16]

Uric ´ar, D

M. Uric ´ar, D. Hurych, P. Kr ´ızek, and S. Yogamani. Chal- lenges in designing datasets and validation for autonomous driving. In Proceedings of the 14th International Joint Con- ference on Computer Vision, Imaging and Computer Graph- ics Theory and Applications, VISIGRAPP 20...

2019
[17]

H. Wang, P. Wang, and X. Qian. MPNET: An end-to- end deep neural network for object detection in surveillance video. IEEE Access, 6:30296–30308, 2018. 2

2018
[18]

Yogamani, C

S. Yogamani, C. Hughes, J. Horgan, G. Sistu, P. Varley, D. O’Dea, M. Uric ´ar, S. Milz, M. Simon, K. Amende, et al. Woodscape: A multi-task, multi-camera ﬁsheye dataset for autonomous driving. arXiv preprint arXiv:1905.01489,

1905 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Briot, P

A. Briot, P. Viswanath, and S. Yogamani. Analysis of efﬁ- cient cnn design techniques for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018. 1

work page 2018

[2] [2]

Chennupati, G

S. Chennupati, G. Sistu, S. Yogamani, and S. A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion Workshops, 2019. 1

work page 2019

[3] [3]

J. C. Clarke and A. Zisserman. Detection and tracking of in- dependent motion. Image and Vision Computing, 14(8):565– 572, 1996. 2

work page 1996

[4] [4]

S. Dey, V . Reilly, I. Saleemi, and M. Shah. Detection of independently moving objects in nonplanar scenes via multi- frame monocular epipolar constraint. In Proceedings of the European Conference on Computer Vision, pages 860–873,

work page

[5] [5]

ShuffleSeg: Real-time Semantic Segmentation Network

M. Gamal, M. Siam, and M. Abdel-Razek. Shufﬂeseg: Real-time semantic segmentation network. arXiv preprint arXiv:1803.03816, 2018. 2, 3

work page Pith review arXiv 2018

[6] [6]

Heimberger, J

M. Heimberger, J. Horgan, C. Hughes, J. McDonald, and S. Yogamani. Computer vision in automated parking sys- tems: Design, implementation and challenges. Image and Vision Computing, 68:88–101, 2017. 2

work page 2017

[7] [7]

Horgan, C

J. Horgan, C. Hughes, J. McDonald, and S. Yogamani. Vision-based driver assistance systems: Survey, taxonomy and advances. In 2015 IEEE 18th International Confer- ence on Intelligent Transportation Systems , pages 2032–

work page 2015

[8] [8]

S. D. Jain, B. Xiong, and K. Grauman. Fusionseg: Learn- ing to combine motion and appearance for fully automatic segmention of generic objects in videos. arXiv preprint arXiv:1701.05384, 2017. 1, 3

arXiv 2017