Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Reliable Real Time Ball Tracking for Robot Table Tennis

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims robot table tennis ball tracking can be made reliable in real time by triangulating every camera pair and keeping the largest set of observations that agree on one 3D position, eliminating the need for hand-built outlier…

desk verdict A solid, useful engineering contribution to robot table tennis whose central reliability claim is overstated: the multi-camera consistency check handles independent detection errors well but not correlated false positives like a ball-colored arm, and the paper's own simulation hints at this. read the letter →

arxiv 1908.07332 v1 pith:PNK3PXLM submitted 2019-08-20 cs.RO

Sebastian Gomez-Gonzalez , Yassine Nemmour , Bernhard Schölkopf , Jan Peters This is my paper · ORCID
classification cs.RO
keywords robottabletennisreal-timeballtrackingmulti-camerastereovisionoutlierrejectionsemanticsegmentationconvolutionaldetectionconsensusestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a table tennis ball can be tracked reliably in real time by treating each camera's detection as untrustworthy and relying on multi-camera agreement rather than physics or shape assumptions. The proposed system, RT2, detects the ball in each image with a fast 5x5 convolutional semantic-segmentation unit, then finds the 3D position by triangulating every camera pair and keeping the largest set of observations that agree on a single point. In simulation, the system becomes more accurate and less failure-prone as cameras are added, and on the real four-camera robot its errors stay below 10 centimeters while the previous RTBlob tracker still produces large outliers. The payoff is that a complete robot table tennis policy can remove all its hand-made outlier filters and still match its earlier success rate, freeing researchers to concentrate on policy learning rather than vision cleanup.

What carries the argument

The mechanism that carries the argument is consensus-based outlier rejection implemented in Algorithm 2. From the 2D detections of c calibrated cameras, the algorithm considers every camera pair, triangulates a candidate 3D point, reprojects it through the remaining cameras, and counts how many observations fall within a pixel error threshold epsilon. The largest such set is treated as the inlier set and used for the final 3D estimate. This makes the consistency check, not any physical model, the source of reliability; the paper pairs it with a detection stage using a single $5\times5$ convolutional semantic-segmentation unit, whose probability image is thresholded and flood-filled in Algorithm 1 to locate the ball about 50 times faster than an SSD/MobilNet detector.

What would settle it

Project a second ball-colored object into several camera views at geometrically consistent pixel positions; if the largest consistent subset includes these false observations and the reported 3D position follows the decoy, the paper's core assumption is wrong. In the authors' own simulation, the same test can be run by replacing random per-camera outliers with projections of one fixed false point and checking whether Algorithm 2 returns that point.

Watch

Extended reading notes

Core claim

The central discovery is that a multi-camera tracker can be made reliable without modeling the ball's physics or shape by turning outlier rejection into a consensus problem. Given the 2D ball positions reported by c calibrated cameras, Algorithm 2 triangulates a candidate 3D position from every camera pair, reprojects that candidate into all cameras, and keeps the largest subset of observations whose reprojection error falls below a threshold epsilon. That subset is then used for the final 3D estimate. The paper shows in simulation that this largest-consistent-subset rule reduces both position error and failure probability as the number of cameras grows, and in the real four-camera setup it yields a unimodal error distribution with no reported position off by more than 10 centimeters, in contrast to the RTBlob baseline's multimodal errors. It further shows that an existing table tennis policy can drop all of its outlier-rejection heuristics and still achieve roughly the same success rate, indicating that the vision system, not the policy, can carry the reliability burden.

Load-bearing premise

The system assumes that wrong ball detections in different cameras will not line up behind a single 3D point, so a false set of observations cannot pass the consistency check.

Editorial extensions

If this is right

  • A table tennis policy can remove all of its hand-designed outlier-rejection heuristics and still match its prior success rate, because the vision system itself reports the largest consistent set of observations.
  • Adding more cameras improves both accuracy and failure rate in simulation: with 30 cameras and a 50 percent per-camera outlier probability, the system reports errors around 0.35 centimeters and never fails, whereas four cameras fail 37.7 percent of the time at that outlier rate.
  • Real-time operation is preserved: the sequential implementation of the consensus search takes about 3 milliseconds for 30 cameras, enough for over 200 observations per second.
  • On the real four-camera platform, the RT2 error distribution is unimodal below 10 centimeters, while RTBlob produces a second error mode near 25 to 30 centimeters and occasional errors of tens of meters, which is why the older system needs extra filters.
  • The detection stage runs roughly 50 times faster with a single $5\times5$ convolutional unit than with an SSD/MobilNet detector at 200 by 200 resolution, making high-frequency tracking possible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension left implicit: the consistency argument applies to any multi-camera tracker whose per-camera detector occasionally commits outliers, so the same Algorithm 2 could be retrained for other small objects or other sports without changing the 3D estimation stage.
  • A testable consequence: accuracy should continue to improve with camera count until the consensus search becomes a bottleneck; the paper's $O(c^3)$ runtimes suggest a parallel implementation could extend the real-time regime beyond 30 cameras.
  • If detection errors were correlated across cameras, for example a skin-colored arm visible in several views, the largest consistent subset could lock onto a false point; injecting such correlated outliers in simulation would directly stress the load-bearing assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents RT2, a real-time vision system for robot table tennis. Detection is done by a single-convolutional-unit semantic segmentation network that outputs a probability image, followed by a threshold-based connected-component extraction (Algorithm 1). The 3D position is estimated by Algorithm 2, which enumerates all camera pairs, triangulates each pair, and retains the largest subset of cameras whose re-projection error is below a tolerance. The system is evaluated in simulation with varying camera counts and outlier probabilities, on a real four-camera robot platform against the RTBlob baseline using an end-effector-mounted ball as ground truth, and in a robot table tennis policy from prior work with all outlier-filtering heuristics removed. The central claim is that multi-camera consistency allows the system to find and discard object-detection errors, improving accuracy and robustness as the number of cameras grows.

Significance. If the central claim holds, this is a practically useful contribution: it is open-source, modular with respect to camera count, offers real-time performance up to 30 cameras, and removes the need for bespoke outlier filters in downstream policies. The real-system comparison against RTBlob shows a clearly more concentrated error distribution, and the simulation does demonstrate improved robustness against independent random outlier noise when more cameras are added. However, the paper's load-bearing assumption is only validated for uncorrelated outliers; mutually consistent false positives from a real 3D object are not tested, and one simulation cell (8 cameras at 50% outliers) contradicts the stated monotone improvement claim. The evidence base is therefore short of what the abstract promises.

major comments (3)
  1. [§2.2 (Algorithm 2)] The load-bearing assumption stated in §2.2, that 'it should be hard to find a single 3D position that explains a set of pixel observations containing outliers', fails when outliers are projections of a real 3D point other than the ball. Figure 3 shows that the human arm is the only non-dark object in the probability image, so a false detection on the arm is a genuine 3D point whose projections across cameras are mutually consistent. If more cameras detect the arm than the ball, Algorithm 2 will select the arm's 3D position and report it as the ball. The simulation in §3.1 generates outliers as random independent pixel positions, which are almost never multi-view consistent, so it tests only the best case. The abstract's claim that the system can find and discard detection errors is therefore not established for correlated false positives; please add experiments with a second consistent 3D source, either simulated or by including a ball-colored object visible to multiple cameras in the real setup.
  2. [Table 1 (§3.1)] The claim that accuracy improves monotonically with the number of cameras is contradicted by the 50% outlier row: with 4 cameras the error is 4.67 cm, while with 8 cameras it is 6.84 cm. The authors note the failure rates differ (37.7% versus 4.5%), but reporting error conditioned on non-failure makes the comparison misleading because the 4-camera result is heavily selected by the 'did not fail' condition. Please report an unconditional error measure (e.g., expected squared distance treating failure as a large error) or otherwise explain why the conditional error is an appropriate summary, and address the non-monotonicity directly.
  3. [Figure 4 (§3.2)] The histogram comparison with RTBlob is the main real-platform evidence for the robustness claim, but the figure and text give no information about the number of samples, the duration of the data collection, or confidence intervals. The statements that RT2 'never reported any ball position whose error was larger than 10 cm' and that RTBlob has a 0.1% probability of errors on the order of tens of meters are point estimates without uncertainty. Please report sample counts and provide a statistical comparison of error means or tail probabilities.
minor comments (5)
  1. [Algorithm 2] In Algorithm 2, line 6, the loop variable k shadows the input size k; rename one of them to avoid confusion.
  2. [§3.1] The sentence 'the outlier rate of the image processing algorithms is below 1% in practice' is stated without supporting measurement or citation; please provide data or soften the claim.
  3. [§3.2] The name RT2 is introduced only in the caption of Figure 4; define it in the text near its first use.
  4. [§2.1] The architecture name 'Mobilnet' should be written 'MobileNet' for consistency with the reference.
  5. [Table 1] The table caption reports error and failure probability but does not state the number of simulation runs per cell; adding this would help assess the stability of the entries, especially the 50% outlier row.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central reliability claim is validated against external ground truth and an independent baseline.

full rationale

The paper's core claim that multi-camera consistency can discard object-detection errors is not circular. Algorithm 2 is a largest-consistent-subset robust estimator, and its output is evaluated against independent ground truth: simulated ball positions in Section 3.1 and robot kinematics with the ball attached to the end effector in Section 3.2. The proposed RT2 system is also compared with the RTBlob baseline, and the real-robot error distribution is reported empirically. No parameter is fitted to the target quantity and then renamed as a prediction; the consistency threshold epsilon is part of the algorithm's definition, not a fitted output. The only self-references are to the authors' prior policy [5] used as an evaluation harness and to RTBlob [15] as a baseline. These are not load-bearing for the vision accuracy claim, and the comparisons are empirical rather than definitional. The Section 2.2 assumption that outliers are hard to explain by a single 3D position is a correctness assumption, and the simulation's random-pixel outlier model matches that assumption by design; however, the real-robot evaluation does not depend solely on the simulation, so any limitation regarding correlated outliers is a robustness concern, not circularity. The paper is self-contained against external benchmarks, and the score is therefore 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the geometric inconsistency of outlier detections across cameras, plus hand-set threshold parameters. No new physical entities are introduced. The thresholds Th, Tl, and epsilon are algorithm inputs rather than fitted constants, but their values are not justified in the paper.

free parameters (3)
  • high threshold Th
    Used in Algorithm 1 to decide whether a ball is present; no tuning or sensitivity analysis is reported in the paper.
  • low threshold Tl
    Used in Algorithm 1 to expand the ball pixel set via breadth-first search; the value is not specified in the text.
  • pixel error tolerance epsilon
    Used in Algorithm 2 to decide which observations are consistent; no procedure for choosing epsilon is given, and the simulation and real-system results depend on it.
assumptions (4)
  • domain assumption Outlier pixel observations from object detection are hard to explain by a single 3D point
    Stated in Section 2.2 as the underlying assumption for Algorithm 2; if false, the consistency test can accept a wrong 3D position.
  • domain assumption Camera calibration provides accurate projection matrices and pixel re-projection errors
    Section 2.2 assumes access to project and stereo functions from a stereo vision library; the real-system evaluation relies on calibration accuracy.
  • domain assumption The single 5x5 convolutional unit learns a probability map separating the ball from background
    Section 2.1 uses a single convolutional unit and reports that it works; the paper does not characterize generalization beyond the evaluated setup.
  • domain assumption Detector errors are not systematically correlated across cameras
    The consistency test assumes correlated errors do not form false inlier sets; the 50 percent outlier simulation results in Table 1 suggest this assumption can fail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reliable Real Time Ball Tracking for Robot Table Tennis." pith.science (2026). https://pith.science/paper/PNK3PXLM

@misc{pith2026190807332,
  author       = {Pith},
  title        = {Pith review of: Reliable Real Time Ball Tracking for Robot Table Tennis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNK3PXLM}},
  note         = {Machine review of arXiv:1908.07332}
}
read the original abstract

Robot table tennis systems require a vision system that can track the ball position with low latency and high sampling rate. Altering the ball to simplify the tracking using for instance infrared coating changes the physics of the ball trajectory. As a result, table tennis systems use custom tracking systems to track the ball based on heuristic algorithms respecting the real time constrains applied to RGB images captured with a set of cameras. However, these heuristic algorithms often report erroneous ball positions, and the table tennis policies typically need to incorporate additional heuristics to detect and possibly correct outliers. In this paper, we propose a vision system for object detection and tracking that focus on reliability while providing real time performance. Our assumption is that by using multiple cameras, we can find and discard the errors obtained in the object detection phase by checking for consistency with the positions reported by other cameras. We provide an open source implementation of the proposed tracking system to simplify future research in robot table tennis or related tracking applications with strong real time requirements. We evaluate the proposed system thoroughly in simulation and in the real system, outperforming previous work. Furthermore, we show that the accuracy and robustness of the proposed system increases as more cameras are added. Finally, we evaluate the table tennis playing performance of an existing method in the real robot using the proposed vision system. We measure a slight increase in performance compared to a previous vision system even after removing all the heuristics previously present to filter out erroneous ball observations.

Figures

Figures reproduced from arXiv: 1908.07332 by the authors.

Figure 1
Figure 1. Robot table tennis setup used to evaluate the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Ball detection with a Mobilnet deep network [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ball detection using a single convolutional unit [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Histogram of the error of the presented vision [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages

  1. [1]

    https://gitlab.tuebingen.mpg.de/sgomez/ball_ tracking

    Open source implementation of the ball tracking sys- tem. https://gitlab.tuebingen.mpg.de/sgomez/ball_ tracking

    work page

  2. [2]

    B. Chen D. Kalenichenko A. Howard, M. Zhu et al. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861 , 2017

    arXiv 2017

  3. [3]

    Maddison D

    C. Maddison D. Silver, A. Huang et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

    work page 2016

  4. [4]

    Opalach G

    A. Opalach G. Pingali and Y. Jean. Ball tracking and virtual replays for innovative tennis broadcasts. In In- ternational Conference on Pattern Recognition , volume 4, pages 152–156. IEEE, 2000

    work page 2000

  5. [5]

    Gomez-Gonzalez, G

    S. Gomez-Gonzalez, G. Neumann, B. Sch¨ olkopf, and J. Peters. Adaptation and robust learning of probabilistic movement primitives. ArXiv e-prints, August 2018

    work page 2018

  6. [6]

    Sch¨ olkopf Gomez-Gonzalez S., G

    B. Sch¨ olkopf Gomez-Gonzalez S., G. Neumann and J. Pe- ters. Using probabilistic movement primitives for strik- ing movements. In International Conference of Humanoid Robots, pages 502–508. IEEE, 2016

    work page 2016

  7. [7]

    Y. Chen W. Tsai H. Chen, M. Tien and S. Lee. Physics- based ball tracking and 3d trajectory reconstruction with applications to shooting location estimation in basketball video. Journal of Visual Communication and Image Repre- sentation, 20(3):204–216, 2009

    work page 2009

  8. [8]

    Kuniyoshi I

    Y. Kuniyoshi I. Noda H. Kitano, M. Asada and E. Osawa. Robocup: The robot world cup initiative. In International conference on Autonomous agents , pages 340–347. ACM, 1997

    work page 1997

Show all 22 references
  1. [9]

    L. Lou K. K¨ uhnlenz H. Li, H. Wu and O. Ravn. Ping- pong robotics with high-speed vision system. In Inter- national Conference on Control Automation Robotics & Vi- sion, pages 106–111. IEEE, 2012

    2012

  2. [10]

    B. Wang Y. Zhou H. Liu, Z. Li and Q. Zhang. Table ten- nis robot with stereo vision and humanoid manipulator ii: Visual measurement of motion-blurred ball. In Inter- national Conference on Robotics and Biomimetics , pages 2430–2435. IEEE, 2013

    2013

  3. [11]

    Heyden and M

    A. Heyden and M. Pollefeys. Multiple view geometry. Emerging topics in computer vision , pages 45–107, 2005

    2005

  4. [12]

    C. Sun M. Zhu J. Huang, V. Rathod et al. Speed/accuracy trade-offs for modern convolutional ob- ject detectors. In IEEE CVPR, 2017

    2017

  5. [13]

    J. Kober K. M¨ ulling and J. Peters. A biomimetic approach to robot table tennis. Adaptive Behavior , 19(5):359–376, 2011

    2011

  6. [14]

    Kroemer K

    O. Kroemer K. M¨ ulling, J. Kober and J. Peters. Learning to select and generalize striking movements in robot ta- ble tennis. The International Journal of Robotics Research, 32(3):263–279, 2013

    2013

  7. [15]

    Lampert and J

    C. Lampert and J. Peters. Real-time detection of col- ored objects in multiple camera streams with off-the-shelf hardware components. Journal of Real-Time Image Pro- cessing, 7(1):31–41, 2012

    2012

  8. [16]

    A.J. Hoane M. Campbell and F. Hsu. Deep blue. Artificial intelligence, 134(1-2):57–83, 2002

    2002

  9. [17]

    Mathias T

    M. Mathias T. Pohlen, A. Hermans and B. Leibe. Full- resolution residual networks for semantic segmentation in street scenes. arXiv preprint, 2017

    2017

  10. [18]

    D. Erhan C. Szegedy S. Reed C. Fu W. Liu, D. Anguelov and A. Berg. Ssd: Single shot multibox detector. In Euro- pean conference on computer vision, pages 21–37. Springer, 2016

    2016

  11. [19]

    W. Wan M. Zhou Z. Yu X. Chen, Q. Huang et al. A robust vision module for humanoid robotic ping-pong game. International Journal of Advanced Robotic Systems , 12(4):35, 2015

    2015

  12. [20]

    H. Lu X. Tong and Q. Liu. An effective and fast soccer ball detection and tracking method. In International Con- ference on Pattern Recognition, volume 4, pages 795–798. IEEE, 2004

    2004

  13. [21]

    Scholkopf Y

    B. Scholkopf Y. Huang and J. Peters. Learning optimal striking points for a ping-pong playing robot. In IEEE International Conference on Intelligent Robots and Systems (IROS), pages 4587–4592. IEEE, 2015

    2015

  14. [22]

    H. Kim Y. Seo, S. Choi and K. Hong. Where are the ball and players? soccer game analysis with color-based track- ing and image mosaick. In International Conference on Image Analysis and Processing , pages 196–203. Springer, 1997

    1997

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.