Pith. sign in

REVIEW 39 references

pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild

T0 review · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new dataset and Gymnasium environment enable detection of pool balls from arbitrary-view photos and simulated shot selection, where a geometry-based Oracle outperforms standard RL agents.

arxiv 2504.12045 v1 pith:GTZB457A submitted 2025-04-16 cs.CV cs.LG

Jonas Myhre Schi{o}tt , Viktor Sebastian Petersen , Dim P. Papadopoulos This is my paper · ORCID
classification cs.CVcs.LG
keywords poolballballsshotsingletabletaskbaseline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Imagine taking a photo of a pool table from an unusual camera position and having a program point out where to hit the cue ball. The authors build this in two steps. First, they collect 195 images of 8-ball matches from online videos, annotate all balls and the diamond-shaped markers on the table rails, and fine-tune a YOLO object detector. From the marker detections they estimate a homography that maps the photo onto a top-down template, which gives them the ball positions in table coordinates. They report a mean location error of 0.4 cm between different camera views of the same situation.

Second, they create a simulated 8-ball environment using the Gymnasium library and the Pymunk physics engine. The cue ball is always hit in the center, collisions are perfectly elastic, and there is no spin. They try standard reinforcement learning agents (PPO, TD3, A2C, DDPG, SAC) and find that none can pocket all balls without a foul in the full-table setting. They then introduce 'hitpoints', a geometric calculation of the aiming points that pocket a ball, optionally including bank and kick shots via mirrored tables. Their Oracle baseline, which picks the best hitpoint, reaches a per-shot success rate of 94.7% and clears an entire table in one turn 30% of the time in the simulator.

The headline numbers are all measured inside this simplified simulator, not on a real pool table. The paper does not show that the same shot suggestions would work in reality, and it does not ship code or data in the text. Its value today is as a benchmark and dataset for future work.

Extended reading notes

Core claim

The paper's central load-bearing assertion is that from a single image in the wild, pix2pockets can detect balls and table with AP50 91.2 and ball-location error 0.4 cm, and that in a simplified 8-ball simulator the Oracle baseline selects shots with 94.7% per-shot success and clears a full table 30% of the time, while standard RL algorithms fail. If true, this means the dataset and environment are usable benchmarks for sports-vision and RL shot selection.

Load-bearing premise

The most fragile premise is that the simplified Pymunk physics, stated in Sec 4.2 ('we choose to simplify the physics as much as possible... the cue ball is always hit in its center and every collision is perfectly elastic'), is an adequate model of real 8-ball pool, so that simulator success rates (94.7% per-shot, 30% full turn) transfer to real tables. This is never validated against physical shots, and spin, friction, and imperfect collisions are all absent; if real-world dynamics differ, the shot suggestions from the Oracle could fail even though the simulation says they succeed.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The shot-suggestion results rest on hand-designed reward coefficients, a cushion penalty of 0.33, a chosen action discretization, and an unformalized ball-center interpolation. The simulator's simplified physics (perfectly elastic collisions, no spin) is a domain assumption that is not validated against real play. Detection and ball location rely on the diamond-system layout and manual annotations. No new physical entities are introduced.

free parameters (4)
  • Reward function coefficients = +100/-100 win/loss, +10 ball hit, +50 pocket, r(v)=1000/(v+10)-50, r(d)=-50d/D, -80 scratch
    Hand-designed reward shaping in Sec 5.2; affects RL baseline success rates and the 'all RL fail' claim.
  • Cushion penalty multiplier = 0.33 per cushion hit
    Introduced for mirror-table bank/kick shots in Sec 5.2; shifts the Oracle's shot choice and success rate.
  • Action discretization = 36,000 angles (0.01 deg), 30 powers
    Chosen as environment hyperparameters in Sec 5.2; the 0.01-degree precision is stated as necessary for 2-ball success and affects all success rates.
  • Ball-center interpolation = linear between bounding-box center and top point based on camera angle (formula not given)
    Modeling choice in Sec 4.1 that directly affects the 0.4 cm ball-location error estimate.
assumptions (4)
  • ad hoc to paper The simplified Pymunk physics (perfectly elastic collisions, center hits, no spin) sufficiently captures real 8-ball pool outcome dynamics for shot suggestions.
    Sec 4.2 states the physics is simplified; none of the paper's shot-success claims are validated against real play.
  • domain assumption The Diamond System dot pattern appears on all pool tables and can be detected with enough accuracy for homography estimation.
    Sec 3 uses dot annotations as table-boundary markers; in-the-wild images with occluded or nonstandard tables would break the pipeline.
  • domain assumption The 0.4 cm mean projection error measured on 25 multi-view situations is representative of the system's location accuracy on arbitrary user images.
    Sec 5.1 computes projection error on a small controlled set from championship videos, not on arbitrary in-the-wild photos.
  • domain assumption Manual annotations of balls and dots are accurate enough to serve as training and evaluation ground truth.
    Sec 3 relies entirely on manual Roboflow annotations; annotation accuracy is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild." pith.science (2026). https://pith.science/paper/GTZB457A

@misc{pith2026250412045,
  author       = {Pith},
  title        = {Pith review of: pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTZB457A}},
  note         = {Machine review of arXiv:2504.12045}
}
read the original abstract

Computer vision models have seen increased usage in sports, and reinforcement learning (RL) is famous for beating humans in strategic games such as Chess and Go. In this paper, we are interested in building upon these advances and examining the game of classic 8-ball pool. We introduce pix2pockets, a foundation for an RL-assisted pool coach. Given a single image of a pool table, we first aim to detect the table and the balls and then propose the optimal shot suggestion. For the first task, we build a dataset with 195 diverse images where we manually annotate all balls and table dots, leading to 5748 object segmentation masks. For the second task, we build a standardized RL environment that allows easy development and benchmarking of any RL algorithm. Our object detection model yields an AP50 of 91.2 while our ball location pipeline obtains an error of only 0.4 cm. Furthermore, we compare standard RL algorithms to set a baseline for the shot suggestion task and we show that all of them fail to pocket all balls without making a foul move. We also present a simple baseline that achieves a per-shot success rate of 94.7% and clears a full game in a single turn 30% of the time.

Figures

Figures reproduced from arXiv: 2504.12045 by the authors.

Figure 1
Figure 1. pix2pockets. We introduce a new task for shot suggestions in pool games using a single input image. First, we detect the table and estimate the position of the balls. Then, they are fed into a pool environment, and a Reinforcement Learning agent predicts the best available shot (i.e., cue angle and shot power). Several papers have proposed to detect the balls and edges on the pool ta￾ble using a restricted setup [33… view at source ↗
Figure 2
Figure 2. Our Dataset. (a) It contains 195 annotated images of tables captured from various angles with diverse lighting conditions. (b) We annotate 5748 ob￾jects with accurate segmentation masks. The maximum number of objects in the image varies from class to class. (c) Bounding box annotated examples. Note how sometimes the balls are not completely visible from the given view. 3 Dataset We present our dataset collected to t… view at source ↗
Figure 3
Figure 3. Full pipeline. The input image Iin is run through the Ball Location Model to estimate the ball positions on the table, which is then handed to the Shot Suggestion Model. First, we obtain the dot detections d and the ball detections b on Iin. We use d to find the table lines and thus estimate a mapping H from Iin to a template T. Then, we use H to estimate the center point p for the balls b, resulting in the position… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Training size. The AP50 of models with different training set sizes, showing diminishing gains after 80 images. (b) Shot Accuracy. To determine the shot precision, we test the performance for different σ values. In the 1-Ball environment, a precision of 0.25 degree…
Figure 5
Figure 5. Figure 5: Projection error. To estimate the projection error, the front-view and 45-view projections are compared to the top-view ground truth. The projection result is shown on the top-view image for accuracy assessment. The blue lines indicate the distance from the estimated c…
Figure 6
Figure 6. Figure 6: Example of shot suggestions. The first column shows two initial states. The second, third, and fourth columns show the action predictions of TD3, Masked PPO, and the Oracle, respectively. Each action image is highlighted, indicating whether the shot was successful (gre…
Figure 7
Figure 7. Figure 7: Mirror table. To include kick and bank shots, we establish an approach to mirror the balls and pockets to a separate space on each table side. Aiming for a ball or pocket in the mirrored space M is equivalent to hitting a cushion and then a ball on the real table. The …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages

  1. [1]

    Alaniz, S.: Deep reinforcement learning with model learning and monte carlo tree search in minecraft (2018)

    work page 2018

  2. [2]

    Alciatore, D.G.: Break statistics (2024)

    work page 2024

  3. [3]

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym (2016)

    2016

  4. [4]

    In: Proceedings of the 8th International Conference on Pattern Recog- nition Applications and Methods

    Burić, M., Pobar, M., Ivašić-Kos, M.: Adapting yolo network for ball and player detection. In: Proceedings of the 8th International Conference on Pattern Recog- nition Applications and Methods. vol. 1, pp. 845–851 (2019)

    work page 2019

  5. [5]

    In: ROVISP 2003: Proc

    Chua, S.C., Wong, E.K., Koo, V.C.: Pool balls identification and calibration for a pool robot. In: ROVISP 2003: Proc. Intl. Conf. Robotics, Vision, Information and Signal Processing. pp. 312–315 (2003)

    work page 2003

  6. [6]

    Image Rochester NY (2010)

    Derpanis, K.G.: Overview of the ransac algorithm. Image Rochester NY (2010)

    work page 2010

  7. [7]

    In: Proceedings of the IEEE international conference on computer vision

    Felsen, P., Agrawal, P., Malik, J.: What will happen next? forecasting player moves in sports videos. In: Proceedings of the IEEE international conference on computer vision. pp. 3342–3351 (2017)

    work page 2017

  8. [8]

    CoRR (2018)

    Fujimoto, S., van Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. CoRR (2018)

    work page 2018

Show all 39 references
  1. [9]

    Kuwait Journal of Science 45(1) (2018)

    Gao, J., He, Q., Gao, H., Zhan, Z., Wu, Z.: Design of an efficient multi-objective recognition approach for 8-ball billiards vision system. Kuwait Journal of Science 45(1) (2018)

    2018

  2. [10]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

    Giancola, S., Amine, M., Dghaily, T., Ghanem, B.: Soccernet: A scalable dataset for action spotting in soccer videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 1711–1721 (2018)

    2018

  3. [11]

    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor (2018)

    2018

  4. [12]

    Hsu, C.C., Tsai, H.C., Chen, H.T., Tsai, W.J., Lee, S.Y.: Computer-assisted billiard self-training using intelligent glasses. pp. 119–126 (2017). https://doi.org/10.1109/ ISPAN-FCST-ISCC.2017.36

    2017

  5. [13]

    IEEE Transactions on Multimedia8(4), 749–760 (2006)

    Huang, C.L., Shih, H.C., Chao, C.Y.: Semantic analysis of soccer video using dy- namic bayesian network. IEEE Transactions on Multimedia8(4), 749–760 (2006)

    2006

  6. [14]

    The International FLAIRS Conference Proceedings 35 (May 2022), http://dx.doi.org/10.32473/flairs.v35i.130584

    Huang, S., Ontañón, S.: A closer look at invalid action masking in policy gradient algorithms. The International FLAIRS Conference Proceedings 35 (May 2022), http://dx.doi.org/10.32473/flairs.v35i.130584

    2022 doi

  7. [15]

    Journal of Intel- ligent and Robotic Systems50, 399–417 (12 2007)

    Landry, J.F., Dussault, J.P.: Ai optimization of a billiard player. Journal of Intel- ligent and Robotic Systems50, 399–417 (12 2007)

    2007

  8. [16]

    In: van den Herik, H.J., Ciancarini, P., Donkers, H.H.L.M.J

    Leckie, W., Greenspan, M.: Monte-carlo methods in pool strategy game trees. In: van den Herik, H.J., Ciancarini, P., Donkers, H.H.L.M.J. (eds.) Computers and Games. pp. 244–255. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)

    2007

  9. [17]

    In: 2011 18th IEEE international conference on image processing

    Legg, P.A., Parry, M.L., Chung, D.H., Jiang, R.M., Morris, A., Griffiths, I.W., Marshall, D., Chen, M.: Intelligent filtering by semantic importance for single-view 3d reconstruction from snooker video. In: 2011 18th IEEE international conference on image processing. pp. 2385–...

    2011

  10. [18]

    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning (2019)

    2019

  11. [19]

    CoRR (2016), http://arxiv.org/abs/1602.01783 pix2pockets 15

    Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. CoRR (2016), http://arxiv.org/abs/1602.01783 pix2pockets 15

    2016 arXiv

  12. [20]

    arXiv preprint arXiv:1312.5602 (2013)

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

    2013 arXiv

  13. [21]

    In: International conference on machine learning

    Oh, J., Chockalingam, V., Lee, H., et al.: Control of memory, active perception, and action in minecraft. In: International conference on machine learning. pp. 2790–

  14. [22]

    Pearce, T., Zhu, J.: Counter-strike deathmatch with large-scale behavioural cloning (2021)

    2021

  15. [23]

    Journal of Machine Learning Research 22(268), 1–8 (2021)

    Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable- baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22(268), 1–8 (2021)

    2021

  16. [24]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Ramanathan, V., Huang, J., Abu-El-Haija, S., Gorban, A., Murphy, K., Fei-Fei, L.: Detecting events and key actors in multi-person videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3043–3053 (2016)

    2016

  17. [25]

    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection (2016)

    2016

  18. [26]

    Samarasinghe,S.M.:DevelopmentofPoolbot:anAutonomousPoolPlayingRobot. Ph.D. thesis, Asian Institute of Technology (2016)

    2016

  19. [27]

    CoRR (2017), http://arxiv.org/abs/1707.06347

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR (2017), http://arxiv.org/abs/1707.06347

    2017 arXiv

  20. [28]

    Seweryn, K., Chęć, G., Łukasik, S., Wróblewska, A.: Improving object detection quality in football through super-resolution techniques (2024)

    2024

  21. [29]

    Nature 529, 484–503 (2016)

    Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529, 484–503 (2016)

    2016

  22. [30]

    CoRR (2017)

    Silver,D.,etal.:Masteringchessandshogibyself-playwithageneralreinforcement learning algorithm. CoRR (2017)

    2017

  23. [31]

    In: Proceedings of the 19th National Conference on Innovative Applications of Artificial Intelligence - Volume 2

    Smith, M., Lee-Urban, S., Muñoz Avila, H.: Retaliate: learning winning policies in first-person shooter games. In: Proceedings of the 19th National Conference on Innovative Applications of Artificial Intelligence - Volume 2. AAAI Press (2007)

    2007

  24. [32]

    In: Proceedings of the national conference on artificial intelligence

    Smith, M.: Running the table: An ai for computer billiards. In: Proceedings of the national conference on artificial intelligence. vol. 21, p. 994. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999 (2006)

    2006

  25. [33]

    Computational Visual Media (Jun 2016)

    Sousa, L., Alves, R., Rodrigues, J.M.F.: Augmented reality system to assist inex- perienced pool players. Computational Visual Media (Jun 2016)

    2016

  26. [34]

    In: Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining

    Sun, X., Davis, J., Schulte, O., Liu, G.: Cracking the black box: Distilling deep sports analytics. In: Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining. pp. 3154–3162 (2020)

    2020

  27. [35]

    Tesauro, G., Murray, A.F.: TD-Gammon: A Self-Teaching Backgammon Program (1995)

    1995

  28. [36]

    Journal of Artificial Intelligence Research 71, 41–88 (2021)

    Tuyls, K., Omidshafiei, S., Muller, P., Wang, Z., Connor, J., Hennes, D., Graham, I., Spearman, W., Waskett, T., Steel, D., et al.: Game plan: What ai can do for football, and what football can do for ai. Journal of Artificial Intelligence Research 71, 41–88 (2021)

    2021

  29. [37]

    Nature575(7782), 350–354 (2019)

    Vinyals, O., Babuschkin, I., Czarnecki, W.e.a.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature575(7782), 350–354 (2019)

    2019

  30. [38]

    Weatherford, S.: Pool cue guide (2013)

    2013

  31. [39]

    IEEE Access (2019)

    Yoon, Y., Hwang, H., Choi, Y., Joo, M., Oh, H., Park, I., Lee, K.H., Hwang, J.H.: Analyzing basketball movements and pass relationships using realtime object tracking techniques based on deep learning. IEEE Access (2019)

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.