Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

The paper claims a gameplay video bug report can be reduced to one frame: after re-encoding, keyframes preserve the bug in 98.79% of confirmed visual-bug videos, and a zero-shot vision-language model ranks the right frame first with 0.89 ac

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A keyframe-plus-GPT-4o pipeline retrieves the single most representative frame for a reported gameplay bug, with F1@1 of 0.79 and Accuracy@1 of 0.89 on industrial bug-report videos.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful industrial case study of keyframe + VLM bug-frame retrieval, but the headline Accuracy@1 overstates per-bug success and Table 2's numbers don't add up. the 4 major comments →

arxiv 2508.04895 v1 pith:MVBERFSG submitted 2025-08-06 cs.SE

Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

Wentao Lu , Alexander Senchenko , Abram Hindle , Cor-Paul Bezemer This is my paper
classification cs.SE
keywords bug frame retrievalgameplay video analysisvision-language modelskeyframe extractiongame QA automationbug report triagezero-shot retrievalvisual bugs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a video bug report can be reduced to a single still frame without asking a human to watch the footage. The pipeline re-encodes each gameplay video, keeps only the encoder's keyframes (about 1.90% of all frames), and asks a vision-language model to rank those keyframes against the written bug description. On 350 real developer-submitted videos from one FPS game, the top-ranked frame matches a manually verified bug frame in 89% of the videos that contain a visible bug, with an overall F1 score of 0.79. Performance is strongest for Lighting & Shadow (F1 = 0.94), Physics & Collision (0.86), and UI & HUD (0.83), and weakest for Animation & VFX (0.51), which the paper attributes to bugs that only exist across time. If the result holds, QA teams can triage many visual bugs by glancing at one image instead of replaying a clip.

Core claim

The discovery is that a static-frame retrieval task, not a full video-understanding task, is enough for most visual bug reports. After re-encoding videos to a uniform format, keyframes preserve at least one bug frame in 98.79% of the 247 manually confirmed bug videos, and zero-shot ranking by a vision-language model retrieves a correct top-1 frame in 169 of those 247 videos. The authors define the pipeline's output as a ranked list of keyframes, with the top of the list meant to serve as instant visual evidence for the reported bug. They further show that repeated runs are stable only at the aggregate level: 183 of 350 videos are correct in all three runs, so they recommend majority voting o

What carries the argument

The central mechanism is keyframe extraction followed by zero-shot visual ranking: FFmpeg re-encodes each video with H.264, and the encoder's inserted keyframes serve as a deterministic down-sampling of the video, cutting it to a median of 1.90% of its original frames. A vision-language model (GPT-4o) then ranks those keyframes against a summarised bug description and returns the most representative frame. For videos with more than 50 keyframes, the frames are processed in consecutive batches and merged by a second VLM pass that acts as a judge, re-ranking all candidate frames with the batch-level explanations.

Load-bearing premise

The pipeline assumes the reported bug is visible in at least one static keyframe after re-encoding; if the bug exists only as motion over time—flickering, frame-rate drops, or animation glitches—no ranking of still frames can find it.

What would settle it

Build a test set of only temporal bugs (a light that flickers only during playback, an FPS counter dropping without any scene change, a two-frame animation pop-in) and require the top-1 static frame alone to carry the bug. If accuracy on that set is at chance, the static-frame ceiling the paper acknowledges is confirmed; if accuracy stays high, the one-frame-is-enough claim extends beyond the paper's own category analysis.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • QA engineers can check the top-1 or top-3 keyframes instead of scrubbing a gameplay video, reducing each review to a glance at a few still images.
  • The approach is cheap enough for routine use: the paper reports running the entire evaluation for under $100.
  • For bugs with strong visual contrast and clear text-to-image alignment (lighting glitches, physics clipping, placeholder UI elements), automated retrieval is reliable enough to be used directly.
  • For flickering, frame-rate drops, and animation/VFX glitches, static-frame retrieval is the wrong tool; the paper recommends multi-frame or video-level modeling for these temporal cases.
  • Aggregating at the report level rather than the video level raises successful retrieval from 68.42% to 82.14% of multi-video bug reports.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same keyframe-plus-ranking recipe could transfer to other software domains where bug reports contain screen recordings, such as mobile apps or desktop UI tools, provided the bug is visible in a static frame.
  • A testable extension is to run the pipeline only on video categories the paper identifies as temporally hard (Animation & VFX, Performance) and use the paper's own category F1 scores as a prior to route those clips to manual review instead.
  • The run-to-run variance suggests that three runs with majority voting may be a cheaper reliability boost than switching to a larger or fine-tuned model, since the paper observed that aggregate F1 stays flat around 0.79 even as per-video results fluctuate.
  • Another extension is to combine static retrieval with a lightweight temporal check: only videos where the VLM returns low-confidence or no frame get a second pass over short clips, which would directly target the failure modes the paper identifies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an automated pipeline for retrieving a single representative bug frame from gameplay videos: FFmpeg is used to extract keyframes (re-encoding videos for better coverage), GPT-4o is used to summarize the textual bug description, rank keyframes, and aggregate batch-level rankings. The system is evaluated on 350 real-world JIRA bug-report videos from a popular FPS game, reporting an overall Accuracy@1 of 0.89 and F1@1 of 0.79, with category-specific performance and an analysis of run-to-run consistency. The paper argues this reduces manual QA video review without requiring captions/transcripts.

Significance. If the reported performance holds, the paper makes a practically useful contribution: it combines well-understood keyframe extraction with a strong VLM in a zero-shot retrieval setting, evaluated on industrial (rather than scraped) data. The task itself, retrieving a single representative frame from a bug-report video, is clearly motivated and not addressed by prior work. The paper also provides an honest analysis of VLM non-determinism and category-level weaknesses (e.g., Animation & VFX, Performance). However, the significance is heavily undermined by serious internal inconsistencies in the evaluation table and the metric definitions, which must be corrected before the headline claims can be accepted.

major comments (4)
  1. [Table 2, Section 5.2] The evaluation table is internally inconsistent. The Overall row reports 247 bug videos, but TP@1+FN = 190, and TP+TN+FP+FN = 324, while the evaluation set is 350 videos. Per-category, the four-cell sums range from 43 to 50, not 50 as implied by the sampling design. Moreover, the 'Invalid' column sums to 57, yet RQ1 (Section 5.1) finds only 1.21% of 247 (about 3) videos fail keyframe coverage after re-encoding. These contradictions make the Accuracy@1 and F1@1 values in the abstract and introduction impossible to verify. The authors must reconcile every video in the dataset, clarify what 'Invalid' represents, and provide a coherent confusion matrix.
  2. [Section 4.0.1, Eq. (1)] The metric named 'Accuracy@N' is defined as TP@N/(TP@N+FN), which is recall over the subset of videos whose keyframes actually contain a bug frame. It excludes invalid videos entirely. The paper itself reports a 'successful retrieval rate' of 68.42% (169/247) in Section 5.2. Presenting the abstract's 'Accuracy of 0.89' without qualifying that it excludes invalid videos is misleading for practitioners because, on a random bug video, the system succeeds only ~68% of the time. Please either report end-to-end accuracy (counting invalid videos as failures) or explicitly distinguish the conditional metric in every claim.
  3. [Section 4.0.1] The definition of false positives is non-standard and likely inflates or deflates F1 unpredictably. The text states that 'false positives include those arising from invalid cases, and any retrieved frame that does not correspond to a valid bug frame, regardless of its rank. Therefore, FP@1 = FP@2 = FP@3 = FP.' This implies a video with a correct top-1 frame and a wrong frame at rank 2 would be both TP@1 and FP, double-counting a single video. F1@1 should be computed from a top-1 confusion matrix only. Please clarify and recompute using a conventional position-specific definition.
  4. [Section 5.2] No baseline is reported on the same data. Since each bug video has, on average, about 10 keyframes of which ~4.5 show the bug, a random keyframe baseline would be ~45% top-1 recall on covered videos. Providing a random selection or a simple heuristic (e.g., first/last keyframe) baseline is essential to judge whether the VLM adds value beyond chance. The current comparison to a prior glitch-detection benchmark uses a different task and a different 100-image sample, and thus does not serve as a retrieval baseline.
minor comments (4)
  1. [Section 5.3] The pass@3 calculation is arithmetically incorrect: 1 - (1 - 0.68)^3 ≈ 0.967, not 0.98. Please correct this value and verify the derived conclusion.
  2. [Section 5.1 vs Section 1] The average number of bug-depicting keyframes is reported as 4.79 in Section 1 (RQ1) and as 4.57 in Section 5.1. Please make these consistent.
  3. [Section 5.2] The text mentions 'Camera & UI' when the category is named 'UI & HUD' in Table 1. Please unify the terminology.
  4. [Throughout] Since the metric TP/(TP+FN) is recall, consider renaming 'Accuracy@N' to 'Recall@N' or clearly define it as recall on the subset of bug-containing videos with keyframe coverage, to avoid misinterpretation.

Circularity Check

0 steps flagged

No significant circularity: the reported F1/accuracy are direct measurements against manual ground truth, with no fitted parameter or load-bearing self-citation.

full rationale

The paper's central claims are empirical system measurements: FFmpeg keyframe coverage (RQ1) and GPT-4o retrieval accuracy (RQ2/RQ3) are evaluated against manually annotated bug frames. The pipeline uses GPT-4o for summarization, ranking, and batch aggregation, but the correctness labels are external human annotations, so the claimed F1@1=0.79 and Accuracy@1=0.89 are not equivalent by construction to the model's own inputs. No parameter is fitted to a subset and then 'predicted'; the model is zero-shot. The self-citations to VideoGameQA-Bench/GlitchBench/VideoGameBunny (Taesiri et al., with co-author Bezemer) justify model selection and benchmark context, but they are not load-bearing: the reported performance is measured directly, and removing those citations would not alter the derivation. The unusual Accuracy@N=TP/(TP+FN) formula and the internal inconsistency of Table 2 (247 bug videos vs 190 TP+FN vs 324 total counts) are correctness/threats-to-validity concerns about what '0.89' means, not circular reductions of a prediction to its inputs. The paper's own limitations (e.g., temporal bugs not captured by static frames, VLM non-determinism, potential manual evaluation bias) are stated and do not constitute circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The ledger is small because the contribution is empirical system evaluation, not a derivation or model. The free parameters are all experimental-design choices that shape the headline aggregate numbers. The axioms are the assumptions that keyframes and text summaries faithfully represent the video and bug description, and that the manual labels are trustworthy.

free parameters (3)
  • Per-category evaluation sample size (50 per category) = 50 videos per category (350 total)
    Hand-chosen equal sampling across 7 categories whose true sizes range from 66 to 861 reports; the aggregate F1 of 0.79 is a balanced-sample statistic, not a population-weighted one.
  • Majority-vote decision threshold = 2 of 3 runs agree
    Hand-chosen rule for confirming retrieval results in RQ3; not derived from cost/benefit data.
  • FFmpeg re-encoding preset = H.264 medium preset
    Determines keyframe density and bug coverage; authors state they did not tune it beyond defaults, so it is a free design choice rather than a fit.
axioms (5)
  • domain assumption Keyframes extracted after FFmpeg H.264 re-encoding preserve at least one frame of any visual bug (98.79% coverage on 247 verified videos).
    Invoked in Section 3.1 and RQ1; the retrieval stage only sees keyframes, so the pipeline's ceiling is keyframe coverage.
  • domain assumption GPT-4o-generated bug-description summaries accurately capture the core issue and whether it is visible.
    Section 3.2; verified on only 100 summaries (97% agreement), and Section 7 admits reliability 'remains uncertain during full pipeline execution'.
  • domain assumption A single author's manual annotation of ground-truth bug frames is correct.
    Section 4; no inter-rater reliability, and Section 7 acknowledges the subjectivity of matching frames to descriptions.
  • domain assumption The seven-category scheme (Table 1) covers the studied bug reports and GPT-4o-mini's automatic category assignment is accurate.
    Section 4; reports were auto-assigned to categories, with only the 350-sample subset manually verified.
  • domain assumption The second-pass LLM-as-judge aggregation yields a globally correct ranking from batch-level rankings.
    Section 3.4; relies on the LLM-as-judge literature rather than any verified guarantee; only 2 outlier videos exceed one batch.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models." pith.science (2026). https://pith.science/paper/MVBERFSG

@misc{pith2026250804895,
  author       = {Pith},
  title        = {Pith review of: Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVBERFSG}},
  note         = {Machine review of arXiv:2508.04895}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern game studios deliver new builds and patches at a rapid pace, generating thousands of bug reports, many of which embed gameplay videos. To verify and triage these bug reports, developers must watch the submitted videos. This manual review is labour-intensive, slow, and hard to scale. In this paper, we introduce an automated pipeline that reduces each video to a single frame that best matches the reported bug description, giving developers instant visual evidence that pinpoints the bug. Our pipeline begins with FFmpeg for keyframe extraction, reducing each video to a median of just 1.90% of its original frames while still capturing bug moments in 98.79 of cases. These keyframes are then evaluated by a vision--language model (GPT-4o), which ranks them based on how well they match the textual bug description and selects the most representative frame. We evaluated this approach using real-world developer-submitted gameplay videos and JIRA bug reports from a popular First-Person Shooter (FPS) game. The pipeline achieves an overall F1 score of 0.79 and Accuracy of 0.89 for the top-1 retrieved frame. Performance is highest for the Lighting & Shadow (F1 = 0.94), Physics & Collision (0.86), and UI & HUD (0.83) bug categories, and lowest for Animation & VFX (0.51). By replacing video viewing with an immediately informative image, our approach dramatically reduces manual effort and speeds up triage and regression checks, offering practical benefits to quality assurance (QA) teams and developers across the game industry.

Figures

Figures reproduced from arXiv: 2508.04895 by Abram Hindle, Alexander Senchenko, Cor-Paul Bezemer, Wentao Lu.

Figure 1
Figure 1. Figure 1: Examples of bug frames identified and retrieved in gameplay video sourced from reddit [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview diagram of our methodology (example video sourced from Reddit [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Six samples of visual bugs in game videos: (a) Physics & Collision, (b) Rendering & Texture, (c) Camera & UI, (d) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Total number of frames vs keyframes per video [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of number of frames in final ranked [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RESP: Reference-guided Sequential Prompting for Visual Glitch Detection in Video Games

    cs.CV 2026-04 unverdicted novelty 7.0

    RESP uses reference-guided sequential prompting with VLMs to improve frame-level and video-level visual glitch detection in games by establishing per-video baselines.

  2. TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos

    cs.CV 2026-05 unverdicted novelty 6.0

    TempGlitch is a controlled benchmark showing that 12 evaluated VLMs perform near chance level on detecting five types of temporal glitches in gameplay videos, with denser sampling and larger models providing no reliab...

Reference graph

Works this paper leans on

57 extracted references · 32 canonical work pages · cited by 2 Pith papers · 4 internal anchors

  1. [1]

    2025. FFmpeg. https://ffmpeg.org/

    work page 2025

  2. [2]

    Anthropic. 2024. Introducing the next generation of Claude . https://www. anthropic.com/news/claude-3-family

    work page 2024

  3. [3]

    Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2021. Automated Video Game Testing Using Synthetic and Humanlike Agents. IEEE Transactions on Games 13, 1 (2021), 50–67. https://doi.org/10.1109/TG.2019.2947597

    arXiv 2021

  4. [4]

    Atlassian. 2025. Jira. https://www.atlassian.com/software/jira Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

    work page 2025

  5. [5]

    Elham Azizi and Loutfouz Zaman. 2023. Automatic Bug Detection in Games using LSTM Networks. In 2023 IEEE Conference on Games (CoG) . 1–4. https: //doi.org/10.1109/CoG57401.2023.10333253

    arXiv 2023

  6. [6]

    Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software En- gineering (Atlanta, Georgia)(SIGSOFT ’08/FSE-16). Association for Computing Ma- chinery, New York, NY, USA, 308–318. https:/...

    work page arXiv 2008

  7. [7]

    Stefano Campanella, Emanuela Guglielmi, Rocco Oliveto, Gabriele Bavota, and Simone Scalabrino. 2024. Towards the Automatic Replication of Gameplays to Support Game Debugging. In Proceedings of the 1st ACM International Workshop on Foundations of Applied Software Engineering for Games (Porto de Galinhas, Brazil) (FaSE4Games 2024). Association for Computing...

    work page arXiv 2024

  8. [8]

    Norman Cliff. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions. Vol. 114. 494–509

    work page 1993

  9. [9]

    Nathan Cooper, Carlos Bernal-Cárdenas, Oscar Chaparro, Kevin Moran, and Denys Poshyvanyk. 2021. It Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports. In Proceedings of the 43rd International Conference on Software Engineering (Madrid, Spain) (ICSE ’21). IEEE Press, 957–969. https://doi.org/10.1109...

    arXiv 2021

  10. [10]

    Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can LLM be a Personalized Judge? arXiv:2406.11657 [cs.CL] https://arxiv.org/abs/2406.11657

    Pith/arXiv arXiv 2024

  11. [11]

    EA. 2025. EA Forum. https://forums.ea.com/

    work page 2025

  12. [12]

    Sidong Feng and Chunyang Chen. 2022. GIFdroid: automated replay of visual bug reports for Android apps. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1045–1057. https://doi.org/10.1145/ 3510003.3510048

    arXiv 2022

  13. [13]

    Yannakakis

    Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N. Yannakakis. 2024. Large Language Models and Games: A Survey and Roadmap. IEEE Transactions on Games (2024), 1–18. https://doi.org/10.1109/TG.2024.3461510

    arXiv 2024

  14. [14]

    Google. 2024. Introducing Gemini 2.0: our new AI model for the agentic era. https://blog.google/technology/google-deepmind/google-gemini-ai-update- december-2024/

    work page 2024

  15. [15]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as- a-Judge. arXiv:2411.15594 [cs.CL] https://arxiv.org/abs/2411.15594

    Pith/arXiv arXiv 2025

  16. [16]

    Emanuela Guglielmi, Gabriele Bavota, Rocco Oliveto, and Simone Scalabrino

    work page

  17. [18]

    Emanuela Guglielmi, Simone Scalabrino, Gabriele Bavota, and Rocco Oliveto

    work page

  18. [19]

    Sihao Hu, Tiansheng Huang, Gaowen Liu, Ramana Rao Kompella, Fatih Ilhan, Selim Furkan Tekin, Yichang Xu, Zachary Yahn, and Ling Liu. 2025. A Survey on Large Language Model-Based Game Agents. arXiv:2404.02039 [cs.AI] https: //arxiv.org/abs/2404.02039

    Pith/arXiv arXiv 2025

  19. [20]

    Sumandeep Kaur, Lakhwinder Kaur, and Madan Lal. 2024. An effective Key Frame Extraction technique based on Feature Fusion and Fuzzy-C means clustering with Artificial Hummingbird. Scientific Reports 14 (11 2024). https://doi.org/10. 1038/s41598-024-75923-y

    work page 2024

  20. [21]

    Eugene Klishevich, Yegor Denisov-Blanch, Simon Obstbaum, Igor Ciobanu, and Michal Kosinski. 2025. Measuring Determinism in Large Language Models for Software Code Review. arXiv:2502.20747 [cs.SE] https://arxiv.org/abs/2502.20747

    Pith/arXiv arXiv 2025

  21. [22]

    Vasilii Korolkov. 2025. Scene Detection Policies and Keyframe Extraction Strategies for Large-Scale Video Analysis. arXiv:2506.00667 [cs.CV] https: //arxiv.org/abs/2506.00667

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [23]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  23. [24]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv:2412.05579 [cs.CL] https://arxiv.org/abs/2412.05579

    Pith/arXiv arXiv 2024

  24. [25]

    Dayi Lin, Cor-Paul Bezemer, and Ahmed E. Hassan. 2019. Identifying gameplay videos that exhibit bugs in computer games. Empirical Software Engineering (12 2019). https://doi.org/10.1007/s10664-019-09733-6

    work page doi:10.1007/s10664-019-09733-6 2019

  25. [26]

    Charles Lovering, Michael Krumdick, Viet Dac Lai, Seth Ebner, Nilesh Kumar, Varshini Reddy, Rik Koncel-Kedziorski, and Chris Tanner. 2025. Language Model Probabilities are Not Calibrated in Numeric Contexts. arXiv:2410.16007 [cs.AI] https://arxiv.org/abs/2410.16007

    Pith/arXiv arXiv 2025

  26. [27]

    Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2024. Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Asso...

    work page 2024

  27. [28]

    Yannakakis

    David Melhart, Matthew Barthet, and Georgios N. Yannakakis. 2025. Can Large Language Models Capture Video Game Engagement? arXiv:2502.04379 [cs.CV] https://arxiv.org/abs/2502.04379

    work page arXiv 2025

  28. [29]

    microsoft. 2025. Azure. https://azure.microsoft.com/en-ca

    work page 2025

  29. [30]

    Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuehne, and Horst Possegger

    M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuehne, and Horst Possegger. 2025. Meta-prompting for Automating Zero-Shot Visual Recognition with LLMs. In Computer Vision – ECCV 2024, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer Nature Switzer...

    work page 2025

  30. [31]

    Mistral. 2025. Frontier AI. In Your Hands. https://mistral.ai/

    work page 2025

  31. [32]

    Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, and Denys Poshy- vanyk. 2015. Auto-completing bug reports for Android applications. In Proceed- ings of the 2015 10th Joint Meeting on Foundations of Software Engineering . ACM, 673–686. https://doi.org/10.1145/2786805.2786857

    arXiv 2015

  32. [33]

    Aishik Nagar, Shantanu Jaiswal, and Cheston Tan. 2024. Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis. In 2024 International Joint Conference on Neural Networks (IJCNN) . 1–8. https://doi.org/ 10.1109/IJCNN60899.2024.10650020

    arXiv 2024

  33. [34]

    OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence . https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    work page 2024

  34. [35]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  35. [36]

    OpenAI. 2024. OpenAI Platform . https://platform.openai.com/docs/api- reference/chat/create

    work page 2024

  36. [37]

    Ciprian Paduraru, Miruna Paduraru, and Alin Stefanescu. 2021. Automated game testing using computer vision methods. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) . 65–72. https: //doi.org/10.1109/ASEW52652.2021.00024

    work page arXiv 2021

  37. [38]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous

    work page

  38. [39]

    Reddit. 2022. Visual bug when Wraith finisher gets interupted. https://www.reddit.com/r/apexlegends/comments/13i8gmp/visual_bug_ when_wraith_finisher_gets_interupted

    work page 2022

  39. [40]

    Richardson

    Iain E. Richardson. 2010. The H.264 Advanced Video Compression Standard, Second Edition. Wiley

    work page 2010

  40. [41]

    Alexander Senchenko, Naomi Patterson, Hamman Samuel, and Dan Ispir. 2022. SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST) . IEEE, 345–354. https: //doi.org/10.1109/icst53961.2022.00043

    arXiv 2022

  41. [42]

    Mohammad Reza Taesiri and Cor-Paul Bezemer. 2025. VIDEOGAMEBUNNY: Towards vision assistants for video games. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2025-03-01)

    work page 2025

  42. [43]

    Mohammad Reza Taesiri, Tianjun Feng, Cor-Paul Bezemer, and Anh Nguyen

  43. [44]

    Mohammad Reza Taesiri, Abhijay Ghildyal, Saman Zadtootaghaj, Nabajeet Bar- man, and Cor-Paul Bezemer. 2025. VideoGameQA-Bench: Evaluating Vision- Language Models for Video Game Quality Assurance. arXiv:2505.15952 [cs.CV] https://arxiv.org/abs/2505.15952

    arXiv 2025

  44. [45]

    Mohammad Reza Taesiri, Finlay Macklon, and Cor-Paul Bezemer. 2022. CLIP meets GamePhysics: towards bug identification in gameplay videos using zero- shot transfer learning. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 270–28...

    arXiv 2022

  45. [46]

    Xuchen Tan, Deenu Yadav, Faiz Ahmed, and Maleknaz Nayebi. 2025. ImageR: Enhancing Bug Report Clarity by Screenshots. arXiv:2505.01925 [cs.SE] https: //arxiv.org/abs/2505.01925

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  46. [47]

    Andrew Truelove, Shiyue Rong, Eduardo Santana de Almeida, and Iftekhar Ahmed. 2023. Finding the Needle in a Haystack: Detecting Bug Occurrences in Gameplay Videos. arXiv:2311.10926 [cs.SE] https://arxiv.org/abs/2311.10926

    Pith/arXiv arXiv 2023

  47. [48]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

    GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). 22444–22455

    work page

  48. [49]

    Dingbang Wang, Zhaoxu Zhang, Sidong Feng, William G. J. Halfond, and Tingt- ing Yu. 2025. An Empirical Study on Leveraging Images in Automated Bug Report Reproduction. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 27–38. https://doi.org/10.1109/MSR66628.2025.00019 Wentao Lu, Alexander Senchenko, Abram Hindle, and Co...

    arXiv 2025

  49. [50]

    1992.Individual Comparisons by Ranking Methods

    Frank Wilcoxon. 1992.Individual Comparisons by Ranking Methods. Springer New York, New York, NY, 196–202. https://doi.org/10.1007/978-1-4612-4380-9_16

    doi:10.1007/978-1-4612-4380-9_16 1992

  50. [51]

    Yanfu Yan, Nathan Cooper, Oscar Chaparro, Kevin Moran, and Denys Poshy- vanyk. 2024. Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports. In Proceedings of the IEEE/ACM 46th In- ternational Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24) . Association for Computing Machinery, New York, NY, USA, ...

    arXiv 2024

  51. [52]

    Yan Zhao, Weihao Zhang, Enyi Tang, Haipeng Cai, Xi Guo, and Na Meng. 2021. A Lightweight Approach of Human-Like Playtesting. arXiv:2102.13026 [cs.SE] https://arxiv.org/abs/2102.13026

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  52. [53]

    Alicia Vidler and Toby Walsh. 2025. Playing games with Large language models: Randomness and strategy. arXiv:2503.02582 [cs.AI] https://arxiv.org/abs/2503. 02582

    Pith/arXiv arXiv 2025

  53. [58]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S...

    work page 2023

  54. [2022]

    Towards Using Gameplay Videos for Detecting Issues in Video Games

    Towards Using Gameplay Videos for Detecting Issues in Video Games. arXiv:2204.04182 [cs.SE] https://arxiv.org/abs/2204.04182

    work page internal anchor Pith review Pith/arXiv arXiv

  55. [2023]

    Using gameplay videos for detecting issues in video games.Empirical Softw. Engg. 28, 6 (Oct. 2023), 32 pages. https://doi.org/10.1007/s10664-023-10365-0

    work page doi:10.1007/s10664-023-10365-0 2023

  56. [2024]

    Is Temperature the Creativity Parameter of Large Language Models? arXiv:2405.00492 [cs.CL] https://arxiv.org/abs/2405.00492

    Pith/arXiv arXiv

  57. [2025]

    ACM Trans

    Automatic Identification of Game Stuttering via Gameplay Videos Analysis. ACM Trans. Softw. Eng. Methodol. 34, 2, Article 38 (Jan. 2025), 29 pages. https: //doi.org/10.1145/3695992

    work page doi:10.1145/3695992 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.