Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

Abram Hindle; Alexander Senchenko; Cor-Paul Bezemer; Wentao Lu

REVIEW 4 major objections 4 minor 2 cited by

Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a gameplay video bug report can be reduced to one frame: after re-encoding, keyframes preserve the bug in 98.79% of confirmed visual-bug videos, and a zero-shot vision-language model ranks the right frame first with 0.89 ac

desk verdict A genuinely useful industrial case study of keyframe + VLM bug-frame retrieval, but the headline Accuracy@1 overstates per-bug success and Table 2's numbers don't add up. read the letter →

arxiv 2508.04895 v1 pith:MVBERFSG submitted 2025-08-06 cs.SE

Wentao Lu , Alexander Senchenko , Abram Hindle , Cor-Paul Bezemer This is my paper

classification cs.SE

keywords bugframeretrievalgameplayvideoanalysisvision-languagemodelskeyframeextractiongameQAautomationreporttriagezero-shotvisualbugs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a video bug report can be reduced to a single still frame without asking a human to watch the footage. The pipeline re-encodes each gameplay video, keeps only the encoder's keyframes (about 1.90% of all frames), and asks a vision-language model to rank those keyframes against the written bug description. On 350 real developer-submitted videos from one FPS game, the top-ranked frame matches a manually verified bug frame in 89% of the videos that contain a visible bug, with an overall F1 score of 0.79. Performance is strongest for Lighting & Shadow (F1 = 0.94), Physics & Collision (0.86), and UI & HUD (0.83), and weakest for Animation & VFX (0.51), which the paper attributes to bugs that only exist across time. If the result holds, QA teams can triage many visual bugs by glancing at one image instead of replaying a clip.

What carries the argument

The central mechanism is keyframe extraction followed by zero-shot visual ranking: FFmpeg re-encodes each video with H.264, and the encoder's inserted keyframes serve as a deterministic down-sampling of the video, cutting it to a median of 1.90% of its original frames. A vision-language model (GPT-4o) then ranks those keyframes against a summarised bug description and returns the most representative frame. For videos with more than 50 keyframes, the frames are processed in consecutive batches and merged by a second VLM pass that acts as a judge, re-ranking all candidate frames with the batch-level explanations.

What would settle it

Build a test set of only temporal bugs (a light that flickers only during playback, an FPS counter dropping without any scene change, a two-frame animation pop-in) and require the top-1 static frame alone to carry the bug. If accuracy on that set is at chance, the static-frame ceiling the paper acknowledges is confirmed; if accuracy stays high, the one-frame-is-enough claim extends beyond the paper's own category analysis.

Watch

Extended reading notes

Core claim

The discovery is that a static-frame retrieval task, not a full video-understanding task, is enough for most visual bug reports. After re-encoding videos to a uniform format, keyframes preserve at least one bug frame in 98.79% of the 247 manually confirmed bug videos, and zero-shot ranking by a vision-language model retrieves a correct top-1 frame in 169 of those 247 videos. The authors define the pipeline's output as a ranked list of keyframes, with the top of the list meant to serve as instant visual evidence for the reported bug. They further show that repeated runs are stable only at the aggregate level: 183 of 350 videos are correct in all three runs, so they recommend majority voting o

Load-bearing premise

The pipeline assumes the reported bug is visible in at least one static keyframe after re-encoding; if the bug exists only as motion over time—flickering, frame-rate drops, or animation glitches—no ranking of still frames can find it.

Editorial extensions

If this is right

QA engineers can check the top-1 or top-3 keyframes instead of scrubbing a gameplay video, reducing each review to a glance at a few still images.
The approach is cheap enough for routine use: the paper reports running the entire evaluation for under $100.
For bugs with strong visual contrast and clear text-to-image alignment (lighting glitches, physics clipping, placeholder UI elements), automated retrieval is reliable enough to be used directly.
For flickering, frame-rate drops, and animation/VFX glitches, static-frame retrieval is the wrong tool; the paper recommends multi-frame or video-level modeling for these temporal cases.
Aggregating at the report level rather than the video level raises successful retrieval from 68.42% to 82.14% of multi-video bug reports.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the same keyframe-plus-ranking recipe could transfer to other software domains where bug reports contain screen recordings, such as mobile apps or desktop UI tools, provided the bug is visible in a static frame.
A testable extension is to run the pipeline only on video categories the paper identifies as temporally hard (Animation & VFX, Performance) and use the paper's own category F1 scores as a prior to route those clips to manual review instead.
The run-to-run variance suggests that three runs with majority voting may be a cheaper reliability boost than switching to a larger or fine-tuned model, since the paper observed that aggregate F1 stays flat around 0.79 even as per-video results fluctuate.
Another extension is to combine static retrieval with a lightweight temporal check: only videos where the VLM returns low-confidence or no frame get a second pass over short clips, which would directly target the failure modes the paper identifies.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this if you care about game QA automation. The pipeline is straightforward: re-encode with FFmpeg, extract keyframes, ask GPT-4o to rank them against a summarized JIRA description, then re-rank batch results. What is new is the evaluation on real developer-submitted gameplay videos and JIRA reports from an FPS title, plus a three-run consistency analysis with a majority-vote recommendation. That consistency analysis is the most honest part of the paper; it shows per-video instability even when aggregate F1 stays at 0.79.

The qualitative results are plausible. For bugs with strong static visual signs—lighting, missing textures, UI—the method probably works well in practice. The authors are also upfront that flickering, frame-rate drops, and animation/VFX bugs need temporal context and cannot be solved with static keyframes. That limitation is not hidden; it is in Section 5.2 and the discussion.

The soft spots are mostly in the reporting. Table 2 is internally inconsistent. It lists 247 bug videos overall, but only 169 TP and 21 FN in the retrieval outcomes, so the Accuracy@1 denominator is 190, not 247. It also reports 57 'Invalid' videos, which contradicts RQ1's claim of 98.79% keyframe coverage (about 3 failures out of 247). The paper's own 'successful retrieval rate' of 68.42% is exactly 169/(169+21+57), which suggests the honest per-bug-video success rate is about 68%, not 89%. The abstract's 0.89 is recall conditioned on keyframes actually containing the bug. That is a defensible metric if clearly defined, but it is not the headline number a practitioner should use to decide whether the tool will save time. The category counts also don't sum cleanly to 350, and the 50-per-category balanced sample means the aggregate numbers overrepresent categories like Lighting & Shadow (rare in the wild) and underrepresent anything common.

Other issues: no baseline comparison on the same data (random keyframe, CLIP, etc.), single-annotator ground truth, and the fact that the same model summarizes, ranks, and judges. That last one is a real concern but not a circularity that invalidates the result, since the ground truth is human-annotated.

I'd send this to a serious referee. The data and the consistency analysis are worth preserving, and the method is plausible enough to matter. But the arithmetic needs to be fixed before the numbers can be trusted, and the headline should be reframed around the per-bug-video success rate.

Referee Report

4 major / 4 minor

Summary. The paper proposes an automated pipeline for retrieving a single representative bug frame from gameplay videos: FFmpeg is used to extract keyframes (re-encoding videos for better coverage), GPT-4o is used to summarize the textual bug description, rank keyframes, and aggregate batch-level rankings. The system is evaluated on 350 real-world JIRA bug-report videos from a popular FPS game, reporting an overall Accuracy@1 of 0.89 and F1@1 of 0.79, with category-specific performance and an analysis of run-to-run consistency. The paper argues this reduces manual QA video review without requiring captions/transcripts.

Significance. If the reported performance holds, the paper makes a practically useful contribution: it combines well-understood keyframe extraction with a strong VLM in a zero-shot retrieval setting, evaluated on industrial (rather than scraped) data. The task itself, retrieving a single representative frame from a bug-report video, is clearly motivated and not addressed by prior work. The paper also provides an honest analysis of VLM non-determinism and category-level weaknesses (e.g., Animation & VFX, Performance). However, the significance is heavily undermined by serious internal inconsistencies in the evaluation table and the metric definitions, which must be corrected before the headline claims can be accepted.

major comments (4)

[Table 2, Section 5.2] The evaluation table is internally inconsistent. The Overall row reports 247 bug videos, but TP@1+FN = 190, and TP+TN+FP+FN = 324, while the evaluation set is 350 videos. Per-category, the four-cell sums range from 43 to 50, not 50 as implied by the sampling design. Moreover, the 'Invalid' column sums to 57, yet RQ1 (Section 5.1) finds only 1.21% of 247 (about 3) videos fail keyframe coverage after re-encoding. These contradictions make the Accuracy@1 and F1@1 values in the abstract and introduction impossible to verify. The authors must reconcile every video in the dataset, clarify what 'Invalid' represents, and provide a coherent confusion matrix.
[Section 4.0.1, Eq. (1)] The metric named 'Accuracy@N' is defined as TP@N/(TP@N+FN), which is recall over the subset of videos whose keyframes actually contain a bug frame. It excludes invalid videos entirely. The paper itself reports a 'successful retrieval rate' of 68.42% (169/247) in Section 5.2. Presenting the abstract's 'Accuracy of 0.89' without qualifying that it excludes invalid videos is misleading for practitioners because, on a random bug video, the system succeeds only ~68% of the time. Please either report end-to-end accuracy (counting invalid videos as failures) or explicitly distinguish the conditional metric in every claim.
[Section 4.0.1] The definition of false positives is non-standard and likely inflates or deflates F1 unpredictably. The text states that 'false positives include those arising from invalid cases, and any retrieved frame that does not correspond to a valid bug frame, regardless of its rank. Therefore, FP@1 = FP@2 = FP@3 = FP.' This implies a video with a correct top-1 frame and a wrong frame at rank 2 would be both TP@1 and FP, double-counting a single video. F1@1 should be computed from a top-1 confusion matrix only. Please clarify and recompute using a conventional position-specific definition.
[Section 5.2] No baseline is reported on the same data. Since each bug video has, on average, about 10 keyframes of which ~4.5 show the bug, a random keyframe baseline would be ~45% top-1 recall on covered videos. Providing a random selection or a simple heuristic (e.g., first/last keyframe) baseline is essential to judge whether the VLM adds value beyond chance. The current comparison to a prior glitch-detection benchmark uses a different task and a different 100-image sample, and thus does not serve as a retrieval baseline.

minor comments (4)

[Section 5.3] The pass@3 calculation is arithmetically incorrect: 1 - (1 - 0.68)^3 ≈ 0.967, not 0.98. Please correct this value and verify the derived conclusion.
[Section 5.1 vs Section 1] The average number of bug-depicting keyframes is reported as 4.79 in Section 1 (RQ1) and as 4.57 in Section 5.1. Please make these consistent.
[Section 5.2] The text mentions 'Camera & UI' when the category is named 'UI & HUD' in Table 1. Please unify the terminology.
[Throughout] Since the metric TP/(TP+FN) is recall, consider renaming 'Accuracy@N' to 'Recall@N' or clearly define it as recall on the subset of bug-containing videos with keyframe coverage, to avoid misinterpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported F1/accuracy are direct measurements against manual ground truth, with no fitted parameter or load-bearing self-citation.

full rationale

The paper's central claims are empirical system measurements: FFmpeg keyframe coverage (RQ1) and GPT-4o retrieval accuracy (RQ2/RQ3) are evaluated against manually annotated bug frames. The pipeline uses GPT-4o for summarization, ranking, and batch aggregation, but the correctness labels are external human annotations, so the claimed F1@1=0.79 and Accuracy@1=0.89 are not equivalent by construction to the model's own inputs. No parameter is fitted to a subset and then 'predicted'; the model is zero-shot. The self-citations to VideoGameQA-Bench/GlitchBench/VideoGameBunny (Taesiri et al., with co-author Bezemer) justify model selection and benchmark context, but they are not load-bearing: the reported performance is measured directly, and removing those citations would not alter the derivation. The unusual Accuracy@N=TP/(TP+FN) formula and the internal inconsistency of Table 2 (247 bug videos vs 190 TP+FN vs 324 total counts) are correctness/threats-to-validity concerns about what '0.89' means, not circular reductions of a prediction to its inputs. The paper's own limitations (e.g., temporal bugs not captured by static frames, VLM non-determinism, potential manual evaluation bias) are stated and do not constitute circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger is small because the contribution is empirical system evaluation, not a derivation or model. The free parameters are all experimental-design choices that shape the headline aggregate numbers. The axioms are the assumptions that keyframes and text summaries faithfully represent the video and bug description, and that the manual labels are trustworthy.

free parameters (3)

Per-category evaluation sample size (50 per category) = 50 videos per category (350 total)
Hand-chosen equal sampling across 7 categories whose true sizes range from 66 to 861 reports; the aggregate F1 of 0.79 is a balanced-sample statistic, not a population-weighted one.
Majority-vote decision threshold = 2 of 3 runs agree
Hand-chosen rule for confirming retrieval results in RQ3; not derived from cost/benefit data.
FFmpeg re-encoding preset = H.264 medium preset
Determines keyframe density and bug coverage; authors state they did not tune it beyond defaults, so it is a free design choice rather than a fit.

assumptions (5)

domain assumption Keyframes extracted after FFmpeg H.264 re-encoding preserve at least one frame of any visual bug (98.79% coverage on 247 verified videos).
Invoked in Section 3.1 and RQ1; the retrieval stage only sees keyframes, so the pipeline's ceiling is keyframe coverage.
domain assumption GPT-4o-generated bug-description summaries accurately capture the core issue and whether it is visible.
Section 3.2; verified on only 100 summaries (97% agreement), and Section 7 admits reliability 'remains uncertain during full pipeline execution'.
domain assumption A single author's manual annotation of ground-truth bug frames is correct.
Section 4; no inter-rater reliability, and Section 7 acknowledges the subjectivity of matching frames to descriptions.
domain assumption The seven-category scheme (Table 1) covers the studied bug reports and GPT-4o-mini's automatic category assignment is accurate.
Section 4; reports were auto-assigned to categories, with only the 350-sample subset manually verified.
domain assumption The second-pass LLM-as-judge aggregation yields a globally correct ranking from batch-level rankings.
Section 3.4; relies on the LLM-as-judge literature rather than any verified guarantee; only 2 outlier videos exceed one batch.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models." pith.science (2026). https://pith.science/paper/MVBERFSG

@misc{pith2026250804895,
  author       = {Pith},
  title        = {Pith review of: Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVBERFSG}},
  note         = {Machine review of arXiv:2508.04895}
}

read the original abstract

Modern game studios deliver new builds and patches at a rapid pace, generating thousands of bug reports, many of which embed gameplay videos. To verify and triage these bug reports, developers must watch the submitted videos. This manual review is labour-intensive, slow, and hard to scale. In this paper, we introduce an automated pipeline that reduces each video to a single frame that best matches the reported bug description, giving developers instant visual evidence that pinpoints the bug. Our pipeline begins with FFmpeg for keyframe extraction, reducing each video to a median of just 1.90% of its original frames while still capturing bug moments in 98.79 of cases. These keyframes are then evaluated by a vision--language model (GPT-4o), which ranks them based on how well they match the textual bug description and selects the most representative frame. We evaluated this approach using real-world developer-submitted gameplay videos and JIRA bug reports from a popular First-Person Shooter (FPS) game. The pipeline achieves an overall F1 score of 0.79 and Accuracy of 0.89 for the top-1 retrieved frame. Performance is highest for the Lighting & Shadow (F1 = 0.94), Physics & Collision (0.86), and UI & HUD (0.83) bug categories, and lowest for Animation & VFX (0.51). By replacing video viewing with an immediately informative image, our approach dramatically reduces manual effort and speeds up triage and regression checks, offering practical benefits to quality assurance (QA) teams and developers across the game industry.

Figures

Figures reproduced from arXiv: 2508.04895 by the authors.

**Figure 2.** Overview diagram of our methodology (example video sourced from Reddit [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Six samples of visual bugs in game videos: (a) Physics & Collision, (b) Rendering & Texture, (c) Camera & UI, (d) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Total number of frames vs keyframes per video [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Distribution of number of frames in final ranked [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

RESP: Reference-guided Sequential Prompting for Visual Glitch Detection in Video Games
cs.CV 2026-04 unverdicted novelty 7.0 of 10

RESP uses reference-guided sequential prompting with VLMs to improve frame-level and video-level visual glitch detection in games by establishing per-video baselines.
TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos
cs.CV 2026-05 unverdicted novelty 6.0 of 10

TempGlitch is a controlled benchmark showing that 12 evaluated VLMs perform near chance level on detecting five types of temporal glitches in gameplay videos, with denser sampling and larger models providing no reliab...

Reference graph

Works this paper leans on

57 extracted references · 32 canonical work pages · cited by 2 Pith papers

[1]

2025. FFmpeg. https://ffmpeg.org/

work page 2025
[2]

Anthropic. 2024. Introducing the next generation of Claude . https://www. anthropic.com/news/claude-3-family

work page 2024
[3]

Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2021. Automated Video Game Testing Using Synthetic and Humanlike Agents. IEEE Transactions on Games 13, 1 (2021), 50–67. https://doi.org/10.1109/TG.2019.2947597

arXiv 2021
[4]

Atlassian. 2025. Jira. https://www.atlassian.com/software/jira Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

work page 2025
[5]

Elham Azizi and Loutfouz Zaman. 2023. Automatic Bug Detection in Games using LSTM Networks. In 2023 IEEE Conference on Games (CoG) . 1–4. https: //doi.org/10.1109/CoG57401.2023.10333253

arXiv 2023
[6]

Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software En- gineering (Atlanta, Georgia)(SIGSOFT ’08/FSE-16). Association for Computing Ma- chinery, New York, NY, USA, 308–318. https:/...

work page arXiv 2008
[7]

Stefano Campanella, Emanuela Guglielmi, Rocco Oliveto, Gabriele Bavota, and Simone Scalabrino. 2024. Towards the Automatic Replication of Gameplays to Support Game Debugging. In Proceedings of the 1st ACM International Workshop on Foundations of Applied Software Engineering for Games (Porto de Galinhas, Brazil) (FaSE4Games 2024). Association for Computing...

work page arXiv 2024
[8]

Norman Cliff. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions. Vol. 114. 494–509

work page 1993

Show all 57 references

[9]

Nathan Cooper, Carlos Bernal-Cárdenas, Oscar Chaparro, Kevin Moran, and Denys Poshyvanyk. 2021. It Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports. In Proceedings of the 43rd International Conference on Software Engi...

2021
[10]

Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can LLM be a Personalized Judge? arXiv:2406.11657 [cs.CL] https://arxiv.org/abs/2406.11657

2024 arXiv
[11]

EA. 2025. EA Forum. https://forums.ea.com/

2025
[12]

Sidong Feng and Chunyang Chen. 2022. GIFdroid: automated replay of visual bug reports for Android apps. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 10...

2022
[13]

Yannakakis

Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N. Yannakakis. 2024. Large Language Models and Games: A Survey and Roadmap. IEEE Transactions on Games (2024), 1–18. https://doi.org/10.1109/TG.2024.3461510

2024
[14]

Google. 2024. Introducing Gemini 2.0: our new AI model for the agentic era. https://blog.google/technology/google-deepmind/google-gemini-ai-update- december-2024/

2024
[15]

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as- a-Judge. arXiv:2411.15594 [cs.CL] https://arxiv.org/a...

2025 arXiv
[16]

Emanuela Guglielmi, Gabriele Bavota, Rocco Oliveto, and Simone Scalabrino
[18]

Emanuela Guglielmi, Simone Scalabrino, Gabriele Bavota, and Rocco Oliveto
[19]

Sihao Hu, Tiansheng Huang, Gaowen Liu, Ramana Rao Kompella, Fatih Ilhan, Selim Furkan Tekin, Yichang Xu, Zachary Yahn, and Ling Liu. 2025. A Survey on Large Language Model-Based Game Agents. arXiv:2404.02039 [cs.AI] https: //arxiv.org/abs/2404.02039

2025 arXiv
[20]

Sumandeep Kaur, Lakhwinder Kaur, and Madan Lal. 2024. An effective Key Frame Extraction technique based on Feature Fusion and Fuzzy-C means clustering with Artificial Hummingbird. Scientific Reports 14 (11 2024). https://doi.org/10. 1038/s41598-024-75923-y

2024
[21]

Eugene Klishevich, Yegor Denisov-Blanch, Simon Obstbaum, Igor Ciobanu, and Michal Kosinski. 2025. Measuring Determinism in Large Language Models for Software Code Review. arXiv:2502.20747 [cs.SE] https://arxiv.org/abs/2502.20747

2025 arXiv
[22]

Vasilii Korolkov. 2025. Scene Detection Policies and Keyframe Extraction Strategies for Large-Scale Video Analysis. arXiv:2506.00667 [cs.CV] https: //arxiv.org/abs/2506.00667

2025 arXiv
[23]

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.03326 (2024)

2024 arXiv
[24]

Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv:2412.05579 [cs.CL] https://arxiv.org/abs/2412.05579

2024 arXiv
[25]

Dayi Lin, Cor-Paul Bezemer, and Ahmed E. Hassan. 2019. Identifying gameplay videos that exhibit bugs in computer games. Empirical Software Engineering (12 2019). https://doi.org/10.1007/s10664-019-09733-6

2019 doi
[26]

Charles Lovering, Michael Krumdick, Viet Dac Lai, Seth Ebner, Nilesh Kumar, Varshini Reddy, Rik Koncel-Kedziorski, and Chris Tanner. 2025. Language Model Probabilities are Not Calibrated in Numeric Contexts. arXiv:2410.16007 [cs.AI] https://arxiv.org/abs/2410.16007

2025 arXiv
[27]

Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2024. Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Be...

2024
[28]

Yannakakis

David Melhart, Matthew Barthet, and Georgios N. Yannakakis. 2025. Can Large Language Models Capture Video Game Engagement? arXiv:2502.04379 [cs.CV] https://arxiv.org/abs/2502.04379

2025
[29]

microsoft. 2025. Azure. https://azure.microsoft.com/en-ca

2025
[30]

Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuehne, and Horst Possegger

M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuehne, and Horst Possegger. 2025. Meta-prompting for Automating Zero-Shot Visual Recognition with LLMs. In Computer Vision – ECCV 2024, Aleš Leonardis, Elisa Ricci, Stefan Roth, ...

2025
[31]

Mistral. 2025. Frontier AI. In Your Hands. https://mistral.ai/

2025
[32]

Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, and Denys Poshy- vanyk. 2015. Auto-completing bug reports for Android applications. In Proceed- ings of the 2015 10th Joint Meeting on Foundations of Software Engineering . ACM, 673–686. https://doi.org/10.1145/278680...

2015
[33]

Aishik Nagar, Shantanu Jaiswal, and Cheston Tan. 2024. Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis. In 2024 International Joint Conference on Neural Networks (IJCNN) . 1–8. https://doi.org/ 10.1109/IJCNN60899.2024.10650020

2024
[34]

OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence . https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

2024
[35]

OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

2024
[36]

OpenAI. 2024. OpenAI Platform . https://platform.openai.com/docs/api- reference/chat/create

2024
[37]

Ciprian Paduraru, Miruna Paduraru, and Alin Stefanescu. 2021. Automated game testing using computer vision methods. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) . 65–72. https: //doi.org/10.1109/ASEW52652.2021.00024

2021
[38]

Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous
[39]

Reddit. 2022. Visual bug when Wraith finisher gets interupted. https://www.reddit.com/r/apexlegends/comments/13i8gmp/visual_bug_ when_wraith_finisher_gets_interupted

2022
[40]

Richardson

Iain E. Richardson. 2010. The H.264 Advanced Video Compression Standard, Second Edition. Wiley

2010
[41]

Alexander Senchenko, Naomi Patterson, Hamman Samuel, and Dan Ispir. 2022. SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST...

2022
[42]

Mohammad Reza Taesiri and Cor-Paul Bezemer. 2025. VIDEOGAMEBUNNY: Towards vision assistants for video games. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2025-03-01)

2025
[43]

Mohammad Reza Taesiri, Tianjun Feng, Cor-Paul Bezemer, and Anh Nguyen
[44]

Mohammad Reza Taesiri, Abhijay Ghildyal, Saman Zadtootaghaj, Nabajeet Bar- man, and Cor-Paul Bezemer. 2025. VideoGameQA-Bench: Evaluating Vision- Language Models for Video Game Quality Assurance. arXiv:2505.15952 [cs.CV] https://arxiv.org/abs/2505.15952

2025
[45]

Mohammad Reza Taesiri, Finlay Macklon, and Cor-Paul Bezemer. 2022. CLIP meets GamePhysics: towards bug identification in gameplay videos using zero- shot transfer learning. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsyl...

2022
[46]

Xuchen Tan, Deenu Yadav, Faiz Ahmed, and Maleknaz Nayebi. 2025. ImageR: Enhancing Bug Report Clarity by Screenshots. arXiv:2505.01925 [cs.SE] https: //arxiv.org/abs/2505.01925

2025 arXiv
[47]

Andrew Truelove, Shiyue Rong, Eduardo Santana de Almeida, and Iftekhar Ahmed. 2023. Finding the Needle in a Haystack: Detecting Bug Occurrences in Gameplay Videos. arXiv:2311.10926 [cs.SE] https://arxiv.org/abs/2311.10926

2023 arXiv
[48]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). 22444–22455
[49]

Dingbang Wang, Zhaoxu Zhang, Sidong Feng, William G. J. Halfond, and Tingt- ing Yu. 2025. An Empirical Study on Leveraging Images in Automated Bug Report Reproduction. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 27–38. https://doi.org/...

2025
[50]

1992.Individual Comparisons by Ranking Methods

Frank Wilcoxon. 1992.Individual Comparisons by Ranking Methods. Springer New York, New York, NY, 196–202. https://doi.org/10.1007/978-1-4612-4380-9_16

1992 doi
[51]

Yanfu Yan, Nathan Cooper, Oscar Chaparro, Kevin Moran, and Denys Poshy- vanyk. 2024. Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports. In Proceedings of the IEEE/ACM 46th In- ternational Conference on Software Engineering (Lisbon,...

2024
[52]

Yan Zhao, Weihao Zhang, Enyi Tang, Haipeng Cai, Xi Guo, and Na Meng. 2021. A Lightweight Approach of Human-Like Playtesting. arXiv:2102.13026 [cs.SE] https://arxiv.org/abs/2102.13026

2021 arXiv
[53]

Alicia Vidler and Toby Walsh. 2025. Playing games with Large language models: Randomness and strategy. arXiv:2503.02582 [cs.AI] https://arxiv.org/abs/2503. 02582

2025 arXiv
[58]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information ...

2023
[2022]

arXiv:2204.04182 [cs.SE] https://arxiv.org/abs/2204.04182

Towards Using Gameplay Videos for Detecting Issues in Video Games. arXiv:2204.04182 [cs.SE] https://arxiv.org/abs/2204.04182

arXiv
[2023]

Using gameplay videos for detecting issues in video games.Empirical Softw. Engg. 28, 6 (Oct. 2023), 32 pages. https://doi.org/10.1007/s10664-023-10365-0

2023 doi
[2024]

Is Temperature the Creativity Parameter of Large Language Models? arXiv:2405.00492 [cs.CL] https://arxiv.org/abs/2405.00492

arXiv
[2025]

ACM Trans

Automatic Identification of Game Stuttering via Gameplay Videos Analysis. ACM Trans. Softw. Eng. Methodol. 34, 2, Article 38 (Jan. 2025), 29 pages. https: //doi.org/10.1145/3695992

2025 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

2025. FFmpeg. https://ffmpeg.org/

work page 2025

[2] [2]

Anthropic. 2024. Introducing the next generation of Claude . https://www. anthropic.com/news/claude-3-family

work page 2024

[3] [3]

Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2021. Automated Video Game Testing Using Synthetic and Humanlike Agents. IEEE Transactions on Games 13, 1 (2021), 50–67. https://doi.org/10.1109/TG.2019.2947597

arXiv 2021

[4] [4]

Atlassian. 2025. Jira. https://www.atlassian.com/software/jira Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models

work page 2025

[5] [5]

Elham Azizi and Loutfouz Zaman. 2023. Automatic Bug Detection in Games using LSTM Networks. In 2023 IEEE Conference on Games (CoG) . 1–4. https: //doi.org/10.1109/CoG57401.2023.10333253

arXiv 2023

[6] [6]

Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software En- gineering (Atlanta, Georgia)(SIGSOFT ’08/FSE-16). Association for Computing Ma- chinery, New York, NY, USA, 308–318. https:/...

work page arXiv 2008

[7] [7]

Stefano Campanella, Emanuela Guglielmi, Rocco Oliveto, Gabriele Bavota, and Simone Scalabrino. 2024. Towards the Automatic Replication of Gameplays to Support Game Debugging. In Proceedings of the 1st ACM International Workshop on Foundations of Applied Software Engineering for Games (Porto de Galinhas, Brazil) (FaSE4Games 2024). Association for Computing...

work page arXiv 2024

[8] [8]

Norman Cliff. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions. Vol. 114. 494–509

work page 1993