{"id":"b4fa9259-2c08-43a3-97a1-64c59351a481","arxiv_id":"2507.22099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 1,000-video benchmark, PhysiXFails, with a 17-category taxonomy of physics failures, shows prompt-tuned large multimodal models outperform video anomaly detectors at detecting and naming physics rule violations in game software.","lead":"This paper builds a 1,000-video benchmark of physics engine failures drawn from games and tests eight families of anomaly and multimodal models on detecting them. The best method, a specially prompted Gemini model, reaches 89.5% accuracy at naming the violated rule, while most off-the-shelf detectors stay near chance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ground-truth labels and unmatched negative clips for PhysiXFails are the load-bearing assumption; no inter-annotator agreement or matched controls are reported, so the method rankings may reflect clip selection rather than physics-failure detection ability.","rationale":"I agree with the reader that ground-truth label validity is the weakest assumption. It is load-bearing because the paper's contributions—taxonomy percentages (Section IV), VD/VI accuracy tables (Tables III-VI), and the multi-violation amplification effect (Section VI.D)—all cascade from it. I considered two other threats. First, undisclosed thresholds: Tables III-VI report binary accuracy/F1 but the per-method decision thresholds are not stated; this is a real threat but is fixable by reporting threshold/ROC analysis and does not undermine the dataset itself. Second, GPT-4o-in-the-loop circularity: Section IV.A used GPT-4o to refine labels and categories, and PhyGenEval/Gemini are later evaluated on those labels; this is best understood as a component of label validity rather than a separate flaw. The multi-violation comparison is also a selection-artifact risk, but it is a secondary finding; the main benchmark claim stands or falls with label validity. The paper partially acknowledges the risk in Section VIII.A, but acknowledgment without measurement is not enough. A CONDITIONAL verdict remains appropriate: the dataset is potentially valuable, but acceptance should require release, inter-annotator agreement, and a matched-negative check. Hence my read does not change the reader's verdict.","tokens_in":22074,"tokens_out":5526,"duration_ms":70822,"concrete_test":"Release the full dataset with source URLs and timestamps, then have two independent annotators who are blind to source and to the paper's labels classify a stratified random sample (e.g., 200 buggy + 200 non-buggy clips) as physics-failure vs. not, and assign taxonomy categories. Report Cohen's kappa and category-level agreement. Separately, construct a matched negative set: for each of 100 buggy clips, take a clip from the same game/title and similar scene in which the reported bug does not occur (or a clip immediately preceding the glitch if available). Re-run DEVIL-Gemini-VD and Gemini-Custom-VI on this matched set. If kappa < 0.6, or if the accuracy of the top methods drops by more than 5 points on matched negatives relative to the current unmatched evaluation, the benchmark's ground truth and the reported rankings are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the community gets a reusable benchmark and that LMM-based methods outperform traditional CV approaches—depends on the validity of the 1,000 PhysiXFails labels. Section III.A sources the 500 buggy clips from community bug reports (429 YouTube, 54 Reddit, 17 GlitchBench) and the 500 non-buggy clips 'all sourced from YouTube.' Section IV.A says the taxonomy was produced by open coding with 'initial annotations ... iteratively refined using a large multimodal model (LMM) such as GPT-4o.' No inter-annotator agreement is reported, and Section VIII.A concedes 'subjective judgments may have influenced the classification of physics failures.' If a buggy clip is actually an intentional game mechanic or a rendering artifact, or if a non-buggy clip contains an unnoticed violation, every accuracy/F1/AUC in Tables III-VI is mismeasured. The unmatched negative set worsens this: positives are user-curated, often dramatic bug compilations while negatives are ordinary gameplay, so a detector can score high by recognizing video oddity or editing style instead of physical inconsistency. The 39-video multi-violation comparison (Section VI.D) inherits the same label uncertainty and adds a selection confound because the multi-failure set is not matched to the single-failure set. This is not an internal contradiction in the paper; it is an unverified empirical precondition. The right fix is to show label reliability and matched negatives before accepting the benchmark's rankings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PhysiXFails, a dataset of 1,000 runtime video clips (500 buggy and 500 non-buggy) drawn from community bug reports and YouTube, along with a 17-category taxonomy of physics failure manifestations, an evaluation of eight detection approaches (video anomaly detection, prompt-based LMMs, and fine-tuned multimodal models), and a developer survey with 32 practitioners. The central claims are that LMM-based methods generally outperform traditional computer vision approaches, that custom Gemini prompts achieve particularly high violation-identification accuracy (up to 89.5%), and that multi-violation videos are sometimes easier to detect than single-violation ones. The authors release the dataset, code, and questionnaire responses.","tokens_in":22323,"tokens_out":5204,"duration_ms":58909,"significance":"If the benchmark and its labels are valid, this is a useful contribution: it is the first large-scale dataset targeted specifically at physics failures in PE-based software, the taxonomy offers a structured vocabulary for future work, and the evaluation provides a broad comparison across several model families. The inclusion of a developer study adds practical context, and the public release of dataset and code supports reproducibility. The main value, however, rests on the validity of the ground-truth labels and on whether the reported method rankings measure physics-failure detection rather than dataset-selection artifacts.","major_comments":[{"comment":"The ground-truth labels for the 500 buggy and 500 non-buggy clips are the basis for every accuracy, F1, and AUC reported in Tables III-VI, yet no inter-annotator agreement is reported. Section IV.A states that 'initial annotations were iteratively refined using a large multimodal model (LMM) such as GPT-4o,' and Section VIII.A concedes that 'subjective judgments may have influenced the classification of physics failures.' This is a load-bearing assumption for the benchmark's validity. Please report quantitative label-reliability statistics (e.g., Cohen's kappa on an independently annotated random subset), and explicitly state the role of GPT-4o in per-video labeling as opposed to taxonomy refinement.","section":"Section III.A and Section IV.A"},{"comment":"The negative set is not matched to the positive set. Non-buggy videos are 'all sourced from YouTube,' while 71 buggy videos come from Reddit and GlitchBench in addition to YouTube; moreover, the buggy videos are user-curated bug reports, which may differ systematically from ordinary gameplay footage in editing style, scene selection, and visual content. Consequently, a detector could achieve high discrimination by exploiting these distributional differences rather than by identifying physical inconsistencies. Please either construct a matched negative set (e.g., same games, comparable scene types, similar production characteristics) or provide a controlled analysis, such as same-game pairs or a human-verified matched subset, to rule out this confound.","section":"Section III.A"},{"comment":"The 'multi-violation amplification effect' is based on only 39 videos (PhysiXMultiFails) compared against a much larger, unmatched set of single-violation videos. The multi-violation subset is not matched on game, video length, scene complexity, or source, so the reported 9.3-percentage-point accuracy improvement for PhyGenEval could be a selection artifact rather than a genuine property of multi-failure detection. Please either match the multi-violation videos to single-violation clips on these dimensions or limit the claim to the current subset and explicitly discuss the mismatch as a threat to the finding.","section":"Section VI.D"},{"comment":"Because the taxonomy and per-video labels were iteratively refined with GPT-4o, and the top-performing VI methods use Gemini (another general LMM), the evaluation may reward agreement with GPT-4o's classification biases instead of independently established physical ground truth. This is not an accusation of circularity in the derivation, but a correctness-risk concern. Please clarify whether the LMM was used only to suggest candidate categories or also to assign video-level labels, and provide a human-labeled validation sample (e.g., 100 videos annotated by independent human judges without LMM involvement) to confirm that the taxonomy and method rankings hold under human gold labels.","section":"Section IV.A and Tables III-VI"}],"minor_comments":[{"comment":"The text refers to 'Tables V and VI' when discussing full-dataset violation detection and identification, but the full-dataset results appear in Tables III and IV; the caption of Table IV also says 'PHYSIXMULTIFAILS' even though the table reports full-dataset results. Please renumber and relabel the tables consistently.","section":"Section VI.A"},{"comment":"The model is called 'VideoPhy' in the text and figures (e.g., Figure 2) but 'VideoCon-Physics' in Table I. Please use a single name consistently throughout.","section":"Section V.A and Table I"},{"comment":"The subfigures inside Figure 11 are labeled as separate figures (Figure 2 through Figure 10) in the caption, and the caption text says 'This figure presents the density plots' while the main text refers to 'Figure 11.' Please align the subfigure numbering and caption references.","section":"Figure 11"},{"comment":"The developer questionnaire contains several typos, including 'sofwtare,' 'Maintainence,' and 'Sparsity.' Please proofread the appendix before publication.","section":"Appendix"},{"comment":"References [7] and [27] are the same PhyFu paper, and reference [2] appears twice in the introduction. Please deduplicate and renumber.","section":"References"},{"comment":"In the first paragraph of Section VI.D, '76.3% accuracy on PHYSIXMULTIFAILS compared to 67.0% on PHYSIXFAILS' should be compared against the single-violation accuracy of 66.9% shown in Table V; please use consistent numbers in the text and tables.","section":"Section VI.D"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution in scope, and the dataset release is commendable. The main risk is not internal inconsistency but an unverified empirical precondition: label reliability and matched negative controls. I believe the authors can address this within a major revision by adding inter-annotator agreement statistics, clarifying the GPT-4o role, and either matching the negative set or explicitly analyzing the confound. If they cannot provide such evidence, the benchmark's claims and rankings would remain unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper builds the first 1,000-clip benchmark of physics failures in PE-based software, a 17-category taxonomy, and a head-to-head comparison of eight method families including video anomaly detectors, fine-tuned multimodal metrics, and prompt-tuned LMMs. The benchmark and taxonomy are genuinely new relative to GlitchBench and VideoPhy, and the comparison is useful: prompt-tuned Gemini variants reach ~90% accuracy on violation identification, while most dedicated methods land in the 50–70% range. That is a concrete, if preliminary, data point. The developer survey adds practical context, though it is small (n=32) and junior-heavy.\n\nThe soft spots are centered on the evaluation. Ground-truth labels come from community bug reports and GPT-4o-assisted open coding, with no inter-annotator agreement reported; the paper itself concedes subjective judgment in VIII.A. The non-buggy clips are unmatched YouTube gameplay, which means a detector could be scoring video oddity or editing style rather than physical inconsistency. The multi-violation \"amplification effect\" (PhyGenEval gains 9.3 points) rests on 39 unmatched videos, so it is a hypothesis, not a finding. There is also a mild circularity: GPT-4o helped shape the taxonomy, and GPT-4o-backed models are evaluated against it. Decision thresholds for score-based detectors are not disclosed, which hurts reproducibility.\n\nNone of this kills the paper. The dataset and taxonomy are valuable contributions, and the method comparison is a reasonable first pass. But the label set is the load-bearing assumption, and the authors should be asked to release the data, report inter-annotator agreement, match or stratify the negatives, and run a matched multi-violation analysis before the rankings are taken at face value.\n\nI would send this to peer review — it deserves referee time. The contributions are real, and the flaws are fixable. I'd cite it if I needed a benchmark or baseline in this area, with the caveat that the labels are not yet independently verified.\n\nRegards.","headline":"A genuinely useful benchmark and taxonomy, but the method rankings need label validation and matched controls before they can be trusted.","tokens_in":22910,"tokens_out":3004,"would_cite":true,"duration_ms":32622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PhysiXFails shows physics failures in engine-based software are detectable from video alone, and prompt-tuned multimodal models beat specialized detectors, reaching 89.5% identification accuracy.","keywords":["physics engine testing","physics failure detection","runtime video analysis","large multimodal models","prompt engineering","video anomaly detection","benchmark dataset","failure taxonomy"],"falsifier":"Ask independent physics experts to re-label a random sample of PhysiXFails clips without seeing the original labels; if agreement is low, or if many buggy clips turn out to be intentional game design, the method rankings collapse. To test the multi-violation effect separately, compare the 39 multi-failure clips against a matched set of 39 single-failure clips from the same games; if the 9.3-point improvement disappears, it was a selection artifact.","tokens_in":21830,"feed_emoji":"🎮","tokens_out":10782,"duration_ms":105276,"temperature":0.7,"pith_summary":"This paper tries to establish that physics failures in software built on physics engines can be studied systematically from runtime video alone, with no access to engine internals, and that current detection technology is better than commonly assumed. The authors build a 1,000-clip benchmark of buggy and non-buggy gameplay, organize the bugs into a 17-category taxonomy of violated physical principles, and pit eight families of detectors against two tasks: saying whether any physics rule is broken, and saying which rule is broken. Their central results are that prompt-tuned large multimodal models outperform specialized computer-vision and fine-tuned baselines, reaching 89.5% identification accuracy, and that videos with simultaneous violations are easier for physics-aware models, not harder. The developer survey adds a practical reason to care: most practitioners still detect these failures by eye, so a working automated oracle would fill a real gap in reliability testing for games, simulators, and safety-critical systems.","feed_headline":"Prompt-tuned AI beats specialized tools at spotting physics bugs","feed_subtitle":"Best prompt-tuned AI identifies 89.5% of physics violations in a new 1,000-clip benchmark.","key_machinery":"The load-bearing artifact is PhysiXFails itself: a curated, balanced collection of 1,000 runtime video clips (500 buggy, 500 non-buggy) assembled from community bug reports, gameplay videos, and the GlitchBench set, together with a 39-clip multi-violation subset. Around it sits an open-coding taxonomy that sorts the buggy clips into 17 manifestation categories across 10 physics principles, and a two-task evaluation protocol: violation detection (is any rule broken?) and violation identification (which rule?). Prompt design is the third piece of machinery, as most evaluated methods run three progressive prompt rounds that move from context-rich descriptions to a bare general physics-compliance check, and the custom Gemini prompts add freeform, rule-based, and few-shot variants. This three-part structure is what lets the paper attribute performance gaps to model capability versus prompt design rather than to dataset noise.","core_discovery":"On the paper's own terms, the discovery is that physics failures in PE-based software are a distinct, characterizable class of runtime bugs: they show up as observable deviations from physical laws rather than crashes, and they can be labeled, taxonomized, and detected from video without white-box access. The core evidence is PhysiXFails, a balanced set of 1,000 clips in which the 500 buggy examples fall into 17 manifestation categories dominated by gravity violations (40%) and Newton's-law violations (28.2%). Evaluating adapted baselines and prompt-based systems on it, the paper finds that general multimodal models with custom prompts lead the leaderboard, with DEVIL-Gemini reaching 69.2% accuracy on violation detection and a custom Gemini prompt reaching 89.5% on violation identification, while specialized video-anomaly models often collapse to chance or to always-alarm behavior. The paper further reports that the 39-clip multi-violation subset is easier, not harder, for physics-aware methods, with PhyGenEval improving 9.3 percentage points over its full-dataset accuracy.","pith_inferences":["The same prompt-tuning recipe is likely to transfer to footage from safety-critical simulators, such as autonomous driving or surgical robotics, where physics failures are rarer and more costly, an extension the paper does not test.","The multi-violation 'amplification effect' rests on only 39 clips, so it may be a selection artifact; a matched single-versus-multi comparison would settle whether concurrent violations genuinely sharpen the signal.","Because gravity and Newton's-law violations dominate the corpus, the taxonomy may capture what players notice and report rather than the true distribution of physics-engine bugs; logs or fuzzer-generated failures would probably look different."],"forward_implications":["The benchmark and taxonomy become a common measuring stick for future detectors, and the best 69.2% detection accuracy marks the headroom left to close.","Custom-prompted general multimodal models, at 89.5% identification accuracy, can serve as black-box runtime oracles that need neither engine internals nor per-game retraining.","Physics-aware methods improve on multi-violation clips while generic anomaly detectors degrade, so detector choice should follow whether a method models physical rules rather than just motion statistics.","The developer survey directly motivates putting such detectors into CI/CD pipelines: 84.4% of respondents still rely on visual inspection and 81.3% want real-time detection."],"supporting_citations":[{"why":"Supplies the existing PhyFu fuzzing approach to physics engines that motivates the paper's black-box, runtime-video angle.","marker":"[7]"},{"why":"Provides a weakly supervised video anomaly detection baseline whose near-zero precision on PhysiXFails anchors the claim that traditional computer vision falls short.","marker":"[12]"},{"why":"Supplies the self-supervised video anomaly detection baseline used for comparison.","marker":"[13]"},{"why":"Offers a fine-tuned multimodal anomaly detector whose failure to identify violations demonstrates the model-capability gap.","marker":"[14]"},{"why":"Supplies the VideoPhy model, a fine-tuned physics-commonsense evaluator used as a baseline in both detection and identification tasks.","marker":"[15]"},{"why":"Supplies VideoScore, a video-quality evaluator that fails to discriminate physics bugs, anchoring the negative result.","marker":"[16]"},{"why":"Provides the DEVIL naturalness prompt protocol that, with the Gemini model, achieves the best violation detection accuracy of 69.2%.","marker":"[17]"},{"why":"Supplies the general multimodal model underlying the best-performing prompt variants in the study.","marker":"[19]"},{"why":"Supplies PhyGenEval, the physics-centric evaluation framework whose contextual prompts show strong identification and the multi-violation improvement.","marker":"[20]"},{"why":"Contributes GlitchBench, whose still-available cases supply 17 buggy clips and the methodological precedent for prompting LMMs to spot game glitches.","marker":"[21]"}],"fun_headline_variants":["Prompt-tuned AI outdetects specialized tools for physics engine bugs","Physics bugs caught by prompt tuning, not specialized models","New benchmark shows general AI beats specialized models at physics failure detection","89.5% accuracy: Prompt-tuned Gemini finds physics violations others miss","Physics engine failures: How prompt tuning beats dedicated detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 500 'buggy' clips really are physics failures and the 500 'non-buggy' clips are clean; if these labels are wrong, every reported accuracy measures label noise, not detection skill.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-tuned AI outdetects specialized tools for physics engine bugs","Physics bugs caught by prompt tuning, not specialized models","New benchmark shows general AI beats specialized models at physics failure detection","89.5% accuracy: Prompt-tuned Gemini finds physics violations others miss","Physics engine failures: How prompt tuning beats dedicated detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3196,"prompt_tokens":959,"completion_tokens":2237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2151}},"tokens_in":575,"tokens_out":2237,"duration_ms":15609,"temperature":1.0,"reasoning_tokens":2151,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:03:59.023408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent physics experts to re-label a random sample of PhysiXFails clips without seeing the original labels; if agreement is low, or if many buggy clips turn out to be intentional game design, the method rankings collapse. To test the multi-violation effect separately, compare the 39 multi-failure clips against a matched set of 39 single-failure clips from the same games; if the 9.3-point improvement disappears, it was a selection artifact.","supporting_citations":[{"cited_title":"Mist: Multiple instance self- training framework for video anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Provides a weakly supervised video anomaly detection baseline whose near-zero precision on PhysiXFails anchors the claim that traditional computer vision falls short."},{"cited_title":"Self- supervised sparse representation for video anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised video anomaly detection baseline used for comparison."},{"cited_title":"Videophy: Evaluating physical commonsense for video generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the VideoPhy model, a fine-tuned physics-commonsense evaluator used as a baseline in both detection and identification tasks."},{"cited_title":"Evaluation of text-to-video generation models: A dynamics per- spective,","cited_arxiv_id":null,"evidence_quote":"Provides the DEVIL naturalness prompt protocol that, with the Gemini model, achieves the best violation detection accuracy of 69.2%."},{"cited_title":"Gemini: A family of highly capable multimodal models,","cited_arxiv_id":null,"evidence_quote":"Supplies the general multimodal model underlying the best-performing prompt variants in the study."},{"cited_title":"Glitchbench: Can large multimodal models detect video game glitches?","cited_arxiv_id":null,"evidence_quote":"Contributes GlitchBench, whose still-available cases supply 17 buggy clips and the methodological precedent for prompting LMMs to spot game glitches."}],"review_version":1}