{"id":"cb994990-50ae-4ddf-af3f-d81599b46165","arxiv_id":"2608.02150","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PhyCheck contributes a large, human-verified VideoQA benchmark for physical law compliance and violation, and shows that fine-tuning on its hierarchical evidence questions improves Video-LLM physical understanding.","lead":"PhyCheck is a new video question-answering dataset, built on 6,399 AI-generated videos from VideoPhy2, that tests whether video language models can judge if events obey or violate physical laws. It adds fine-grained evidence questions and a small context-sensitive probe, and shows that fine-tuning Qwen2.5-VL on its data substantially improves physical consistency judgments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection of best training order from a small set of runs plus per-prompt test-set overlap with training data could inflate the headline 81.00% gain; a held-out generalization check is needed.","rationale":"The reader identified inherited VideoPhy2 label noise and unverified answerability of fine-grained questions as the weakest assumptions, and routed the paper CONDITIONAL on releasing data/code and specifying the evaluation protocol. Those are genuine concerns, but the single most load-bearing threat to the paper's central quantitative claim is the unreported selection procedure behind the 81.00% result: with only four supervision settings and the best row reported, the test set may have been used for model selection, and no repeats/error bars are given. This is a methodological concern rather than a claim of intentional misconduct, and it does not change the conditional acceptance posture: the benchmark contribution is still valuable, and the improvement should be re-verified with a held-out protocol on at least one more base model. I partially agree with the reader because the label-inheritance issue is real but is secondary to the evaluation-protocol issue for the paper's headline claim; the reader's request for reproducible splits and protocol details already covers part of my concern, so the verdict stays CONDITIONAL.","tokens_in":12120,"tokens_out":1677,"duration_ms":15784,"concrete_test":"Fix a held-out PhyCheck evaluation split (e.g., 20% of videos, stratified by generation model and physical law category) and re-run the full fine-tuning pipeline exactly as in Table 3, selecting the training order and checkpoint only on a separate validation split. Report mean and standard deviation over at least 3 random seeds. Additionally, test the best model on 200-500 fresh VideoPhy2 videos that were not used in building PhyCheck (or a new generation source) to measure out-of-distribution accuracy. If the held-out gain is comparable to 81.00%, the claim stands; if the in-distribution gain collapses or the OOD accuracy is near the base model's, the headline improvement is largely in-distribution label fitting.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that PhyCheck supervision substantially improves physical understanding, evidenced by Qwen2.5-VL rising from 50.93% to 81.00% on the PhyCheck test set (Table 3). Two load-bearing weaknesses make this number fragile. First, the paper reports that the two training orders (coarse-then-fine vs. fine-then-coarse) differ enormously (51.89% vs. 81.00%), yet the full-training results are reported for a single best-order configuration on the same test set used for model selection; the reader is not told whether the test set was used to choose the order, whether checkpoints were selected on a held-out split, or how many trials/random seeds were run. Given the four-condition matrix, picking the best row can inflate gains by chance. Second, PhyCheck is built from VideoPhy2 by deterministically converting its labels into QAs, so the fine-tuned model is trained and tested on the same underlying annotation distribution; nothing in the paper demonstrates generalization to held-out videos, unseen generation models, new physical laws, or rerendered content. Off-the-shelf baselines (Table 2) are in the low-60s, so a jump to 81.00% on the same distribution is plausible as in-distribution memorization or label-cue learning rather than genuine physical understanding, especially given the marginal improvement from fine-grained VQA alone (51.43%). The pilot's context result (0.98 after context vs. 0.18 before) is also computed on the same 50 pairs that motivated the model's training paradigm, so it is not a clean transfer test. The paper's own caveat about the pilot is appropriate, but the main-table result lacks the same caveat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PhyCheck is a video question-answering dataset constructed from 6,399 videos and 69,825 binary QA pairs, organized into coarse-grained physical-law compliance questions and fine-grained evidence questions, with a 50-pair context-sensitive pilot subset. The coarse-grained labels are deterministically derived from VideoPhy2's followed/violated annotations, while the fine-grained questions are generated by Gemini-3 and GPT-5 and filtered by human annotators. The paper benchmarks several Video-LLMs on PhyCheck, reporting accuracies around 60% and a sharp recall drop on violation examples. It then fine-tunes Qwen2.5-VL on different subsets and training orders and reports that fine-grained-then-coarse-grained supervision raises accuracy from 50.93% to 81.00%. A context pilot suggests the fine-tuned model updates its judgments when external causal context is provided (0.18 to 0.98).","tokens_in":12490,"tokens_out":7117,"duration_ms":61096,"significance":"If the headline result is robust, PhyCheck would be a valuable resource: it is large-scale, hierarchically structured, and offers both evaluation and training data, with strong inter-annotator agreement on the generated fine-grained questions. The paper's public code/data URL and the systematic decomposition of physical laws into visual evidence are concrete strengths. However, the central empirical claim currently lacks the protocol details needed to distinguish genuine physical understanding from in-distribution label learning: no held-out generalization test, no validation-based model selection, no statistical significance testing, and no independent verification of the inherited VideoPhy2 labels. The context result is intriguing but based on 50 pairs. For these reasons the contribution is promising but not yet established.","major_comments":[{"comment":"The reported 81.00% accuracy is selected as the best among four supervision configurations, yet the paper does not state whether the test set was used for this selection, whether checkpoints were chosen on a held-out validation split, or how many random seeds were used. Section 4.1 states only that Qwen2.5-VL was 'fully fine-tuned ... for 12 epochs' on the PhyCheck training set, without giving the number of videos or questions in the train and test splits or the split criterion (e.g., video-level vs. per-generation-model). Because the same test set may have been used for model selection, the gap between 50.93% and 81.00% could be inflated by chance. Please report a fixed train/test split, validation-based checkpoint selection, results over multiple seeds with variance, and a held-out generalization test (e.g., on generation models not seen in training or on new physical-law categories).","section":"Section 4.3, Table 3"},{"comment":"The coarse-grained answers are deterministically derived from VideoPhy2's followed/violated labels, and the fine-grained questions are generated by Gemini-3/GPT-5 and filtered by human binary agreement. This means the training signal and the evaluation ground truth come from the same annotation distribution, and no independent verification of the VideoPhy2 labels (e.g., by fresh human annotators under a protocol that allows them to disagree) is reported. The 99.74% raw inter-annotator agreement measures agreement on the LLM-generated questions, not the correctness of the underlying labels. The 81.00% result is therefore compatible with in-distribution label-cue learning. Please provide an independent human-verified test subset with freshly collected labels, or evaluate on videos and physical laws not used in training, to support the claim of improved physical understanding.","section":"Section 3.2"},{"comment":"The context-sensitive result is computed on the same 50 hand-crafted pairs that define the pilot; the text gives 0.18 video-only and 0.98 context-assisted accuracy for the fine-tuned model but does not report the context-assisted accuracy of the baseline models (only states that 'many struggle'), nor any confidence interval or statistical test. The interpretation that the low video-only score reflects 'epistemic caution' is not uniquely supported; it could be a calibration artifact from fine-tuning on a balanced binary dataset. Please report per-condition numbers for all models on the context subset and validate the finding on a larger or held-out set.","section":"Section 4.5, Appendix C3"}],"minor_comments":[{"comment":"The dataset size is reported as 'approximately 50k human-verified VQA pairs' in the abstract and introduction, but Section 3.1 says PhyCheck contains 69,825 QA pairs; please reconcile these numbers.","section":"Abstract and Section 3.1"},{"comment":"The checkmark notation for the sequential training rows (e.g., '✓ 1⁄bigcircle✓2⁄bigcircle') is unclear; replace it with explicit labels such as 'fine-grained then coarse-grained'.","section":"Section 4.3, Table 3"},{"comment":"Provide the full training hyperparameters (optimizer, learning rate, batch size, number of frames per video, prompt template) and the exact split statistics; without these, the reproducibility of the 81.00% result is limited.","section":"Section 4.1"},{"comment":"The appendix is referenced as 'Appendix C3' but no appendix text is included; either include the appendix or report the numbers inline.","section":"Section 4.5"},{"comment":"The construction of the 50 context pairs is underspecified; specify how many source videos were used, what crops were made, and how the context is presented to models (text-only, image, video, or a combination).","section":"Section 3.3"},{"comment":"The row labeled 'Qwen2.5-Finetune' is not identified in Table 2 or Table 3; add a footnote identifying this as the fine-tuned Qwen2.5-VL model and the training configuration.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is a dataset and training recipe, and the evaluation needs to be strengthened before publication. The main risk is that the 81.00% improvement is a test-set-selected, single-seed result on an in-distribution split. If the authors provide disjoint splits and an external generalization test, the paper could be appropriate for the journal. I would also flag that the dataset's novelty largely rests on re-annotating VideoPhy2; the authors should clarify the incremental contribution over VideoPhy2 itself in the related-work section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid benchmark paper with a headline number that is less solid than it looks. The dataset (69,825 QA pairs over 6,399 synthetic videos, human-verified at 99.74% raw agreement) is a genuinely useful artifact. The fine-grained evidence questions are a real step beyond coarse yes/no or generation-side benchmarks like PhysGame and PhysBench. The central training claim — that fine-to-coarse sequential supervision lifts Qwen2.5-VL from 50.93% to 81.00% on PhyCheck's own test set — is plausible in direction but under-supported in protocol.\n\nCredit where due: the human verification loop is serious, the training orders are compared, and the paper honestly flags the 50-pair context subset as a pilot. The category-wise failure analysis is a nice diagnostic.\n\nSoft spots, in proportion. First, we never learn how many videos/questions are in train vs test, how many random seeds or runs produced Table 3, or whether the test set was used to select the best training order. When two orders give 51.89% and 81.00%, and you pick the winner, you need to show this wasn't cherry-picking. Second, the label pipeline is deterministic from VideoPhy2, so the test distribution is the same as the training distribution; nothing shows generalization to held-out generators, new physical laws, or rerendered content. The 81% might be in-distribution shortcut learning rather than physical understanding — the fact that fine-grained VQA alone adds almost nothing (51.43%) and coarse-only gives 66.83% suggests the gain is mostly from the coarse labels plus ordering, not from the fine-grained evidence per se. Third, no confidence intervals or significance tests anywhere. The context pilot (0.18 to 0.98 on the same 50 pairs) is suggestive but small, and the paper says so.\n\nNone of that makes the paper a rejection. It is an engineering contribution that is reproducible in principle if the authors release data and code. The central claim should be re-verified on at least one more base model and with repeated seeds before the field takes 81% as a fact.\n\nRecommendation: send to peer review. A serious referee should ask for split counts, number of trials, a contamination check, and ideally a cross-model generalization check. I'd bring it to a reading group, and I'd cite the dataset if I were working on video-LLM evaluation.","headline":"A solid, reusable benchmark whose headline 81% fine-tuning gain is plausible but under-supported; it deserves peer review, with protocol details and a second base model requested.","tokens_in":13032,"tokens_out":3125,"would_cite":true,"duration_ms":31911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper aims to establish that PhyCheck, a fine-grained, evidence-grounded video QA dataset, both diagnoses and improves Video-LLMs' physical law understanding, with hierarchical training lifting Qwen2.5-VL from 50.93% to 81.00%…","keywords":["physical law understanding","video question answering","video large language models","evidence-grounded annotation","physics consistency benchmark","hierarchical supervision","synthetic videos","context-sensitive reasoning"],"falsifier":"Run an annotation study where a fresh set of annotators, blind to VideoPhy2 labels and to the LLM-generated questions, independently answer the fine-grained PhyCheck questions from the video alone; if their answers diverge from the dataset's labels beyond a few percent, the evidence-grounded training signal is not what the paper claims.","tokens_in":11824,"feed_emoji":"⚛️","tokens_out":5332,"duration_ms":44307,"temperature":0.7,"pith_summary":"PhyCheck is a video question-answering dataset built on 6,399 synthetic videos from VideoPhy2, re-annotated with about 69,825 binary yes/no questions at two granularities: coarse-grained questions ask whether an event obeys or violates a physical law, and fine-grained questions ask whether specific visual evidence of a violation is present. The paper's central claim is that current video-LLMs only superficially understand physical laws: they score reasonably on conforming examples but their recall collapses on violating examples. Using the dataset as structured supervision, with fine-grained questions trained before coarse-grained ones, fine-tuned Qwen2.5-VL reaches 81.00% accuracy on the PhyCheck test set, up from 50.93% for the base model. The authors also report a 50-pair pilot in which a model trained on PhyCheck becomes more cautious on ambiguous clips yet jumps to 0.98 once external causal context is supplied, suggesting the training fosters context-sensitive recalibration.","feed_headline":"Fine-grained physics questions push video-LLM accuracy from 51% to 81%.","feed_subtitle":"PhyCheck's two-level question set turns Qwen2.5-VL into a stronger judge of which events obey physical laws.","key_machinery":"The load-bearing mechanism is the two-tier, evidence-grounded VQA annotation: each video carries coarse-grained yes/no judgments of law compliance plus fine-grained questions naming a specific visual detail, such as 'a hammer moving without external force.' Template-based generation deterministically assigns answer polarity from VideoPhy2's followed and violated labels, and LLM-assisted drafting (Gemini-3 generates negative prompts; GPT-5 converts them into questions) is filtered by human verification with 99.74% agreement. The training-order finding—fine-grained first, coarse-grained second—is presented as the key that makes the hierarchy effective.","core_discovery":"The discovery is that a benchmark which decomposes physical-law judgments into explicit visual-evidence questions—with a hierarchy where fine-grained questions are learned before coarse-grained consistency questions—serves both as a reliable diagnostic and as training data that materially improves Video-LLMs. Across a spectrum of open-source and proprietary models, most achieve around 60% accuracy on PhyCheck, with high recall on 'conform' examples but low recall on 'violate' examples, confirming that apparent competence hides a bias toward accepting physical statements. Sequential fine-tuning of Qwen2.5-VL on fine-grained then coarse-grained VQA yields 81.00% overall accuracy, and per-category analysis shows the largest gains on law categories requiring implicit reasoning, such as conservation and Newtonian mechanics. The paper presents the context-sensitive pilot as initial evidence that alignment on PhyCheck teaches models to integrate hidden causal factors rather than fixate on superficial visual anomalies.","pith_inferences":["If PhyCheck's supervision signal were shown to generalize, a natural extension is to use its question decomposition as a prompt template at inference time for models not fine-tuned on it, testing whether the evidence chain itself causes the gain.","The reported 81% is on a test set drawn from the same VideoPhy2 distribution as training; nothing in the paper shows transfer to real-world videos or to generators outside the training set, so a cross-distribution benchmark is the obvious next test.","The context-sensitive subset's dramatic jump (0.18 to 0.98) could indicate genuine recalibration or could reflect the model learning to flip its answer whenever supplementary text is present; a control with arbitrary irrelevant context would distinguish these.","The 99.74% agreement only measures agreement between annotators who already saw the LLM-generated questions, not whether fresh viewers can answer them from the video alone."],"forward_implications":["If correct, physical-law understanding in video models is measurable by fine-grained evidence questions, not just coarse verdicts.","A training recipe of evidence questions before consistency questions can lift a Video-LLM's accuracy from about 51% to 81% on this benchmark.","Existing models' high conform-versus-violate recall imbalance indicates a systematic bias, and benchmark builders should include violation-rich splits.","The context-sensitive pilot suggests that recalibrating physical judgments in response to hidden causal factors is trainable.","Category-wise failures, especially in conservation and Newtonian mechanics, point to specific supervision targets for future model training."],"supporting_citations":[{"why":"Supplies the 6,399 synthetic videos and the followed/violated/undetermined labels from which PhyCheck's coarse-grained answers are deterministically derived.","marker":"Bansal et al. [2025]"},{"why":"Defines the Qwen2.5-VL base model that is fine-tuned in the dataset utility validation experiments.","marker":"Bai et al. [2025]"},{"why":"PhysBench is the main comparison benchmark, representing the coarse-grained physical-knowledge evaluation that PhyCheck extends with violation-focused, evidence-grounded questions.","marker":"Chow et al. [2025]"},{"why":"PhysGame is an earlier physical-commonsense benchmark limited to gameplay videos, used to position PhyCheck's broader coverage of physical laws.","marker":"Cao et al. [2024]"},{"why":"MVP provides a shortcut-aware evaluation protocol via minimal video pairs, which PhyCheck contrasts with its fine-grained evidence decomposition.","marker":"Krojer et al. [2025]"},{"why":"Skyra is a specialized AI-generated-video detector whose competitive but not superior performance supports the claim that artifact detection is not equivalent to physical understanding.","marker":"Li et al. [2026]"}],"fun_headline_variants":["Video-LLMs learn physics: fine-tuned on PhyCheck, accuracy jumps to 81%","PhyCheck: two-tier physics questions boost video-LLM from 51% to 81%","From 51% to 81%: PhyCheck fine-tuning teaches video-LLMs physical laws","PhyCheck diagnostic reveals video-LLMs' bias toward conform, not violate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is inherited from VideoPhy2's followed/violated labels, and the fine-grained questions are LLM-drafted and only filtered by human agreement; if those labels or questions are noisy, every accuracy number inherits that noise.","fun_headline_variants_meta":{"raw":{"variants":["Video-LLMs learn physics: fine-tuned on PhyCheck, accuracy jumps to 81%","PhyCheck: two-tier physics questions boost video-LLM from 51% to 81%","From 51% to 81%: PhyCheck fine-tuning teaches video-LLMs physical laws","PhyCheck diagnostic reveals video-LLMs' bias toward conform, not violate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001175,"raw_usage":{"total_tokens":4888,"prompt_tokens":1004,"completion_tokens":3884,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":3782}},"tokens_in":620,"tokens_out":3884,"duration_ms":28115,"temperature":1.0,"reasoning_tokens":3782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:08:02.284751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an annotation study where a fresh set of annotators, blind to VideoPhy2 labels and to the LLM-generated questions, independently answer the fine-grained PhyCheck questions from the video alone; if their answers diverge from the dataset's labels beyond a few percent, the evidence-grounded training signal is not what the paper claims.","supporting_citations":[],"review_version":2}