{"id":"121036dd-c435-43cd-a4d7-c0623efd0e69","arxiv_id":"2608.07014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Across five video LLMs, 12.5-25.5% of multiple-choice items are correct at a lower visual budget but wrong at a higher one, even when aggregate scaling curves look safe.","lead":"This paper shows that average accuracy curves for video AI models can hide large per-question reversals: as the visual budget grows, many questions switch from correct to wrong while many others switch the other way, so the mean looks stable. It provides an item-level audit framework and per-question data to make scaling claims more honest and to support cheaper adaptive inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frame-count trajectories may conflate budget size with which frames are shown, since the grids are not stated to be nested.","rationale":"The paper's core claim is that aggregate scaling curves hide large opposing item-level transitions across visual budgets. The most load-bearing assumption is therefore that observed per-item flips between budget levels are caused by the budget amount and not by uncontrolled changes in which frames are presented. The manuscript never states that frame subsets are nested across budgets, and its own fixed-budget sampling experiment demonstrates that frame selection alone can flip item correctness and that sampling-sensitive items are overrepresented among confused items. This makes the published churn and harmful-transition rates vulnerable to a confounding that is directly testable. The reader's weakest assumption concerned ordinal cost definitions in Appendix C.5; that issue is real but secondary, because the headline matched-grid numbers are computed on frame-only grids where frame count is a well-defined order, and because the same churn phenomenon appears on resolution and iso-budget grids. The reader's verdict of CONDITIONAL is already appropriate; this concern adds a specific validity condition to the central measurement rather than overturning the conclusion in either direction. A nested-grid replication would settle whether the phenomenon is primarily about budget size or about sampling sensitivity, and the paper already ships the infrastructure needed to run it.","tokens_in":25568,"tokens_out":7389,"duration_ms":90021,"concrete_test":"Run Qwen2.5-VL-7B on Video-MME v1 short with a strictly nested frame grid: define 256 candidate frames uniformly spanning each video; use 16f = every 16th candidate, 32f = every 8th, 64f = every 4th, 128f = every 2nd, and 256f = all candidates. Recompute best-fixed accuracy, oracle headroom, adjacent churn, and visual-confusion rate on this nested grid, and compare them with the published non-nested grid. If confusion and churn do not drop materially (e.g., within the reported confidence intervals), the headline effect is robust to sampling nesting; if they drop toward zero, the central claim must be re-expressed as item-level sensitivity to frame selection rather than to budget scale.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical quantity is the paired trajectory over frame budgets, but the paper does not establish that increasing the frame budget presents a superset of the lower-budget frames. Appendix C.2 and C.4 describe \"uniformly subsample the requested budget\" from the 1-fps JPEG cache and \"uniform decoding\" from raw video at each requested budget; no nesting or aligned-start condition is stated. If 16f, 32f, 64f, 128f, and 256f are independently spaced subsets (e.g., linspace over the video), then a \"correct at 16f, wrong at 32f\" transition changes both the number of frames and which timestamps are visible. The paper itself shows at fixed 64f that uniform vs. random sampling flips 4.7% of items, and confused items are 3.83x enriched for sampling sensitivity (Sec. 6.2, Table 6). Consequently, churn and \"harmful higher-budget transition\" rates (12.5-25.5%, Table 2) could be substantially attributable to resampling artifacts rather than to the visual budget amount. This is the load-bearing measurement assumption behind the headline claim; the cost-ordering caveat of Appendix C.5 is secondary because frame-only and resolution grids mostly avoid mixed cost comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an item-level paired-trajectory audit for Video LLM scaling: for each frozen model–item pair, the authors record correctness or a continuous score under several visual budgets and then compute oracle headroom, visual confusion, harmful transitions, and text overwrite. Across five open Video LLMs, four MCQA splits, open-ended QA and summarization, and controlled dialogue generation, they report substantial configuration complementarity (8.8–18.9 accuracy points of oracle headroom and 12.5–25.5% visual confusion on the matched MCQA grid) even where aggregate scaling curves look stable or saturating. They also document protocol dependence, a sampling intervention that recovers 29.0% of terminal regressions, and a confidence cascade that matches fixed-128f accuracy at 31.7% lower equivalent frame cost.","tokens_in":25804,"tokens_out":9452,"duration_ms":101836,"significance":"The paper is methodologically strong in several respects: the central quantities are direct counts from released trajectories, the identity A(c')−A(c)=R−H is an exact algebraic decomposition, confidence intervals use paired item bootstraps, policy contrasts use McNemar tests, and the raw-video/cache replication gives the same +8.78-point headroom in both pipelines. The full disclosure of the Qwen3-VL-8B re-audit and the release of approximately 0.13M per-item records with provenance tags are exemplary transparency practices. If the central measurement concern below is resolved, the paper makes a significant contribution: it provides a reusable audit protocol that turns aggregate scaling curves into per-item response matrices and shows that mean improvements can conceal large opposing flows. The main caveat is the frame-nesting issue, which affects the causal interpretation of 'higher budget harms'; this is addressable and does not require rejecting the paper's empirical program.","major_comments":[{"comment":"The frame-budget grids are not stated to be nested. Appendix C.4 describes cached cells as 'uniformly subsample the requested budget' from a 1-fps JPEG cache and raw cells as 'uniform decoding' from raw video, with no condition that the frame set at budget c is a subset of the frame set at budget c′>c. Under independent uniform draws, the flow counts Rc,c′ and Hc,c′ in Eq. (1) mix an increase in budget size with a change in which timestamps are visible. This is not hypothetical: the paper's own fixed-budget sampling comparison at 64f (Appendix E.2, §6.2) flips 4.7% of items between uniform and random sampling, and visually confused items are 3.83× enriched for sampling sensitivity. Because the reported 5–15% adjacent churn (§4, Figure 2) and 12.5–25.5% visual confusion (Table 2) are of the same order as the sampling-induced flip rate, the headline interpretation 'correct at a lower budget but wrong at a higher one' is not yet identified with the budget amount. Please either state and verify a nested sampling rule (for example floor(i×T/N) on a common timeline) for every reported cell, or repeat the matched-grid analysis with explicitly nested frame sets or with fixed frame positions and report whether the headroom and confusion rates survive. This is load-bearing for the central claim and should be addressed before publication.","section":"§3, Appendix C.4"},{"comment":"The reference cascade is selected in-sample. The margin threshold θ=0.91 and the 16f→32f→128f design are chosen from a full sweep on the same Video-MME v1-short split (Table 18), and the headline 31.7% cost reduction is measured on that split. The paper is careful to call it a 'reference' cascade and to separate deployable from oracle-gated policies, but as written §7 and the Abstract present the cascade as an operational demonstration. Please add a held-out or cross-validated evaluation, or explicitly relabel the numbers as a retrospective illustration for the demonstrated policy class. This does not affect the item-level heterogeneity measurement, but it does affect one of the claimed applications.","section":"§7, Tables 7 and 18, Appendix H"}],"minor_comments":[{"comment":"The Abstract's '31.7% lower shared frame cost' refers to equivalent-frame cost; Appendix H shows that the wall-clock savings are only realized when frame sharing or KV-cache reuse is available, and that the no-share cascade is 13.7% slower than fixed-128f. Consider stating 'equivalent-frame cost' in the Abstract to avoid a misreading as wall-clock savings.","section":"Abstract and Appendix H"},{"comment":"The cost ordering for mixed frame-resolution comparisons (total pixel count) is a definitional assumption. The main text should state explicitly that the headline matched-grid results are frame-only at fixed resolution, so they do not depend on the total-pixel ordering.","section":"Appendix C.5"},{"comment":"Since Qwen3-VL-8B required a re-audit and InternVL3-8B showed a +1.78-point run-to-run difference at 64f, the paper should state whether all final Table 2 numbers come from a single audited runner or from a protocol that permits such reruns; this would help readers interpret the exact-count gates.","section":"Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually transparent and the released artifact is a genuine community asset. The frame-nesting issue is the main technical obstacle: if the grids are nested, a sentence in Appendix C.4 will suffice; if not, a nested-grid robustness analysis is needed. Once that is addressed and the cascade is relabeled as in-sample, I would be glad to see the paper accepted. No concerns about novelty, scope, or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2608.07014. First, it is a serious measurement paper: the authors actually did the matched-grid, per-item work, shipped a large artifact, and were unusually transparent about re-audits. Second, the headline numbers carry a caveat the paper does not fully resolve: the frame-budget trajectories are not shown to be nested, so \"correct at 16f, wrong at 32f\" may partly be a resampling artifact, not a budget-size effect.\n\nWhat is new: the paired item-level trajectory audit, the rescued/harmed flow decomposition, and the matched-grid comparison across five models with protocol provenance. The identity A(c') - A(c) = R - H is a genuinely useful way to reframe scaling evaluation. The empirical work is careful: 5,000-pair bootstrap CIs, McNemar and permutation tests, raw/cache replication with exactly matching +8.78 point headroom, and disclosed re-audits for Qwen3 and InternVL3.5. That is real evidence, and the release of 0.13M per-item records is the right kind of contribution. The citation pattern is unremarkable; the relevant scaling and frame-selection literature is covered.\n\nThe soft spots, in order of importance. First, the nesting issue. Appendix C.2/C.4 say cached cells \"uniformly subsample the requested budget\" and raw cells decode uniformly, but nowhere does the paper state that the 16f/32f/64f/128f/256f grids are nested or aligned. If they are independent evenly spaced subsets, then a budget transition changes both the number of frames and their timestamps. The paper itself shows at fixed 64f that uniform vs. random sampling flips 4.7% of items, and confused items are 3.83x enriched for sampling sensitivity. So a meaningful slice of the 12.5-25.5% harmful-transition rate could be a frame-selection artifact. I do not think this kills the central claim—the fixed-resolution oracle gaps, the iso-compute results, and the persistent regressions in the sampling intervention all survive—but the headline rates need a nested-grid replication or at minimum an explicit statement and sensitivity analysis.\n\nSecond, the cascade threshold is tuned on the same split it is evaluated on. That is fine as an in-sample illustration, but it is presented as an operational result; it needs held-out validation or softer framing. Third, Table 2 lacks confidence intervals, and the artifact has no direct URL or commit hash in the text. Minor, but for a reproducibility-first paper, the missing link is awkward.\n\nOverall, this is a solid paper with one load-bearing assumption that needs checking. The authors are clearly thinking clearly and honestly. Send it to peer review; make the nesting question a required revision, then the numbers can be trusted.","headline":"A careful, high-effort measurement paper that convincingly shows aggregate Video-LLM scaling curves hide item-level churn, but the headline harmful-transition rates need a nested-grid check before being taken at face value.","tokens_in":26340,"tokens_out":4525,"would_cite":true,"duration_ms":47617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that aggregate scaling curves for Video LLMs conceal large, opposing item-level changes as visual budgets grow, and provides a per-item trajectory protocol that exposes the hidden churn.","keywords":["video LLMs","scaling curves","item-level churn","oracle headroom","visual confusion","text overwrite","confidence cascade","evaluation protocol"],"falsifier":"Run a large video LLM benchmark under a matched frame grid and check whether every item's correctness trajectory is monotone: if no item is correct at a lower budget and wrong at a higher one, and the item oracle equals best fixed accuracy, the central claim would be refuted for that model–benchmark pair. A cheaper check uses the released per-item response matrix to compute the fraction of items with rescued or harmed transitions; the claim predicts a substantial fraction (12.5–25.5% confusion on matched MCQA), so a result near zero would falsify it.","tokens_in":25359,"feed_emoji":"🎬","tokens_out":4598,"duration_ms":42181,"temperature":0.7,"pith_summary":"This paper argues that aggregate scaling curves overstate how safely video-language models improve with larger visual budgets. Tracking the same frozen model on the same item under controlled frame counts and resolutions shows large opposing movements: an item that is wrong at 16 frames can be right at 128, while another flips in the opposite direction. Across five open Video LLMs, matched multiple-choice grids show item-level oracle headroom of 8.8–18.9 accuracy points, and 12.5–25.5% of items are correct at a lower budget but wrong at a higher one. The pattern persists in open-ended QA, summarization, and dialogue generation, and across sampling, resolution, and raw-video/cache pipelines. The paper's contribution is an evaluation protocol: per-item response trajectories plus matched-grid metrics that make cancellation visible, along with a released artifact and a confidence cascade that matches fixed-128f accuracy at 31.7% lower average shared frame cost.","feed_headline":"Scaling curves hide 12.5–25.5% harmful flips in video LLMs","feed_subtitle":"Per-item trajectories expose 8.8–18.9-point oracle headroom and enable a 31.7% cheaper confidence cascade.","key_machinery":"The central object is the paired budget trajectory τ_i(C)=(y_{i,c})_{c∈C}: the correctness (or task score) of one frozen model on one item across an ordered configuration grid. Two functionals are built from it: best fixed accuracy A_fixed = max_c (1/n)Σ_i y_{i,c}, and item oracle A_oracle = (1/n)Σ_i max_c y_{i,c}; their difference is oracle headroom. The paper's key identity is A_{c′}−A_c = R_{c,c′}−H_{c,c′}, with R and H the rescued and harmed fractions, while pairwise churn is R+H, so a small net change implies nothing about either directional flow. Visual confusion is defined as an item correct at a lower-cost configuration but wrong at a higher one, with text overwrite as the special case where the lower-cost correct configuration is text only. This construction turns the response matrix into an audit object and a routing surface for confidence cascades.","core_discovery":"The central discovery is that the mean scaling curve is a difference of two large opposing flows, not a summary of stable behavior. For any ordered budget pair c