{"id":"e8284b13-7c29-4ed5-b90a-21e93a0929ce","arxiv_id":"2412.10360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.","lead":"Apollo is a systematic study of how to build video-understanding AI models, testing sampling, encoders, training schedules, and data mixes on hundreds of models. It reports that small 2-4B models reliably predict design choices for larger ones and releases Apollo models that are competitive at small scale, though some top-line benchmark claims are overstated.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling Consistency extrapolation to 30B+ models is mathematically invalid: the Fig. 3 fit R2(s,7B)=0.12 ln(s)+0.78 is anchored to a 7B reference, so it cannot be evaluated at s>7 to predict R2(7B,32B); the curve already exceeds 1 near x=7.","rationale":"The reader identified the four-point fit and the family/size confound as the weakest point. My stress-test agrees but sharpens the issue: the extrapolation to 32B/72B in App. D is not just a 'trend continues' assumption; it is a mathematical category error. The fitted function is R2(s, 7B), i.e., correlation between a model of size s and a fixed 7B model. You cannot substitute s=32 into this function to obtain R2(7B, 32B) — the reference model would change. Even as a descriptive fit for s<7, the curve should pass through R2(7,7)=1; it instead gives ~1.01 with natural log, and exceeds 1 for larger s. The appendix's numbers (>0.86 for 32B, >0.84 for 72B) are therefore not derivable from Fig. 3. This matters because the abstract and conclusion use Scaling Consistency to claim design decisions transfer to much larger models (and the paper even suggests Apollo-7B competes with 30B models). The empirical Apollo results are not invalidated, and the design findings may still be correct, but the central scientific claim is overstated in a way that requires correction. The reader's CONDITIONAL verdict already requires 'address the confound in the scaling analysis'; my analysis strengthens that requirement and specifies that the 32B/72B extrapolation should be removed or replaced with direct evidence. I do not see a reason to move to REJECT: the paper contains substantial empirical work, a novel benchmark, and a rich set of ablations that remain useful; the issue is with the generality of the headline claim, not with the integrity of the experiments. Hence UNCHANGED (still CONDITIONAL).","tokens_in":39780,"tokens_out":10623,"duration_ms":88669,"concrete_test":"Evaluate the fitted curve y = 0.12 ln(x) + 0.78 at x = 7 and x = 32. Since R2(7B,7B) is identically 1, a fitted value of ~1.01 at x=7 and >1.2 at x=32 proves that this curve cannot be the predictor used for the 32B/72B claims; if the authors used a different extrapolation, they should report its formula and fit it to direct 7B-vs-32B/72B correlation data. As a complementary empirical check, retrain the 21 design variations with Qwen2-4B (or Qwen2.5-3B) and recompute R2 to Qwen2-7B; if R2 falls below ~0.9, the size effect is confounded with the Qwen1.5-vs-Qwen2 family change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core scientific claim is Finding 1 (Sec. 3): design decisions made on smaller models transfer reliably to larger ones. The evidence is Fig. 3, a log-linear fit y = 0.12 ln(x) + 0.78 to R2 values between four sizes (0.5, 1.5, 4, 7B) and the 7B reference. Appendix D then states that 'models above the 3–4 billion parameter range can be expected to have high correlation even with much larger models, such as 32B (> R2 = 0.86) or 72B (> R2 = 0.84)'. This extrapolation is not merely unsupported; it misuses the fitted curve. The fitted curve describes correlation with a fixed 7B reference as a function of the smaller model's size. Extending the x-axis beyond 7B does not yield the correlation between 7B and a 32B or 72B model — that would require new data with those larger models as references. Moreover, the curve is inconsistent with the boundary condition R2(7B,7B)=1: at x=7 it gives ~1.01, and at x=32 it exceeds 1.2, an impossible value for R2. The in-range evidence is also confounded: the 4B point is Qwen1.5-4B while the 0.5B and 1.5B points are Qwen2, so the jump from 0.831 to 0.938 is not a clean size effect. Because the paper uses this result to justify running all design explorations at Qwen2.5-3B — a family absent from the scaling study — the weakness propagates to every design finding. The Apollo models' benchmark scores stand on their own, but the claimed scientific contribution of Scaling Consistency for much larger models is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic study of video understanding in large multimodal models (LMMs), covering video sampling, vision encoders, token resampling, token integration, training schedules, and data composition. It introduces ApolloBench, a 400-question benchmark subset intended to be 41x faster and more video-centric than existing suites, and proposes 'Scaling Consistency,' the claim that design decisions validated on smaller models (roughly 2-4B parameters) transfer reliably to larger models. Guided by these findings, the authors train Apollo-1.5B, Apollo-3B, and Apollo-7B and report strong performance on MLVU, Video-MME, LongVideoBench, and other benchmarks, including claims of state-of-the-art results among 7B models.","tokens_in":40132,"tokens_out":3624,"duration_ms":31705,"significance":"If Scaling Consistency holds, the paper would provide a practically important methodology: researchers could validate video-LMM design choices on 2-4B models and ~500K samples instead of training large models, substantially reducing compute. The systematic exploration also yields concrete, actionable findings (e.g., fps sampling over uniform sampling, SigLIP-SO400M as a strong single encoder, Perceiver Resampler over average pooling) and makes raw experimental results available in the appendix. ApolloBench, despite its potential for evaluation-gaming concerns, is a useful contribution toward cheaper and more video-discriminative evaluation. However, the central scientific claim is currently supported by a very small and confounded dataset, and the headline benchmark claims are overstated relative to the paper's own Table 4.","major_comments":[{"comment":"The Scaling Consistency extrapolation is not valid as stated. The fitted curve y = 0.12 log(x) + 0.78 describes R2 between models of size x and a fixed 7B reference. Extending the x-axis beyond 7B cannot produce the correlation between a 7B model and a 32B or 72B model; that would require new data with the larger model as the reference. Moreover, the curve violates the boundary condition R2(7B,7B)=1, as it reaches about 1.01 at x=7 and exceeds 1.2 at x=32, which is impossible for a correlation coefficient. The Appendix D claim that models above 3-4B can be expected to have R2 > 0.86 with 32B or R2 > 0.84 with 72B models is therefore unsupported and should be removed or replaced with actual measurements.","section":"Sec. 3, Fig. 3, App. D"},{"comment":"The in-range evidence for Scaling Consistency is confounded by model family. The 0.5B and 1.5B points use Qwen2 models, while the 4B point uses Qwen1.5-4B. The jump from R2=0.831 (1.5B) to R2=0.938 (4B) may reflect family differences rather than a pure size effect. The paper even states that Qwen1.5-4B has similar or lower performance than Qwen2-1.5B, so the correlation difference cannot be attributed to size alone. A clean test would include a Qwen2-4B model or a Qwen1.5-1.5B model.","section":"Sec. 3, Fig. 3, App. D"},{"comment":"The design exploration is carried out with Qwen2.5-3B, but Qwen2.5-3B is absent from the Scaling Consistency study, which uses Qwen2-0.5B, Qwen2-1.5B, Qwen1.5-4B, and Qwen2-7B. The claim that 'these findings exhibit a strong correlation (R2 > 0.9) with results on larger models and across different model families' does not follow from the presented data, since the specific model family used for all subsequent ablations was never part of the correlation analysis. This weakens every Finding 2-10 that relies on the Qwen2.5-3B proxy.","section":"Sec. 4, first paragraph; Sec. 3"},{"comment":"The claim that Apollo-7B is 'state-of-the-art compared to 7B LMMs' is contradicted by the paper's own Table 4. Qwen2-VL-7B achieves 68.5 on TempCompass vs Apollo-7B's 64.9, and 63.3/69.0 on Video-MME with/without subtitles vs Apollo-7B's 61.3/63.3. Apollo-7B does lead on MLVU and several other columns, but the unqualified SOTA claim overstates the results. The abstract and Section 6 should either name the specific benchmarks where Apollo-7B is SOTA or qualify the claim relative to per-benchmark comparisons.","section":"Table 4; Abstract; Sec. 6"},{"comment":"The training-schedule comparison reports only the best-performing model among three hyperparameter settings per schedule ('For each training schedule, three hyperparameters were tested, and we report the best-performing model'). Reporting best-of-N inflates scores and makes it difficult to assess whether the three-stage advantage is robust or an artifact of hyperparameter selection. Reporting all runs, or at least the median and variance, would substantially strengthen Finding 8.","section":"Sec. 5.1, Table 3"}],"minor_comments":[{"comment":"Typo: 'Perciver Resampler' should read 'Perceiver Resampler'.","section":"Sec. 6, architecture description"},{"comment":"The Introduction reports Apollo-7B's Video-MME (w/o subtitles) as 61.2, while Table 4 reports 61.3. Please reconcile the numbers.","section":"Intro, Sec. 6, Table 4"},{"comment":"ApolloBench is used both as the main ablation metric and as a final evaluation benchmark. Since ApolloBench was curated with a separate set of ten open models and Apollo is also evaluated on external benchmarks, this is not circular reasoning, but the paper should state more explicitly that ApolloBench rankings may not perfectly reflect rankings on external benchmarks.","section":"Sec. 2.3"},{"comment":"The left plot's fitted curve is labeled with '7B 0.5B' in a way that is ambiguous about which points are used for the fit. Please clarify whether the 7B self-correlation point (1.0) is included in the fit.","section":"Fig. 3 caption"},{"comment":"The numbers '> R2 ≃ 0.86' for 32B and '> R2 ≃ 0.84' for 72B appear inconsistent with the fitted equation y = 0.12 log(x) + 0.78, which gives values above 1 for x > 7. Please either remove these extrapolated values or derive them from a correctly specified model.","section":"App. D"}],"recommendation":"major_revision","confidential_remarks":"The paper has useful empirical content and a valuable release of raw results, but the central Scaling Consistency claim needs substantially stronger evidence before it can be regarded as established. In particular, the extrapolation in Appendix D should be removed, and the authors should either add a Qwen2-4B or Qwen2.5-3B point to the correlation study or explicitly limit the claim to the tested range. The SOTA claims in the abstract and Section 6 should also be corrected against Table 4. With these changes, the paper could be a solid contribution; as it stands, the overclaims are likely to mislead readers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid empirical study of video-LMM design choices, but the headline claims outrun the evidence. The core Scaling Consistency result is the load-bearing idea, and as stated it doesn't hold. The four-point fit in Fig. 3 (left) is y = 0.12 ln x + 0.78; at x = 7 that already exceeds 1, so the curve is not a valid R2 model even in range. The 4B point is Qwen1.5, the others are Qwen2, so the size effect is confounded with model family. Extrapolating to 32B/72B in App. D is therefore unsupported, and the stress-test note is right that this is a misuse of the fitted curve.\n\nThat doesn't invalidate the design findings themselves. fps sampling over uniform, SigLIP + InternVideo2 as encoder pairing, the 3-stage schedule, and the 10-14% text data mixture are all plausible and well documented. The paper ships raw results for all 84 models, which is genuinely appreciated. ApolloBench is self-curated but also used alongside established benchmarks, so the circularity risk is low.\n\nThe abstract's SOTA claim for Apollo-7B is too strong: the paper's own Table 4 shows Qwen2-VL-7B beating Apollo-7B on Video-MME and TempCompass. The 'state-of-the-art compared to 7B' phrasing should be corrected to something like 'competitive with the strongest 7B models.'\n\nOther soft spots: Table 3 reports best-of-three hyperparameter settings, which is a little cherry-picked; there are no error bars or repeated seeds anywhere, and given the compute involved that may be forgiveable, but it should at least be acknowledged. The critical-size threshold is vague—somewhere between 2 and 4B—and the dataset-size plateau at ~500K rests on a small number of points.\n\nWhat is genuinely new is the breadth: systematic comparisons across sampling, encoders, pooling, integration, schedules, and data mixtures, plus the ApolloBench analysis that shows how much existing benchmarks reward text-only or single-frame guessing. That is useful for anyone building a video-LMM.\n\nI would send this to peer review; it deserves referee time and the corrected version would be a valuable reference. But the scaling claim needs a major rework: more points in the same family, a proper statistical treatment, and no extrapolations beyond the observed range. The SOTA statements also need to be aligned with Table 4.\n\nFor your reading group, it's a maybe—good for discussing how to responsibly make scaling claims from small-N correlations.\n\nRecommendation: major revision, then seriously consider.","headline":"A wide and useful design-space study whose headline Scaling Consistency claim is overreached; the SOTA and extrapolation claims need major revision, but the fps/encoder/data findings are worth taking seriously.","tokens_in":40778,"tokens_out":2034,"would_cite":true,"duration_ms":21302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that design choices for video large multimodal models can be made on small models and datasets and transferred reliably to larger ones, and that the resulting Apollo model family sets strong results on video benchmarks.","keywords":["video large multimodal models","Scaling Consistency","ApolloBench","frames per second sampling","vision encoder selection","token resampling","training schedules","video benchmark design"],"falsifier":"Train the same 21 design variations within one LLM family at 0.5B, 1.5B, 3B, 7B and, if resources allow, 30B+, then compare their rankings on ApolloBench; if the 3B-to-30B ranking correlation drops well below the reported 4B-to-7B value of $R^2 = 0.938$, Scaling Consistency is falsified. A first check is whether replacing the Qwen1.5-4B point with a same-family Qwen2-4B preserves the reported fit.","tokens_in":39509,"feed_emoji":"🎬","tokens_out":11329,"duration_ms":85334,"temperature":0.7,"pith_summary":"The paper sets out to find what actually drives video understanding in large multimodal models (LMMs) instead of inheriting image-model conventions. Its central claim is Scaling Consistency: design and training decisions made with models of roughly 2–4 billion parameters and datasets of about 500K samples transfer reliably to larger models, so the design space can be explored cheaply. Using that shortcut, the authors compare video sampling, vision encoders, token resamplers, training schedules, and data mixtures, and extract concrete rules such as frames-per-second sampling beating uniform frame sampling. They also introduce ApolloBench, a 400-question multiple-choice benchmark filtered so that text-only or single-frame inputs cannot answer it, which they report is $41\\times$ faster to evaluate and highly correlated with existing video benchmarks. Guided by these findings, Apollo models are claimed to outperform most existing 7B models at 3B scale and to match or beat many 30B models at 7B scale, with Apollo-7B scoring 70.9 on MLVU and 63.3 on Video-MME.","feed_headline":"3B video-AI models can guide 7B design choices","feed_subtitle":"Apollo-7B, built on small-model checks, scores 70.9 on MLVU and 63.3 on Video-MME.","key_machinery":"The mechanism that carries the argument is Scaling Consistency, defined as the transfer of design-decision rankings from small to large LMMs once a critical size is passed; the paper measures this with rank-correlation ($R^2$) over 21 hand-selected model variations and reports the fit $R^2 = 0.12\\log(x) + 0.78$ against the 7B model. The second load-bearing piece is ApolloBench, a 400-question multiple-choice suite filtered so that questions answerable by text-only or single-frame inputs are removed, which makes the small-scale comparisons cheap and more video-specific. These two pieces work together: ApolloBench supplies the fast, discriminative signal, and Scaling Consistency says the signal seen at 2-4B scale will still order larger models correctly, letting the paper's other findings (fps sampling, encoder choice, Perceiver resampler, three-stage training, 10-14% text data) be read as design rules rather than small-model accidents.","core_discovery":"The paper's central discovery, stated on its own terms, is Scaling Consistency: for video large multimodal models there is a critical scale—roughly 2–4 billion parameters and around 500K training samples—above which design decisions transfer reliably to larger models. The evidence is a set of 84 models built by crossing 21 design variations with four LLM sizes (Qwen2-0.5B, Qwen2-1.5B, Qwen1.5-4B, Qwen2-7B), where the agreement between the 4B and 7B configurations reaches $R^2 = 0.938$ and the correlation grows roughly log-linearly with model size. The paper also claims that on this small-scale testbed, frames-per-second sampling beats uniform sampling, SigLIP-SO400M is the best single vision encoder and combines best with InternVideo2, the Perceiver Resampler beats average pooling, and a three-stage schedule with about 10-14% text data in the fine-tuning mix works best. On the evaluation side, it claims many existing video benchmarks can be partially answered from text or a single frame, and it curates ApolloBench, 400 multiple-choice questions that resist such shortcuts and are $41\\times$ faster to score while staying highly correlated with the larger suites. Guided by these choices, the paper presents Apollo-3B and Apollo-7B, which it claims outperform most existing 7B models and rival 30B models, with Apollo-7B scoring 70.9 on MLVU and 63.3 on Video-MME.","pith_inferences":["Editorial extension: the paper extrapolates the log-linear correlation beyond 7B to 32B/72B, but no models above 7B were measured; a direct same-family scaling study at 30B+ would test whether that extrapolation holds.","Editorial extension: because the 4B point uses a different LLM family than the smaller sizes, a same-family replication (for example, Qwen2-0.5/1.5/3/7B) would cleanly separate model size from model family effects.","Editorial extension: the benchmark-decontamination principle behind ApolloBench (filter out questions answerable from text or a single frame) could be applied when building future video benchmarks, not just to the questions selected here.","Editorial extension: if Scaling Consistency holds for unseen choices, small-model-first experimentation could become the default workflow for video-LMM research, with large models used mainly for final confirmation."],"forward_implications":["Video-LMM design studies can be run on 2–4B models and about 500K samples, reserving large-scale training for final verification, which cuts the compute of design exploration by a large factor.","Frames-per-second sampling should replace uniform frame sampling for long-video training, with 8–32 tokens per frame as the recommended operating range.","A recipe combining SigLIP-SO400M and InternVideo2 encoders, a Perceiver Resampler, and clip timestamps between video tokens is a strong default architecture.","ApolloBench provides a cheaper development-time evaluation that ranks models similarly to the full video benchmark suites it was drawn from.","Careful design choices can let a 3B or 7B model compete with open models two to four times its size."],"supporting_citations":[{"why":"Supplies the traditional scaling-law baseline that Scaling Consistency relaxes.","marker":"Hoffmann et al. (2022)"},{"why":"Extends scaling laws to mixed-modal LMM pretraining, motivating the relaxation attempted here.","marker":"Aghajanyan et al. (2023)"},{"why":"Provides another observed scaling-law result for multimodal pretraining that the study builds on.","marker":"Yu et al. (2023)"},{"why":"Introduces the Perceiver Resampler that Apollo adopts as its token compression mechanism.","marker":"Jaegle et al. (2021)"},{"why":"Supplies SigLIP-SO400M, the best single vision encoder and a component of Apollo's dual-encoder setup.","marker":"Zhai et al. (2023)"},{"why":"Supplies InternVideo2, the video encoder combined with SigLIP for Apollo's final architecture.","marker":"Wang et al. (2024d)"},{"why":"Provides Video-MME, one of the benchmarks used for redundancy analysis, curation, and final evaluation.","marker":"Fu et al. (2024)"},{"why":"Provides MLVU, one of the key long-video benchmarks used for correlation analysis and Apollo's headline scores.","marker":"Zhou et al. (2024)"}],"fun_headline_variants":["Scaling Consistency: small video models, big design clues","Apollo: 3B insights, 7B performance","Video AI: test at 3B, deploy at 7B","Small-scale video LMM experiments guide large-scale design","Why 3B video models lead to 7B wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ranking of design choices obtained on 2-to-4-billion-parameter models trained on roughly 500K samples is the same ranking that would be obtained on much larger models, including sizes beyond the 7B models actually tested.","fun_headline_variants_meta":{"raw":{"variants":["Scaling Consistency: small video models, big design clues","Apollo: 3B insights, 7B performance","Video AI: test at 3B, deploy at 7B","Small-scale video LMM experiments guide large-scale design","Why 3B video models lead to 7B wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2715,"prompt_tokens":1162,"completion_tokens":1553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":778,"completion_tokens_details":{"reasoning_tokens":1468}},"tokens_in":778,"tokens_out":1553,"duration_ms":596003,"temperature":1.0,"reasoning_tokens":1468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:55:43.711192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 21 design variations within one LLM family at 0.5B, 1.5B, 3B, 7B and, if resources allow, 30B+, then compare their rankings on ApolloBench; if the 3B-to-30B ranking correlation drops well below the reported 4B-to-7B value of $R^2 = 0.938$, Scaling Consistency is falsified. A first check is whether replacing the Qwen1.5-4B point with a same-family Qwen2-4B preserves the reported fit.","supporting_citations":[],"review_version":1}