{"id":"08b18438-ec17-4549-bea6-13abce5845bd","arxiv_id":"2506.08228","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Motion forecasting models improve with compute as a power law, with optimal model size growing 1.5x faster than dataset size, and closed-loop driving failures also decreasing with scale.","lead":"This technical report from Waymo measures how an autoregressive transformer for predicting multiple vehicles' future motion improves as training compute, model size, and data grow. It finds that driving models follow language-model-style power laws, but optimally need about 50 times fewer parameters than LLMs, and that larger models also drive better in closed-loop simulation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The compute-optimal exponents rest on a two-stage fit over only seven compute bands, and the low-compute parabola minima are weakly constrained by the curvature the paper itself attributes to an irreducible loss term; the quoted ±0.08/±0.06 uncertainties may therefore be optimistic.","rationale":"The reader's weakest assumption names the parametric form L(N,D)=E+A/N^α+B/D^β and the FLOPs model as the load-bearing premise. I partially agree, but I isolate a more specific failure mode within that premise: even if Equation 4 is the correct family, the two-stage iso-FLOP extraction can be unreliable when the loss is near an irreducible constant. The paper's own Figure 5 documents exactly that curvature for L_opt(C), and with only seven compute bands and roughly twelve runs per band, the parabola-minimum step has limited resolving power. The three-sigma bands in Figure 4 are obtained by propagating fit-parameter covariance only, not by resampling the band structure, varying the architecture grid, or comparing against a direct joint fit. An honest conditional verdict is therefore the right call: the trend is plausible and the authors are appropriately cautious, but the headline exponents should be stress-tested before the 'small model, large dataset' recommendation is treated as a scaling law. The reader's other concerns—closed-loop error bars, absent code/data, and the dataset-hour discrepancy—are real but secondary in my reading, because they affect interpretation and reproducibility rather than the numerical core of the scaling-law claim.","tokens_in":21384,"tokens_out":9747,"duration_ms":132050,"concrete_test":"Run a robustness audit of the Figure 4 extraction: refit the N_opt(C) and D_opt(C) power laws after (a) dropping the lowest-compute band, (b) adding a constant term a C^b + c, and (c) leave-one-band-out refits; then independently fit Equation 4 directly to all 84 final losses, i.e., Hoffmann et al.'s approach 3, and derive exponents from the fitted α, β and the Appendix A FLOPs formula. If the N_opt or D_opt exponents shift by more than 0.05–0.08 under any of these checks, the quoted exponents should be presented as provisional rather than as a predictive scaling law.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—N_opt ∝ C^{0.63±0.08} and D_opt ∝ C^{0.44±0.06} (Section 4.1, Figure 4)—is produced by first fitting a parabola in log N and log D inside each of seven iso-FLOP bands, then fitting a power law through those seven minima. Appendix B propagates only the parabola-fit covariance into the displayed error bands; it does not include model-form or grid-selection error. This is consequential because Section 4.2 and Figure 5 show that over the same compute range L_opt(C) is significantly better described by a C^b + L∞ than by a pure power law, i.e., the loss is already approaching an irreducible floor. Near a floor, the loss-versus-log-N curve is flatter in the low-compute bands, so the parabola minimum x_opt is poorly identified there; the second-stage pure power law then depends heavily on the least informative points. The 84 runs also amount to only about 12 configurations per band, and model size is varied by changing layer count at two fixed width-to-depth ratios, so the minima sit on a coarse discrete grid. The derived ratio 'model size should grow 1.5x as fast as dataset size' and the '50x smaller than an LLM' comparison inherit this fragility. This is not a claim that Equation 4 is wrong; it is a request to demonstrate that the two-stage extraction is stable on this narrow, curvature-dominated range before the exponents are treated as predictive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical scaling-law study of an encoder-decoder autoregressive transformer for joint motion forecasting and planning, trained on a proprietary Waymo dataset of roughly 447,000 driving hours. Following the IsoFLOP methodology of Hoffmann et al. (2022), the authors train 84 models across 7 compute budgets, fit parabolic loss curves in log-parameter and log-data space within each iso-FLOP band, and extract compute-optimal exponents N_opt ∝ C^0.63±0.08 and D_opt ∝ C^0.44±0.06 (Section 4.1, Figure 4). They further report that compute-optimal loss is better described by a power law plus a constant than by a pure power law (Section 4.2), that open-loop distance metrics improve with compute (Section 5.1), that closed-loop failure count decreases with pretraining compute (Section 5.2), that inference-time sampling can make smaller models competitive up to a crossover point (Section 6), and that training on other agents' trajectories transfers to the ego-agent (Section 7). The paper is careful in several places to label the open-loop and closed-loop trends as preliminary, noting that more compute orders of magnitude are needed to establish power-law behavior.","tokens_in":21773,"tokens_out":4165,"duration_ms":46061,"significance":"If the central scaling exponents are reliable, this is one of the first large-scale demonstrations that compute-optimal scaling in an autonomous-driving prediction and planning task differs quantitatively from language modeling, with substantially smaller optimal models and a stronger preference for data. The paper also contributes a controlled comparison between open-loop and closed-loop scaling within a single architecture and loss, which is valuable for a field that has mostly seen confounded cross-model comparisons. Credit should be given for the explicit caveats in Sections 5.1 and 9 about the limited compute range, the reporting of fit-error propagation formulas in Appendix B, and the unusually large proprietary dataset. The main risk is that the headline exponents are extracted from a narrow, curvature-dominated compute range using a two-stage fitting procedure whose error bars do not include model-form or grid-selection uncertainty; this concern is central to the paper's primary quantitative claim and needs to be addressed before the exponents can be treated as predictive.","major_comments":[{"comment":"The quoted exponents N_opt ∝ C^0.63±0.08 and D_opt ∝ C^0.44±0.06 rest on a two-stage fit: a parabola in log x within each of seven iso-FLOP bands, followed by a power-law fit through the seven minima. Appendix B propagates only the parabola-fit covariance into the displayed error bands, but it does not account for model-form uncertainty or for the coarse model grid: model size is varied by changing layer count at only two width-to-depth ratios, giving roughly a dozen configurations per band. Because Section 4.2 shows that the loss is already approaching an irreducible floor over this same range, the parabola minima in the low-compute bands are weakly constrained, and the second-stage power law depends heavily on the least informative points. I ask the authors to demonstrate stability of the exponents by, for example, fitting the full loss surface directly to Eq. (4), by excluding or down-weighting the low-compute bands, or by bootstrap resampling over the 84 runs; otherwise the ±0.08 and ±0.06 uncertainties are likely optimistic.","section":"Section 4.1, Figure 4, Appendix B"},{"comment":"The paper states that adding a constant to the power-law fit, L(C) = aC^b + L∞, explains the compute-optimal loss data significantly better than a pure power law. This is not merely a remark about the loss curve; it directly bears on the exponent extraction in Section 4.1, because the same loss values determine the parabola minima used to estimate N_opt(C) and D_opt(C). If the loss is close to an irreducible floor, the low-compute bands may not lie on the same pure power-law surface assumed by the two-stage procedure. I request a sensitivity analysis that fits the parabola with an explicit floor, or that re-extracts the exponents after removing the lowest-compute bands and reports whether N_opt ∝ C^0.63 and D_opt ∝ C^0.44 remain within the stated uncertainties.","section":"Section 4.2, Figure 5"},{"comment":"The closed-loop claim — that the number of failures η decreases as a power law of pretraining compute — is presented with a fitted curve but without reporting the fit parameters, their uncertainties, the number of compute budgets used, the number of scenarios in the closed-loop validation set, or confidence intervals on η. Since η is a low-count failure metric and each model's progress-bias hyperparameter α is tuned individually to match assertiveness, the plot alone does not establish that the trend is statistically distinguishable from noise or that the α-calibration is not absorbing model differences. I ask that the closed-loop analysis report the quantitative fit constants and error bars, and ideally a bootstrap or confidence-interval calculation over scenarios.","section":"Section 5.2, Figure 10"}],"minor_comments":[{"comment":"The abstract says a '500 thousand hours' driving dataset, while Table 1 reports 447 thousand hours; please make the rounding explicit or use a single number consistently.","section":"Abstract and Section 3, Table 1"},{"comment":"In the minADE definition, the notation ||y_t^k − ŷ_t||_2 is used for what should be a displacement error averaged over time; consider clarifying that the norm is taken over spatial coordinates.","section":"Section 5.1, Eq. (5)"},{"comment":"The progress-bias parameter α is introduced without stating its units or range; since α is calibrated per model, a brief description of the calibration procedure and how assertiveness is measured would help reproducibility.","section":"Section 5.2, Eq. (7)"},{"comment":"The '10 observed miles equivalent to 2–3 demonstrated miles' claim would be more informative with confidence intervals from the iso-loss fits, and with an explicit reminder that the data are collected on the same AV platform, as the text already notes.","section":"Section 7, Figure 15"},{"comment":"The appendix correctly notes that first-order error propagation is biased for nonlinear fits, but it does not quantify how large this bias is for the reported 3σ bands; a short numeric comparison would be useful.","section":"Appendix B"},{"comment":"The qualitative trajectory figures would benefit from captions stating which model sizes and which sample counts are shown, and whether the same scenarios are used across panels.","section":"Figures 9 and 13"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern from the reader is valid and lands on a load-bearing point: the compute-optimal exponents are extracted from seven iso-FLOP bands over a compute range where the loss is already curvature-dominated, and the reported uncertainties omit model-form and grid-selection error. This is fixable within the manuscript's scope by adding a direct joint fit to Eq. (4) or a robustness analysis, so I do not recommend rejection. The closed-loop section also needs quantitative reporting. I would not require public release of the proprietary data or code, but the internal stability checks should be reported in the paper. The paper's cautious language in Sections 5.1 and 9 is a strength and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is the first systematic scaling-law study for joint AV motion forecasting, and it deserves attention. The main result—compute-optimal model size grows as C^0.63 and data as C^0.44, so models should be much smaller than LLMs at equal compute—is new and consequential if it holds. The paper does several things well: the isoFLOP setup is standard and carefully described, the loss curves are shown unsmoothed, the authors explicitly caution that open-loop metrics are not established as power laws, and they are transparent about the closed-loop simulator and its calibration. The inference-time sampling crossover and the cross-agent transfer result are useful additions.\n\nThe soft spots are real but not fatal. The exponents come from a two-stage fit over seven compute bands, and the paper's own Figure 5 shows that L(C) is significantly better described by a power law plus a constant. That means the loss is already nearing an irreducible floor, so the parabola minima in the low-compute bands are weakly constrained; the second-stage power law leans on the least informative points. The displayed error bars propagate only the parabola-fit covariance, not model-form or grid-selection uncertainty. I'd say the qualitative conclusion—models for driving should be much smaller, with more data, than for language—is well supported, but the exact exponents and the '50x smaller' factor should be treated as indicative. The closed-loop result relies on an internal simulator with per-model calibrated alpha and no error bars, which is a genuine limitation for the strongest claim. No code or data is released, and the abstract says ~500k hours while Table 1 says 447k hours; that mismatch should be fixed.\n\nWho this is for: anyone working on scaling for robotics, AV planning, or data-efficient training. It's a solid empirical contribution that will be cited. It deserves peer review—the methodology is sensible, the limitations are acknowledged, and the open-loop-to-closed-loop transfer result is important to test further. I'd send it to a serious venue with a request for a robustness analysis of the exponent extraction (bootstrap, direct joint fit, or more compute bands) and error bars on the closed-loop curve. I wouldn't accept the numbers as law yet, but I'd want this paper on record.","headline":"First credible scaling-law study for AV motion forecasting; the exponents are plausible but more fragile than the quoted error bars suggest.","tokens_in":22401,"tokens_out":2862,"would_cite":true,"duration_ms":34226,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Joint motion forecasting and planning follows a language-model-style power-law scaling law, with compute-optimal driving models about 50 times smaller than language models.","keywords":["scaling laws","motion forecasting","autonomous driving","compute-optimal training","closed-loop evaluation","inference-time compute","power-law loss","joint trajectory prediction"],"falsifier":"Retrain the same model family at a compute budget at least ten times larger than the largest budget used here and compare the measured validation loss with the extrapolated $L(C)=aC^b+L_\\infty$ curve; if the data point lands more than three propagated standard deviations above the curve, the power-law extrapolation fails. A complementary check is to vary encoder and decoder depth independently while holding FLOPs fixed: if the loss-minimizing architecture shifts, the Appendix A FLOPs model is mis-ranking models and the $N_{\\mathrm{opt}}$ and $D_{\\mathrm{opt}}$ exponents are artifacts of that formula.","tokens_in":21183,"feed_emoji":"🚗","tokens_out":10524,"duration_ms":116132,"temperature":0.7,"pith_summary":"This report argues that joint motion forecasting and planning for autonomous vehicles obeys the same empirical scaling laws that have been documented for language models. It shows that the cross-entropy loss of an encoder-decoder autoregressive transformer, trained on roughly 447,000 hours of driving data, falls as a power law of training compute, and that the compute-optimal allocation grows model size about 1.5 times as fast as dataset size. The report also claims that these loss improvements transfer to closed-loop driving, with the number of failed simulated scenarios decreasing as a power law of pretraining compute, and that sampling more trajectories at inference time lets smaller models compete with larger ones up to a crossover point. If the claims hold, training compute for driving should be spent on relatively small, data-hungry models, and open-loop loss can serve as a practical proxy for simulation safety.","feed_headline":"Motion forecasting follows language-model-style power-law scaling","feed_subtitle":"Optimal driving models grow 1.5x faster in size than data as compute scales; bigger pretraining cuts failures.","key_machinery":"The argument is carried by a single parametric ansatz: the loss surface is $L(N,D)=E+A/N^\\alpha+B/D^\\beta$, where $N$ counts non-embedding transformer parameters, $D$ counts training examples, $A$ and $B$ are fitted constants, and $E$ is the irreducible data entropy. The machinery around it is an iso-FLOP study: 84 models from 900K to 118M parameters, trained in seven compute bands, with final validation losses fit to parabolas in $\\log N$ and $\\log D$ to locate each band's optimum, then power-law fits over the optima to extract the exponents. The models themselves are MotionLM-style encoder-decoder autoregressive transformers that predict discrete Verlet-wrapped displacement tokens for eight agents jointly, with a single flattened agent-time self-attention pass; the FLOPs count in Appendix A determines how parameter and data sizes map to compute. The loss ansatz does the work: it converts a vague belief that 'bigger is better' into two concrete exponents with error bars, which are the paper's main quantitative claims.","core_discovery":"On its own terms, the report's central discovery is a scaling law: for an encoder-decoder autoregressive transformer trained to jointly predict the future trajectories of eight road agents, validation cross-entropy follows $L(N,D) = E + A/N^\\alpha + B/D^\\beta$, with compute-optimal exponents $N_{\\mathrm{opt}} \\propto C^{0.63\\pm0.08}$ and $D_{\\mathrm{opt}} \\propto C^{0.44\\pm0.06}$. This means that at a given training-compute budget, the best driving model is roughly 50 times smaller than a compute-optimal language model at the same budget, while requiring far more data. The same compute-optimal series shows consistent power-law improvement in open-loop distance metrics (minADE and wADE) and, after short fine-tuning to route-conditioned single-agent planning, a power-law decrease in closed-loop simulation failures. Finally, the report shows that increasing inference-time sampling improves small-model metrics until a crossover, beyond which a larger model is more inference-compute efficient, and that models trained without the ego vehicle zero-shot predict ego-agent trajectories at a rate of about ten observed miles per two to three demonstrated miles.","pith_inferences":["A testable extension: if the fitted entropy constant $E$ reflects the limited perception feature set (the paper names this as a hypothesis), then adding richer inputs such as camera or LiDAR tokens should lower the floor of the loss curve and could change the exponents; rerunning the smallest iso-FLOP band with vision inputs would test this directly.","The zero-shot transfer from other agents' trajectories suggests a data-collection strategy the paper only gestures at: passive multi-platform driving logs could substitute for ego-demonstrated miles, and the observed 10 observed miles per 2 to 3 demonstrated miles ratio gives a rough cost-equivalence to test on larger, truly passive corpora.","The inference-compute crossover implies an adaptive serving policy: pick the smallest model whose sampling curve has not yet saturated and increase samples per scenario up to that model's crossover; this is a direct, testable extension of the paper's inference-scaling analysis.","If the 1.5x model-to-data growth ratio carries over to other continuous-control planning tasks, robotics teams should bias fixed compute budgets toward data collection over model capacity, but that generalization is not established by this report."],"forward_implications":["At a fixed training-compute budget, the optimal driving model is about 50 times smaller than the optimal language model, so compute invested in driving models should buy data, not a much larger network.","As the compute budget grows, model parameters should increase roughly 1.5 times faster than training examples, which means the model-size axis grows somewhat faster than the data axis.","Open-loop loss, minADE, and wADE improve together with compute, so loss-based hill climbing is a reliable stand-in for forecast quality.","Closed-loop simulation failures fall as a power law of pretraining compute, meaning open-loop scaling studies can be used to predict simulator safety rather than only offline accuracy.","Increasing inference-time samples makes smaller models competitive with larger ones up to a crossover, after which the larger model is the better use of inference FLOPs."],"supporting_citations":[{"why":"Supplies the iso-FLOP methodology and the Chinchilla exponents that this paper reproduces and contrasts with.","marker":"Hoffmann et al. (2022)"},{"why":"Establishes the power-law loss scaling framework and the N_opt, D_opt allocation problem.","marker":"Kaplan et al. (2020)"},{"why":"Provides the power-law-plus-constant form and the interpretation of the added term as irreducible data entropy.","marker":"Henighan et al. (2020)"},{"why":"Gives the MotionLM architecture and discrete joint motion-token formulation that all models in the study inherit.","marker":"Seff et al. (2023)"},{"why":"Parameterizes loss in encoder and decoder parameters, which the paper simplifies into Eq. (4).","marker":"Ghorbani et al. (2021)"},{"why":"Defines the open-loop metrics (minADE, minFDE, miss rate, mAP) and the perception feature configuration used for evaluation.","marker":"Ettinger et al. (2021)"},{"why":"Supplies the early-fusion scene encoder backbone used to encode roadgraph, traffic lights, and agent histories.","marker":"Nayakanti et al. (2022)"},{"why":"Provides the NMS/K-means rollout aggregation algorithm used to convert sampled trajectories into the evaluated mode set.","marker":"Varadarajan et al. (2021)"}],"fun_headline_variants":["Motion forecasting obeys power-law scaling with compute","Compute-optimal driving models grow 1.5x faster in size","Inference-time sampling lets small models rival big ones","Closed-loop driving improves with more compute","Ego planning learned from general driving data zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true validation loss is exactly $L(N,D)=E+A/N^\\alpha+B/D^\\beta$ with the paper's FLOPs formula correctly ranking architectures, so the exponents fitted on 84 runs across a two-order-of-magnitude compute range remain valid when extrapolated beyond it.","fun_headline_variants_meta":{"raw":{"variants":["Motion forecasting obeys power-law scaling with compute","Compute-optimal driving models grow 1.5x faster in size","Inference-time sampling lets small models rival big ones","Closed-loop driving improves with more compute","Ego planning learned from general driving data zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1724,"prompt_tokens":1046,"completion_tokens":678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":602}},"tokens_in":662,"tokens_out":678,"duration_ms":7254,"temperature":1.0,"reasoning_tokens":602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:17:12.360128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same model family at a compute budget at least ten times larger than the largest budget used here and compare the measured validation loss with the extrapolated $L(C)=aC^b+L_\\infty$ curve; if the data point lands more than three propagated standard deviations above the curve, the power-law extrapolation fails. A complementary check is to vary encoder and decoder depth independently while holding FLOPs fixed: if the loss-minimizing architecture shifts, the Appendix A FLOPs model is mis-ranking models and the $N_{\\mathrm{opt}}$ and $D_{\\mathrm{opt}}$ exponents are artifacts of that formula.","supporting_citations":[],"review_version":1}