{"id":"fbd891fe-cd6e-49d4-8b31-565e12d3e899","arxiv_id":"2507.01823","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Reward-level distillation and FP16 quantization compress a 317M-parameter TD-MPC2 agent to 1M parameters, reaching 28.45 normalized score on MT30, though most of the gap over the original 18.93 comes from a longer training budget.","lead":"This paper distills a large 317M-parameter TD-MPC2 reinforcement learning agent into a 1M-parameter student on the MT30 benchmark, using a reward-prediction matching loss and post-training quantization. A generalist could read it to see whether large multi-task robot-learning models can be compressed enough for real-world devices with limited compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline '+50.2% over 18.93' compares a 1M-step, batch-256 student with a 200K-step, batch-1024 checkpoint; against the paper's own same-budget from-scratch baseline (27.36), distillation gains only 0.76-1.09 points, and no error bars or seeds are reported to rule out run-to-run noise.","rationale":"The reader's weakest-assumption analysis identifies the same empirical weak point, and I agree. The paper's method is coherent and a code release is promised, though no executable artifact is shipped in the manuscript, so reproducibility currently rests on the text. The independent support is limited: no formal verification, no multi-seed statistics, and no external benchmark baselines beyond the TD-MPC2 reproduction. The strongest evidence in the paper's favor is the internally consistent ordering at 200K/256 (17.85 vs 14.04), which is a larger relative gain and in the expected direction; this suggests the method may provide sample-efficiency benefits in low-update regimes. However, the headline claim is about the final SOTA score after 1M steps, and there the gain over the same-budget control is small and unquantified. The 'best FP16 quantized' number beating its unquantized source is a red flag that the evaluation protocol may be noisy, since deterministic post-training quantization of the same weights should not systematically improve the normalized score by 0.33 unless run-to-run variance is non-negligible. The most defensible response is to require repeated-seed evaluation before accepting the SOTA claim; hence the verdict remains CONDITIONAL, which is unchanged from the reader. This is not a rejection of the core idea: reward distillation may well be useful, and the 200K-batch-256 result is promising, but the central claim as written is not established by the current evidence.","tokens_in":8381,"tokens_out":6910,"duration_ms":74237,"concrete_test":"Run at least 5 independent seeds for each of three conditions, holding everything else fixed to the reported setup: (A) from-scratch 1M/256 trained for 1M steps; (B) distilled 1M/256 trained for 1M steps with the reported d_coef; (C) FP16 quantization of each distilled seed. Report per-seed MT30 normalized scores and per-task paired differences between A and B/C. If the mean A-B difference is less than the pooled inter-seed standard deviation, or if a paired non-parametric test does not reach p<0.05, the distillation and SOTA claims are not supported. The fair comparison to the 200K/1024 18.93 baseline should also be replaced by the same-budget baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the 28.45 result is both fairly benchmarked and reproducible. Table 2 provides the same-budget control: a from-scratch 1M-parameter model trained for 1M steps with batch size 256 scores 27.36, and the unquantized distilled run scores 28.12. The gap attributable to distillation is therefore 0.76 normalized points, not the 9.52 implied by comparing 28.45 to the previously reported 18.93, which was trained for only 200K steps at batch size 1024. The best FP16-quantized score (28.45) is itself a single run that exceeds its unquantized source (28.12), a direction inconsistent with typical quantization effects and consistent with evaluation noise. Every configuration in Tables 2-4 is a single run with no standard deviation, no seeds, and no per-task error analysis. On MT30, run-to-run variation at this scale is routinely several points; without repeated seeds the 'new state-of-the-art' claim is not distinguishable from a favorable draw. The paper's own Section 4.2 also reports a full-data regime where distillation underperforms from-scratch (25.44 vs 26.94 at 1024/337K), so the claimed advantage is not uniform across regimes. Thus the load-bearing assumption, that the measured 0.76-1.09 advantage is real and reproducible, is unsupported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TD-MPC-Opt, a method for distilling a 317M-parameter TD-MPC2 teacher into a 1M-parameter student on the MT30 benchmark, using an additional reward-prediction MSE loss (L_distill) and post-training FP16 quantization. The central claim is that the resulting 1M-parameter student achieves a state-of-the-art normalized score of 28.45, surpassing the previously reported 1M-parameter model score of 18.93 by 50.2%, and that distillation improves over from-scratch training in some regimes. Experiments compare distillation against from-scratch baselines at several batch sizes and training lengths, and explore quantization (FP16, mixed precision, INT8). The paper also reports limitations including no physical-system validation and evaluation only on MT30.","tokens_in":8686,"tokens_out":2867,"duration_ms":32054,"significance":"If the central claim were robustly supported, the paper would make a useful practical contribution: reward-level distillation from a large world model to a 1M-parameter student, combined with FP16 quantization, could enable deployment of multi-task model-based RL agents in resource-constrained settings. The paper's strengths include a simple, clearly described distillation objective, evaluation on a standard benchmark (MT30), and a code release. However, the current experimental evidence does not establish the headline improvements: the main comparison is confounded by different training budgets, all key results are single runs without error bars, and one regime (1024/337K) shows distillation underperforming from-scratch training. The significance of the claimed state-of-the-art result therefore remains unverified.","major_comments":[{"comment":"This is the load-bearing comparison for the paper's central claim, so it must be presented fairly.","section":"Abstract and Section 5, Discussion"},{"comment":"This issue directly affects reproducibility of the central claim.","section":"Tables 2 and 4; Section 4.2"},{"comment":"This is a substantive inconsistency in the results that needs to be addressed.","section":"Section 4.2, Table 2"},{"comment":"This is a methodological caveat that affects the strength of the state-of-the-art claim.","section":"Section 3.1, Table 1, and Section 5"}],"minor_comments":[{"comment":"The abstract reports a '+50.2%' improvement, while the contribution bullet in Section 1 reports '+48.5%' for the same comparison; these numbers should be reconciled.","section":"Abstract and Section 1 contribution list"},{"comment":"The dataset size is given as 345,690,000 transitions in Section 3.4 but as '349 million transitions' in Section 4.2; these should be made consistent.","section":"Section 3.4 and Section 4.2"},{"comment":"Table 1 is titled 'Impact of d_coef' but its first two rows are not d_coef values; they are ablation variants (reward + next-state with linear projection or PCA). The table should be restructured or retitled to avoid confusion.","section":"Table 1"},{"comment":"Figure 3 shows training trajectories at five checkpoints but provides no error bars or multiple-run information; since the underlying scores are single runs, the visual 'consistently superior performance' claim is unsupported. This should be clarified in the caption or text.","section":"Figure 3"},{"comment":"The text says 'reward + next-state (linear projection) 7.69' and 'reward + next-state (PCA) 8.78' in Table 1, but Section 4.6 describes these as attempts at latent next-state distillation; the terminology ('next-state latent' vs 'next-state') should be harmonized.","section":"Section 4.1"},{"comment":"The symbol 'd coef' is sometimes written with a space and sometimes as 'd_coef'; use a single notation consistently.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the distilled model may have practical value, but the experimental evidence as presented is not sufficient to support the headline claims. The most important fix is to add repeated seeds and error bars for all key configurations, and to present the same-budget comparisons (e.g., 256/1M from scratch vs distilled) as the primary evidence. If the 0.76-1.09 point advantage persists across seeds and the full-data underperformance is explained, the paper could become acceptable. The self-citation [18] is not a circularity problem, but it is a preprint with no identifier; the authors should provide a citable version if available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim does not survive contact with the paper's own tables. The '+50.2%' compares a 1M-step, batch-256, FP16-distilled student (28.45) with a 200K-step, batch-1024 from-scratch checkpoint (18.93). Against the paper's same-budget from-scratch baseline (27.36), the distilled student gains 0.76–1.09 points, and no seeds or error bars are reported. That gap is within the typical run-to-run variance on MT30. The FP16 score being higher than the unquantized source (28.45 vs 28.12) is another sign of noise. In the full-data regime (337K, batch 1024) distillation actually underperforms from-scratch (25.44 vs 26.94). So the central claim of 'significant improvement' is not supported.\n\nWhat the paper does well: it runs a clean sweep of batch sizes, distillation lengths, d_coef, teacher sizes, and quantization on a meaningful benchmark, and it reports the ugly regimes (337K underperformance, next-state distillation failures). The reward-only distillation recipe is simple and easy to reproduce; code is promised. That is a useful empirical map, even if the headline is oversold.\n\nThe method itself is not circular: a frozen teacher's reward predictions are a regression target, and d_coef is a hyperparameter. Self-citation to [18] is fine.\n\nSoft spots: single runs everywhere, no confidence intervals. The teacher-size comparison (48M vs 317M) is a single pair of runs and may be confounded by teacher quality rather than size. The d_coef sweep is also single-run. The small same-budget gain needs replication. Also, the abstract and discussion should use the honest comparison, not the misleading one.\n\nVerdict: This paper deserves a serious referee, not a desk reject. The idea is straightforward but the empirical claims need to be redone with seeds and error bars, and the comparison baseline fixed. For a reading group, it's a useful case study in how benchmarking choices can flip a conclusion. I would not cite it in its current form, but I would engage with a revised version.","headline":"The +50.2% headline is an apples-to-oranges comparison; the same-budget distillation gain is small and unquantified, but the empirical sweep is worth a careful revision.","tokens_in":9252,"tokens_out":2878,"would_cite":false,"duration_ms":28153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distilled 1M-parameter RL agent scores 28.45 on MT30, up from 18.93","keywords":["model-based reinforcement learning","knowledge distillation","multi-task learning","TD-MPC2","MT30 benchmark","model compression","FP16 quantization","continuous control"],"falsifier":"Train the distilled 1M/256/1M student and the from-scratch 1M/256 baseline across at least five seeds each and compare the distributions of MT30 normalized scores; if the confidence intervals overlap substantially or the distilled mean does not exceed the from-scratch mean, the claimed distillation benefit and the 28.45 state-of-the-art are not established.","tokens_in":8141,"feed_emoji":"🤖","tokens_out":10304,"duration_ms":96182,"temperature":0.7,"pith_summary":"This paper argues that a large model-based reinforcement learning agent can be compressed into a 1M-parameter student that outperforms the original compact model on the MT30 multi-task benchmark. The method adds a reward-distillation term to the standard TD-MPC2 loss, training the student to match the teacher's reward predictions on the same state-action inputs. With a batch size of 256 and 1M training steps, the distilled student reaches a normalized score of 28.45 after FP16 quantization, compared with 18.93 for the original 1M checkpoint. The paper also finds that smaller batch sizes help distillation, that reward-only distillation beats attempts to also match next-state latents, and that FP16 quantization roughly halves model size with no performance loss. If these results hold, they point to a practical recipe for turning large world models into deployable multi-task controllers.","feed_headline":"Distilled 1M-parameter RL agent scores 28.45 on MT30, up from 18.93","feed_subtitle":"Reward-prediction distillation from a 317M teacher beats the original compact model and halves its size.","key_machinery":"The load-bearing mechanism is the reward distillation loss $L_{\\mathrm{distill}} = \\mathrm{MSE}(R_{\\mathrm{teacher}}(s,a), R_{\\mathrm{student}}(s,a))$, added to the original TD-MPC2 loss as $L_{\\mathrm{total}} = L_{\\mathrm{orig}} + d_{\\mathrm{coef}} \\cdot L_{\\mathrm{distill}}$. The teacher is a frozen 317M-parameter TD-MPC2 checkpoint from [12]; the student is the 1M-parameter TD-MPC2 backbone. The distillation coefficient $d_{\\mathrm{coef}}$ (optimal around 0.4) controls how much the teacher's reward knowledge shapes the student, and the paper combines this with a batch size of 256 and extended 1M-step training. FP16 post-training quantization halves the model size to 3.9 MiB while slightly improving the score, making the mechanism a full pipeline from large teacher to deployable compact agent.","core_discovery":"The central discovery is that matching only the reward predictions of a frozen 317M-parameter TD-MPC2 teacher is enough to transfer a large part of its multi-task competence to a 1M-parameter TD-MPC2 student. The student is trained with the original TD-MPC2 objective (consistency, reward, and value losses) plus an MSE penalty between teacher and student reward outputs, scaled by a coefficient d_coef, with values near 0.4 performing best. After 1M steps with batch size 256, the distilled student scores 28.12; applying FP16 post-training quantization pushes the normalized MT30 score to 28.45, which the paper reports as the new state of the art for this benchmark and a +50.2% improvement over the previously reported 18.93 for the 1M model trained from scratch. The paper also reports that a 200K-step distilled run with batch size 256 reaches 17.85 versus 14.04 from scratch, that a 1024-batch run does not benefit from distillation, and that adding next-state latent matching via linear projection or PCA degrades scores to 7.69 or 8.78, respectively.","pith_inferences":["Editorial inference: The reported 0.76-point gap between the distilled 1M/256 model (28.12) and the from-scratch 1M/256 baseline (27.36) is based on single runs; with typical MT30 run-to-run variance, this difference may not be robust, so the most convincing evidence for distillation is the 200K-step regime, where the distilled model beats from scratch by 3.81 points.","Editorial inference: The surprising FP16 gain (28.12 to 28.45) could be a seed effect rather than a real benefit of quantization; a multi-seed comparison is needed before treating quantized models as superior to their float32 counterparts.","Editorial inference: A testable extension would be to distill from the same teacher into a student with a larger latent dimension (closer to the teacher's 1376) and add a latent-matching term; if the dimension-mismatch explanation is right, such a student should beat reward-only distillation on tasks that require next-state prediction, such as acrobot-swingup and cheetah-jump.","Editorial inference: The finding that batch size 256 beats 1024 may depend on the teacher's own training batch size; matching the student's batch to the teacher's could be an alternative explanation for the effect, and it can be tested directly."],"forward_implications":["A 1M-parameter world model can match or exceed the MT30 performance of a 317M-parameter teacher, so multi-task continuous control competence is highly compressible when distillation uses reward predictions.","Reward-only distillation is sufficient for strong transfer on MT30; attempts to also match next-state latent representations currently hurt performance because of the teacher-student dimension gap.","FP16 post-training quantization can cut model size by about half without degrading (and in this study slightly improving) normalized score, making FP16 a safe default compression step for these agents.","Training with batch size 256 outperforms batch size 1024 for distillation, suggesting that frequent weight updates matter more than large batches in this low-resource regime.","Longer distillation (1M steps) lets the student overtake a from-scratch baseline trained under identical conditions, whereas short distillation (200K steps) only helps at small batch sizes."],"supporting_citations":[{"why":"Supplies the 317M-parameter teacher checkpoint, the MT30 benchmark and dataset, the 1M-parameter student architecture, and the 18.93 baseline that the distilled model is claimed to surpass.","marker":"[12]"},{"why":"Defines the TD-MPC algorithm (temporal difference learning combined with model predictive control) that both teacher and student implement.","marker":"[13]"},{"why":"Introduces the teacher-student knowledge distillation objective that the reward-prediction MSE adapts to model-based RL.","marker":"[14]"},{"why":"Shows distillation can transfer knowledge between RL policies, establishing the precedent for using distillation in this setting.","marker":"[27]"},{"why":"Defines the DM Control suite from which MT30 tasks are drawn, grounding the benchmark's task distribution.","marker":"[32]"},{"why":"Supplies the FP16/mixed-precision technique used for post-training quantization of the distilled model.","marker":"[22]"},{"why":"Provides the quantization methods (post-training, INT8) used to evaluate aggressive model compression.","marker":"[8]"}],"fun_headline_variants":["Distilled 1M RL agent beats 18.93 with 28.45 on MT30","317M teacher distilled to 1M, scores 28.45 on MT30","Reward-matching distillation lifts 1M agent to 28.45","1M RL model hits 28.45 via teacher reward distillation","SOTA 28.45: 1M model from 317M teacher distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single-run scores for the distilled and from-scratch 1M-parameter models are treated as reliable point estimates; if run-to-run variance on MT30 is as large as a few points, the claimed edge of distillation and the 28.45 state-of-the-art could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Distilled 1M RL agent beats 18.93 with 28.45 on MT30","317M teacher distilled to 1M, scores 28.45 on MT30","Reward-matching distillation lifts 1M agent to 28.45","1M RL model hits 28.45 via teacher reward distillation","SOTA 28.45: 1M model from 317M teacher distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000297,"raw_usage":{"total_tokens":1737,"prompt_tokens":976,"completion_tokens":761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":663}},"tokens_in":592,"tokens_out":761,"duration_ms":6531,"temperature":1.0,"reasoning_tokens":663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:42:40.266085+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the distilled 1M/256/1M student and the from-scratch 1M/256 baseline across at least five seeds each and compare the distributions of MT30 normalized scores; if the confidence intervals overlap substantially or the distilled mean does not exceed the from-scratch mean, the claimed distillation benefit and the 28.45 state-of-the-art are not established.","supporting_citations":[{"cited_title":"Temporal difference learning for model predictive control","cited_arxiv_id":null,"evidence_quote":"Defines the TD-MPC algorithm (temporal difference learning combined with model predictive control) that both teacher and student implement."},{"cited_title":"Distilling the knowledge in a neural network","cited_arxiv_id":null,"evidence_quote":"Introduces the teacher-student knowledge distillation objective that the reward-prediction MSE adapts to model-based RL."}],"review_version":1}