{"id":"07ff8736-9188-457d-945f-04b0fe2f07e0","arxiv_id":"2412.00395","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Pretraining a transformer purely on synthetic dynamics sampled from a reproducing kernel Hilbert space yields zero-shot and fine-tuned state predictors that beat non-pretrained baselines on cart-pole and Furuta systems.","lead":"A transformer model trained only on synthetic data generated from smooth random functions can predict the future states of physical systems it has never seen, such as cart-pole and Furuta pendulums. The paper suggests that synthetic pretraining could give control systems the same data efficiency and robustness benefits that foundation models bring to language and vision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The data-efficiency and robustness claims confound pretraining with model capacity: the only transformer baseline has ~17x fewer parameters, and no matched-capacity from-scratch transformer is evaluated.","rationale":"The reader's weakest_assumption emphasizes the representativeness of the RKHS synthetic prior for action-driven real dynamics, but the reader's rationale also explicitly notes that 'the pretraining benefit is confounded with model capacity because the only non-pretrained transformer baseline is much smaller.' I agree this capacity confound is a serious internal-validity problem and arguably more fundamental than prior coverage: even if the RKHS prior were perfectly representative, the experiments as designed cannot prove that pretraining—rather than scale—produces the reported P2/P3 benefits. I therefore partially agree with the reader. The concern does not require changing the CONDITIONAL verdict; it sharpens the condition: the central feasibility claim should be accepted only after a matched-capacity from-scratch transformer comparison is run. The proposed test would settle this by directly controlling for architecture size, layer count, and training-data amount, leaving pretraining as the only difference.","tokens_in":12074,"tokens_out":4802,"duration_ms":51335,"concrete_test":"Train a 20-layer, ~3.4M-parameter decoder-only transformer from scratch (no RKHS pretraining) on the same cart-pole and Furuta datasets at 2%, 10%, and 100% data levels, with 20 repeats, and compare its MSE and variance to the existing Pre and Ft models. If the from-scratch large transformer matches or outperforms Ft, the reported data-efficiency and robustness advantages are not attributable to synthetic pretraining; if Ft remains superior, the pretraining contribution is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 states the proposed transformer has 20 layers and approximately 3.4 million parameters, while the sole transformer baseline, ST, has 8 layers and approximately 200k parameters (Section 6.1). ST is described as 'not pretrained and thus shows the impact of pretraining and the foundation model paradigm,' but this comparison cannot isolate pretraining: any advantage of Ft over ST could come from model capacity, optimization dynamics, or the larger pretraining dataset rather than from the synthetic RKHS pretraining itself. This is load-bearing for the strongest claim because the paper's central contribution is that pretraining on purely synthetic data yields foundation-model benefits (P1)–(P3). The claim 'fine-tuning a pretrained model with little data beats non-pretrained baselines' is only meaningful if the non-pretrained baseline has the same capacity and is trained from scratch on the same task data. The paper does not provide such a baseline. Section 6.3 even reports that on the full hardware cart-pole dataset, the best ST run slightly outperforms Ft, which is consistent with capacity, not pretraining, driving part of the advantage. A secondary issue is that pretraining uses zero action (u≡0) and zero placeholder actions (Eq. 4), so zero-shot generalization to action-driven hardware is not competitive; however, the capacity confound is more fundamental because it threatens the attribution of the fine-tuning gains regardless of the synthetic prior's coverage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes training a transformer-based foundation model for state prediction in dynamical systems, using purely synthetic data generated by sampling dynamics functions from an RKHS, discretizing with Euler's method, and filtering by total variation. The resulting model is evaluated zero-shot and after fine-tuning on simulated cart-pole systems and on hardware cart-pole and Furuta pendulum data. The authors claim that this approach yields the typical foundation-model benefits: generalization across unseen systems, data efficiency in fine-tuning, and robustness across runs. The paper includes comparisons against linear regression, a feedforward network, and a smaller non-pretrained transformer.","tokens_in":12493,"tokens_out":2176,"duration_ms":24636,"significance":"If substantiated, the result would be a meaningful step toward foundation models for low-level control, addressing the lack of Internet-scale dynamical-systems data by using a synthetic pretraining distribution. The paper is clearly written and includes hardware validation, repeated runs, and a concrete data-generation pipeline. However, the central attribution of the reported gains to pretraining is weakened by the fact that the only transformer baseline has about 17 times fewer parameters than the proposed model, so capacity and pretraining are confounded. The paper also explicitly concedes that zero-shot performance is not competitive on action-driven hardware systems, which narrows the scope of the generalization claim. The code is not yet released, limiting reproducibility.","major_comments":[{"comment":"The comparison between the fine-tuned model (Ft) and the smaller transformer (ST) cannot be used to isolate the effect of pretraining. Ft has 20 layers and approximately 3.4 million parameters, while ST has 8 layers and approximately 200k parameters; the paper states that ST is \"not pretrained and thus shows the impact of pretraining,\" but any performance difference could instead be due to model capacity, optimization, or the larger pretraining corpus. This confound is load-bearing for the paper's central claims (P1)–(P3) because those claims are about the benefits of synthetic-data pretraining. The observation in §6.3 that the best ST run slightly outperforms Ft on the full hardware cart-pole dataset is consistent with capacity rather than pretraining driving part of the advantage. Please add a matched-capacity from-scratch transformer baseline (same architecture size, trained only on the task data), or otherwise control for capacity, before attributing the fine-tuning gains to pretraining.","section":"§6.1, §6.3"},{"comment":"The pretraining distribution is restricted to systems without control inputs (u ≡ 0 and zero placeholder actions in Eq. (4)), yet the problem formulation and the downstream evaluation include action-driven dynamics. The paper concedes in §6.3 that zero-shot performance on the hardware systems is \"not competitive\" because pretraining considered zero action. This is a substantive scope limitation: the claimed generalization across dynamical systems is demonstrated only for autonomous systems, while action-driven systems require fine-tuning. The paper should state this limitation prominently and temper the claim that the model generalizes across dynamical systems generally, or add experiments that include actions in the pretraining distribution.","section":"§4, §6.3"}],"minor_comments":[{"comment":"There is a typo in the abstract: \"large-scale pretaining\" should be \"large-scale pretraining.\"","section":"Abstract"},{"comment":"In the sentence \"models the capture the inherent 'dynamics' of the problem,\" \"models the capture\" should be \"models that capture.\"","section":"§1"},{"comment":"The paper defines robustness in §3 as consistency of MSE across runs, but the introduction lists robustness under distribution shifts, adversarial inputs, and noisy data. The evaluation only measures variance across initializations and data subsets; please clarify that the robustness claim is about run-to-run consistency, not the broader robustness properties listed in (P3).","section":"§3, §6.1"},{"comment":"The indexing in the model description is slightly confusing: after giving the input sequence in Eq. (4), the text says the model generates {x̂_2,...,x̂_{c+m+1}} but only {x̂_{c+1},...,x̂_{c+m}} are used as valid predictions. Rewording or a small figure would help the reader verify the alignment of context and prediction windows.","section":"§5"},{"comment":"The paper states \"The code will be released upon acceptance.\" Since the reproducibility of the experiments is part of the contribution, please consider releasing the code and data at submission time or providing a detailed appendix with hyperparameters and training details.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The capacity confound is the main technical issue: the paper's strongest claim—that pretraining on synthetic RKHS data yields foundation-model benefits—is not cleanly supported without a matched-capacity from-scratch baseline. The paper is otherwise well-executed and the hardware experiments are valuable. I would like to see either a matched-capacity baseline or a clear reframing of the claims as \"a larger pretrained transformer outperforms smaller baselines,\" which would substantially weaken the contribution. The action-free pretraining limitation should also be acknowledged more prominently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is that a TimesFM-style transformer pretrained purely on synthetic dynamics sampled from an RKHS transfers to unseen cart-pole and Furuta systems, and fine-tunes efficiently with little data. That feasibility claim is plausible and worth testing further. The paper is honest: it names parallel works, discloses that zero-shot hardware performance is not competitive because pretraining used zero actions, and clearly describes the synthetic data pipeline, including the total variation filtering and binning.\n\nThe main soft spot is the pretraining-vs-capacity confound. The only transformer baseline (ST) has 8 layers and roughly 200k parameters, while the proposed model has 20 layers and roughly 3.4M. The paper says ST \"shows the impact of pretraining,\" but that comparison cannot isolate pretraining from capacity or optimization. The fine-tuning gains over ST could come from the larger model alone. The paper even reports that on the full hardware cart-pole dataset, the best ST run slightly outperforms Ft, which is consistent with capacity rather than pretraining driving the advantage. This threatens the attribution of the data-efficiency and robustness claims, which are central to the paper.\n\nThe synthetic prior coverage is a second, somewhat softer issue. Pretraining on zero-action dynamics explains why zero-shot on action-driven hardware is poor; the paper acknowledges this, but it also means the generalization claim is mostly demonstrated for autonomous systems. The total variation filtering is a disclosed selection bias but its effect on the learned representation is not analyzed. Code and data are not yet available, so the figures cannot be independently checked. These are addressable, not fatal.\n\nWho is this for? Researchers working on foundation models for control or on synthetic pretraining for dynamical systems. They will get a clear, well-written starting point with a concrete architecture and data-generation recipe, but they should not take the P1–P3 claims at face value until the capacity confound is resolved. This deserves a serious referee; I would recommend acceptance only after a matched-capacity from-scratch transformer is added and the pretraining attribution is re-examined.","headline":"A plausible feasibility study of pretraining a transformer on purely synthetic RKHS dynamics, but the central pretraining-vs-capacity comparison is confounded and needs a matched-capacity baseline before the foundation-model claims hold.","tokens_in":12940,"tokens_out":1394,"would_cite":true,"duration_ms":15434,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer trained only on synthetic smooth dynamics transfers to real cart-pole and Furuta states, and fine-tunes from little data.","keywords":["foundation model","dynamical systems","synthetic data pretraining","reproducing kernel Hilbert space","state prediction","transformer","fine-tuning","zero-shot generalization"],"falsifier":"Train the same pipeline and evaluate it zero-shot on a dynamical system with genuinely non-smooth dynamics, such as a bouncing ball or an oscillator with Coulomb friction, comparing prediction error to the reported cart-pole results; if errors are orders of magnitude larger and small-data fine-tuning cannot recover, the RKHS synthetic prior is not representative of the target class.","tokens_in":11817,"feed_emoji":"⚙️","tokens_out":8458,"duration_ms":76169,"temperature":0.7,"pith_summary":"The paper asks whether a foundation model for control can be built from purely synthetic data, since no Internet-scale dataset of physical dynamics exists. It claims yes: a decoder-only transformer pretrained on trajectories generated from random smooth functions sampled from a reproducing kernel Hilbert space learns a general predictor that transfers to unseen dynamical systems. The authors validate the claim on simulated and physical cart-pole setups and a physical Furuta pendulum, and show that fine-tuning with a small slice of target data beats specialist models trained from scratch. If the claim holds, pretraining for control tasks does not require collecting a giant real-world dataset; a well-chosen synthetic prior plus a transformer architecture provides the generalization, data efficiency, and run-to-run stability that foundation models promise elsewhere.","feed_headline":"Synthetic-only training predicts real pendulum systems","feed_subtitle":"Trained on synthetic smooth dynamics, it transfers to real cart-pole and Furuta states and fine-tunes from little data.","key_machinery":"The load-bearing mechanism is the synthetic pretraining prior: dynamics functions are drawn from a reproducing kernel Hilbert space with an RBF kernel, with random supporting points and coefficients, norm-scaled to a uniformly chosen target, discretized by Euler's method, and filtered by total variation so the input and output halves of each trajectory have comparable complexity and no single behavior dominates. The second mechanism is the decoder-only transformer, which embeds state-action pairs through residual blocks, applies causal multi-head self-attention, and predicts future states under random masking and output patching. It is trained to map a context window plus future-action placeholders to future states, so the same network can be evaluated zero-shot or fine-tuned on a target system.","core_discovery":"The core discovery is that the missing dataset problem for dynamical systems can be circumvented by sampling dynamics functions from an RKHS with an RBF kernel, scaling each function's norm, converting it to a continuous-time vector field, and discretizing with Euler's method. The resulting trajectories are filtered and binned by total variation to keep the pretraining distribution balanced. After pretraining, a 3.4-million-parameter transformer predicts future states of unseen systems in simulation and on hardware, and fine-tuning it with small data subsets improves accuracy beyond that of a linear regression baseline, a feedforward network, and a smaller transformer trained from scratch. The paper frames this as demonstrating generalization, data efficiency, and lower-variance performance across repeated runs.","pith_inferences":["If this transfer holds, the same recipe could initialize state estimators and model-predictive controllers across many hardware systems, replacing per-system identification with a short fine-tuning step.","The zero-action pretraining prior is likely the main reason zero-shot performance drops on action-driven hardware; a direct testable extension is to include nonzero control inputs in the synthetic data generation.","Comparing the same pipeline with other synthetic function families, such as trigonometric or polynomial vector fields, would reveal whether smoothness alone or the specific RBF-kernel structure carries the transfer."],"forward_implications":["A pretrained dynamics model can predict future states of a cart-pole system it has never seen, matching or beating linear regression and a feedforward net in low-data regimes in simulation.","Fine-tuning the pretrained model on as little as 2% of a target system's trajectories yields lower prediction error than training a specialist model from scratch on the same data.","The fine-tuned model shows less run-to-run variance across 20 training repetitions than the baselines, including on hardware cart-pole and Furuta pendulum data.","Zero-shot performance degrades when control actions are present, because pretraining used zero action, but fine-tuning closes most of that gap on hardware."],"supporting_citations":[{"why":"Supplies the decoder-only transformer architecture that the paper adapts for state-action trajectory prediction.","marker":"Das et al., 2024"},{"why":"Supplies the RKHS function-sampling and norm-scaling practice used to generate synthetic dynamics functions.","marker":"Fiedler et al., 2021"},{"why":"Provides the RKHS background and theoretical justification for representing sampled functions as kernel expansions.","marker":"Steinwart and Christmann, 2008"},{"why":"Defines the foundation-model properties of generalization, data efficiency, and robustness that the paper aims to demonstrate.","marker":"Bommasani et al., 2022"},{"why":"Motivates few-shot fine-tuning and data efficiency as transferable foundation-model benefits.","marker":"Brown et al., 2020"},{"why":"Provides the custom cart-pole hardware system and its parameters used in the physical experiments.","marker":"Hose et al., 2024"}],"fun_headline_variants":["Synthetic data alone trains controller foundation model","Synthetic-to-real transfer for pendulum dynamics","Zero real data: pretrained model predicts pendulums","Foundation model from synthetic dynamics generalizes to hardware","Synthetic-only pretraining powers real system predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that random smooth RKHS functions, discretized with Euler's method and filtered by total variation, cover the behavior of real dynamical systems well enough that a transformer trained on them transfers to cart-pole and Furuta pendulums; if the prior misses the target class, the zero-shot and fine-tuning advantages disappear.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data alone trains controller foundation model","Synthetic-to-real transfer for pendulum dynamics","Zero real data: pretrained model predicts pendulums","Foundation model from synthetic dynamics generalizes to hardware","Synthetic-only pretraining powers real system predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1113,"prompt_tokens":838,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":454,"tokens_out":275,"duration_ms":3114,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:25:35.751509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same pipeline and evaluate it zero-shot on a dynamical system with genuinely non-smooth dynamics, such as a bouncing ball or an oscillator with Coulomb friction, comparing prediction error to the reported cart-pole results; if errors are orders of magnitude larger and small-data fine-tuning cannot recover, the RKHS synthetic prior is not representative of the target class.","supporting_citations":[],"review_version":1}