{"id":"b76798c2-d24b-43f8-9109-04886228a91c","arxiv_id":"2506.03128","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"A zero-shot time series forecaster, COSMIC, learns to use covariates via synthetic augmentation and beats prior pretrained models on most covariate-included and no-covariate benchmarks.","lead":"COSMIC is a pretrained time series forecasting model that uses in-context learning to exploit external covariates without any task-specific training. It is trained entirely on covariate-free data using a new synthetic augmentation scheme, and it outperforms prior zero-shot models on most covariate-included benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Covariate-use claim rests on untested transfer from simple synthetic impact functions to real, potentially nonlinear/nonstationary covariate-target relationships; the paper's own D.5 linear baseline outperforming COSMIC on Benchmark I underscores that the learned covariate mechanism may be…","rationale":"The reader's weakest_assumption correctly identifies the simple, stable covariate-target relationship assumption and Section 6's caveat. My stress-test agrees and adds specificity: the impact function space F is the sole source of covariate supervision, so the method's generalizability is bounded by F unless real datasets happen to match it. The paper provides positive evidence on 11 real dataset groups and a convincing ablation (Table 1) that augmentation is what enables covariate use, which supports the claim on the reported benchmarks. However, the broad abstract claim 'state-of-the-art performance ... with and without covariates' is not scoped to locally linear relationships, and the evaluation does not include datasets engineered to be outside F. Appendix D.5's result that a simple linear in-context model beats COSMIC on Benchmark I is an internal data point that reinforces the concern: on the full benchmark, COSMIC's learned covariate mechanism does not even beat a trivial external linear model, although COSMIC is more robust on Benchmark II and avoids catastrophic failures. This does not falsify the reported benchmark results, so the CONDITIONAL verdict stands, but the paper should either demonstrate transfer to nonlinear/nonstationary covariate dependencies or restrict the central claim accordingly. The proposed synthetic test would settle whether the learned mechanism is a general in-context estimator or a memorizer of the restricted synthetic family.","tokens_in":46910,"tokens_out":14360,"duration_ms":184910,"concrete_test":"Construct a controlled covariate benchmark with three known failure modes for F: (i) multiplicative effects y_t = base_t * (1 + beta x_t), (ii) regime-switching coefficients where beta changes at a known changepoint inside the context or forecast horizon, and (iii) interactions between two covariates, e.g. y_t = base_t + beta_1 x1_t + beta_2 x2_t + gamma x1_t x2_t. Generate at least 100 series per setting with realistic base signals, supply past and future covariates, and compare COSMIC-with-covariates against COSMIC-without-covariates on WQL and MASE. Also compare against task-specific PatchTST/TFT with covariates trained on the same generated data. If COSMIC's covariate gain largely disappears or turns negative on settings (i)-(iii) while task-specific models improve, the central claim must be scoped to locally linear, non-interacting, stable covariate effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that COSMIC 'effectively leverages covariates in zero-shot forecasting' requires that the synthetic covariate-target relationships used in Informative Covariate Augmentation (Section 4) transfer to real-world relationships. But the impact function space F is explicitly restricted to piecewise-linear, threshold-triggered functions with constant coefficients over the whole series: f_t = a0 + <a, x_{t-l:t}> + eps on active steps, otherwise 0. The model is therefore trained only on relationships of this exact form. The paper acknowledges in Section 6 that complex relationships will not be captured, but the abstract and contribution claims do not carry this caveat. The evaluation does not establish how far beyond F the method generalizes: many covariate-included datasets involve electricity prices, weather, and multi-target traffic/air-quality series, where covariate effects are likely nonlinear, interacting, or regime-dependent. This concern is sharpened by the paper's own Appendix D.5: a simple ridge-regression in-context covariate model combined with a no-covariate pretrained forecaster outperforms COSMIC's learned covariate integration on Benchmark I, while COSMIC only wins on Benchmark II (which excludes target-as-covariate datasets). That pattern is exactly what one would expect if COSMIC's learned covariate mechanism adds little beyond simple local linear structure. Since the headline SOTA claim with covariates is the paper's main contribution, the transferability of F to real covariate-target dependencies is the most load-bearing assumption and it is not directly stress-tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COSMIC, an encoder-decoder transformer for zero-shot time series forecasting that accepts covariates through in-context learning. To avoid relying on scarce real-world covariate datasets, the authors propose Informative Covariate Augmentation, which generates synthetic covariates and adds piecewise-linear, quantile-threshold impact functions to standard no-covariate pretraining series. COSMIC is evaluated on 11 covariate-included dataset groups (split into Benchmark I and Benchmark II) and on the Chronos no-covariate benchmark. The paper reports first-place rankings on 9 of 11 WQL and 7 of 11 MASE dataset groups among pretrained zero-shot models, and competitive performance relative to task-specific models. The central claims are that COSMIC achieves state-of-the-art zero-shot forecasting both with and without covariates, and that the augmentation enables effective covariate usage.","tokens_in":47233,"tokens_out":5505,"duration_ms":65279,"significance":"If confirmed, COSMIC would be a useful contribution because it provides a way to train covariate-aware zero-shot forecasters without any covariate-included pretraining data, and its evaluation design is broad: multiple model sizes, both covariate and no-covariate benchmarks, an augmentation ablation (Table 1), analysis of Moirai's training overlap, past-only covariate experiments, inference-time measurements, and a comparison against an external linear in-context model (Appendix D.5). The synthetic-only pretraining recipe is appealing for practical deployment. The main weakness is that the headline state-of-the-art claim is not supported against the paper's own simple linear baseline on Benchmark I, and the question of how far the learned covariate mechanism generalizes beyond the synthetic impact-function family is left unmeasured.","major_comments":[{"comment":"The paper's own comparison against a ridge-regression in-context covariate model combined with a no-covariate pretrained forecaster shows that this simple baseline outperforms COSMIC on Benchmark I for both MASE and WQL, while COSMIC wins only on Benchmark II (which excludes datasets where other targets are used as covariates). This directly qualifies the abstract's and Section 5.1's claim that COSMIC achieves state-of-the-art zero-shot forecasting with covariates. Because Benchmark I is the headline covariate-included evaluation, the authors should either include this baseline in the main results and revise the contribution claim, or justify carefully why Benchmark II is the appropriate basis for the claim and make that framing explicit throughout the paper.","section":"Appendix D.5, Figure 21"},{"comment":"The impact function family F is restricted to piecewise-linear, threshold-triggered functions with coefficients that are constant over the whole series, and Section 4 explicitly assumes the covariate-target relationship is stable throughout the context and forecast horizon. Section 6 concedes that highly complex relationships cannot be captured. However, the paper does not provide a stress test or an analysis of where COSMIC's covariate mechanism breaks down as the relationship moves outside F, such as nonlinear, lag-varying, regime-dependent, or nonstationary effects. Given that the real-world datasets in Table 3 include electricity, traffic, and air-quality series where such effects are plausible, the evaluation does not establish how far the learned in-context covariate mechanism generalizes beyond the augmentation distribution. I would ask for a synthetic transfer experiment with out-of-family impact functions, or for a tempering of the generalization claims in the abstract and conclusion.","section":"Section 4, Impact Function"},{"comment":"The quantitative support for the phrase 'effectively leverages covariates' is weaker than the rank-based claims suggest. Providing covariates improves aggregate MASE by only about 1% (Figure 6), and Table 1 shows that both the augmentation and the covariate-access benefit are concentrated in a few datasets, such as PEMS08 and ETTh. Moreover, no error bars or multiple seeds are reported anywhere in the experimental section, so small aggregate differences — for example COSMIC Base WQL 0.546 versus Chronos Bolt Base 0.552 in Figure 3 — cannot be separated from noise. Reporting confidence intervals, standard deviations across evaluation rolls, or repeated-seed results for the key comparisons would be necessary to support the 'state-of-the-art' wording.","section":"Section 5.1, Figures 3 and 6, Table 1"}],"minor_comments":[{"comment":"There are several typos, including 'Copmarision' in Section 5.1, 'zeros-hot' in Appendix C, 'pice-wise' in Appendix A, 'Aggreagted' in multiple figure captions, and 'covaraites' in Section 4; these should be corrected before publication.","section":"Throughout"},{"comment":"Algorithm 2 returns the condition symbol inconsistently: line 12 writes '(, ⊕, z, q)' while line 13 writes '(>, y, 0)'. The notation should be harmonized, and the text around the algorithm should define what 'first order' and 'piece-wise' mean since these names appear in Table 2 but not in the main text.","section":"Appendix A, Algorithm 2"},{"comment":"The failure-case analysis introduces Rideshare and KDD2022 as datasets that are not part of the covariate-included benchmark in Table 3. The paper does state this, but it would help readers if Figure 21 and the surrounding text made explicit that these are auxiliary datasets chosen to probe the linear in-context model's limitations rather than part of the main evaluation.","section":"Appendix D.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically solid and the augmentation idea is interesting, but the headline claim of state-of-the-art covariate-based zero-shot forecasting is currently contradicted by the authors' own linear in-context baseline on Benchmark I. The fix is feasible: the linear baseline should be moved into the main comparison or the claims should be scoped to Benchmark II, and a transfer stress test should be added. I would not reject the paper, but the current version overstates what is established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"COSMIC is a legitimate step forward: a zero-shot forecaster that uses covariates without any covariate-labeled pretraining data, thanks to a clean synthetic augmentation trick. Worth engaging, but the SOTA-with-covariates claim is stronger than the evidence.\n\nWhat's actually new and good: Informative Covariate Augmentation constructs training samples where covariates have known, local, simple effects on the target, so the model learns in-context covariate-target inference without needing scarce real covariate datasets. The architecture—patched encoder-decoder with variate separation tokens—is sensible, and the ablation in Table 1 is convincing: without augmentation, providing covariates at inference does almost nothing; with it, covariate use improves forecasts on several datasets. No-covariate zero-shot performance roughly matches Chronos, so the capability doesn't come at that cost. The evaluation is broad: 11 covariate-included dataset groups, two horizons, task-specific and local baselines, past-only vs. past-and-future covariate settings, plus a useful analysis of an external linear in-context model.\n\nSoft spots, in proportion. The load-bearing assumption is that simple synthetic impact functions transfer to real covariate-target relationships. The function space is piecewise-linear threshold-triggered with constant coefficients; the paper acknowledges complex relationships won't be captured, but the abstract and contributions don't. The sharpest internal evidence is Appendix D.5: on Benchmark I (all covariate datasets), a ridge regression in-context model combined with a no-covariate forecaster beats COSMIC. COSMIC wins on Benchmark II, which excludes target-as-covariate datasets, so the learned mechanism does help where covariates are truly exogenous. Still, that pattern is exactly what you'd expect if COSMIC mostly learns simple local linear structure. A direct transfer test—e.g., synthetic nonlinear or regime-switching covariate effects—would make the claim much stronger. This is a caveat, not a fatal flaw; the paper is honest about it in Section 6.\n\nOther issues are reproducible ones: no error bars or multiple seeds, no released code or weights, and some baselines (Chronos Bolt) are unpublished. Those should be fixed in revision. The evaluation itself is not circular—training for predictive covariates is a learning strategy, and the test data is real.\n\nWho this is for: anyone working on pretrained time series models or practical forecasting with exogenous variables. It deserves a serious referee; I'd accept and ask for code/weights, error bars, and a transfer experiment that goes beyond the F family. I'd also tone down the SOTA claim in the abstract.","headline":"COSMIC is a genuine step forward in covariate-aware zero-shot forecasting, but the SOTA-with-covariates headline overstates what the evidence, including the paper's own linear in-context baseline, supports.","tokens_in":47783,"tokens_out":2385,"would_cite":true,"duration_ms":28436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62M10"],"pacs":[],"model":"deepseek-v4-flash","headline":"COSMIC is a pretrained transformer that performs zero-shot forecasting with covariates by inferring covariate-target relationships from context, trained on synthetically augmented covariate-free data.","keywords":["zero-shot forecasting","covariates","in-context learning","time series transformers","informative covariate augmentation","probabilistic forecasting","pretrained forecasting models"],"falsifier":"A controlled benchmark where the target depends on a covariate through a stable but nonlinear map, such as threshold-switching or multiplicative interaction, with ample evidence of the relationship in the context: if COSMIC's covariate-input forecasts do not beat its no-covariate forecasts, the claim that it learns covariate-target relations in context would be falsified.","tokens_in":1533,"feed_emoji":"📈","tokens_out":1423,"duration_ms":65192,"temperature":0.7,"pith_summary":"Zero-shot time series models typically ignore covariates or use them poorly. This paper introduces COSMIC, a pretrained transformer that forecasts a target from its history plus any number of covariate series, inferring the covariate-target relationship from the context alone at inference time. Because real covariate-labeled training data is scarce, the authors train COSMIC on covariate-free series to which they add synthetic, informative covariate effects, an approach they call Informative Covariate Augmentation, so the model learns to read these relationships without ever seeing a real covariate dataset. They report state-of-the-art zero-shot results with covariates, best probabilistic score on 9 of 11 dataset groups, and no loss of accuracy on no-covariate benchmarks, suggesting covariates can be a free improvement for pretrained forecasters.","feed_headline":"COSMIC: zero-shot forecasts that actually use covariates","feed_subtitle":"Synthetic augmentation teaches a pretrained transformer to read covariate-target links from context alone.","key_machinery":"The load-bearing object is Informative Covariate Augmentation, which manufactures training samples by sampling covariates from a corpus or a synthetic generator and adding an impact function, a sparse piecewise-linear function of recent covariate lags active on quantile-selected time steps, to an otherwise covariate-free target series. Around it, COSMIC is an encoder-decoder transformer that z-score normalizes each series, patches them with shared weights, and inserts distinct separator tokens and rotary embeddings so the attention stack can tell the target from each covariate and align them in time. The output layer emits nine quantiles per horizon step, trained by quantile loss. The augmentation is what gives the model a training signal that covariates predict the future target; without it, covariates do not reduce the loss and the model has no reason to learn the in-context behavior.","core_discovery":"COSMIC is a pretrained encoder-decoder transformer that accepts a target series plus any number of past-only or past-and-future covariate series, normalizes each individually, patches them into tokens, and interleaves them with separator tokens so the attention stack can relate covariates to the target. At inference it reads the covariate-target relationship from the provided context and applies it to the forecast horizon, outputting nine quantiles per time step. The paper claims this yields state-of-the-art zero-shot probabilistic forecasts on covariate-included benchmarks, first place on 9 of 11 dataset groups in weighted quantile loss, and matches or exceeds prior zero-shot models on no-covariate benchmarks, while remaining competitive with task-specific models that are trained per dataset.","pith_inferences":["A natural extension is to teach other auxiliary reasoning tasks, such as known interventions, calendar effects, or target-target correlations, by synthesizing them into the training corpus in the same way, without needing labeled examples.","Because the paper's analysis shows COSMIC uses covariates more strongly when more evidence of the relationship appears in context, larger models and longer contexts may close more of the remaining gap to task-specific covariate models.","The exclusion of static covariates suggests a hybrid design: a task-specific global encoder for static features combined with COSMIC's in-context machinery could handle datasets where the covariate signal lives outside the horizon."],"forward_implications":["Pretrained forecasters can now be deployed on covariate-rich domains such as energy, retail, and healthcare without any dataset-specific training or manual feature engineering.","Covariate-aware zero-shot models can be built from covariate-free corpora, so the scarcity of public covariate-labeled time series need not block this capability.","Providing locally informative covariates, including past-only ones, can improve forecasts, and on datasets where covariates carry no local signal, performance stays roughly flat.","COSMIC's probabilistic forecasts are competitive with task-specific models trained per dataset, at least on the aggregate weighted quantile loss benchmark, despite seeing none of the task data.","The ability to consume covariates does not degrade no-covariate zero-shot performance, since the same model matches or beats prior zero-shot models on the covariate-free benchmark."],"supporting_citations":[{"why":"Supplies the Chronos training corpus, the evaluation protocol, and the metric aggregation procedure used throughout the paper.","marker":"Ansari et al. (2024a)"},{"why":"Provides Moirai, the only other pretrained model that supports covariates in zero-shot forecasting, and several covariate-included evaluation datasets.","marker":"Woo et al. (2024)"},{"why":"Provides the transformer architecture that COSMIC extends with patching, separator tokens, and rotary embeddings.","marker":"Vaswani et al. (2017)"},{"why":"Provides the efficient T5 configuration on which COSMIC's hyperparameters and architecture are based.","marker":"Tay et al. (2021)"},{"why":"Introduces the patching of time series into tokens, which makes adding multiple covariate signals to the context computationally affordable.","marker":"Nie et al. (2022)"},{"why":"Motivates the in-context learning capability that the paper relies on for inferring covariate-target relationships at inference time.","marker":"Brown et al. (2020)"},{"why":"Provides TimesFM, a pretrained zero-shot forecasting baseline that does not support covariates and is compared against in the benchmarks.","marker":"Das et al. (2024)"}],"fun_headline_variants":["COSMIC zero-shot forecasts read covariates from context","COSMIC: in-context learning unlocks covariates for zero-shot","COSMIC reads covariate-target links from context alone","Zero-shot forecasting that learns covariates from context alone","COSMIC: zero-shot forecasting that actually uses covariates"],"cache_read_input_tokens":49792,"weakest_assumption_plain":"The covariate-target relationship must be simple, roughly linear or piecewise-linear with small lags, and stable across the context and forecast horizon, and the synthetic covariate relationships used in pretraining must transfer to real-world covariates.","fun_headline_variants_meta":{"raw":{"variants":["COSMIC zero-shot forecasts read covariates from context","COSMIC: in-context learning unlocks covariates for zero-shot","COSMIC reads covariate-target links from context alone","Zero-shot forecasting that learns covariates from context alone","COSMIC: zero-shot forecasting that actually uses covariates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000872,"raw_usage":{"total_tokens":3702,"prompt_tokens":797,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":2829}},"tokens_in":413,"tokens_out":2905,"duration_ms":24676,"temperature":1.0,"reasoning_tokens":2829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:08:15.552009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled benchmark where the target depends on a covariate through a stable but nonlinear map, such as threshold-switching or multiplicative interaction, with ample evidence of the relationship in the context: if COSMIC's covariate-input forecasts do not beat its no-covariate forecasts, the claim that it learns covariate-target relations in context would be falsified.","supporting_citations":[{"cited_title":"Unified Training of Universal Time Series Forecasting Transformers","cited_arxiv_id":null,"evidence_quote":"Provides Moirai, the only other pretrained model that supports covariates in zero-shot forecasting, and several covariate-included evaluation datasets."},{"cited_title":"N., Kaiser, L., and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Provides the transformer architecture that COSMIC extends with patching, separator tokens, and rotary embeddings."},{"cited_title":"W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D","cited_arxiv_id":null,"evidence_quote":"Provides the efficient T5 configuration on which COSMIC's hyperparameters and architecture are based."},{"cited_title":"H., Sinthong, P., and Kalagnanam, J","cited_arxiv_id":null,"evidence_quote":"Introduces the patching of time series into tokens, which makes adding multiple covariate signals to the context computationally affordable."},{"cited_title":"A decoder-only foundation model for time-series forecasting","cited_arxiv_id":null,"evidence_quote":"Provides TimesFM, a pretrained zero-shot forecasting baseline that does not support covariates and is compared against in the benchmarks."}],"review_version":1}