{"id":"0aba361b-c207-45ad-86b1-f3e564dbcaba","arxiv_id":"2505.20716","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Removing embedding layers from most of fifteen forecasting models keeps or improves accuracy on four ETT datasets and often reduces training time and memory.","lead":"This paper removes the embedding layers from fifteen published time series forecasting models and reruns them on four benchmarks. It finds that most models stay accurate or improve without embeddings, which could simplify forecasting architectures and save compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No-embedding variant is built with unspecified permutation/concatenation preprocessing, so the ablation may not isolate embedding removal.","rationale":"The paper's strongest claim is causal, and the load-bearing premise is that the without-embedding variant is exactly the original model with embedding layers removed. I agree with the reader's weakest_assumption: the permutation/concatenation preprocessing is an unvalidated replacement, not a minimal ablation. The paper itself flags the extra operations only when explaining degradations (Section 5), which shows the intervention is not just bypassing the embedding. This confound is severe enough that the main quantitative claims cannot be accepted as-is. However, the concern is empirically checkable by inspecting the released code and running a controlled bypass; therefore the appropriate disposition is CONDITIONAL rather than outright rejection. The reader already reached CONDITIONAL, so I recommend UNCHANGED. Independent support (public code, reproducible tables) is limited by the lack of protocol detail; the confidence-interval appendix contains typos and impossible intervals, reinforcing that the evidence is not yet airtight.","tokens_in":18254,"tokens_out":5456,"duration_ms":55208,"concrete_test":"Inspect the released repository for one claimed large gain, e.g., ETSformer on ETTh1 H=720 (Table 2: 0.987 to 0.627): trace the exact tensor shape entering the first encoder layer in both with-embedding and without-embedding modes, and identify the precise permutation/concatenation applied. Then build a minimal controlled ablation that bypasses the embedding without ad hoc preprocessing: feed the length-96 raw series directly to the downstream layers with the embedding projection removed and, where dimensions differ, use a fixed random projection or zero-padding to the required width, rerunning with the same seeds. If the large MSE drop vanishes or reverses under the controlled bypass, the headline result is an artifact of the ad hoc preprocessing rather than a genuine effect of removing data embeddings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: removing data embedding layers is what produces the observed accuracy and efficiency gains. The intervention, however, is not a clean deletion of the embedding layer. The authors state in Section 1 that 'in the absence of embedding layers, some preprocessing steps, such as permutation and concatenation are performed to reconcile the input with the model expected dimensions,' and Section 5's 'Configurations with degraded performance' confirms that the no-embedding pipeline 'manually permutes, concatenates, and processes the raw input data.' This means the no-embedding model is a different network than the original minus the embedding. The permutation/concatenation can change the input length, channel order, or positional alignment, and can introduce additional tensors and different GPU kernel shapes. For example, if concatenation tiles a length-96 sequence to fill a dimension-512 slot required by an inverted-embedding model (SOFTS, EDformer), the downstream model sees repeated copies of the same signal, which is effectively a crude averaging/ensemble operation. The paper never validates that this preprocessing is information-preserving or that it is equivalent to bypassing the embedding, and the exact operation is not specified per model. Without this validation, the observed gains cannot be attributed to the absence of data embedding layers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether data embedding layers in modern time series forecasting models are necessary. The authors take fifteen published forecasting models spanning Transformer-, MLP-, and hybrid/decomposition-based architectures, run them with their original embedding layers on four ETT benchmark datasets at horizons 96, 192, 336, and 720, and compare against variants with the embedding layers \"bypassed\" using permutation and concatenation preprocessing. The central claim is that removing data embedding layers usually does not degrade forecasting accuracy and in many cases improves both accuracy and computational efficiency; the paper further claims that these gains can exceed the performance differences typically reported between competing state-of-the-art models. The main evidence is presented as MSE/MAE tables (Tables 2-5), runtime and memory measurements, and confidence intervals for a subset of models in Table 6.","tokens_in":18510,"tokens_out":7034,"duration_ms":68111,"significance":"If the central claim survives scrutiny, it is a practically valuable negative result: it challenges a commonly used architectural component and could motivate simpler and cheaper forecasting pipelines. The breadth of the study—fifteen models, four datasets, two accuracy metrics, and runtime/memory measurements—is a genuine strength, as is the effort to provide code and to include confidence intervals for selected models. However, the current manuscript does not yet establish the causal claim that removing embeddings produces the observed improvements, because the no-embedding intervention is under-specified and the confidence-interval evidence contains internal contradictions. The paper's value is therefore conditional on a careful revision of the ablation protocol and the statistical reporting.","major_comments":[{"comment":"The no-embedding condition is not a clean deletion of the embedding layer. The manuscript states that \"in the absence of embedding layers, some preprocessing steps, such as permutation and concatenation are performed to reconcile the input with the model expected dimensions,\" and later that the raw input is \"manually permutes, concatenates, and processes.\" Because the exact operations are never specified per model, the differences in Tables 2-5 cannot be attributed to the embedding layer alone. For models whose inverted embedding projects a length-96 sequence to a dimension-512 latent space, concatenation or tiling to fill the expected input shape effectively feeds repeated copies of the input to downstream layers, which is not equivalent to simply removing the embedding. The authors must specify the preprocessing for each of the fifteen models, demonstrate that it is information-preserving or otherwise controlled for input shape, and ideally validate the replacement against a simple fixed baseline such as a linear projection with a near-identity initialization.","section":"Section 5, \"Configurations with degraded performance\"; Section 1"},{"comment":"The claim that the confidence intervals for with-embedding and without-embedding models \"do not overlap\" is contradicted by the numbers in Table 6. For ETTh1, Times2D intervals (0.436, 0.442) and (0.429, 0.439) overlap; PDF intervals (0.451, 0.468) and (0.445, 0.459) overlap; and SOFTS intervals also overlap. Additionally, Table 6 contains internally impossible intervals: PDF on ETTm1 with embedding reports MSE 0.392 with 95% CI (0.394, 0.455), and SOFTS on ETTm1 with embedding reports MSE 0.408 with CI (0.403, 0.394). The statistical-significance argument in the text is therefore invalid and must be recomputed and restated; the main tables alone do not currently provide evidence that the improvements are statistically reliable.","section":"Section 5, \"Confidence intervals\"; Table 6"},{"comment":"The manuscript does not specify how many independent runs are used for the main results in Tables 2-5. Several reported improvements are as small as 0.001-0.003 in MSE or MAE, and some configurations show degradation; without a seed count or variance information, single-run numbers cannot support the aggregate claim that removing embeddings improves accuracy in \"over 95% of\" configurations. The paper should state the number of runs and report error bars or standard deviations for all principal comparisons, or explicitly restrict significance claims to the confidence-interval experiments.","section":"Section 4 and Tables 2-5"}],"minor_comments":[{"comment":"The claim that improvements occur in \"over 95%\" of configurations is not backed by a precise count or a definition of a configuration. Some table entries show degradation (e.g., Crossformer on ETTh1 at H=96, MICN on ETTm1 at H=96, PatchTST on ETTm1 at H=96); the authors should provide the exact numerator and denominator and state whether a configuration means a model-horizon pair or a model-horizon-metric cell.","section":"Section 5, \"Accuracy typically improves without embeddings\""},{"comment":"There are numerous formatting and transcription errors in the confidence-interval tables: the header \"Time2D\" should read \"Times2D,\" Table 6 contains impossible intervals as noted above, and Tables 8-9 contain malformed entries such as \"0.461 0.458, 0.464)\" and extra digits like \"0.4152\" and \"0.5532.\" These errors undermine reader trust in the numerical results and should be corrected throughout.","section":"Table 6 and Appendix A.4"},{"comment":"The limitations section does not mention that the no-embedding condition relies on permutation/concatenation preprocessing, which is a potential confound for the paper's main causal claim. This limitation should be acknowledged explicitly and, ideally, addressed with sensitivity analyses.","section":"Section 1 and Appendix A.1"},{"comment":"Table 1 lists iTransformer, RLinear, and TimeMixer, but these models do not appear in the ablation tables; the paper should clarify whether they are excluded from the fifteen evaluated models or are only reference points, and, if excluded, state the reason.","section":"Section 4 and Table 1"},{"comment":"The GitHub repository name \"neuripsdataembedidng\" appears to contain a typo and should be verified; the link should point to a stable, accessible repository since the manuscript relies on it for reproducibility.","section":"Abstract and GitHub link"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-reported reproducibility checklist states that experimental details and code are fully disclosed, but the manuscript does not provide the exact no-embedding preprocessing for each model, and the confidence-interval tables contain impossible intervals. During revision, the editor may wish to request the raw result files and the exact code diff used to create the 'without embedding' variants. The paper may also benefit from a line-by-line proofreading of all tables before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this is the first systematic ablation of embedding layers across a broad set of modern time series models, and the directional finding is probably right — embeddings often don't help and sometimes hurt. But the paper's central causal claim is undercut by a design flaw: the \"without embedding\" condition is not a clean deletion. It replaces the embedding with ad hoc permutation/concatenation preprocessing to satisfy the model's expected input shape. That makes the comparison a different model, not just the original minus an embedding. The stress-test note is on target here, and the paper never specifies these operations per model or validates that they are information-preserving. So we can't cleanly attribute the gains to the absence of embeddings.\n\nWhat's genuinely good: the breadth — fifteen models, four standard datasets, reproduction of official code, and a consistent trend across most configurations. The authors also include an explicit limitations section and are careful to scope their claim to forecasting on regularly sampled data. The computational efficiency story (runtime and memory) is a useful practical addition, even if it's partly confounded by the same preprocessing change.\n\nThe soft spots beyond the main confound: most tables report single runs; confidence intervals are computed for only four models on two datasets, and Table 6 contains an impossible interval (MSE 0.392 with CI (0.394, 0.455)). That kind of internal inconsistency makes the statistical claims hard to trust. Also, all four datasets are ETT variants — same domain, same sensors, regular sampling. The \"gains exceed SOTA differences\" point is evocative but rests on a small convenience table.\n\nNet assessment: the paper addresses an important question and the direction of the result is plausible, but it needs a clean ablation protocol and proper uncertainty quantification before the quantitative claims can be accepted. Worth sending to referees, with the expectation of major revision.\n\nRecommendation: send to peer review — a serious referee could push the authors to either specify and verify the replacement preprocessing or redesign the ablation to isolate embeddings (e.g., keep the same projection but make it identity, random, or untrained).","headline":"First broad ablation of embedding layers in time series models, with a plausible directional finding, but the no-embedding condition is confounded by ad hoc preprocessing and the statistical evidence is shaky.","tokens_in":18976,"tokens_out":2260,"would_cite":false,"duration_ms":23552,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Removing embedding layers generally improves time series forecasts.","keywords":["time series forecasting","data embedding","ablation study","model simplification","multivariate time series","forecasting accuracy","computational efficiency","embedding layers"],"falsifier":"Run the same fifteen-model ablation but, instead of permutation and concatenation, substitute a fixed random linear projection with the embedding's output dimension, so the downstream network sees identically shaped input with no learned embedding. If accuracy no longer improves (or degrades) relative to the original models, the paper's attribution to embedding removal would be undermined. A second check would be to test on a dataset with irregular sampling or a longer input window and find configurations where embedding-free variants consistently lose.","tokens_in":18077,"feed_emoji":"📉","tokens_out":6650,"duration_ms":58578,"temperature":0.7,"pith_summary":"This paper asks whether the embedding layers common in modern time-series forecasting models actually pay for themselves. Across fifteen state-of-the-art models and four benchmark datasets, the authors ablate the embedding components and find that removing them does not hurt accuracy in the vast majority of configurations, and often improves MSE and MAE while reducing training time and memory use. The paper's central point is that these gains frequently exceed the tiny differences, a few thousandths in error metrics, that separate competing state-of-the-art models. If the finding holds, it implies that a large share of recent architectural complexity in forecasting is unnecessary, and that simpler embedding-free variants deserve to be standard baselines. The authors are careful to say this does not prove embeddings are never useful, only that their widespread inclusion should be justified empirically.","feed_headline":"Removing embedding layers generally improves time series forecasts","feed_subtitle":"Across 15 models and 4 benchmarks, simpler embedding-free variants match or beat the originals and train faster.","key_machinery":"The load-bearing mechanism is the embedding-layer ablation itself: for each of the fifteen models, the authors identify which of the five embedding families, value, temporal, positional, inverted, or patch, is used, bypass that layer, and reconcile the input dimensions with the downstream model through permutation and concatenation of the raw input. Training and evaluation are otherwise kept identical, so any change in MSE, MAE, training time, or memory is attributed to the presence or absence of the embedding layer. The classification of embedding techniques supplies the vocabulary for deciding what counts as an embedding in each architecture.","core_discovery":"On the paper's own terms, the discovery is that data embedding layers, including value, temporal, positional, inverted, and patch embeddings, are often redundant in multivariate time-series forecasting. After reproducing fifteen published models and rerunning each with its embedding layer bypassed, the paper reports that over 95% of model-horizon configurations improve in MSE or MAE without the embedding, with average error reductions around 0.02 to 0.03 on the ETT datasets. In several cases, such as ETSformer on ETTh1 at horizon 720, the improvement is large, with MSE down by 0.360. Removing embeddings also cuts training time and memory in most configurations; a minority of models, such as EDformer, become slower or more memory-hungry because the replacement preprocessing creates extra tensors and misaligns with GPU kernel tile sizes. The paper concludes that raw multivariate inputs often carry enough information on their own, and recommends that the community assess embeddings critically before adding complexity.","pith_inferences":["A natural extension is to test whether the same conclusion holds for other architectural components, such as normalization layers (for example, RevIN) or residual connections, since the paper explicitly leaves those interactions out of scope.","The finding suggests that learned embeddings may act as a form of overparameterization or implicit regularization rather than as necessary feature extractors; a controlled study of variance and training dynamics would clarify the mechanism.","Because the evaluation is limited to four regularly sampled ETT datasets, the result's scope is uncertain for irregularly sampled series, high-dimensional exogenous inputs, or non-stationary regimes.","A direct test would compare the permutation-and-concatenation replacement against a randomly initialized fixed linear projection of matched output dimension; if the improvements vanish, the reshaping itself, not the absence of learned embeddings, would be the active ingredient."],"forward_implications":["Embedding-free variants of popular forecasting models can serve as stronger, cheaper baselines than the original releases.","Reported differences between state-of-the-art models of a few thousandths in MSE or MAE are smaller than the typical gain from removing embeddings, so model rankings may be less meaningful than architecture simplifications.","Longer forecasting horizons tend to benefit more from embedding removal, suggesting simplified designs are especially relevant for long-term forecasting.","Computational savings in training time and memory make embedding-free variants attractive for memory-constrained or resource-limited deployment.","The result calls for empirical justification of any new embedding component introduced in future forecasting architectures."],"supporting_citations":[{"why":"Supplies Times2D, one of the state-of-the-art models whose embedding removal yields large MSE improvements, anchoring the paper's main comparison.","marker":"[15]"},{"why":"Introduces the patch-embedding design that the ablation removes in PatchTST, providing the baseline architecture and its embedding rationale.","marker":"[16]"},{"why":"Defines SOFTS and its inverted embedding, one of the five embedding families the paper identifies and ablates.","marker":"[5]"},{"why":"Presents MICN, a multi-scale convolution model whose token and temporal embeddings are among the components removed.","marker":"[18]"},{"why":"Describes ETSformer's convolutional value embedding, the configuration with the largest reported accuracy gain after removal.","marker":"[21]"},{"why":"Provides Crossformer's dual patch-plus-positional embedding, which shows the biggest long-horizon improvements when removed.","marker":"[23]"},{"why":"Reports LiNo's state-of-the-art numbers that differ by only thousandths, establishing the baseline against which embedding-removal gains are compared.","marker":"[22]"},{"why":"Introduces PDF, a hybrid decomposition model whose embedding removal shows modest but consistent gains.","marker":"[3]"}],"fun_headline_variants":["Embeddings often hurt time series forecasts","Drop embeddings, improve forecast accuracy","Simpler models beat embedding-heavy forecasts","Blank embeddings boost 95% of forecast setups","No embeddings needed for better time series"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing an embedding layer with permutation and concatenation of the raw input is equivalent to simply deleting the embedding; if that reshaping itself changes what information reaches the model, the measured gains cannot be cleanly attributed to removing embeddings.","fun_headline_variants_meta":{"raw":{"variants":["Embeddings often hurt time series forecasts","Drop embeddings, improve forecast accuracy","Simpler models beat embedding-heavy forecasts","Blank embeddings boost 95% of forecast setups","No embeddings needed for better time series"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1136,"prompt_tokens":878,"completion_tokens":258,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":494,"tokens_out":258,"duration_ms":3367,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:47:52.150012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fifteen-model ablation but, instead of permutation and concatenation, substitute a fixed random linear projection with the embedding's output dimension, so the downstream network sees identically shaped input with no learned embedding. If accuracy no longer improves (or degrades) relative to the original models, the paper's attribution to embedding removal would be undermined. A second check would be to test on a dataset with irregular sampling or a longer input window and find configurations where embedding-free variants consistently lose.","supporting_citations":[{"cited_title":"Times2d: Multi-period de- composition and derivative mapping for general time series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies Times2D, one of the state-of-the-art models whose embedding removal yields large MSE improvements, anchoring the paper's main comparison."},{"cited_title":"A time series is worth 64 words: Long-term forecasting with transformers","cited_arxiv_id":null,"evidence_quote":"Introduces the patch-embedding design that the ablation removes in PatchTST, providing the baseline architecture and its embedding rationale."},{"cited_title":"SOFT S: Efficient multivariate time series forecasting with series-core fusion","cited_arxiv_id":null,"evidence_quote":"Defines SOFTS and its inverted embedding, one of the five embedding families the paper identifies and ablates."},{"cited_title":"MICN: Multi-scale local and global context modeling for long-term series forecasting","cited_arxiv_id":null,"evidence_quote":"Presents MICN, a multi-scale convolution model whose token and temporal embeddings are among the components removed."},{"cited_title":"ETSformer: Ex- ponential smoothing transformers for time-series forecas ting, 2023","cited_arxiv_id":null,"evidence_quote":"Describes ETSformer's convolutional value embedding, the configuration with the largest reported accuracy gain after removal."},{"cited_title":"Crossformer: Transformer utilizing cross-dimension depen- dency for multivariate time series forecasting","cited_arxiv_id":null,"evidence_quote":"Provides Crossformer's dual patch-plus-positional embedding, which shows the biggest long-horizon improvements when removed."},{"cited_title":"Lino: Advancing recursive residual decomposition of linea r and nonlinear patterns for robust time series forecasting, 2025","cited_arxiv_id":null,"evidence_quote":"Reports LiNo's state-of-the-art numbers that differ by only thousandths, establishing the baseline against which embedding-removal gains are compared."},{"cited_title":"Pe- riodicity decoupling framework for long-term series forec asting","cited_arxiv_id":null,"evidence_quote":"Introduces PDF, a hybrid decomposition model whose embedding removal shows modest but consistent gains."}],"review_version":1}