{"id":"c14929b2-a0ae-42f1-a97e-986d5b060f13","arxiv_id":"2501.10448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LiPFormer combines a simplified patch-wise Transformer with a CLIP-style contrastive module that exploits future covariates to forecast time series more accurately and efficiently.","lead":"LiPFormer is a lightweight patch-based transformer for time series forecasting that drops layer normalization and feed-forward networks, and adds a contrastively pretrained module that uses future context such as weather or calendar features. On nine benchmarks it reports accuracy comparable to or better than heavier state-of-the-art models while cutting parameters, training time, and memory, and it runs faster on a CPU-only edge device.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LN/FFN removal is justified only by ablations on ETTh1 and ETTm2; the evidence is noisy and variants are not retuned, so the lightweight architecture's accuracy premise is unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the decision to remove Layer Normalization and Feed-Forward Networks rests on a narrow ablation. This is the most load-bearing concern because it directly supports both halves of the central claim: the accuracy advantage is attributed (Section IV-E1, IV-E4) to removing components that 'do not substantially contribute to accuracy,' while the efficiency advantage is achieved by deleting them. If that premise fails on the other seven datasets, the model could require LN/FFN to match SOTA accuracy, and the win-win claim would reduce to a dataset-specific trade-off. The paper's own admission in Section IV-B2 that accuracy 'marginally declines' on large-channel datasets, together with the internally inconsistent inference-time statements (abstract says 1/3, introduction says less than 1%, Table VII shows ratios from 0.10 to 0.62), reinforces that the empirical evidence is not yet robust. I am not advocating rejection: the efficiency numbers are plausible, the cross-patch attention idea is novel, and the weak data enriching module is a reasonable contribution with positive transplant results (Table XII). But the central claim depends on a generalizable accuracy benefit that the current ablation does not establish. The concrete test—broad, retuned, multi-seed ablations—would settle whether the LN/FFN premise holds, and therefore whether the central claim should be accepted, revised, or restricted. Since the reader already conditioned acceptance on additional evidence, the verdict remains CONDITIONAL. I credit the paper for releasing efficiency measures and for comparing against strong baselines like PatchTST and TiDE, but the missing error bars and the narrow ablation keep correctness risk high.","tokens_in":1245,"tokens_out":995,"duration_ms":186281,"concrete_test":"Run the Section IV-E4 ablation (LiPFormer vs LiPFormer+LN, +FFNs, +LN+FFNs) on ETTm1, ETTh2, Weather, Electricity, Traffic, Cycle, and Electricity-Price, with per-variant hyperparameter retuning (e.g., grid search over learning rate and dropout) and at least 5 random seeds. If adding LN or FFNs produces a mean MSE improvement exceeding the run-to-run standard deviation on any of these datasets, the premise that these components are ineffective for numeric time series is unsupported and the central lightweight-accuracy claim must be re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LiPFormer improves accuracy while cutting parameters and compute, enabled by eliminating Layer Normalization and Feed-Forward Networks. Section IV-E4 (Table X) supports this with ablations on only ETTh1 and ETTm2. The evidence is weak: on ETTm2 at L=96, adding LN reduces MSE (0.158 vs 0.160); at L=336, adding LN also helps (0.266 vs 0.273); the large degradation from adding LN+FFN is concentrated in ETTh1 at L=96 (0.754 vs 0.359). Moreover, the variants are not retuned: hyperparameters such as dropout=0.5, learning rate, and depth were selected for the LN/FFN-free model, so adding modules puts them at an optimization disadvantage. If on the other seven datasets (Weather, Electricity, Traffic, Electricity-Price, Cycle, ETTm1, ETTh2) adding LN/FFN improves accuracy, the claimed win-win of lightweight design without accuracy loss fails. The paper itself concedes in Section IV-B2 that performance 'marginally declines' on large-channel datasets, making the general superiority already fragile. Without a broader, retuned ablation, the key architectural deletion is not established as generally beneficial, and the headline accuracy advantage cannot be separated from dataset-specific tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LiPFormer is a patch-wise Transformer for multivariate long-term time series forecasting that removes Layer Normalization, Feed-Forward Networks, and positional encoding from the backbone, replacing them with two lightweight attention mechanisms (cross-patch and inter-patch) and single-layer MLPs. The paper also proposes a weak-data-enriching module in which future covariates are contrastively aligned with target sequences during pretraining, and the frozen covariate encoder is then used to guide predictions. Experiments on nine datasets report accuracy and efficiency comparisons against iTransformer, TimeMixer, FGNN, PatchTST, DLinear, and TiDE, including CPU-only edge-device inference measurements. The paper claims that LiPFormer significantly outperforms state-of-the-art methods in accuracy while substantially reducing parameter count, training time, and GPU memory.","tokens_in":25453,"tokens_out":4934,"duration_ms":49770,"significance":"If the accuracy and efficiency claims hold, LiPFormer would be a useful contribution to lightweight on-device time series forecasting, and the weak-data-enriching idea could transfer to other Transformer backbones. The paper has several strengths: it evaluates on nine datasets, reports training/inference time, MACs, and parameter counts, includes a real CPU-only deployment experiment, and demonstrates that the covariate encoder can be plugged into other models (Table XII). However, the load-bearing evidence is currently incomplete: the core architectural deletions are supported only by a two-dataset, non-retuned ablation; several reported numbers are internally inconsistent; and no error bars or multi-seed statistics are provided for the accuracy claims.","major_comments":[{"comment":"The claim that removing Layer Normalization and Feed-Forward Networks is beneficial for time series forecasting is not established by the presented evidence. The ablation is run on only ETTh1 and ETTm2, and within those two datasets the result is not consistent: on ETTm2 at L=96 adding LN improves MSE (0.158 vs. 0.160), and at L=336 adding LN also improves MSE (0.266 vs. 0.273). The variants are also not retuned: hyperparameters such as dropout=0.5 and depth were selected for the LN/FFN-free model, so the added modules are at an optimization disadvantage. Since the lightweight architecture is the central contribution, this ablation needs to be extended to more datasets with retuned variants and repeated runs.","section":"IV-E4, Table X"},{"comment":"The inference-time claim is internally inconsistent. Section I states that deployment on a CPU-only edge device shows LiPFormer inference time is 'less than 1%' of Transformer models, while the abstract says 1/3. Table VII shows that on ETTh1 with input length 96, LiPFormer takes 0.55s versus 1.47s for Transformer, i.e., about 37%, not less than 1%; on Weather at input length 96 the ratio is about 47%. The later text in Section IV-E1 also says 'nearly a tenfold increase' for ETTh1 input length 336 (Table VII gives 0.62/5.82, about 10.7%), which is different from both previous statements. The efficiency claims need to be restated consistently with the actual table.","section":"Section I vs. Table VII and Abstract"},{"comment":"Tables III and VIII contain conflicting results for the default configuration. Table VIII reports that pl=48 on ETTm1 at forecast length 720 gives MSE 0.526, while Table III reports LiPFormer with the same default setting on ETTm1 at L=720 gives MSE 0.408. Moreover, in Table VIII pl=48 is not the best patch length on ETTm1 at L=720 (pl=6 gives 0.417, pl=12 gives 0.414, pl=24 gives 0.507), which contradicts the statement in Section IV-E2 that fixed patch length does not lead to performance loss and that pl=48 is generally suitable. These numbers must be reconciled, and the recommendation of pl=48 needs to be justified on all reported datasets.","section":"IV-E2, Table VIII vs. Table III"},{"comment":"No error bars or multi-seed statistics are provided. Section IV-A2 says every experiment uses the same random seed and reports mean scores, but with one seed there is no meaningful mean or variance. Many accuracy differences against the strongest baselines are small (e.g., on Electricity at L=96, LiPFormer MSE 0.131 vs. PatchTST 0.129; on ETTm1 at L=96, 0.296 vs. 0.290). Given the abstract's claim of 'significantly outperforms,' the paper should report results over multiple seeds with standard deviations and, where appropriate, a significance test, especially for the differences that drive the headline conclusion.","section":"IV-A2 and Table III"},{"comment":"The pretraining procedure needs a clearer statement about data boundaries. The contrastive pretraining in Section III-B uses target sequences, i.e., future ground-truth values, to train the Target Encoder, and at inference only the frozen Covariate Encoder is used. The paper should state explicitly that the pretraining and the selection of the frozen covariate encoder use only the training (and possibly validation) splits, and never the test-period future values, to rule out leakage into the reported test metrics. Currently the manuscript does not specify which temporal split is used for the contrastive pretraining.","section":"III-B and Section IV-C"}],"minor_comments":[{"comment":"The sentence 'Every experiment is executed with the same random seed and the mean scores are reported' is self-contradictory; please clarify what the mean is taken over.","section":"IV-A2"},{"comment":"The phrase 'only takes only 1/3 inference time' contains a duplicated 'only'; please also align the fraction with Table VII.","section":"Abstract"},{"comment":"The sentence 'Fixed patch length does not lead to its performance loss on different datasets' is not supported by Table VIII, especially for ETTm1 at L=720; please rephrase after reconciling the numbers.","section":"IV-E2"},{"comment":"The text says that removing the covariate encoder makes MSE 'decrease by 34%' and MAE 'decrease by 17%'; since removal harms performance, the direction should be 'increase' or 'degrade by,' and the numbers should be checked against the figure.","section":"IV-E6, Figure 6"},{"comment":"There is a typo: 'the we constructed four model model variants' should read 'we constructed four model variants.'","section":"IV-E4"},{"comment":"Both TSMixer and MLP-Mixer appear to be cited with the same reference number [33], and TSMixer is also cited as [47]; please disambiguate these references.","section":"II, References [33] and [47]"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful idea and a broad empirical study, but the verification is currently not solid enough for publication. The internal inconsistencies in Tables VII/VIII and Section I, and the limited LN/FFN ablation, are the main blockers. I would encourage the editor to request a revised version with corrected numbers, multi-seed experiments, and a full ablation, rather than reject, because the core direction appears plausible and the plug-and-play covariate encoder experiments (Table XII) are a genuinely useful result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution but the sell is oversold. The new bits—trend-sequence cross-patch attention and a plug-and-play contrastive module for future covariates—are worth a look. The efficiency gains are large and credible: roughly 66K parameters versus 6.9M for PatchTST, MACs down by 50x or more on large-channel datasets, and real edge-device runtimes. The weak data enriching module is also a useful generalization: it plugs into Informer, Transformer, and Autoformer and improves them on the Electri-Price dataset.\n\nThe soft spots are real. First, the Introduction says inference on the edge device takes 'less than 1%' of Transformer's time, but Table VII shows LiPFormer at roughly one-third of the time (0.55s vs 1.47s on ETTh1 L=96). That's a factor of three, not one hundred. Second, every run uses a single seed and no error bars; several headline comparisons are within 1-2% (e.g., Electricity L=96: 0.131 vs 0.129 for PatchTST; L=336 exact tie), so 'significantly outperforms' is not supported by the reported evidence. Third, the case for removing LayerNorm and FFNs rests on ablations on just ETTh1 and ETTm2, and on ETTm2 adding LN actually improves MSE at L=96 (0.158 vs 0.160) and L=336 (0.266 vs 0.273). The variants are not retuned, so the conclusion that these components are 'not effective' is shaky. Fourth, on Traffic, LiPFormer loses to PatchTST at every horizon, and the paper concedes performance 'marginally declines' on large-channel datasets; that is more than marginal on Traffic. Finally, given the lightweight claims, the absence of comparisons with linear attention mechanisms (Performer and the like) is a gap.\n\nThe dual-encoder pretraining uses ground truth only to align the target encoder during pretraining; at inference the covariate encoder sees only future covariates, so I do not see test-set leakage there.\n\nBottom line: the architecture is clever and the efficiency story is strong, but the accuracy claims are not yet solid. Anyone working on edge forecasting or on exogenous covariates will find the module worth studying. It deserves a serious referee—a good reviewer would ask for multi-seed runs, a broader retuned ablation of the LN/FFN deletion, and a corrected inference-time figure. I would accept it for review, not desk reject.","headline":"Genuinely novel lightweight design with a plug-in covariate module, but the accuracy claim is oversold and the inference-time numbers contradict the abstract.","tokens_in":25995,"tokens_out":6925,"would_cite":false,"duration_ms":60576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LiPFormer claims that a patch-wise Transformer can beat state-of-the-art forecasters after removing Layer Normalization and Feed Forward Networks, with contrastively trained weak future-context labels adding accuracy at low cost.","keywords":["time series forecasting","patch-wise Transformer","lightweight architecture","weak data enriching","contrastive learning","future covariates","edge deployment","long-term forecasting"],"falsifier":"Run the same add-back ablation on the seven datasets where it was not shown, especially the high-channel Electricity and Traffic sets: if adding Layer Normalization, Feed Forward Networks, or both does not degrade accuracy there, the claim that these components are ineffective for time series would not hold.","tokens_in":24977,"feed_emoji":"⚡","tokens_out":9127,"duration_ms":85991,"temperature":0.7,"pith_summary":"The paper tries to establish that accurate long-horizon time series forecasting does not need a heavyweight Transformer. It proposes LiPFormer, a patch-wise Transformer that removes Layer Normalization and Feed Forward Networks, drops positional encoding, and replaces them with a cross-patch attention mechanism plus two single-layer MLPs. A separate weak-data-enriching module pre-trains a pair of encoders with contrastive learning, aligning cheap future context (weather forecasts, holidays, time of day) with the future sequence to be predicted. On nine benchmark datasets the paper reports higher accuracy than current methods while using far fewer parameters, less training time, and less GPU memory, and on a CPU-only edge device it reports about one-third of the inference time of a classic Transformer. The underlying claim is that a lean architecture plus weak future supervision can outperform heavier models, which would make on-device forecasting practical.","feed_headline":"Lean patch Transformer plus cheap future labels beats heavy forecasters","feed_subtitle":"It removes heavy Transformer blocks and adds weak future-context supervision to cut cost and raise forecast accuracy.","key_machinery":"Cross-patch attention is the core mechanism: for a patch length $pl$, it builds $pl$ trend sequences by taking the value at a fixed position from each patch and ordering those values chronologically, then runs self-attention over these sequences together with inter-patch attention over patch embeddings. Because trend sequences span the whole input, they carry global order and trend information, which is what allows the model to drop positional encoding and Layer Normalization. The second load-bearing mechanism is the weak-data-enriching module, a dual-encoder contrastive framework in which one encoder embeds future covariates (numerical and textual, either explicit like weather forecasts or implicit like hour of day) and the other embeds the ground-truth target sequence; a symmetric cross-entropy loss over the $b \\times b$ pairwise similarity matrix aligns them. At prediction time the frozen covariate representation is mapped by a learnable linear layer and added to the base predictor output, so the forecast is a weighted combination of signal learned from history and signal learned from cheap future context.","core_discovery":"The central claim is that two heavy components of the Transformer, Layer Normalization and Feed Forward Networks, are not effective for numerical time series and can be removed without sacrificing accuracy. LiPFormer keeps a patch-wise multi-head self-attention backbone, but introduces cross-patch attention, which picks out the data point at a fixed position in every patch and arranges those points into global trend sequences; this captures order and trend information that previously required positional encoding. It also uses inter-patch attention over patch embeddings, and replaces FFNs with two linear single-layer MLPs, reducing parameter scale from $\\mathcal{O}(8 \\times hd^2)$ to $\\mathcal{O}(hd \\times pl)$. The weak-data-enriching module then pre-trains a covariate encoder and a target encoder with a symmetric contrastive loss over a $b \\times b$ similarity matrix, so that explicit or implicit future covariates become aligned with the target sequence; at inference the frozen covariate embedding is linearly mapped and added to the base prediction. The paper reports that this architecture reaches the best or second-best result in 64 of 72 multivariate settings and 26 of 32 univariate settings, while cutting training and inference time by roughly half relative to a leading patch-wise baseline and running within CPU-memory limits where a classic Transformer fails.","pith_inferences":["This reader would want to test the same add-back ablation on the high-channel datasets (Electricity and Traffic), since the paper itself notes that LiPFormer's accuracy margin narrows there; if LayerNorm or FFNs help on those data, the architecture's core premise is domain-dependent rather than universal.","A natural extension is to treat weak-data enriching as a general pretraining strategy for other sequence models, not just Transformers; the dual encoder only needs a covariate embedding and a target embedding of matching dimension, so RNN- or MLP-based forecasters could benefit too.","The edge-deployment numbers compare LiPFormer with a classic Transformer on CPU; a fairer test against other lightweight patch-wise forecasters on the same device would clarify how much of the speed gain comes from removing LN/FFN versus from the patch design itself.","One could isolate the value of weak labels by corrupting future covariates (for example, shuffling weather forecasts across days) and measuring how much accuracy drops, which would separate genuine apriori context from the pretraining objective alone."],"forward_implications":["On a CPU-only edge device, LiPFormer's inference time is reported at about one-third that of a classic Transformer, and it stays within memory where the standard model runs out at longer input lengths.","Removing Layer Normalization and Feed Forward Networks can improve accuracy on numerical series: adding them back, individually or together, degrades reported MSE and MAE in the ablations.","Weak future context can be synthesized: when no explicit covariates exist, encoding time-of-day, weekday, and month via the pretrained dual encoders improves forecasts on the ETT datasets.","The weak-data-enriching module is transportable: attaching it to other Transformer-based forecasters improved their MSE and MAE on the dataset with explicit future covariates.","Longer input histories continue to help, indicating that the cross-patch and inter-patch attention design captures long-range dependencies rather than relying on a fixed window."],"supporting_citations":[{"why":"Supplies the linear-baseline insight and the instance-normalization trick that motivate replacing nonlinear Transformer blocks with linear MLPs.","marker":"[30]"},{"why":"Defines the patch-wise tokenization scheme LiPFormer inherits and serves as the strongest patch-wise baseline to beat.","marker":"[31]"},{"why":"Provides the vanilla self-attention formulation and the quadratic-complexity baseline used for efficiency comparisons.","marker":"[23]"},{"why":"Provides the symmetric contrastive dual-encoder objective that the weak-data-enriching module adapts to future covariates.","marker":"[54]"},{"why":"Argues that Layer Normalization is of limited utility for time series, supporting its removal from the backbone.","marker":"[38]"},{"why":"Supplies the temporal feature encoding used to construct implicit weak labels when explicit future covariates are unavailable.","marker":"[27]"},{"why":"Supplies the mixer-style MLP feature-mixing idea behind the cross-patch attention design.","marker":"[47]"}],"fun_headline_variants":["Patch transformer drops LayerNorm and FFN for faster forecasts","LiPFormer: lean transformer with weak data beats heavy baselines","Edge-ready forecasting: transformer without heavy blocks wins","Slim patch transformer uses cheap context to outdo big models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design depends on the claim that Layer Normalization and Feed Forward Networks are not effective for numerical time series, yet the ablation that supports this claim was run on only two datasets (ETTh1 and ETTm2), leaving the other seven benchmarks untested.","fun_headline_variants_meta":{"raw":{"variants":["Patch transformer drops LayerNorm and FFN for faster forecasts","LiPFormer: lean transformer with weak data beats heavy baselines","Edge-ready forecasting: transformer without heavy blocks wins","Slim patch transformer uses cheap context to outdo big models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1346,"prompt_tokens":1087,"completion_tokens":259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":191}},"tokens_in":703,"tokens_out":259,"duration_ms":3836,"temperature":1.0,"reasoning_tokens":191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:18.661857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same add-back ablation on the seven datasets where it was not shown, especially the high-channel Electricity and Traffic sets: if adding Layer Normalization, Feed Forward Networks, or both does not degrade accuracy there, the claim that these components are ineffective for time series would not hold.","supporting_citations":[{"cited_title":"Are transformers effective for time series forecasting?","cited_arxiv_id":null,"evidence_quote":"Supplies the linear-baseline insight and the instance-normalization trick that motivate replacing nonlinear Transformer blocks with linear MLPs."},{"cited_title":"A time series is worth 64 words: Long-term forecasting with transformers,","cited_arxiv_id":null,"evidence_quote":"Defines the patch-wise tokenization scheme LiPFormer inherits and serves as the strongest patch-wise baseline to beat."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Provides the vanilla self-attention formulation and the quadratic-complexity baseline used for efficiency comparisons."},{"cited_title":"Learning transferable visual models from natural language supervi- sion,","cited_arxiv_id":null,"evidence_quote":"Provides the symmetric contrastive dual-encoder objective that the weak-data-enriching module adapts to future covariates."},{"cited_title":"A transformer-based framework for multivariate time series representation learning,","cited_arxiv_id":null,"evidence_quote":"Argues that Layer Normalization is of limited utility for time series, supporting its removal from the backbone."},{"cited_title":"Informer: Beyond efficient transformer for long sequence time-series forecasting,","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal feature encoding used to construct implicit weak labels when explicit future covariates are unavailable."}],"review_version":1}