REVIEW 3 major objections 6 minor 50 references
Enhancing Masked Time-Series Modeling via Dropping Patches
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that randomly dropping sub-sequence patches before masked reconstruction slows the collapse of Transformer representations to a rank-1 subspace, and that this mechanism improves time-series forecasts.
desk verdict A simple drop-then-mask pre-training trick with a broad empirical study, but the rank-collapse proof is invalid and the efficiency claims are overstated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the random patch-drop operation applied before mask-and-reconstruct, with positional encodings assigned before removal. The theoretical machinery is the rank-collapse residual bound inherited from the cited attention theory: with stacked self-attention layers, the representation matrix's deviation from the rank-1 all-rows-equal matrix shrinks at a rate controlled by a constant γ; the paper's Corollary 2 asserts that after uniformly dropping L−L′ rows, the attention matrix's row-variation terms rescale roughly by L/L′ and L′/L, making the bound on γ larger and therefore slowing the decay in the proof's reasoning. Because attention is quadratic in token count, dropping a fraction r reduces pre-training compute and memory by about (1−r)^2 while preserving sequence position information through pre-dropping positional encodings.
What would settle it
Train the same PatchTST backbone with and without DropPatch on a fixed dataset such as ETTm1, using the same visible-token budget (for PatchTST, raise the mask ratio to 0.78 to match DropPatch's 0.6 drop), and record the residual norm $\|X - \mathbf{1}x^\top\|$ after each encoder layer during pre-training. If the residual under DropPatch does not decay more slowly layer by layer than under PatchTST—or decays faster—then the rank-collapse mechanism asserted by Corollary 2 is contradicted even if forecast errors improve.
Extended reading notes
Core claim
The central discovery claim is that removing a random subset of input patches before masking—rather than masking more patches—fixes the dilemma that low mask ratios let the model reconstruct repetitive time series without understanding them, while high mask ratios scatter attention. DropPatch randomly drops a fixed ratio r (0.6 in the main experiments) of patches before patch-level masking; positional encodings are computed before dropping so sequence positions are preserved. In experiments, this simple change surpasses PatchTST across in-domain, cross-domain, few-shot, and cold-start benchmarks, with pre-training time and memory reduced by roughly a factor of 2 on the datasets reported. Mechanism studies show attention heads become more focused (larger KL divergence from uniform), more diverse across heads, and less redundant. The proof attempt connects the drop operation to the rank-collapse theory of Transformers, aiming to show that the constant controlling the residual contraction grows after row dropping, so representations degenerate to a rank-1 matrix more slowly.
Load-bearing premise
The proof needs the pre-softmax attention scores to be a small row-wise perturbation of a row constant (S_{ij} = µ_i + δ_{ij} with tiny δ_{ij}), and it needs the inference that a larger upper-bound constant γ means the true residual shrinks more slowly; if either fails, the rank-collapse-slowing explanation loses its support.
Editorial extensions
If this is right
- If DropPatch works as claimed, masked time-series pre-training can be made faster and lighter simply by dropping a fixed fraction of patches before masking, without changing the architecture or loss.
- The reported gains in cold-start and few-shot settings imply that dropped-patch pretraining yields representations that transfer to target datasets with little data and short lookback windows.
- The attention analyses imply that fewer, sharper attention patterns are learned, reducing redundancy across heads and scales.
- Because only surviving patches are processed, the method scales to longer sequences or larger batches under the same GPU budget, which the paper frames as a step toward time-series foundation models.
- The proof's corollary, if valid, extends the rank-collapse mitigation beyond time series to any Transformer representation learning that uses row dropping.
Reading between the lines
- A testable extension the paper leaves open is whether the benefit comes from the stochasticity of random selection or simply from the shorter sequence; training with a fixed, learned, or clustered subset of patch positions would separate the two.
- The same rank-collapse argument would predict that DropPatch helps other masked-modeling settings, such as masked image or speech modeling; measuring residual-rank decay there would test the generality claimed in the paper's impact section.
- The drop ratio interacts with mask ratio, so the effective visible/masked balance changes; a principled account of the optimal (r, m) trade-off could remove the need to tune drop ratio per dataset.
- Efficiency gains of roughly (1−r)^2 in attention cost suggest DropPatch could be used as a routine pre-training accelerator even where forecasting accuracy is unchanged.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DropPatch, an enhancement to masked time-series pre-training (built on PatchTST) that randomly drops a fixed fraction of sub-sequence patches before the masking and reconstruction step. The authors claim two advantages: improved pre-training efficiency (fewer tokens attended to) and better forecasting performance across in-domain, cross-domain, few-shot, and cold-start settings. They also claim a theoretical result: randomly dropping patches slows the rate at which Transformer representations collapse to a rank-1 subspace, based on an analysis in the final section. The empirical evaluation compares DropPatch against several self-supervised and supervised baselines on 12 datasets plus two synthesized multi-dataset collections, reporting MSE/MAE, efficiency measurements, and several attention/representation analyses.
Significance. If the empirical results hold, the contribution is practically useful: DropPatch is a simple, drop-in pre-training modification that often improves over PatchTST and reduces pre-training cost, with code released and standard deviations reported. The evaluation is broad (in-domain, cross-domain, few-shot, cold start, synthesized foundation-model-style pretraining) and the paper is careful to fix hyperparameters across settings. However, the theoretical claim advertised in the abstract and conclusion is not established by the proof in Corollary 2: the argument compares upper bounds on residual contraction rather than actual contraction rates, so it does not demonstrate that DropPatch slows rank collapse. The empirical findings are the strength of the paper; the theoretical section requires either a rigorous replacement or an explicit downgrade to a conjecture.
major comments (3)
- [Theoretical Analysis, Corollary 2, Eqs. (28)-(30)] The final inference of Corollary 2 is invalid. The proof shows that the lower bound on the constant γ increases after row dropping, and then concludes that because γ′ > γ, the upper bound on r_{L+1}/r_L increases, "causing the residual to shrink more slowly layer by layer." Lemma 2 provides only an upper-bound recurrence, r_{L+1} ≤ (4γβ/√d) r_L^3. A larger γ loosens this upper bound; it does not constrain the actual residual, which could decay at the same rate or even faster. To prove that DropPatch slows rank collapse, one would need a lower bound on the actual contraction factor of the dropped model, or a direct comparison of the actual residuals under the two models, not a comparison of upper bounds. This gap directly undermines the abstract's statement that "it is proved that DropPatch slows down the rate...", and it is the load-bearing step of the theoretical contribution.
- [Theoretical Analysis, Corollary 2, Eqs. (20)-(28)] The derivation of the scaling γ′ ≥ γ · (L/L′) sqrt(L/L′) is not rigorous. In Eqs. (24)-(26), the quantity max_{j,j′} Σ_i |A_{ij} − A_{ij′}| is treated as if it simply scales by L′/L after row subsampling, but the maximizing pair (j,j′) for the subsampled matrix need not coincide with the maximizing pair of the original matrix, and the row-sum decomposition used in Eqs. (24)-(25) is not justified for a fixed column pair after arbitrary row deletion. Furthermore, Eq. (28) combines the row-wise factor L/L′ with the column-wise factor L′/L as if the numerator and denominator in the definition of γ in Eq. (27) were independent; the joint maximization over i and j,j′ makes such a product formula unsubstantiated. This step requires a fully specified derivation before the claimed lower bound on γ′ can be accepted.
- [Theoretical Analysis, Corollary 2, after Eq. (22)] The proof invokes concentration inequalities with only the condition "if Δ_i are bounded and weakly dependent," without stating the required tail bound, the dependence of the high-probability statement on L, L′, and ε, or the control of the O(ε²) approximation terms under the probabilistic claim. In addition, the assumption S_{ij} = μ_i + δ_{ij} with Σ_j δ_{ij} = 0 and |δ_{ij}| ≤ ε for small ε is stated as a generic premise, but no justification is given for why it holds for time-series patch representations or learned attention weights. As written, the "with high probability" conclusion is not a theorem and the scope of the theoretical result is unclear.
minor comments (6)
- [Throughout] There are several typographical errors: "mesure" (Method section), "dopping" (Conclusion), "time-seires" (Conclusion), "efficienty" (Main Results), and "out theoretical analysis" (Conclusion).
- [Table 1 and accompanying text] The claim that DropPatch "surpasses existing methods in 13 out of 14 metrics" overstates the result: on the averaged metrics, DropPatch ties PatchTST on ECL MSE (0.157 vs. 0.157), and on individual horizons (e.g., ETTm2/96 and ETTm2/192 in Table 10) PatchTST is slightly better. The sentence should specify that the count refers to averaged metrics, or should be reworded to reflect the tie.
- [Abstract and Introduction] The phrase "square-level advantage" for pre-training efficiency is not supported by the measured numbers in Table 7, where the speedup in time per epoch is between roughly 1.4× and 1.7×. The quadratic reduction applies only to the attention-cost component, not to the end-to-end training time, and the claim as stated is misleading.
- [Method, Eqs. (1)-(2)] The description "previously dropped positional encodings are added back to these embeddings" is ambiguous: after random dropping, the dropped patches are absent, so it is unclear whether positional encodings of the dropped positions are added to zero tensors or whether the remaining patches retain their original positional encodings. This should be clarified in the text.
- [Figure 2] Figure 2 is dense, and the two steps "Random Dropping" and "Random Masking" are visually similar; annotating the sequence with explicit step numbers or distinct colors would improve readability.
- [Cold Start, text after Table 6] The text says "we present the averaged results across the target datasets," but Table 6 reports per-dataset rows; the sentence should read "we present results for each target dataset" or the table should include an additional averaged row.
Circularity Check
No significant circularity; the theoretical gap in Corollary 2 is a soundness issue, not an input-output equivalence, and the empirical claims rest on external benchmarks.
full rationale
I found no step in which a claimed prediction or theorem reduces by construction to an input or to a self-citation. The main theoretical claim (Corollary 2) does not fit any free parameter to the target conclusion; it attempts a derivation from an explicit small-perturbation assumption on pre-softmax scores and from the rank-collapse recurrence imported from Dong, Cordonnier, and Loukas (2021), an external source. The problematic step is the final inference that a larger gamma-prime makes the upper bound on r_{L+1}/r_L looser and therefore that the actual residual shrinks more slowly; a loosened upper bound does not constrain the actual decay, so the proof fails as a soundness matter. That is a logical gap, not an equivalence between premise and conclusion, and it does not meet the threshold for circularity. The empirical claims are tested against public datasets and previously reported baseline numbers, with drop and mask ratios fixed across tasks; no fitted parameter is relabeled as a prediction. References to rank collapse and attention entropy collapse are external, and no load-bearing self-citation appears. Hence the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- drop_ratio_r =
0.6
- mask_ratio_m =
0.4
assumptions (3)
- standard math The rank-collapse bound of Dong et al. (2021): for stacked self-attention layers, ∥res(SAN(X))∥_{1,∞} ≤ C∥res(X)∥^3_{1,∞} with C = 4γβ/√d (Eq. 4-5).
- ad hoc to paper The attention score decomposition S_{ij} = μ_i + δ_{ij} with Σ_j δ_{ij}=0 and |δ_{ij}| ≤ ε for small ε (Corollary 2 proof).
- ad hoc to paper Concentration inequalities apply to the random row selection so that expectations hold with high probability, without stated conditions (Corollary 2 proof, after Eq. 22).
Cite this review
Pith. "Pith review of Enhancing Masked Time-Series Modeling via Dropping Patches." pith.science (2026). https://pith.science/paper/PKG23Q6W
@misc{pith2026241215315,
author = {Pith},
title = {Pith review of: Enhancing Masked Time-Series Modeling via Dropping Patches},
year = {2026},
howpublished = {\url{https://pith.science/paper/PKG23Q6W}},
note = {Machine review of arXiv:2412.15315}
}
read the original abstract
This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by a square-level advantage; 2) It provides additional advantages for modeling in scenarios such as in-domain, cross-domain, few-shot learning and cold start. This paper conducts comprehensive experiments to verify the effectiveness of the method and analyze its internal mechanism. Empirically, DropPatch strengthens the attention mechanism, reduces information redundancy and serves as an efficient means of data augmentation. Theoretically, it is proved that DropPatch slows down the rate at which the Transformer representations collapse into the rank-1 linear subspace by randomly dropping patches, thus optimizing the quality of the learned representations
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al
Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[2]
Amos, I.; Berant, J.; and Gupta, A. 2023. Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors. arXiv preprint arXiv:2310.02980
arXiv 2023
-
[3]
Baevski, A.; Hsu, W.-N.; Xu, Q.; Babu, A.; Gu, J.; and Auli, M. 2022. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, 1298--1312. PMLR
work page 2022
-
[4]
Bao, H.; Dong, L.; Piao, S.; and Wei, F. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254
arXiv 2021
-
[5]
D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al
Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901
2020
-
[6]
Candanedo, L. 2017. Appliances Energy Prediction . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5VC8G
doi:10.24432/c5vc8g 2017
-
[7]
M.; Feldheim, V.; and Deramaix, D
Candanedo, L. M.; Feldheim, V.; and Deramaix, D. 2017. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings, 140: 81--97
2017
-
[8]
Chen, J.; Agarwal, A.; Abdelkarim, S.; Zhu, D.; and Elhoseiny, M. 2022. Reltransformer: A transformer-based long-tail visual relationship recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19507--19517
work page 2022
Show all 50 references
-
[9]
Cheng, M.; Liu, Q.; Liu, Z.; Zhang, H.; Zhang, R.; and Chen, E. 2023. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv preprint arXiv:2303.00320
2023
-
[10]
Choromanski, K.; Likhosherstov, V.; Dohan, D.; Song, X.; Gane, A.; Sarlos, T.; Hawkins, P.; Davis, J.; Mohiuddin, A.; Kaiser, L.; et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794
2020 arXiv
-
[11]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
2018 arXiv
-
[12]
Dong, J.; Wu, H.; Zhang, H.; Zhang, L.; Wang, J.; and Long, M. 2024. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36
2024
-
[13]
Dong, Y.; Cordonnier, J.-B.; and Loukas, A. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, 2793--2803. PMLR
2021
-
[14]
I.; Hyndman, R
Godahewa, R.; Bergmeir, C.; Webb, G. I.; Hyndman, R. J.; and Montero-Manso, P. 2021. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643
2021 arXiv
-
[15]
Goswami, M.; Szafer, K.; Choudhry, A.; Cai, Y.; Li, S.; and Dubrawski, A. 2024. MOMENT: A Family of Open Time-series Foundation Models. arXiv preprint arXiv:2402.03885
2024 arXiv
-
[16]
He, K.; Chen, X.; Xie, S.; Li, Y.; Doll \'a r, P.; and Girshick, R. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16000--16009
2022
-
[17]
He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729--9738
2020
-
[18]
Hendrycks, D.; and Gimpel, K. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415
2016 arXiv
-
[19]
Hogue, J. 2019. Metro Interstate Traffic Volume . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5X60B
2019 doi
-
[20]
Jin, X.; Park, Y.; Maddix, D.; Wang, H.; and Wang, Y. 2022. Domain adaptation for time series forecasting via attention sharing. In International Conference on Machine Learning, 10280--10297. PMLR
2022
-
[21]
Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019. Similarity of neural network representations revisited. In International conference on machine learning, 3519--3529. PMLR
2019
-
[22]
Liang, Y.; Wen, H.; Nie, Y.; Jiang, Y.; Jin, M.; Song, D.; Pan, S.; and Wen, Q. 2024. Foundation Models for Time Series Analysis: A Tutorial and Survey. arXiv preprint arXiv:2403.14735
2024 arXiv
-
[23]
Liu, M.; Zeng, A.; Chen, M.; Xu, Z.; Lai, Q.; Ma, L.; and Xu, Q. 2022. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35: 5816--5828
2022
-
[24]
Liu, Y.; Hu, T.; Zhang, H.; Wu, H.; Wang, S.; Ma, L.; and Long, M. 2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625
2023 arXiv
-
[25]
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
2019 arXiv
-
[26]
Liu, Y.; Zhang, H.; Li, C.; Huang, X.; Wang, J.; and Long, M. 2024. Timer: Transformers for Time Series Analysis at Scale. arXiv preprint arXiv:2402.02368
2024 arXiv
-
[27]
H.; Sinthong, P.; and Kalagnanam, J
Nie, Y.; Nguyen, N. H.; Sinthong, P.; and Kalagnanam, J. 2022. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730
2022 arXiv
-
[28]
P.; and Lucchi, A
Noci, L.; Anagnostidis, S.; Biggio, L.; Orvieto, A.; Singh, S. P.; and Lucchi, A. 2022. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35: 27198--27211
2022
-
[29]
Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32
2019
-
[30]
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9
2019
-
[31]
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67
2020
-
[32]
Salam, A.; and El Hibaoui, A. 2023. Power Consumption of Tetouan City . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5B034
2023 doi
-
[33]
N.; Kaiser, .; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[34]
Vito, S. 2016. Air Quality . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C59K5F
2016 doi
-
[35]
Vyas, A.; Katharopoulos, A.; and Fleuret, F. 2020. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33: 21665--21674
2020
-
[36]
Wang, Z.; Xu, X.; Zhang, W.; Trajcevski, G.; Zhong, T.; and Zhou, F. 2022. Learning latent seasonal-trend representations for time series forecasting. Advances in Neural Information Processing Systems, 35: 38775--38787
2022
-
[37]
Woo, G.; Liu, C.; Kumar, A.; Xiong, C.; Savarese, S.; and Sahoo, D. 2024. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592
2024 arXiv
-
[38]
Woo, G.; Liu, C.; Sahoo, D.; Kumar, A.; and Hoi, S. 2022. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575
2022 arXiv
-
[39]
Wu, H.; Xu, J.; Wang, J.; and Long, M. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34: 22419--22430
2021
-
[40]
Xie, Z.; Geng, Z.; Hu, J.; Zhang, Z.; Hu, H.; and Cao, Y. 2023. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14475--14485
2023
-
[41]
Xie, Z.; Zhang, Z.; Cao, Y.; Lin, Y.; Bao, J.; Yao, Z.; Dai, Q.; and Hu, H. 2022. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9653--9663
2022
-
[42]
Yue, Z.; Wang, Y.; Duan, J.; Yang, T.; Huang, C.; Tong, Y.; and Xu, B. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8980--8987
2022
-
[43]
Zeng, A.; Chen, M.; Zhang, L.; and Xu, Q. 2023. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, 11121--11128
2023
-
[44]
Zerveas, G.; Jayaraman, S.; Patel, D.; Bhamidipaty, A.; and Eickhoff, C. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2114--2124
2021
-
[45]
Zhai, S.; Likhomanenko, T.; Littwin, E.; Busbridge, D.; Ramapuram, J.; Zhang, Y.; Gu, J.; and Susskind, J. M. 2023. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, 40770--40803. PMLR
2023
-
[46]
Zhang, Q.; Wang, Y.; and Wang, Y. 2022. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 35: 27127--27139
2022
-
[47]
Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; and Zhang, W. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 11106--11115
2021
-
[48]
Zhou, T.; Ma, Z.; Wen, Q.; Wang, X.; Sun, L.; and Jin, R. 2022. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, 27268--27286. PMLR
2022
-
[49]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[50]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.