Pith. sign in

REVIEW 3 major objections 7 minor 14 references

Beyond Magnitude and Shape: A Direction-Aware Loss for Time Series Forecasting

T0 review · 3 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MSE-trained forecasters systematically mispredict the direction of small moves; adding a scale-invariant cosine loss on difference vectors—CosDir—consistently improves directional accuracy across 13 datasets and 15 backbones while preservin

desk verdict A genuinely useful plug-in directional loss with rock-solid empirical breadth, but the abstract's small-move story is contradicted by the paper's own Figure 4. read the letter →

arxiv 2608.01857 v1 pith:O237DI76 submitted 2026-08-03 cs.LG cs.AI

Seunghan Lee , Jaehoon Lee , Jun Seo , Junhyeok Kang , Sangjun Han , Sungdong Yoo , Minjae Kim , Tae Yoon Lim
show 4 more authors
Dongwan Kang Hwanil Choi Soonyoung Lee Wonbin Ahn
This is my paper
classification cs.LGcs.AI
keywords timeseriesforecastingdirectionalaccuracycosinesimilaritylossscale-invariantuncertaintyweightingfunctiondeeplearningdirectionofchange
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that standard magnitude losses such as MSE are blind to whether a forecast moves up or down, especially on small moves, because their gradients vanish as move size shrinks. The authors propose CosDir, a plug-in loss term that computes the cosine similarity between the predicted and true step-to-step difference vectors over the forecast horizon, penalizing angular disagreement regardless of scale. Across more than 100K experiments on 13 datasets and 15 forecasting backbones, adding CosDir to MSE raises directional accuracy (DA) by about one to two percentage points on average without degrading MSE, and an uncertainty-weighted variant CosDir-UW learns the direction–magnitude mixing ratio per dataset. If correct, the work identifies a neglected axis of forecasting quality—direction of change—and offers a simple, architecture-agnostic way to train for it.

What carries the argument

The central object is the horizon difference vector: for each channel, the H-length vector of step-to-step changes Δŷ and Δy, anchored at the last observed value. CosDir adds the penalty 1 − cos(Δŷ, Δy) to the base loss (e.g., MSE), averaged over channels. Because cosine similarity depends only on orientation, the penalty is invariant to rescaling the whole window; its gradient carries a 1/‖Δŷ‖ prefactor, so small-move windows receive a directional update rather than a vanishing one. CosDir-UW replaces the fixed λ with two learnable log-variances and regularizer (s1+s2)/2, making the effective directional weight λ_eff = e^{s1-s2} adapt to the dataset.

What would settle it

On a fixed benchmark, train the same 15 backbones with MSE and MSE+CosDir and compute DA within the decile of smallest true |Δy| steps: if CosDir does not improve (or worsens) DA in that decile relative to MSE, the claimed mechanism—that scale invariance re-injects directional signal where MSE's gradient vanishes—is contradicted. A second check would compare CosDir against a direct sign-classification auxiliary on a synthetic series with balanced up/down labels: if the direct sign term beats CosDir on DA, the cosine surrogate is not the tight direction surrogate the paper claims.

Watch

Extended reading notes

Core claim

Adding a scale-invariant cosine term between the predicted and true step-to-step difference vectors raises directional accuracy across 15 backbones and 13 datasets, with gains concentrated on large, decision-relevant moves. The paper argues that MSE-trained models neglect direction because the squared-error gradient is dominated by large-amplitude points, leaving small moves unlearned. CosDir's normalized cosine term keeps a directional gradient at every scale without altering the magnitude objective. Its uncertainty-weighted variant, CosDir-UW, learns the mixing ratio per dataset through two log-variances, yielding an effective weight λ_eff = e^{s1-s2} that tracks a tuned fixed λ.

Load-bearing premise

The load-bearing premise is that cosine similarity between the horizon difference vectors is a valid and sufficiently tight surrogate for the per-step sign-agreement metric DA; the paper does not quantify the gap, and a prediction can score high cosine alignment while still getting small steps' signs wrong.

Editorial extensions

If this is right

  • Adding CosDir to an MSE base improves DA on 93% of (backbone, dataset) cells across 13 datasets and 15 backbones, with average DA rising from 55.60% to 57.10% (CosDir) and 57.32% (CosDir-UW).
  • Directional gains concentrate on the largest true moves—the ones that drive decisions—and a direction-following strategy based on CosDir accumulates more payoff than one based on MSE, even after transaction costs.
  • CosDir-UW's learned balance is a dataset property: 96% of the variance in the learned log-weight lies between datasets, and its median tracks a per-dataset tuned fixed λ (Spearman ρ = 0.60), so no hyperparameter sweep is needed.
  • CosDir does not trade off magnitude for direction; on financial datasets it keeps MSE at or slightly below the MSE-only baseline, whereas shape- and frequency-focused auxiliary losses degrade MSE.
  • CosDir is complementary to existing structure-aware losses: combining it with a frequency-domain loss (FreDF) gives the best DA among the combinations tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A straightforward but untested consequence is that the same cosine-on-differences term could regularize probabilistic forecasting, where one would align sampled or quantile trajectories rather than point forecasts; the paper lists probabilistic extension as future work.
  • Because CosDir's gradient equalizes across windows by a 1/‖Δŷ‖ factor, it may also act as a helpful regularizer for trend-change detection in anomaly or regime-shift settings, where sign of change is the label.
  • The paper optimizes a continuous cosine surrogate for a discrete sign-agreement metric; a direct synthetic test with known sign labels would show how tight the surrogate is, since high cosine alignment can coexist with sign errors on small steps.
  • CosDir-UW's λ_eff could be repurposed as a measure of directional predictability of a dataset, potentially informing when direction-aware training is worthwhile.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes CosDir, an auxiliary loss that adds a scale-invariant cosine penalty between the horizon difference vectors of the prediction and the target to a base MSE loss, and CosDir-UW, an extension that learns the direction–magnitude mixing ratio via uncertainty weighting. The authors claim that MSE-trained forecasters systematically mispredict the direction of small moves, that CosDir re-injects directional gradient where MSE is blind, and that the method improves directional accuracy (DA) across 13 datasets and 15 backbones while preserving magnitude accuracy. The paper reports over 100K experiments, significance tests, per-seed standard deviations, comparisons with several alternative losses, ablations, and a payoff analysis.

Significance. If the reported gains are reliable, the paper offers a simple, backbone-agnostic plug-in loss for improving directional accuracy in time series forecasting, supported by an unusually broad empirical study: 13 datasets, 15 backbones, up to six horizons, multiple random seeds, and comparisons with shape, frequency, and directional losses. The per-seed standard deviations and paired Wilcoxon tests strengthen the empirical core, and the release of code is a positive feature. However, the paper's stated failure mode—small moves—is not supported by its own Figure 4, and the ablation does not cleanly isolate scale invariance as the source of the gains. These issues affect the central narrative and the mechanistic explanation, though the core DA improvement appears defensible.

major comments (3)
  1. [Abstract; Introduction; Analysis 'Where the gains come from' (Fig. 4)] The central motivation is that MSE-trained forecasters fail on the direction of small moves and that CosDir's scale invariance re-injects learning signal exactly there. However, Figure 4(a) shows that both MSE and MSE+CosDir are near chance on the smallest-move decile, and CosDir's advantage grows monotonically with move size. The text explicitly states: 'CosDir does not manufacture signal on the near-random smallest moves.' This is an internal inconsistency. Either provide additional evidence that small-move DA improves (e.g., decile analysis aggregated across datasets/backbones), or revise the abstract, introduction, and conclusion to state that the gains concentrate on larger, decision-relevant moves. As written, the paper's stated mechanism and its primary evidence disagree.
  2. [Eq. (2) vs. Eq. (1)] CosDir optimizes cosine similarity of the entire horizon difference vectors, while DA is a per-step sign agreement. The paper offers no bound or empirical analysis relating these two quantities. A forecast can achieve high cosine alignment while still having sign errors on small steps, so optimizing Eq. (2) need not improve Eq. (1). This surrogate gap is especially consequential given Figure 4(a) shows no DA improvement on small moves. Please add a formal or empirical analysis of the surrogate gap (e.g., a per-step decomposition of the gradient, or a synthetic example where the cosine surrogate and DA diverge), or substantially weaken the claim that CosDir is a tight surrogate for directional accuracy.
  3. [Table 7; Appendix E, Eq. (7)] The ablation is intended to show that scale invariance is the source of CosDir's gains. Yet the sign-classification baseline (BCE, Eq. (7)) is not scale-invariant in the sense emphasized by the paper, but it nearly matches CosDir: average DA 57.76 vs. 57.98 (Table 7). The claim that 'the gain comes from scale invariance rather than from adding a directional term' is therefore not supported. The paper should either include a scale-invariant sign surrogate (e.g., BCE applied to the normalized predicted difference) or reinterpret the ablation: adding any directional term helps, and scale invariance provides only a small additional gain.
minor comments (7)
  1. [Eq. (2)] The stabilizer ε is introduced but its value and sensitivity are never reported. Please state the value used and, if possible, include a sensitivity check or a sentence on why it is benign.
  2. [Eq. (3); 'Hyperparameter-free' claims] The claim that CosDir-UW exposes 'no loss hyperparameter' is overstated: the regularization coefficient 1/2 in Eq. (3), the initial values of s1 and s2, and the same ε remain modeling choices that can affect the learned ratio. Please qualify the claim.
  3. [Figure 6(b); 'From a fixed to an adaptive mixing ratio'] Spearman ρ=0.60 is a moderate correlation, not a tight 'match.' The text says the learned ratio 'matches' or 'tracks' the per-dataset tuned λ; please soften this to 'correlates moderately' or provide a quantitative equivalence test.
  4. [Appendix J] The statement that p<10^-3 'stays significant under a Bonferroni correction over the cells reported in each table' needs the number of tests and the resulting corrected threshold. Please report them.
  5. [Table 5 caption] The notation 'MSE/(MSE+CosDir)/(MSE+CosDir-UW)' is easy to misread. Consider adding a sample row or clarifying in the caption that each cell contains three values in that order.
  6. [Table 41] The Traffic H=720 cell for TiDE is reported as '–'. State explicitly why this cell is missing (e.g., the model did not produce output for this configuration).
  7. [Experimental protocol] The claim of 'over 100K experiments' should be broken down (main results, ablations, hyperparameter sweeps) so readers can verify the count and understand the computational scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CosDir is an auxiliary training objective evaluated with an independent metric; no fitted parameter is promoted to a prediction.

full rationale

The paper's chain is an empirical proposal rather than a derivation that reduces to its inputs. CosDir (Eq. 2) augments MSE with a cosine penalty on first-difference vectors, while DA (Eq. 1) is a separate per-step sign-agreement metric. The paper never claims that minimizing the cosine surrogate is equivalent to maximizing DA; it empirically reports that adding the surrogate improves DA on test splits. The mixing weight lambda is fixed at 0.5 in the main experiments ('Unless noted, lambda=0.5'), and CosDir-UW learns s1,s2 via uncertainty weighting (Eq. 3), not by fitting DA; the comparison of learned lambda_eff to a tuned lambda (Fig. 6, App. H) is an empirical check, not a fitted-input-called-prediction. No load-bearing self-citation appears: related losses (TILDE-Q, MADL, FreDF, Soft-DTW) are external, and no uniqueness theorem or prior ansatz by the present authors is invoked. The internal tension between the abstract's 'small moves' motivation and Figure 4's finding that gains concentrate on larger moves is a consistency/mechanism concern, not circularity: the reported DA improvements are measured on held-out data against an independent metric. The paper even explicitly concedes the small-move limitation, which further separates the stated mechanism from the demonstrated outcome. No step in the derivation is definitionally equivalent to its target metric or to a self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new theoretical entities. It relies on the standard math of cosine similarity and on the domain assumption that sign agreement (DA) is the target. The main free parameter is the mixing weight lambda, which the paper itself shows is dataset-dependent; CosDir-UW learns it from data, which reduces but does not eliminate the modeling choice.

free parameters (3)
  • lambda (direction-magnitude weight) for CosDir = 0.5 (default), tuned per dataset in Appendix H from 0.3 to 3.0
    Chosen by hand as a default; the paper shows the best value varies by dataset, motivating CosDir-UW.
  • epsilon stabilizer in Eq. (2) = not given
    Small constant to avoid division by zero; hand-chosen.
  • k (scaling) in ablation variants = 10 for magnitude-weighted sign, 5 for BCE
    Hand-chosen constants in the ablation study; not part of the main method.
assumptions (3)
  • standard math Cosine similarity is differentiable almost everywhere and a valid smooth surrogate for direction
    Used in Eq. (2) and Property 2; the paper relies on it for end-to-end training.
  • domain assumption Direction of change is decision-relevant and DA is an appropriate metric
    Stated in the Introduction and used to justify the loss.
  • domain assumption The first-difference operator with y_0 = last observed value is the right notion of step change
    Defined in Preliminaries; it determines what 'direction' means.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Magnitude and Shape: A Direction-Aware Loss for Time Series Forecasting." pith.science (2026). https://pith.science/paper/O237DI76

@misc{pith2026260801857,
  author       = {Pith},
  title        = {Pith review of: Beyond Magnitude and Shape: A Direction-Aware Loss for Time Series Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O237DI76}},
  note         = {Machine review of arXiv:2608.01857}
}
read the original abstract

The direction of change --- whether a series will move up or down --- is often as important as its exact value in decisiondriven applications such as risk management and financial forecasting. However, most forecasting losses optimize either point magnitude or shape and frequency structure, and none explicitly targets the direction of change. In this paper, we find that MSE-trained forecasters fail on the direction of small moves. To address this, we propose CosDir, a simple yet effective direction-aware loss that aligns the difference vectors of the prediction and the target via cosine similarity. Being scale-invariant, CosDir keeps a directional gradient on small moves, re-injecting learning signal exactly where MSE neglects it. CosDir is a lightweight, plug-in term that attaches to any backbone without architectural modification. Since the best ratio for mixing the directional and magnitude terms differs across datasets, we further propose CosDir-UW, an extension that makes this ratio adaptive by learning it during training, matching a per-dataset tuned weight with no hyperparameter. We conduct over 100K experiments, demonstrating that our method consistently and significantly improves directional accuracy while preserving magnitude accuracy, and that it outperforms various loss functions. Code is available at: https://github.com/seunghan96/cosdir.

Figures

Figures reproduced from arXiv: 2608.01857 by the authors.

Figure 1
Figure 1. Same MSE, different decision values. Two forecasts share the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of CosDir. (a) For each channel we take the step-to-step changes of the forecast and the target, which stack [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. CosDir vs. CosDir-UW. CosDir mixes direction [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Directional gain grows with move size and turns into payoff. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Direction without sacrificing magnitude. On finan [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: CosDir-UW learns a dataset property. (a) The learned balance λeff = e s1−s2 clusters tightly by dataset across the eight main backbones, with 96% of its variance lying between datasets. (b) The dataset-level median λeff tracks the tuned λ. λ RealVar RealVol RealVol60 S…
Figure 7
Figure 7. Figure 7: Robustness across backbones and datasets. For each [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 4 linked inside Pith

  1. [1]

    Fastdifferentiablesortingandranking

    Blondel,M.;Teboul,O.;Berthet,Q.;andDjolonga,J.2020. Fastdifferentiablesortingandranking. InInternationalCon- ference on Machine Learning (ICML). Chen, S.-A.; Li, C.-L.; Yoder, N.; Arik, S. O.; and Pfister, T

    2020

  2. [5]

    arXiv preprint arXiv:2305.10721

    Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721. Lin, S.; Lin, W.; Wu, W.; Zhao, F.; Mo, R.; and Zhang, H

    arXiv

  3. [6]

    Liu, S.; Yu, H.; Liao, C.; Li, J.; Lin, W.; Liu, A

    SegRNN: Segment recurrent neural network for long-term time series forecasting.arXiv preprint arXiv:2308.11200. Liu, S.; Yu, H.; Liao, C.; Li, J.; Lin, W.; Liu, A. X.; and Dustdar, S. 2022a. Pyraformer: Low-complexity pyramidal attentionforlong-rangetimeseriesmodelingandforecasting. InInternational Conference on Learning Representations (ICLR). Liu, Y.; H...

    arXiv

  4. [7]

    The per-seed standarddeviationsofthemainDAresultsaresmall,sothey areomittedfromthehorizon-resolvedTable5forreadability. Loss DA (↑) MSE0.5642 ±0.0006 +FreDF0.5729 ±0.0006 +TILDE-Q0.5642 ±0.0004 +DBLoss0.5618 ±0.0006 +MADL0.5629 ±0.0003 +CosDir (Ours)0.5798 ±0.0009 +CosDir-UW (Ours)0.5829 ±0.0007 Table 16: CosDir vs. various loss functions. Loss DA (↑) MSE...

    arXiv

  5. [11]

    InInternational Conference on Learning Representations (ICLR)

    Crossformer: Transformer uti- lizing cross-dimension dependency for multivariate time se- ries forecasting. InInternational Conference on Learning Representations (ICLR). Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; andZhang,W.2021. Informer:Beyondefficienttransformer forlongsequencetime-seriesforecasting. InProceedingsof the AAAI Conferen...

    2021

  6. [12]

    It improves DA over the baseline but, unlike CosDir, it op- timizes a hard up/down label that ignores how strongly the trajectories agree (Table 7)

    This is the natural sign-classification auxiliary. It improves DA over the baseline but, unlike CosDir, it op- timizes a hard up/down label that ignores how strongly the trajectories agree (Table 7). All three inject directional information in a non-scale- invariant way, unlike CosDir’s scale-invariant cosine, so the gap to CosDir isolates the effect of s...

    arXiv 1920

  7. [13]

    2023), it reports the change in DA and MSE of each auxiliary loss relative to the base loss

    On the five financial datasets with FreTS (Yi et al. 2023), it reports the change in DA and MSE of each auxiliary loss relative to the base loss. CosDir and CosDir-UW raise DA the most while keeping MSE at or below the baseline, whereas the shape and frequency losses buy little direction and clearly worsen the very MSE they were meant to preserve. Auxilia...

    arXiv 2023

  8. [1970]

    Jiang,W.2021.Applicationsofdeeplearninginstockmarket prediction: Recent progress.Expert Systems with Applica- tions, 184: 115537

    Efficient capital markets: A review of theory and empirical work.The Journal of Finance, 25(2): 383–417. Jiang,W.2021.Applicationsofdeeplearninginstockmarket prediction: Recent progress.Expert Systems with Applica- tions, 184: 115537. Kendall,A.;Gal,Y.;andCipolla,R.2018. Multi-tasklearn- ing using uncertainty to weigh losses for scene geometry and semanti...

    2021

Show all 14 references
  1. [2017]

    InAdvances in Neural Information Processing Systems (NeurIPS)

    At- tention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS). Wang, H.; Pan, L.; Chen, Z.; Yang, D.; Zhang, S.; Yang, Y.; Liu,X.;Li,H.;andTao,D.2025.FreDF:Learningtoforecast in the frequency domain. InInternational Conference on Learning Represent...

    2025

  2. [2019]

    LeGuen,V.;andThome,N.2019

    Short-term residential load forecasting based on LSTM recurrent neural network.IEEE Transactions on Smart Grid, 10(1): 841–851. LeGuen,V.;andThome,N.2019. Shapeandtimedistortion loss for training deep time series forecasting models. InAd- vancesinNeuralInformationProcessingSys...

    2019

  3. [2022]

    Zhang, Y.; and Yan, J

    Less is more: Fast multivariate time series fore- casting with light sampling-oriented MLP structures.arXiv preprint arXiv:2207.01186. Zhang, Y.; and Yan, J

    arXiv

  4. [2023]

    predict,then optimize

    Long-term forecasting with TiDE: Time-series denseencoder.TransactionsonMachineLearningResearch (TMLR). Elmachtoub,A.N.;andGrigas,P.2022. Smart“predict,then optimize”.Management Science, 68(1): 9–26. Fama, E. F

    2022

  5. [2024]

    InInternational Confer- ence on Learning Representations (ICLR)

    iTransformer: Inverted transformers are ef- fective for time series forecasting. InInternational Confer- ence on Learning Representations (ICLR). Liu, Y.; Wu, H.; Wang, J.; and Long, M. 2022b. Non- stationary transformers: Exploring the stationarity in time series forecasting....

    2024

  6. [2025]

    Yi, K.; Zhang, Q.; Fan, W.; Wang, S.; Wang, P.; He, H.; An, N.; Lian, D.; Cao, L.; and Niu, Z

    CaReTS: A Multi-Task Framework Unifying Classification and Re- gression for Time Series Forecasting.arXiv preprint arXiv:2511.09789. Yi, K.; Zhang, Q.; Fan, W.; Wang, S.; Wang, P.; He, H.; An, N.; Lian, D.; Cao, L.; and Niu, Z

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.