Pith. sign in

REVIEW 3 major objections 7 minor 11 references

When Directional Accuracy Lies: A Base-Rate-Honest Benchmark for LoRA-Adapted TimesFM on Equity Forecasting

T0 review · 3 major / 7 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LoRA-adapted TimesFM shows no directional skill over the always-up base rate in equity forecasting.

desk verdict A careful, honest negative-result paper with a genuinely reusable benchmark protocol, but the abstract overclaims a per-sector significance that the paper's own tables do not support. read the letter →

arxiv 2607.12248 v2 pith:BIZYFB3I submitted 2026-07-14 q-fin.ST cs.LG

Taizhen Cheung , SA Kwon This is my paper
classification q-fin.STcs.LG
keywords directionalaccuracybase-ratebiasLoRAadaptationTimesFMequityforecastingwalk-forwardvalidationexcessfoundationmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper contends that high raw directional accuracy in equity forecasting can be a pure base-rate artifact: in a rising market, an always-up rule is hard to beat by a model that has learned nothing. The authors build a reproducible, frozen-data, walk-forward benchmark and apply it to a tech-heavy and a broad equity universe, measuring excess accuracy over the always-up baseline. Across both universes, the LoRA-adapted TimesFM shows no positive excess accuracy at any horizon, and per-sector adapters are significantly worse than a single pooled adapter. The only measurable benefit of fine-tuning is lower point-forecast error, which nonetheless fails to beat naive baselines and provides no tradeable edge. The paper argues that the methodological protocol matters more than the specific negative result.

What carries the argument

The central mechanism is the excess-accuracy metric: model directional accuracy minus always-up accuracy computed on identical windows. This subtraction removes the base-rate component of raw accuracy and leaves only skill. It is paired with a frozen, checksum-versioned dataset, expanding walk-forward folds, a stratified held-out-ticker split, and paired significance tests (McNemar and Diebold-Mariano) under FDR control. The always-up baseline is the load-bearing comparator; per-sector vs. pooled comparisons use Diebold-Mariano on forecast loss.

What would settle it

Re-run the identical protocol on a point-in-time universe whose test window includes a sustained bear market (e.g., 2008-2009 or a 2022 drawdown), or on overlapping windows to restore long-horizon power; if pooled LoRA shows significantly positive excess accuracy over the always-up baseline at any horizon after FDR control, the paper's central negative claim would be falsified. A simpler check: compute excess accuracy on just the down-market subperiod of the existing folds; if it climbs decisively above zero, the conclusion depends on the bull-market window.

Watch

Extended reading notes

Core claim

The paper claims that the apparent ~80% directional accuracy of an early LoRA adapter was a base-rate artifact: recreated on identical windows, the always-up rule reaches ~0.70 and the fine-tuned model scores below it. On the honest benchmark, pooled LoRA's excess accuracy over the always-up base rate is centered on zero at every horizon and negative at six months on both NASDAQ-100 and S&P 500, while zero-shot TimesFM sits below the base rate everywhere. The pre-registered test shows per-sector specialization is significantly worse than a single pooled adapter (Diebold-Mariano p<0.001 at h=128). The only supported benefit of fine-tuning is lower point-forecast error, which beats neither nai

Load-bearing premise

The benchmark uses current-membership stock lists rather than point-in-time constituents, which inflates the up base rate and long-horizon upward drift; if the test windows had included a major bear market, the always-up baseline would be far lower and the 'no directional skill' conclusion might not replicate.

Editorial extensions

If this is right

  • Any report of high directional accuracy for foundation models on equities should be recomputed as excess over the always-up base rate on identical windows before skill is claimed.
  • Per-sector specialization strategies for such models are unlikely to help and may hurt, because each adapter trains on less data; the paper finds this in a pre-registered test.
  • Fine-tuning a large pretrained time-series model on equities can sharpen point forecasts without producing tradable directional skill, so point-error gains should not be sold as forecasting edge.
  • The protocol itself—frozen data, walk-forward folds, held-out tickers, seeded runs, paired tests—provides a template that prevents the base-rate trap in future evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: the same excess-accuracy logic applies to any high-base-rate classification setting (e.g., fraud detection, medical screening with rare events), not just equities; raw accuracy should always be reported against a no-input baseline.
  • The paper's finding that specialization hurts suggests a testable hypothesis: that adapter capacity is the binding constraint, so a larger-rank per-sector adapter or a longer training budget might reverse the result; the paper does not test this.
  • Because the universe comparison is confounded, a matched subsample of the S&P 500 with the same number of stocks as the NASDAQ-100 would isolate whether the negative result is robust to size; the paper lists this as future work but does not run it.
  • If the always-up baseline is so hard to beat, a stronger baseline for future work is a market-timing rule conditioned on the index itself; the paper does not include one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a benchmark protocol to avoid the 'base-rate trap' in evaluating directional equity forecasts from a LoRA-adapted TimesFM. It uses frozen, checksum-versioned price data on NASDAQ-100 and S&P 500, expanding walk-forward folds with held-out tickers, and computes excess accuracy over an always-up baseline. Three findings are claimed to replicate: (1) the original ~80% directional accuracy is a ~0.70 base rate that the fine-tuned model does not beat; (2) pooled LoRA shows no directional skill over always-up at any horizon; (3) per-sector adapters are significantly worse than a single pooled adapter (DM p<0.001 at h=128). The only positive effect is lower point-forecast MAE than zero-shot TimesFM, with no edge over naive baselines. The contribution is framed as methodological.

Significance. If correct, the paper provides a valuable cautionary template for financial foundation-model evaluation: raw accuracy without an always-up baseline is uninformative in trending markets. Strengths include a detailed, seeded reproducibility protocol, explicit walk-forward splits that never select on test windows, paired tests with FDR control, and replication across two universes with honest reporting of power limitations. The central no-skill result is plausible and consistent with the point estimates. However, the abstract's headline p<0.001 for per-sector specialization is not supported by the directional evidence, and this must be corrected before the paper can be accepted.

major comments (3)
  1. [Abstract; §6.3; §9] The claim that per-sector specialization is 'significantly worse' (p<0.001 at h=128) is not supported by the tests reported. Table 6 reports Diebold–Mariano statistics on point-forecast loss, not on directional accuracy or excess accuracy, yet §6.3 first states directional accuracies (0.599 vs 0.641 vs 0.658) and then uses the DM loss test as confirmation. At h=128, only folds 0 and 2 are individually significant (fold 1 p=.30), and no combined test is shown. §6.5 explicitly disclaims directional significance at h=128. Either provide a valid paired test on the directional metric (e.g., McNemar over pooled discordant pairs or a bootstrap CI for per-sector minus pooled accuracy) and a combined test across folds, or revise the abstract and conclusion to say per-sector adapters have higher point-forecast loss in pooling's favor, with the directional difference being descriptive only.
  2. [§6.5; §9] The conclusion that pooled LoRA shows 'no directional skill at any horizon' overstates the evidence. §6.5 reports discordant-pair counts of only {5,5,20} (S&P) and {3,10,10} (NASDAQ) at h=128 and says no significance claim is made for directional tests at that horizon. With this power, the honest statement is that excess accuracy is consistent with zero, and is negative in point estimate, but long-horizon skill cannot be reliably excluded. Please soften the 'at any horizon' wording or provide additional long-horizon evidence.
  3. [§6.4; Table 7] The comparison to an 'always-up point forecast' is undefined. Always-up is introduced in §5.5 as a directional rule ('predict up every window'), which does not produce a price forecast unless an additional convention is specified. Table 7 lists 'always up' MAE values (15.56 S&P, 24.91 NASDAQ) and §6.4 concludes pooled LoRA 'does not beat the naive always-up point forecast.' Without defining the point forecast implied by always-up (e.g., persistence, zero h-period return, or last price), this comparison cannot be checked. Please either define the baseline explicitly (and use its proper name) or replace it with the random-walk/persistence baselines that do have point forecasts.
minor comments (7)
  1. [Table 4] Specify whether the legacy condition is run on NASDAQ-100 or S&P 500. Base rates differ across the two universes, so the universe label is essential for reproducing RQ1.
  2. [Table 5] The text repeatedly states that bootstrap intervals span zero, but the table shows only point estimates. Please include the intervals and p-values, or provide a supplementary table/figure with the actual numbers.
  3. [§3] The paper calls RQ2 'pre-registered,' but no pre-registration URL, time stamp, or appendix is given. Please provide the registration record or rephrase as 'planned confirmatory analysis.'
  4. [§5.7] The 'sample-size-capped Newey–West variance' is not defined. Please state the cap used for the number of lags and how it depends on sample size.
  5. [§5.4] The reproducibility section mentions 'bit-identical Table-A accuracies,' but no Table A is included in the manuscript. Please add it or describe the verification numbers in the text.
  6. [§5.7; §6.2] The exploratory family under Benjamini–Hochberg FDR control is not enumerated. Please state how many tests were in the family and report FDR-adjusted p-values for the headline excess-accuracy comparisons.
  7. [Abstract] Typo: 'butraw' should be 'but raw.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark is self-contained, test windows are never used for selection, and the main comparisons are empirical rather than definitional.

full rationale

The paper's central derivation is an empirical benchmark, not a chain that reduces to its inputs. The headline metric is defined as excess accuracy = model accuracy minus always-up accuracy on identical windows (Section 5.6); this is a relative performance measure, but the model accuracy and the baseline accuracy are computed independently from predictions and the realized direction, so the finding of zero excess accuracy is not forced by construction. The legacy 80% reconstruction (Section 6.1) uses a separate 2014+ training window and evaluates raw accuracy against the always-up rate on the same held-out stocks; this is a controlled empirical reproduction of a historical condition, not a tautology. The per-sector versus pooled test (Section 6.3) compares independently trained adapters on held-out data, so the result that per-sector is worse is an empirical outcome, not an equation identity. The protocol explicitly guards against the classic circularity: model selection and early stopping use only validation folds (Section 5.1), normalization is fit only on pre-target history (Section 5.3), the test window is never used for selection, and all runs use a fixed seed. References are all external works (TimesFM, LoRA, RevIN, Fu et al., statistical tests), and no self-citation carries any load-bearing argument. There is a reporting mismatch worth flagging but not circular: Table 6 reports Diebold-Mariano tests on point-forecast loss while the abstract's per-sector p<0.001 is phrased as supporting directional specialization, and at h=128 only two of three folds are individually significant while Section 6.5 disclaims directional significance at that horizon. That is a statistical precision issue, not a case where a fitted parameter is renamed as a prediction or where a claimed derivation is equivalent to its input. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and no known result is merely renamed. The paper is self-contained against external benchmarks and honestly reports its limitations; no circular step is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or forces; it makes empirical claims based on a fixed training configuration and standard statistical assumptions. The free parameters are LoRA/training hyperparameters and the seed, which are explicitly acknowledged in the limitations. The most load-bearing assumption is the survivorship-biased universe, which the paper itself flags.

free parameters (6)
  • LoRA rank = 32
    Chosen hyperparameter in Table 3; affects adapter capacity and could influence directional skill, though not fitted to test data.
  • LoRA alpha = 64
    Chosen hyperparameter in Table 3; scaling factor for LoRA updates.
  • Directional loss weight = 0.3
    Loss is MSE + 0.3 * directional loss; this hyperparameter directly controls emphasis on direction and could affect the directional results.
  • Seed = 42
    Single-seed deterministic training; the paper acknowledges results are not multi-seed robust (Section 8).
  • Learning rate = 1e-4
    AdamW learning rate in Table 3; training hyperparameter.
  • Dropout = 0.05
    LoRA dropout in Table 3; training hyperparameter.
assumptions (5)
  • domain assumption TimesFM is an appropriate base model for equity forecasting
    The study assumes the pretrained foundation model is a reasonable starting point; if TimesFM is fundamentally unsuitable, the negative result may not extend to other base models.
  • domain assumption The always-up rule is the correct baseline for directional accuracy
    Excess accuracy is defined as model accuracy minus always-up accuracy; this is the core methodological choice and is appropriate in a trending market but would need rethinking in other regimes.
  • domain assumption Current-membership snapshot is a valid test universe
    The paper uses current constituents, which inflates the base rate; acknowledged as survivorship bias in Section 8.
  • standard math Diebold-Mariano and McNemar tests are valid for this data
    The paper uses paired significance tests with Newey-West variance; assumptions include stationarity and appropriate loss differential behavior.
  • ad hoc to paper Single-seed training is sufficient to support the conclusions
    The paper acknowledges this as a limitation (Section 8) but still draws strong negative conclusions from a single seed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Directional Accuracy Lies: A Base-Rate-Honest Benchmark for LoRA-Adapted TimesFM on Equity Forecasting." pith.science (2026). https://pith.science/paper/BIZYFB3I

@misc{pith2026260712248,
  author       = {Pith},
  title        = {Pith review of: When Directional Accuracy Lies: A Base-Rate-Honest Benchmark for LoRA-Adapted TimesFM on Equity Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BIZYFB3I}},
  note         = {Machine review of arXiv:2607.12248}
}
read the original abstract

Large pretrained time-series models such as TimesFM are attractive for financial forecasting, but raw directional accuracy is a misleading scoreboard in equity markets. An early LoRA adapter in this project appeared to reach roughly 80% directional accuracy; we show this is not evidence of skill. Over a long horizon in a rising market, a trivial "always-up" rule attains comparably high accuracy without using the input at all. To separate genuine skill from this base-rate artifact, we build a reproducible, frozen-data benchmark with expanding walk-forward folds, a stratified held-out-ticker split, honest baselines (zero-shot TimesFM, always-up, random-walk, persistence, AR(1)), and paired significance tests (McNemar, Diebold-Mariano) under Benjamini-Hochberg FDR control. We apply the identical method to two universes -- a tech-heavy NASDAQ-100 and a broad S&P 500 -- reporting excess accuracy over the always-up base rate. Three findings replicate. First, when the historical ~80% condition is recreated, the high number is a base rate of ~0.70 that the fine-tuned model scores below. Second, pooled LoRA shows no directional skill over the base rate at any horizon on either universe (negative at the six-month horizon). Third, per-sector specialization is significantly worse than a single pooled adapter (Diebold-Mariano p<0.001 on held-out stocks at h=128). Fine-tuning's only measurable benefit is a statistically significant reduction in point-forecast error relative to zero-shot TimesFM, which nonetheless does not beat naive baselines and confers no tradeable directional edge. The contribution is methodological: a defensible, fully seeded protocol that prevents the base-rate trap, together with the replicated negative result it produces.

Figures

Figures reproduced from arXiv: 2607.12248 by the authors.

Figure 1
Figure 1. Excess directional accuracy versus horizon, held-out stocks, both universes. The dashed [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Raw directional accuracy of pooled LoRA tracks the always-up base rate at every horizon; [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Point-forecast MAE on held-out stocks: pooled LoRA is below zero-shot but ties the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Quantile reliability (held-out stocks): empirical coverage versus nominal for pooled and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 2 linked inside Pith

  1. [1]

    A. Das, W. Kong, R. Sen, and Y. Zhou. A decoder-only foundation model for time-series forecasting (TimesFM).ICML, 2024

    2024

  2. [2]

    E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models.arXiv:2106.09685, 2021. 10

    arXiv 2021

  3. [3]

    Kim et al

    T. Kim et al. Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift.ICLR, 2021

    2021

  4. [4]

    T. Fu, M. Hirano, and K. Imajo. Financial Fine-tuning a Large Time Series Model. arXiv:2412.09880, 2024

    arXiv 2024

  5. [5]

    R. J. Hyndman and A. B. Koehler. Another look at measures of forecast accuracy.International Journal of Forecasting, 22(4):679–688, 2006

    2006

  6. [6]

    Gneiting and A

    T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation.JASA, 102(477):359–378, 2007

    2007

  7. [7]

    F. X. Diebold and R. S. Mariano. Comparing predictive accuracy.Journal of Business & Economic Statistics, 13(3):253–263, 1995

    1995

  8. [8]

    Harvey, S

    D. Harvey, S. Leybourne, and P. Newbold. Testing the equality of prediction mean squared errors.International Journal of Forecasting, 13(2):281–291, 1997

    1997

Show all 11 references
  1. [9]

    Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages.Psychometrika, 12(2):153–157, 1947

    1947

  2. [10]

    H. R. K¨ unsch. The jackknife and the bootstrap for general stationary observations.Annals of Statistics, 17(3):1217–1241, 1989

    1989

  3. [11]

    Benjamini and Y

    Y. Benjamini and Y. Hochberg. Controlling the false discovery rate.JRSS-B, 57(1):289–300, 1995. 11

    1995

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.