{"id":"35fd786f-ace8-4ac2-ab78-2d09030151a8","arxiv_id":"2608.01599","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A regime-conditional audit reveals that volatility forecasts with comparable aggregate accuracy hide substantial state-dependent bias and tail underprediction.","lead":"This paper proposes a model-agnostic audit framework that checks volatility forecasts for hidden bias across latent market regimes, showing that models with similar RMSE can differ sharply in stress-period reliability. It offers a new evaluation lens for risk management, moving beyond average-error metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Latent-regime contribution may collapse to stress-score thresholding; test whether learned embeddings add information beyond the stress score.","rationale":"The reader identified the same weakest assumption: the asset-relative regimes must capture economically meaningful states beyond the stress score. My stress-test agrees fully: the alignment procedure makes the regime labels a function of the stress score, so the paper must demonstrate that the learned embeddings add information. The proposed concrete test—comparing against a simple stress-score threshold partition—directly targets that assumption. If the test fails, the paper's contribution shrinks to a stress-conditioned evaluation, which is useful but not novel as 'latent' regimes. If it passes, the central claim is supported. The reader's verdict of CONDITIONAL is exactly right: the paper should be accepted only if the authors provide this comparison. Therefore I recommend no change to the reader's verdict, and I agree with the stated weakest assumption. My load-bearing concern is not a rejection of the empirical finding but a precise condition on the core methodological contribution. The paper itself even states (Section VI) that regimes are 'sample-dependent audit partitions,' which reinforces the need to show they are not just stress-score bins. I see no other concern of comparable weight: the ad-hoc metrics and missing repository are secondary and addressable, as the reader noted. Thus the verdict remains CONDITIONAL, contingent on the stress-threshold benchmark.","tokens_in":10920,"tokens_out":3143,"duration_ms":40405,"concrete_test":"Re-run the audit on the same data and models, but replace the learned-embedding regimes with a direct partition of the training-period stress score into K=3 equal-frequency bins (or K-means on the one-dimensional stress score), assigning held-out windows by their stress score. Compare (a) the adjusted Rand index between the learned regimes and the stress-threshold regimes; (b) the per-model worst-regime bias and its bootstrap significance; (c) the model rankings induced by RBA_stable and RCRG_norm. If the stress-threshold regimes reproduce the same regimes, the same 'hidden' biases, and the same rankings (e.g., ARI > 0.8 and rank correlation > 0.9), then the latent-regime representation adds no information beyond the observed stress score, and the central novelty claim fails. If the partitions differ substantially and produce different reliability conclusions, the latent construction is","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that aggregate metrics hide conditionally unreliable forecasts in 'latent' market regimes. The regimes are built (Section III-C) by clustering learned embeddings and then aligning clusters by a training-period stress score. But the stress score is itself a composite of volatility, drawdown, absolute return, and stress proxies—essentially the same variables used as features for the embeddings. The alignment step orders clusters by this score, so the final calm/intermediate/stress labels are monotone in the score. It is therefore unclear whether the learned representations contribute anything beyond a direct thresholding of the stress score. Table III shows that the regimes differ in RV, IV, drawdown, and tail share, but these are exactly the components of the stress score; that characterization does not establish that the latent embedding adds incremental information. If the regimes are merely quantiles of a visible stress score, then the 'latent' framing is not substantiated, and the empirical findings reduce to a conditional evaluation on an observed economic variable—valuable but no longer a regime-discovery contribution. This is the load-bearing assumption: without evidence that the representation captures structure beyond the stress score, the framework's central novelty (contribution 3) is in question, and the robustness results in Table VII do not address it because all variants still use the same stress-score alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a model-agnostic audit framework for volatility forecasts. It learns asset-normalized time-series embeddings with a TS2Vec-style encoder and an adversarial asset classifier, clusters the embeddings within each asset, aligns the resulting clusters by a training-period stress score into calm/intermediate/stress regimes, and then evaluates a diverse set of volatility forecasters conditionally on these regimes. The empirical application uses daily data for BTC, ETH, and five US ETFs, and the main finding is that models with competitive aggregate RMSE can exhibit systematic regime-conditional bias, tail underprediction, and unfavorable economic-loss profiles. The paper also introduces two summary diagnostics, RBA_stable and RCRG_norm, and reports block-bootstrap confidence intervals for regime-specific bias. The central claim is that aggregate accuracy can mask conditionally unreliable forecasts.","tokens_in":11275,"tokens_out":5675,"duration_ms":68767,"significance":"If the framework is taken at face value, it addresses a real gap: standard aggregate metrics such as RMSE and MAE do compress away the location of forecast failures, and for volatility forecasting those failures are most consequential in stressed states. The paper has several strengths: the regimes are constructed on training information only and assigned out of sample, which avoids the most naive forms of leakage; the use of block-bootstrap confidence intervals for regime bias is appropriate given overlapping forecast horizons; the evaluation spans a broad set of model families, including finance-native HAR benchmarks, recurrent, convolutional, and transformer-style architectures; and the repository is provided for reproducibility. The paper also carefully avoids overclaiming that the regimes are universal market states, acknowledging they are sample-dependent audit partitions. However, the central novelty of the paper — the 'latent' regime construction — is not yet substantiated, because the alignment step uses a stress score built from essentially the same variables used to learn the embeddings. A simple stress-score thresholding baseline would clarify whether the learned repres","major_comments":[{"comment":"The paper's central novelty (contribution 3) is the construction of 'asset-relative aligned regimes' from learned embeddings. However, the alignment step orders clusters by a training-period stress score whose components — volatility, drawdown, absolute return, and stress-proxy variables — are also inputs to the embedding (Section III-A). The final calm/intermediate/stress labels are therefore monotone in a visible economic score, and it is unclear whether the latent representation adds any information beyond a direct thresholding of that score. The paper should add a baseline in which regimes are defined by within-asset quantiles of the training-period stress score (with the same K=3), and then compare the audit conclusions (Table V, Table VI, and Table VII) under that baseline. If the latent regimes give materially different conditional-bias findings or asset-regime dependence, contrib","section":"Section III-C and Table VII"},{"comment":"The headline summary statistics RBA_stable and RCRG_norm are defined with hyperparameters that are never reported: lambda_stab, epsilon, and the precise definition of the valid-regime set K_valid. Similarly, the asymmetric loss used for the economic-loss rankings is defined as lambda > 1, but the actual value of lambda used in Table V and Figure 2 is not given. Since the model rankings in Figure 2 depend on these choices, the main empirical claims are not reproducible from the manuscript alone. Please report the exact values and provide a sensitivity analysis over lambda_stab and lambda. The stress score used for alignment is also described only qualitatively ('combining volatility, drawdown, absolute return, and stress-proxy information'); its exact formula and weighting must be specified, both for reproducibility and for the thresholding baseline suggested above.","section":"Sections III-E and III-F, Tables V and VI"},{"comment":"The empirical comparison is described only at a coarse level. The paper does not give the feature set, training procedure, hyperparameter choices, or validation scheme for each of the sixteen forecasting models. While the repository link is a useful step, the manuscript itself needs at least a table summarizing each model's key configuration (or a reference to a fixed, versioned configuration file) so that the reported RMSE and regime-bias rankings can be independently checked. This is especially important because the paper draws conclusions about model families — e.g., that transformer-style models are less responsive to volatility extremes — which depend on the specific implementations and hyperparameters used.","section":"Section IV-A and Table V"}],"minor_comments":[{"comment":"The first column header appears garbled as 'RegimenCrypto'. It should be split into separate columns, e.g., 'Regime', 'n', and 'Crypto share', so that the sample sizes and asset composition are legible.","section":"Table III"},{"comment":"The author list for the domain-adversarial training reference is misprinted as 'Y. Gani, E. Ustinova, ...'; the correct first author is Yaroslav Ganin, and the standard citation is Ganin et al., JMLR 2016.","section":"References [28]"},{"comment":"The asymmetric loss uses a parameter lambda > 1, but no concrete value is given in the text. The value might be in the repository, but it should appear in the paper because Table V and Figure 2 report ranking under this loss.","section":"Section III-F"},{"comment":"The robustness table reports median RBA and RCRG values without uncertainty estimates. Given that the main results rely on bootstrap inference, it would be helpful to provide bootstrap intervals for these medians as well, even if only for the main specification.","section":"Table VII"},{"comment":"The right panel is labeled 'Economic stress gradient' but does not clearly identify which color scale corresponds to the stress proxy. Adding a color bar with numeric values would improve readability.","section":"Figure 1"},{"comment":"The term 'historical baselines' is vague. The table later lists 'CurrentHistRV', but the reader is not told exactly how this baseline is constructed (e.g., rolling historical average or last observed RV). Please define it explicitly.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but the 'latent regime' framing is the main selling point and is currently not supported by a decisive experiment. The missing stress-threshold baseline is the key issue: without it, the contribution could be reframed as conditional evaluation on an observed stress score, which is useful but much less novel. I do not see evidence of circularity in the forecasting evaluation itself, since the regimes are not used to train the forecasters and are assigned out of sample. I would encourage the editor to ask for the baseline comparison and the hyperparameter reporting before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it: it makes a point worth taking seriously—aggregate RMSE can hide regime-specific volatility-forecast failures—and its central methodological novelty is shakier than the abstract suggests, because the \"latent\" regimes may be just a reordering of an observable stress score.\n\nThe package is new as a combination: adversarial representation learning plus within-asset clustering plus stress-score alignment for audit partitions. The paper runs a real empirical sweep across 16 forecasters on crypto and ETFs, with block-bootstrap confidence intervals for regime bias. The main observation—several models that look equally good on RMSE differ a lot in worst-regime bias, and some accurate models are significantly biased in stress/calm states—is supported by the tables. That is a useful result for applied forecasting work.\n\nThe biggest issue is the one the stress-test flags: regime alignment uses a training-period stress score built from the same volatility/drawdown/return features that feed the embeddings. To show the learned representation adds anything beyond thresholding that score, they would need a simple baseline: partition by stress-score terciles within each asset, run the same audit, and show the cluster-based regimes find different or sharper failures. As is, the \"latent\" claim is not substantiated. Table III only shows the regimes differ on the stress components, which is circular. This is the load-bearing weakness, and it is addressable.\n\nAlso, RBA_stable and RCRG_norm have undefined hyperparameters (lambda_stab, epsilon) and no sensitivity analysis. The repo is linked but not verified; the full pipeline depends on it. A minor point: the robustness variants all use the same stress-score alignment, so they do not resolve the main concern.\n\nThe paper is for anyone working on volatility-forecast evaluation and risk management. It is a solid conditional-evaluation reminder with a credible empirical demonstration, but the methodological packaging oversells \"latent.\" A serious referee could fix this in a round of revision.\n\nSend it to peer review. It deserves a careful referee, not a desk reject, because the core finding is reproducible in spirit and the missing baseline is a straightforward addition.","headline":"Useful audit framework held back by an overclaimed 'latent' regime story and under-specified metrics; the empirical core on conditional bias is sound enough to warrant a serious revision.","tokens_in":11629,"tokens_out":2990,"would_cite":false,"duration_ms":34865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Volatility forecasts that look accurate on average can be badly biased in specific market regimes.","keywords":["volatility forecasting","forecast evaluation","latent regimes","regime-conditional bias","tail underprediction","asymmetric loss","time-series representation learning","asset-relative regimes"],"falsifier":"Replace the learned embeddings with a direct stress-score quantile partition (e.g., thirds of the training-period stress score) and rerun the audit. If the RBA_stable, tail-underprediction, and economic-loss results are nearly identical, the latent-regime machinery is not adding information over the stress score alone.","tokens_in":10898,"feed_emoji":"📉","tokens_out":4845,"duration_ms":54411,"temperature":0.7,"pith_summary":"Volatility forecasts are usually judged by average error. This paper argues that such aggregate metrics hide where and when forecasts become unreliable, and proposes a model-agnostic audit that checks forecast errors inside latent market regimes learned separately for each asset. Applying the audit to daily volatility forecasts for cryptocurrencies and ETFs, the paper shows that models with similar RMSE can differ sharply in regime-specific bias, tail underprediction, and underprediction-sensitive losses. The takeaway is that a forecast can look accurate on average while being economically dangerous in stressed states, so evaluation should ask not only 'which model is most accurate' but 'where does accuracy fail'.","feed_headline":"Average accuracy hides where volatility forecasts fail","feed_subtitle":"An audit of crypto and ETF forecasts shows models with similar RMSE can differ sharply in stress-state bias and tail underprediction.","key_machinery":"The load-bearing object is the asset-relative aligned regime partition. Learned embeddings of 64-day market-state windows are normalized within each asset and passed through an adversarial encoder that reduces linearly recoverable asset identity; K-means then clusters each asset's training embeddings into three states, and the clusters are ordered by a stress score into calm, intermediate, and stress regimes. The audit statistics RBA_stable and RCRG_norm compare the largest regime bias against the aggregate bias, and tail-weighted asymmetric loss penalizes underprediction in high-volatility outcomes.","core_discovery":"The paper's central claim is that aggregate accuracy can mask substantial regime-dependent failures in volatility forecasting. It constructs asset-relative regimes — calm, intermediate, and stress — from window embeddings of market features, using within-asset normalization, adversarial reduction of asset identity, K-means clustering on training data, and alignment by a training-period stress score. On held-out data, it then measures regime-conditional bias, tail-underprediction, and asymmetric economic loss for a wide range of forecasters, from HAR/IV baselines to recurrent, convolutional, and Transformer models. The result is that RMSE ranking, conditional-reliability ranking, and economic","pith_inferences":["A natural test of the central contribution would be to run the same audit with a plain stress-score quantile partition in place of the learned embeddings; if the reliability signal is unchanged, the latent representation is adding little.","Because regimes are asset-relative, the audit avoids pooled clusters that mainly encode asset identity, suggesting the method could transfer to multi-asset settings where universal regimes are poorly defined.","The regime definitions are sample-dependent audit partitions, so extending the audit to longer samples or intraday targets could reveal additional failure modes, but the stability of the latent regimes across such settings is an open question.","If regulators or risk managers adopt conditional reliability reporting, model selection pressure would shift toward forecasters that remain calibrated in stress states, which could change which models are deployed."],"forward_implications":["A model's rank under RMSE can differ from its rank under regime-conditional reliability and under tail-weighted economic loss.","Low aggregate RMSE does not guarantee low worst-regime bias: the audit finds statistically supported conditional bias in at least one regime for several accurate models.","Simple finance-native HAR-RV variants can become more attractive under underprediction-sensitive losses than more complex neural or Transformer models.","The audit is model-agnostic and out-of-sample: regimes are built only from training information, so any forecaster can be audited without retraining.","Choosing a volatility forecaster for risk management should weigh where errors occur, not just average error."],"supporting_citations":[{"why":"Provides the conditional-predictive-ability perspective that motivates comparing forecast errors conditional on observed information sets.","marker":"[4]"},{"why":"Supplies the subgroup-reliability audit idea used to justify regime-conditional evaluation beyond aggregate performance.","marker":"[5]"},{"why":"Gives the TS2Vec-style contrastive encoder used to learn window embeddings for the audit partitions.","marker":"[19]"},{"why":"Supplies the gradient-reversal adversarial classifier used to reduce asset identity in the learned embeddings.","marker":"[28]"},{"why":"Underpins the claim that conditional reliability matters beyond point accuracy in forecast evaluation.","marker":"[1]"},{"why":"Supports the use of asymmetric loss in evaluating forecast rationality and motivates underprediction-sensitive losses.","marker":"[2]"}],"fun_headline_variants":["Latent regimes expose hidden forecast bias","Aggregate accuracy masks conditional failure","Which market state does your forecast break?","Good RMSE, bad stress: volatility audit"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The asset-relative regimes are aligned by a training-period stress score; if the learned embeddings add no information beyond a simple stress-score threshold, the central contribution of 'latent' regimes is not substantiated.","fun_headline_variants_meta":{"raw":{"variants":["Latent regimes expose hidden forecast bias","Aggregate accuracy masks conditional failure","Which market state does your forecast break?","Good RMSE, bad stress: volatility audit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1400,"prompt_tokens":706,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":653}},"tokens_in":450,"tokens_out":694,"duration_ms":9007,"temperature":1.0,"reasoning_tokens":653,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:20:59.574543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the learned embeddings with a direct stress-score quantile partition (e.g., thirds of the training-period stress score) and rerun the audit. If the RBA_stable, tail-underprediction, and economic-loss results are nearly identical, the latent-regime machinery is not adding information over the stress score alone.","supporting_citations":[{"cited_title":"Tests of conditional predictive ability,","cited_arxiv_id":null,"evidence_quote":"Provides the conditional-predictive-ability perspective that motivates comparing forecast errors conditional on observed information sets."},{"cited_title":"Multicalibration: Calibration for the (Computationally- identifiable) masses,","cited_arxiv_id":null,"evidence_quote":"Supplies the subgroup-reliability audit idea used to justify regime-conditional evaluation beyond aggregate performance."},{"cited_title":"Ts2vec: Towards universal representation of time series,","cited_arxiv_id":null,"evidence_quote":"Gives the TS2Vec-style contrastive encoder used to learn window embeddings for the audit partitions."},{"cited_title":"Domain- adversarial training of neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-reversal adversarial classifier used to reduce asset identity in the learned embeddings."},{"cited_title":"Evaluating interval forecasts,","cited_arxiv_id":null,"evidence_quote":"Underpins the claim that conditional reliability matters beyond point accuracy in forecast evaluation."},{"cited_title":"Estimation and testing of forecast rationality under flexible loss,","cited_arxiv_id":null,"evidence_quote":"Supports the use of asymmetric loss in evaluating forecast rationality and motivates underprediction-sensitive losses."}],"review_version":1}