Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis

Joseph Nyangon; Ruth Akintunde

REVIEW 1 major objections 5 minor 2 references

Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis

T0 review · 1 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that two-stage outlier removal plus PCA features reduces day-ahead price forecast error in California, with test RMSE 5.83, beating raw features (5.87) and IQR-only (6.69).

desk verdict A routine PCA/outlier-cleaning forecast comparison on CAISO data whose central comparison is unverifiable because the paper never says whether cleaning or PCA used test-period information. read the letter →

arxiv 2412.07787 v1 pith:6WYZODMT submitted 2024-11-25 econ.EM cs.AIcs.ETcs.LGcs.SYeess.SY

Joseph Nyangon , Ruth Akintunde This is my paper · ORCID

classification econ.EMcs.AIcs.ETcs.LGcs.SYeess.SY

keywords electricitypriceforecastingprincipalcomponentanalysisrobustPCAanomalydetectionoutlierremovalCAISOday-aheadmarketheteroskedasticityrenewableenergyintegration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the way price data are cleaned matters as much as the model for day-ahead electricity price forecasting in California. Using six years of hourly day-ahead prices and demand from the California ISO (CAISO), the authors first remove outliers with the interquartile-range rule, then apply a sparse-matrix robust PCA that separates the data into a normal low-rank part and a sparse anomalous part. They fit linear regressions to the raw features and to PCA-transformed versions of the same features. Their best test result is the PCA-transformed model after both cleaning stages, with test RMSE 5.83 dollars per megawatt-hour and $R^2=0.84$, slightly better than the same cleaning with raw features (5.87) and clearly better than IQR-only cleaning (6.69). If this holds, a simple, transparent pipeline can improve day-ahead forecast reliability for grid operators and renewable integration.

What carries the argument

The load-bearing object is the robust PCA decomposition, written as $\min_{L,S}\ \|L\|_* + \lambda \|S\|_1$ subject to $A = L + S$, where $A$ is the data matrix, $\|L\|_*$ is the nuclear norm, and $\|S\|_1$ encourages sparsity. With $\lambda = 1/\sqrt{n}$, the procedure separates the matrix into a low-rank component capturing normal price-and-load patterns and a sparse component containing anomalies; the paper removes the sparse component's flagged points, after first applying an interquartile-range filter. On the cleaned data, PCA produces uncorrelated linear combinations of the seven raw features (yesterday's price, yesterday's load, yesterday's average load, month, day of week, day of month, day of year), which are then used as regressors in a linear model.

What would settle it

Recompute the pipeline with a strict temporal split: estimate the IQR bounds and the sparse-matrix outlier mask on the 2016-2019 training years only, freeze that mask, and apply it to the 2020-2021 test year. If the PCA-feature model's test RMSE no longer beats the raw-feature model's 5.87, or rises above it, the paper's central claim about PCA's contribution fails; if the ordering Model 4 < Model 3 < Model 2 reproduces, the claim is supported.

Watch

Extended reading notes

Core claim

On CAISO hourly day-ahead prices and demand from 2016 to 2021, the paper fits four linear regressions. Removing outliers by IQR alone cuts test RMSE from 26.28 to 6.69; adding the sparse-matrix robust PCA cleaning drops it to 5.87 with the same raw features. Replacing the seven raw features with PCA-transformed features after the same two-stage cleaning gives the best test result, RMSE 5.83 and $R^2=0.84$, a slight edge over the raw-feature version. The paper reads this as evidence that combining anomaly removal with PCA feature extraction improves forecast accuracy and tames the heteroskedastic spikes that dominate California's day-ahead market.

Load-bearing premise

The central assumption is that the price-cleaning step only uses the training part of the data; if it also uses test-period prices to decide which points to delete, the reported forecast errors are optimistic and the model comparison is circular.

Editorial extensions

If this is right

The two-stage cleaning pipeline (IQR plus sparse-matrix robust PCA) followed by PCA features produces day-ahead price forecasts with test RMSE 5.83 dollars per megawatt-hour, versus 6.69 for IQR-only cleaning and 26.28 for no cleaning.
Most of the accuracy gain comes from outlier removal: the raw-feature model with both cleaning stages reaches 5.87, so PCA contributes only a small final increment.
The method is transparent and reproducible, needing only historical price, load, and calendar features, with no deep learning or market microstructure inputs.
For grid operators and renewable schedulers, lower day-ahead forecast error implies less expensive real-time balancing and more reliable schedules under volatile prices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The PCA-specific edge over raw features is thin (5.83 vs 5.87 on one test split), and the paper reports no uncertainty measures, so the honest reading is that PCA's added value is plausible but unproven until re-run over multiple years or with confidence intervals.
Because the cleaning rules are described on the full 2016-2021 sample, a strict out-of-sample check should re-estimate the outlier mask on training years only; if the edge vanishes, the improvement may be a leakage artifact.
The pipeline should transfer to other volatile markets; re-running it on a different ISO's prices would show whether the RMSE ordering across models is a general property of outlier-plus-PCA cleaning or specific to California's spike pattern.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The paper is a workmanlike application of existing tools—IQR plus robust PCA outlier removal, then PCA features in a linear regression—to hourly CAISO day-ahead prices from 2016–2021. The methods are not new; the value is in the case-study comparison. It does some things well: it uses real market data, reports descriptive statistics, flags wildfire and negative-price effects, and cites the PCA/RPCA literature appropriately. If the evaluation protocol were clean, the finding would be a modest confirmation that outlier cleaning helps and PCA features do not degrade forecasts.

The soft spot is load-bearing. Section III.C describes outlier removal on the full dataset, and no train/test split is ever defined. The paper reports training/test RMSE in Table II but never says when the split occurs, whether the IQR bounds, the RPCA sparse matrix, or the PCA transformation were estimated using only training data. If any of those steps saw the test period, the test RMSE is partly fitted and the 5.83 vs 5.87 difference between Models 4 and 3 is circular. That is not a minor omission; it determines whether the central claim holds.

Other issues are real but secondary. The top-two RMSE gap is 0.04, with no error bars or significance test. There are no external baselines. The RPCA objective function is garbled in the text. No code or data are provided. These reinforce the skeptical verdict.

I am not arguing the paper is dishonest—it reads like a straightforward empirical exercise that simply omitted the evaluation protocol. If the authors can show that all cleaning and feature construction used only the training period, the paper becomes a modest applied contribution worth publishing somewhere. As submitted, the evidence does not support the headline claim. I would not cite it. It could be a useful reading-group example of how easy it is to leak test information in forecasting, but not for its results.

I would send it to a serious referee rather than desk-reject it, because the flaw is fixable and the underlying question is legitimate. But the referee should ask for the split, the leakage check, significance testing, and code/data. As is, reject.

Referee Report

1 major / 5 minor

Summary. The paper proposes an anomaly-detection and feature-transformation pipeline for day-ahead electricity price forecasting in the CAISO market. Using 2016-2021 hourly price and load data, the authors first remove outliers via the interquartile range (IQR) method and then via a robust PCA (RPCA) sparse-matrix procedure implemented in SAS Viya. They then fit multiple linear regression models with raw features and with PCA-transformed features, and compare their in-sample and out-of-sample RMSE and R2 values. The central claim is that the model using PCA features after both outlier-removal steps (Model 4) achieves the best test performance, with a test RMSE of 5.83 versus 5.87 for the raw-feature model (Model 3), and that this demonstrates that PCA-based methods significantly enhance forecasting accuracy.

Significance. If the reported results were obtained under a valid out-of-sample protocol, the paper would provide a modest but potentially useful demonstration that combining RPCA-based outlier removal with PCA feature extraction can improve day-ahead price forecasts in a volatile market. However, the manuscript as written does not establish this. The evaluation protocol is not described: the outlier-removal and PCA steps are applied to the full dataset before any train/test split is introduced, so the reported test RMSE values may be optimistically biased by test-period information. Moreover, the improvement of Model 4 over Model 3 is only 0.04 RMSE (about 0.7%) with no error bars or significance tests, and no baselines such as persistence or ARIMA are included. The absence of code and data further prevents verification. The paper does have some strengths: it uses a realistic six-year CAISO dataset, presents descriptive statistics that reveal known market features (e.g., extreme price spikes), and describes a transparent two-stage outlier-removal idea. Yet these strengths do not compensate for the load-bearing evaluation issues.

major comments (1)

[Section III.C] The RPCA equation is garbled: the displayed expression '/0'0/012 ∥5∥∗+ 7∥8∥% 9:;<2=> >? @=5+8' is not readable, and the definitions of the nuclear norm, the ℓ1 norm, and the balancing parameter λ are not clearly stated. The paper references the SAS documentation for details, but the equation is the methodological core of the anomaly-detection step. This makes it impossible to understand or reproduce the method from the text. Please rewrite the optimization problem in standard notation, e.g., min_{L,S} ||L||_* + λ||S||_1 subject to X = L + S, and specify how λ was chosen for the CAISO dataset.

minor comments (5)

[Table I] The table lists 8760 observations for 2016, but 2016 is a leap year and should have 8784 hours; the stated total of 52,441 observations is also inconsistent with the correct per-year counts. Please correct these numbers.
[Section I] The kernel density estimation equation in the introduction is garbled and uses undefined notation; it is not connected to the rest of the paper and should be either removed or rewritten with full definitions.
[Section IV.A] The text says 'we examined up to 5 PCA features' but does not report the number of components actually retained in Model 4 or the criterion used (e.g., cumulative explained variance). Please state this explicitly.
[References] Several references contain encoding artifacts (e.g., 'California?s' in [25] and [31]), and there is an unusually high number of self-citations; please ensure all citations are necessary and correctly formatted.
[Figure 3] The caption and text describe the figure as showing outcome of the SAS Sparse Matrix strategy 'alone and in combination' with IQR, but only one panel appears; please clarify or provide the missing panel.

Circularity Check

1 steps flagged · score 5.0 of 10

Outlier removal and PCA appear to be applied to the full 2016-2021 dataset before any train/test split is described, so Model 4's reported test RMSE is not a demonstrated out-of-sample prediction.

fitted input called prediction [Section III.C (outlier removal) and Section IV.A / Table II (test results)]
"Further refining our analysis, we implemented the Robust Principal Component Analysis (RPCA) technique using SAS Viya platform... The resulting sparse matrix, named sparsemat2, as outlined in [34], identified outliers, which were then excluded from our analysis. ... Model 4, which employed SAS PCA features after outlier removal using traditional and SAS Sparse Matrix methods, achieved the highest performance on the test set: a slight improvement when SAS PCA features are applied instead of the 7 raw features."

The methodology section describes outlier removal on the full 2016-2021 dataset before any train/test split is defined, and the results section then reports test-set RMSE for models built on the cleaned data. Because the IQR bounds and the RPCA sparse matrix are estimated on the full sample, test-period outliers are identified and removed before evaluation; the PCA transformation likewise uses full-sample covariance. The test RMSE of Model 4 (5.83) is therefore computed on a filtered and transformed test set, and the claimed improvement over Model 3 (5.87) is partly forced by the fact that the cleaning step already used the test data.

full rationale

The paper's equations and PCA machinery are standard and do not reduce to an identity, and the self-citations in the reference list are background material rather than load-bearing premises. The central circularity concern is the evaluation protocol: Section III.C applies IQR and RPCA anomaly detection to the full dataset before any split is described, and Table II reports training/test RMSE for models built on the cleaned data. If, as the text suggests, the test-period outliers and PCA loadings are determined using the full sample, then the test RMSE is computed on data whose anomalies were already removed using information that includes the test period. This makes the reported improvement of Model 4 over Model 3 partly a consequence of the cleaning protocol rather than a genuine out-of-sample forecast. Because the paper never demonstrates that the cleaning and feature extraction use only training data, the central claim is not established as a valid predictive result. The score of 5 reflects a partial circularity: the prediction is not purely a fit of a parameter renamed as a result, but the evaluation is contaminated by full-sample data cleaning, so the claimed accuracy gain is not independent of the method's own inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a modest set of statistical assumptions and two unreported tuning choices. No new entities are posited; the sparse matrix is an artifact of the RPCA decomposition. The largest unverified load-bearing assumption is that outlier removal is performed without leaking test-period information into the training process.

free parameters (2)

Number of retained PCA components = Up to 5, exact number not reported
The paper says 'up to 5 PCA features' but never states how many were chosen for Model 4 or whether selection used validation data, so the reported test performance may depend on an unreported tuning choice.
RPCA regularization lambda = 1/sqrt(n), cited from literature
Lambda balances the low-rank and sparse components. It is set from a cited formula rather than fitted, but no sensitivity analysis is given, and the garbled equation means the exact implementation is unclear.

assumptions (5)

domain assumption PCA captures the dominant variance structure of hourly price and load features.
The forecasting improvement assumes linear low-dimensional structure in the features; no variance-explained or component-loading evidence is provided.
domain assumption Price spikes and negative prices are anomalies to be removed rather than signal.
The method removes sparse outliers via RPCA and IQR; if these events carry predictive information, such as wildfire or shortage signals, removal could bias forecasts.
domain assumption CAISO 2016-2021 hourly data are complete and representative.
The paper relies on CAISO data but provides no data-cleaning details or missing-value handling; Table I has year counts that do not match 24x365, suggesting undisclosed data processing.
domain assumption Linear regression is an adequate model for day-ahead price after cleaning.
The paper uses ordinary least squares without testing nonlinearity, heteroskedasticity, or serial correlation, despite mentioning heteroskedasticity as a key challenge.
standard math The RPCA low-rank plus sparse decomposition assumptions hold.
The recovery guarantee cited from Zhao et al. requires the low-rank component to be incoherent and the sparse component to be sufficiently sparse; the paper does not verify these conditions on CAISO data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis." pith.science (2026). https://pith.science/paper/6WYZODMT

@misc{pith2026241207787,
  author       = {Pith},
  title        = {Pith review of: Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WYZODMT}},
  note         = {Machine review of arXiv:2412.07787}
}

read the original abstract

Accurate and reliable electricity price forecasting has significant practical implications for grid management, renewable energy integration, power system planning, and price volatility management. This study focuses on enhancing electricity price forecasting in California's grid, addressing challenges from complex generation data and heteroskedasticity. Utilizing principal component analysis (PCA), we analyze CAISO's hourly electricity prices and demand from 2016-2021 to improve day-ahead forecasting accuracy. Initially, we apply traditional outlier analysis with the interquartile range method, followed by robust PCA (RPCA) for more effective outlier elimination. This approach improves data symmetry and reduces skewness. We then construct multiple linear regression models using both raw and PCA-transformed features. The model with transformed features, refined through traditional and SAS Sparse Matrix outlier removal methods, shows superior forecasting performance. The SAS Sparse Matrix method, in particular, significantly enhances model accuracy. Our findings demonstrate that PCA-based methods are key in advancing electricity price forecasting, supporting renewable integration and grid management in day-ahead markets. Keywords: Electricity price forecasting, principal component analysis (PCA), power system planning, heteroskedasticity, renewable energy integration.

Figures

Figures reproduced from arXiv: 2412.07787 by the authors.

**Figure 2.** Boxplot after outlier removal using the traditional strategy (1.5*IQR) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Comparison of boxplots with outlier removal using traditional and SAS’ Sparse Matrix strategy This dual-method approach to outlier detection and removal in electricity price ensures a robust and accurate analysis, crucial for informed decision-making in the power markets [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[5]

A price-based demand response scheduling model in day-ahead electricity market,

Q. Duan, “A price-based demand response scheduling model in day-ahead electricity market,” in 2016 IEEE Power and Energy Society General Meeting (PESGM), Jul. 2016, pp. 1–5. [6] A. Sadeghi-Mobarakeh, M. Kohansal, E. E. Papalexakis, and H. Mohsenian-Rad, “Data mining based on random forest model to predict the California ISO day-ahead market prices,” in 20...

work page arXiv 2016
[2024]

copperplate

Email: 1 jnyangon@udel.edu and 2 Ruth.Akintunde@sas.com Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis Joseph Nyangon, Ph.D.1 Senior Member, IEEE and Ruth Akintunde, Ph.D.2 U.S. Department of Energy, Washington, DC., United States 2 Research and Development Division, SAS...

arXiv 2016

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [5]

A price-based demand response scheduling model in day-ahead electricity market,

Q. Duan, “A price-based demand response scheduling model in day-ahead electricity market,” in 2016 IEEE Power and Energy Society General Meeting (PESGM), Jul. 2016, pp. 1–5. [6] A. Sadeghi-Mobarakeh, M. Kohansal, E. E. Papalexakis, and H. Mohsenian-Rad, “Data mining based on random forest model to predict the California ISO day-ahead market prices,” in 20...

work page arXiv 2016

[2] [2024]

copperplate

Email: 1 jnyangon@udel.edu and 2 Ruth.Akintunde@sas.com Anomaly Detection in California Electricity Price Forecasting: Enhancing Accuracy and Reliability Using Principal Component Analysis Joseph Nyangon, Ph.D.1 Senior Member, IEEE and Ruth Akintunde, Ph.D.2 U.S. Department of Energy, Washington, DC., United States 2 Research and Development Division, SAS...

arXiv 2016