pith. sign in

arxiv: 2606.20079 · v1 · pith:2TNCS534new · submitted 2026-06-18 · 💱 q-fin.RM

How to spot outliers: an Ensemble Anomaly Detection Framework

Daniil Peysakhovich , Rafa{\l} Sieradzki This is my paper

Pith reviewed 2026-06-26 15:04 UTC · model grok-4.3

classification 💱 q-fin.RM
keywords ensemble anomaly detectionrisk valuationoutlier detectioncredit derivativesmodel risk managementunsupervised methodsBasel IIIFRTB
0
0 comments X

The pith

Ensemble of anomaly detectors identifies risk valuation errors more reliably than any single method.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Ensemble Quality Assessment Framework (EQAF) to detect errors in risk valuation outputs from data-feed failures or system issues. Using real credit-derivatives data and a protocol that injects eight types of realistic anomalies, it demonstrates that a calibrated combination of methods reaches F1 scores of 61-79 percent. This beats the best individual detector, which tops out at 6-66 percent, and the gains hold across different risk measures and threshold choices. Statistical methods alone miss frozen or stale values, so deterministic rules based on domain knowledge are essential for complete coverage. Such tools support regulatory needs for automated checks on internal models.

Core claim

Using proprietary daily credit-derivatives data covering 183 trades over 129 days, the EQAF ensemble achieves F1 scores between 61 and 79 percent on eight operationally realistic anomaly scenarios, outperforming individual methods whose best F1 ranges from 6 to 66 percent across four risk-measure datasets, with additional AUC-ROC gains of 4-6 points; purely statistical detectors fail on stale-value anomalies, requiring domain-specific deterministic rules.

What carries the argument

The Ensemble Quality Assessment Framework (EQAF), a layered unsupervised architecture combining complementary outlier-detection methods for real-time monitoring of risk calculation integrity.

Load-bearing premise

The controlled injection of eight operationally realistic anomaly scenarios accurately reflects the distribution and detectability of actual errors in production risk valuation systems.

What would settle it

Comparing the ensemble's detection performance on a collection of real, naturally occurring errors in risk valuation outputs against the F1 scores obtained from the injected anomaly protocol.

read the original abstract

Errors in risk valuation outputs arising from data-feed failures, model misconfiguration, or system malfunctions can propagate undetected through an investment bank's risk infrastructure and generate material operational losses. Using proprietary daily credit-derivatives data from a major global investment bank covering 183 trades across 129 trading days, we design, implement, and empirically evaluate the Ensemble Quality Assessment Framework (EQAF), a layered unsupervised architecture that combines complementary outlier-detection methods to monitor risk calculation integrity in real time. Using a controlled anomaly-injection protocol with eight operationally realistic scenarios, we show that the calibrated ensemble achieves F1 scores of 61-79%, substantially outperforming the best individual method (6-66%) across four distinct risk-measure datasets. Improvements of 4-6 percentage points in AUC-ROC confirm that this advantage is robust to threshold selection. We further demonstrate that purely statistical detection methods systematically fail to identify stale-value anomalies, a class of frozen-feed errors in which valuation outputs are identical to prior observations and therefore indistinguishable from normal data, and that domain-specific deterministic rules are architecturally indispensable. These findings have direct implications for model risk management under Basel III and the Fundamental Review of the Trading Book (FRTB), where automated and auditable quality controls for internal risk models are increasingly required.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces the Ensemble Quality Assessment Framework (EQAF), a layered unsupervised ensemble combining complementary outlier-detection methods with domain-specific deterministic rules to monitor integrity of risk valuation outputs. Using proprietary daily credit-derivatives data on 183 trades over 129 days and a controlled injection of eight asserted 'operationally realistic' anomaly scenarios, it reports that the calibrated ensemble attains F1 scores of 61-79% (vs. 6-66% for the best single method) across four risk-measure datasets, with 4-6 point AUC-ROC gains, and argues that purely statistical methods fail on stale-value anomalies.

Significance. If the injection scenarios accurately capture the statistical properties and detectability of real production errors, the results would supply a practical, auditable architecture for real-time quality control in bank risk systems and directly support Basel III/FRTB model-risk requirements. The explicit demonstration that deterministic rules are required for certain anomaly classes is a concrete contribution. The proprietary dataset and lack of released injection code, however, constrain reproducibility and external validation.

major comments (2)
  1. [Abstract] Abstract: the headline F1 (61-79%) and AUC gains rest entirely on the claim that the eight anomaly-injection scenarios are 'operationally realistic' and share the same distribution, cross-trade correlations, and detectability profile as actual data-feed, model-configuration, and stale-value failures; the manuscript supplies no parameter values, generation code, or validation against real error logs, rendering the transferability of the performance numbers unverifiable.
  2. [Abstract] Abstract and evaluation description: ensemble weights and decision thresholds are calibrated on the same injected-anomaly data used for final F1/AUC reporting, with no mention of hold-out sets, nested cross-validation, or external benchmarks; this creates dependence between fitting and evaluation that directly affects the reported outperformance margins.
minor comments (1)
  1. [Abstract] Abstract: the four distinct risk-measure datasets are referenced but not named or characterized (e.g., by summary statistics or correlation structure).

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their thorough review and valuable feedback on our manuscript describing the Ensemble Quality Assessment Framework (EQAF). We address each of the major comments below and indicate the revisions we plan to make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline F1 (61-79%) and AUC gains rest entirely on the claim that the eight anomaly-injection scenarios are 'operationally realistic' and share the same distribution, cross-trade correlations, and detectability profile as actual data-feed, model-configuration, and stale-value failures; the manuscript supplies no parameter values, generation code, or validation against real error logs, rendering the transferability of the performance numbers unverifiable.

    Authors: We agree that the transferability of our results hinges on the realism of the anomaly scenarios. These scenarios were constructed in collaboration with risk practitioners to emulate real-world issues such as data feed failures and stale valuations based on observed patterns in the production environment. However, the proprietary nature of the dataset and internal operational logs prevents us from releasing the generation code or performing public validation against actual error records. In the revised version, we will expand the methodology section with additional qualitative details on how each scenario was generated (e.g., the specific rules for injecting stale values by duplicating previous outputs across correlated trades) to enhance transparency while maintaining confidentiality. revision: partial

  2. Referee: [Abstract] Abstract and evaluation description: ensemble weights and decision thresholds are calibrated on the same injected-anomaly data used for final F1/AUC reporting, with no mention of hold-out sets, nested cross-validation, or external benchmarks; this creates dependence between fitting and evaluation that directly affects the reported outperformance margins.

    Authors: This is a valid concern regarding the evaluation protocol. The current approach tunes the ensemble on the injected data to demonstrate the potential of the framework in a controlled setting. To strengthen the claims, we will revise the paper to include a hold-out validation procedure, for example by reserving a subset of trading days or trades for testing after calibration, or by employing nested cross-validation where feasible given the time-series nature of the data. We will report the results of this more rigorous evaluation in the updated manuscript. revision: yes

standing simulated objections not resolved
  • The inability to release the proprietary dataset or the anomaly injection code due to confidentiality requirements of the data provider.

Circularity Check

0 steps flagged

No circularity in empirical evaluation protocol

full rationale

The paper reports an empirical study of an ensemble anomaly detector evaluated via controlled synthetic anomaly injection on proprietary data. No mathematical derivation chain exists, and the provided text contains no self-definitional equations, fitted parameters renamed as independent predictions, or load-bearing self-citations that reduce claims to inputs by construction. Performance numbers are presented as outcomes of the described injection protocol rather than tautological restatements of calibration choices.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Framework rests on standard unsupervised anomaly detection methods plus domain rules for stale values; central performance claims depend on the representativeness of the injection protocol and calibration choices.

free parameters (1)
  • ensemble calibration parameters and decision thresholds = tuned per dataset for reported F1
    Calibrated on the anomaly-injected data to reach the stated 61-79% F1 range.
axioms (1)
  • domain assumption The eight operationally realistic anomaly scenarios cover the relevant error types that occur in live risk systems.
    Invoked when claiming the ensemble advantage generalizes beyond the test protocol.

pith-pipeline@v0.9.1-grok · 5757 in / 1289 out tokens · 45458 ms · 2026-06-26T15:04:08.660982+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages

  1. [1]

    Outlier ensembles: a position paper

    Aggarwal, C.C., 2013. Outlier ensembles: a position paper. ACM SIGKDD Explorations Newsletter 15 (1), 49–58. https://doi.org/10.1145/2481244.2481252

    work page doi:10.1145/2481244.2481252 2013

  2. [2]

    Aggarwal, C. C. (2017). Outlier analysis (2nd ed.). Springer. https://doi.org/10.1007/978-3-319-47578-3

    work page doi:10.1007/978-3-319-47578-3 2017

  3. [3]

    Theoretical foundations and algorithms for outlier ensembles

    Aggarwal, C.C., Sathe, S., 2015. Theoretical foundations and algorithms for outlier ensembles. ACM SIGKDD Explorations Newsletter 17 (1), 24–47. https://doi.org/10.1145/2830544.2830549

    work page doi:10.1145/2830544.2830549 2015

  4. [4]

    , Delbaen , Freddy F

    Artzner, P., Delbaen, F., Eber, J.-M., Heath, D., 1999. Coherent measures of risk. Mathematical Finance 9 (3), 203–228. https://doi.org/10.1111/1467-9965.00068

    work page doi:10.1111/1467-9965.00068 1999

  5. [5]

    Detecting anomalies in financial data using machine learning algorithms

    Bakumenko, A., Elragal, A., 2022. Detecting anomalies in financial data using machine learning algorithms. Systems 10 (5), 130. https://doi.org/10.3390/systems10050130

    work page doi:10.3390/systems10050130 2022

  6. [6]

    Sound practices for the management and supervision of operational risk

    Basel Committee on Banking Supervision, 2003. Sound practices for the management and supervision of operational risk. Bank for International Settlements, Basel

    2003

  7. [7]

    International Convergence of Capital Measurement and Capital Standards: A Revised Framework

    Basel Committee on Banking Supervision, 2006. International Convergence of Capital Measurement and Capital Standards: A Revised Framework. Bank for International Settlements, Basel

    2006

  8. [8]

    Basel III: A global regulatory framework for more resilient banks and banking systems

    Basel Committee on Banking Supervision, 2011. Basel III: A global regulatory framework for more resilient banks and banking systems. Bank for International Settlements, Basel

    2011

  9. [9]

    Minimum capital requirements for market risk (Rev

    Basel Committee on Banking Supervision, 2019. Minimum capital requirements for market risk (Rev. 2019). Bank for International Settlements, Basel

    2019

  10. [10]

    Data and Information Quality: Dimensions, Principles and Techniques

    Batini, C., Scannapieco, M., 2016. Data and Information Quality: Dimensions, Principles and Techniques. Springer, Cham. https://doi.org/10.1007/978-3-319-24106-7

    work page doi:10.1007/978-3-319-24106-7 2016

  11. [11]

    Operational risk is more systemic than you think: evidence from U.S

    Berger, A.N., Curti, F., Mihov, A., Sedunov, J., 2022. Operational risk is more systemic than you think: evidence from U.S. bank holding companies. Journal of Banking & Finance 143, 106619. https://doi.org/10.1016/j.jbankfin.2022.106619

    work page doi:10.1016/j.jbankfin.2022.106619 2022

  12. [12]

    SR 11-7: Supervisory guidance on model risk management

    Board of Governors of the Federal Reserve System, 2011. SR 11-7: Supervisory guidance on model risk management. Board of Governors of the Federal Reserve System, Washington, DC. 35

    2011

  13. [13]

    Breunig, Hans-Peter Kriegel, Raymond T

    Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J., 2000. LOF: identifying density-based local outliers. ACM SIGMOD Record 29 (2), 93–104. https://doi.org/10.1145/335191.335388

    work page doi:10.1145/335191.335388 2000

  14. [14]

    Anomaly detection: A survey,

    Chandola, V., Banerjee, A., Kumar, V., 2009. Anomaly detection: a survey. ACM Computing Surveys 41 (3), 1–58. https://doi.org/10.1145/1541880.1541882

    work page doi:10.1145/1541880.1541882 2009

  15. [15]

    Business complexity and risk management: evidence from operational risk events in U.S

    Chernobai, A., Ozdagli, A., Wang, J., 2021. Business complexity and risk management: evidence from operational risk events in U.S. bank holding companies. Journal of Monetary Economics 117, 418–440. https://doi.org/10.1016/j.jmoneco.2020.02.004

    work page doi:10.1016/j.jmoneco.2020.02.004 2021

  16. [16]

    Anomaly detection in financial time series by principal component analysis and neural networks

    Crépey, S., Lehdili, N., Madhar, N., Thomas, M., 2022. Anomaly detection in financial time series by principal component analysis and neural networks. Algorithms 15 (10), 385. https://doi.org/10.3390/a15100385

    work page doi:10.3390/a15100385 2022

  17. [17]

    Are the largest banking organizations operationally more risky? Journal of Money, Credit and Banking 54 (5), 1223–1259

    Curti, F., Frame, W.S., Mihov, A., 2022. Are the largest banking organizations operationally more risky? Journal of Money, Credit and Banking 54 (5), 1223–1259. https://doi.org/10.1111/jmcb.12933

    work page doi:10.1111/jmcb.12933 2022

  18. [18]

    Capital and risk: new evidence on implications of large operational losses

    De Fontnouvelle, P., DeJesus-Rueff, V., Jordan, J.S., Rosengren, E.S., 2006. Capital and risk: new evidence on implications of large operational losses. Journal of Money, Credit and Banking 38 (7), 1819–1846. https://doi.org/10.1353/mcb.2006.0088

    work page doi:10.1353/mcb.2006.0088 2006

  19. [19]

    Finding a needle in a haystack: A machine learning framework for anomaly detection in payment systems

    Desai A, Kosse A, Sharples J. Finding a needle in a haystack: A machine learning framework for anomaly detection in payment systems. J Financ Data Sci. 2025; 11:100163. https://doi.org/10.1016/j.jfds.2025.100163

    work page doi:10.1016/j.jfds.2025.100163 2025

  20. [20]

    From detection to action: a human-in-the-loop toolkit for anomaly reasoning and management

    Ding, X., Seleznev, N., Kumar, S., Bruss, C.B., 2023. From detection to action: a human-in-the-loop toolkit for anomaly reasoning and management. Proceedings of the 4th ACM International Conference on AI in Finance, pp. 1–10. https://doi.org/10.1145/3604237.3626872

    work page doi:10.1145/3604237.3626872 2023

  21. [21]

    unduly burdensome

    European Banking Authority. (2025). Final report: Draft regulatory technical standards on establishing a risk taxonomy on operational risk; specifying the condition of “unduly burdensome” for the calculation of the annual operational risk loss; and specifying how institutions shall determine adjustments to their loss data set following mergers or acquisit...

    2025

  22. [22]

    Guidance on supervisory interaction with financial institutions on risk culture: a framework for assessing risk culture

    Financial Stability Board, 2014. Guidance on supervisory interaction with financial institutions on risk culture: a framework for assessing risk culture. Financial Stability Board, Basel

    2014

  23. [23]

    Expert Systems with Applications193, 116429 (2022).https: //doi.org/https://doi.org/10.1016/j.eswa.2021.116429,https:// www.sciencedirect.com/science/article/pii/S0957417421017164

    Hilal, W., Gadsden, S.A., Yawney, J., 2022. Financial fraud: a review of anomaly detection techniques and recent advances. Expert Systems with Applications 193, 116429. https://doi.org/10.1016/j.eswa.2021.116429

    work page doi:10.1016/j.eswa.2021.116429 2022

  24. [24]

    Risk Management and Financial Institutions, 5th ed

    Hull, J.C., 2018. Risk Management and Financial Institutions, 5th ed. Wiley, Hoboken; 2018. ISBN 978-1- 119-44811-2

    2018

  25. [25]

    Value at Risk: The New Benchmark for Managing Financial Risk, 3rd ed

    Jorion, P., 2007. Value at Risk: The New Benchmark for Managing Financial Risk, 3rd ed. McGraw-Hill, New York; ISBN 978-0-07-146495-6

    2007

  26. [26]

    Machine learning in banking risk management: a literature review

    Leo, M., Sharma, S., Maddulety, K., 2019. Machine learning in banking risk management: a literature review. Risks 7 (1), 29. https://doi.org/10.3390/risks7010029 36

    work page doi:10.3390/risks7010029 2019

  27. [27]

    Isolation forest,

    Liu, F.T., Ting, K.M., Zhou, Z.-H., 2008. Isolation Forest. In: Proceedings of the IEEE 8th International Conference on Data Mining (ICDM 2008), pp. 413–422. https://doi.org/10.1109/ICDM.2008.17

    work page doi:10.1109/icdm.2008.17 2008

  28. [28]

    What is operational risk? FRBSF Economic Letter

    Lopez JA. What is operational risk? FRBSF Economic Letter. 2002;(2002-02). Federal Reserve Bank of San Francisco; 1–3. Available from: https://www.frbsf.org/research-and-insights/publications/economic-letter/ 2002/01/what-is-operational-risk/ [accessed 31 May 2026]

    2002

  29. [29]

    Data quality assessment

    Pipino, L.L., Lee, Y.W., Wang, R.Y., 2002. Data quality assessment. Communications of the ACM 45 (4), 211–218. https://doi.org/10.1145/505248.506010

    work page doi:10.1145/505248.506010 2002

  30. [30]

    System for automated quality control (SaQC) to enable traceable and reproducible data streams in environmental science

    Schmidt, L., Schäfer, D., Geller, J., Lünenschloss, P., Palm, B., Rinke, K., Bumberger, J., 2023. System for automated quality control (SaQC) to enable traceable and reproducible data streams in environmental science. Environmental Modelling & Software 165, 105809. https://doi.org/10.1016/j.envsoft.2023.105809

    work page doi:10.1016/j.envsoft.2023.105809 2023

  31. [31]

    JPMorgan Chase London Whale A: Risky Business

    Zeissler, A.G., Ikeda, D., Metrick, A., 2019. JPMorgan Chase London Whale A: Risky Business. Journal of Financial Crises 1 (2), 40–59; https://doi.org/10.17132/2693-3179.1013

    work page doi:10.17132/2693-3179.1013 2019