A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.
Model Validation Practice in Banking: A Structured Approach for Predictive Models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper presents a comprehensive overview of model validation practices and advancement in the banking industry based on the experience of managing Model Risk Management (MRM) since the inception of regulatory guidance SR11-7/OCC11-12 over a decade ago. Model validation in banking is a crucial process designed to ensure that predictive models, which are often used for credit risk, fraud detection, and capital planning, operate reliably and meet regulatory standards. This practice ensures that models are conceptually sound, produce valid outcomes, and are consistently monitored over time. Model validation in banking is a multi-faceted process with three key components: conceptual soundness evaluation, outcome analysis, and on-going monitoring to ensure that the models are not only designed correctly but also perform reliably and consistently in real-world environments. Effective validation helps banks mitigate risks, meet regulatory requirements, and maintain trust in the models that underpin critical business decisions.
citation-role summary
background 1
citation-polarity summary
fields
stat.ML 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
How to Choose a Threshold for an Evaluation Metric for Large Language Models
A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.