What is Model Risk Quantification and why is it needed?

Understanding Model Risk Quantification

Model risk quantification (MRQ) is the process of assessing and measuring the uncertainty, limitations, and ‘potential for errors’ when using a model. It provides organisations with a structured approach to understanding how much risk a model introduces and enables the implementation of appropriate controls to mitigate its impact.

In the MRQ definition above I purposefully did not say “potential errors within a model”. It’s important to be clear that what we are actually interested in is “potential for error when using and interpreting the outputs of a model”. It is a subtle but, I believe, very important difference. There may be no obvious errors within a model, but it is important to recognise that this type of model still has the potential for adverse consequences when being used by an organisation. This is what we mean by Model Risk, and it arises when a model fails to perform as expected, leading to inaccurate outputs that can result in financial loss, poor decision-making, regulatory non-compliance, or reputational damage.

Common Sources of Model Risk

Models can ‘go wrong’ for a variety of reasons, which are not limited to:

• Data: Poor data quality, missing values, or biased data can distort model outcomes.

• Assumptions: Incorrect or oversimplified assumptions can lead to poor models or poor interpretation of model outputs.

• Modelling: A model that is too closely fitted to past data may not generalise well to new data.

• Implementation: Mistakes in translating a developed model into production can lead to unexpected results.

• Changes Over Time: Changes in the underlying data or business environment can make a model ineffective over time.

Why is Model Risk Quantification Needed?

Several key factors drive the necessity for banks to prioritise MRQ:

• Regulatory Compliance: Regulatory bodies globally, including the Federal Reserve (SR 11-7), the European Central Bank (TRIM), the Central Bank of the UAE (MMS and MMG) and the Bank of England's PRA (SS1/23), increasingly mandate rigorous model risk management and quantification. These frameworks emphasise the need for comprehensive model validation and an understanding of model limitations.

• Enhanced Decision-Making: Banks depend on models for significant strategic and operational decisions. Without adequate MRQ, there is a potential for over-reliance on models with inherent flaws, leading to suboptimal outcomes. Quantifying potential uncertainties allows for more informed decision-making with a clearer understanding of associated risks.

• Mitigating Financial and Reputational Losses: History demonstrates the substantial losses that can arise from model failures. Implementing MRQ practices helps identify and address model weaknesses proactively, thereby reducing the likelihood of significant financial or reputational damage.

• Strengthening Governance and Risk Culture: A robust MRQ framework fosters a stronger model risk management culture within an organisation, promoting transparency, accountability, and consistency in model management. It enables firms to prioritise model reviews, allocate resources effectively, and strengthen governance standards.

How is Model Risk Quantified?

Model risk quantification involves a combination of qualitative assessments and quantitative techniques across different areas from where model risk can arise. Some key approaches include:

• Data Quality Risk: Data profiling and anomaly detection to identify inconsistencies and missing values in datasets.

• Assumption Risk: Sensitivity analysis to measure how variations in model assumptions impact outcomes.

• Performance and Stability Risk: Out of time validations to compare model predictions with actual outcomes over different time periods.

• Implementation and Operational Risk: Code review and automated testing to detect implementation errors in model deployment.

• Regulatory and Compliance Risk: Regulatory stress testing to ensure that models meet required standards under adverse conditions.

• Complexity and Interpretability Risk: Explainability techniques such as SHAP or Partial Dependency Plots to assess whether model outputs can be understood and interpreted by stakeholders.

Should These Dimensions Be Combined or Kept Separate?

While each model risk dimension provides unique insights, consolidating them into an overall model risk measure can offer a more comprehensive view. A combined risk score can aid in decision-making by providing a single, interpretable metric for stakeholders.

However, maintaining visibility into the individual components is crucial for diagnosing specific areas of risk and implementing targeted mitigation strategies. A hybrid approach where individual dimensions are assessed separately, but also contribute to an aggregate model risk score, offers the best of both worlds ensuring both transparency and holistic risk management.

Conclusion

Model Risk Quantification is not merely a regulatory requirement but a fundamental component of best practice model risk management practices. By thoroughly understanding and quantifying the potential risks embedded within a portfolio of models, financial institutions can fully trust their models and decision-making processes, mitigate potential losses, and foster a stronger model risk management culture. The implementation of a structured framework that provides a comprehensive view of model usage, performance, and associated model risks can significantly contribute to achieving these objectives.

Discover how Focus can transform your model risk management. Contact us to learn more.