Introduction

Credit risk models, often referred to as scorecards and scores, are the linchpin of financial stability for both individuals and businesses. These models guide crucial decisions about lending, capital allocation, credit loss forecasting, and strategic planning for banks and lenders. Given their profound impact, credit risk models are under increasing scrutiny and regulation, with a growing emphasis on Model Risk Governance. This discipline ensures models are developed, utilized, and managed with a comprehensive understanding and without unintended consequences.

From model owners to regulators, all stakeholders need clear and concise reporting to navigate and govern an organization’s models effectively. This article delves into why robust reporting is essential and how it empowers stakeholders to make informed, confident decisions. 

Understanding Key Audiences 

Stakeholders in credit risk model development include: 

• Model Owners: Ensure the models serve their intended purpose. 

• Model Users: Need to understand and trust the model outputs. 

• Validators and Auditors: Require evidence of correct development processes and compliance. 

• Executives and Regulators: Demand clarity on reliability, performance, strong governance, and compliance. 
  

Essential Components of Model Reporting 

To meet the diverse needs of these stakeholders, comprehensive model reporting should encompass the model's development, functionality, and performance. Here, we categorize key reports into four areas: Data, Design and Development, Model Structure and Functionality, and Results and Performance. 

Data 

The foundation of any model lies in the integrity and reliability of the data used. Reports should cover data sources, validation checks, and preprocessing steps. Transparency in data enhances confidence in the model. The primary question to address is, “Is the data reliable and accurate?”

• Univariate Data Field Reports: These reports provide an overview of each data field, including splits by population segments and previous Accept/Reject decisions, Good, Bad outcomes etc. They provide an overview of the data, allowing quick and easy identification of any potential outliers or unexpected trends. Reports that use graphs, charts and colours are often far more effective in conveying information in a meaningful way than simply reporting rows and columns of numbers.   

Other useful data reports include Field Reductions reports and Correlation reports. 

• Field Reductions Reports: Detail how data fields are streamlined by removing redundant or highly correlated variables. A bar graph can illustrate which fields were excluded and why, providing a clear rationale for each decision. 

• Correlation Reports: Show relationships between variables using heatmaps. For instance, a correlation matrix can use a colour gradient to represent the strength of relationships between pairs of characteristics. 

Design and Development 

Model validators and auditors need to ensure the design aligns with expectations and regulations. For example, have the right type of exclusions been identified and applied correctly?  Do record volumes look correct?  Key reports include: 

• Population Flow Reports:  Help ensure that the design has been coded correctly.  Having the ability to easily report on defined segments of the population also enables checks on the integrity of the data and design. 

• Exclusions Reports: Detail the volumes and reasons for data exclusions. For Origination model developments this can be reported per ‘origination month’ to ensure volumes make sense and any trends can be explained.  

Model Structure and Functionality 

Understanding how a model functions is crucial. The following reports provide insights into model structure and how the model actually works.  Some reports are model agnostic and can be generated against any type of predictive model, whether tree-ensemble, neural network or statistical regression based, for example.  Other reports are specific to the methodology used, for example, Weight of Evidence (WoE) group reports for models developed using WoE based logistic regression: 

• Model Parameters Reports: These reports list all parameters used in the model, including their settings and justifications. A comprehensive table can show the parameters for different model types, alongside explanations for each setting. 

• Model Log Reports: Chronicle the development process, documenting every change and decision. A timeline can effectively communicate the evolution of the model from conception to deployment. 

• Partial Dependency Plot (PDP) Reports: Illustrate the effect of individual variables on the model output. These reports might use line graphs to show how changes in variables like ‘debt to income ratio’ impact the predicted risk score, providing a clear visual interpretation of complex interactions. 

• Weight of Evidence (WoE) Group Reports: Specific to logistic regression models using WoE, these reports break down how different variable groups contribute to the model. 

• SHAP Analysis Reports: Explain drivers of model outputs, particularly useful for complex models like Random Forest and XGBoost. SHAP (SHapley Additive exPlanations) values are often visualized using summary plots, which highlight the contribution of each feature/characteristic to a particular prediction. 

• Permutation Feature Importance Reports: To understand the main drivers of a model, the Permutation Feature importance report shows the importance of each scored field in every model across all populations. This type of report helps explain the main drivers of the model and also checks for consistency across the Development and Holdout samples to highlight any potential overfitting to the Development data.   

Performance and Results 

How a model performs is vital information for any model.  Essential reports include: 

• Gini Summary Tables: Summarize model performance across different populations and holdout samples. A table can show the Gini coefficients for multiple models and segments, alongside colour-coded performance indicators for easy comparison. 

• Gini Curve Reports: Graphically represent model performance with curves that illustrate how well the model discriminates between good and bad outcomes. Each line on the graph represents a different model or population, making it easy to compare performance visually. 

• Score Alignment Reports: Ensure score consistency across different segments and time periods, highlighting potential biases. These reports might use bar graphs to compare scores across demographic groups, ensuring fairness and stability over time. 

• Characteristic Alignment Reports: Verify that model inputs align correctly. Misaligned characteristics can impact model accuracy and therefore these reports can help maintain predictive power. Alignment ensures that risk levels match expected outcomes and therefore aiding risk assessment. 

• Score Distribution Reports: Graphically display score distributions across populations and outcomes. Histograms or density plots can show how scores are spread across different customer segments, highlighting areas where the model performs well or needs adjustment. 

• Calibration Curves and Volume Distributions: Check consistency between development and holdout samples. Calibration curves might use line graphs to compare predicted probabilities against actual outcomes, ensuring the model's predictions are accurate across different samples. 

Summary 

Effective reporting is pivotal in the model development process, fostering understanding and trust among stakeholders. Well-designed reports, supported by automation tools, ensure efficient report generation and allow more focus on interpreting results rather than producing reports. 

To learn more about how Modeller can support your model development and reporting needs, please contact us here.