modelAccuracy

Compute R-square, RMSE, correlation, and sample mean error of predicted and observed EADs

Since R2021b

modelAccuracy is renamed to modelCalibration. modelAccuracy is not recommended. Use modelCalibration instead.

Syntax

AccMeasure = modelAccuracy(eadModel,data)

[AccMeasure,AccData] = modelAccuracy(___,Name=Value)

Description

AccMeasure = modelAccuracy(eadModel,data) computes the R-square, root mean square error (RMSE), correlation, and sample mean error of observed vs. predicted exposure at default (EAD) data. modelAccuracy supports comparison against a reference model and also supports different correlation types. By default, modelAccuracy computes the metrics in the EAD scale. You can use the ModelLevel name-value argument to compute metrics using the underlying model's transformed scale.

[AccMeasure,AccData] = modelAccuracy(___,Name=Value) specifies options using one or more name-value arguments in addition to the input arguments in the previous syntax.

Input Arguments

eadModel — Exposure at default model

Regression object | Tobit object | Beta object

Loss given default model, specified as a previously created Regression, Tobit, or Beta object using fitEADModel.

Data Types: object

data — Data

table

Data, specified as a NumRows-by-NumCols table with predictor and response values. The variable names and data types must be consistent with the underlying model.

Data Types: table

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: [AccMeasure,AccData] = modelAccuracy(eadModel,data(TestInd,:),DataID='Testing',CorrelationType='spearman')

CorrelationType — Correlation type

"pearson" (default) | character vector with value of 'pearson', 'spearman', or 'kendall' | string with value of "pearson", "spearman", or "kendall"

Correlation type, specified as CorrelationType and a character vector or string.

Data Types: char | string

DataID — Data set identifier

"" (default) | character vector | string

Data set identifier, specified as DataID and a character vector or string. The DataID is included in the output for reporting purposes.

Data Types: char | string

ModelLevel — Model level

'ead' (default) | character vector with value 'ead', 'conversionMeasure', or 'conversionTransform' | string with value "ead", "conversionMeasure", or "conversionTransform"

Model level, specified as ModelLevel and a character vector or string.

Note

Regression models support all three model levels, but a Tobit or Beta model supports model levels only for "ead" and "conversionMeasure".

Data Types: char | string

ReferenceEAD — EAD values predicted for data by reference model

[] (default) | numeric vector

EAD values predicted for data by the reference model, specified as ReferenceEAD and a NumRows-by-1 numeric vector. The modelAccuracy output information is reported for both the eadModel object and the reference model.

Data Types: double

ReferenceID — Identifier for the reference model

'Reference' (default) | character vector | string

Identifier for the reference model, specified as ReferenceID and a character vector or string. ReferenceID is used in the modelAccuracy output for reporting purposes.

Data Types: char | string

Output Arguments

AccMeasure — Accuracy measure

table

Accuracy measure, returned as a table with columns 'RSquared', 'RMSE', 'Correlation', and 'SampleMeanError'. AccMeasure has one row if only the eadModel accuracy is measured and it has two rows if reference model information is given. The row names of AccMeasure report the model ID and data ID (if provided).

AccData — Accuracy data

table

Accuracy data, returned as a table with observed EAD values, predicted EAD values, and residuals (observed minus predicted). Additional columns for predicted and residual values are included for the reference model, if provided. The ModelID and ReferenceID labels are appended in the column names.

More About

Model Accuracy

Model accuracy measures the accuracy of the predicted probability of EAD values using different metrics.

• R-squared — To compute the R-squared metric, modelAccuracy fits a linear regression of the observed EAD values against the predicted EAD values:

  $$EA{D}_{obs}=a+b\ast EA{D}_{pred}+\epsilon$$

  The R-square of this regression is reported. For more information, see Coefficient of Determination (R-Squared).

• RMSE — To compute the root mean square error (RMSE), modelAccuracy uses the following formula where N is the number of observations:

  $$RMSE=\sqrt{\frac{1}{N}{\displaystyle {\sum }_{i=1}^N(EA{D}_i^{obs}-EA{D}_i^{pred}}{)}^2}$$

• Correlation — This metric is the correlation between the observed and predicted EAD:

  $$corr(EA{D}_{obs},EA{D}_{pred})$$

  For more information and details about the different correlation types, see corr.

• Sample mean error — This metric is the difference between the mean observed EAD and the mean predicted EAD or, equivalently, the mean of the residuals:

  $$SampleMeanError=\frac{1}{N}{\displaystyle {\sum }_{i=1}^N(EA{D}_i^{obs}-EA{D}_i^{pred})}$$

References

[1] Baesens, Bart, Daniel Roesch, and Harald Scheule. Credit Risk Analytics: Measurement Techniques, Applications, and Examples in SAS. Wiley, 2016.

[2] Bellini, Tiziano. IFRS 9 and CECL Credit Risk Modelling and Validation: A Practical Guide with Examples Worked in R and SAS. San Diego, CA: Elsevier, 2019.

[3] Brown, Iain. Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Applications. SAS Institute, 2014.

[4] Roesch, Daniel and Harald Scheule. Deep Credit Risk. Independently published, 2020.

Version History

Introduced in R2021b

R2023a: modelAccuracy function is renamed to modelCalibration function

The modelAccuracy function is renamed to modelCalibration function. The use of modelAccuracy is not recommended, use modelCalibration instead.

R2022b: Support for Beta model

The eadModel input supports an option for a Beta model object that you can create using fitEADModel.