modelAccuracy

Compute R-square, RMSE, correlation, and sample mean error of predicted and observed EADs

Since R2021b

`modelAccuracy` is renamed to `modelCalibration`. `modelAccuracy` is not recommended. Use `modelCalibration` instead.

Syntax

``AccMeasure = modelAccuracy(eadModel,data)``
``[AccMeasure,AccData] = modelAccuracy(___,Name=Value)``

Description

````AccMeasure = modelAccuracy(eadModel,data)` computes the R-square, root mean square error (RMSE), correlation, and sample mean error of observed vs. predicted exposure at default (EAD) data. `modelAccuracy` supports comparison against a reference model and also supports different correlation types. By default, `modelAccuracy` computes the metrics in the EAD scale. You can use the `ModelLevel` name-value argument to compute metrics using the underlying model's transformed scale.```
````[AccMeasure,AccData] = modelAccuracy(___,Name=Value)` specifies options using one or more name-value arguments in addition to the input arguments in the previous syntax.```

Input Arguments

collapse all

Loss given default model, specified as a previously created `Regression`, `Tobit`, or `Beta` object using `fitEADModel`.

Data Types: `object`

Data, specified as a `NumRows`-by-`NumCols` table with predictor and response values. The variable names and data types must be consistent with the underlying model.

Data Types: `table`

Name-Value Arguments

Specify optional pairs of arguments as `Name1=Value1,...,NameN=ValueN`, where `Name` is the argument name and `Value` is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: ```[AccMeasure,AccData] = modelAccuracy(eadModel,data(TestInd,:),DataID='Testing',CorrelationType='spearman')```

Correlation type, specified as `CorrelationType` and a character vector or string.

Data Types: `char` | `string`

Data set identifier, specified as `DataID` and a character vector or string. The `DataID` is included in the output for reporting purposes.

Data Types: `char` | `string`

Model level, specified as `ModelLevel` and a character vector or string.

Note

`Regression` models support all three model levels, but a `Tobit` or `Beta` model supports model levels only for `"ead"` and `"conversionMeasure"`.

Data Types: `char` | `string`

EAD values predicted for `data` by the reference model, specified as `ReferenceEAD` and a `NumRows`-by-`1` numeric vector. The `modelAccuracy` output information is reported for both the `eadModel` object and the reference model.

Data Types: `double`

Identifier for the reference model, specified as `ReferenceID` and a character vector or string. `ReferenceID` is used in the `modelAccuracy` output for reporting purposes.

Data Types: `char` | `string`

Output Arguments

collapse all

Accuracy measure, returned as a table with columns `'RSquared'`, `'RMSE'`, `'Correlation'`, and `'SampleMeanError'`. `AccMeasure` has one row if only the `eadModel` accuracy is measured and it has two rows if reference model information is given. The row names of `AccMeasure` report the model ID and data ID (if provided).

Accuracy data, returned as a table with observed EAD values, predicted EAD values, and residuals (observed minus predicted). Additional columns for predicted and residual values are included for the reference model, if provided. The `ModelID` and `ReferenceID` labels are appended in the column names.

collapse all

Model Accuracy

Model accuracy measures the accuracy of the predicted probability of EAD values using different metrics.

• R-squared — To compute the R-squared metric, `modelAccuracy` fits a linear regression of the observed EAD values against the predicted EAD values:

`$EA{D}_{obs}=a+b\ast EA{D}_{pred}+\epsilon$`

The R-square of this regression is reported. For more information, see Coefficient of Determination (R-Squared).

• RMSE — To compute the root mean square error (RMSE), `modelAccuracy` uses the following formula where N is the number of observations:

`$RMSE=\sqrt{\frac{1}{N}{\sum }_{i=1}^{N}\left(EA{D}_{i}^{obs}-EA{D}_{i}^{pred}{\right)}^{2}}$`

• Correlation — This metric is the correlation between the observed and predicted EAD:

`$corr\left(EA{D}_{obs},EA{D}_{pred}\right)$`

For more information and details about the different correlation types, see `corr`.

• Sample mean error — This metric is the difference between the mean observed EAD and the mean predicted EAD or, equivalently, the mean of the residuals:

`$SampleMeanError=\frac{1}{N}{\sum }_{i=1}^{N}\left(EA{D}_{i}^{obs}-EA{D}_{i}^{pred}\right)$`

References

[1] Baesens, Bart, Daniel Roesch, and Harald Scheule. Credit Risk Analytics: Measurement Techniques, Applications, and Examples in SAS. Wiley, 2016.

[2] Bellini, Tiziano. IFRS 9 and CECL Credit Risk Modelling and Validation: A Practical Guide with Examples Worked in R and SAS. San Diego, CA: Elsevier, 2019.

[3] Brown, Iain. Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Applications. SAS Institute, 2014.

[4] Roesch, Daniel and Harald Scheule. Deep Credit Risk. Independently published, 2020.

Version History

Introduced in R2021b

expand all