Bias Mitigation in Credit Scoring by Reweighting
Bias mitigation is the process of removing bias from a data set or a model in order to make it fair. Bias mitigation usually follows a bias detection step, where a series of metrics are computed based on a data set or model predictions. Bias mitigation has three stages: pre-processing, in-processing, and post-processing. This example demonstrates a pre-processing method to mitigate bias in a credit scoring workflow. The example uses bias detection and bias mitigation functionality from the Statistics and Machine Learning Toolbox™. For a detailed example on bias detection, see the following example: Explore Fairness Metrics for Credit Scoring Model.
The bias mitigation method in this example is Reweighting which essentially reweights observations within a data set to guarantee fairness between different subgroups within a sensitive attribute. As a result of reweighting, the Statistical Parity Difference (SPD) of all subgroups goes to 0 and the Disparate Impact metric becomes 1. This example demonstrates how reweighting works in a credit scoring workflow.
Load Data
Load the CreditCardData data set and discretize the 'CustAge' predictor.
load CreditCardData.mat AgeGroup = discretize(data.CustAge,[min(data.CustAge) 30 45 60 max(data.CustAge)], ... 'categorical',{'Age < 30','30 <= Age < 45','45 <= Age < 60','Age >= 60'}); data = addvars(data,AgeGroup,'After','CustAge'); head(data)
CustID CustAge AgeGroup TmAtAddress ResStatus EmpStatus CustIncome TmWBank OtherCC AMBalance UtilRate status
______ _______ ______________ ___________ __________ _________ __________ _______ _______ _________ ________ ______
1 53 45 <= Age < 60 62 Tenant Unknown 50000 55 Yes 1055.9 0.22 0
2 61 Age >= 60 22 Home Owner Employed 52000 25 Yes 1161.6 0.24 0
3 47 45 <= Age < 60 30 Tenant Employed 37000 61 No 877.23 0.29 0
4 50 45 <= Age < 60 75 Home Owner Employed 53000 20 Yes 157.37 0.08 0
5 68 Age >= 60 56 Home Owner Employed 53000 14 Yes 561.84 0.11 0
6 65 Age >= 60 13 Home Owner Employed 48000 59 Yes 968.18 0.15 0
7 34 30 <= Age < 45 32 Home Owner Unknown 32000 26 Yes 717.82 0.02 1
8 50 45 <= Age < 60 57 Other Employed 51000 33 No 3041.2 0.13 0
Split the data set into training and testing data. Use the training data to fit the model and the testing data to predict from the model.
rng('default'); c = cvpartition(size(data,1),'HoldOut',0.3); data_Train = data(c.training(),:); data_Test = data(c.test(),:);
Compute Fairness Metrics at Predictor and Model Level
Compute the fairness metrics for the training data by creating a fairnessMetrics object and then generating a metrics report using report. Since you are only working with data and there is no fitted model, only two bias metrics are computed for StatisticalParityDifference and DisparateImpact. The two group metrics computed are GroupCount and GroupSizeRatio. The fairness metrics are computed for two sensitive attributes, Age ('AgeGroup') and Residential Status ('ResStatus').
trainingDataMetrics = fairnessMetrics(data_Train, 'status', 'SensitiveAttributeNames',{'AgeGroup', 'ResStatus'}); tdmReport = report(trainingDataMetrics)
tdmReport=7×4 table
SensitiveAttributeNames Groups StatisticalParityDifference DisparateImpact
_______________________ ______________ ___________________________ _______________
AgeGroup Age < 30 0.039827 1.1357
AgeGroup 30 <= Age < 45 0.096324 1.3282
AgeGroup 45 <= Age < 60 0 1
AgeGroup Age >= 60 -0.19181 0.34648
ResStatus Home Owner 0 1
ResStatus Tenant 0.01689 1.0529
ResStatus Other -0.02108 0.93404
figure tiledlayout(2,1) nexttile plot(trainingDataMetrics,'spd') nexttile plot(trainingDataMetrics,'di')
Looking at the DisparateImpact bias metric for both AgeGroup and ResStatus, you can see that there is a much larger variance in the AgeGroup predictor as compared to the ResStatus predictor. This suggests that users are treated more unfairly when it comes to their age as compared to their residential status. This example focuses on the AgeGroup predictor and attempts to reduce bias among its subgroups.
To begin, fit a credit scoring model and compute the model-level bias metrics. This provides a baseline for comparison.
Since CustAge and AgeGroup are essentially the same predictor and this is a sensitive attribute, you can exclude it from the model. Additionally, you can use 'status' as the response variable and 'CustID' as the ID variable.
PredictorVars = setdiff(data_Train.Properties.VariableNames, ... {'CustAge','AgeGroup','CustID','FairWeights','status'}); sc1 = creditscorecard(data_Train,'IDVar','CustID', ... 'PredictorVars',PredictorVars,'ResponseVar','status'); sc1 = autobinning(sc1); sc1 = fitmodel(sc1,'VariableSelection','fullmodel');
Generalized linear regression model:
logit(status) ~ 1 + TmAtAddress + ResStatus + EmpStatus + CustIncome + TmWBank + OtherCC + AMBalance + UtilRate
Distribution = Binomial
Estimated Coefficients:
Estimate SE tStat pValue
________ ________ ________ __________
(Intercept) 0.73924 0.077237 9.5711 1.058e-21
TmAtAddress 1.2577 0.99118 1.2689 0.20448
ResStatus 1.755 1.295 1.3552 0.17535
EmpStatus 0.88652 0.32232 2.7504 0.0059516
CustIncome 0.95991 0.19645 4.8862 1.0281e-06
TmWBank 1.132 0.3157 3.5856 0.00033637
OtherCC 0.85227 2.1198 0.40204 0.68765
AMBalance 1.0773 0.31969 3.3698 0.00075232
UtilRate -0.19784 0.59565 -0.33214 0.73978
840 observations, 831 error degrees of freedom
Dispersion: 1
Chi^2-statistic vs. constant model: 66.5, p-value = 2.44e-11
pointsinfo1 = displaypoints(sc1)
pointsinfo1=38×3 table
Predictors Bin Points
_______________ _________________ _________
{'TmAtAddress'} {'[-Inf,9)' } -0.17538
{'TmAtAddress'} {'[9,16)' } 0.05434
{'TmAtAddress'} {'[16,23)' } 0.096897
{'TmAtAddress'} {'[23,Inf]' } 0.13984
{'TmAtAddress'} {'<missing>' } NaN
{'ResStatus' } {'Tenant' } -0.017688
{'ResStatus' } {'Home Owner' } 0.11681
{'ResStatus' } {'Other' } 0.29011
{'ResStatus' } {'<missing>' } NaN
{'EmpStatus' } {'Unknown' } -0.097582
{'EmpStatus' } {'Employed' } 0.33162
{'EmpStatus' } {'<missing>' } NaN
{'CustIncome' } {'[-Inf,30000)' } -0.61962
{'CustIncome' } {'[30000,36000)'} -0.10695
{'CustIncome' } {'[36000,40000)'} 0.0010845
{'CustIncome' } {'[40000,42000)'} 0.065532
⋮
pd1 = probdefault(sc1,data_Test);
Set the threshold value that controls the allocation of "goods" and "bads."
threshold = 
0.35;
predictions1 = double(pd1>threshold);Create a fairnessMetrics object to compute fairness metrics at the model level and then generate a metrics report using report.
modelMetrics1 = fairnessMetrics(data_Test, 'status', 'Predictions', predictions1, 'SensitiveAttributeNames','AgeGroup'); mmReport1 = report(modelMetrics1)
mmReport1=4×7 table
ModelNames SensitiveAttributeNames Groups StatisticalParityDifference DisparateImpact EqualOpportunityDifference AverageAbsoluteOddsDifference
__________ _______________________ ______________ ___________________________ _______________ __________________________ _____________________________
Model1 AgeGroup Age < 30 0.54312 2.6945 0.47391 0.5362
Model1 AgeGroup 30 <= Age < 45 0.19922 1.6216 0.35645 0.22138
Model1 AgeGroup 45 <= Age < 60 0 1 0 0
Model1 AgeGroup Age >= 60 -0.15385 0.52 -0.18323 0.16375
Measure accuracy of model using validatemodel.
validatemodel(sc1)
ans=4×2 table
Measure Value
________________________ _______
{'Accuracy Ratio' } 0.33751
{'Area under ROC curve'} 0.66876
{'KS statistic' } 0.26418
{'KS score' } 1.0403
figure tiledlayout(2,1) nexttile plot(modelMetrics1,'spd') nexttile plot(modelMetrics1,'di')
Reweight Data at Predictor and Model Level
Use fairnessWeights to reweight the training data to remove bias for the sensitive attribute 'AgeGroup'.
fairWeights = fairnessWeights(data_Train, 'AgeGroup', 'status'); data_Train.FairWeights = fairWeights; head(data_Train)
CustID CustAge AgeGroup TmAtAddress ResStatus EmpStatus CustIncome TmWBank OtherCC AMBalance UtilRate status FairWeights
______ _______ ______________ ___________ __________ _________ __________ _______ _______ _________ ________ ______ ___________
1 53 45 <= Age < 60 62 Tenant Unknown 50000 55 Yes 1055.9 0.22 0 0.95879
2 61 Age >= 60 22 Home Owner Employed 52000 25 Yes 1161.6 0.24 0 0.75407
3 47 45 <= Age < 60 30 Tenant Employed 37000 61 No 877.23 0.29 0 0.95879
4 50 45 <= Age < 60 75 Home Owner Employed 53000 20 Yes 157.37 0.08 0 0.95879
7 34 30 <= Age < 45 32 Home Owner Unknown 32000 26 Yes 717.82 0.02 1 0.82759
8 50 45 <= Age < 60 57 Other Employed 51000 33 No 3041.2 0.13 0 0.95879
9 50 45 <= Age < 60 10 Tenant Unknown 52000 25 Yes 115.56 0.02 1 1.0992
10 49 45 <= Age < 60 30 Home Owner Unknown 53000 23 Yes 718.5 0.17 1 1.0992
Use fairnessMetrics to compute fairness metrics for the training data after reweighting and use report to generate a fairness metrics report..
trainingDataMetrics_AfterReweighting = fairnessMetrics(data_Train, 'status', 'SensitiveAttributeNames','AgeGroup','Weights','FairWeights'); tdmrReport = report(trainingDataMetrics_AfterReweighting)
tdmrReport=4×4 table
SensitiveAttributeNames Groups StatisticalParityDifference DisparateImpact
_______________________ ______________ ___________________________ _______________
AgeGroup Age < 30 -2.9976e-15 1
AgeGroup 30 <= Age < 45 -5.5511e-16 1
AgeGroup 45 <= Age < 60 0 1
AgeGroup Age >= 60 -2.9421e-15 1
By applying the reweighting algorithm to the AgeGroup predictor, you can completely remove the disparate impact for AgeGroup. Then use this debiased data to fit a model to produce predictions with an overall reduced disparate impact at the model level.
Use creditscorecard to fit a new credit scoring model with the new fair weights and compute model-level bias metrics.
sc2 = creditscorecard(data_Train,'IDVar','CustID', ... 'PredictorVars',PredictorVars,'WeightsVar','FairWeights','ResponseVar','status'); sc2 = autobinning(sc2); sc2 = fitmodel(sc2,'VariableSelection','fullmodel');
Generalized linear regression model:
logit(status) ~ 1 + TmAtAddress + ResStatus + EmpStatus + CustIncome + TmWBank + OtherCC + AMBalance + UtilRate
Distribution = Binomial
Estimated Coefficients:
Estimate SE tStat pValue
________ ________ ________ __________
(Intercept) 0.74055 0.076222 9.7158 2.5817e-22
TmAtAddress 1.3416 0.9108 1.473 0.14075
ResStatus 2.0467 1.7669 1.1584 0.24672
EmpStatus 0.91879 0.32197 2.8536 0.0043222
CustIncome 0.91038 0.33216 2.7407 0.00613
TmWBank 1.1067 0.30826 3.5901 0.0003305
OtherCC 0.42264 3.5078 0.12049 0.9041
AMBalance 1.1347 0.3447 3.2919 0.00099504
UtilRate -0.39861 0.77284 -0.51577 0.60601
840 observations, 831 error degrees of freedom
Dispersion: 1
Chi^2-statistic vs. constant model: 46.6, p-value = 1.85e-07
pointsinfo2 = displaypoints(sc2)
pointsinfo2=34×3 table
Predictors Bin Points
_______________ _________________ ________
{'TmAtAddress'} {'[-Inf,9)' } -0.21328
{'TmAtAddress'} {'[9,23)' } 0.07168
{'TmAtAddress'} {'[23,Inf]' } 0.14763
{'TmAtAddress'} {'<missing>' } NaN
{'ResStatus' } {'Tenant' } 0.016048
{'ResStatus' } {'Home Owner' } 0.091092
{'ResStatus' } {'Other' } 0.28326
{'ResStatus' } {'<missing>' } NaN
{'EmpStatus' } {'Unknown' } -0.10352
{'EmpStatus' } {'Employed' } 0.33653
{'EmpStatus' } {'<missing>' } NaN
{'CustIncome' } {'[-Inf,30000)' } -0.37618
{'CustIncome' } {'[30000,40000)'} 0.047483
{'CustIncome' } {'[40000,42000)'} 0.10244
{'CustIncome' } {'[42000,47000)'} 0.14652
{'CustIncome' } {'[47000,Inf]' } 0.40015
⋮
pd2 = probdefault(sc2,data_Test); predictions2 = double(pd2>threshold);
Use fairnessMetrics to compute fairness metrics at the model level and report to generate a fairness metrics report.
modelMetrics2 = fairnessMetrics(data_Test, 'status', 'Predictions', predictions2, 'SensitiveAttributeNames','AgeGroup'); mmReport2 = report(modelMetrics2)
mmReport2=4×7 table
ModelNames SensitiveAttributeNames Groups StatisticalParityDifference DisparateImpact EqualOpportunityDifference AverageAbsoluteOddsDifference
__________ _______________________ ______________ ___________________________ _______________ __________________________ _____________________________
Model1 AgeGroup Age < 30 0.39394 2.1818 0.37391 0.39377
Model1 AgeGroup 30 <= Age < 45 0.094298 1.2829 0.22947 0.11509
Model1 AgeGroup 45 <= Age < 60 0 1 0 0
Model1 AgeGroup Age >= 60 -0.13333 0.6 -0.18323 0.1511
Measure accuracy of model using validatemodel.
validatemodel(sc2)
ans=4×2 table
Measure Value
________________________ _______
{'Accuracy Ratio' } 0.27735
{'Area under ROC curve'} 0.63868
{'KS statistic' } 0.22702
{'KS score' } 0.90741
figure tiledlayout(2,1) nexttile plot(modelMetrics2,'spd') nexttile plot(modelMetrics2,'di')
The process of reweighting removed all the bias from the training data. When you use the new data to fit a model, the overall bias in the model is reduced when compared to a model trained with biased data. As a consequence of this reduction in bias, there is a drop in model accuracy. You can choose to make tradeoff to improve fairness.
References
[1] Nielsen, Aileen. "Chapter 4. Fairness PreProcessing." Practical Fairness. O'Reilly Media, Inc., Dec. 2020.
[2] Mehrabi, Ninareh, et al. “A Survey on Bias and Fairness in Machine Learning.” ArXiv:1908.09635 [Cs], Sept. 2019. arXiv.org, https://arxiv.org/abs/1908.09635.
[3] Wachter, Sandra, et al. Bias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU Non-Discrimination Law. SSRN Scholarly Paper, ID 3792772, Social Science Research Network, 15 Jan. 2021. papers.ssrn.com, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3792772.
See Also
creditscorecard | autobinning | fitmodel | displaypoints | probdefault
