How to stack multiple regression type ML models to form an ensemble?

Hi everyone,

I have set up different regression type machine learning models (GaussianSVM, medium decision tree, linear robust model showed best accuracy) to predict a target value using identical (numerical) features in all three models. I am now wondering, if and how it is possible to stack models to further increase accuracy. I am thinking of using these three models as first layer and another linear model or SVM for the second layer of the ensemble.

Does it make sense to use different features in the models of the first layer of the ensemble and combine / weight them in the second ensemble layer? Do all features have to be numerical or could one model use classifying features? Where exactly would I find the 'weights' of the first layer models to be used as features in the second layer of the ensemble and how would I incorporate the classification model?

Since I am quite new in ML with no IT background, I would appreciate a simple step by step approach ;o)

Thank you so much!

Answer 1

[Anshika Chaurasia]

Hi Kathrin,

It is my understanding that you want to implement stacking ensemble technique.

In your case, train Gaussian SVM, medium decision tree and linear robust regression models and get the predictions from them. Then feed these predictions as input features to the second layer of another linear model or SVM. This is how you can stack your multiple regression type ML model.

For the first layer input you could experiment by playing with same or different features. By doing this you will know which feature is important for which model and how it is affecting the accuracy.

In stacked ensemble you have to give prediction as features to the second layer input. So, there is no need of weights of first layer.

You could incorporate classification model in second layer if you want. In that case, the classification model will take predictions of first layer as input.

Refer to Ensemble Learning Toolbox for the implementation of stacking ensemble technique.

[Kathrin]

Hi Anshika,

thank you for your answer and the link to the toolbox.

I will try to apply it.

Kind regards,

Kathrin