What would be the case "trainbr"( Bayesian regularized NN) perform better than others gradient based optimizers(even Adam)?

What would be the case "trainbr"( Bayesian regularized NN) perform better than others gradient based optimizers(even Adam)?

Matlab Deep Learning tool box provide "trainbr" optimizer and as well as others gradient optimizers.

I am aware of Bayesian regularization (BR) optimizer ("trainbr") is based on Levenberg-Marquardt optimizations, therefore, computationally expressive to train even simple MLP model compared to other gradient based optimizers(such as Adam). For now let's put aside the computational expenses to train the model but only focus the performance of the model.

My dataset has input of both categorical and numerical inputs (mixed) and the output is numerical, when I trained simple NN using “trainbr” it gives me better prediction accuracy than other gradient based optimizers. I am wondering what would be a reasonable explanation for this situation to happen.

Answer 1

[Avadhoot]

Hi Wabi,

There could be several reasons why "trainbr" performed better than all the other alternative strategies in your case. I have mentioned a few prominent causes below:

1. Regularization: "trainbr" provided in-build regularization as it is a part of its strategy. This prevents overfitting and leads to more accurate predictions on smaller datasets.
2. Noise handling: "trainbr" is robust and resistant to noise in the dataset, which helps in case the dataset contains significant amount of noisy data.
3. Data Complexity: If a dataset contains complex data of various datatypes where the relationships between variables are complex and non-linear, "trainbr" helps as it has an implicit averaging functionality which helps in minimizing the ill effects of complex data.
4. Smaller datasets: "trainbr" can work with smaller datasets as well as it calculates a posterior distribution based on the prior probabilities and the likelihood.
5. Convergence: "trainbr" is based on Levenberg-Marquardt optimizations which is a trust-region method. It leads to faster and more stable convergence in case of problems well-suited to second order methods. Here it outperforms algorithms like ADAM which are based on first order methods.

For more information on the functionality of "trainbr", refer to the below documentation:

https://www.mathworks.com/help/deeplearning/ref/trainbr.html

I hope this proves helpful