Differences of grid search and random search when using 'fitrnet'

I was trying to use the 'fitrnet' to optimize the hyperparameters for regression neural network and I have been attempting all the optimizers provided in the toolbox.

Just a very small question regarding the 'gridsearch' used in the 'fitrnet'. On the mathworks website, it writes that "'gridsearch' searches in a random order, using uniform sampling without replacement from the grid.", which is quite confusing. Because as far as I understand, the grid search should calculate the loss function for each point in the multi-dimensional space. However here it mentions that it searches in a random order... Or should I interpret it as it chooses the combinations of the hyperparameters randomly from the grid while the random search does not select the combinations from the grid?

Does anyone know about this? Because it is a new function introduced only in R2021b, there is no answer to this on the internet, and the description on mathworks document is very confusing...

Many thanks in advance.

Answer 1

[Siraj]

Hii! @Xinlin Ye

It is my understanding that you want to know the difference between grid search and random search when using “fitrnet”.

I believe that you are trying to optimize the neural network by using the “OptimizeHyperparameters” argument. This argument causes “fitrnet” to search for hyperparameters that give a model with low cross-validation error.

Grid search creates a grid with “NumGridDivisions” values per dimension. Then 'gridsearch' performs uniform sampling without replacement from the hyperparameter grid. This means that it randomly selects combinations of hyperparameters from the grid, ensuring that each combination is used only once. The order in which the combinations are evaluated is random.

For grid search “ NumGridDivisions” specifies the number of values in each dimension. The value can be a vector of positive integers giving the number of values for each dimension, or a scalar that applies to all dimensions. This field is ignored for categorical variables, because for categorical values all distinct categories are included in the grid.

Whereas “random search” searches at random until the maximum number of objective function evaluations reaches. The maximum number of objective function evaluations is specified by the parameter “MaxObjectiveEvaluations”.

For more details, please refer to the following link.

https://in.mathworks.com/help/stats/fitrnet.html#mw_c3fbad89-4bde-419e-a252-28c29de1faf6

Hope this helps.