How does one apply max(0,z) element-wise in a vectorized way (apply ReLu linear rectifier element-wise)

I wanted to implement something similar to applying a mathematical function element-wise to a matrix. For example something like:

kernel_matrix = max(0, W' * X + b) %linear rectifier element-wise vectorized code

in a manner that the code is vectorized. Is this possible in matlab? So far the max documentation doesn't seem super helpful.

Ideally, it would be important that this element-wise way of doing this is supported by GPU arrays too.

So for example if our matrix we want to pass through the ReLu is:

>> A

A =

       8     1     6
       3     5     7
       4     9     2

and the following is the threshold the neurons/ReLu should activate:

B =

     1     7     9
     3     1     1
     8     4     4

then the ReLu should only activate when A(i,j) - B(i,j) >= 0 to produce:

activation =

       8     0     0
       3     5     7
       0     9     0

[Roger Stafford]

It isn't clear what matrix the term "element-wise" is to apply to. Please give information about the sizes of the various variables here and what "element-wise" is to apply to.

[Brando Miranda]

it should just work just like

exp(A)

on any matrix, the dimensions or size, elements don't matter.

I think figured out how to do it though. Will answer now.

Answer 1

[Brando Miranda]

The only way that occurred to me how to do it was to simply compare the matrix with the number 0 (or the individual thresholds if you are dealing with a ReLu) and then multiply the original matrix with the logical matrix. In particular:

Z = rand(D,N); %pre-activation matrix
logical_matrix = (Z >= 0); %which entries should activate neurons
A = Z .* logial_matrix;

if you are dealing with a threshold though and want to activate for individual threshold, then one could do instead:

X = rand(D,N); %X original data matrix
logical_matrix = ((W' * Z - B) >= 0); %which entries should activate neurons based on threshold B
A = (W' * Z + B) .* logial_matrix;

the idea is to implement ReLu(w' * x + b) for element wise matrices which the above does.

I have not checked if its fast in GPUs but it should be better than doing a for loop if one has access to GPU. Will check shortly.

Answer 2

[Joss Knight]

Not quite sure whether your point is clear because this is exactly how the two-argument form of max works - element-wise.

>> max(0, magic(3)-5)

ans =

       3     0     1
       0     0     2
       0     4     0