How large are your variables? I would hazard a guess that they are large, in the 10's or 100's MB or larger.
One of the problems with Fortran is that temporary variables typically get memory from the stack (relatively small memory available), rather than the heap (relatively large memory available). So calculations that immediately get used as arguments or in other calculations are temporary variables that take stack memory, whereas calculations that immediately get assigned to a variable do not ... such calculations are done in-place in the left-hand-side variable. So look at your two formulations:
matD = matmul(transpose(matB),matC*matB)
Here transpose(matB) is a temporary result that is only used as an argument, so the memory for it will typically be taken from the stack. Likewise, matC*matB is a temporary result that is only used for an argument, so its memory will also be taken from the stack. If they are too large, you will blow the stack memory (a fixed amount of memory much smaller than the heap) and the program will crash.
Now look at your second formulation:
matC = matC*matB
matD = matmul(transpose(matB),matC)
The matC*matB calculation is assigned to a variable. Thus, the result does not require additional stack memory ... the results can be done "in-place" in the matC variable (since the * is "element-wise multiplication" in Fortran). There is still an issue with the transpose(matB) calculation taking stack memory, but it appears that was not a tipping point for the crash.
As to why your Fortran stand-alone program worked fine but the mex routine using the same code failed, it is likely simply the result of having different amounts of stack memory available for the two different implementations. E.g., had you started MATLAB with some type of command that would increase its stack space, the mex routine may have worked fine.
That is one reason I tend to avoid certain constructs like using "any" on a temporary calculation result. E.g.,
if( any( A == B ) ) then
! etc
endif
If A and B are large, then the result A == B will be large and come from the stack, potentially crashing the program. To guard against this, I often fall back to F77 style programming because it is protected against this. E.g., the above could be coded as follows:
flag = .false.
do i=1,n
if( A(i) == B(i) ) then
flag = .true.
exit
endif
enddo
if( flag ) then
! etc
endif
Seems kind of backwards to have to resort to F77 style code to protect against stack overflows, but there it is. There might be compiler directives to tell the compiler to create temporary and local variables from the heap, but doing so ties your code to non-standard compiler directives. Rather than doing this, I tend to just fall back on F77 style coding because I know it will be safe for large variables. I don't want to have to rely on compiler directives, or increasing the stack size when starting the program, etc. I would rather write code from the outset that I know will work regardless of stack size (unless of course the heap runs out).
Another potential issue is the matB, matC, and matC variables themselves. If they are non-allocated variables then their memory will also typically come from the stack. So you should never do something like this in Fortran (actually you can have this exact problem in other languages such as C or C++ as well):
subroutine whatever(large_number)
integer large_number
real*8 X(large_number)
The reason is that X will take stack memory, and potentially crash the program. Local large variables should always be allocated since that memory will come from the heap. E.g.,
subroutine whatever(large_number)
integer large_number, status
real*8, allocatable :: X(:)
allocate( X(large_number), stat = status )
if( status /= 0 ) then
! handle the memory allocation error
endif
! code to use X
deallocate(X)
