The issue most likely comes from the incompatability regarding LP64 and ILP64 versions of BLAS/LAPACK libs.
Most libs are in LP64 mode by default in which "long int", "long long", "Pointer", and "size_t" are 64 bit, but "int" is 32 bit. MATLAB, however, only supports ILP64 where even "int" is 64 bit.
In your example, openBLAS, LAPACK and so on are most likely using the LP64 interface, but when called from within MATLAB, where we use BLAS/LAPACK libs that are compiled in ILP64 mode, whe have this type mismatch. Ideally, you just re-compile all the external pieces you need in ILP64 mode (SuperLU and ARPACK should have an option for that) and make use of the BLAS/LAPACK version shipping with MATLAB.
If that is not possible, you have to create a wrapper layer as following:
- create a wrapper layer via C++ MEX for the calls to the "armadillo" library
- link the needed LP64 libraries (including the LP64 BLAS/LAPACK versions) into the C++ MEX compilation such that calls to "amadillo" via the C++ MEX wrapper uses the LP64 libs (pretty much the same like you did for the standalone example)
- call the MEX layer from within MATLAB but OUT-OF-PROCESS (https://www.mathworks.com/help/matlab/matlab_external/out-of-process-execution-of-c-mex-functions.html)
- doing so, ensures that MATLAB itself uses the default ILP64 BLAS/LAPACK libs, but calls to "amadillo", from within MATLAB via C++ MEX, can use the LP64 libs as needed
Something that has been suggested wrongly in the past is to "LD_PRELOAD" the LP64 BLAS/LAPACK libs before starting MATLAB. Although that might work in very restricted cases, this would make MATLAB use those same LP64 libs inside MATLAB which will cause issues as you experienced when using core MATLAB functions and most likely will result in crashes. It is not supported to interact with LP64 versions of BLAS/LAPACK from inside of MATLAB.
