How can I launch a parallel pool on Amazon Web Services cloud using Java compiled library - getting error Undefined function 'dct_psfcns' for input arguments of type 'char'

I have written the following function:

-------------------

function [output]=poolset

parpool;

p=parallel.defaultClusterProfile;

       [output]=p;
  end

-----------------

and am getting the following java error:

-----------------

ubuntu@ip-172-31-44-19:/var/lib/tomcat7/webapps/computing/WEB-INF/lib$ java -cp /var/lib/tomcat7/webapps/computing/WEB-INF/lib/CPFunctions-1.0.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/cp_commons-0.0.1-SNAPSHOT.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/cp_computing_domain-0.0.1-SNAPSHOT.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/javabuilder-1.0.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/gson-2.2.4.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/log4j-api-2.0-rc1.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/cp_computing_service_test.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/cp_computing_service-0.0.1-SNAPSHOT.jar:/var/lib/tomcat7/webapps/computing/WEB-INF/lib/junit-4.8.2.jar:./*.jar org.junit.runner.JUnitCore com.cp.service.computing.CPFunctionsPoolSetExExamples
JUnit version 4.8.2
ERROR StatusLogger Unable to locate a logging implementation, using SimpleLogger
.Error using parpool (line 99)
Undefined function 'dct_psfcns' for input arguments of type 'char'.

Error in poolset (line 14)

... Matlab M-code Stack Trace ...
file /home/ubuntu/.mcrCache8.4/CPFunc1/toolbox/distcomp/cluster/parpool.m, name parpool, line 99.
file /home/ubuntu/.mcrCache8.4/CPFunc1/Users/J/Box Sync/CapitalPreferences_LTD/Preference Engine/Algorithm Development/Matlab Files/poolset.m, name poolset, line 14.
com.mathworks.toolbox.javabuilder.MWException: Undefined function 'dct_psfcns' for input arguments of type 'char'.
  at com.mathworks.toolbox.javabuilder.internal.MWMCR.mclFeval(Native Method)
  at com.mathworks.toolbox.javabuilder.internal.MWMCR.access$600(MWMCR.java:23)
  at com.mathworks.toolbox.javabuilder.internal.MWMCR$6.mclFeval(MWMCR.java:839)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.mathworks.toolbox.javabuilder.internal.MWMCR$5.invoke(MWMCR.java:737)
  at com.sun.proxy.$Proxy2.mclFeval(Unknown Source)
  at com.mathworks.toolbox.javabuilder.internal.MWMCR.invoke(MWMCR.java:412)
  at CPFunctions.PoolSetClass.poolset(PoolSetClass.java:228)
  at com.cp.service.computing.CPFunctionsPoolSetExExamples.testPoolSetEx(CPFunctionsPoolSetExExamples.java:27)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
  at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
  at org.junit.runners.Suite.runChild(Suite.java:128)
  at org.junit.runners.Suite.runChild(Suite.java:24)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
  at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
  at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
  at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
  at org.junit.runner.JUnitCore.runMain(JUnitCore.java:98)
  at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:53)
  at org.junit.runner.JUnitCore.main(JUnitCore.java:45)
Undefined function 'dct_psfcns' for input arguments of type 'char'.
................

The error is occurring in the function when I try to assign the path for the pool profile to a variable. Do I need to add in the matlab function 'dct_psfcns' to my compiler? I cannot confirm whether or not the pool launched. This was a test function. Our end goal is to be able to dynamically adjust the number of workers based on the number of cores we have in our AWS instance, adjusting for load.

Thank you in advance for the assistance?

Answer 1

[Elwin Chan]

Hi there,

On which platform did you do the compilation? Was that also linux? dct_psfcns is a mex file that ships with Parallel Computing Toolbox. Its absence from your compiled .jar suggests that you may have compiled on a different platform from linux.

The error occurred during pool creation (in the parpool.m function), so the pool did not launch.

Just out of interest, can you elaborate more on your use of AWS here? Is your intention that the pool that is launched is running on the 'local' profile? Or some other cluster type?

Thanks,

Elwin

[J. Womack]

Hello Elwin,

Thank you for your response.

We are using AWS as the backend for a SaaS service. One server is our dedicated computing engine, which is where we are trying to launch the pool. We are trying to set up a local profile, but have not been able to get any parallel functions to run. My developer told me we're using a Linux box, so that's a part of the problem. I am compiling on a PC.

At the end of the day, we want to be able to set up a function that can initialize a pool using a load factor as the only input (i.e. if you have 16 cores with 32 logical units we can operate at 60% capacity and adjust upward as our load increases).

Is t here a way to load Linux components as part of a jar compiled on a PC?

[Elwin Chan]

Hi there,

The easiest thing is to compile the .jar on a linux box so that it picks up the correct platform-specific files. Your other option is to manually include the dct_psfcns function for linux into your generated .jar. You can find this in <matlabroot>\toolbox\distcomp\distcomp\dct_psfcns.mexa64

If you look at the Application Compiler app, you'll find the "Files required for your library to run" section - this is where you should add the dct_psfcns.mexa64 file.

Unfortunately, there's no guarantee that this won't happen for the next MEX file that cannot be found, so this may be an iterative process.

Thanks, Elwin