Resolve MATLAB Pod Issues

The MATLAB^® pod comprises four Docker^® containers, one of which is MATLAB Runtime.

When the services are running successfully and have no issues, then the status of pods is Running.

However, if the services are having issues, pods can still show the status Running. This can be because, for example, the MATLAB Docker image was built incorrectly, or there are issues connecting with the network license manager server.

Run the following command to display pod status, replacing your-namespace with the namespace you used for MATLAB Online Server™:

kubectl get pods --namespace your-namespace

This command outputs information similar to that shown in the following example:

 NAME                                               READY    STATUS    RESTARTS      AGE 
your-namespace-core-ui-core-ui-5bd9b7c88b-fqbl4     1/1     Running      0          111s 
your-namespace-authnz-authnz-7c7999589c-x74d9       1/1     Running      0          111s 
your-namespace-gateway-669dd7865f-t6l74             1/1     Running      0          110s 
your-namespace-license-548f7b6655-8skdt             1/1     Running      0          108s 
your-namespace-matlab-pool-8c679f76-c4h4n           2/2     Running      0          107s 
your-namespace-matlab-pool-8c679f76-cckkp           2/2     Running      0          107s 
your-namespace-matlab-pool-ui-579fc8ff49-2bz25      1/1     Running      0          107s 
your-namespace-ingress-nginx-controller-kcwwd       1/1     Running      0          104s
your-namespace-resource-resource-894447df7-r9ngb    1/1     Running      0          103s

Problems and Resolutions

Check the possible problems and solutions in this section to help diagnose and resolve common issues.

License Checkout Issues

Although a license checkout error is a failing scenario, the MATLAB pod status still shows as Running because a prewarmed pool of MATLAB instances might run successfully but fail to check out a license when a user signs in.

To determine if a license checkout error is the issue, perform the following assessment:

Run the following command to get the MATLAB container logs, replacing matlab_pod_name with the name of your MATLAB Online Server MATLAB pod and your-namespace with the namespace you used for MATLAB Online Server:
```
kubectl logs matlab_pod_name --namespace your-namespace -c matlab
```
Check for License Manager Error -<n>: License Checkout Failed in the logs. If this error has occurred, then it is highly likely that MATLAB is having issues checking out the license.
Logs can be lengthy. You can use a text-searching tool to get the information you need faster. For example, you can search the logs for a license checkout error:
```
kubectl logs matlab_pod_name --namespace your-namespace -c matlab | grep 'License Checkout'
```
You can also search for a license manager error:
```
kubectl logs matlab_pod_name --namespace your-namespace -c matlab | grep 'License Manager Error'
```
If you see License Manager Error -<n>: License Checkout Failed in the logs, check the specific error message, and then consult the following table.

Error Message Resolution Steps

Invalid license file syntax Check to make sure that the license file syntax is appropriate (that is, check the network license manager logs). Then, making sure that you use the correct license file, try again.

Cannot connect to license server

Try to connect to the license server locally.
Check to see if the network firewall rules for the license server are set correctly to allow connection from the MATLAB Online Server cluster.
Network policy on a MATLAB pod is enabled by default, which allows it to talk to the network license manager only on ports 27000 and 27001 by default (controlled by license.dat). Make sure your network license manager server is listening on those ports. If your server is listening on other ports, you need to change the following configuration in the matlab-pool override and then restart the pod:
networkPolicy: enabled: true additionalAllowedPorts: - port: <flexlm_port_here> protocol:

Incorrect MATLAB Image

The MATLAB Image is built from a local MATLAB install and can sometime have issues if it was built from an incorrect MATLAB location or version that is not supported by MATLAB Online Server.

To determine if an incorrect MATLAB image is the problem, perform the following assessment.

Run the following command to display a pod status list, replacing your-namespace with the namespace you used for MATLAB Online Server:
```
kubectl get pods --namespace your-namespace
```

Examine the pod status information, as shown in this example:

NAME                                               READY   STATUS    RESTARTS   AGE 
your-namespace-authnz-authnz-569df55f78-pqklk      1/1     Running   0          47h 
your-namespace-core-ui-core-ui-84c59ddd45-fwf4t    1/1     Running   0          47h 
your-namespace-gateway-84dbf7c8d-6gjcz             1/1     Running   0          47h 
your-namespace-license-85ffb6c595-q6x4f            1/1     Running   0          47h 
your-namespace-matlab-pool-apparmor-loader-jzqsg   1/1     Running   0          40s 
your-namespace-matlab-pool-f479b5b96-59xmj         3/4     Running   0          40s 
your-namespace-matlab-pool-helpsearch-546cb5fd8-jkx4p  1/1     Running   0          40s 
your-namespace-matlab-pool-ui-7b6d688f98-rfdz5	1/1     Running   0          40s 
your-namespace-ingress-nginx-controller-5c6b5f775f-7ls87  1/1     Running   0          27h
your-namespace-resource-resource-8554f87bbf-g4ssq   1/1     Running   0          47h

If you see MATLAB pods in a 3/4 Running state, as shown in the example log, run the following command to learn why. Replace matlab_pod_name with your MATLAB Online Server MATLAB pod name and your-namespace with the namespace you used for MATLAB Online Server:

kubectl logs matlab_pod_name --namespace your-namespace -c matlab

This command outputs a log similar to this example:

Starting MATLAB 
+ exec /MATLAB/bin/matlab -prewarm -c '"27000@flexlm.mwcloudtest.com"' -nodesktop -softwareopengl -nosplash 
MATLAB is selecting SOFTWARE OPENGL rendering. 
...
 
< M A T L A B (R) > 
Copyright 1984-2024 The MathWorks, Inc. 
...
 
Warning: Unrecognized command line option: prewarm.  
  
To get started, type doc. 
For product information, visit www.mathworks.com.

The logs show that MATLAB was not able to start successfully. To verify that this is what happened, you can run the following command to look at the logs of resource-proxy, which is another container in the MATLAB pod that proxies requests to MATLAB.

Issue this command, replacing matlab_pod_name with your MATLAB Online Server MATLAB pod name and your-namespace with the namespace you used for MATLAB Online Server:

kubectl logs matlab_pod_name --namespace your-namespace -c resource-proxy

Examine the logs output by this command. If you see the following messages, the backend (in this case, MATLAB) is not ready and is not functioning properly.

YYYY-MM-DD 19:06:51,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:06:51,507 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:06:54,498 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:04,499 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:11,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:11,507 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:14,500 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:24,500 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:31,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:31,507 INFO matlabonlineserver.resource_proxy Backend is not ready

In the example log, the message "Backend is not ready" indicates that MATLAB is not yet ready.

For resolution, check to see if you are building the MATLAB image from the correct MATLAB install location.

Storage Mount Errors

When all services and MATLAB are in a Running state, but you are encountering issues using MATLAB Online™, it is possible that there was a problem with mounting the storage on MATLAB at the time of user sign on.

To determine if storage mounting on MATLAB is the problem, perform the following assessment:

Run the following command to generate pod logs, replacing matlab_pod_name with the name of your MATLAB Online Server MATLAB pod and your-namespace with the namespace you used for MATLAB Online Server.
```
kubectl logs matlab_pod_name --namespace your-namespace -c resource-proxy
```

Examine the output from this command:

YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Setting up the profile: nfsHome 
YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Setting up the mount: home 
YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Starting NFS mount 
YYYY-MM-DD 20:44:39,125 INFO matlabonlineserver.storage Executing: mount -o rw,
relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,
acdirmin=600,acdirmax=600,hard,nocto,noacl,proto=tcp,timeo=600,retrans=2,mountproto=tcp,
local_lock=none,nolock nfs:/exports/home/oin /mounts/homeoin 
YYYY-MM-DD 20:44:49,167 INFO matlabonlineserver.storage mount exit value= 137, mount result= , mount error=

If you see the mount error message, as displayed in the last line of the example shown, check that the ports allowed by the network policy for MATLAB Pool include ports for your NFS server setup. For more details on network policies, see Configure Network Policies.

MATLAB Pod Not in Running State

If none of the possible issues described in this section help you to resolve the problem, try the following:

See Resolve MATLAB Pod Issues.
Contact MathWorks Support.
1. Run the following kubectl commands, replacing license_pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:
  kubectl describe pod license-pod-name --namespace your-namespace kubectl logs license-pod-name --namespace your-namespace
2. Keep the output nearby to discuss with a support technician.