Main Content

Resolve Pod Creation Stuck Issues

During MATLAB® Online Server™ installation and configuration, pods can sometimes get stuck in one of the following states:

To see pod status, run the following command, replacing your-namespace with the namespace you used for MATLAB Online Server:

kubectl get pods --namespace your-namespace

Consult the following sections for instructions on correcting the pod state when the pod is stuck during creation.

ImagePullBackOff

To find out the exact error message for why the pod status is in ImagePullBackOff status, run the following command, replacing pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:

kubectl describe pod pod_name --namespace your-namespace

In the information returned, look at the Events section. The following table lists the most likely reasons for the ImagePullBackOff error.

Message TypeSample Error MessageReason

Invalid container image or no pull access for a private image

Error response from daemon: repository foobartest4 not found: does not exist or no pull access

This means either that the specified registry name is nonexistent, or you do not have access to the container.

Make sure you have the correct name. If the name is correct, then check that the container registry for this image does not require authentication or, if it requires authentication, make sure that you have the secret properly configured.

As a test, use the following command to try to pull the same image from your local machine:

docker pull <docker-image-name:docker-image-tag>

Invalid container image tag

Warning Failed 10m (x4 over 12m) kubelet, mos-gar-3-pool-1-9781becc-bdb3 Failed to pull image "redis:foobar": rpc error: code = Unknown desc = Error response from daemon: manifest for redis:foobar not found

This means that a specified image cannot be found.

To test if the image has an appropriate tag, try to pull the image locally:

docker pull <docker-image-name:docker-image-tag>

CrashLoopBackOff

This pod status means that your pod is starting, crashing, starting again, and then crashing again.

To get more information about the problem, run the following command, replacing pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:

kubectl describe pod pod_name --namespace your-namespace

In the information returned, look at the Events section, which is the main part of the log that describes what is happening with the pod. Examples of output for the Events section can be found in Check Pod Status.

Next, look at the logs by running the following command, replacing pod_name with the name of the pod that is stuck:

kubectl logs pod_name

For a multi-container pod (for example, the MATLAB pod), run the command with the container name:

kubectl logs pod_name -c container_name

Logs can help you find a reason why the container is crashing. Look at the logs for Java exception or error messages. See Get Help for assistance.

Pending

To get more information about why the pod is in a pending state, run the following command, replacing pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:

kubectl describe pod pod_name --namespace your-namespace

The pod might be in a Pending state because there is not enough CPU, not enough memory, or not enough CPU and memory.

If there is not enough CPU, for example, you might find the following entry in the log:

Events:
  Type     Reason            Age               From            Message
  ----     ------            ----              ----              ----
  Warning  FailedScheduling  2s (x6 over 11s) default-scheduler  0/2 
nodes are available: 4 Insufficient cpu

Possible conditions:

  • You have requested more CPU than any of the nodes has. For example, if each node in the two-node cluster has 2 CPU cores and you request 4 CPU cores, even though, in total, the cluster capacity is 4 CPU cores, a single node does not meet the request of 4 CPU cores.

    Every pod needs to be scheduled on a single node with 4 CPU cores and because all nodes have only 2 CPU cores, the request cannot be met. Even if you turn on more nodes in your cluster, Kubernetes® still is not able to schedule your request.

  • There is no more capacity in the cluster for the CPU cores you have requested. For example, if each node in the two-node cluster has 1 CPU core and you request 1 CPU core, then two pods can be scheduled on each node because the requested CPU matches the node’s capacity.

    However, if a third pod requests 1 CPU core, there is not enough CPU in the cluster, and you get the FailedScheduling state. If this happens, you can turn on more nodes (with 1 CPU core) in the cluster, and your pod can be scheduled.

ContainerCreating

When a pod is just starting, ContainerCreating is a normal state. However, it is not normal for a pod to be stuck in the ContainerCreating state for longer than a few seconds (approximately 10 to 20 seconds).

To get more information about the problem, run the following command, replacing pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:

kubectl describe pod pod_name --namespace your-namespace

In the information returned, look at the Events section to find a possible reason for the pod to continue to be in the ContainerCreating state.

Get Help

If none of the above issues described in this section help you to resolve the issue, try the following:

  • See Resolve MATLAB Pod Issues.

  • Contact MathWorks Support.

    1. Run the following kubectl commands, replacing license_pod_name with the name of the pod that is stuck and your-namespace with the namespace you used for MATLAB Online Server:

      kubectl describe pod license-pod-name --namespace your-namespace 
kubectl logs license-pod-name --namespace your-namespace

    2. Keep the output nearby to discuss with a support technician.

Related Topics