Application Debugging

Connect to the JVM of a Self Managed Commerce Application

This section outlines how to connect to a Self Managed Commerce application JVM in a Kubernetes deployment. It is intended for developers who are familiar with Java and the Elastic Path application stack. By default the JMX, and JVM (Java Virtual Machine) debug ports are not made accessible via endpoints for security reasons.

Using JMX Console

Procedure

To connect to a JMX port for a deployed Self Managed Commerce application with JConsole or JVisualVM:

1. Set the parameter enableJmx in the Jenkins job deploy-or-delete-commerce-stack.

2. (Optional) Enable JMX user and password authentication by setting the parameter enableJmxAuth.

3. Deploy a Self Managed Commerce stack using the job deploy-or-delete-commerce-stack.

4. (Optional) Find the JMX username and password, if JMX user and password authentication is enabled.

   The username and password are auto-generated and uploaded to a Kubernetes ConfigMap under the keys JMX_USER and JMX_PASSWORD.

   To access the ConfigMap, access the Kubernetes dashboard of the same Kubernetes cluster as the Self Managed Commerce deployment. Switch to the same namespace as the deployment and open the ConfigMap named ep-configmap.

5. Forward the JMX port of the applications in the cluster to your local computer.

   See the following command as an example. Replace ${NAMESPACE} and app=ep-cm with the values for your application.

   cmPodName=$(kubectl get pods -n ${NAMESPACE} -l 'app=ep-cm' -o jsonpath='{.items[*].metadata.name}')
   kubectl port-forward pod/${cmPodName} -n ${NAMESPACE} 8888:8888

6. If authentication is disabled, you can open the JMX connection directly from the terminal.

   Run the following command to run JConsole:

   jconsole localhost:8888

   Run the following command to run JVisualVM, if required:

   jvisualvm --openjmx 127.0.0.1:8888

   If authentication is enabled, run the Java profiler of choice and connect manually using the UI providing the correct username and password from step 4.

note

Disable JMX connections on production environments to avoid performance issues.

Using remote debugger

To connect to the JVM of a Self Managed Commerce application in debug mode from your IDE:

1. Enable the parameter enableDebug in the Jenkins job deploy-or-delete-commerce-stack

2. Deploy a Self Managed Commerce stack using the job deploy-or-delete-commerce-stack

3. Forward the debug port of the application in the cluster to your local computer.

   See the following command as an example. Replace ${NAMESPACE} and app=ep-cm with the values for your application.

   cmPodName=$(kubectl get pods -n ${NAMESPACE} -l 'app=ep-cm' -o jsonpath='{.items[*].metadata.name}')
   kubectl port-forward pod/$cmPodName -n ${NAMESPACE} 1081:1081

4. Configure the remote debugger in your IDE to connect to localhost:1081.

   note

   You can expose both the JMX and debug ports in one step by port forwarding with both ports appended to the command. See the following command as an example:

   cmPodName=$(kubectl get pods -n ${NAMESPACE} -l 'app=ep-cm' -o jsonpath='{.items[*].metadata.name}')
   kubectl port-forward pod/$cmPodName -n ${NAMESPACE} 8888:8888 1081:1081

Getting the Heap Dump of a Self Managed Commerce Application

The heap dump files are stored on a persistent volume in the Kubernetes cluster, and the volume is mounted on all of the Self Managed Commerce containers. When a heap dump is triggered by the Java virtual machine (JVM), the heap dump is saved in the pod under a path like /hdump/<commerce_service_container_name>/java_pid<pid>.hprof.

Overview of copying heap dump files

Use kubectl to connect to a container and copy the file to your local computer. If you know the correct file to copy, use the kubectl cp command to copy it from a running container. We recommend to compress the large files.

All Self Managed Commerce containers now share a single Elastic File System (EFS) volume mount. With this enhancement, all heap dumps are available on any Self Managed Commerce container that you can connect to through kubectl. For example, you can get the heap dump files generated by the Elastic Path batch service container from the Elastic Path integration container.

The volume structure is as follows:

/hdump/

  CORTEX
    java_pid12.hprof

  BATCH
    java_pid23.hprof

  SEARCH
    java_pid34.hprof

Setting up the environment variables

1. Set the environment variable for the Kubernetes namespace that Self Managed Commerce is deployed into from the kubernetesNickName parameter used in the Jenkins job.

   The kubernetesNickNameValue is the Kubernetes namespace in which the Self Managed Commerce stack is deployed. This will be the value of the Jenkins parameter kubernetesNickname, from the Jenkins job deploy-or-delete-commerce-stack, that is used when deploying the Self Managed Commerce stack.

   See the following as an example:

   commerce_namespace="kubernetesNickNameValue"

2. Confirm the service and container names for the Self Managed Commerce service that you would like to get a heap dump from.

   Use these to find the right heap dump file in the next steps. If you know the service and container names already, you can skip to the next step. If you are not certain, see the following command as an example:

   epApps=$(kubectl get deploy -n ${commerce_namespace} -o json | jq -r '.items[].metadata | select(.labels.app | startswith("ep-")) | select (.labels.app != "ep-activemq") | .name')
   echo "${epApps[@]}" | cut -c4- | rev | cut -c12- | rev

   Take note of the exact names of the service.

3. Set the environment variable for the service and container names. The commerce_service_container_name variable is the uppercase version of the commerce_service that you have set.

   See the following as an example:

   commerce_service="cortex"
   commerce_service_container_name="CORTEX"

4. Set the environment variable for the Self Managed Commerce service that you would like to get a heap dump from.

   See the following as an example:

   service_pod_name=$(kubectl get pods -n ${commerce_namespace} -l "app=ep-${commerce_service}" -o jsonpath="{.items[0].metadata.name}")

Obtaining the heap dump file

1. Run exec into the container.

   See the following as an example:

   kubectl exec -it ${service_pod_name} -n ${commerce_namespace} /bin/bash

2. Zip the contents of the existing heap dump file and exit the container.

   See the following as an example:

   cd /hdump/${commerce_service_container_name}
   gzip java_pid<pid>.hprof
   ls /hdump/${commerce_service_container_name}
   exit

3. Copy the zip file.

   See the following as an example:

   kubectl cp -n ${commerce_namespace} ${service_pod_name}:/hdump/${commerce_service_container_name}/java_pid<pid>.hprof.gz java_pid<pid>.hprof.gz

Identifying the heap dump file (Optional)

1. Check the status of the Commerce kubernetes service pod.

   See the following as an example:

   kubectl describe pod ${service_pod_name} -n ${commerce_namespace}

2. Check whether the container is restarted due to a memory issue.

   Last State: Terminated is displayed in the output, and Reason: OOMKilled is displayed in the container description. It may not be possible to know the correct process ID if the container does not exist or a new Java process is started.

   See the following as an example:

   Containers:
     cortex:
       ...
       State:          Running
          Started:      Thu, 10 Oct 2021 11:14:13 +0200
        Last State:     Terminated
          Reason:       OOMKilled
          Exit Code:    137
          Started:      Thu, 10 Oct 2021 11:04:03 +0200
          Finished:     Thu, 10 Oct 2021 11:14:11 +0200

3. Run exec into the container.

   See the following as an example:

   kubectl exec -it ${service_pod_name} -n ${commerce_namespace} /bin/bash

4. Get the process ID of java on the container.

   See the following as an example:

   jps -v

5. If the container does not exist or a new process is started, you can obtain the path to the heap dump file from the log aggregation tool that you have installed. Log aggregation tools are not provided in CloudOps for Kubernetes.

   The following example log shows you the path of the heap dump:

   Heap dump file created [70236796 bytes in 0.611 secs]
   Dumping heap to /hdump/CM/java_pid64.hprof ...
   java.lang.OutOfMemoryError: Java heap space

   If you cannot identify the heap dump file from the existing Java process, or from your log aggregation tool, you can guess the heap dump file based on the timestamp that is created.