Manage ActiveMQ Deployments

Updating the ActiveMQ Deployment(s) when changes are made to the Elastic Path ActiveMQ Docker image definition

When changes are made to the Elastic Path ActiveMQ Docker image definition, update your ActiveMQ deployment(s) to consume the changes in your deployed Self Managed Commerce stack by doing the following steps:

Step 1: Build the image

Build a new ActiveMQ image using the build-activemq Jenkins job to update the changes to the Elastic Path ActiveMQ Docker image definition.

Step 2: Update the ActiveMQ deployment(s)

If you gave the image built a new tag, use the new imageTag from the build-activemq Jenkins job, and rerun the create-or-delete-activemq-container Jenkins job for your existing ActiveMQ deployment with the new imageTag specified.

If you did not give the image built a new tag, then you can use kubectl to restart the ActiveMQ deployment(s) to update the new version of the image. Use the following commands to restart the ActiveMQ deployment(s):

1. Confirm the exact name of the Kubernetes namespace where your ActiveMQ and Self Managed Commerce services are running. This will be the value you specified as the kubernetesNickName parameter in the Jenkins jobs used to deploy the Self Managed Commerce services.

2. Set the namespace environment variable.

   # Be sure to replace <kubernetesNickNameValue> with your actual 'kubernetesNickName' value
   activemq_namespace="<kubernetesNickNameValue>"

3. Determine whether you have one or two ActiveMQ deployments in your namespace.

   # get the deployments
   kubectl get deployments -n ${activemq_namespace} -l 'app=ep-activemq'

4. Run the following commands to update the first ActiveMQ deployment:

   # get the deployment name
   activemq_deployment_name_0=$(kubectl get deployments -n ${activemq_namespace} -l 'app=ep-activemq' -o jsonpath='{.items[0].metadata.name}')
   
   # restart the first deployment
   kubectl rollout restart -n ${activemq_namespace} ${activemq_deployment_name_0}
   kubectl rollout status -n ${activemq_namespace} ${activemq_deployment_name_0}

5. If you have a second ActiveMQ deployment, run the following commands to update the second ActiveMQ deployment:

   # get the deployment name
   activemq_deployment_name_1=$(kubectl get deployments -n ${activemq_namespace} -l 'app=ep-activemq' -o jsonpath='{.items[1].metadata.name}')
   
   # restart the second deployment
   kubectl rollout restart -n ${activemq_namespace} ${activemq_deployment_name_1}
   kubectl rollout status -n ${activemq_namespace} ${activemq_deployment_name_1}

Recovering an ActiveMQ pod stuck in Terminating

Force deletion can corrupt the message store

Force-deleting an ActiveMQ pod removes the Pod record without confirming that the broker has stopped. Kubernetes can then start a replacement while the original broker continues to write to the same KahaDB journal. This can corrupt the journal, cause message loss, and prevent the replacement broker from starting.

Use the recovery procedure below when messages must be retained or when the environment cannot be easily recreated.

For a disposable environment, an engineer may decide to accept the risk and force-delete the pod. Do this only if queued messages do not need to be retained and the environment can be recreated if the broker fails to start.

ActiveMQ stores messages in a KahaDB journal that only one broker may write at a time. Kubernetes keeps the Pod object in Terminating until the kubelet confirms that the container has exited. If the node is unreachable that confirmation never arrives, and the pod appears stuck -- but the broker may still be running, and still writing.

The KahaDB file lock does not prevent this failure. The lock is held through the original broker's connection to Amazon Elastic File System (EFS). If that connection is partitioned or its lease expires, the lock can be released while the broker is still running.

Recovery has three phases. Prevent a replacement broker from starting, make sure the original broker process has exited, and then start a new broker. Complete all three phases to restore messaging.

Phase 1: Prevent a replacement broker from starting

Kubernetes treats a Terminating pod as inactive, so a replacement may already be running. Scale the deployment to zero. This stops any replacement and prevents another from starting while you confirm that the original broker has stopped.

# Be sure to replace <kubernetesNickNameValue> with your actual 'kubernetesNickName' value
activemq_namespace="<kubernetesNickNameValue>"

# List the ActiveMQ deployments in your namespace.
# The NAME column gives the <deployment-name> used below.
kubectl get deployments -n ${activemq_namespace} -l 'app=ep-activemq'

# Scale down the deployment that owns the stuck pod.
kubectl scale -n ${activemq_namespace} deployment/<deployment-name> --replicas=0

# Wait until no ActiveMQ pod reports Running before you continue.
kubectl get pods -n ${activemq_namespace} -l 'app=ep-activemq' -o wide

Phase 2: Make sure the broker process has exited

Establish the state of the node that the pod is running on.

# List the ActiveMQ pods. The NAME column gives the <pod-name> of the stuck pod,
# and the NODE column gives the <node-name> that it is running on.
kubectl get pods -n ${activemq_namespace} -l 'app=ep-activemq' -o wide

# Check that node. The STATUS column reads Ready, NotReady, or Unknown.
kubectl get node <node-name>

# Read the Events section at the end of the output. It reports what is
# holding up the termination, such as a slow unmount or a container
# that has not exited.
kubectl describe pod -n ${activemq_namespace} <pod-name>

Then use the first of the three options below that applies. Each option is more disruptive than the one before it.

Option 1: Wait for the kubelet

Use this when the node reports Ready.

The kubelet is still in control. The pod clears on its own when the termination grace period expires, and an EFS unmount can add to that time. Wait, and do not intervene.

Option 1 is complete when the pod no longer appears in kubectl get pods.

If the node changes to NotReady or Unknown, use Option 2.

If the node remains Ready but the pod events show no progress, use Option 2.

Option 2: Signal the broker on the node

Use this when Option 1 does not clear the pod. First, try to reach the broker with kubectl exec. This affects only ActiveMQ, not the other workloads on the node.

Signal the broker process directly. On a stuck pod, SIGTERM may not have reached the broker. The signal lets the broker complete its KahaDB checkpoint. A SIGKILL can instead stop it in the middle of a journal write.

# Find the broker process inside the container.
kubectl exec -n ${activemq_namespace} <pod-name> -c activemq -- \
  /bin/bash -c "ps -ef | grep '[a]ctivemq.jar start'"

# Ask the broker to shut down cleanly. Never use kill -9.
kubectl exec -n ${activemq_namespace} <pod-name> -c activemq -- \
  /bin/bash -c 'kill -TERM <broker-pid>'

# Wait for the process to exit. This is the KahaDB checkpoint and flush.
# The session ends when the container exits. That is the expected outcome.
kubectl exec -n ${activemq_namespace} <pod-name> -c activemq -- \
  /bin/bash -c 'while kill -0 <broker-pid> 2>/dev/null; do sleep 2; done'

If kubectl exec fails, open a shell on the node with AWS Session Manager. For more information, see Accessing a shell on an EKS Cluster Member.

# Find the broker process from the node.
ps -ef | grep '[a]ctivemq.jar start'

# Confirm that the process belongs to the stuck pod.
# This must print HOSTNAME=<pod-name>.
sudo cat /proc/<broker-pid>/environ | tr '\0' '\n' \
  | grep "^HOSTNAME=<pod-name>$"

# Ask the broker to shut down cleanly. Never use kill -9.
sudo kill -TERM <broker-pid>

# Wait for the process to exit.
while sudo kill -0 <broker-pid> 2>/dev/null; do sleep 2; done

Option 2 is complete when the broker process no longer appears.

If the broker does not exit, do not use kill -9. Check the broker logs for ongoing KahaDB shutdown activity.

If neither kubectl exec nor a node shell is available, move the node's other workloads before you terminate it:

# Preview the workloads that the drain would affect.
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force \
  --pod-selector='app!=ep-activemq' --timeout=5m --dry-run=server

Review the listed workloads before continuing. Draining terminates them on this node. Kubernetes restarts controller-managed workloads elsewhere, but active Jenkins agent pods are deleted and their builds fail. Wait for important builds to finish, or be prepared to retry them. When the disruption is acceptable, run the drain:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force \
  --pod-selector='app!=ep-activemq' --timeout=5m

The drain honors Pod Disruption Budgets and deletes emptyDir data. Review any reported blockers. If the drain times out, continue to Option 3. If you stop here, run kubectl uncordon <node-name>.

Option 3: Terminate the instance

Use this when Option 2 cannot reach the broker. Terminating the instance is then the safest remaining way to confirm that the broker has stopped. Other workloads on the node may be interrupted, so move them first where possible.

1. Terminate the instance through the Auto Scaling group or the EC2 console. Confirm that it reaches the terminated state.

2. Wait for the Node object to disappear. Kubernetes removes it after the instance no longer exists, then garbage-collects the pods that were bound to it.

   kubectl get node <node-name>

3. If the Node object is still there after a few minutes, delete it.

   kubectl delete node <node-name>

Option 3 is complete when the instance is terminated and the pod no longer appears.

If the pod still has not cleared

Confirm that no broker process remains.

If the pod has not cleared, and you have confirmed that the broker process is gone, you can now remove the Pod record. Run this command only after confirming that the broker process is no longer running.

kubectl delete pod -n ${activemq_namespace} <pod-name> --force --grace-period=0

Phase 3: Start the broker again

The deployment is still scaled to zero, so no broker is running until you complete this phase.

kubectl scale -n ${activemq_namespace} deployment/<deployment-name> --replicas=1
kubectl rollout status -n ${activemq_namespace} deployment/<deployment-name> --timeout=5m
kubectl logs -n ${activemq_namespace} -l 'app=ep-activemq' --tail=100

If the rollout status command times out, check the logs. KahaDB recovery may still be in progress.

Check the logs for KahaDB recovery errors before returning the broker to service.