Tuning Pod Disruption Budgets
A PodDisruptionBudget (PDB) limits how many pods in a Deployment may be voluntarily evicted at one time. During a nodegroup rebuild or a Cluster Autoscaler scale-down event, the Kubernetes eviction API honours the PDB before removing a pod. This page explains the PDB variables that you can tune in CloudOps for Kubernetes and how to choose the values appropriate for your team.
tip
The PodDisruptionBudget (PDB) tuning can affect the following
- How long CloudOps for Kubernetes upgrades takes, when the upgrade requires a nodegroup rebuild.
- How long nodegroup rebuilds maintenance procedures take.
- The availability and capacity of Commerce services during the above maintenance procedures.
- The behaviour of Cluster Autoscaler scale-down events.
CloudOps for Kubernetes includes PDBs for the Cortex, Search Slave, and Integration Commerce services. By default, Cortex and Search Slave have PDBs enabled and Integration does not.
Policy options
CloudOps for Kubernetes exposes two PDB policy fields per service. Only one may be set at a time; set the other to "" (empty string). The default is maxUnavailable: 1.
The maxUnavailable policy
maxUnavailable sets the maximum number of pods that may be unavailable at any time. A value of "1" means at most one pod is evicted at a time. The remaining replicas continue to serve traffic.
This policy serialises evictions. With six replicas and maxUnavailable: 1, the drain evicts one pod, waits, then evicts the next.
The minAvailable policy
minAvailable sets the minimum number of pods that must remain running at all times. A value of "1" means at least one pod is always available, and the drain may evict all remaining replicas concurrently.
This policy allows parallel evictions. With six replicas and minAvailable: 1, five pods can be evicted at the same time.
caution
Do not set minAvailable: 1 on a service that runs as a single replica. The minAvailable: 1 requirement would prevent the service's one pod from being evicted, preventing the node from draining and causing the maintenance procedure to fail and exit with an error once the PDB timeout is reached.
Built-in profiles
CloudOps for Kubernetes ships two resource profiles that set these variables.
Development (dev.tfvars)
EP_CORTEX_PDB_MAX_UNAVAILABLE = "1"
EP_CORTEX_PDB_MIN_AVAILABLE = ""
EP_SEARCH_SLAVE_PDB_MAX_UNAVAILABLE = "1"
EP_SEARCH_SLAVE_PDB_MIN_AVAILABLE = ""
maxUnavailable: 1 is safe at any replica count, including single-replica deployments. It serialises evictions, so the service runs at near-full capacity throughout the drain.
Production-small (prod-small.tfvars)
EP_CORTEX_PDB_MAX_UNAVAILABLE = ""
EP_CORTEX_PDB_MIN_AVAILABLE = "1"
EP_SEARCH_SLAVE_PDB_MAX_UNAVAILABLE = ""
EP_SEARCH_SLAVE_PDB_MIN_AVAILABLE = "1"
minAvailable: 1 only works with deployments that have at least two replicas. It allows concurrent evictions, which shortens the overall drain time. At least one pod remains available throughout.
Behaviour at different replica counts
The following table shows how many pods can be evicted simultaneously for each profile.
| Replicas | maxUnavailable: 1 (dev) | minAvailable: 1 (prod-small) |
|---|---|---|
| 1 | 1 — only pod is evicted normally | 0 — drain stalls; do not use |
| 2 | 1 | 1 |
| 6 | 1 | 5 |
| 20 | 1 | 19 |
With minAvailable: 1 and six replicas, a nodegroup rebuild completes roughly six times faster than with maxUnavailable: 1, at the cost of running at one-sixth capacity during the drain.
Choosing a policy
The right policy depends on your availability and speed requirements.
Use maxUnavailable: 1 when:
- The services run as single replicas (for example, development or staging environments).
- The services run with multiple replicas but you want to maintain close to full capacity during maintenance windows. The maintenance processes, such as node-group rebuilds, will take longer.
Use minAvailable: 1 when:
- The service runs two or more replicas.
- Fast nodegroup rebuilds are a priority and you accept reduced capacity during the drain window.
- Your traffic can be handled by a single pod for the duration of the drain.
These two goals pull in opposite directions. A more aggressive PDB (lower minAvailable) shortens the drain window but reduces available capacity. A more conservative PDB (lower maxUnavailable) maintains capacity but extends the drain window. Choose based on your business requirements.
Custom values
Neither built-in profile may be the right fit for every environment. Both maxUnavailable and minAvailable accept any positive integer or a percentage string (for example, "50%"). You can set them to any value that suits your replica count and availability requirements.
For example, with 10 replicas you might set EP_CORTEX_PDB_MIN_AVAILABLE = "3" to keep three pods available at all times during a drain. This is more conservative than minAvailable: 1 but faster than maxUnavailable: 1.
caution
When using custom values, verify that the value is satisfiable given your current and minimum replica counts. A minAvailable value that is equal to or greater than the number of healthy pods will block eviction entirely. For example, minAvailable: 3 with only two replicas running gives allowedDisruptions = 0 and the node drain will stall until the timeout is reached.
This risk is higher in environments where the replica count can drop, such as those with autoscaling enabled. If the autoscaler scales the service down to fewer replicas than your minAvailable value during a maintenance window, the drain will block. Size your minAvailable value against the minimum replica count, not the expected or peak replica count.
Unhealthy pod eviction
CloudOps for Kubernetes sets unhealthyPodEvictionPolicy: AlwaysAllow on each PDB by default. This allows pods that are already unhealthy (for example, stuck in CrashLoopBackOff or failing their readiness probe) to be evicted regardless of the budget. Without this setting, a single unhealthy pod can permanently stall a node drain when the disruption budget is exhausted.
Set EP_*_PDB_UNHEALTHY_POD_EVICTION_POLICY to "IfHealthyBudget" to revert to the Kubernetes default. Do this only if your workload requires it.
Configuration reference
Full variable descriptions are in Configure Application Containers.
For authoritative Kubernetes documentation on PodDisruptionBudgets, see: