When working with CloudOps for Kubernetes, it is recommended to use a single cluster per AWS account or Azure subscription. In a typical scenario, this means that there will be one cluster which will host all non production environments, and one cluster which will host only the production environment.
The lower environment cluster is usually referred to as the
hub cluster, and will host the build and deploy infrastructure such as Jenkins and Nexus. This cluster can also be used to host the lower environments, such as
CI, and so on. For most customer scenarios, this cluster can also be used to host
pre-prod. Each environment tier should be placed into a separate namespace, which will allow for separation of resources, and if desired this can be used to limit inter-environment communication using Cilium or another Kubernetes service mesh implementation. In this configuration all environments will share compute resources, which will provide the most cost effective method for deploying multiple environments. The environments will also share a single Kubernetes control plane, which greatly simplifies environment management.
When you work with CloudOps for Kubernetes and deploy multiple types of Elastic Path environments, we recommend having one hub Kubernetes cluster. Dedicate this Kubernetes cluster to building and deploying code. This cluster can also be used to host the lower environments, such as
CI, and so on. For production or production-like environments such as
pre-prod, if production like testing and isolation is required, create additional Kubernetes clusters dedicated to hosting Elastic Path environments. By running in this way, the Jenkins and Nexus containers inside the hub Kubernetes cluster are less likely to compete for resources with Elastic Path deployments, similar to a production environment. For production environments, we recommend deploying the Kubernetes cluster into a separate AWS account or Azure subscription using docker-compose, to ensure complete isolation of the environment.
The preceding diagram is an example of a CloudOps for Kubernetes deployment with test, development, and staging Elastic Path Commerce environments running in a shared Kubernetes cluster. In this configuration, each Elastic Path stack is deployed in a dedicated namespace, which houses all of the necessary containers to run an Elastic Path Commerce or Acccount Management stack. For most use cases, this environment configuration is sufficient to preserve isolation between environments, and each environment can be tested indepenedently of the others. This cluster is managed by the Jenkins server in the default namespace, and has a Nexus server deployed to allow code builds and Maven tests to be executed. This cluster is typically referred to as a
Hub Kubernetes Cluster, and it is the cluster the bootstrap container creates in
Using multiple Kubernetes Clusters in the same account
In some scenarios, users will need a greater degree of isolation between environments, usually for access control or performance testing requirements. Typically this is not required. Kubernetes provides effective resource separation between namespaces. If a shared cluster host many environments, or you require highly accurate performance data from a lower environment tier, you might need to have a secondary cluster for some environments. Greater environment isolation can be achieved by utilizing separate Kubernetes clusters for each type of Elastic Path Commerce environment or having a dedicated cluster for a pre-production environment. This is done by running the Jenkins job
create-additional-kubernetes-cluster, which creates a new secondary cluster in the same AWS account, or the same resource group and subscription if you are using Azure.
When deploying multiple clusters, the first cluster, which is created using docker-compose in
setup mode, is designated as the
Hub Kubernetes Cluster. This Cluster has a Jenkins server that is used for building code, docker containers, and managing terraform, as well as being able to run Maven and Cucumber tests against Elastic Path Commerce deployments. All clusters that are created using the
create-additional-kubernetes-cluster Jenkins job are referred to as
Secondary Kubernetes Clusters.
Secondary Kubernetes Clusters also have a Jenkins server deployed to them, but they have access to a much more limited set of Jenkins jobs, specifically related to deploying and restarting environments. This is to help keep the build process simple, while still allowing for more access control between clusters.
For more information on the Jenkins job
For example, the following diagram shows a typical lower environment deployment with a
Hub Kubernetes Cluster being used to host the build infrastructure, as well as a Dev and QA Elastic Path Commerce stack, and a secondary cluster hosting the Stage environment.