Overview of Multi-Cluster Architecture
When working with CloudOps for Kubernetes, we recommend using a single cluster per AWS account. In a typical scenario, this means that there will be one AWS account and cluster which will host all non-production environments, and another AWS account and cluster which will host only the production environment.
The lower environment cluster is usually referred to as the hub cluster, and will host the build and deploy infrastructure such as Jenkins and Nexus. This cluster can also be used to host the lower environments, such as Dev, QA, CI, and so on. For most customer scenarios, this cluster can also be used to host stage or pre-prod. Each environment tier should be placed into a separate namespace, which will allow for separation of resources, and if desired this can be used to limit inter-environment communication using Cilium or another Kubernetes service mesh implementation. In this configuration all environments will share compute resources, which will provide the most cost effective method for deploying multiple environments. The environments will also share a single Kubernetes control plane, which greatly simplifies environment management.
When you work with CloudOps for Kubernetes and deploy multiple types of Elastic Path environments, we recommend having one hub Kubernetes cluster. Dedicate this Kubernetes cluster to building and deploying code. This cluster can also be used to host the lower environments, such as Dev, QA, CI, and so on. For production or production-like environments such as stage or pre-prod, if production like testing and isolation is required, create additional Kubernetes clusters dedicated to hosting Elastic Path environments. By running in this way, the Jenkins and Nexus containers inside the hub Kubernetes cluster are less likely to compete for resources with Elastic Path deployments, similar to a production environment. For production environments, we recommend deploying the Kubernetes cluster into a separate AWS account, to ensure complete isolation of the environment.
The preceding diagram is an example of a CloudOps for Kubernetes deployment with test, development, and staging Self Managed Commerce environments running in a shared Kubernetes cluster. In this configuration, each Elastic Path stack is deployed in a dedicated namespace, which houses all of the necessary containers to run a Self Managed Commerce or Acccount Management stack. For most use cases, this environment configuration is sufficient to preserve isolation between environments, and each environment can be tested indepenedently of the others. This cluster is managed by the Jenkins server in the default namespace, and has a Nexus server deployed to allow code builds and Maven tests to be executed. This cluster is typically referred to as a Hub Kubernetes Cluster, and it is the cluster created during initial setup.
Using multiple Kubernetes Clusters in the same account
In some scenarios, users will need a greater degree of isolation between environments, usually for access control or performance testing requirements. Typically this is not required. Kubernetes provides effective resource separation between namespaces. If a shared cluster host many environments, or you require highly accurate performance data from a lower environment tier, you might need to have a secondary cluster for some environments. Greater environment isolation can be achieved by utilizing separate Kubernetes clusters for each type of Self Managed Commerce environment or having a dedicated cluster for a pre-production environment. This is done by running the Jenkins job create-additional-kubernetes-cluster, which creates a new secondary cluster in the same AWS account.
When deploying multiple clusters, the first cluster, created during initial setup, is designated as the Hub Kubernetes Cluster. This Cluster has a Jenkins server that is used for building code, docker containers, and managing Terraform and runs Maven and Cucumber tests against Self Managed Commerce deployments.
Create additional clusters using the create-additional-kubernetes-cluster Jenkins job. These clusters are referred to as Secondary Kubernetes Clusters, which have a Jenkins server deployed to them. They also have access to a limited set of Jenkins jobs, specifically related to deploying and restarting environments. This is to help keep the build process simple, while still allowing for more access control between clusters.
For more information about the Jenkins job create-additional-kubernetes-cluster, see Creating a new Kubernetes Cluster.
For example, the following diagram shows a typical lower environment deployment with a Hub Kubernetes Cluster being used to host the build infrastructure, as well as a Dev and QA Self Managed Commerce stack, and a secondary cluster hosting the Stage environment.
