Creating a Self Managed Commerce Database Server
Each Self Managed Commerce environment requires a dedicated database. This topic guides you through the options to create the database service that will host the Self Managed Commerce database.
Introduction
There are several strategies you can use for creating the Self Managed Commerce database service using CloudOps for Kubernetes:
- During development, use the MySQL Docker image to deploy a non-production database server that runs in a container. Data saved in a containerized database is lost when the container is deleted.
- Alternatively, during development, create a single AWS Aurora RDS Database and create separate databases on that server. Ensure that you use different database names for each Self Managed Commerce deployment and developer.
- For staging and production, create a separate AWS Aurora RDS Database for each deployment.
- Using a database in an existing external compatible database.
- Restoring a database snapshot when creating a new database instance.
These strategies are all automated by CloudOps for Kubernetes by Jenkins jobs.
Creating Containerized MySQL Server
warning
This method of running a MySQL server is not production-grade. For more information about creating a suitable approach for production, see the Creating a Cloud Database Server section.
To create a containerized MySQL server, run the
create-or-delete-mysql-containerjob with the following parameters:deleteContainercloudOpsForKubernetesRepoURLcloudOpsForKubernetesBranchkubernetesNicknameimageTagclusterName
The job creates a container running MySQL 5.7 within the Kubernetes namespace specified by kubernetesNickname.
The username and password are randomly generated and stored in a Kubernetes Secret, in the same Kubernetes namespace.
The Kubernetes Service created for the MySQL server is of ClusterIP type and there is no Ingress for it. By default, the MySQL container is only accessible from other containers in the same namespace.
Parameters for the create-or-delete-mysql-container Jenkins Job
containerUser
Indicates whether the MySQL container image was built to run as the root user or the non-root user. It cannot be empty.
Set to non-root if the container image was built to run as non-root (for example, it was built with release 4.2.x or newer of the Elastic Path Docker repository).
Set to root if the container image was built to run as root (for example, it was built with release 4.1.x or earlier of the Elastic Path Docker repository).
deleteContainer
When this parameter is selected, the specified MySQL service and resources, including the container, will be deleted instead of created.
cloudOpsForKubernetesRepoURL
The URL to your hosted copy of the cloudops-for-kubernetes Git repository.
cloudOpsForKubernetesBranch
The branch of the cloudops-for-kubernetes Git repository to use during the deployment.
kubernetesNickname
This job uses the value of kubernetesNickname as the Kubernetes namespace where the MySQL container is created and connection information is saved as secrets. If the namespace does not exist, this job creates it. To be compatible with Terraform, Kubernetes, and DNS domain name requirements, ensure that the kubernetesNickname is lowercase and alphanumeric. When you deploy the Elastic Path stack that will use this MySQL container, be certain to specify the same value for kubernetesNickname.
If you specify a value where a MySQL container is already deployed, then that MySQL container will be updated and may be replaced and data lost.
imageTag
The tag of the mysql Docker image to use.
clusterName
The name of the Kubernetes cluster in which to deploy the MySQL service.
Creating a Cloud Database Server
The Jenkins job create-and-manage-database-server supports two types of databases for creation and management. The job can create and manage Aurora MySQL RDS cluster, or the job can create a PostgresSQL RDS instance with a standby instance in a separate Availability Zone (AZ). PostgresSQL as a database type is only supported in Self Managed Commerce version 8.3 and later.
The job creates a database server with a default and standby node on the private subnets that were created along with your EKS cluster. By default, each node is db.r5.xlarge. Each node has four vCPUs and 32 GB RAM. The server is configured for geo-redundant daily backups that are preserved for 14 days. The kubernetesNickname parameter should be unique across your AWS account to avoid conflicts in database server naming.
The username and password are randomly generated and stored in a Kubernetes Secret.
The job sets the server settings for Aurora MySQL RDS clusters:
character_set_server=UTF8MB4
tx_isolation=READ-COMMITTED
The job sets the server settings for PostgresSQL RDS servers:
default_transaction_isolation=read committed
Parameters for the create-and-manage-database-server Jenkins Job
databaseType
Set this choice to the type of database server that AWS RDS should create and manage.
deleteServer
Deletes the server instead of creating it.
applyUpgradeImmediately
This optional parameter can be helpful when modifying an existing database instance. When creating a new instance, this parameter has no effect. Set this parameter to true to immediately apply any specified changes to the RDS instance class or engine version.
Warning
The database instance and any Self Managed Commerce services that rely on the database instance will be unavailable while the changes are applied.
Ensure that you set this parameter to true if you are making changes to the engine version. You cannot make changes to the engine version using this Jenkins job if the parameter is set to false. This is due to a limitation of the AWS Terraform provider implementation.
Set this parameter to false if you want to make changes to the instance class without downtime. Those changes will apply during the next RDS maintenance window.
Note
Engine version upgrades may still be applied without downtime in the RDS maintenance windows if you upgrade the engine version in the AWS RDS console. If you follow that approach, you must ensure that those changes have been completed entirely before running this job.
createPostgresDatabaseResources
This parameter is only applied if the databaseType parameter is set to postgres-rds. Set this parameter to true when creating a new Postgres RDS instance and you want to create a new database/schema and a corresponding PostgreSQL role for the database/schema. You can set this parameter to false when updating an existing Postgres RDS instance to avoid trying to recreate database objects that already exist.
When creating a new Postgres RDS instance, the createPostgresDatabaseResources must be set to true. When creating a new Postgres RDS instance, If createPostgresDatabaseResources is set to false, then the resulting configuration will be incomplete and will not work: data-pop operations will fail and the Self Managed Commerce stack will fail to standup.
useRecommendedVersion
Select this parameter to use the recommended engine version of Postgres RDS. This parameter is ignored if the databaseType parameter is not set to postgres-rds. If you are using non-recommended versions of the Postgres RDS engine, set this parameter to false.
postgresRDSEngineVersion
This parameter is ignored if the databaseType parameter is set to aurora-mysql. If the useRecommendedVersion Jenkins job parameter is set to false, use this version of the Postgres RDS engine instead.
kubernetesNickname
This job uses the value of kubernetesNickname as the Kubernetes namespace where the database connection information is stored, and in the name of the database server. If the namespace does not exist, this job creates it. To be compatible with Terraform, Kubernetes, and DNS domain name requirements, ensure that the kubernetesNickname is lowercase and alphanumeric. The namespace is what ties the different services of a Self Managed Commerce environment together and allows them to find and communicate with each other.
The kubernetesNickname value must be unique across your AWS account to avoid conflicts in database server naming. If you specify a value used by an existing database service, then the configuration of that service will be updated.
When you deploy the Elastic Path stack that will use this database server, specify the same value for kubernetesNickname. The kubernetesNickname value used should match the namespace you have used for the deployment of ActiveMQ and the namespace you have planned for the Elastic Commerce stack to be deployed into.
backupWindowPreference
Specify a preferred backup window using a 24-hour clock. The window should be a period of at least 30 minutes, such as 11:00-13:00. If you do not want to specify a preferred backup window, leave the parameter value as AmazonsDefault. Amazon Aurora assigns a default 30-minute backup window.
For more information about the backup window, see the RDS documentation on the default backup window.
enablePerformanceInsights
When you set this parameter to true, you enable the free tier of the Amazon Performance Insights database performance tuning and monitoring feature for the two instances created.
backupRetentionDays
The number of days that RDS retains restore data. The number of days can be from 1 to 35. RDS backs up your cluster volume automatically and retains restore data for the length of the backup retention period.
For more information about the backup retention days, see the RDS documentation on the backup retention period.
enableAuditLogging
When selected, RDS advanced auditing is enabled, and audit data is captured in Amazon CloudWatch. KMS based encryption of CloudWatch logs for the database logs group will be turned on. You can provide the ARN of a custom KMS key in encryptionKey. A KMS key is generated if you do not provide one.
note
If you are providing a custom KMS key ensure that the RDS log group has permission to use the key. For more information, see Set Permissions on the Key.
useLTSVersion
Select this parameter to use a Long Term Support (LTS) engine version of Aurora MySQL. This parameter is ignored if the databaseType parameter is not set to aurora-mysql. If you are using non-LTS versions of the Aurora engine, set this parameter to false.
auroraEngineVersion
This parameter is ignored if the databaseType parameter is set to postgres-rds. If the useLTSVersion Jenkins job parameter is set to false, use this version of Aurora engine instead.
encryptDatabase
Optional. When selected, the RDS database uses at rest encryption.
encryptionKey
Optional. If encryptDatabase or enableAuditLogging is set to true, you can provide the Amazon Resource Name of a custom AWS Key Management Service key. A KMS key is generated if you do not provide one.
preventKeyDeletion
Optional. Select this parameter if you are deleting your database server and want to ensure that your generated KMS key will not be deleted. If you have database snapshots or backups of the database server created by this job that you need to read in the future, you might need the KMS key. This parameter only runs when deleteServer is selected.
dbInstanceClass
note
The RDS DB instance class does not immediately change when you update this value for an existing database server. Instead, the change is applied in the next maintenance window. Go to the Maintenance & backups tab, to see the scheduled changes in your RDS database server in the Amazon RDS console.
Specify the RDS DB instance class to use. Choose a DB instance class that meets your processing power and memory requirements. The value must be a valid RDS instance class that is available in your AWS region. The smallest supported option is db.r5.xlarge.
For more information on the available instance classes, see DB instance classes.
cloudOpsForKubernetesBranch
The branch of the cloudops-for-kubernetes Git repository to use during the deployment.
cloudOpsForKubernetesRepoURL
The URL to your hosted copy of the cloudops-for-kubernetes Git repository.
Restoring a database snapshot in a new database instance
The Jenkins job create-and-manage-database-server can also restore a database snapshot in a new database instance.
To use this feature you must fill out the parameters in the create-and-manage-database-server job with values that match the snapshot that you are restoring. The auroraEngineVersion field must be populated and useLTSVersion set to false. The snapshot must have been created for the selected database type and engine - no conversion will be performed. Snapshots can only be used when creating a new instance; an error will occur if you try to restore a snapshot to an existing database instance.
After ensuring that the regular create-and-manage-database-server job parameters are correctly filled, fill out the parameters in the section Restore Database from Snapshot Options.
Parameters for Restore Database from Snapshot Options
snapshotToRestore
Specify a valid Amazon database snapshot available in the AWS account to use when creating the database instance; the value for this field can be found in the Amazon RDS console in the column Snapshot Name.
databaseName
The name of the database in the snapshot. You can find the name of an existing Self Managed Commerce database by inspecting the kubernetes secret ep-<mynamespace>-mysql-secret.
databaseRootPassword
The root password of the database snapshot. You can find the root password in an existing Self Managed Commerce database by inspecting the kubernetes secret ep-<mynamespace>-mysql-secret.
databaseUsername
The username of the database snapshot. You can find the username in an existing Self Managed Commerce database by inspecting the kubernetes secret ep-<mynamespace>-mysql-secret.
databasePassword
The password of the database snapshot. You can find the password in an existing Self Managed Commerce database by inspecting the kubernetes secret ep-<mynamespace>-mysql-secret.
Creating a cloud database server (deprecated in favour of the create-and-manage-database-server Jenkins job)
To create an AWS Aurora RDS MySQL Database cluster, run the create-or-delete-mysql-server job in Jenkins with the following parameters:
deleteServeruseLTSVersionauroraEngineVersionencryptDatabaseencryptionKeypreventKeyDeletionbackupRetentionDaysbackupWindowPreferencedbInstanceClassenableAuditLoggingcloudOpsForKubernetesRepoURLcloudOpsForKubernetesBranchkubernetesNicknameclusterName
The job creates an Aurora RDS MySQL cluster with a default and standby node on the private subnets that were created along with your EKS cluster. By default, each node is a db.r5.xlarge. Each node has four vCPUs and 32 GB RAM. The server is running MySQL 5.7 and is configured for geo-redundant daily backups that are preserved for 14 days. The kubernetesNickname parameter should be unique across your AWS account to avoid conflicts in database server naming.
The username and password are randomly generated and stored in a Kubernetes Secret.
The job sets the server settings:
character_set_server=UTF8MB4
tx_isolation=READ-COMMITTED
This Job also creates a new RDS security group that allows access to the database from within the VPC that was created along with the EKS cluster.
Parameters for the create-or-delete-mysql-server Jenkins Job
applyImmediately
This parameter only works if you change either the RDS instance class or the RDS engine version.
Set this parameter to true to immediately have any specified changes to the RDS Aurora instance class or engine version applied.
Warning
The database instance and any Self Managed Commerce services that rely on the database instance will be unavailable while the changes are applied.
Ensure that you set this parameter to true if you are making changes to the engine version. You cannot make changes to the engine version using this Jenkins job if the parameter is set to false. This is due to a limitation of the AWS Terraform provider implementation.
Set this parameter to false if you want to make changes to the instance class without downtime. Those changes will apply during the next RDS maintenance window.
Note
Engine version upgrades may still be applied without downtime in the RDS maintenance windows if you do the engine version upgrade in the AWS RDS console. If you follow that approach, you must ensure that those changes have been completed entirely before running this job.
enablePerformanceInsights
When you set this parameter to true, you enable the free tier of the Amazon Performance Insights database performance tuning and monitoring feature for the two instances created.
deleteServer
Deletes the server instead of creating it.
useLTSVersion
This parameter ensures the validation of the dbInstanceClass parameter against Long Term Support (LTS) versions of Aurora MySQL.
If you are using non-LTS versions of the Aurora engine, set this parameter to false.
This is used when running a high performance configuration of Self Managed Commerce on Arm-based AWS Graviton2 processor instance types specified in the dbInstanceClass Jenkins job parameter.
auroraEngineVersion
If the useLTSVersion Jenkins job parameter is set to false, use this version of Aurora engine instead.
This is used when running a high performance configuration of Self Managed Commerce on Arm-based AWS Graviton2 processor instance types specified in the dbInstanceClass Jenkins job parameter.
encryptDatabase
Optional. When selected, the RDS database uses at rest encryption.
encryptionKey
Optional. If encryptDatabase or enableAuditLogging is set to true, you can provide the Amazon Resource Name of a custom AWS Key Management Service key. A KMS key is generated if one is not provided.
preventKeyDeletion
Optional. Select this parameter if you are deleting your MySQL server and want to ensure that your generated KMS key will not be deleted. If you have database snapshots or backups of the MySQL server created by this job that you need to read in the future, you might need the KMS key. This parameter is only run when deleteServer is selected.
backupRetentionDays
The number of days that Amazon Aurora retains restore data. The number of days can be from 1 to 35. Amazon Aurora backs up your cluster volume automatically and retains restore data for the length of the backup retention period.
For more information about the backup retention days, see Overview of backing up and restoring an Aurora DB cluster.
backupWindowPreference
Specify a preferred backup window using a 24-hour clock. The window should be a period of at least 30 minutes, such as 11:00-13:00. If you do not want to specify a preferred backup window, leave the parameter value as AmazonsDefault. Amazon Aurora assigns a default 30-minute backup window.
For more information about the backup window, see Overview of backing up and restoring an Aurora DB cluster.
dbInstanceClass
note
The Amazon Aurora DB instance class does not immediately change when you update this value for an existing Aurora RDS MySQL cluster. Instead, the change is applied in the next maintenance window. You can see the scheduled changes in your RDS cluster in the Amazon RDS console by clicking the "Maintenance & backups" tab.
Specify the Amazon Aurora DB instance class to use. Choose a DB instance class that meets your processing power and memory requirements. The value must be a valid Aurora instance class that is available in your AWS region. The smallest supported option is db.r5.xlarge.
For more information on the available instance classes, see DB instance classes.
enableAuditLogging
When selected, Amazon Aurora advanced auditing is enabled, and audit data is captured in Amazon CloudWatch. KMS based encryption of CloudWatch logs for the database logs group will be turned on. You can provide the ARN of a custom KMS key in encryptionKey. A KMS key is generated if you do not provide one.
note
If you are providing a custom KMS key ensure that the RDS log group has permission to use the key. For more information, see Set Permissions on the Key.
cloudOpsForKubernetesRepoURL
The URL to your hosted copy of the cloudops-for-kubernetes Git repository.
cloudOpsForKubernetesBranch
The branch of the cloudops-for-kubernetes Git repository to use during the deployment.
kubernetesNickname
This job uses the value of kubernetesNickname as the Kubernetes namespace where the database connection information is stored, and in the name of the database server. If the namespace does not exist, this job creates it. To be compatible with Terraform, Kubernetes, and DNS domain name requirements, ensure that the kubernetesNickname is lowercase and alphanumeric.
The kubernetesNickname value must be unique across your AWS account to avoid conflicts in database server naming. If you specify a value used by an existing database service, then the configuration of that service will be updated.
When you deploy the Elastic Path stack that will use this MySQL server, be certain to specify the same value for kubernetesNickname.
clusterName
The name of the Kubernetes cluster that stores database connection information. To use a MySQL server created by this job, an Elastic Path stack must be deployed in the same Kubernetes cluster.
Using an Existing Database Server
You can use an existing database server if the database instance is accessible to services running in the Kubernetes cluster. The Jenkins job use-existing-database-server creates the necessary resources required for an existing database server to be used by an Elastic path Commerce stack. This job can use an existing database or create a new database in the database server. Only MySQL Aurora and Postgres RDS are supported.
Run the Jenkins job use-existing-database-server to allow an Elastic path Commerce stack in CloudOps for Kubernetes to use an existing database server. This creates the necessary Kubernetes secrets that the Self Managed Commerce stack will use to connect to the external database instance.
For information on migrating a database from CloudOps for AWS to CloudOps for Kubernetes, see Migrating a Database from CloudOps for AWS to CloudOps for Kubernetes.
Parameters for the use-existing-database-server Jenkins Job
deleteDatabase
When checked, this deletes the database instead of creating it.
databaseType
The database type of the existing database server. Only MySQL Aurora and Postgres RDS are supported.
databaseRootUsername
The root username of the external database instance. If the instance was created by the create-and-manage-database-server job, the root username can be found in the Kubernetes secret named ep-<kubernetesNickname>-mysql-secret. If the instance was created through the AWS RDS console, then the root username can be found on the configuration page of the instance on the AWS RDS console.
databaseRootPassword
The root password of the external database instance. It can be found in the Kubernetes secret created by the Jenkins job create-and-manage-database-server or configured in the AWS RDS console at server creation time.
databaseServerUrl
The endpoint of the server to connect to. It is found in the web console of your cloud provider. The format of an Amazon RDS cluster endpoint is sample-database.cluster-asdf.us-west-2.rds.amazonaws.com.
useExistingDatabase
When checked, this job will use the existing database and user credentials provided in databaseName, databaseUsername, and databasePassword instead of creating a new one.
databaseName
The name of the database in the external database server. If useExistingDatabase is false, the job creates a database with this name. In this case, it can also be left blank and a database name will be generated by this job. If useExistingDatabase is true, the database name provided must already exist and will be used for this job instead.
databaseUsername
The user of the database in the external database server. If useExistingDatabase is false, the job creates a user with this name. In this case, it can also be left blank and a username will be generated by this job. It should not be the same as databaseRootUsername. If useExistingDatabase is true, the username provided must already exist and will be used for this job instead.
databasePassword
The password of the database user. If useExistingDatabase is false, a new user will be created for the new database. In this case, it can also be left blank and a password will be generated by this job. It should not be the same as databaseRootPassword.
kubernetesNickname
The nickname for this group of resources. The nickname corresponds to the namespace that your resources will be deployed in and must exactly match the kubernetesNickname value you specify for the other components of the given Self Managed Commerce environment. The nickname ties the different services of a Self Managed Commerce environment together and allows them to find and communicate with each other. For more information, see Planning a Self Managed Commerce deployment.
cloudOpsForKubernetesBranch
The branch of the cloudops-for-kubernetes Git repository to use during the deployment.
cloudOpsForKubernetesRepoURL
The URL to your hosted copy of the cloudops-for-kubernetes Git repository.