Customizing Jenkins

Adding or Customizing Jenkins Jobs

It is very common to add new Jenkins jobs or override Jenkins jobs provided in CloudOps for Kubernetes. CloudOps for Kubernetes includes a framework for importing custom Jenkins jobs into the Jenkins interface. The framework has several benefits, such as:

• It avoids the need to modify the CloudOps for Kubernetes cloud-ops-kubernetes Git repository, which can simplify the work required to apply future CloudOps for Kubernetes upgrades and patches.
• It allows you to implement and configure your Jenkins jobs as code, ensuring that they are saved and changes are tracked in a Git repository that you control completely.
• It ensures that your Jenkins jobs will not be lost or disappear from the CloudOps for Kubernetes Jenkins instance when the Jenkins helm chart gets updated.

For more information about this recommended approach for adding new or overriding provided Jenkins jobs, see the CloudOps for Kubernetes Extensions Framework documentation.

CloudOps for Kubernetes Jenkins Architecture

A Jenkins server is deployed during the initial setup process. The implementation provided is based on the following community tools:

• Jenkins Helm chart
• Jenkins Kubernetes plugin
• The img daemon-less image builder

It is an option to customize and redeploy Jenkins if you require changes to meet your needs.

Configuring Jenkins

The configuration for Jenkins is split into the following:

• The bootstrap/jenkins-helm-values.yaml.tmpl file. This provides the majority of the configuration for Jenkins and includes:
  • The admin credentials to access Jenkins.
  • Resource settings (RAM/CPU) for the Jenkins server. Resource settings for the agents are defined in files in jenkins/agents/kubernetes.
  • The plugins to install.
  • The configuration for the Jenkins plugin Kubernetes.
  • The persistent volume settings.
  • The Groovy based init script to run:
    • Creates the Jenkins Credential for the git server, based on the values set in docker-compose.yml for the bootstrap process.
    • Adds the ssh host key to the server, also from the docker-compose.yml file.
    • Creates and starts the bootstrap job inside Jenkins, where the job itself is defined in cloudops-for-kubernetes/jenkins/jobs/bootstrap/bootstrap.groovy.
• The files in the cloudops-for-kubernetes/jenkins/jobs directory which define the Jenkins jobs.
• The files in the cloudops-for-kubernetes/jenkins/docker directory which provides the container images that are used for the Jenkins agents.
• The jenkins-secrets Kubernetes secret. This is normally created by the bootstrap process and it provides many settings used by the Jenkins jobs.

Configuring Jenkins Using the Web Interface

You can make configuration changes using the Jenkins web interface. However, some of those changes might be lost when Jenkins restarts or is redeployed. If that happens then follow the below procedure to prevent this by setting TF_VAR_jenkins_overwrite_config to false.

1. Identify a maintenance window during which no Jenkins jobs will be running to apply the change. The Jenkins server pod will be restarted when the change is applied, causing any running jobs to fail.
2. Obtain the most recent copy of the docker-compose.override.yml file for your CloudOps for Kubernetes cluster.
3. Set the TF_VAR_jenkins_overwrite_config variable to false. If the variable does not exist in your copy of docker-compose.override.yml, you will need to add it.
4. (Optional) Set variable TF_VAR_rebuild_nodegroups to false to avoid unnecessary pod redeployment.
5. Follow the procedure described in Updating Cluster Configuration to apply the change.

note

Some CloudOps for Kubernetes version upgrades may require resetting the Jenkins configuration. Ensure you are following the upgrade documentation specific to your CloudOps for Kubernetes version.

tip

Record any custom configuration changes made using the Jenkins web interface so they can be manually restored if necessary.

Updating the Container Images for the Jenkins Agents

Jenkins agents in CloudOps for Kubernetes use the container images in the jenkins/agents/docker directory. If these container images need to be rebuilt, run the build-jenkins-agents job.

Updating the Jenkins Jobs

To update the implementation of Jenkins jobs, push the code to the CloudOps for Kubernetes repository and branch set in the docker-compose.override.yml file used during the bootstrap process.

If the configuration of the Jenkins job is changed by editing the jenkins/jobs/bootstrap/bootstrap.groovy file, run the bootstrap Jenkins job for the changes to be imported into the Jenkins server. The bootstrap job updates any jobs where the configuration has changed.

Updating Jenkins

By default, Jenkins is deployed during the initial setup process.

If you need to update Jenkins or its plugins, test the changes first, ideally in an environment where errors will not impact development or production operations. You can perform tests to validate that the newer plugins do not have breaking changes either for the bootstrap container or any of the Jenkins jobs.

Defining Additional Plugins

The Jenkins deployment includes a number of plugins needed by the CloudOps for Kubernetes environment. These plugins are defined in the bootstrap/jenkins-helm-values.yaml.tmpl file under the installPlugins section.

If you need additional plugins to be installed as part of the deployment process, these should be added to the bootstrap/jenkins-helm-values.yaml.tmpl file under the additionalPlugins section. Defining custom plugins in this section will allow for easier merging of changes that may be made in future versions of CloudOps for Kubernetes.

Plugin Upgrade Testing Process

1. Deploy a new environment using the bootstrap container

2. Manually upgrade Jenkins and its plugins using the Jenkins web interface

3. Collect the names and versions of the Jenkins plugins

   • The easiest way to get the plugin names and versions is to run the following code in the Jenkins Script Console:

     Jenkins.instance.pluginManager.plugins.each{
       plugin ->
         println ("${plugin.getShortName()}: ${plugin.getVersion()}")
     }

4. Update the code to use the new version of Jenkins and the plugins

   • The bootstrap/jenkins-helm-values.yaml.tmpl and jenkins.tf files would have to be modified

5. Deploy another new environment, using this newly updated code

6. Validate that all the Jenkins jobs in this new environment function as expected.

7. Update the Jenkins server version and plugins in the code of cloudops-for-kubernetes

8. Run the bootstrap container in "create" mode to apply the code changes

9. Remove the environments created for this task