Jenkins Upgrade in release 3.5 · CloudOps for Kubernetes

Overview

CloudOps for Kubernetes release 3.5.0 introduces a newer version of Jenkins. All newer versions of Jenkins, and more specifically Jenkins plugins used for running pipelines, have changed the file structure of the Jenkins workspace. The Jenkins developers made these changes to address security vulnerabilities. The effect of these changes is that the location of the cloudops-for-kubernetes Git project inside of the workspace cannot be known or predicted.

Upgrade Instructions

Reviewing and updating your custom Jenkins pipelines is a key part of the upgrade process, but other steps are also required. Review and follow all of the instructions in Update to Version 3.5 when upgrading to CloudOps for Kubernetes release 3.5.0.

Jenkins Pipeline Changes

Jenkins pipelines in previous versions of CloudOps for Kubernetes relied on the ability to predict the location of the cloudops-for-kubernetes Git project inside of the workspace. Most commonly, the Jenkins pipelines relied on finding the necessary files to configure the Jenkins agents.

note

The root of the issue is that the location of the CloudOps for Kubernetes Git repository is not known or predictable in newer Jenkins versions. Previously, we relied on the path to code being ${env.JENKINS_HOME}/workspace/${env.JOB_NAME}@script/cloudops-for-kubernetes. With newer Jenkins versions we cannot predict the path or determine it based on environment variables.

Previous Approach

Below is an example, typical of pipelines in previous CloudOps for Kubernetes versions, that no longer works with the newer versions of Jenkins.

def podYamlFromFile = new File("${env.JENKINS_HOME}/workspace/${env.JOB_NAME}@script/cloudops-for-kubernetes/jenkins/agents/kubernetes/docker-0.5gb-0.25core-1container.yaml").text.trim();
String podYaml = podYamlFromFile.replace('${dockerRegistryAddress}', "${dockerRegistryAddress}").replace('${jenkinsAgentImageTag}', "${jenkinsAgentImageTag}")

pipeline {
  agent {
    kubernetes {
      label "${randomLabel}"
      defaultContainer "jnlp"
      yaml "${podYaml}"
    }
  }
  ...
}

In newer versions of Jenkins, the path ${env.JENKINS_HOME}/workspace/${env.JOB_NAME}@script/cloudops-for-kubernetes is no longer a valid way to find the location of the CloudOps for Kubernetes Git repository.

New Approach

All out of the box pipelines have been updated to use a new approach to configure the Jenkins agents, where the agent configuration files are now loaded using a function, named kubernetesPodTemplate, that is defined in a Jenkins global shared library. Below is an example of the new approach.

library "shared-library@${params.cloudOpsForKubernetesBranch}"

pipeline {
  agent {
    kubernetes {
      yaml kubernetesPodTemplate(file: "docker-0.5gb-0.25core-1container.yaml", binding: ['dockerRegistryAddress': dockerRegistryAddress, 'agentImageTag': jenkinsAgentImageTag])
      defaultContainer "docker"
      inheritFrom "jenkins-agent"
    }
  }
  ...
}

Updating Custom Pipelines

While pipelines provided by CloudOps for Kubernetes are updated to use the new approach, custom pipelines that you previously created could be affected by this Jenkins behaviour change.

Reviewing Custom Pipelines

It is important to review and update all custom pipelines that you rely on to ensure they are compatible with the new Jenkins behaviour. Any pipelines that are not updated will fail when run with newer Jenkins versions.

To check if a custom pipelines is impacted, search in the pipeline groovy code for the string /workspace. Any reference to paths inside of the CloudOps for Kubernetes git repository under cloudops-for-kubernetes will need to be reviewed and udpated.

Changes to make

If you have custom piplines impacted by this change, the fix will depend on which workspace files you are referencing and for what purpose.

Where referencing agent configuration files

Custom pipelines that reference an agent configuration file in the workspace, such as ${env.JENKINS_HOME}/workspace/${env.JOB_NAME}@script/cloudops-for-kubernetes/jenkins/agents/kubernetes/docker-0.5gb-0.25core-1container.yaml, will need to be updated to use the new approach. See the example above, or refer to out of the box pipelines for examples.

The names of the agent files provided out of the box can be seen in the CloudOps for Kubernetes git project, under path resources/com/elasticpath.

tip

If you need to define a custom agent configuration file, you can do so by adding a new file to the resources/com/elasticpath directory in the CloudOps for Kubernetes git repository.

Where referencing other files in the workspace

If custom pipelines reference other files in the workspace then you will need to investigate the correct solution.

Timing The Change

Do not make these changes to your custom pipelines until after the Jenkins upgrade has been completed as part of the CloudOps for Kubernetes 3.5.x upgrade. The changes described above are not backwards compatible with previous versions of CloudOps for Kubernetes.

The recommended approach for implementing custom piplelines is to check them into a Git project, as opposed to editing them in the Jenkins UI.

tip

Prepare your pipeline updates for CloudOps for Kubernetes 3.5.x in a new branch while pipelines for CloudOps for Kubernetes 3.4.x are in an existing branch.

Checking your custom pipelines into a Git project is also important to ensure that you can recreate them in, for example, a disaster recovery situation. For more information about the recommended approach to custom pipelines, see Adding or Customizing Jenkins Jobs.