Creating and editing jobs in Jenkins is a tedious and error-prone task. A Jenkins server with a handful of plugins will have dozens of text fields and checkboxes for various configurable settings. Creating a new job typically involves editing lots of form fields, and those changes are not saved to version control. I despise the Jenkins configuration screen; it’s tedious, error-prone, and repetitive. Even worse, it matches the definition of a “pet” in DevOps lingo.
The Job DSL Plugin for Jenkins provides a mechanism for defining Jenkins jobs as code. The code is written in Groovy, so developers can define Jenkins jobs with the help of variables, conditional statements, and loops. The Job DSL API integrates with the most popular Jenkins plugins.
Wiring up the Job DSL plugin into a useful platform for developers takes some work. Like most APIs, the plugin defines the interfaces for creating jobs, but leaves the orchestration details to the team to define. Below, I detail three powerful tips for integrating the Job DSL platform into a team’s development workflow.
Tip 1: The Mother Seed Job
With the Job DSL plugin, a Jenkins job executes a Groovy script that creates more jobs. Unfortunately, this has a chicken-or-the-egg problem: what creates the first job? I suggest creating a starter job that is installed using the same provisioning framework used to deploy Jenkins itself (Puppet, Chef, Ansible, shell script, etc.).
My team refers to our job as the “Mother Seed Job”, and our bootstrapping process works like this:
1) Manually create a Job DSL Groovy script that loads all the other groovy scripts
2) Create a Jenkins job that runs the new Job DSL script
3) Jenkins saves the configuration for the job on the file system as an XML file. Copy the XML file for this job and inject it into the provisioning pipeline. For example, in Puppet via the puppet-jenkins module:
[pcsh lang=”groovy” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
::jenkins::job { 'mother-seed-job':
config => template("my-module/mother-seed-job.xml.erb")
}[/pcsh]
The provisioning tool can also import the XML with the jenkins-cli.jar provided by Jenkins:
[pcsh lang=”groovy” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
java –jar jenkins-cli.jar -s https://jenkins-server create-job mother-seed-job < mother-seed-job.xml[/pcsh]
In the first step above, how does the mother seed job find other jobs to create? My team created a schema like the example below, where each line item becomes a new “seed job” (e.g. stable-project Seed Job) in Jenkins that creates its own jobs.
[pcsh lang=”groovy” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
Environments {
all [
[projectName: "stable-project", url: "https://github.com/repo1/stable-project-Jenkins-automation"],
]
dev [
[projectName: "new-project", url: "https://github.com/repo2/new-project-Jenkins-automation"],
]
prod [
]
}[/pcsh]
This setup has a slew of benefits:
The config is saved to version control
- Empower developers to add new projects by adding a single line to the configuration. Similarly, it’s easy to remove all the jobs for a deprecated project.
- Since the job watches for updates to the repo, any changes (i.e. added or removed jobs) are deployed automatically.
Each Jenkins server automatically gets the jobs it needs when it first boots. This makes the Jenkins server a truly disposable asset, rather than a “pet”.
- Some of our teams destroy and recreate their Jenkins server after every sprint to emphasize the importance of submitting jobs to version control.
Project jobs can be in one Jenkins server and moved or copied to another with a simple code change.
- In the example above, new-project doesn’t need to be in production Jenkins yet, but can be added to production in the future by moving the line from ‘dev’ to ‘all’.
Tip 2: Create job builders
To reduce the effort needed to create new jobs, I suggest creating a library of generic job builders. An object oriented mindset will encourage teams to discuss and capture norms and repeated patterns as reusable code.
For example, if a norm is that build logs should only be kept for 30 days, that configuration should be part of the “base” builder. Additional builders can be created to extend the base, templatizing common patterns is a team’s jobs.
CFPB’s jenkins-automation repo provides a good example of this practice. It has BaseJobBuilder identifying the norms and other builders that add functionality on top of that. Let’s use their repo to implement a job that tests if google is working:
[pcsh lang=”groovy” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
import Jenkins.automation.builders.SiteMonitorJobBuilder
new SiteMonitorJobBuilder(
name: "google-pulse-check",
description: "Ensure Google is up",
cronSchedule: "@hourly",
urls: ["https://google.com"]
).build(this);[/pcsh]
Without a Builder to inherit, the groovy code for this task would have been more than 50 lines. It’s a very simple example, but it shows how job builders can help decouple the repetition often present in Jenkins jobs.
Tip 3: Use the REST API Runner
One of the challenges of developing new DSL jobs is testing that the Groovy code creates the expected job configuration. With the mother seed job pattern I described above, groovy code is automatically run when committed to version control. However, as a developer I prefer to not commit code that I’m not sure is working.
It’s possible to write tests for DSL code, which can run with a command like `./gradlew test`, but I also prefer to get the job on a Jenkins server (in a Vagrant box, for example) and assess the result. Fortunately, there’s a way to do this without committing to version control!
The gradlew toolchain provides a mechanism for pushing a job via Jenkins’ RESTful API, which is detailed in the job-dsl-gradle-example github repo. For example, the following command deploys the Google pulse check job to my local testing Jenkins server.
[pcsh lang=”groovy” tab_size=”4″ message=”” hl_lines=”” provider=”manual”]
./gradlew rest -Dpattern=jobs/google-pulse-check.groovy -DbaseUrl=https://localhost
[/pcsh]
I’ve found that a workflow where jobs can be deployed “early and often” for validation increases developer confidence and efficiency.
The three tips above are not requirements to enjoy the benefits of the Job DSL plugin. However, integrating the tool with a large set of jobs and a large development team takes a bit of effort. For our teams, the suggestions above enable a completely automated platform that is based on reusable and testable code.