If you’re using GitHub Advanced Security with your own runners or with Actions Runner Controller (ARC), you’ve likely encountered the need to create an environment that supports CodeQL analysis. Today, I’ll walk you through the steps to create a custom CodeQL Docker image for use with ARC. If you’re running virtual machines, the same steps apply but you’ll need to adapt these for how you build your virtual machine images.
Prerequisites
Before you can create an image, there are some articles you should be familiar with:
These articles contain the specifics of what you need to run CodeQL, but here are some important highlights:
- Support on Linux is limited to Ubuntu 22.04 and 24.04 running on the x64 (AMD64) architecture with
glibc2.17 or better. While you may be able to get CodeQL to run on other distributions or architectures, it is not officially supported. - You’ll want your container to have at least 8GB of RAM and two vCPUs (but I recommend at least twice that for most codebases).
- You must have
gitinstalled and it must be in the PATH. - You’ll also often need the languages you want to analyze to be installed. Generally speaking, for a universal image you will want at least Python 2, Python 3, Java, JavaScript (Node.js), and .NET to be available.
Thankfully, the default Actions Runner image (ghcr.io/actions/actions-runner:latest) meets several of these requirements, so it gives us a good starting point. The documents don’t explicitly mention it, but you will also want to install CodeQL on the image. The binaries are more than 1.6GB, so you want to ensure that it doesn’t need to be downloaded every time you run a job!
The three paths
There are three options for creating an image that will support CodeQL. The first option is to install each of the language dependencies manually. If you’re doing this, you also want to ensure that you create JAVA_HOME_$VERSION_$PLATFORM environment variables for each version of Java you install (for example, JAVA_HOME_11_X64), as well as JAVA_HOME pointing to a default version. This has two downsides, however. First, the workflows might not be able to take advantage of any language binaries that aren’t properly installed into the tool cache, so you may download some files a second time. Second, CodeQL has changed the way it’s versioned (which impacts how it’s stored in the tool cache) in the past. Any time this happens, you may need to update your folder structures to match.
The second option is the one I mention in my earlier post about
how to create a tool cache. This involves creating a tool cache and populating it with CodeQL and each of the language dependencies (using actions/setup-java, actions/setup-python, etc). You will still need to configure the Java environment variables, but these Actions will do most of the rest of the work for you. An additional benefit of this approach is that your workflows can take advantage of the tool cache, speeding up the setup of your jobs.
If you take this approach, be aware that the tool cache on the GitHub-hosted runners is built manually and designed to match how the setup- Action expects to find the files structured. For some of the default tools, the hosted runners use symbolic links to point to the actual binaries. This is part of why the article deletes the existing tool cache. I wanted to ensure that the dependencies are installed in a predictable way each time.
The third option is a hybrid of the two approaches. I want to use this one today because it demonstrates how to create a Docker image using some of the same techniques that the Actions team uses to create the official GitHub hosted runner VM images. It reuses part of the image building process to craft the Docker image.
No, it is not possible to directly use the official GitHub hosted runner VM images as a base for your Docker image. The images are not available publicly. They are also configured to build a complete virtual machine disk image, which is not the same thing as a Docker image. The two formats are not compatible with each other, so the image could not be directly used for Docker or Kubernetes.What you need to know about the process
The scripts are part of the
actions/runner-images repository. The scripts are called to build up the virtual machine images, but they are well-organized and with some minor adjustments can be used as part of a Docker build process. The files you need are in the images/ubuntu directory.
Before the scripts are run to install the binaries, a few things have already happened:
- The environment variable
HELPER_SCRIPTSprovides the full path to theimages/ubuntu/scripts/helpersdirectory. - The environment variable
INSTALLER_SCRIPT_FOLDERprovides the full path to theimages/ubuntu/toolsetsdirectory. - The
toolsets-2404.jsonfile in thetoolsetsdirectory contains the list of tool versions and related information. This file is copied totoolset.json, which gets referenced in the scripts. - The permissions for all of the shell scripts are updated to be executable.
- The file
${HELPER_SCRIPTS}/invoke-tests.shis copied to create the file/usr/local/bin/invoke-tests. This script is used to run some PowerShell scripts that test the installation of the tools. - Some environment variables have been configured and written to
/etc/environment, includingAGENT_TOOLSDIRECTORY, the path to the tool cache. - Some prerequisites (such as
lsb-releaseandunzip) are installed. - The
rootuser is configured to be able to runsudo. This is important because the scripts have places where the code assumes the current user isroot.
Building a Dockerfile
To use just the installation scripts you need for CodeQL, you will need to reproduce these steps in your Dockerfile. As always, there’s a few ways we can do this (and plenty of areas we can optimize!), but I want to focus on implementing the process itself. I’ll use some advanced Dockerfile features such as multistage builds and Heredocs, so the Dockerfile will start with a syntax declaration that ensures the Dockerfile is interpreted correctly. From there, I’ve annotated the Dockerfile to explain each step in the process.
1# syntax=docker/dockerfile:1
2
3## Settings that we can change at build time, including the base image
4## You can provide either a specific version or a completely different base image.
5ARG RUNNER_VERSION=latest
6ARG BASE_IMAGE=ghcr.io/actions/actions-runner:${RUNNER_VERSION}
7ARG AGENT_TOOLSDIRECTORY=/tools
8
9## First stage of the build will create an updated base image.
10FROM ${BASE_IMAGE} AS tools
11
12## Allow this variable to be set as part of this stage
13ARG AGENT_TOOLSDIRECTORY
14
15## Configure the environment variable for the tools directory
16## so the runner can use it (and more importantly, because
17## the scripts expect it to be set to know where to install the tools).
18ENV AGENT_TOOLSDIRECTORY=${AGENT_TOOLSDIRECTORY}
19
20## Change to the root user to install the tools (base image is using `runner`)
21USER root
22
23## Use the heredoc syntax to run a series of commands using the Bash shell
24## instead of the default `sh` shell. The script ends when a line with just
25## EOF is encountered. By doing a lot of steps inside a single RUN command,
26## it creates a single layer with the final layout instead of multiple
27## layers representing diffs between each command.
28RUN << EOF bash
29 ## Configure the Bash error handling
30 set -euxo pipefail
31
32 ## Update the package list so that we can install some packages
33 apt-get update
34
35 ## Quietly install the prerequisites we need to run the scripts
36 apt-get install -qqq -y wget lsb-release unzip
37
38 ## Clone the repository. The repository is 19M, I like small and fast!
39 ## As a blobless clone, it just downloads the list of files and directories
40 ## (trees), reducing the size of the clone to 8.4M. Since it is also
41 ## a shallow clone (depth 1), it only downloads the latest commit -- 332K.
42 ## By making it sparse, I only download the blobs for files in the root
43 ## directory, `images`, and `images/ubuntu` (three "cone"). The final size
44 ## is about 1.2M, or less than 10% of the original repository size!
45 git clone --sparse --filter=blob:none --depth 1 https://github.com/actions/runner-images
46
47 ## Move into the repository directory, but remember where we are for later.
48 pushd runner-images
49
50 ## Perform a sparse checkout. This will cause the blobs for the "cone"
51 ## `images/ubuntu` to be downloaded and placed in the working directory.
52 ## You will now have all of the directories and files below `images/ubuntu`
53 ## (plus the files -- but not the directories -- from `images`)
54 git sparse-checkout add images/ubuntu
55
56 ## Move into the `images/ubuntu` directory to set everything else up.
57 ## I'm not using pushd because I won't need to return back to the parent
58 cd images/ubuntu
59
60 ## Set some environment variables for the script. AGENT_TOOLSDIRECTORY
61 ## is already set for the image. These just need to be set for this script.
62 ## Because Docker tries to populate any variables in the RUN command,
63 ## I use a slash to escape the dollar sign. That gets removed when
64 ## the script is executed. For example ${AGENT_TOOLSDIRECTORY} is
65 ## interpreted by Docker and populated before the script is run, but
66 ## \${INSTALLER_SCRIPT_FOLDER} is converted to ${INSTALLER_SCRIPT_FOLDER}
67 ## when the final script is executed inside the container.
68 ## I want ${PWD} to be interpreted by the script (not Docker), so I
69 ## escape it as well.
70 export DEBIAN_FRONTEND=noninteractive
71 export HELPER_SCRIPTS=\${PWD}/scripts/helpers
72 export INSTALLER_SCRIPT_FOLDER=\${PWD}/toolsets
73
74 ## The directory is expected to exist
75 mkdir -p "${AGENT_TOOLSDIRECTORY}"
76
77 ## Create the toolset file that the scripts will use
78 cp "\${INSTALLER_SCRIPT_FOLDER}/toolset-2404.json" "\${INSTALLER_SCRIPT_FOLDER}/toolset.json"
79
80 ## And make sure all of the .sh scripts are executable, since many
81 ## are missing that bit in the repository
82 find "\${PWD}/scripts" -type f -name "*.sh" -exec chmod +x {} \;
83
84 ## Instead of setting up PowerShell and making sure we can run the test
85 ## scripts, I will create an empty script so that nothing breaks
86 ## when the command is called
87 echo "#!/bin/bash" > /usr/local/bin/invoke_tests
88 chmod +x /usr/local/bin/invoke_tests
89
90 ## Root use needs to be able to run sudo, but I want to leave the
91 ## sudoers file like I found it. So, I will back it up, then
92 ## add the line that allows `root` to run `sudo
93 cp /etc/sudoers /etc/sudoers.bak
94 echo 'root ALL=(ALL) ALL' >> /etc/sudoers
95
96 ## FINALLY! We can now run the scripts to install the tools!
97
98 ## Configure Java and set it up in the tool cache (lots of symlinks!)
99 ./scripts/build/install-java-tools.sh
100
101 ## Install and configure the latest version of CodeQL (huge!)
102 ./scripts/build/install-codeql-bundle.sh
103
104 ## Install Node.js. This one is not being installed in the tool cache.
105 ## so you might consider doing that yourself using the logic in
106 ## the `actions/setup-node` Action.
107 ./scripts/build/install-nodejs.sh
108
109 ## Wait! What? No Python? I'll explain in a bit ...
110
111 ## Restore the sudoers file back to its original state
112 mv -f /etc/sudoers.bak /etc/sudoers
113
114 ## And remove invoke_tests, since it doesn't need to remain in the image.
115 rm /usr/local/bin/invoke_tests
116
117 ## Update the permissions on the tools directory so that the
118 ## default user, `runner`, can access it.
119 chown -R runner:docker "${AGENT_TOOLSDIRECTORY}"
120
121 ## Return back to the parent directory so we can also remove
122 ## the entire Git repository.
123 popd
124 rm -rf runner-images
125
126 ## Finally, clean up the package cache and temporary files.
127 rm -rf /var/lib/apt/lists/*
128 rm -rf /tmp/*
129EOF
130
131## And now we can return the image to using the `runner` user.
132USER runnerHopefully all of those steps make sense. Obviously, some of these things can be setup or cached outside of the Dockerfile, but this updates the image to have most of the necessary bits for CodeQL.
Building the image
If you’re using Docker, you can build the image using the following command:
1docker build --target tools -t codeql-base:latest --platform linux/amd64 .The platform option is important because CodeQL only officially supports the x64 (AMD64) architecture. In fact, these scripts are also really designed for x64 at the moment. As a result, you need to ensure that you are always building only for that architecture. Of course, if you’re on an Intel/AMD machine, that platform will be the default.
If you’re using Actions, you can use the default workflow that Actions provides for “Docker Publish”. You may want to explicitly set the platform here as well:
1# Build and push Docker image with Buildx (don't push on PR)
2# https://github.com/docker/build-push-action
3- name: Build and push Docker image
4 id: build-and-push
5 uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09 # v5.0.0
6 with:
7 context: .
8 push: ${{ github.event_name != 'pull_request' }}
9 tags: ${{ steps.meta.outputs.tags }}
10 labels: ${{ steps.meta.outputs.labels }}
11 target: tools
12 platforms: linux/amd64
13 cache-from: type=gha
14 cache-to: type=gha,mode=maxI’ve already talked about the various options you have available for your
Docker layer caching, so I won’t repeat that here. For product images, I generally recommend the registry cache. While you can use the GitHub Actions cache, a single build of this image will use almost 2.5GB of space, which is a pretty big chunk of the 10GB limit for the cache. That means it takes just 3 PR branches (plus main) to start to overrun the cache limit if you’re using GHA caching.
One last thing – you may have noticed that nothing in the Dockerfile script (aside from escaping the dollar signs) is really specific to Docker. In fact, you could incorporate most of that code directly into the scripts used to build a virtual machine if you’re building standalone, non-ephemeral runners. The same logic applies. Of course, you might be able to adopt more of the Packer approach that’s in that repo, but that’s a topic for another day.
I think we missed a step
So, why didn’t I run the script to install and configure Python? It turns out the base image we’re using already has Python 3 installed, so we don’t need to do that a second time. Unfortunately, the script doesn’t configure Python 2, so that is something that we would need to implement manually. Unfortunately, Python 2 is no longer available in the Ubuntu repositories, so it requires building from source.
That’s why this Dockerfile is set up for multiple stages. Docker’s BuildKit can run multiple tasks in parallel, so we can allow this additional work to happen in a separate stage. That way, we can take just the binaries we need and copy them into the final image. This is a bit more work, so I’ll cover that in the next post.