Dynamic Build Matrices in GitHub Actions

Today’s post is more of a short tip. Using a matrix strategy in a build makes it easy to repeat the same set of steps with different sets of parameters. For each set of parameters, a new run will be created. These are commonly used to create a build (or test) for multiple platforms, split tests into multiple runs, or execute code in multiple environments or containers.

Actions are far more configurable than most people realize, thanks to the power of expressions. Most of the fields in a workflow that can accept a value will also accept an expression. The expression can provide a single value (such as a string) or a complex object (using fromJson). Creating dynamic values often requires some amount of compute to run the script or programming logic. As a result, the first step is to create a job. Next, in the steps perform any logic or calculations in the language of your choice. You can then append a line to a special file to create an output. That file is stored in the environment variable GITHUB_OUTPUT. It expects to receive an entry in the form name=value. To be able to retrieve that value, make sure to provide an id for the step. You can then use outputs at the job level to declare a job-level output based on your step output.

Fun trick: on Linux, you can use the jq app to create properly escaped JSON object strings. Just pass in a named variable and value using --arg name value. You can then reference the value of the variable as $name and it will be expanded in the final result. If you use --argjson name value, you can provide a JSON value. For Windows, consider PowerShell and ConvertTo-Json.

Putting that all together, assume I have a bash script (createTargets.sh) that dynamically creates and returns a JSON array of environment names:

1["dev", "qa", "prod"]

I want to create a job output called mymatrix with a single field, target. That field should contain the generated array. The output JSON from that step needs to look like this:

1{
2    "target": ["dev", "qa", "prod"]
3}

I’ll also create a step with the id of dataStep. That allows me to reference the step to create the output variable, mymatrix, from an output created in dataStep. The output is created by appending (>>) a name/value pair to a special file, $GITHUB_OUTPUT. The resulting job looks like this:

 1jobs:
 2  setup:
 3    runs-on: ubuntu-latest
 4    outputs:
 5      mymatrix: ${{ steps.dataStep.outputs.myoutput }}
 6    steps:
 7      - id: dataStep
 8        run: |
 9          TARGETS=$(./createTargets.sh)
10          echo "myoutput=$(jq -cn --argjson environments "$TARGETS" '{target: $environments}')" >> $GITHUB_OUTPUT          

I use jq to create a JSON encoded object that is properly escaped. The -cn parameter is short for -c -n. The -c (--compact-output) argument outputs the JSON on a single line without formatting. The -n (--null-input) argument indicates that jq does not need to process any input. I then use --argjson to create a variable called environments whose value is set from the TARGETS variable. I then define the object using jq’s object syntax.

The output variable definition is then appended to $GITHUB_OUTPUT, creating an output value that can be referenced by the job (or other steps):

1myoutput={"target":["dev","qa","prod"]}

The job defines an output, mymatrix, and sets the value to this step output:

1outputs:
2  mymatrix: ${{ steps.dataStep.outputs.myoutput }}

To consume that value in the next job (we’ll call it run-matrix), I need two things. First, I need to declare that the run-matrix job needs the first job, setup. This ensures the setup step runs first and makes its output variables available. Next, I need to dynamically define the entire matrix object in the workflow file using the output variable from the setup job. To do this, I need to convert the JSON-encoded string in the output variable to an object using fromJson.

The resulting job (which just outputs each target name):

1run-matrix:
2  needs: setup
3  runs-on: ubuntu-latest
4  strategy:
5    matrix: ${{ fromJson(needs.setup.outputs.mymatrix) }}
6  steps:
7    - run: echo ${{matrix.target}}

When you assign an object to a matrix, each property in the object becomes a new matrix variable. In this case, I’m assigning the entire object to the matrix. As a result, I can reference the target property as matrix.target. The dynamically created matrix is equivalent to the following static matrix:

1run-matrix:
2  needs: setup
3  runs-on: ubuntu-latest
4  strategy:
5    matrix:
6      target: [dev, qa, prod]
7  steps:
8    - run: echo ${{matrix.target}}

Easy, right? You’re also not limited to just defining the matrix dynamically. Many of the fields can use expressions. For example, I can dynamically define my runs-on to execute the steps on multiple operating systems using a matrix:

1  test:
2    strategy:
3      matrix:
4        platform: [windows, ubuntu, macos]
5
6    runs-on: ${{ matrix.platform }}-latest

There is one major exception to this – uses. Both reusable workflows and Actions use in steps must be static string values. This is because the process of executing the Action requires the system to extract the external workflows and Actions. It then resolves those to a specific version of the code, which are downloaded to runners before the steps execute. As a result, there’s currently no way to dynamically define the uses. The best way to work around that? Rely on scripts for the execution step. Using a run step, a script can run any required logic and dynamically call other scripts.

May the Fourth be with you!