Kirill Müller
I wanted to try out the new fromJSON()
that allows dynamic build matrices in GitHub Actions for quite some time now.
Today was the day.
GitHub Actions allows automating build and deployment processes (CI/CD), tightly integrated with GitHub.
A build matrix is a way to define very similar workflows that differ only by configuration parameters.
Usually, a build matrix is defined directly in the .yaml
files together with the workflows.
This blog post shows how to define these build matrices dynamically, so that the “source of truth” for the matrix definition is outside the .yaml
file.
The configuration for a workflow is a YAML file that has a context and expression syntax with very few basic functions.
Two very powerful functions are toJSON()
and fromJSON()
:
toJSON()
can capture pieces of the workflow configuration as JSON and pass it to your workflow codefromJSON()
allows injecting arbitrary configuration pieces created from JSON codeThe basic setup comprises of two jobs: one that creates the workflow definition as JSON and stores it as output, and another dependent job that injects this output via fromJSON()
into its matrix definition.
A third job is defined for testing if outputs are passed correctly between jobs.
The original blog post contains a somewhat terse description. This blog post shows a walkthrough how I converted a static to a dynamic build matrix in the DBItest project.
In DBItest, we test compatibility of new or updated tests with backend packages. Each backend is run in a build matrix, which is defined as follows:
jobs:
backend:
strategy:
fail-fast: false
matrix:
package:
- duckdb
- RSQLite
- RMariaDB
- RPostgres
- RKazam
The relevant backends are defined in the Makefile
, we want to get the list from there so that we can use a single source of truth.
This is a very simple build matrix, ideally suited for first experiments. The techniques shown here are applicable to build matrices of any complexity and size.
Our goal is to create the package:
section from the above matrix in JSON format.
To derive the JSON format, I use the sed
stream editor, my beloved hammer that I use whenever I see a text transformation task in the shell:
echo '{ "package" : ['
## { "package" : [
sed -n "/^REVDEP *:= */ { s///; p }" revdep-dev/Makefile | sed 's/ /, /g' | xargs -n 1 echo | sed -r 's/^([^,]*)(,?)$/"\1"\2/'
## "RMariaDB",
## "RSQLite",
## "RPostgres",
## "RKazam",
## "duckdb"
echo "]}"
## ]}
This is not pretty, but still valid JSON when put together.
We can prettify with jq .
, later we will use jq -c .
to condense to a single line.
(
echo '{ "package" : ['
sed -n "/^REVDEP *:= */ { s///; p }" revdep-dev/Makefile | sed 's/ /, /g' | xargs -n 1 echo | sed -r 's/^([^,]*)(,?)$/"\1"\2/'
echo "]}"
) | jq .
{
"package": [
"RMariaDB",
"RSQLite",
"RPostgres",
"RKazam",
"duckdb"
]
}
We verify the YAML version by piping to json2yaml
which can be installed with npm install json2yaml
:
---
package:
- "RMariaDB"
- "RSQLite"
- "RPostgres"
- "RKazam"
- "duckdb"
These tools are preinstalled on the workers. This avoids time-consuming installation procedures in this first job that needs to be run before the main jobs can even start.1
Once we have derived the JSON, we’re ready to define a job that creates the matrix.
This must be done in the same workflow file where the matrix is defined, ideally before the main job.
The job runs on ubuntu-latest
, and also must clone the repository.
In the bash
snippet, the $matrix
variable contains the JSON.
It is shown and pretty-printed before it is provided as output via echo ::set-output ...
.
jobs:
matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
- id: set-matrix
run: |
matrix=$((
echo '{ "package" : ['
sed -n "/^REVDEP *:= */ { s///; p }" revdep-dev/Makefile | sed 's/ /, /g' | xargs -n 1 echo | sed -r 's/^([^,]*)(,?)$/"\1"\2/'
echo " ]}"
) | jq -c .)
echo $matrix
echo $matrix | jq .
echo "::set-output name=matrix::$matrix"
backend:
# Original workflow
# ...
Before plugging in the generated JSON into our build job, we add another check job to verify if the generated JSON is transported correctly across job boundaries.
The needs: matrix
declares that the job must wait before the first matrix
job succeeds.
The job’s output is queried via ${{ needs.matrix.outputs.matrix }}
, the quotes ensure that bash
processes this correctly.
We install and use json2yaml
to double-check what the YAML snippet looks like.
jobs:
matrix:
# job defined above
check-matrix:
runs-on: ubuntu-latest
needs: matrix
steps:
- name: Install json2yaml
run: |
sudo npm install -g json2yaml
- name: Check matrix definition
run: |
matrix='${{ needs.matrix.outputs.matrix }}'
echo $matrix
echo $matrix | jq .
echo $matrix | json2yaml
backend:
# Original workflow
# ...
Finally, we’re ready to use the generated JSON as a build matrix.
The workflow now uses matrix: ${{fromJson(needs.matrix.outputs.matrix)}}
instead of the hard-coded matrix:
jobs:
matrix:
# see above
check-matrix:
# see above
backend:
needs: matrix
strategy:
fail-fast: false
matrix: ${{fromJson(needs.matrix.outputs.matrix)}}
# rest unchanged
This gives a workflow as shown in the image below. Click on the image to view the workflow on GitHub.
For R packages, I see two use case where dynamic matrices can be useful:
What are your use cases for dynamic build matrices? Drop us a line at mail@cynkra.com!
Even with this simple build matrix it took more time than I would have hoped to get the bits and pieces right.
Quoting is hard.
Setting up the check-matrix
job really saves time, I wish I had done this from the start.
Both fromJson()
and fromJSON()
appear to work.
The internal functions from the expression syntax seem to be case-insensitive throughout.
For older versions, jq
needs to be called as jq .
to act as pretty-printer.
For newer versions this can be omitted.
Today I also learned that workflows can be temporarily disabled. This is useful in situations where you experiment with a workflow and want to avoid running other workflows for every test.
You can use any tool or ecosystem you are familiar with to come up with the JSON definition. To avoid long installation times, use a specific image for your step via uses: docker://...
or implement a container action, also possible in the same repository. ↩
Kirill Müller
Setting up a load-balanced Jitsi Meet instancePatrick Schratz
DevOps Expert (f/m/d, 60-100%)cynkra team
Maintaining multiple identities with GitKirill Müller
Relational data models in RAngel D'az, Kirill Müller
tempdisagg: converting quarterly time series to dailyChristoph Sax
tsbox 0.2: supporting additional time series classesChristoph Sax
DevOps System Engineer (40-60%)cynkra team
Introducing dm: easy juggling of tables and relationsBalthasar Sager
tsbox 0.1: class-agnostic time seriesChristoph Sax
Data Scientist/Engineer (40-100%)cynkra team
Time series of the world, unite!Christoph Sax
Done “Establishing DBI”!?Kirill Müller