A matrix strategy lets you use variables in a single job definition to automatically create multiple job runs that are based the combinations of the variables. For example, you can use a matrix strategy to test your code in multiple versions of a language or on multiple operating systems.
I was experimenting with workflows and discovered that I need to use matrix strategies but with dynamic inputs. At this point I believe you already know basic of Github workflow and matrix strategies and how it works. That is why you are reading this blog to implement dynamic matrix.
Let’s look at the requirements for workflow before implementation.
- Workflow should be able to take deployment targets as input (comma separated values)
- Workflow can be triggered with different branches or commit hash (for rollback strategy).
- Workflow should be cache dependencies to improve build time.
Finally we have all the requirements. Now we need to implement basic workflow which will take input from user and should trigger multiple jobs with input at same time. Simple right?
Then lets get started
We will start by creating repository first. After you create repository or go to existing repository navigate to Actions and click on New workflow. Search fir node and you should see something like this. Click on configure on first workflow.
Once you click on Configure you should see below screen where you can name your file and will have some sample configuration in YAML file. Once you see below file you can commit and we will edit it according to our requirements
And from here it gets interesting, for martrix to work Github requires input as array so that Github can start jobs in combinations with different inputs. Let me elaborate look at following sample config.
JOB_NAME:
needs: prebuild
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
OS: [ubuntu-latest, windows-latest]
NODE_VERSION: [10,16]With above config we have specified 2 inputs to matrix one is OS and second is NODE_VERSION and as we can see both are specified in array. With this configurations Github will trigger multiple jobs with below combinations. And JOB_NAME job is triggered. Then we can specify required steps to execute.
{OS: ubuntu-latest, NODE_VERSION: 10}{OS: ubuntu-latest, NODE_VERSION: 16}{OS: windows-latest, NODE_VERSION: 10}{OS: windows-latest, NODE_VERSION: 16}
Above example is defined input directly in workflow yaml file. Lets see how we can make it dynamic. We will take a look at multiple approaches to get it working dynamically.
Approach 1: Use input to workflow as stringified JSON
Approach 2: Use input with comma seperated and with quotation mark
Approach 1: Use normal input without any formatting and add custom action to transform input
Let’s discuss these approaches in details
Approach 1:
Add below changes to workflow for adding input for stringified values
name: Stringified input workflow
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "['ubuntu-latest','windows-latest']"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "['10','16']"
jobs:
build:
runs-on: ubuntu-latest
name: '${{ matrix.OS }} ${{ matrix.NODE_VERSION }}'
strategy:
fail-fast: false
matrix:
OS: ${{ fromJSON(github.event.inputs.OS) }}
NODE_VERSION: ${{ fromJSON(github.event.inputs.NODE_VERSION) }}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"This above workflow input and run would like this:
If you see we have input as array and which will trigger multiple jobs in combinations. With this approach, any one with programming language / JSON knowledge will be comfortable but still this is error prone if input is not specified properly.
Approach 2:
Add below changes to workflow for adding input for matrix values in single quotation mark.
name: Input with quotation mark
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "'ubuntu-latest','windows-latest'"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "'10','16'"jobs:
prebuild:
runs-on: ubuntu-latest
outputs:
os: ${{ steps.set-matrix.outputs.os }}
nodeVersion: ${{ steps.set-matrix.outputs.nodeVersion }}
steps:
- id: set-matrix
run: |
echo "::set-output name=os::${{format('[{0}]',github.event.inputs.OS)}}"
echo "::set-output name=nodeVersion::${{format('[{0}]',github.event.inputs.NODE_VERSION)}}"
build:
needs: prebuild
runs-on: ubuntu-latest
name: "${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
strategy:
fail-fast: false
matrix:
OS: ${{fromJSON(needs.prebuild.outputs.os)}}
NODE_VERSION: ${{fromJSON(needs.prebuild.outputs.nodeVersion)}}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
This is almost same implementation as we discussed in approach 1 but without square brackets. Take a look at this snippet from above YAML
steps:
- id: set-matrix
run: |
echo "::set-output name=os::${{format('[{0}]',github.event.inputs.OS)}}"
echo "::set-output name=nodeVersion::${{format('[{0}]',github.event.inputs.NODE_VERSION)}}"This is what will add [] to the inputs for matrix. Here we have used one of the expression available for us to use in Github actions. That is format , if you want to read more on this Github expressions#format.
Approach 3:
If I say this is more dynamic approach of all 3 it would not be wrong. So with this approach we will add our own custom Github action in same repo and convert input from string to stringified version of array for matrix input. Lets start I am using node js you should be able to do this with any supported language, basic principle will remain same.
- We will write custom github action which will take input from our main workflow for transformation.
- As we know matrix need input in array format, we will convert provided input in stringified array format and spits it as output.
- From second step (transformation step) output we will feed it to matrix.
This is how we will achieve fully dynamic worflow with matrix.
You can follow this official document for creating action.
Lets get started, first create 2 files action.yamland index.js in ./github/actions/transform directory.
1. action.yaml file will contain specification for our action like how input should be what will be the actions output.
2. index.js file is for actual implementation for transformation.
- copy and paste below yaml config in your
action.yamlfile - This takes inputs as
osandnodeversionand provide output of transformed value with same name
# This is custom action, for transforming input to stringified output
name: Workflow input transformer
description: This custom action will transform input to stringified output
inputs:
os:
description: provide which operating systems to use
required: true
default: 'ubuntu-latest,windows-latest'
nodeversion:
description: provide which node version systems to use
required: true
default: '10,16'
outputs:
os:
description: This is transformed output in array of string from provided os
nodeversion:
description: This is transformed output in array of string from provided linux versions
runs:
using: node16
main: index.jsNow lets jump into coding this custom action.
- Before we start we will need
@actions/corepackage, to work and access above mentioned inputs. (Note: you can do this without package by reading it from env variables but we stick with package).
Runnpm i @actions/corefor installing this package. - Copy and paste below code in index.js file. And we will go through how it works.
const core = require("@actions/core");try {
// `os` input defined in action metadata file with default value "ubuntu-latest,windows-latest"
const os = core.getInput("os", { require: true });// `nodeversion` input defined in action metadata file with default value "10,16"
const nodeVersion = core.getInput("nodeversion", { required: true });core.info(
`Inputs to transform ${JSON.stringify({
os,
nodeVersion
})}`
);// create arrays by splitting input on ','
const osArray = os.split(",");
const nodeVersionArray = nodeVersion.split(",");// Reduce array to string with single quotation mark
const osInputStringified = osArray.map(_os => `'${_os}'`).toString();
const nodeVersionInputStringified = nodeVersionArray.map(_nodeVersion => `'${_nodeVersion}'`).toString(); core.setOutput("os", `[${osInputStringified}]`);
core.setOutput("nodeversion", `[${nodeVersionInputStringified}]`);
} catch (error) {
core.setFailed(
`Action failed to transform input with error: ${error.message}`
);
}
- Get input provided from main workflow with help of
@actions/corepackage. You cancore.getInput(“os”, { require: true }). We have mentioned required as true in second parameter ofgetInputwhich will exit process with error code 1 if input is not provided. - we will split input to create it as array based on
,separator. - Then we create stringified version of input by mapping each entry and transforming with help of template literals.
const osInputStringified = osArray.map(_os => `'${os}'`).toString();
const nodeVersionInputStringified = nodeVersionArray.map(_nodeVersion => `'${_nodeVersion}'`).toString();4. Lastly we set it as output of action wrapped in [] square brackets to represent it as array. core.setOutput(“os”, `[${osInputStringified}]`);
5. Now only thing remaining is to setup workflow and call this action in workflow to transform our inputs. Take a look at our final dynamic workflow file. Take a look at below configurations.
name: Dynamic input matrix workflow
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "ubuntu-latest,windows-latest"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "10,16"jobs:
prebuild:
runs-on: ubuntu-latest
outputs:
os: ${{ steps.set-matrix.outputs.os }}
nodeVersion: ${{ steps.set-matrix.outputs.nodeVersion }}
steps:
- uses: actions/checkout@v2
with:
ref: main
- name: Use Node.js (16)
uses: actions/setup-node@v1
with:
node-version: 16
registry-url: "https://registry.npmjs.org"
- name: Install package
run: npm install
- name: transform input
id: transformInput
uses: ./.github/actions/transformer/
with:
os: ${{ github.event.inputs.OS }}
nodeversion: ${{ github.event.inputs.NODE_VERSION}}
- id: set-matrix
run: |
echo "::set-output name=os::${{ steps.transformInput.outputs.os }}"
echo "::set-output name=nodeVersion::${{ steps.transformInput.outputs.nodeversion }}"
build:
needs: prebuild
runs-on: ubuntu-latest
name: "${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
strategy:
fail-fast: false
matrix:
OS: ${{fromJSON( needs.prebuild.outputs.os )}}
NODE_VERSION: ${{fromJSON( needs.prebuild.outputs.nodeVersion )}}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
This is how workflow will look like
Source code available here.
Conclusion: you can write fully dynamic Github workflow with matrix. And for this you would need custom action to transform input. Now it is up to requirement and imagination of developer to add inputs to workflows. 🙂
Note: This is very basic example here you can see weird behaviors if you add spaces in between inputs, random characters. In real word you should take care of all these scenarios for filtering and transforming your input.
Pro tip:
You can reduce custom action flow to very few lines like combining all operations on input. This is not a good practice and also not good from readability perspective but it is fun like recursive functions (hard to understand). 😅
core.setOutput("os", `[${(core.getInput("os")||"").split(',').map(_os=>`'${_os}'`).toString()}]`);