Dynamic matrix in Github action

VS-404
8 min readApr 29, 2022

--

Github actions

A matrix strategy lets you use variables in a single job definition to automatically create multiple job runs that are based the combinations of the variables. For example, you can use a matrix strategy to test your code in multiple versions of a language or on multiple operating systems.

I was experimenting with workflows and discovered that I need to use matrix strategies but with dynamic inputs. At this point I believe you already know basic of Github workflow and matrix strategies and how it works. That is why you are reading this blog to implement dynamic matrix.

Let’s look at the requirements for workflow before implementation.

  1. Workflow should be able to take deployment targets as input (comma separated values)
  2. Workflow can be triggered with different branches or commit hash (for rollback strategy).
  3. Workflow should be cache dependencies to improve build time.

Finally we have all the requirements. Now we need to implement basic workflow which will take input from user and should trigger multiple jobs with input at same time. Simple right?

Then lets get started

We will start by creating repository first. After you create repository or go to existing repository navigate to Actions and click on New workflow. Search fir node and you should see something like this. Click on configure on first workflow.

Create node workflow
Create new workflow for node js

Once you click on Configure you should see below screen where you can name your file and will have some sample configuration in YAML file. Once you see below file you can commit and we will edit it according to our requirements

Create new workflow for node js
Stringified input workflow sample file

And from here it gets interesting, for martrix to work Github requires input as array so that Github can start jobs in combinations with different inputs. Let me elaborate look at following sample config.

JOB_NAME:
needs: prebuild
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
OS: [ubuntu-latest, windows-latest]
NODE_VERSION: [10,16]

With above config we have specified 2 inputs to matrix one is OS and second is NODE_VERSION and as we can see both are specified in array. With this configurations Github will trigger multiple jobs with below combinations. And JOB_NAME job is triggered. Then we can specify required steps to execute.

  1. {OS: ubuntu-latest, NODE_VERSION: 10}
  2. {OS: ubuntu-latest, NODE_VERSION: 16}
  3. {OS: windows-latest, NODE_VERSION: 10}
  4. {OS: windows-latest, NODE_VERSION: 16}

Above example is defined input directly in workflow yaml file. Lets see how we can make it dynamic. We will take a look at multiple approaches to get it working dynamically.

Approach 1: Use input to workflow as stringified JSON

Approach 2: Use input with comma seperated and with quotation mark

Approach 1: Use normal input without any formatting and add custom action to transform input

Let’s discuss these approaches in details

Approach 1:

Add below changes to workflow for adding input for stringified values

name: Stringified input workflow
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "['ubuntu-latest','windows-latest']"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "['10','16']"
jobs:
build:
runs-on: ubuntu-latest
name: '${{ matrix.OS }} ${{ matrix.NODE_VERSION }}'
strategy:
fail-fast: false
matrix:
OS: ${{ fromJSON(github.event.inputs.OS) }}
NODE_VERSION: ${{ fromJSON(github.event.inputs.NODE_VERSION) }}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"

This above workflow input and run would like this:

Workflow
Stringified workflow input
Workflow run
Workflow run

If you see we have input as array and which will trigger multiple jobs in combinations. With this approach, any one with programming language / JSON knowledge will be comfortable but still this is error prone if input is not specified properly.

Approach 2:

Add below changes to workflow for adding input for matrix values in single quotation mark.

name: Input with quotation mark
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "'ubuntu-latest','windows-latest'"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "'10','16'"
jobs:
prebuild:
runs-on: ubuntu-latest
outputs:
os: ${{ steps.set-matrix.outputs.os }}
nodeVersion: ${{ steps.set-matrix.outputs.nodeVersion }}
steps:
- id: set-matrix
run: |
echo "::set-output name=os::${{format('[{0}]',github.event.inputs.OS)}}"
echo "::set-output name=nodeVersion::${{format('[{0}]',github.event.inputs.NODE_VERSION)}}"
build:
needs: prebuild
runs-on: ubuntu-latest
name: "${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
strategy:
fail-fast: false
matrix:
OS: ${{fromJSON(needs.prebuild.outputs.os)}}
NODE_VERSION: ${{fromJSON(needs.prebuild.outputs.nodeVersion)}}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"

This is almost same implementation as we discussed in approach 1 but without square brackets. Take a look at this snippet from above YAML

steps:
- id: set-matrix
run: |
echo "::set-output name=os::${{format('[{0}]',github.event.inputs.OS)}}"
echo "::set-output name=nodeVersion::${{format('[{0}]',github.event.inputs.NODE_VERSION)}}"

This is what will add [] to the inputs for matrix. Here we have used one of the expression available for us to use in Github actions. That is format , if you want to read more on this Github expressions#format.

Input with quotation
Input with quotation
Workflow run

Approach 3:

If I say this is more dynamic approach of all 3 it would not be wrong. So with this approach we will add our own custom Github action in same repo and convert input from string to stringified version of array for matrix input. Lets start I am using node js you should be able to do this with any supported language, basic principle will remain same.

  1. We will write custom github action which will take input from our main workflow for transformation.
  2. As we know matrix need input in array format, we will convert provided input in stringified array format and spits it as output.
  3. From second step (transformation step) output we will feed it to matrix.

This is how we will achieve fully dynamic worflow with matrix.

You can follow this official document for creating action.
Lets get started, first create 2 files action.yamland index.js in ./github/actions/transform directory.
1. action.yaml file will contain specification for our action like how input should be what will be the actions output.
2. index.js file is for actual implementation for transformation.

  • copy and paste below yaml config in your action.yaml file
  • This takes inputs as os and nodeversion and provide output of transformed value with same name
# This is custom action, for transforming input to stringified output
name: Workflow input transformer
description: This custom action will transform input to stringified output
inputs:
os:
description: provide which operating systems to use
required: true
default: 'ubuntu-latest,windows-latest'
nodeversion:
description: provide which node version systems to use
required: true
default: '10,16'
outputs:
os:
description: This is transformed output in array of string from provided os
nodeversion:
description: This is transformed output in array of string from provided linux versions
runs:
using: node16
main: index.js

Now lets jump into coding this custom action.

  • Before we start we will need @actions/core package, to work and access above mentioned inputs. (Note: you can do this without package by reading it from env variables but we stick with package).
    Run npm i @actions/core for installing this package.
  • Copy and paste below code in index.js file. And we will go through how it works.
const core = require("@actions/core");try {
// `os` input defined in action metadata file with default value "ubuntu-latest,windows-latest"
const os = core.getInput("os", { require: true });
// `nodeversion` input defined in action metadata file with default value "10,16"
const nodeVersion = core.getInput("nodeversion", { required: true });
core.info(
`Inputs to transform ${JSON.stringify({
os,
nodeVersion
})}`
);
// create arrays by splitting input on ','
const osArray = os.split(",");
const nodeVersionArray = nodeVersion.split(",");
// Reduce array to string with single quotation mark
const osInputStringified = osArray.map(_os => `'${_os}'`).toString();
const nodeVersionInputStringified = nodeVersionArray.map(_nodeVersion => `'${_nodeVersion}'`).toString();
core.setOutput("os", `[${osInputStringified}]`);
core.setOutput("nodeversion", `[${nodeVersionInputStringified}]`);
} catch (error) {
core.setFailed(
`Action failed to transform input with error: ${error.message}`
);
}
  1. Get input provided from main workflow with help of @actions/core package. You can core.getInput(“os”, { require: true }) . We have mentioned required as true in second parameter of getInput which will exit process with error code 1 if input is not provided.
  2. we will split input to create it as array based on , separator.
  3. Then we create stringified version of input by mapping each entry and transforming with help of template literals.
const osInputStringified = osArray.map(_os => `'${os}'`).toString();
const nodeVersionInputStringified = nodeVersionArray.map(_nodeVersion => `'${_nodeVersion}'`).toString();

4. Lastly we set it as output of action wrapped in [] square brackets to represent it as array. core.setOutput(“os”, `[${osInputStringified}]`);

5. Now only thing remaining is to setup workflow and call this action in workflow to transform our inputs. Take a look at our final dynamic workflow file. Take a look at below configurations.

name: Dynamic input matrix workflow
on:
workflow_dispatch:
inputs:
OS:
description: provide all OS in stringified array format
required: true
default: "ubuntu-latest,windows-latest"
NODE_VERSION:
description: provide all node versions in stringified array format
required: true
default: "10,16"
jobs:
prebuild:
runs-on: ubuntu-latest
outputs:
os: ${{ steps.set-matrix.outputs.os }}
nodeVersion: ${{ steps.set-matrix.outputs.nodeVersion }}
steps:
- uses: actions/checkout@v2
with:
ref: main
- name: Use Node.js (16)
uses: actions/setup-node@v1
with:
node-version: 16
registry-url: "https://registry.npmjs.org"
- name: Install package
run: npm install
- name: transform input
id: transformInput
uses: ./.github/actions/transformer/
with:
os: ${{ github.event.inputs.OS }}
nodeversion: ${{ github.event.inputs.NODE_VERSION}}
- id: set-matrix
run: |
echo "::set-output name=os::${{ steps.transformInput.outputs.os }}"
echo "::set-output name=nodeVersion::${{ steps.transformInput.outputs.nodeversion }}"
build:
needs: prebuild
runs-on: ubuntu-latest
name: "${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"
strategy:
fail-fast: false
matrix:
OS: ${{fromJSON( needs.prebuild.outputs.os )}}
NODE_VERSION: ${{fromJSON( needs.prebuild.outputs.nodeVersion )}}
steps:
- name: running
run: echo "running job with ${{ matrix.OS }} ${{ matrix.NODE_VERSION }}"

This is how workflow will look like

Dynamic input workflow
Dynamic input workflow
Dynamic input workflow run
Dynamic input workflow run

Source code available here.

Conclusion: you can write fully dynamic Github workflow with matrix. And for this you would need custom action to transform input. Now it is up to requirement and imagination of developer to add inputs to workflows. 🙂

Note: This is very basic example here you can see weird behaviors if you add spaces in between inputs, random characters. In real word you should take care of all these scenarios for filtering and transforming your input.

Pro tip:
You can reduce custom action flow to very few lines like combining all operations on input. This is not a good practice and also not good from readability perspective but it is fun like recursive functions (hard to understand). 😅

core.setOutput("os", `[${(core.getInput("os")||"").split(',').map(_os=>`'${_os}'`).toString()}]`);

--

--

VS-404
VS-404

Responses (1)