Generating a Task Matrix by Looping over Repo Files with GitHub Actions

github

I’ve been having more fun with GitHub actions after Automating Code Changes via GitHub Actions Making Pull Requests. Let’s generate a job matrix from a list of files.

Why would I need that? In opensearch-project/opensearch-build we create manifest files that are used to produce an OpenSearch distribution. These files are created manually, one for every version. Each needs to be sanity-checked when created or changed.

These checks can be executed in parallel, so we can create a GitHub Actions matrix like so.

name: manifests
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    strategy:
        matrix:
            manifest:
                - manifests/opensearch-1.1.0.yml
                - manifests/opensearch-1.0.0.yml
    steps:
      - uses: actions/checkout@v2   
      - run: |
        ./check-manifest $

We’ll definitely forget to update the matrix when a new file is created, so let’s just list those files dynamically, and generate a matrix from the list.

name: manifests
on: [push, pull_request]
jobs:
  list-manifests:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v2
      - id: set-matrix
        run: echo "::set-output name=matrix::$(ls manifests/*.yml | jq -R -s -c 'split("\n")[:-1]')"
  check:
    needs: list-manifests
    runs-on: ubuntu-latest
    strategy:
        matrix:
            manifest: ${{ fromJson(needs.list-manifests.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v2   
      - run: |
        ./check-manifest ${{ matrix.manifest }}

Here’s how this works.

1. A shell command ls manifests/*.yml lists all .yml files.
2. A pipe to | jq -R -s -c 'split("\n")[:-1]' transforms the file list into a JSON array (from StackOverflow#10234327). Note that jq is installed on all GHA Linux images.
3. The matrix output is set to the JSON array of files using set-output with echo "::set-output name=matrix::value.
4. The manifest values are loaded from the JSON array using fromJson and become part of the updated workflow.

This is so awesome that it generates the matrix during the build!

Profit

See opensearch-project/opensearch-build#386 for a working example.