Commenting on Pull Requests in GitHub Actions

The OpenSearch API specification is authored in OpenAPI and used to auto-generate OpenSearch language clients. I wanted to know how much of the API was described in it vs. the actual API implemented in the default distribution of OpenSearch that includes all plugins. To do so, I have exposed an iterator over REST handlers in OpenSearch core, and wrote a plugin that rendered a very minimal OpenAPI spec at runtime. All that was left was to compare the manually authored OpenAPI spec in opensearch-api-specification to the runtime one, added in opensearch-api-specification#179. The comparison workflow output a total and relative number of APIs described.

To surface this information in pull requests I wanted to add a comment in the API coverage workflow. This required a PAT token, so I initially authored that workflow with pull_request_target in opensearch-api-specification#196.

- name: Gather Coverage
  id: coverage
  shell: bash
  run: |
    ...
    cat >>"$GITHUB_OUTPUT" <<EOL
    current=$current
    total=$total
    percent=$percent
    EOL
- uses: peter-evans/create-or-update-comment@v4
  if: github.event_name == 'pull_request_target'
  with:
    issue-number: $
    body: |
        API specs implemented for $/$ ($%) APIs.

This is actually insecure because we run a JavaScript tool to generate the API spec with source code coming from the pull request, and a user can execute arbitrary code this way and gain access to the secure token. One solution is to run the tool from main, but we can do better.

In opensearch-api-specification#251 I split the coverage workflow in one that gathered information into a JSON file and uploaded it as an artifact of the pull request workflow, and another that downloaded the artifact and commented on the pull request.

name: Gather API Coverage
on: [push, pull_request]

- name: Gather Coverage
  id: coverage
  shell: bash
  run: |
    ...
    cat >>"coverage.json" <<EOL
    {
      "pull_request":$,
      "current":$current,
      "total":$total,
      "percent":$percent
    }
    EOL
- uses: actions/upload-artifact@v4
  with:
    name: coverage
    path: coverage.json

name: Comment with API Coverage

on:
  workflow_run:
    workflows: ["Gather API Coverage"]
    types:
      - completed

jobs:
  comment:
    runs-on: ubuntu-latest
    if: >
      github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success'
    steps:
      - name: Download Coverage Report
        uses: actions/download-artifact@v4
        with:
          github-token: $
          name: coverage
          run-id: $

      - name: 'Comment on PR'
        uses: actions/github-script@v3
        with:
          github-token: $
          script: |
            const fs = require('fs');
            var data = JSON.parse(fs.readFileSync('./coverage.json'));
            console.log(data);
            await github.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: data.pull_request,
              body: `API specs implemented for ${data.current}/${data.total} (${data.percent}%) APIs.`
            });

This is a very convenient pattern of passing structured data from a pull request workflow to one that has access to write data in a GitHub repository. Check out the latest versions of the coverage workflows if you are going to use them.

Daniel Doubrovkine

Commenting on Pull Requests in GitHub Actions opensearch

Commenting on Pull Requests in GitHub Actions