The slack-ruby-client library, an open source Ruby gem I maintain, runs a scheduled GitHub Actions workflow that regenerates code from Slack’s API definitions and opens a pull request with the diff. The commit message and CHANGELOG entry used to be a generic “Update API (2026-08-11)”, which told a reviewer nothing about what actually changed. Here’s how we taught the workflow to describe its own diffs, using GitHub Copilot CLI, which open source maintainers can get for free.
The Idea
The workflow already computes a diff before opening the pull request. Instead of a boilerplate commit message, we pipe that diff through an LLM and ask it to summarize what changed, then use the response as the commit message and PR body.
- name: Check for changes
id: changes
run: |
if git diff --quiet; then
echo "changed=false" >> "$GITHUB_OUTPUT"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
fi
- name: Prepare diff for AI summary
if: steps.changes.outputs.changed == 'true'
run: |
git diff --stat | sed 's/^/ /' > /tmp/diff_stat.txt
git diff | head -c 20000 | sed 's/^/ /' > /tmp/diff.txt
The sed 's/^/ /' indent isn’t decorative. actions/ai-inference substitutes template variables as raw text into a prompt YAML file before parsing it, so a multi-line diff starting at column 0 breaks the indentation of the enclosing content: |- block scalar. Pre-indenting the file to match keeps the YAML valid no matter what the diff looks like.
The Prompt File
actions/ai-inference supports .prompt.yml files, a small convention for keeping the system/user prompt out of the workflow YAML.
messages:
- role: system
content: |-
You write CHANGELOG entries describing an automated API update to slack-ruby-client, a
Ruby gem whose Web API endpoint methods, argument validations, specs, and bin commands
are code-generated from vendored Slack API method definitions (via a git submodule and
rake task).
Given a diffstat and a diff of the regenerated files, respond with ONLY a single JSON
object (no markdown code fences, no other text) in exactly this shape:
{"entries": [string, ...]}
Rules for entries:
- Each entry is a single short line, imperative mood, no leading bullet/dash and no
trailing period (the caller adds both).
- Group related changes together into one entry per notable change or affected method
group, rather than one entry per file.
- Focus on developer-visible API surface changes: new/removed/renamed methods, new/
removed arguments, validation changes.
- role: user
content: |-
Asking for a single line per group of changes, rather than per file, matters, a single Slack API update can touch dozens of generated files for one conceptual change (adding an argument to five related methods, say), and nobody wants five identical CHANGELOG lines about it.
Wiring It into the Workflow
- name: Generate changelog entries with AI
if: steps.changes.outputs.changed == 'true'
id: ai
uses: actions/ai-inference@v1
with:
prompt-file: ./.github/prompts/changelog-entries.prompt.yml
file_input: |
diff_stat: /tmp/diff_stat.txt
diff: /tmp/diff.txt
- name: Build changelog entries
id: entries
run: |
entries=""
if [ "$" = "true" ]; then
entries="$(grep -v '^```' "$" 2>/dev/null | jq -r '.entries[]? // empty' 2>/dev/null)"
fi
if [ -z "$entries" ]; then
entries="Update API ($)"
fi
Two defensive touches worth calling out. First, the workflow strips markdown code fences the model sometimes adds despite being told not to, before jq tries to parse the JSON. Second, if parsing fails or produces nothing, the workflow falls back to the original generic message rather than failing the whole run.
Once we have the entries, we use them both as the commit message/PR body and to insert one CHANGELOG line per group, all referencing the same (predictable, since GitHub allocates them sequentially) PR number:
lines = entries.map do |entry|
"* [##{pr_number}](#{pr_url}): #{entry} - [@slack-ruby-ci-bot](https://github.com/apps/slack-ruby-ci-bot)."
end.join("\n")
You Can’t Use GitHub Models
If you’re setting this up today, skip GitHub Models as the backend, it was retired on July 30, 2026 and actions/ai-inference no longer talks to it at any version. The v1 tag, in particular, doesn’t float to the rewrite, it’s stuck resolving to the old GitHub-Models-only code, so don’t expect a version bump to fix things automatically. The current major version, v3, speaks exclusively to GitHub Copilot CLI instead, which changes a few things about the setup:
- name: Set up Node
uses: actions/setup-node@v6
- name: Install Copilot CLI
run: npm install -g @github/copilot
- name: Generate changelog entries with AI
if: steps.changes.outputs.changed == 'true'
id: ai
uses: actions/ai-inference@v3
with:
prompt-file: ./.github/prompts/changelog-entries.prompt.yml
model: ''
file_input: |
diff_stat: /tmp/diff_stat.txt
diff: /tmp/diff.txt
env:
COPILOT_GITHUB_TOKEN: $
Copilot CLI has to be installed on the runner (the action doesn’t bundle it any more), authentication is via COPILOT_GITHUB_TOKEN rather than the old token input, and responseFormat/jsonSchema inputs are gone entirely, hence asking for bare JSON in the prompt and parsing it defensively, as above. We also set model: '' explicitly; the action’s built-in default (gpt-4.1) isn’t necessarily available on every Copilot plan, and an empty string tells the CLI to fall back to whatever model it has access to.
For the token, if your org has a paid Copilot Business or Enterprise plan, you can enable “Allow use of Copilot CLI billed to the organization” and grant the workflow the copilot-requests: write permission, then the built-in GITHUB_TOKEN just works, no secret needed. Free organizations (like slack-ruby) don’t have that option, since free Copilot access for open source maintainers is granted to individual maintainer accounts, not organizations, so the fallback is a personal fine-grained PAT with the “Copilot Requests” account permission, stored as a COPILOT_PAT secret.
Testing Before Merging
Both workflows have a github.repository == 'slack-ruby/...' guard so they don’t run on forks by accident, which also means we can’t just dispatch them there to test. We got around this by pushing a disposable branch to our own fork, temporarily overriding the guard (if: true), adding a throwaway step to force a non-empty diff (the real scraping step usually finds nothing new on a given day), and dispatching manually.
gh workflow enable "Update API" --repo dblock/slack-ruby-client
gh workflow run "Update API" --repo dblock/slack-ruby-client --ref test-copilot-cli
gh run watch <run-id> --repo dblock/slack-ruby-client --exit-status
This is how I caught both the missing-token and the unavailable-model issues above, neither of which was obvious from reading the action’s README.