GitHub Action that runs a Sengol evaluation suite and gates CI/CD on compliance obligations.
The action installs sengol, runs sengol gate against your server, and writes the results to the GitHub Actions step summary, a sticky PR comment, or a JUnit XML file. Workflow-command annotations are emitted alongside all modes so failed obligations surface inline on the PR diff. Your pipeline fails when an obligation pass rate falls below its declared threshold.
The gate verdict and the signed audit record are the same object — the PR comment is the developer view of evidence that already exists, never a second source of truth.
Your sengol.yaml does not carry a ControlBook, regulations, control evaluators, judges or pass
thresholds. The Sengol server owns them for each agent. On every run the gate reads the agent's
risk tier and labels from the server, resolves which controls apply, and runs the evaluators
those controls name with the settings the ControlBook gives them. Your sengol.yaml may list
extra quality evaluators; a suite.controlbook key makes the gate exit 2.
If the server cannot be reached, the step fails. Nothing is cached and there is no fallback, so a build never passes against a ControlBook it did not read. The server is expected to be highly available.
You do not copy or paste a ControlBook signing key. The gate verifies the ControlBook against the install's public keys, fetched over verified TLS from your server on every run.
Minimal — runs the suite, fails the workflow when the gate fails, writes a markdown summary and posts a sticky PR comment:
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
config: sengol.yaml
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Required permission: The action posts a PR comment via
GITHUB_TOKEN. Addpull-requests: writeto your workflow's permissions block:permissions: pull-requests: write contents: readFork PRs receive a read-only token automatically — the gate degrades gracefully (::notice + step summary) and never fails because of the comment.
JUnit XML output (for surfacing failures in test-report integrations):
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
output-format: junit
junit-output-path: eval-results.xml
env:Flake tolerance — retry LLM evaluators 3 times, require 2/3 to pass:
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
repeat: '3'
min-pass: '2'
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Push results to a downstream exporter (e.g. MLflow):
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
export: mlflow
env:
MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
MLFLOW_TRACKING_TOKEN: ${{ secrets.MLFLOW_TRACKING_TOKEN }}Run a regression suite against a target agent after the gate passes:
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
regression-runner: agent.main:handle
regression-agent: acme-support-agent
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Pin sengol-version and enable pip caching via actions/setup-python to avoid re-downloading on every run:
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
sengol-version: '2.0.0' # pin to a specific release
config: sengol.yamlDocker prebuilt-image variant (no install step, sub-second startup) is Phase 7 backlog.
| Input | Required | Default | Description |
|---|---|---|---|
api-url |
yes (or SENGOL_API_URL) |
(none) | Base URL of your Sengol server. The server owns the ControlBook and control evaluators for the agent; the step fails if it cannot be reached. |
config |
no | sengol.yaml |
Path to your sengol.yaml. |
python-version |
no | 3.11 |
Python version installed before pip install sengol. |
sengol-version |
no | >=2.0,<3 |
Pin a specific release or range (e.g. 2.0.0, >=2.0,<3). |
extras |
no | anthropic-judge,openai-judge |
Comma-separated sengol extras to install. The default ships both LLM-judge provider SDKs so judges wire up out of the box. Add model for statistical evaluators (PSI/AUC/Fairness); set to "" for a bare install (deterministic evaluators only). |
fail-on-gate-failure |
no | true |
Exit non-zero when the gate fails. |
output-format |
no | github-summary |
One of github-summary, junit. |
junit-output-path |
no | eval-results.xml |
Path for JUnit XML when output-format=junit. |
export |
no | (none) | Comma-separated result-exporter names (e.g. mlflow). Options come from the sengol.yaml exporters: block. Exporters are best-effort sinks — a failure never fails the gate. |
pr-comment |
no | true |
Post/update a sticky PR comment with gate results on pull_request events. Requires pull-requests: write. Degrades gracefully on fork PRs. |
repeat |
no | 1 |
Run LLM evaluators N times per input for flake tolerance. Deterministic evaluators always run once. All attempt records are written to evidence. |
min-pass |
no | 0 |
Require M of N (repeat) LLM-evaluator attempts to pass. 0 means all must pass (equivalent to repeat value). |
dataset-path |
no | '' |
Path to a JSONL trace dataset (sengol gate --dataset-path). Empty uses the dataset from sengol.yaml. |
dataset-id |
no | '' |
Dataset identifier recorded on the evaluation run. Empty derives it from config. |
regression-runner |
no | '' |
module:function passed to sengol regression run --runner (e.g. agent.main:handle). Empty skips the regression step. Requires a sengol version with sengol regression run. |
regression-agent |
no | '' |
Agent id passed to sengol regression run --agent. Required when regression-runner is set. |
| Output | Description |
|---|---|
passed |
"true" / "false" — gate verdict for downstream steps. Always set. |
score |
GoldScore (0.0–1.0) for the evaluation period. Empty string ('') when GoldScore is not configured. |
report_url |
Path or URL to the generated PDF evidence pack. Empty string ('') when PDF generation is not enabled. |
no_agent_activity |
"true" when at least one obligation found no agent activity in the evaluation period. Empty string ('') otherwise. |
regression |
"true" when current pass-rate regressed vs the configured suite.baseline_pass_rate in sengol.yaml. "false" when at or above baseline. Empty string ('') when no baseline is configured. |
GitHub Actions composite action note: All declared outputs are resolved by the runner even when the underlying step does not emit a value, producing an empty string rather than a truly absent output. Check with
!= ''rather than testing for absence:- if: steps.sengol.outputs.regression == 'true' run: echo "Pass-rate regressed vs baseline" - if: steps.sengol.outputs.no_agent_activity != '' run: echo "Agent was offline during evaluation period"
The SDK's set_outputs() never writes an empty string to $GITHUB_OUTPUT — the empty string consumers see is the GitHub Actions runner's own default for unmapped outputs.
Add baseline_pass_rate to your sengol.yaml to enable regression detection:
suite:
name: acme-compliance
evaluators:
- FaithfulnessEvaluator
baseline_pass_rate: 0.95 # alert when pass-rate drops below thisThe regression output is then "true" / "false" on every run, and the pass-rate delta appears in the step summary and PR comment.
api-urlinput (orSENGOL_API_URL) — your Sengol server. The action fails first, before installing anything, when neither is set. A repository or organization variable works well.- Authentication — GitHub OIDC (below).
SENGOL_API_TOKENremains the fallback. ANTHROPIC_API_KEY— only when your suite includes LLM-judge evaluators.
The server signs and stores the evidence, so the action needs no signing key, audit store URI or tenant id.
This removes the long-lived SENGOL_API_TOKEN from CI. LLM-judge suites still need
ANTHROPIC_API_KEY.
Before (every workflow): a long-lived SENGOL_API_TOKEN stored as a repo/org secret,
shared across every run, with no per-run identity.
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
config: sengol.yaml
env:
SENGOL_API_TOKEN: ${{ secrets.SENGOL_API_TOKEN }} # stored secret, never rotates itselfAfter (Phase 7c — ADR-0077): GitHub already issues a short-lived OIDC token to every
workflow run. The action exchanges it for a Sengol JWT — no long-lived Sengol auth token
stored in CI. Add id-token: write and drop SENGOL_API_TOKEN. The example below is a
complete, runnable job (permissions: is a job-level key — it cannot sit next to a
- uses: step):
jobs:
governance:
runs-on: ubuntu-latest
permissions:
id-token: write # required for the OIDC exchange
pull-requests: write # required for the sticky PR comment
contents: read
steps:
- uses: actions/checkout@v4
- uses: sengol-io/sengol-github-action@2.0.0
with:
api-url: ${{ vars.SENGOL_API_URL }}
config: sengol.yaml
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Audience. The exchange uses an OIDC audience that must match on both sides. The default
is sengol. If your Sengol server is configured with a custom SENGOL_GITHUB_OIDC_AUDIENCE,
the workflow must request a GitHub token with the same audience by setting
SENGOL_GITHUB_OIDC_AUDIENCE as a step env var — GitHub only changes the token's aud
claim when the action requests it explicitly. The action.yml exposes no audience input;
it is configured purely through this env var:
env:
SENGOL_GITHUB_OIDC_AUDIENCE: my-org-audience # only if the server uses a non-default audience
# ...plus the secrets aboveIf the server still has the default SENGOL_GITHUB_OIDC_AUDIENCE unset, the exchange
returns 503 and the action falls back to SENGOL_API_TOKEN (see below).
Fallback. The action detects OIDC availability (ACTIONS_ID_TOKEN_REQUEST_URL) at
runtime: if id-token: write isn't granted, or the server doesn't accept the exchange, it
falls back to SENGOL_API_TOKEN automatically. For that fallback to work, the
SENGOL_API_TOKEN secret must remain set in the workflow env — only remove it once you
have confirmed the OIDC exchange succeeds. Every exchange is logged server-side with the
repository, ref, commit, and workflow run ID (github_workflow_audit_log — see
docs/reference/console.md in the sengol repo).
Apache 2.0. See LICENSE.