Skip to content

[AAASM-6204] 🐛 (examples): Declare the dry-run posture the offline Python demos depend on - #631

Merged
Chisanan232 merged 4 commits into
mainfrom
v0.0.1/AAASM-6204/fix/python_examples_offline_posture
Sep 28, 2026
Merged

Chisanan232 merged 4 commits into
mainfrom
v0.0.1/AAASM-6204/fix/python_examples_offline_posture

Conversation

@Chisanan232

@Chisanan232 Chisanan232 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Two things, in the order they have to happen.

  1. metadata/sdk-versions.yaml advances the python SDK pin from 0.0.1rc6 to 0.0.1rc7 — the current published release. The generator rewrites the 40 files that derive from it (18 manifests, 20 bounded README blocks, the live scenario's Dockerfile and verify-live.yml), and all 18 lockfiles are relocked with a single-package upgrade.
  2. All 16 Python examples declare the dry-run posture they actually depend on, in src/main.py and in tests/test_smoke.py.

Step 2 is not optional cleanup. Without it the pin advance turns 18 red PRs into 16 red example lanes.

Why

Every Python example calls init_assembly in sdk-only mode with no enforcement_mode, so it inherits the advertised default. On rc.6 that default silently failed open when registration could not reach a gateway, which is the only reason the offline demos ran at all.

rc.7 closed that hole (AAASM-4130). _local_posture_is_enforce now treats a None posture exactly like enforce for the SDK's own error posture, and only the explicit dry-run postures fail open locally:

def _local_posture_is_enforce(enforcement_mode: str | None) -> bool:
    """Whether the SDK's *local* failure posture must fail closed (AAASM-4130).

    ``enforcement_mode is None`` is the advertised default: it registers the agent
    under the gateway's server-side default, which is live ``enforce``. So for the
    SDK's own error posture ``None`` must behave exactly like ``enforce`` -- an
    unreachable runtime or an unauthoritative query denies rather than silently
    proceeding. Only the explicit dry-run postures (``observe`` / ``disabled``) fail
    open locally.
    """
    return enforcement_mode is None or enforcement_mode == ENFORCE_MODE

So on the current published SDK every demo raises out of registration before reaching its first tool call, while its README promises:

No API key or running gateway is required for the offline demo.

This is not an SDK regression to work around. rc.6 was the version that was wrong — it let an agent proceed ungoverned under the advertised default. The examples were written against that hole and never declared the posture they depend on. Declaring it is the fix.

observe rather than disabled, because it is the documented dry-run posture and it keeps the offline demos symmetric with the live scenario, which already passes an explicit enforce at scenarios/live-core-enforcement/python-agent/agent.py:95.

Why the change lands in src/main.py too, not only the tests

The call in src/main.py sits inside the quickstart region that scripts/extract_snippets.py lifts into the published snippet. Fixing only the tests would turn CI green and leave the snippet a reader copies still broken. snippets/python/*.py are the regenerated output, and the third commit corrects the one README whose production note told the reader to remove arguments one at a time — following it verbatim would now leave a fail-open posture in a production agent.

How to verify

Every claim below was reproduced locally against the published rc.7 wheel, not against an unreleased build. Both rc.6 and rc.7 are on PyPI today.

Per example:

cd python/<name>
uv sync --extra dev --locked --no-build
uv run --locked --no-build pytest tests/ -v

Results on rc.7 with this branch — 16 of 16 suites pass:

agno-tool-policy                          6 passed
autogen-tool-policy                       6 passed
crewai-research-crew                      6 passed
custom-tool-policy                        7 passed
google-adk                                7 passed
haystack-tool-policy                      6 passed
langchain-basic-agent                     6 passed
langchain-research-agent                  8 passed
langgraph                                 5 passed
llamaindex-tool-policy                    6 passed
microsoft-agent-framework-tool-policy      4 passed
openai-agents-sdk                         7 passed
pydantic-ai                               7 passed
semantic-kernel-tool-policy               6 passed
smolagents-tool-policy                    8 passed
strands-agents-tool-policy                6 passed

Anti-vacuity: with the posture line stripped from all 16 test files and nothing else changed, 16 of 16 fail, one failing test each.

agno-tool-policy                          1 failed, 5 passed
autogen-tool-policy                       1 failed, 5 passed
crewai-research-crew                      1 failed, 5 passed
custom-tool-policy                        1 failed, 6 passed
google-adk                                1 failed, 6 passed
haystack-tool-policy                      1 failed, 5 passed
langchain-basic-agent                     1 failed, 5 passed
langchain-research-agent                  1 failed, 7 passed
langgraph                                 1 failed, 4 passed
llamaindex-tool-policy                    1 failed, 5 passed
microsoft-agent-framework-tool-policy      1 failed, 3 passed
openai-agents-sdk                         1 failed, 6 passed
pydantic-ai                               1 failed, 6 passed
semantic-kernel-tool-policy               1 failed, 5 passed
smolagents-tool-policy                    1 failed, 7 passed
strands-agents-tool-policy                1 failed, 5 passed

Reverting the pin to rc.6 on the same machine passes either way, which is what isolates the pin as the only variable.

The metadata drift gate, all five steps, run locally from the repo root:

python scripts/generate_example_metadata.py   -> No changes -- every example is already in sync with the SoT.
python scripts/extract_snippets.py            -> Extracted 26 snippet(s) across 3 SDK(s). No changes.
git diff --exit-code                          -> clean
python scripts/generate_example_metadata.py --check
                                              -> --check: every audited surface matches the SoT.
python -m unittest scripts.test_generate_example_metadata
                                              -> Ran 39 tests, OK

uv lock --check is clean on all 18 locked directories. Each relock moved exactly one package and preserved revision = 3.

Three examples (google-adk, semantic-kernel-tool-policy, strands-agents-tool-policy) cannot be installed locally with the exact CI flag combination, because watchdog publishes no macOS-arm64 wheel for this interpreter and pybars4 ships no wheel at all, so --no-build refuses them. That is pre-existing and unrelated: it reproduces identically on unmodified main at rc.6. Their numbers above come from the same command without --no-build, so the same test code ran on rc.7 and only dependency acquisition differed. CI runs ubuntu, where manylinux wheels exist. The command the READMEs actually give users is uv sync --extra dev, which has no such restriction.

What this means for the 18 open Dependabot PRs

They were recorded as blocked on a future release. That was wrong. Their two failures are both release-independent:

  • metadata drift — the generator regenerates pyproject.toml from the SoT and reverts Dependabot's rc.7 back to rc.6.
  • the per-example lane — uv sync --locked refuses because Dependabot bumped the manifest without relocking.

This PR resolves both by satisfaction rather than by closure. A separate ticket covers the structural half: Dependabot's pip ecosystem edits a generated pin, and advancing it correctly also rewrites a Dockerfile and a workflow that pip will never touch, so those PRs cannot satisfy the drift gate by construction.

Fourth commit: the SonarCloud duplication gate (AAASM-6206)

The quality gate initially failed on one condition, new_duplicated_lines_density 4.1% against a 3% threshold, localised to 3 new duplicated lines in each half of one pair:

dup=3  new=3  snippets/python/llamaindex-tool-policy.py
dup=3  new=3  python/llamaindex-tool-policy/src/main.py

The reported block is 31 lines — the whole extracted quickstart region, not the 3 added lines. snippets/python/llamaindex-tool-policy.py is a verbatim extraction of that region by scripts/extract_snippets.py, drift-gated by example-metadata-check.yml, so the two files are identical by construction. I verified that directly: all 16 Python snippet/source pairs are byte-identical, modulo dedent, on main as well as on this branch.

main reports duplicated_lines 0 and duplicated_blocks 0 for that same file, with an empty duplications list, so nothing about the content became duplicated here — only the detection changed. .sonarcloud.properties already excludes three sibling pairs (agno, crewai, haystack) from copy-paste detection for exactly this reason; its comment records that the list was "taken from the duplication report as of 2026-07-23", and llamaindex was below the detector's minimum block size on that day. One added argument plus its comment pushed it over.

So the fourth commit adds both halves of that pair to the existing list and replaces the dated-snapshot comment with the invariant. Both files stay fully analysed by every other rule, and every hand-written source file stays under full duplication analysis — nothing is removed from analysis scope, and no paid SonarCloud capacity is involved. The gate now passes all five conditions, with the PR's total duplication back to 101 lines / 7 blocks, exactly matching main.

AAASM-6206 records the structural half: the exclusion list is wildcard-free by platform restriction, so it is a hand-maintained snapshot with no gate, and the next region to cross the boundary will fail the same way.

Coverage caveat: the two rc.7 pins this PR's checks do not exercise

Two of the 58 files whose pin this advances are not reachable from any check on this PR:

  • scenarios/live-core-enforcement/python-agent/Dockerfile — a RUN pip install line.
  • .github/workflows/verify-live.yml — a run: string.

verify-live.yml is schedule (daily 06:00 UTC) plus workflow_dispatch only; it has no pull_request trigger. And verify-scenarios.yml's live: live-core-enforcement (real gateway) job is gated if: github.event_name == 'workflow_dispatch', which is the one skipped check on this PR. Its sibling validate: live-core-enforcement (offline, no gateway) does pass, but it only installs PyYAML to validate policy documents and never runs the agent, so it carries no posture exposure either.

Those two pin edits are therefore verified by inspection only here, and will first be executed by the next scheduled live run or a manual dispatch. Calling this PR fully end-to-end validated would overstate it.

Not done here

No release is created, and none is needed — rc.7 is already published.

Refs: AAASM-6204

Chisanan232 and others added 3 commits September 28, 2026 14:33
`metadata/sdk-versions.yaml` is the single source of truth for the SDK
versions every example advertises, and its own bump policy says to edit it
on a new SDK release and then run the generator. agent-assembly 0.0.1rc7
has been on PyPI (13 files, not yanked) while this file still said rc6 —
and the node pin was already at 0.0.1-rc.7, so the two ecosystems had
drifted apart.

Bumping the pin here and running `scripts/generate_example_metadata.py`
rewrites 40 files: 18 `pyproject.toml` manifests, 20 bounded README
blocks, `scenarios/live-core-enforcement/python-agent/Dockerfile` and
`.github/workflows/verify-live.yml`. Every changed line is the version
string alone. The 18 `uv.lock` files are relocked with
`uv lock --upgrade-package agent-assembly`, so exactly one package moves
in each: no incidental resolution changes, and `revision = 3` is
preserved in all 18.

This is one commit rather than three because neither half is green on its
own — an advanced manifest with a stale lock fails `uv sync --locked`, and
an advanced lock with a stale manifest fails it too. The pin, its
generated output and its locks are a single bisectable unit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Python example calls init_assembly in sdk-only mode with no
enforcement_mode, so it inherits the advertised default. On 0.0.1rc6 --
the version these examples are pinned to -- that default silently failed
open when registration could not reach a gateway, which is the only
reason the offline demos ran at all.

0.0.1rc7 closed that hole (AAASM-4130). _local_posture_is_enforce now
treats a None posture exactly like enforce for the SDK's own error
posture, and only the explicit dry-run postures fail open locally. So on
the current published SDK every one of these demos raises out of
registration before reaching its first tool call, while its README
promises "No API key or running gateway is required for the offline
demo."

This is not an SDK regression to work around. rc.6 was the version that
was wrong: it let an agent proceed ungoverned under the advertised
default. The examples were written against that hole and never declared
the posture they actually depend on. Declaring it is the fix.

observe rather than disabled: it is the documented dry-run posture, and
it keeps the offline demos symmetric with the live scenario, which
already passes an explicit enforce at
scenarios/live-core-enforcement/python-agent/agent.py:95.

The change lands in src/main.py as well as the tests, and deliberately
so. The call in src/main.py sits inside the quickstart region that
scripts/extract_snippets.py lifts into the published snippet, so fixing
only the tests would turn CI green while leaving the snippet a reader
copies still broken. snippets/python/*.py are the regenerated output.

Verified against the published rc.7 wheel, not an unreleased build: all
16 suites pass with the posture and all 16 fail without it, one failing
test each. Reverting to rc.6 on the same machine passes either way,
which is what isolates the pin as the only variable.

Refs: AAASM-6204
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The production note told the reader to remove the sdk-only mode argument
and said nothing about enforcement_mode, which did not exist here until
the previous commit. Following it verbatim now leaves an explicit observe
posture in a production agent -- a dry-run posture that carries on when
the SDK cannot reach the gateway, which is precisely the fail-open
behaviour rc.7 closed.

This is the only one of the 16 READMEs that instructs the reader to
remove arguments one by one; the rest show a complete production call, so
they are unaffected.

Refs: AAASM-6204
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…etection

`snippets/python/llamaindex-tool-policy.py` is a verbatim extraction of the
`region: quickstart` slice of `python/llamaindex-tool-policy/src/main.py`, so
the two are identical by construction — `scripts/extract_snippets.py` produces
one from the other and `example-metadata-check.yml` fails on any drift between
them. Three sibling pairs (agno, crewai, haystack) are already excluded from
CPD for exactly this reason.

llamaindex was missing only because `sonar.cpd.exclusions` was seeded from the
duplication report of one particular day, and on that day this region sat below
the detector's minimum block size. Adding the `enforcement_mode` argument and
its comment pushed it over, so the gate reported 3 new duplicated lines in each
half of a pair that is already 100% identical on main — where Sonar reports
zero duplication for the same file.

Both files remain fully analysed by every other rule, and all hand-written
source remains under full duplication analysis.

Refs AAASM-6206

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@Chisanan232
Chisanan232 merged commit 21f3dd0 into main Sep 28, 2026
45 checks passed
@Chisanan232
Chisanan232 deleted the v0.0.1/AAASM-6204/fix/python_examples_offline_posture branch September 28, 2026 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant