Skip to content

test(bub): run real Bub with the PowerContext plugin against a real Server - #2649

Open
Fengzdadi wants to merge 1 commit into
oceanbase:masterfrom
Fengzdadi:test/bub-service-chain-e2e
Open

Fengzdadi wants to merge 1 commit into
oceanbase:masterfrom
Fengzdadi:test/bub-service-chain-e2e

Conversation

@Fengzdadi

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

No tracking issue. Bub had no real-host service chain; this PR adds one. Its recall case guards the fix for #2487 (#2488).

Rationale for this change

The Bub plugin tests in e2e/bub/tests/ call the hooks with hand-built bub.hooks.interception objects. The harness-validation job type-checks and unit-tests the replay harness but never starts a Bub turn, and the Harbor replay uses real models outside CI. Nothing ran Bub's own framework against a Server. As a result, #2487 (Bub's channel frame in the recall query) only surfaced through manual runs.

What changes are included in this PR?

  • tests/e2e/test_bub_service_chain.py runs the real bub run CLI with the plugin discovered through its bub entry point.
    • Bub 0.6.1 comes from e2e/bub/.venv, which make harness-sync builds. Each case gets an isolated HOME/BUB_HOME.
    • The Server runs in-process with SQLite, bearer auth, and no memory schedule. HostModelFixture serves both Bub's model and the Server's generation model.
    • Bub runs as a subprocess from the harness environment, and the Server stays in pytest's main environment. The harness environment installs only powercontext[client]. The split also mirrors a deployment, with the host process on one side and the Server on the other.
    • Each workspace holds an empty .env. Bub loads the nearest .env from its working directory upward. Without the empty file, a run under a checkout would pick up that checkout's POWERCONTEXT_BUB_* values.
  • The cases are taken from integrations/bub/README.md and the evaluation integration docs:
    • Recall: a prompt captured in one Bub session becomes Memory through the plugin's own end-of-turn flush. A later session's first model call carries it after the PowerContext host-supplied context label.
    • Credentials: capture removes a known credential environment value (Bub's provider key) from the captured prompt. Neither the stored Sources nor the generation model receive it.
    • Workspace binding: an operator-bound workspace key reaches its Scope. A resumed session replays its earlier turn and still gets fresh context. Another workspace stays on the Server default, an explicit POWERCONTEXT_BUB_SCOPE_ID takes precedence, and no workspace path reaches the Server.
    • Settings: turning capture_events off stops capture and flush only. Recall keeps running.
    • Tools: powercontext.remember, powercontext.search, and powercontext.context reach the bound Scope, not the default.
  • CI:
    • A new bub suite and bub-host job in master.yml run make harness-sync, then run the cases with POWERCONTEXT_TEST_BUB_HOST=1.
    • Without the Bub environment, that variable turns the skips into failures.
    • The job uploads bub.log on failure.
    • .github/ci-paths.toml selects the suite for changes to the Bub plugin, the harness lock, the test, the shared wire fixtures, and the shared runtime. tests/test_ci_changes.py covers the new selection rows.
  • integrations/capabilities.toml lists the new test as Bub evidence. The only test listed before, the workload catalog, runs none of Bub's declared capabilities, and this test runs all of them. Editing master.yml already selects full CI, so the manifest edit adds no CI cost.

Are there any user-facing changes?

No. This PR changes tests and CI only.

How was this change tested?

  • uv run pytest tests/e2e/test_bub_service_chain.py against real Bub 0.6.1 on macOS: all 5 cases pass.

  • Each mutation below was applied to the plugin, the cases were run, and the mutation was reverted:

    Mutation Failing cases
    Recall with the framed prompt instead of the user's text (bug(bub): recall query includes Bub's channel frame, so lexical recall returns empty #2487) recall
    Drop the context message recall, workspace binding
    Capture without render_capture_event redaction credentials
    No workspace binding key workspace binding, tools
    Ignore the explicit Scope workspace binding
    Skip recall once the request has earlier turns workspace binding, settings
    Capture the prompt regardless of capture_events settings
    Gate recall on capture_events settings, recall, workspace binding, tools
    No end-of-turn flush recall, credentials, settings
    Tools resolve the Server default Scope tools
    Send the raw workspace path as the binding key workspace binding
  • With POWERCONTEXT_TEST_BUB_HOST=1 and no Bub environment, all 5 cases error. Without the variable, they skip.

  • A parent .env that sets a bogus POWERCONTEXT_BUB_SCOPE_ID and turns capture on leaves all 5 cases passing. Without the workspaces' empty .env, the settings case fails.

  • The workspace-path assertion searches decoded request strings, so JSON escaping cannot hide a Windows path.

  • make check, make harness-check, and uv run pytest tests/test_ci_changes.py tests/test_ci_pipeline.py tests/test_integration_manifest.py all pass.

  • The diff was reviewed twice before submission:

    • An independent review found no P1 or P2 issues. Its three P3s are fixed: the path check passed vacuously on Windows, a parent .env could reach Bub, and a comment described the frame bug.
    • Self-reviews with code-review at high effort led to these fixes: output is kept when bub run times out, the path check also covers resolved paths, the settings case no longer pins the exact capture event sequence, Bub no longer inherits proxy variables, and the log upload skips pytest's *current links.
    • Left as is:
      • The host gate does not check the Bub version, because CI always installs the locked environment.
      • Like the other service chains, the in-process Server reads POWERCONTEXT_SERVER_* variables from the runner's environment. CI sets none of them.

AI usage statement

Claude Code (Claude Opus 5.5) wrote the test, the CI wiring, and this description, and ran the local verification.

…erver

Add a service-chain E2E that runs `bub run` from the Bub 0.6.1 harness
environment against an in-process Server and deterministic model fixtures.
It covers cross-session recall, capture redaction of known credential
values, workspace binding across a resumed session, the capture setting,
and the PowerContext tools. A new bub suite and bub-host job run it in CI
and refuse to skip without the Bub environment.

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AlexStocks AlexStocks added the 1.5.2 v1.5.2 release tracking label Nov 15, 2026

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings at 8444d21. The real Bub service-chain coverage addresses the integration regression gap without adding runtime database tables or public interfaces. Independently exercised pinned Bub 0.6.1 against the real SQLite Server with controlled model endpoints, checked CI selection and result gates, and confirmed that restoring the framed recall query makes the new recall case fail as intended.

@hidb4ai hidb4ai removed the 1.5.2 v1.5.2 release tracking label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants