Skip to content

Usage read model / Typert remote / Profile contract #39

Closed
5 of 5 issues completed
Closed

Description

@DoodleBears

Problem Statement

A Human cannot tell which provider and model a PersonaBot actually used or how many tokens each consumed across Orchestrator, Assignment, and DSH Subagent work. The current token card combines a multi-model Turn into a mixed label, and rebuilding its daily table from surviving Session logs would erase statistics after ordinary Session deletion. Permission to use a model is not evidence that the model was called.

Solution

Show retained, observed model usage in the PersonaBot Profile. Attribute actual DSH model attempts to the owning PersonaBot, execution role, exact provider/model, and Host-local day. Show input, output, cache-read, and cache-write tokens separately; mark missing provider usage as unknown. Keep daily aggregate statistics after ordinary Session deletion, remove identifiable usage after thorough PersonaBot purge, and let the Human filter the Profile chart and totals by route and role.

User Stories

  1. As a Human, I want to see which provider/model a PersonaBot actually used, so that I can verify its execution rather than infer use from its allowed plan.
  2. As a Human, I want usage grouped by exact provider and model, so that similarly named models from different providers stay distinct.
  3. As a Human, I want to see input tokens, so that I can understand prompt consumption.
  4. As a Human, I want to see output tokens, so that I can understand generation consumption.
  5. As a Human, I want to see cache-read and cache-write tokens separately, so that cached activity is visible rather than hidden in one total.
  6. As a Human, I want missing provider usage marked unknown, so that I do not mistake an absent report for zero consumption.
  7. As a Human, I want Orchestrator, Assignment, and DSH Subagent work attributed to the same owning PersonaBot, so that its complete workload is visible.
  8. As a Human, I want to filter those three execution roles, so that I can distinguish routine conversation from delegated work.
  9. As a Human, I want to filter by provider/model, so that I can inspect one route or compare several routes used by a Bot.
  10. As a Human, I want a daily activity chart that initially shows 26 weeks, so that recent use is easy to scan.
  11. As a Human, I want to choose another bounded time range and see an all-time total, so that I can inspect both short-term and cumulative use.
  12. As a Human, I want a Turn that calls multiple models to contribute to each model separately, so that no usage disappears into mixed.
  13. As a Human, I want reported usage from retries and failed attempts counted when the provider reports it, so that the statistic follows actual model consumption.
  14. As a Human, I want an ordinary Session deletion to preserve daily usage totals without preserving identifiable deleted-Session detail, so that I can clear conversation history without falsifying statistics.
  15. As a Human, I want archiving a PersonaBot to preserve its usage for inspection, so that archived activity remains understandable.
  16. As a Human, I want a thorough PersonaBot purge to remove its identifiable usage, so that the Bot's data can be cleared completely.
  17. As a Human, I want empty, partial, unknown, and temporarily degraded states explained in the Profile, so that I know whether a blank chart means no calls or a reporting problem.
  18. As a Human, I want a refreshed Profile to show newly recorded usage without reading private DSH logs in the browser, so that the UI stays within the trusted Host boundary.

Implementation Decisions

  • DSH SessionEvents and provider-reported attempt usage are the source facts. Trusted Session ownership attributes each Orchestrator, Assignment, and DSH Subagent attempt to a PersonaBot. BotHarness owns the application-defined retained statistic, not the DSH transcript or provider credentials (ADR-0094).
  • Aggregate by PersonaBot, Host-local calendar day, execution role, exact provider, and exact model. Keep uncached input, output, cache-read, and cache-write buckets plus an unknown-usage indicator or count. The reported provider route of each actual attempt wins over the Bot's configured Model Plan.
  • A Turn using multiple routes contributes its attempts to their individual route buckets. Reported usage from failed or retried attempts is included exactly once. Unknown usage is not converted into zero or an invented estimate.
  • Incremental folding is idempotent per durable attempt. Reconciliation may correct only intervals with complete source evidence; it cannot clear retained history or double-count remaining Session logs after an ordinary Session is deleted.
  • Ordinary Session deletion preserves the daily aggregate while removing deleted-Session identifiable detail from the retained statistic. A thorough PersonaBot purge removes its identifiable aggregates; archiving leaves them readable.
  • The Host exposes bounded typed queries through the existing Typert remote and API Gateway. Queries include route/role/date filters, totals, daily series, freshness, and explicit empty/partial/degraded information. The browser consumes this read model; it does not read DSH Session logs or the operational database directly. Refresh may start with bounded polling; later push frames may invalidate but never own usage facts.
  • The existing Profile activity area is the consumer: default 26-week chart, selectable range, all-time total, and provider/model plus role breakdown. Detailed results are paginated or range-limited. Model permissions and actual use appear as different concepts.
  • v1.0 reports token buckets, not monetary estimates. The Host's local time zone defines the day boundary. Cost, currency, and price-era decisions remain in 价格与计费口径:官方来源、币种/时段/缓存规则 #35.
  • Delivery begins with one real PersonaBot model attempt visible under its exact provider/model in Profile. Expand to multi-route Turns, Assignment and Subagent attribution, retention across deletion, and purge, validating each runnable slice before the next. Each UI PR includes screenshots and an operable Human QA state.

Testing Decisions

  • Test observable usage through the highest existing seam: execute a real DSH model request, read the Host's Profile usage query, and confirm the visible route and reported buckets. Do not assert private table layout as the primary acceptance criterion.
  • Extend existing usage, bridge-method, and PersonaBot Profile tests for query shape and UI states. Use DSH adapter and Assignment tests for trusted ownership and role attribution. Focus coverage on behavior: exact provider/model, four token buckets, unknown usage, filters, and bounded date ranges.
  • Exercise a multi-model Turn, reported retry or failure, duplicate/late delivery, ordinary Session deletion followed by reconciliation, PersonaBot archiving, and thorough purge. Confirm totals change only when the underlying observed consumption or purge semantics require it.
  • Verify the first cross-layer slice against an isolated DSH instance with a real model reply. Give the Human a Profile path to inspect the model breakdown and screenshot the resulting UI.

Out of Scope

Further Notes

Activity

  1. added this to the v1.1 milestone on Sep 18, 2026
  2. changed the title [-]Dashboard 与 botharness/usage 桥契约[/-] [+]Usage read model / Typert remote / Dashboard contract[/+] on Sep 20, 2026
  3. added
    V1.1Milestone v1.1; mirrors the GitHub milestone
    area:fullstackEnd-to-end Host and Client implementation
    feature:usageUsage and cost attribution
    scope:designDesign decision or UX contract
    and removed
    V1.1Milestone v1.1; mirrors the GitHub milestone
    on Sep 22, 2026
  4. modified the milestones: v1.1, v1.0 on Sep 28, 2026
  5. 5 remaining items

  6. changed the title [-]Usage read model / Typert remote / Dashboard contract[/-] [+]Usage read model / Typert remote / Profile contract[/+] on Sep 29, 2026
  7. DoodleBears commented on Sep 29, 2026

    @DoodleBears
    ContributorAuthor

    Agent work outcome

  8. removed
    agent:activeOne or more coding-agent tasks currently claim this issue; see claim comments for task IDs
    on Sep 29, 2026
  9. added
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    scope:designDesign decision or UX contract
    on Sep 29, 2026
  10. DoodleBears commented on Sep 29, 2026

    @DoodleBears
    ContributorAuthor

    Agent work outcome update

  11. DoodleBears commented on Oct 1, 2026

    @DoodleBears
    ContributorAuthor

    Usage delivery completion

    All implementation children are now merged: #499, #502, #503, #507 and the coordinated chart UI #592. The final slice #507 passed Human QA and merged via #641 (dfe8b637b0515959e1ab706d9b99dbb0c1c9024c). Profile defaults to seven days, filters actual models or providers and execution categories, shows retained all-time totals, and refreshes on demand without polling. Ordinary Session deletion preserves token statistics; PersonaBot purge follows the established privacy contract. Real DSH evidence is included in each UI PR.

    Task: codex/local/01a0e445-0c0f-70e1-bad4-fd99c8cfaa21. #488 remains open because #505 still needs full Profile Backup/Restore and PersonaBot Export prerequisites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:fullstackEnd-to-end Host and Client implementationfeature:usageUsage and cost attributionready-for-agentFully specified, ready for an AFK agentwayfinder:grillingWayfinder grilling ticket

    Type

    No type

    Projects

    • Status
      Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions