feat(cockpit): deploy a hosted read-only Cockpit to Azure App Service

[placerda]

Summary

Add an optional agentops cockpit deploy experience that deploys a secure,
read-only AgentOps Cockpit to Azure App Service.

The hosted Cockpit should provide a shared operational view of available
Microsoft Foundry and Azure Monitor telemetry without requiring each user to
install AgentOps locally, write KQL, or manage connection strings.

The existing local agentops cockpit experience must remain unchanged.

Problem

The current Cockpit is intentionally local and workspace-oriented. It reads
configuration, Doctor history, and evaluation results from the local
agentops.yaml and .agentops/ directories.

This works well for an individual developer, but it does not provide a simple
shared experience for a team that wants to inspect available telemetry across
Azure resources.

Operators currently need to navigate multiple Foundry projects, Application
Insights resources, and Log Analytics workspaces. They may also need to
understand resource IDs, workspace configuration, and KQL before they can
identify:

• which projects and agents are producing telemetry
• request volume, failures, and latency
• input and output token usage
• which expected telemetry dimensions are unavailable

Goal

Provide a guided Azure deployment that results in one authenticated Cockpit URL
for a selected Azure scope.

The end-user experience should be:

agentops cockpit deploy

The command should guide the operator through selecting the subscription,
resource group, and telemetry scope, deploy the required Azure resources, and
return the Cockpit URL.

After deployment, users should sign in with Microsoft Entra ID and see only
read-only operational information available to the Cockpit identity.

Proposed MVP

Deployment

• Deploy the existing FastAPI Cockpit to Azure App Service.
• Use versioned Bicep under infra/ and an azd-compatible deployment flow.
• Enable a system-assigned managed identity on the App Service.
• Configure Microsoft Entra authentication for access to the web application.
• Restrict sign-in to an explicitly configured tenant and, when provided, an
  allowed Entra group.
• Return the application URL and Azure portal resource link after deployment.
• Keep the deployment repeatable and safe to rerun.

Deployment architecture

The Cockpit panels are part of the FastAPI application. They are not deployed
as separate Azure dashboards or Workbook resources.

Use the following responsibilities:

Layer	Responsibility
agentops cockpit deploy	Guided user experience and validation
Azure Developer CLI (azd)	Provision and deploy orchestration
Bicep	Versioned definition of Azure resources and RBAC
Azure Resource Manager	Deployment engine generated from Bicep

The expected flow is:

1. agentops cockpit deploy checks Azure and azd authentication.
2. The user selects or confirms the subscription, resource group, location, and
   telemetry scope.
3. The command runs an infrastructure preview and presents the resources and
   role assignments before confirmation.
4. azd provision deploys the Bicep infrastructure.
5. azd deploy packages and deploys the FastAPI Cockpit.
6. The command verifies the health endpoint and returns the Cockpit and Azure
   portal URLs.

The Bicep deployment should create:

• an App Service plan
• a Linux App Service for the FastAPI application
• a system-assigned managed identity
• App Service Authentication configuration
• application settings containing only non-secret configuration
• least-privilege role assignments for the selected Azure scope

The deployment should reuse existing telemetry resources. It should not create
or modify the customer's Foundry projects, Application Insights resources, Log
Analytics workspaces, or diagnostic settings.

Creating Microsoft Entra app registrations is not supported directly by ARM or
Bicep. The deployment experience should either accept an existing client ID or
offer an explicit CLI-assisted registration step when the deployer has the
required tenant permission. This choice must be resolved before implementation.

The same Bicep can later support a Deploy to Azure entry point. Directly
maintaining an ARM JSON template is not recommended because Bicep is the source
of truth.

Authorization

Assign only the roles required for the selected scope:

Purpose	Recommended built-in role
Discover Azure resources	Reader
Read Azure Monitor and Application Insights data	Monitoring Reader
Query selected Log Analytics workspaces	Log Analytics Reader

Role assignments should default to the selected resource group. Subscription
scope must require an explicit choice.

The deployment must not store Azure credentials, workspace keys, or Application
Insights connection strings. Runtime access should use
DefaultAzureCredential, which resolves to the App Service managed identity.

Hosted Cockpit

Add an explicit hosted mode that does not assume access to a local workspace.
The initial hosted view should query Azure live and provide:

• project, agent, model, and time-range filters when those dimensions exist
• request or invocation volume
• failure rate
• latency
• input and output token usage
• a telemetry coverage status that distinguishes:
  • resource inaccessible
  • telemetry not configured
  • telemetry configured but no data found
  • expected attribution field unavailable

Missing data must not be represented as zero or as a successful state.

The UI should hide resource IDs, workspace query syntax, and KQL from the normal
operator workflow. Advanced troubleshooting may expose the source resource and
the reason data is unavailable.

Proposed panels

Add one new top-level Cockpit tab named Observe. Keep the existing Cockpit
experience intact.

The Observe tab contains four internal views:

1. Overview
2. Agents
3. Models and usage
4. Telemetry coverage

A shared filter bar should apply the selected Azure scope, project, agent,
model, and time range to every Observe view. Switching views must preserve the
current filters and time range.

Local and hosted behavior

The Observe tab should run in both Cockpit modes using the same query,
normalization, API, and UI components:

Mode	Command	Azure authentication	Scope configuration
Local	agentops cockpit	DefaultAzureCredential, normally Azure CLI after az login	Current agentops.yaml, environment, and optional CLI scope
Hosted	agentops cockpit deploy	App Service managed identity	Azure scope saved as non-secret App Service configuration

Local mode retains the existing workspace, Doctor, and evaluation-history
experience. Hosted mode does not have access to the user's local
agentops.yaml or .agentops/ files, so it must not show local history as if it
were available.

Observe is cloud-backed in both modes. Given the same Azure permissions, scope,
and time range, local and hosted mode should produce equivalent normalized
metrics. Differences in Azure permissions must be shown in Telemetry coverage.

Query execution and performance

Do not execute a full telemetry fan-out when the Cockpit process starts.

Use progressive loading:

1. Opening the Cockpit loads the application shell without querying telemetry.
2. Opening Observe loads cached Azure resource discovery and executes only the
   aggregated Overview queries.
3. Agents, Models and usage, and Telemetry coverage query their additional data
   only when the user opens the corresponding view.
4. Agent detail and trace correlation load only after the user selects an
   agent.

Use the following defaults:

• 24-hour initial time range
• explicit Apply action for scope and filter changes
• five-minute automatic refresh for the active Observe view
• manual Refresh action
• two-minute result cache keyed by scope, view, filters, and time range
• 15-minute resource-discovery cache
• bounded parallelism across workspaces
• per-query timeout and partial-result reporting
• cancellation or result suppression for stale requests

Queries should:

• constrain TimeGenerated before parsing or aggregation
• select only required tables and columns
• apply project, agent, model, and resource filters as early as possible
• summarize in Azure Monitor instead of returning raw rows to the Cockpit
• use bounded result sizes for detail tables
• avoid wildcard table searches and unbounded cross-workspace unions
• report query duration, source count, partial failures, and last refresh time

Cross-workspace execution should be planned from the discovered resource set and
limited to workspaces relevant to the selected scope. A slow, inaccessible, or
cross-region workspace must not block all other results. The UI should render
available results and identify incomplete sources in Telemetry coverage.

Visual direction

Use a compact, GitHub-inspired operational dashboard style while retaining the
existing AgentOps visual identity:

• dark and light themes with accessible contrast
• thin solid trend lines
• subtle dashed grid lines
• restrained legends and labels
• compact metric cards
• tooltips with exact timestamp, value, and series
• responsive charts that do not require horizontal scrolling

Time-series charts should support a vertical area gradient beneath each line.
The fill starts with a low-opacity version of the series color near the line and
fades to transparent toward the chart baseline. The line remains solid so the
gradient never hides the actual value.

Multiple series may overlap only when they remain distinguishable. Use
accessible colors, line styles, and legend labels rather than relying on color
alone. The gradient is presentation only and must not imply confidence,
forecast, or an additional metric.

1. Overview

Provide the operational summary across all readable projects:

• projects with telemetry
• agents observed
• total invocations
• failure rate
• p95 latency
• input and output tokens
• telemetry coverage percentage
• invocation, failure, latency, and token trends

Each card should show its source and last refresh time. Selecting a project,
agent, or model should filter the other panels rather than opening a separate
dashboard.

2. Agents

Show one normalized row per observed agent:

Field	Behavior
Project	Foundry project when reported or discovered
Agent	Agent name, ID, or Not reported
Identity	Whether gen_ai.agent.id is available
Source	Foundry, external, or unknown
Last seen	Latest observed invocation, not lifecycle status
Invocations	Count for the selected period
Failure rate	Derived from observed operations
p95 latency	Derived when duration is available
Tokens	Input and output totals when reported
Model	Reported deployment or model

Selecting an agent should open a detail view with trends and deep links to the
corresponding Foundry trace or Application Insights transaction when available.

External agents are included only when their telemetry reaches an authorized
Application Insights or Log Analytics resource. OpenTelemetry gen_ai.*
attributes provide the preferred normalization contract.

3. Models and usage

Aggregate available model activity by project, agent, deployment, and model:

• request volume
• input tokens
• output tokens
• total tokens
• failures
• p95 latency
• last observed activity

This panel must describe tokens as observed usage. It must not present token
counts as billing records. Estimated cost is excluded from the MVP unless a
separate, explicit pricing configuration and estimation contract is approved.

4. Telemetry coverage

Provide a matrix that explains whether each project or agent can populate the
other panels:

Check	Example states
Azure resource access	Available, denied, or not found
Application Insights	Connected or not discovered
Recent traces	Found or no data in selected period
Agent attribution	ID and name available or not reported
Model attribution	Model available or not reported
Token usage	Input/output available or not reported
Trace correlation	Trace and conversation IDs available or not reported

This is the main troubleshooting surface. It should provide a concise reason and
recommended next action without requiring the operator to inspect KQL.

Mapping to customer needs

Need	Cockpit panel	MVP behavior
Central view across projects	Overview	Aggregate all telemetry resources readable in the configured scope
Tokens by agent, model, and project	Agents and Models and usage	Show only dimensions present in telemetry
Latency, failures, and volume	Overview and Agents	Aggregate observed operations and allow drill-down
Agents outside Foundry	Agents	Include external OTel sources using compatible gen_ai.* attributes
Agent identity	Agents and Telemetry coverage	Use gen_ai.agent.id when emitted and expose missing attribution
No-code operator experience	All panels	Hide KQL and resource wiring behind guided deployment and filters
Agent lifecycle	Not available from invocation telemetry	Show Last seen only and require an authoritative inventory or lifecycle source for future support
Billing-accurate cost	Not available from token telemetry	Treat tokens as observed usage and require Azure Cost Management billing data for future support
APIM quotas and enforcement	Not available from agent telemetry	Require separate APIM management and diagnostic data sources

Data-source boundaries

The MVP uses Azure resource metadata and runtime telemetry from Application
Insights and Log Analytics. Those sources can describe observed executions, but
they are not authoritative for every requested dimension:

• Invocation traces show that an agent was active. They do not reliably prove
  when an agent was created, updated, disabled, deleted, or promoted. Lifecycle
  support requires an authoritative Foundry inventory or lifecycle-event source.
• Token attributes describe observed model usage. They are not billing records.
  Billed cost can depend on model and version, region, deployment type, cached
  tokens, provisioned throughput, PAYG spillover, discounts, and pricing
  agreements. Billing-accurate cost requires Azure Cost Management data, and
  that data might not include agent-level attribution.
• APIM quotas and enforcement are defined by APIM policies and exposed through
  APIM management APIs, metrics, and diagnostic logs. Agent traces cannot prove
  which policy was applied, what quota remains, or which gateway consumer was
  throttled.

These capabilities are possible future integrations, but they should not be
inferred from missing or incomplete trace data.

Security model

Two independent controls are required:

1. App Service Authentication controls who can open the Cockpit.
2. The App Service managed identity controls which Azure resources the Cockpit
   can read.

All authenticated Cockpit users share the Azure data scope granted to the
managed identity. The UI and documentation must state that behavior clearly.

The hosted Cockpit remains read-only. It must not create or modify Foundry
projects, telemetry settings, agents, model deployments, alerts, or APIM
policies.

Acceptance criteria

• agentops cockpit deploy provides a guided deployment to Azure App
  Service.
• The local agentops cockpit behavior remains unchanged.
• The hosted app requires Microsoft Entra authentication.
• Runtime Azure access uses managed identity without stored credentials.
• RBAC defaults to resource-group scope and least-privilege read roles.
• Subscription-wide access requires explicit confirmation.
• The hosted mode starts without agentops.yaml or a local .agentops/
  directory.
• The Cockpit can discover readable telemetry resources in the configured
  scope.
• The hosted UI adds one top-level Observe tab with Overview, Agents, Models
  and usage, and Telemetry coverage internal views.
• Observe works in both local and hosted Cockpit modes through the same
  query, normalization, API, and UI components.
• Local Observe uses the developer's DefaultAzureCredential; hosted
  Observe uses the App Service managed identity.
• Hosted mode never presents unavailable local workspace history as cloud
  data.
• Observe filters and time range are shared and preserved when switching
  between views.
• Starting the Cockpit does not execute telemetry queries.
• Opening Observe executes only the resource-discovery and aggregated
  Overview workload.
• Inactive views and detail queries are loaded on demand.
• Query results and resource discovery use bounded caches and expose last
  refresh time.
• Cross-workspace queries use bounded parallelism, timeouts, and
  partial-result reporting.
• Time-series charts support an accessible area-gradient presentation with
  solid trend lines, subtle grids, exact-value tooltips, and light/dark
  themes.
• The panels show volume, errors, latency, tokens, and telemetry coverage
  when the source data supports them.
• Agent and trace drill-down provides Foundry or Application Insights deep
  links when available.
• Missing permissions, missing configuration, and no data are presented as
  distinct states.
• Last seen is not labeled or interpreted as agent lifecycle.
• Token usage is not labeled or interpreted as billing data.
• Deployment output includes the Cockpit URL and Azure portal link.
• Bicep validation, unit tests, and deployment dry-run coverage are added.
• Documentation explains required deployer permissions, runtime RBAC, and
  the shared managed-identity data scope.

Out of scope

• A Microsoft-hosted AgentOps SaaS
• Cross-tenant access
• Per-user delegated Azure RBAC
• A persistent application database
• Copying local .agentops/ history to the hosted service
• Creating or changing Application Insights and diagnostic settings
• Generating telemetry that agents or platforms do not emit
• Billing-accurate cost calculation
• Agent lifecycle inventory
• APIM gateway policy, quota, or enforcement management

Open questions

1. Should the first deployment require an existing Entra app registration, or
   should the CLI offer to create one when the deployer has tenant permission?
2. Is system-assigned managed identity sufficient for the MVP, or should a
   user-assigned identity be supported for stable identity across App Service
   replacement?
3. Should the initial Azure scope be limited to one resource group to keep
   deployment and permission review simple?
4. Which Azure Monitor tables and semantic-convention fields are stable enough
   to be part of the first supported telemetry contract?

Design principles

• Complement Foundry and Azure Monitor rather than duplicating their native
  experiences.
• Prefer a guided workflow over requiring KQL or Azure resource IDs.
• Keep the hosted runtime stateless and read-only.
• Never imply that unavailable telemetry was collected successfully.
• Make the authorized Azure scope visible to every Cockpit user.