Skip to content

Reduce no-match automation telemetry volume - #767

Open
malhotra5 wants to merge 1 commit into
mainfrom
reduce-no-match-automation-telemetry
Open

malhotra5 wants to merge 1 commit into
mainfrom
reduce-no-match-automation-telemetry

Conversation

@malhotra5

Copy link
Copy Markdown
Member

Summary

Stop sending automation_event_matched product telemetry to PostHog when an accepted webhook event matches zero automations.

The service still records the event in integration_events and still emits the internal observability span event with matched_count=0, so operational evidence remains available without paying PostHog volume for no-op webhook traffic.

Why

The Agent Canvas PostHog spike starting Nov 12 was dominated by no-match GitHub workflow-run deliveries. Most of these events had matched_count == 0, so capturing every no-op match check created high product analytics volume without representing user-visible automation work.

Tests

  • uv run pytest tests/test_ingest.py -k 'telemetry or product_telemetry'
  • uv run pre-commit run --files openhands/automation/ingest.py tests/test_ingest.py

This PR was created by an AI agent (OpenHands) on behalf of the user.

@malhotra5 can click here to continue refining the PR

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Warning

Your comment is too long (maximum is 65536 characters), so the coverage report was not added. See the job log for how to reduce it.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed the diff on head b27b878 against main, the surrounding ingest/telemetry/trigger-matcher code, and the test suite.

Assessment

  • The if matched: gate around capture_automation_event("automation_event_matched", ...) only suppresses the PostHog capture when zero automations matched. The integration_events row (record.matched_count) and the internal observability span event (automation.event.matched with automation.event.matched_count) are both emitted before the gate, so operational evidence for no-match deliveries is retained as described.
  • No lost product telemetry beyond the intended event: automation_run_created / automation_conversation_continued only fire inside the matched loop, and grep confirms no other emitter or consumer of automation_event_matched.
  • The new test deterministically drives the no-match path (on: "message" vs event key app_mention per matches_trigger semantics) and asserts real behavior end-to-end through accept_event with the telemetry HTTP client captured, including the persisted matched_count == 0 row.

Verification

  • unit-tests, backend, windows-tests, and all image/description checks pass on this head. Local lint and type checks (ruff format/check, pycodestyle, pyright) pass on both changed files. DB-backed tests require a PostgreSQL testcontainer that cannot run in this review sandbox, so suite correctness is verified via the green unit-tests CI run on the exact head plus local static verification.
  • Non-blocking note: the pr-title / Lint PR title (conventional) check fails because the title lacks a Conventional Commits prefix (e.g. feat:/chore:). This affects release-notes label grouping only; consider a retitle before merge.

No material correctness, security, or design findings.

✅ APPROVED

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants