PostHog: configure prompt evals (Hog + LLM-as-judge) for PR/repo explainers

[shilpi1958]

Summary

Instrument $ai_generation (done in code). Configure PostHog AI Evals so we continuously score PR + repo explainer prompts.

Prerequisite

• POSTHOG_API_KEY (+ optional POSTHOG_HOST) set when running the CLI
• Generations appear under AI Observability → Generations with properties:
  • explainer_mode: pr | repo
  • prompt_name: pr_explainer | repo_explainer

Evals to create (AI Evals → Evaluations)

1. Code-based (Hog) — format / structure — sample ~100%

Filter: prompt_name is set (or all $ai_generation from this product).

Checks:

• Output starts with # title
• Has ## Quick check and at least one **Q1.
• Has A/B/C options and <details> Answer with Correct:
• For explainer_mode = repo: sections What it does, How it's put together, Why it matters
• For explainer_mode = pr: sections What changed, Why it was done this way, Why it matters
• Soft length: fail if output > ~6k chars (verbosity regression)

2. LLM-as-a-judge — non-technical pitch — sample ~10–20%

Filter: explainer_mode = repo (and optionally pr).

Pass if:

• Plain language suitable for a product analyst
• No dense src/… / main() architecture dump unless clearly for an engineer profile
• Concise (bullets over essay)
• Grounded (no invented product claims vs input)

3. LLM-as-a-judge — quiz quality — sample ~10%

Pass if Quick check tests purpose/takeaway, not filename trivia.

Acceptance

• Provider API key configured in PostHog → Settings → AI Observability
• Hog format eval enabled and passing on recent generations
• At least one LLM judge eval enabled with pass-rate report
• Dashboard or eval report reviewed after Express/Prettier-style runs

Docs

https://posthog.com/docs/ai-evals

[shilpi1958]

Closing the code side as done in #18:

• $ai_generation capture with `explainer_mode` / `prompt_name`
• Funnel events + `.env` / Action secret wiring
• Tests for capture/shutdown

Still manual in PostHog UI (not a code issue): create the Hog + LLM-as-judge evals under AI Evals using the checklist in this issue body, and add a provider key under Settings → AI Observability.