Skip to content

fix(qa): mark the dashboard's placeholder scores as simulated - #5

Open
sergiobuilds wants to merge 1 commit into
LilMGenius:ralphthonfrom
sergiobuilds:fix/label-simulated-scores
Open

sergiobuilds wants to merge 1 commit into
LilMGenius:ralphthonfrom
sergiobuilds:fix/label-simulated-scores

Conversation

@sergiobuilds

Copy link
Copy Markdown

/api/personas/:id/qa-simulation returns twenty followers, five dimensions each, a total and a TOP 5 ranking. It reads as an evaluation. It is deterministicScore, a string hash over persona, follower, dimension, and the content name — the draft is never opened.

The CLI path is the opposite: virtual-follower runs a real LLM evaluation and even forces a Read on the saved report to confirm it exists. Two things named "QA score", one real and one a hash, and nothing in the type, the response, or the UI separates them.

Measured

A. same filename, content replaced wholesale
   garbage draft    -> f14:413  f15:394  f03:385  f09:361 ...
   polished draft   -> f14:413  f15:394  f03:385  f09:361 ...
   byte-identical

B. identical content, different filename
   alpha.md  -> top1 f14 / 413
   beta.md   -> top1 f18 / 387

C. filename that exists nowhere
   HTTP 200, {hook:64, empathy:53, share:90, cta:95, platform_fit:70}

Rewriting the draft moves nothing. Renaming it moves everything. That is the whole finding.

(One correction to the public write-up going around: the endpoint does 404 when a persona has none of its three files, so it is the content name that can be fictional, not the persona.)

After

The scores stay — the layout needs something to render, and a stable placeholder is better than an empty grid. What changes is that they stop claiming to be an evaluation:

  • the response carries simulated: true, scoring: "deterministic-hash", and a scoringNote naming /qa as the real path
  • the QA Simulation page renders a banner above the board when simulated is set
  • deterministicScore carries a comment saying it never opens the draft, so the next reader does not have to rediscover it

bun run build passes. tsc --noEmit reports the same errors as ralphthon, file for file (31 in PersonaDetail.tsx, 3 in VirtualFollower.tsx, all pre-existing) — this branch adds none.

Replacing the hash with a real evaluation is the actual fix; this is the honest label until then.

- `/qa-simulation` returned a 20-follower, five-dimension board and a TOP 5 ranking that read as an evaluation, but `deterministicScore` hashes persona, follower, dimension, and content *name* and never opens the draft: replacing a file's content leaves every score byte-identical while renaming it changes them, and a filename that exists nowhere still scores; the response now carries `simulated`, `scoring`, and `scoringNote`, and the QA Simulation page renders a banner pointing at `/qa` for a real virtual-follower evaluation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant