fix(qa): mark the dashboard's placeholder scores as simulated - #5
Open
sergiobuilds wants to merge 1 commit into
Open
sergiobuilds wants to merge 1 commit into
sergiobuilds wants to merge 1 commit into
Conversation
- `/qa-simulation` returned a 20-follower, five-dimension board and a TOP 5 ranking that read as an evaluation, but `deterministicScore` hashes persona, follower, dimension, and content *name* and never opens the draft: replacing a file's content leaves every score byte-identical while renaming it changes them, and a filename that exists nowhere still scores; the response now carries `simulated`, `scoring`, and `scoringNote`, and the QA Simulation page renders a banner pointing at `/qa` for a real virtual-follower evaluation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/api/personas/:id/qa-simulationreturns twenty followers, five dimensions each, a total and a TOP 5 ranking. It reads as an evaluation. It isdeterministicScore, a string hash over persona, follower, dimension, and the content name — the draft is never opened.The CLI path is the opposite:
virtual-followerruns a real LLM evaluation and even forces a Read on the saved report to confirm it exists. Two things named "QA score", one real and one a hash, and nothing in the type, the response, or the UI separates them.Measured
Rewriting the draft moves nothing. Renaming it moves everything. That is the whole finding.
(One correction to the public write-up going around: the endpoint does 404 when a persona has none of its three files, so it is the content name that can be fictional, not the persona.)
After
The scores stay — the layout needs something to render, and a stable placeholder is better than an empty grid. What changes is that they stop claiming to be an evaluation:
simulated: true,scoring: "deterministic-hash", and ascoringNotenaming/qaas the real pathsimulatedis setdeterministicScorecarries a comment saying it never opens the draft, so the next reader does not have to rediscover itbun run buildpasses.tsc --noEmitreports the same errors asralphthon, file for file (31 inPersonaDetail.tsx, 3 inVirtualFollower.tsx, all pre-existing) — this branch adds none.Replacing the hash with a real evaluation is the actual fix; this is the honest label until then.