Skip to content

Detect low-confidence spans from token logprobs, live mode, release binaries - #2

Merged
Mattbusel merged 1 commit into
mainfrom
feat/logprob-detector
Sep 25, 2026
Merged

Mattbusel merged 1 commit into
mainfrom
feat/logprob-detector

Conversation

@Mattbusel

Copy link
Copy Markdown
Owner

Makes the tool deliver on the repo name, honestly.

What's new

  • --logprobs-file <path>: reads an OpenAI-style Chat Completions response (full response, its logprobs object, or a bare token array; - for stdin). Each token's probability is exp(logprob). Tokens are grouped into words; a word is flagged when any of its alphanumeric tokens is below --threshold (default 0.5). Adjacent flagged words merge into spans, and each span reports its weakest token plus the top_logprobs alternatives. Punctuation/whitespace never trigger, special tokens like <|eot_id|> are dropped.
  • --live "<prompt>": calls any OpenAI-compatible endpoint (OPENAI_API_KEY, optional OPENAI_BASE_URL, --model, --max-tokens, --save). Uses ureq behind a default live feature; --no-default-features builds offline-only.
  • --format json and --fail-on-flag (exit 2) for scripts and CI.
  • Offline samples: four real responses from meta-llama/Llama-3.1-8B-Instruct via the Hugging Face router (novita provider), temperature 0, top_logprobs: 3, fetched today. Not edited. cargo run --example detect runs over all of them.
  • Release workflow: tag v* builds x86_64 Linux, x86_64 + aarch64 macOS and x86_64 Windows binaries and attaches them with SHA256SUMS.
  • README rewritten around detection, with a "What it cannot tell you" section using the bundled moonwalk.json, where the model confidently names the wrong astronaut and the detector does not flag it. CONTRIBUTING now has real commands (it referenced Python tests that do not exist). CHANGELOG added, version 0.2.0, license = "MIT" in Cargo.toml plus a License section and badge.

Bug fix

TerminalRenderer::render printed its output to stdout as a side effect and also returned it, so a library caller of visualize_tokens(.., "terminal", ..) got unwanted output. The renderer now only returns the string; the CLI prints it.

Testing

  • 11 new unit tests in src/detect.rs (input shapes, error messages, subword grouping, merging, threshold edge, digits split by a space token, alternatives formatting, and a test against the bundled sample) plus one for the live request body.
  • Live mode was exercised end to end against a local HTTP server replaying a saved response (checked path, bearer header, logprobs: true, --save round trip). It was not run against api.openai.com (no key on this machine); the request format matches the one used to fetch the samples from the HF router.
  • CI now also runs clippy with --no-default-features and a detector smoke test.

…orkflow

Adds the detector the repo name promises: --logprobs-file reads a Chat
Completions response and flags words whose token probability is below
--threshold, reporting the alternatives the model weighed. --live calls any
OpenAI-compatible API using OPENAI_API_KEY. Bundles four real Llama 3.1 8B
responses so the demo runs offline, adds --format json and --fail-on-flag,
a release workflow for Linux/macOS/Windows binaries, MIT license metadata,
and fixes TerminalRenderer printing to stdout as a side effect.
@Mattbusel
Mattbusel merged commit b3b7dc9 into main Sep 25, 2026
1 check passed
@Mattbusel
Mattbusel deleted the feat/logprob-detector branch September 25, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant