Repository navigation
Detect low-confidence spans from token logprobs, live mode, release binaries - #2
Merged
Merged
Conversation
…orkflow Adds the detector the repo name promises: --logprobs-file reads a Chat Completions response and flags words whose token probability is below --threshold, reporting the alternatives the model weighed. --live calls any OpenAI-compatible API using OPENAI_API_KEY. Bundles four real Llama 3.1 8B responses so the demo runs offline, adds --format json and --fail-on-flag, a release workflow for Linux/macOS/Windows binaries, MIT license metadata, and fixes TerminalRenderer printing to stdout as a side effect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the tool deliver on the repo name, honestly.
What's new
--logprobs-file <path>: reads an OpenAI-style Chat Completions response (full response, itslogprobsobject, or a bare token array;-for stdin). Each token's probability isexp(logprob). Tokens are grouped into words; a word is flagged when any of its alphanumeric tokens is below--threshold(default 0.5). Adjacent flagged words merge into spans, and each span reports its weakest token plus thetop_logprobsalternatives. Punctuation/whitespace never trigger, special tokens like<|eot_id|>are dropped.--live "<prompt>": calls any OpenAI-compatible endpoint (OPENAI_API_KEY, optionalOPENAI_BASE_URL,--model,--max-tokens,--save). Usesureqbehind a defaultlivefeature;--no-default-featuresbuilds offline-only.--format jsonand--fail-on-flag(exit 2) for scripts and CI.meta-llama/Llama-3.1-8B-Instructvia the Hugging Face router (novita provider), temperature 0,top_logprobs: 3, fetched today. Not edited.cargo run --example detectruns over all of them.v*builds x86_64 Linux, x86_64 + aarch64 macOS and x86_64 Windows binaries and attaches them with SHA256SUMS.moonwalk.json, where the model confidently names the wrong astronaut and the detector does not flag it. CONTRIBUTING now has real commands (it referenced Python tests that do not exist). CHANGELOG added, version 0.2.0,license = "MIT"in Cargo.toml plus a License section and badge.Bug fix
TerminalRenderer::renderprinted its output to stdout as a side effect and also returned it, so a library caller ofvisualize_tokens(.., "terminal", ..)got unwanted output. The renderer now only returns the string; the CLI prints it.Testing
src/detect.rs(input shapes, error messages, subword grouping, merging, threshold edge, digits split by a space token, alternatives formatting, and a test against the bundled sample) plus one for the live request body.logprobs: true,--saveround trip). It was not run against api.openai.com (no key on this machine); the request format matches the one used to fetch the samples from the HF router.--no-default-featuresand a detector smoke test.