Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
bin/
.git/
*.jsonl
*.test
coverage.*
26 changes: 26 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,29 @@ jobs:
# gofmt, go vet, go test -race, govulncheck and a build (see Makefile)
- name: Run every check
run: make check

container:
name: Container image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

# Builds the deployment image and runs it the way Fly.io will: as the
# non-root user, writing the audit log to a volume at /data.
- name: Build and smoke-test the image
run: |
docker build --build-arg VERSION=ci -t discreet:ci .
test "$(docker inspect --format '{{.Config.User}}' discreet:ci)" = "nonroot:nonroot"
docker volume create discreet-data
docker run -d --name discreet -p 8080:8080 -v discreet-data:/data \
-e DISCREET_HMAC_KEY="$(openssl rand -hex 32)" discreet:ci
curl -fsS --retry 10 --retry-delay 1 --retry-all-errors http://localhost:8080/healthz
out=$(curl -fsS http://localhost:8080/demo/run -H 'Content-Type: application/json' \
-d '{"purpose":"claims summary","text":"Member S1234567D on 9123 4567"}')
printf '%s' "$out" | grep -q '"sent_to_model":"Member <NRIC_1> on <PHONE_1>"'
docker exec discreet /discreet audit verify -log /data/discreet-audit.jsonl
docker logs discreet 2>&1 | grep -vq 'S1234567D'

59 changes: 59 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Deploy demo

# Deploys the public mock-mode demo to Fly.io from main only, after
# Michael's merge. Does nothing until the FLY_API_TOKEN secret exists.
on:
push:
branches:
- main
workflow_dispatch:

permissions:
contents: read

concurrency:
group: deploy
cancel-in-progress: false

jobs:
deploy:
name: Deploy to Fly.io
runs-on: ubuntu-latest
env:
FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
steps:
- name: Skip without a Fly.io token
id: gate
run: |
if [ -z "$FLY_API_TOKEN" ]; then
echo "::notice::FLY_API_TOKEN is not set; skipping deployment (see docs/DEPLOY.md)."
echo "run=false" >> "$GITHUB_OUTPUT"
else
echo "run=true" >> "$GITHUB_OUTPUT"
fi

- name: Checkout
if: steps.gate.outputs.run == 'true'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Set up flyctl
if: steps.gate.outputs.run == 'true'
uses: superfly/flyctl-actions/setup-flyctl@fc53c09e1bc3be6f54706524e3b82c4f462f77be # 1.5

- name: Deploy
if: steps.gate.outputs.run == 'true'
run: flyctl deploy --remote-only --build-arg VERSION="${GITHUB_SHA::7}"

- name: Smoke test the live demo
if: steps.gate.outputs.run == 'true'
run: |
app=$(sed -n 's/^app = "\(.*\)"/\1/p' fly.toml)
url="https://$app.fly.dev"
curl -fsS --retry 5 --retry-delay 5 --retry-all-errors "$url/healthz"
out=$(curl -fsS "$url/demo/run" -H 'Content-Type: application/json' \
-d '{"purpose":"claims summary","text":"Member S1234567D on 9123 4567"}')
printf '%s' "$out" | grep -q '"sent_to_model":"Member <NRIC_1> on <PHONE_1>"'
printf '%s' "$out" | grep -q '"upstream":"mock"'
echo "Live demo OK: $url"
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ All notable changes to this project are documented here. The format follows [Kee
- Evidence files: `CONTROLS.md`, `THREAT_MODEL.md`, `EVALS.md`, `RISKS.md`, `docs/HOW-THIS-WAS-BUILT.md` and the first two decision records.
- Vetted reviews every pull request (shadow mode until 2026-10-14).
- Detection engine (`internal/detect`, issue D3): implements the sg-pii-rules specification with Go's `regexp` (RE2), including the five named validators (NRIC/FIN, NRIC look-alikes, payment cards, postal sectors, calendar dates), `value` groups and the overlap rule. Passes all 369 shared conformance cases.
- Deployment (issue D10): a distroless, non-root container image pinned by digest and smoke-tested in CI on every pull request; `fly.toml` for a mock-only public demo in Singapore that sleeps when idle; a deploy workflow from `main` that waits for the `FLY_API_TOKEN` secret and smoke-tests the live URL; `docs/DEPLOY.md` with the one-time steps only Michael can do (ADR 0009).
- Benchmark (issue D9, `make bench`): 5,000 synthetic samples from a generator written independently of the rules, scored per entity and per writing style, plus latency against a baseline. Results in `EVALS.md`: 76.5% recall overall (100% on every supported style), 92.3% precision (all false alarms are bare eight-digit numbers as `PHONE`), about 3 ms added per request, almost all of it the audit write flushed to disk.
- Four-pane demo page (issue D8) at `/`: what the app sent, what the AI saw, what it answered, and what the app got back, plus that request's audit record. Five made-up scenarios (claims, appointment, card dispute, mistyped NRIC, a refused eligibility decision). It runs through the same pipeline as the API, always on the mock model, even for token holders; strict Content-Security-Policy; works on phones, light and dark.
- A real model behind access tokens (issue D6): an OpenAI-compatible client (OpenAI, or Ollama with `max_tokens`) used only for callers with a valid `DISCREET_ACCESS_TOKENS` token; everyone else gets the mock. One pinned model with checked prices (`gpt-6-luna`); each request reserves its worst-case cost (at most US$0.0076) against a hard daily budget (default US$0.20) recomputed from the audit log; 30 requests a minute per client address (ADR 0007). No test calls a paid API.
Expand Down
19 changes: 19 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Discreet's container: a static Go binary on Google's distroless image,
# running as a non-root user. Base images are pinned by digest.
FROM golang:1.27.1-trixie@sha256:3b77fc618ec235a1ab412de7737f120dd507c57e8d87de4cbb7994fb94275ed5 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=0.0.0-dev
RUN CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=${VERSION}" -o /out/discreet ./cmd/discreet \
&& mkdir -p /out/data

FROM gcr.io/distroless/static-debian13:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3
COPY --from=build /out/discreet /discreet
# /data holds the audit log; owned by the non-root user so it can write.
COPY --from=build --chown=nonroot:nonroot /out/data /data
USER nonroot:nonroot
ENV DISCREET_ADDR=:8080 DISCREET_AUDIT_LOG=/data/discreet-audit.jsonl
EXPOSE 8080
ENTRYPOINT ["/discreet", "serve"]
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,23 @@ make check # gofmt, go vet, go test -race, govulncheck, build

## Try the demo

**Live (once deployed):** <https://discreet-demo.fly.dev>. Mock model only; see [`docs/DEPLOY.md`](docs/DEPLOY.md).

**Three-minute demo path:** open the page, press *Load a made-up example* to cycle through five scenarios, and compare pane 1 (what the app sent) with pane 2 (what the AI saw). The fifth scenario asks for an automated eligibility decision: it's refused before any model is called, and the audit record shows why.

**Five-minute quickstart, on a clean machine with Docker:**

```sh
git clone https://github.com/makoydev/discreet && cd discreet
docker build -t discreet .
docker run -p 8080:8080 -e DISCREET_HMAC_KEY="$(openssl rand -hex 32)" discreet
# open http://localhost:8080/
```

Or with Go 1.27: `make check && ./bin/discreet serve` after setting `DISCREET_HMAC_KEY`.

### Running from source

Run `discreet serve` (below) and open <http://localhost:8080/>: four panes show what the app sent, what the AI saw, what it answered and what came back, with the audit record. It always uses the free mock model.

## Run it
Expand Down
2 changes: 2 additions & 0 deletions THREAT_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,6 @@
| The public demo is used to spend money | Mock model unless a valid access token is present (a wrong or missing token gets the mock, test); worst-case cost reserved against a hard daily budget that survives restarts (tests, including 100 parallel requests); 30 requests a minute per address (ADR 0007) | D6 (done), D10 |
| An access token leaks from memory or logs | Only SHA-256 hashes kept, compared in constant time; tokens and provider keys never logged; the caller's token is never forwarded | D6 (done) |
| The demo page is used to inject script or exfiltrate what people type | Strict Content-Security-Policy (own scripts only, connections only to this server), user text inserted as text never HTML, no third-party scripts; always the mock model (test) | D8 (done) |
| The public demo is reconfigured to reach a paid model | `fly.toml` never sets an upstream; `docs/DEPLOY.md` forbids setting one as a secret; without an upstream every request gets the mock (ADR 0007, ADR 0009) | D10 (done) |
| The container is used to escalate on the host | Distroless image (no shell), non-root user, static binary, images pinned by digest, built and smoke-tested in CI | D10 (done) |
| A compromised dependency or Action | Standard library only (ADR 0001), SHA-pinned Actions, Dependabot, `govulncheck`, CodeQL | D1 |
49 changes: 49 additions & 0 deletions docs/DEPLOY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Deploying the public demo (Fly.io)

The public demo runs **mock-only**: `fly.toml` never sets `DISCREET_UPSTREAM_URL`, so nothing anyone sends can reach a paid model (ADR 0007, ADR 0009). It sleeps when idle, so it costs well under US$1 a month (one 256 MB shared machine plus a 1 GB volume).

Deployment happens from `main` only, by `.github/workflows/deploy.yml`, after Michael merges `next` → `main`. Until the `FLY_API_TOKEN` secret exists, the workflow skips itself.

## One-time setup (Michael, about 10 minutes)

These steps need your account and card, so only you can do them. Run them from the `discreet` folder.

1. Install the Fly.io command-line tool and sign in. Sign-up asks for a card.
```sh
brew install flyctl
fly auth signup # or: fly auth login
```
2. In the Fly.io dashboard (Billing), turn on a usage alert so you're emailed before any surprise.
3. Create the app. If the name `discreet-demo` is taken, choose another and change `app` in `fly.toml` to match.
```sh
fly apps create discreet-demo
```
4. Create the 1 GB volume for the audit log, in Singapore:
```sh
fly volumes create discreet_data --region sin --size 1 -a discreet-demo
```
5. Set the audit log's HMAC key. It's generated and sent straight to Fly; it never appears on screen or in the repository.
```sh
fly secrets set DISCREET_HMAC_KEY="$(openssl rand -hex 32)" -a discreet-demo
```
6. Create a deploy token for this app only, and store it as a GitHub secret. Paste the token when `gh` asks for it.
```sh
fly tokens create deploy -a discreet-demo --expiry 2160h # 90 days
gh secret set FLY_API_TOKEN -R makoydev/discreet
```
7. Deploy: either merge `next` → `main`, or run the workflow by hand:
```sh
gh workflow run deploy.yml -R makoydev/discreet
```
The workflow finishes with a smoke test against `https://discreet-demo.fly.dev`: the page answers, the model saw only placeholders, and the model was the mock.

## Never do this on the public app

- Don't set `DISCREET_UPSTREAM_URL` or `DISCREET_UPSTREAM_API_KEY` with `fly secrets set`. The public demo must never reach a paid model.
- Don't share `DISCREET_HMAC_KEY`. Without it, nobody can test whether a guessed NRIC matches a fingerprint in the audit log.

## Checking the audit log on the server

```sh
fly ssh console -a discreet-demo -C "/discreet audit verify -log /data/discreet-audit.jsonl"
```
4 changes: 4 additions & 0 deletions docs/HOW-THIS-WAS-BUILT.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,16 @@
| 2026-10-05 | Demo page in headless Chrome at 1280 px (light and dark) and 390 px, clicking through the examples | No Content-Security-Policy violations or script errors; one 404 for the browser's default icon request, fixed with an inline empty icon; audit log verified afterwards (7 records) |
| 2026-10-05 | Benchmark (5,000 samples) and latency (2,000 requests per run) | Results in EVALS.md. A suspicion that the audit flush dominated latency was measured rather than assumed: 2.9 ms of the 3.0 ms |
| 2026-10-05 | Deliberate break: the generator ignoring its seed | The determinism test failed; restored from backup |
| 2026-10-05 | Fly.io `fly.toml` keys (docs.fly.io, primary); volume ownership for non-root images (Fly community forum, secondary); base image digests from Docker Hub and gcr.io; flyctl action v1.5 | Config written to the documented keys; ownership to be confirmed by the first deploy's smoke test |
| 2026-10-05 | The container image built and run locally the way Fly runs it (Docker started by Michael): non-root, audit log on a volume | 9.97 MB image; demo answered with placeholders only; `audit verify` passed inside the container; no planted value in its logs |
| 2026-10-05 | Five deliberate breaks of the engine and vendored rules (EVALS.md) | Four caught at once. The reversed overlap tie-break was not: no shared case covers it. Added toy-detector overlap tests (now 3 fail on that break) and opened vetted#44 for the shared suite |

## What the AI got wrong, and how it was caught

| Date | What went wrong | How it was caught | Fix |
| --- | --- | --- | --- |
| 2026-10-05 | The merge helper treated "no check pending" as "done" while CI hadn't started yet, saw only the fast Vetted check, and stopped waiting | It then refused to merge because a required check hadn't passed, the safe outcome | The helper now waits until every check required on `next` has appeared and finished |
| 2026-10-05 | A local smoke test piped JSON through zsh's `echo`, which turns `\\n` escapes into real line breaks, so a correct response looked like invalid JSON | The parse error, checked against the raw response before suspecting Discreet | Saved the response to a file instead; the CI and deploy workflows use `printf '%s'` so their checks don't depend on the shell's `echo` |
| 2026-10-05 | A deliberate break of the access-token check left two variables unused, so Go refused to compile, and the filter used to list failing tests showed nothing, which looked like an uncaught break. The same class of mistake as sg-pii-rules' unbalanced-bracket break earlier that day | Noticing that no test was listed at all, then checking the build | Redid the break so it compiles (any non-empty token matches): the token test fails. Breaks now run `go vet` first, so a build failure can't pass for a result |
| 2026-10-05 | The mock model's reply said "Here is what I received, exactly as a real model would see it", but readers see that text after Discreet restores the real values, so it showed them the opposite of what the model saw | Reading the reply returned to OpenAI's Python client during the compatibility check | Reworded: the mock says it only ever saw placeholders and that Discreet put the real details back |
| 2026-10-05 | A gateway test searched JSON-encoded messages for `<NRIC_1>`, but Go's encoder writes `<` as `\u003c`, so a correct gateway failed the test | The test failure itself, read before changing any gateway code | Fixed the test to read message text directly; the gateway now keeps `<` and `>` unescaped in responses so placeholders stay readable |
21 changes: 21 additions & 0 deletions docs/adr/0009-public-demo-on-fly.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# 0009. The public demo runs mock-only on one small Fly.io machine

Status: drafted by Claude Code, awaiting Michael's review
Date: 2026-10-05

## Context

The Milestone 2 plan (approved 2026-10-02) chose Fly.io: pay-as-you-go, with machines that stop when idle (about US$0.50 a month for the smallest, per Fly's pricing page). The brief requires a public deployment in mock mode, and public endpoints never relay to a paid model.

## Decision

- **Image:** a static Go binary (`CGO_ENABLED=0`) on Google's distroless `static-debian13:nonroot`, no shell or package manager, running as a non-root user. Both base images are pinned by digest. A CI job builds the image on every pull request and smoke-tests it the way Fly runs it: non-root, audit log on a volume at `/data`, `audit verify` passing, no planted value in the logs.
- **Fly:** region `sin` (Singapore), one shared-CPU 256 MB machine, `auto_stop_machines = "stop"` and `min_machines_running = 0`, a 1 GB volume for the audit log, a health check on `/healthz`, and the client address taken from `Fly-Client-IP` for rate limiting.
- **Mock only:** `fly.toml` never sets `DISCREET_UPSTREAM_URL`; `docs/DEPLOY.md` says never to set it as a secret on the public app.
- **Deploys** run from `main` only, through a pinned workflow that skips itself without `FLY_API_TOKEN` and ends with a smoke test against the live URL.

## Consequences

- The first request after idling waits a few seconds while the machine starts.
- Fly gives the volume to the image's user, according to Fly staff on the community forum (secondary source); the first deployment's smoke test confirms it, because the demo can't answer without writing its audit record.
- Fly can only be set up by Michael (account, card, token); until then the demo runs locally or in Docker.
35 changes: 35 additions & 0 deletions fly.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Public demo of Discreet on Fly.io (issue D10, ADR 0009). Mock model only:
# DISCREET_UPSTREAM_URL is deliberately never set here, so the public app
# cannot reach a paid model whatever anyone sends it.
app = "discreet-demo"
primary_region = "sin"

[env]
DISCREET_ADDR = ":8080"
DISCREET_AUDIT_LOG = "/data/discreet-audit.jsonl"
DISCREET_CLIENT_IP_HEADER = "Fly-Client-IP"
DISCREET_RATE_LIMIT_PER_MINUTE = "30"

[http_service]
internal_port = 8080
force_https = true
auto_stop_machines = "stop"
auto_start_machines = true
min_machines_running = 0

[[http_service.checks]]
grace_period = "10s"
interval = "30s"
method = "GET"
path = "/healthz"
timeout = "5s"

[mounts]
source = "discreet_data"
destination = "/data"
initial_size = "1gb"

[[vm]]
memory = "256mb"
cpu_kind = "shared"
cpus = 1
Loading