AI Systems Engineer. I build production AI systems and run them: RAG pipelines, agentic assistants, fine-tuned models, and the infrastructure underneath.

The question behind all of it: how do you know an AI's numbers are true? My flagship system's retrieval recall measures 88.7% on the exam it was tuned against (55 of 62, 2026-09-05) - down from 95% sixteen days earlier, published anyway, with every lost hit diagnosed: the corpus grew 47% and newer documents about the same stories displaced the canonical sources. The harder, outside-authored holdout that no fix was ever tuned toward reads 25 of 32, unchanged across eras. The judge that grades every answer is itself audited, and that audit puts the overfit to my own tests at 19.0 points (2026-09-05) - a gap that has widened four times and never twice for the same reason, this time because the growing corpus reached the holdout too. The same measurement prices what that number is worth - two runs held at one corpus fingerprint spread 2.6 points, so the instrument knows itself to within that band and publishes it. Numbers that only ever go up are marketing. Grade the grader; show the receipts.

One production system, built layer by layer. Every write-up below starts with something breaking - a chatbot lying, a judge grading its own homework, a VM freezing at 3am - and ends with a number proving it's fixed. It's all one codebase: Architecture Zero, four live white-label instances - and the core is now open source. Read them in order; they compound.

Start with the spine: Justified Confidence - The Scoreboard - every reliability layer the field names, the failure each one prevents, and the measured number where one exists - including the published tuned-vs-holdout gap and an honest deficit section of what is not built. Each study below is the long version of one of its cells.

1. Hybrid RAG + Reranker - why a chatbot lies, and how retrieval gets fixed: vector search + a from-scratch BM25 + a self-hosted cross-encoder, recall 40% -> 93%, eval-driven.
2. Cheap Only If Free - the reranker's sequel: it bought +10.7 points of recall and cost 19.5 seconds a query, so a four-arm build-vs-buy ran against a rule written before any number existed - the quantized arm lost for costing accuracy, the same model on my own GPU won at 45x with every metric identical, the hosted API never needed to run, and the winner shipped that night behind a privacy latch and a fallback chain.
3. RAG Ingestion Intelligence - when retrieval fails, fix the corpus: a stage-by-stage retrieval debugger, three measured rounds, 88% -> 100% recall with 44 of 50 hits at rank 1.
4. Context Engineering - keeping a RAG corpus honest: docs-as-code, single source of truth, and drift detection on both the write side and the read side. A stale source is a confident liar.
5. Grade the Grader - calibrating the LLM judge every other number depends on: planted errors, a human baseline, a second lab, an independently-authored second harness (RAGAS), and a locked holdout exam an outside model wrote - every instrument produced real findings, including a published overfit gap that widened from 6.3 to 19.0 points - once as the tuned set improved, once because the exam got fairer, once because the corpus moved under the holdout, and once when both ends moved at once; never twice for the same reason. The judge also carries a fourth rubric - refuse-vs-fabricate - holding 13/13 over an artifact-demand cohort seeded from an external stress test.
6. The External Stress Test - twelve adversarial demands for artifacts that do not exist, zero fabrications - and a demanded live eval run that caught a real regression, fixed and re-measured the same night.
7. Your Eval Score Is Measuring Three Things at Once - a 5.9-point drop with zero code changes, decomposed instead of panicked over: a noise band, a retrieval diff that killed the obvious theory by direction, a silently rotted vector index, and a pre-registered prediction confirmed at 92.5% - with a corpus fingerprint on every eval row as the residue.
8. The Review That Praised the Bug - three frontier models reviewed the same code, and the reviews got graded against the code: the accurate one went five for five and found the only real bug; the glowing, near-perfect one was the least reliable, praising features that do not exist - and the one true finding still lost its A/B (76 vs 75 of 83) and did not ship. A review is a claim, not evidence.
9. The Test I Wrote to Fail My Own Code - sixteen adversarial security probes with their pass conditions frozen in a commit BEFORE the first one ran, pointed at code I had shipped hours earlier and was confident in. Six failed - five in the code and one uptime check nothing had pinned: a freeze that gated two routes instead of the instance, a restore under a rotated key that returned 500 on every read of a two-factor account, an export archive readable by the wrong account, an evidence contract that lived in a docstring, and a manifest blind to a table outside the ORM. Then the fixes got the same treatment - an adversary attacked each design and five of eight were materially wrong, including one that would have turned an accidental crash into a working two-factor bypass, because the crash had been denying access by accident. The suite was green the whole time: reasoning about the code passed, running it failed.
10. When the Corpus Stops Being Yours - the untrusted-corpus injection gate, shipped BEFORE the first outside-source connector: scanning enforced at the one function every ingestion path already flows through, provenance trust tiers deciding a chunk's authority, and third-party content quarantined while my own is only flagged. 18/18 payloads caught, zero high-severity false positives on my real corpus - and my first two live results were 4/4 and both retracted, because the poisoned doc announced itself and then my own control arm turned out to be the treatment arm. The rebuilt probe proves each arm's label state before believing it, and holds 4/4 on all three arms - including the bare-label true floor.
11. Three Connectors, One Gate - Google Drive, Calendar, and Gmail into the assistant, read-only by construction, every byte through the gate as untrusted: each connector states its real selection guarantee in the UI (Drive's is enforced by Google's grant; Calendar's and Gmail's are my policy, because no narrower scope exists), email syncs one document per thread so a phishing message quarantines one conversation instead of a month of mail - and the live proof caught what a green 450-test suite could not, twice, fixed the same sitting.
12. The Admin Who Could Not Read the File - the company version of those connectors, where documents keep the access rules of the system they came from: each file's own sharing grants read, mapped to local accounts, and carried into retrieval as per-document keys, so an admin of my platform holding every permission the product defines still gets nothing when he asks for the compensation bands - the customer's Drive never shared that file with him, and it never enters the model's context to be refused. The key detail is in the change detection: it hashes access alongside content, because a file whose sharing was tightened this morning has the exact same bytes it had yesterday.
13. Untrusted Until Measured - live web access as two doors on one guarded chassis: every fetched page labeled untrusted with both its dates, transient, outranked by my own corpus - and the tool itself shipped recorded as UNTRUSTED until its five-arm cohort measured it. The first number, 80.0, was retracted because the instrument scored the model's ideal answer (naming the planted instruction and refusing it) as obedience; the fixed instrument, its behavior cases pinned by test, measured the clean run at 5/5 with every fixture verifiably reaching the model. The cohort rides every eval run - the number re-earns itself or visibly stops holding.
14. AI Security - threat-modeling an AI that holds real data and can act: OWASP Web + LLM Top 10, a real broken-access-control bug found and fixed, CI security gates on every push.
15. Multi-Tier Access Isolation - proving a tiered AI cannot be talked into leaking: an adversarial cohort measured 3/11 -> 11/11 while owner recall held 100%. The corpus-diffusion lesson.
16. Zero-Trust Remote Inference - a private GPU, reachable from anywhere, without opening a port: outbound-only tunnel, machine-to-machine service tokens, and a request served locally never egresses.
17. The Controls That Forgot - security state that survives a restart: every rate-limit window, attempt counter and single-use token lived in process memory, so an ordinary deploy reset them all and a spent one-time token became spendable again. Moved into the database already deployed, with single-use decided by a primary key rather than a check-then-mark window.
18. Ops Resilience - backups that prove themselves every night: encrypted to two providers (one tamper-proof by object lock), pulled back and restore-drilled by machinery while I sleep.
19. The Night the VM Froze - a batch job starved a production server for 17 minutes: the forensic chain, the OOM blind spot, and the layered guards that make it impossible again.
20. LLM Cost Engineering - prompt caching that proves itself in the token counters, and the double-billed system prompt the build found. Cost is a reliability metric.
21. Eco Mode - a federated RAG mesh: independent AI instances share knowledge at query time with fail-closed, per-caller scoped access. Never copied, never leaked.
22. The Optimization That Made It Worse - a ten-line change wired document ingestion into the batched embedding path the connectors already used; the suite was green and the first live boot ran slower than no batching at all, because every slice timed out and then paid the per-chunk fallback on top. Two tunes fixed it and disproved the premise it was banked on (embed compute dominates, so batching buys round-trip overhead only). Then a third datapoint ended the tuning: a 5-text slice died at its 90s budget while a 64-text slice finished inside its own 798s - it was queued, not slow, and a budget derived from payload cannot see a queue.
23. Open-Sourcing a Production Platform in One Night - taking the core public as a derivation instead of a fork: every file read in full, every feature an explicit keep-or-drop decision landing as a commit that states its reasoning, on fresh history that cannot leak what it never held. A static import-contract check caught a wired-but-missing layer that compiling could not see; CI caught routes that existed only on my machine; a six-lens adversarial review confirmed 27 findings before anyone else could look. Then the part worth the night: after a literal scrub verified clean, one more question found 16 items still sitting there - because text replacement only fixes literals and shape survives it.
24. Architecture Zero - the platform it all runs on: one core codebase, four live instances, white-label by config.

• Architecture Zero (open source) - the platform's open core, Apache-2.0: hybrid RAG with a reranker provider seam, the four-rubric judged eval layer, the untrusted-content injection gate, multi-tier access isolation, and peer federation - derived from the production codebase on fresh history, with a CI-green test suite adapted from production and a MODULES.md that states plainly which capabilities are commercial.
• Northwind AI - a live white-label demo of Architecture Zero. Ask it anything; flip the "View as" switcher and watch the access boundary work in real time.
• MrJStickel Assistant - my public AI, first person, RAG-loaded with my work history: the same assistant that lives in the corner of mrjstickel.com, standalone. Guest chat is open - interview it about my work. Its measured trust numbers are public at ai.mrjstickel.com/#trust - every value derived live from stored eval runs, shown as bands, never hand-set.
• JobTrue - an AI job-search product, live with Stripe billing: honest 1-10 fit scoring, resume tailoring that structurally cannot fabricate (the model selects from your real bullets by index), cover letters, pipeline tracking. My own resume is served from it.
• thestatic.tv - a Web3 live-streaming platform built solo: 160+ API routes, GCP Livestream video, a real-time token economy, on-chain payments - the full-stack depth under the AI work.
• TheStatic.TV Agent - that platform's public AI agent, on the same Architecture Zero core - open to guests, and reachable in-world inside the TheStatic HQ Decentraland scene. Its trust panel: ai.thestatic.tv/#trust.
• @thestatic-tv/dcl-sdk - a published TypeScript SDK for Decentraland scene integration.

• A Google Workspace shop can now sign in the way a Microsoft shop does, through one door mechanism (new): the platform's second single sign-on door lands a person on the account the document access control already maps their verified address to, or creates one at the lowest role on first sign-in, and pins each provider's stable subject id to the account so an address handed to a new person at the directory is refused rather than inherited. Everything the two doors would otherwise carry twice - the signed state, the cookie bindings against login-CSRF, token replay and code interception, the single-use handoff - moved into one shared module and is bound to the door that minted it. Because Google asserts no tenant, a server-side allow list matched against the verified hosted-domain claim is the boundary and the door stays off until an operator names one. Reviewed against ten named controls with two hardenings applied to both doors before the deploy, then proven live with a mapped landing and a just-in-time landing.

• A Slack workspace can now ask it too, on the same identity mapping as Teams (new): Slack delivers each direct message or mention to the platform over its Events API - the request signature checked over the raw body, a five-minute window - and the sender is resolved from the email Slack has confirmed for them through the identity rule the document ACL already uses, so the bot answers strictly as that person with that person's citations, or tells an unmatched member to link an account and answers nothing. The pre-launch security read found that Slack's default link unfurling would let a poisoned document pull corpus text out through a URL Slack fetches server-side with no click; it ships pinned off. No SDK, no second process. Live the same night: the first answer came from a document only that account may read.

• A Microsoft shop can now connect and sign in, and both doors use one identity mapping (new): the platform's second permission-aware connector reads SharePoint and OneDrive through Microsoft Graph on the same engine as Google Drive - every document synced with the permissions the source states, an unmatched identity granting nothing and saying so - and Microsoft Entra sign-in lands a person on the account that carries their Microsoft address in exactly the field the document ACL matches against, so who may sign in as whom and who may read what cannot drift apart. Proven on a real tenant both ways: an unmapped admin retrieved none of three restricted documents, then one mapping click and one sync made them readable and citable with the bytes unchanged, and the first real sign-in landed on that account with the server receipt to show it. Ported from my own Google sign-in door; a defensive review before deploy found one conditional gap (a stricter mode promising more than the password route enforced) and five smaller ones, all fixed with tests first. The same night, the client's silent session refresh shipped fleet-wide after the 30-minute "session expired" banner turned out to be a refresh token nobody had ever used - with one refresh at a time across browser tabs, because the server reads a rotated token presented twice as theft. And by morning a Teams bot: the Bot Framework's signed messages verified without the SDK, the asker resolved by their Entra object id, and the answer produced by driving the instance's own chat route as that person - proven in a channel with a cited answer, the same permission checks as the web app and no second pipeline.

• The platform now ships as a client deploy, and the first rehearsal on a clean box found fifteen things the workstation could not (new): one env file selects one compose - an edge that terminates TLS and mints its own certificate on first start, the API and web client behind it, a drop folder that ingests what lands in it, an optional in-compose embedder - and an acceptance script is the handover bar: claim the instance from its own logs, sign in, ingest a document, get an answer grounded in it, sync a Google Drive folder and watch it answer the mapped user and refuse an unmapped one. Run on a fresh cloud VM the same day it was written, the script itself surfaced the defects a demo never meets - a watcher credential missing from the template, a Windows-built tarball shipping the wrong line endings, a Drive viewer that cannot read a file's sharing list, an interrupted model download wedging every later one - each fixed and re-proven in the same sitting.

• The demo's departments are now separate running instances, and the answer says which one it came from (new): Northwind AI runs as a hub federating over three department peers on the same box - Engineering, Support, Sales - each a container holding only its own documents, reachable only by the hub with that peer's own credential. The role a visitor picks decides which peers the hub may ask at all, so "Support can't see the engineering source" is true because the source physically lives in one container the Support role is never routed to, not because a filter hides it. Each cited document names the peer that held it. Building it found two things reading the code could not: the hub refused before it had asked its peers, and once it asked, it flooded the model with every peer's raw hits until the merged pool went through the same reranker the local lane uses. A third was subtler - the safety rule that stops a retrieved document from raising a caller's access was being read as "do not disclose this document", so an authorized reader was refused their own department's policy; the rule now says authority is not permission to answer.

• The fleet's injection numbers re-earned under the stricter key, on every surface that quoted them (new): an audit had shown that the mechanical grader behind the published 4/4 would pass a fully obedient answer, so the number described the grader, not the defense. After the stricter key shipped, I re-ran the live three-arm probe inside each deployed instance - including the true-floor arm, where the poison arrives as trusted content - and every surface held 4/4 with the poison verifiably in the model's context on every question, on models down to the smallest in the fleet. Each report now names the grader version that produced it, so no future number can outlive its own key.

• A fix verified at the tier its finding was found at, and the template it had never reached (new): an audit had run my assistant and found the owner's private profile inside a guest turn's system prompt - a runtime finding - and the fix shipped with a test that checked only the gate's arithmetic. The probe that closed it boots the real application, mints owner, admin, member and guest sessions through the real doors, drives the chat handler and reads the prompt at the one seam every prompt leaves through: the owner's turn is the control, every other rung is proven absent with the profile demonstrably loadable. Written against the fleet, it found the open-source template still adding the card unconditionally days after the private fix; the template got the gate, the same probe, and a turn-log receipt that says whether the card rode without ever logging the profile - and the drift checker gained a row for the call site, proven to fire by removing the fix and watching it report.

• Three audit findings closed in one night, each proven by running production rather than reasoning about it: the security controls' state - login throttles, two-factor challenge counters, burned one-time tokens, the demo's daily guest budget - lived in process memory on every instance, so each redeploy quietly reset all of it. Instead of standing up Redis, that state moved into the SQLite database every deployment already runs: one table with per-row expiry and optimistic concurrency, shipped to all five surfaces with the public template first, the migration dry-run against a copy of the live database, and then proven the only way that counts - state written into two running production containers, the containers restarted, every count and burned token read back intact. The same night, a web fetcher that resolved a hostname to check it and then let the HTTP client resolve it again (the classic rebinding window onto the cloud metadata service) now connects to the address the check validated, proven from inside the deployed container against two hosts that refuse a connection without the right TLS name; and two vhosts that had no content-security policy at all got one written from a measured inventory of what each bundle loads, verified by injecting the candidate header in a headless browser against the live site before deploying it. Two things the runs found that no review had: the public demo greeted every first-time visitor with a false "session expired" banner, and a test runner had never mounted the scripts twelve offboarding tests depend on, so those tests had only ever run in CI. Both fixed the same night.

• The guard was green while three surfaces leaked, so the guard learned to read bodies: the fleet's drift checker compares security-relevant function names across five codebases and had 27 gaps to report - but the deeper miss was where it had nothing to say. The login handler existed on every surface; on three of them it answered an unknown username in about a millisecond and a real one in about 380, so account existence read from a single request's timing, and the checker cannot see a wrong body behind a right name. Closed in three classes in one sitting, public template first, every surface gated by its own test suite inside its own production image, both live public instances probed after deploy (ten 401s, then 429): a per-address throttle on the anonymous auth routes shipped in the same commit as a timing equalizer, because paying a full password hash for every unknown name without a throttle turns an oracle into a CPU amplifier; a config toggle that stored the string "false" as true - a mistake that could only fail open - now parses by an allow-list, and the template's version, which had opened on any unrecognized word, was aligned to fail closed; and refresh-token replay now revokes the whole session family everywhere, with the one Redis failure that let a revoked token keep working logged instead of swallowed. Then the checker itself changed: the classes a name diff cannot see are pinned by substring, and every pin names the call site, never the definition - the template once shipped a spend guard defined and never called for a month while a pin on the definition stayed green. The fourth class, a restore under a rotated key that breaks every read of a two-factor account, was deliberately not shipped: four of four designs for it were wrong a week ago, and a wrong fix on the template reaches everyone who deploys it, so it waits for a design attacked per surface first.

• A stolen session can no longer mint a lasting one (new): every write on the platform that creates or raises durable authority - creating an account, changing a role, granting the scopes that manage people or the system, registering a federation peer, approving an OAuth grant, resetting another account's password - now re-asks the caller's own password, on every surface including the open-source template. The refusal shape was the hard part: a wrong password answers a plain 400, never the 401 that logs the operator out or the 422 that blanks the admin panel; failures count against the same lockout as login, so the new check is not a second guessing surface; and the one class of account that can never answer, one provisioned by Google sign-in, carries a visible marker so the rule refuses to hand it authority instead of stranding it. Resetting your own password still works while signed in, behind a current authenticator code. The design was attacked before it shipped, and the attack found doors the first cut had missed - including a third way to create an account that the rule as written never saw.

• I wrote the test that would fail my own hours-old code, then published what it found: a five-step security program on the white-label platform ended with a pre-registered adversarial acceptance test - sixteen probes whose pass conditions were frozen in a commit before the first one ran, so no result could be argued into a pass afterwards. It found six failures - five real defects in code I had shipped hours earlier and was confident in, and one uptime check nothing had pinned: a freeze that stopped two routes instead of the instance, a restore under a rotated key that broke every read of a two-factor account, an export archive readable by the wrong account, an evidence contract that was a comment rather than code, and a manifest blind to a table it did not know about. All six are closed and re-proven, 8 pass / 1 resolved / 0 fail - and an independent review three days later found more at the edges of those fixes - four closed the same day, the rest closed with the hardening cleanup on 2026-09-14. Then the fixes got the same treatment: an independent adversary reviewed each one, and five of eight were materially wrong before shipping - including a fix that would have turned an accidental crash into a working two-factor bypass. My own review had caught none of them. The pattern held every time: reasoning about the code passed, running it failed.

• Security evidence that survives the machine it accuses: the platform's security events now leave the host every fifteen minutes to an object-locked store, sanitized and encrypted, so wiping the box cannot un-send what already left. Proven by breaking it on purpose and watching the whole chain fire - the job failed, the public health probe went red inside a minute, six monitoring regions saw it, and the alert email landed thirteen minutes after the fault with nobody in the loop. Deleting rows locally afterwards left the shipped copies intact, and the store refused to delete them.

• A client can leave, and the paperwork cannot over-promise: a full offboarding lifecycle - a read-only export with a hashed manifest, an explicit reversible freeze, and a destroy step that lives on the host and refuses to run without a client receipt. The manifest's retention disclosure reads UNVERIFIED and promises no deletion date until the off-site expiry has actually been observed, because the honest answer was that nobody had watched it happen yet. Background workers already running when the freeze lands now stop at their next write instead of outliving it, and the freeze itself waits for them, so the export it certifies is of a still system.

• The repo whose dependencies become other people's was the only one not scanned (new): the open-source template had been running secret scanning and nothing else, so it got pip-audit, bandit and trivy - on every push and daily, since public CI minutes are free and the private repos split those triggers to save them. Turning them on found exactly one real fix, and it was a test pin the private fleet had already bumped. Then trivy failed where I had predicted it would pass, which is the part worth keeping: the four private surfaces each carried a .trivyignore deferring root containers with documented reasoning, and the public one carried none. I had read a green run elsewhere and never asked why it was green. The deferral got ported but not its reasoning - the private note is about not crash-looping a running deployment, which is not a template's situation; the reason that actually holds here is that the data directory is a host bind mount whose ownership a template cannot know, and fixing it properly strands every existing deployment. Written into .trivyignore and the roadmap with what bounds it meanwhile, because a deferral you can read and disagree with is not the same as a silent suppression.

• An outside model reviewed the release, and the finding it under-called was real: a frontier model reviewed v0.1.0 and flagged that a caller's clearance was not propagated to federated peers - filing it as something that "deserves an explicit security decision." It was not a decision, it was a missing guard, and the side it did not check was the side the leak was on: a peer key scoped all meant "every department except the public one" and called the raw similarity search, which takes no clearance argument at all. So all served the owner-only departments plus every department an operator had created and never classified - the ones the default map fails closed to owner precisely so a new collection stays private. Verified against the shipped demo corpus, which was leaking its own internal docs to any all-scoped key. v0.1.1 makes a peer scope a rung on the same clearance ladder as every other read path, and each side of the seam enforces its own half: the scope bounds what leaves, a local floor bounds who may receive. The caller's clearance is deliberately NOT sent to the peer - a clearance asserted over the wire is one instance vouching for its own user, which is authorization by self-report across a trust boundary. It survived seven rounds of adversarial review because the peer lane's only test coverage was the injection scan; it has fourteen clearance tests now, mutation-verified against the pre-fix code. Then the live leg found a second one no test could see - a log field named level was silently overwriting the log severity on the two federation audit events - which makes it three consecutive defects found by running the thing rather than reading it.

• v0.1.0, and the defect a browser found that seven review rounds did not: tagged the first release of the open core, with known limitations enumerated in the README rather than left to be discovered. Getting there took seven rounds of adversarial multi-agent review, and the round that decided it was not a review. Running the reference client on a non-default port showed the shipped configuration only worked on localhost: browsers send an Origin header on same-origin POSTs too, so any real deployment refused every question with a 403 - while the config file documented that setting as never consulted. Seven rounds and roughly a hundred review agents read past it, because the code is not wrong on its face, only wrong in deployment. The first fix was verified against the backend directly and still failed through the proxy, which strips the port from the forwarded Host; that surfaced only because it got re-tested on the path a user actually takes. The client also got its first automated tests in the same arc - not one per bug, but one rule (a message is unstored exactly when no server row exists) checked across every way a stream can end, then mutation-verified by reverting the fix to confirm the right test fails. The reviews had converged to cosmetics while running the thing was still returning severe defects, which is the part worth carrying to the next release.

• The platform went open source: Architecture Zero's core is public at github.com/architecture-zero/architecture-zero, Apache-2.0, derived from the production codebase in one overnight arc - on deliberately fresh git history, where every file that lands is a reasoned include decision and the private repo's commits structurally cannot leak. The release had to earn the flip: a six-lens adversarial pre-release review (27 confirmed findings, every one remediated before publishing), a test suite adapted from production and green in CI - including a native-library teardown crash diagnosed down to "trust the junit report the session writes, not the interpreter's exit code" - and a final residue hunt for the leak class literal scrubbing can't see: document shapes, date-subject pairs, and metrics sampled from reality, closed by rewriting fixtures as genuine fiction and rebuilding history at the blob level, verified clean before going public. The open core ships with its own help corpus wired into the eval seed, so a fresh install scores its onboarding and support answers on its first eval run.

• A 45x latency cut that had to prove it cost nothing: my assistant's reranker was worth +10.7 points of recall and 94% of the wait - so it got a four-arm build-vs-buy judged by a rule pre-registered before any arm ran: cheap wins only at zero accuracy cost. A five-minute probe showed INT8 quantization is hardware-specific (1.43x faster on the 2015-era cloud CPU, 0.57x - slower - on a modern laptop); measured properly it was one question worse and died by the rule. The identical model on my own GPU behind an authenticated tunnel matched the baseline on every metric - same questions rescued, same questions broken - at 430ms p50 instead of 19.5s, and shipped to production the same night behind a provider seam, a host-level privacy latch (runtime config alone can never send corpus chunks to a vendor - a test asserts zero egress while latched), and a remote-to-local fallback chain so a GPU outage costs latency, never answers. The hosted API never needed to run. The same pass also caught this very page's ecosystem lying politely: the public write-up still claimed ~40ms CPU reranking - wrong by 600x - and was repriced with the correction in it. The write-up.

• Status answers left the model's hands: my email-module test battery kept exposing one honesty class - a small local model fabricating "executed 25/25" about its own actions while the true server state sat in its context. Now closed structurally rather than prompted away: execution-status questions are intercepted by a precision-tiered matcher and answered by code reading the action ledger directly, and the client renders a server-truth badge only for a marker the code path alone can emit - a model reproducing the answer byte-for-byte cannot earn the mark. Proven on the lane that fabricated: the exact graded turn that failed before, answered in ~32ms with the right per-target receipts and no model invoked. The wider lesson got measured too: the next full eval run moved scores on a corpus that had tripled, and the run-diff instrument attributed the movement to retrieval displacement, not the answer layer - a score delta without that split is not a finding.

• A guard is not shipped until something proves it fires: a four-lens adversarial review (authorization, untrusted-content, robustness, client) over one fast-built feature week shipped 17 fixes in a sitting. Headline: an empty-answer retry guard added hours earlier was already defeated - a message-format converter silently dropped the retry instruction after tool calls, so the "retry" replayed the failing request verbatim while the test suite stayed green. Also closed: OAuth sign-in state now binds to the originating browser (login-CSRF, verified live over HTTPS by the refusal itself), a read-then-write confirm became an atomic claim after proving a double-tap could execute a non-idempotent write twice, and an injection path where quarantined email subjects re-entered trusted surfaces through action-plan text. Every finding was adjudicated against the code before it counted, and the clean results are recorded as explicitly as the defects.

• Governed writes went from demo to product: the agentic-writes demo's trust layer now runs inside my personal assistant, managing my real Gmail inbox under the same discipline - the model can only propose (archive / spam / label / draft; no delete, and no send capability exists in the codebase), every proposed target is verified against Gmail server-side so an invented thread id is dropped loudly rather than acted on, repeated commands are content-hash idempotent, and confirmation is a REST endpoint bound to the owner's session with no model-callable equivalent - a jailbroken model cannot confirm its own writes. Refusals, skips, and duplicates are first-class audit rows, not absences. Proven on my real inbox within the hour it shipped: model-proposed triage, owner-confirmed archives executed 10/10 with per-target receipts - and when archived mail looked still-present, the answer came from querying Gmail directly (all 10 verifiably moved; the real gaps were Gmail threading alert/resolved notification pairs separately and an unread-only overview, both fixed and re-shipped the same hour with the lesson written into the tool's own instructions). The ingest-side injection screen also earned its keep in production that hour: a marketing email with instruction-shaped copy was withheld from the model's view while staying actionable.

• The read arc, closed measured: my assistant went from "the corpus is mine" to reading Google Drive, Calendar, Gmail, and the live web in one arc - in the only order that isn't a promise: the injection gate shipped before the first connector, every third-party byte ingests as untrusted through it, each connector states its real selection guarantee in the UI (grant-enforced vs application policy), and email syncs one document per thread so a phishing message quarantines one conversation. The web tool shipped recorded as UNTRUSTED under the project's own rule - no number, no trust - until its five-arm cohort measured it: first result retracted (the instrument had scored the model's ideal refuse-and-report answer as obedience), fixed instrument pinned by test, clean run 5/5 with every fixture verifiably reaching the model. Three case studies came out of the arc, published with the retractions in them.

• The scoreboard, and the operator catching up: published Justified Confidence - The Scoreboard - every reliability layer the field names with a stamped number or an honest gap, the tuned-vs-holdout gap published as it widened, and a priced deficit section. Then the operator side caught up with the public pages the same night: admin trust tabs and full-metrics eval tables on every serving instance (the public pages had been publishing more measurement than the operator could see), a per-feature model pinning matrix fleet-wide (chat / eval writer / eval judge, with the writer-judge same-family collision surfaced at dial time - closing a lane where a UI could silently bypass server-side model pins), and per-answer latency instrumentation with live-derived percentiles - which shrank the scoreboard's own printed deficit the day after it published. The deficit getting shorter in public is the page's design working.

• Governed agentic writes: a demo agent that changes records, not just answers - Claude tool use against a mock of Workday's documented Staffing API, with authorization checked against session identity (never against anything the model says), a server-side confirm-before-execute gate, idempotent writes, field-level access (compensation visible to HR admins only, stripped for everyone else), and refusals audited as faithfully as actions. One trust layer now serves three doors - the web UI, Claude Desktop over an MCP stdio server, and a Slack bot - with identity supplied by the surface, never the conversation; the first live Slack write came from an unmapped user and was refused and audited, which is fail-closed working as built. Measured with a mechanically-graded eval suite - 18/18 on the latest run, prompt-injection cases 5/5 - plus 38 mechanical tests, and the trust layer's own tests pass with no model attached. Two-minute demo.

• The trust layer went fleet-wide: the four-rubric judged evals, honesty cohorts, and reranked hybrid retrieval now run on every serving instance - validated per instance with like-for-like A/Bs on each one's own question set (one peer: correctness +12; the other: right-document retrieval +18.4 with first-position hits doubled). Fixes are proven where they land, not assumed to transfer.

• Live trust panels: both public assistants now publish their measured numbers on public pages - every value derived from stored evaluation runs at request time, shown as bands across identical-configuration runs. The panel's first hour live caught a real measurement bug: bands were blending runs judged under two different rubric versions into one absurd spread. Fixed the same hour by stamping every eval row with a judge instrument version - a score is a property of the system, the corpus, the question set, AND the ruler.

• Multi-turn, measure-gated: scripted-conversation cohorts now measure follow-up handling - the failure class single-turn evals structurally cannot see. The discipline ran end to end: baselines on the real unresolved path (bare follow-ups found the right document 1 of 9 and 3 of 8 times on the two public instances), the follow-up resolver ported strictly on those numbers, and the same exams re-scored 8 of 9 and 7 of 8.

• Grading the grader: the LLM judge behind every trust number is now calibrated five independent ways - planted-error suite 37/37 across four rubrics, human-adjudicated agreement 90% (kappa 0.61), a second judge from a different lab at 99.0% correctness agreement (kappa 0.94, up from 92.7% / 0.68 before the judge was tightened - so an outside lab confirmed the tightening improved the instrument, not just its self-scores), and a second independently-authored harness (RAGAS) re-grading the same stored run with the same judge model: 98.2% agreement at the pass/fail boundary, its sole disagreement an already-known defect row. Every disagreement is hand-adjudicated - the latest six split into one real defect the primary judge had passed, one genuine corpus-staleness catch, and three refusal over-flags by the second judge.

• The exam nobody could study for: the fifth instrument is a locked holdout authored by an outside model straight from the corpus, structurally excluded from every fix loop - and the exam is held to the same standard as the system: the original 18 questions shared a lab with the judge, so a third lab re-authored it as 32 questions spanning the whole corpus, with the retired exam's authorship conflict priced first (2.1 points against a measured 1.5-point noise band). Latest reading (2026-09-05, a pair of runs held at one corpus fingerprint): tuned 90.9%, the holdout 71.9% - a 19.0-point overfit gap published as a first-class number, next to the 2.6-point self-spread those identical runs revealed, so a sub-noise difference cannot be read as movement. This fourth widening has its own cause and is stated as such: both ends moved at once, the tuned score rising while the holdout fell again under a corpus that grew into it. The gap is the reason the capstone write-up stays gated - the bar for it was written down before the run, the run missed it, and the bar did not move. The fairer exam also broke a flattering diagnosis: retrieval was 18-for-18 on the old exam but 25-of-32 on the new one (2026-07-30) - the old exam never reached the corpus tail, so "the gap lives entirely in the answer layer" was the exam talking, not the system. A fairer exam, a worse number, published anyway - then a moving corpus, a worse number again, published again.

• Refuse-vs-fabricate, measured everywhere: questions that demand artifacts the corpus does not hold (raw CSVs, shell commands, secret values, private records) run on every serving instance, each cohort authored by a model that plays no other role in that instance's eval - the system never sets its own honesty exam. The private hub holds 13/13; both public assistants opened at 12/12 on their first measured reads; the public demo holds 100 across three consecutive runs. Any invented path, command, or figure fails, even inside a disclosure-toned answer.

AI / LLM  Python · FastAPI · RAG (ChromaDB, hybrid vector + BM25, cross-encoder reranking) · retrieval + answer-quality eval harnesses (LLM-as-judge: correctness, faithfulness, freshness) · judge calibration (planted-error suites, Cohen's kappa, cross-provider second opinions) · agentic tool use · QLoRA / PEFT fine-tuning · multi-provider routing (Anthropic, OpenAI, Gemini, Mistral, Groq, xAI, DeepSeek, Ollama) · local inference (7B to 30B-class on RTX 5090/3090)

Full-stack  Next.js · React · TypeScript · React Native / Expo · Node.js · Firebase · Stripe

Infrastructure  GCP · Docker · Nginx · Cloudflare · CI security gating (gitleaks, pip-audit, bandit, trivy)

AI / ML engineer roles - building and operating production AI systems - and private AI consulting (your data, your brand, your infrastructure).  Resume · Get in touch