Repository navigation
Conversation
…corpus The server's wrapper (docs/implementation/) starts from the engine itself: a Linux image built from this repo's own scripts (vendor, llama.cpp, ffmpeg), the CLI verified headless in the final image, and Stage 2's proof indexing the pinned Commons corpus and answering natural-language searches with the engine's numbers. Additive: only .gitignore gains lines, and .dockerignore keeps host artifacts out of the build context.
The server talks to DigUp the way the Tauri shell does: spawn digup search-service / index-service / worker as supervised children (the ParentWatch rule), speak their JSON-lines protocols, read status/estimate from the CLI. The engine is never linked, patched, or replaced; the contract tests pin every reply field, event, and exit path the adapter uses, against the shipped Linux payload.
POST /index starts one serialized job (409 busy with the live id); the runner forwards the engine's stop at the safe point, marks interrupted records with partial counts, replays its append-only log across rotations at startup, and reloads+prewarms the searcher after a done job. Root-scope semantics pinned by tests: plain sync retires walked-root ghosts; prune retires out-of-scope ghosts.
FastAPI with typed wire models, the engine's answers verbatim (hits, strong, meaning, ms, warning, the honest empty), engine failures as 5xx quoting the engine, config from the environment only — paths never arrive from a client. localhost by default; in the container the publish line enforces that. The lifespan owns the search-service as a direct child, prewarms at startup, and ends every helper on shutdown.
… by id Vanilla HTML/CSS/JS served same-origin (no build step, no external URLs, pinned by test). Media endpoints take only server-issued media_id handles: /preview (the engine's render JPEGs, symlink-safe to <index>/previews), /text (the passage it indexed), and /file — the file's real bytes, typed by magic bytes, Accept-Ranges and 206 for video seeking (#t=), inline for media kinds, attachment text/plain for code/doc. The UI mirrors shell/ui: keyword-first typing (prefix_last words, then the meaning upgrade after 120 ms with embed_ahead), strong/weaker fold, kind filters, code mode, limit, selection with arrows, Space big look, Open at the video's moment, Estimate panel, activity dot, keys footer — plus the System/Light/Dark appearance choice kept in localStorage, which the OS-following desktop needs no equivalent of. Settings that are OS integrations are documented as deployment env config; reveal and copy are absent, not faked.
server-runtime: engine payload + API + UI on python:3.12-slim, tini as PID 1 (the ParentWatch chain tini->uvicorn->helpers), non-root uid 990, HEALTHCHECK asking the API's own /health, startup prewarm. Compose publishes 127.0.0.1:8787, names its project digup-server, keeps index/models/config/job-log in bind-mounted volumes, and sets stop_grace_period: 90s so a mid-job docker stop finishes the safe-point path (143, interrupted with partial counts) — the 15s default's SIGKILL path recovers by log replay and resumes, both measured (docs stage 8).
The suite's corpus stays exactly 15 files (its rankings normalize over that set), so new test material lives beside it: 20 subjects x 3 Commons photographs fetched as 1400 px renders, pinned in manifest.tsv, out of git, built by build.sh; the H.264 scenes clip travels with it. Provenance and licenses documented; it is testing material for the running server (Compose mount note in its README), not suite input.
…unbook docs .github/workflows/server.yml: build the builder and runtime images, run swift-side payload checks, materialize the corpus, download the pinned models, run the contract suite against the shipped payload, validate the Compose schema, and assert the SIGTERM contract (exit != 137, shutdown complete). The drift-guard step compares ModelFiles.swift's sha256 constants with the workflow's download pins, so an upstream model bump fails CI loudly before the server can silently run a different model. server/README.md and IMPLEMENTATION_STATUS.md carry the runbooks, the merge policy, and what is proven.
…ettings Palettes explicit per theme (measured WCAG ratios), color-scheme per theme, the big-look overlay guarded (#big[hidden]) — it sat at 78% black over the whole page. Search box centered with the hit-limit select at its right end, kind chips as a themed, evenly spaced pill row, gear opens the settings panel: appearance, Index now + prune, Estimate (one honest note line), and the engine's counts. el() takes real child nodes (the [object HTMLElement] text is gone); statusline shows the searchable count by the title.
…e; MP4 by ftyp An ok:false reply (SearchService.swift:106,115) is EngineRefused: raised in the adapter, answered 503 with the engine's sentence — a refusal reply has no hits, and api.py used to crash KeyError on it (measured 500s in the Stage-10 battery). raw() keeps passing the engine's replies verbatim; the runner survives a refused post-job refresh (the job stays done, with a note). The UI ships cache-control: no-cache — measured: without it Chrome served a stylesheet the server no longer had. The magic sniff reads ftyp at offset 4 and the brand there, so scenes.mp4 is video/mp4, not octet-stream.
…d fixes Corpus2 gallery served live: job 0.4 s idempotent (61 files, 65 vectors), six meaning searches all strong 3 (top z 4.46), /file 200·206·416·404 on real bytes, video range + preview frame, estimate ≈ 263.9 s calibrated. Suite: 57 passed, 1 deselected, 241.18 s. The proof battery gates on readiness and retries 503s. .gitignore: the Playwright MCP working folder.
Single-column system: header grid (title left, gear right, the searchable count centered), search row 1fr auto 1fr so the input's center IS the page's center with limit + code at its right end, footer text aligned to the content column, settings anchored under the gear (right edges equal, controls left edges at one x) as a label/control grid with section rules and a close button. The gear is an inline SVG centered 0/0 in its circle, colored by currentColor. Measured at 1280 px and 390 px: center offsets 0, overflowX 0. The gear's svg broke the outside-click close (target !== gear) — closest(). Suite (20): 57 passed, 1 deselected, 236.12 s.
Nine real findings from the adversarial backend review, each fixed with the scenario the reviewer described pinned as a test: - jobs.shutdown() self-deadlocked (a plain Lock re-taken by _mark from inside its own with-block): a docker stop with a job past the join grace hung the lifespan forever. The active job is now read under the lock and marked outside it. - the runner thread died on a TimeoutError from the post-job prewarm (not a RuntimeError): _active stayed set, every later POST /index answered 409 naming a job that could never run. Any escape from _run_one now marks the job interrupted, clears the writer, and keeps the thread alive. - a failed search-service handshake leaked the child (pipes + reader threads unreferenced until its 600 s idle-exit); _spawn_locked reaps on every failure and maps a missing binary to CLIError, so the API quotes the spawn as a 502 instead of dying as a 500. - ok:false WITHOUT an error field passed the refusal guard and hit the API's stripped["hits"] as a KeyError — the guard fires on the shape now. - render held the serialize lock across the engine's ffmpeg work, undoing the engine's own design (render runs beside the read loop on two slots, replies routed by id, "typing never waits on one"). Two slots, lock-free render: a 1.2 s frame no longer freezes keystroke-searches. - Range: bytes=0- (what every media element sends first) buffered the whole file; open/suffix/huge ranges are capped at 8 MiB and streamed from the span's own handle. Type, size and span come from one open (fstat + S_ISREG), root confinement is is_relative_to, and every /file answer carries nosniff. - subprocess.TimeoutExpired is a SubprocessError, not a TimeoutError: mapped in cli.run so a stalled status/estimate quotes the engine (502), not a traceback. - job-log rotation shifted a .1…N chain (O(n) renames, order scramblable by a kill mid-shift: a running line read after its done reported finished jobs as interrupted). One rename to a time_ns suffix; replay is a sort. Replay skips valid JSON that is not a job record (5, null, a record with no id used to brick app start). - HitCache.get() didn't refresh, so eviction was insertion-order: a browsing session evicted its own on-screen handles after ~26 searches (404s, no recovery). move_to_end makes it the LRU the docstring promised. - /status and /health each spawned a digup status (real SQLite counting) per poll; the identity fields ride a StatusCache TTL (DIGUP_SERVER_STATUS_TTL_S, 0 = read every time). SearchReply is extra="allow": engine fields this model has never seen are published, not dropped. Suite: 75 passed, 1 deselected, zero skipped in 228.12 s in digup-server-test:local.
…yncs its aria
The UI review's blockers, verified in a live browser against the running stack:
- Space hijacked every button. Native activation fires on keyup; the document
handler (Quick Look) preventDefault()ed on keydown for everything except a
few tags, so the gear, the close, Index now, Estimate, "all", and the
Open/Download links stopped activating for keyboard users — while the big
look flashed open over the page. Activation targets (closest() over
button/a/[role=button]/input/select/textarea) keep Space; bare page areas
preview. Live: Space on the focused gear opens Settings; Space on a focused
row opens the big look; Space on a focused link is the browser's own.
- a failed job poll killed the loop (unhandled rejection): the job line froze
at its last state and "indexing…" stayed in the footer forever. Polling now
answers "job poll failed: …" honestly, clears the indexing state, and stops;
repeated busy-clicks poll one loop (state.jobPoll), not N.
- FastAPI's list-shaped validation detail printed "[object Object]" on the
footer; api() renders each {loc, msg} and falls back to statusText for a
non-JSON body.
- every settings-close path shares closeSettings(): aria-expanded always
matches the panel's hidden state, opening focuses the first control, closing
returns focus to the gear. Estimate's panel has a generation token, so a
slow crawl's late reply can't write into a panel closed and reopened between.
- roving tabindex (selected 0, rest -1: 40-100 rows no longer turn every
search into a tab storm), aria-selected on rows, the weaker fold says
role=option + aria-expanded/aria-selected, group titles are role=presentation.
- Space in the field on the weaker row toggles the fold (as the document
handler does), the code hint shows for the code-mode empty, and the query
field carries maxlength=512 matching the API's cap.
- CSS: the ticked kind chips mix at 10% (the dark audio pair measured 4.3:1 at
14%, 4.54:1 now — AA), .badge.warn wins the cascade it was losing to .badge,
the audio control is sized where app.js puts it (.audio-picture), and the
keys line unwraps under 720px. A data: favicon silences the browser's
standing 404 in the console (no external URL: the pin tests stay green).
- the test image carries tini as its ENTRYPOINT. The suite's app spawns search-service as a child of the test process, and ParentWatch ends a helper whose parent is PID 1: run the image bare and pytest IS PID 1, so every helper died at once (`exit 0`, no stderr — 59 setup errors, measured). With tini as PID 1 the test process is a live parent, the production shape, and CI needs no --init. - HEALTHCHECK is server/docker/healthcheck.py: 200 healthy, 503 ALIVE (the API answers 503 while the searcher respawns — Docker must not restart a live server over the engine's own refusal; urlopen raises HTTPError on 503 so it is caught), everything else exit 1. The build asserts the script fails with nothing listening; CI polls the live server with the same file. - measured honesty in the Dockerfile: base is Debian 13 (trixie), glibc 2.41-12+deb13u4 (newer than the payload's 2.35 build glibc — forward, never back); no /data/locks (the writer lock is <index>/.digup-server.writer.lock); the inert DIGUP_SERVER_HOST/PORT ENV lines are gone (uvicorn's CMD args win in a container; those envs feed the bare-metal `python -m` entry); `digup --help` claims only that the CLI runs headless — §3's full acceptance is CI's boot step and the proof battery, against a live index. - CI: 180-minute ceiling; the engine chain (ffmpeg, llama.cpp) is built host-side with the repo-pinned actions/cache and the same hash keys ci.yml uses, then seeded into the builder's cache mount through .ci-vendor/ (the vendor scripts land on their own stamps and skip the rebuild; hosted runners start their BuildKit mounts cold, and a fresh 15-minute ffmpeg compile per PR was overrunning the old ceiling). The drift guard runs FIRST and pins ModelFiles.revision + both file names + both sha256s. Zero-skip gate (any SKIPPED line is red; the clip must exist). Compose schema check. The boot step polls /health with the image's own healthcheck.py, asserts exit != 137 + 'Application shutdown complete', and uploads proof/suite logs. - proof.sh counts failures: it piped every gate through tee and exited 0 unconditionally, so a run where the index died and every search answered nothing was 'green'. Gates now record their labels, the pipeline's status is real (pipefail), and the eval ALL row must clear R@1 >= 0.60 / R@5 >= 0.85 (the recorded 0.71/0.88 clear it with margin) before it prints PASSED. - compose: restart: unless-stopped, and DIGUP_PUBLISH_PORT for the host side when 8787 is taken. localhost-only publish unchanged. - .dockerignore excludes the corpus videos/clips, corpus2 media (45 MB rode into the build context) and .playwright-mcp/; .gitignore ignores .ci-vendor/* with a !-exception for the tracked .gitkeep the COPY needs.
- server/README.md: the suite's real number (75 passed / 1 deselected / zero skipped, buckets 17+19+6+13+17+4 collected 76) where two stale counts said 48 and 57; the tini rationale for the test image (the ParentWatch PID-1 trap, measured 59 errors); the -m slow clarification; the quick start's mkdir + chown -R 990 for real Linux hosts (Docker creates bind dirs root-owned, the server crashes EACCES on its lock); restart/publish-port notes; the corpus2 scale note (zGate 2.0 is the engine's gate on every platform - 1-3 hits on 61 files is the corpus, not a filter). - IMPLEMENTATION_STATUS: Stage-11 section, newest-first - suite23/24, the parity identity table (adapter on a sqlite3 .backup snapshot == HTTP, identical hits/strong/z on five queries + keyword-first), the proof battery's 11 gates, the browser verification of the keyboard fixes, the CI/image facts, the commit shape, and the honest open items (CI first green pending, Stages 9-10). - docs/implementation/review-round.md: the full finding -> fix -> test table for backend, UI, and build/CI, plus the deliberate non-fixes with reasons (no auth on a localhost-only server; no pixels cap; no fsync per job-log line; test image runs root; the engine's own limits left alone by the PR's rule). - docs/implementation/stage-11-review-round.md: the stage record - the PID-1 discovery, the parity proof with the probe lesson (a read-only mount of a hot WAL index is not a snapshot: 0 hits, probe artifact), the corpus-scale quantification, the battery gates, the browser checks. - Docs honesty pass, measured not asserted: base image is Debian 13 trixie glibc 2.41-12+deb13u4 (newer than the payload's 2.35 build glibc, forward never back), no /data/locks (the writer lock is <index>/.digup-server.writer.lock), the shipped endpoint list (/preview|/text|/file, no settings endpoints), the timestamp-suffixed rotation shape + the 5 new job tests (13 total), the CORS/localhost wording, the media-surface notes. - Host identity out of the stage records (this is a publication): the three host home paths become ~/Projects paths (a clone works from anywhere), and the exact OS/Xcode build codes (fingerprint details) come out. All build/proof commands in the docs are relative or env-var based: $PWD, $HOME, $RUNNER_TEMP, container paths.
Ocean572
changed the title
… process is the design
…he UI asking once
… resume sums them
…easured on the NVIDIA box
…wn the Gate A commands
…1070 cuda-libs assembles the shipped chain by glob (cp -a keeps symlink+target; 5 lib families + libnccl.so.2, 955 M) — the dotted -dev names are not in a runtime base; the Dockerfile rm -fs Vendor/llama-build/CMakeCache.txt before build-llama: a flavor switch on a shared cache mount must re-configure (stale GGML_CUDA=ON killed it, measured 2026-11-04); gpu-handoff.sh takes CUDA_ARCHS as a build-arg (plain numbers only — apt cmake 3.22.1 rejects 6.1 and native alike), checks the payload in /out, waits /health then retries POST, and samples nvidia-smi at 0.15 s across a real index job (PASS peak > 20% or busy-duty >= 10% with the model in card VRAM; a 2 s sampler saw 0-12% on a provably-GPU job); the nvidia compose override carries target server-runtime-cuda + the matching devel/runtime CUDA bases + CUDA_ARCHS so --profile nvidia up -d --build builds the CUDA image on the box that has the card.
… tags embed the pushed engine
… 131,972 files / 131,758 searchable / 403,831 vectors / 1.39 GB Job 3 record (engine's own): started 01:32:33 → ended 07:13:03 box-local, 20,225.0 s, files_done 7,097, files_failed 1, errors 0. Whole library across the three kept records: 131,543 files completed, 421 recorded failures (video decode refusals), 0 server-side errors; engine working time 59,940 s = 16 h 39 min, 2.195 files/s average — 33.75% of the estimate forecast's 177,575.3 s (GPU-assisted). Three verified ceiling-stop/resume cycles. Sampler peaks: 99% GPU (job 2) / 90% (job 3), model resident 1,780–1,802 MiB; container 207.75% CPU, 3.436 GiB RAM. Final timed searches 0.6875–0.8004 s, five queries, all 10 hits/10 strong/meaning true. Index 1,387,327,274 bytes; / free 7.0 G at end. Full sampler table + arithmetic in server/data/bench-18m-stats.md (git-ignored).
…rge shapes pinned in CI, ghost-row recreation lesson measured and documented docker-compose.lan.yml appends a LAN host binding for the API (env-forced DIGUP_LAN_BIND — refuses to apply without it; the base file's 127.0.0.1 mapping stays: compose merges ports lists). Header states the trade plainly: the server has no auth, so a LAN bind exposes the whole indexed library to the network; home-LAN use only, the §7 shape for anything else is an auth'd proxy. Field-proven on this box merged with the GPU override: healthy, UI + /api/v1 + searches on the host's LAN address, 10 hits/10 strong in 0.71-0.81 s at the 132k-file index. CI now schema-checks seven merge shapes (adds base+lan, lan+nvidia, ghcr+lan; uses RFC 5737 TEST-NET addresses — documentation addresses, nobody's). The server/README.md LAN section documents the shape and pins the recreation rule the field run proved the hard way: recreations must pass the SAME file chain AND env every time — dropped env moved the media bind one folder off the roots the index rows carry, every row fell out of scope, and the API answered 0 hits (keyword queries included) while health/files/vectors/job-log stayed normal; same command with the original env restored 10/10 strong, no prune, no reindex, the three job records untouched.
…58 s (7 merge shapes), publish 1 h 21 m 44 s, both tags republished
…ng compose state, git-ignored by design)
…st, principal log, embed widget, the Nextcloud shared-stack compose shape The host-application future (docs/implementation/stage-12-integrations.md, the binding contract with §11.x numbers quoted in code): one compose stack shares the Nextcloud data dir with the DigUp server read-only, and a Nextcloud PHP app (its own repository, digup-nextcloud) queries the API server-to-server with a read-scope token. Per-user ACL is the host app's boundary (its permission APIs filter every result set); the server gains no user concept and touches nothing. security.py: DIGUP_SERVER_READ_TOKENS=name:token,… — every entry guards the read scope exactly like DIGUP_SERVER_TOKEN (index/estimate stay admin-only), first value wins per name, malformed entries skipped with a named warning; Auth.token_names() answers scopes per token (admin covers the read side); BearerAuth records the accepted principal on the scope; PrincipalLog (outermost) writes 'POST /api/v1/search -> 200 principal=nc-a' for every guarded request — health/docs excluded, unguarded deployments log principal=none. Cors middleware: DIGUP_SERVER_ALLOWED_ORIGINS exact origins, inert by default, preflight 204 grants, decorated answers, and it sets the guard's cross-origin-POST exception ONLY for allowlisted origins (Host allowlist still outranks everything; the cookie-free CSRF argument spelled out). api.py installs all four middlewares always: inert knobs, same opt-in discipline as every §7 feature. server/ui/embed.html/.css/.js — the compact widget for host apps: no external URLs, no build step, server-issued media handles, engine fields verbatim, keyword-first typing with embed_ahead, IntersectionObserver previews, theme via postMessage from the host; the token enters via ?token=, moves to sessionStorage, history.replaceState strips it from the URL; previews are authorized blob fetches, never a token in an <img> URL. deploy/docker-compose.nextcloud.yml — the shared-stack example: data dir mounted :ro at /media/photos, group_add default 33 (the official image writes the data dir as www-data), appdata_<instanceid> exclusion COMPOSED from DIGUP_NEXTCLOUD_INSTANCE_ID (occ config:system:get instanceid — CACHEDIR-tagged preview/thumbnails/versions auto-skip by the engine's own rule on top), token map entry nextcloud:<secret> + admin token env-forced with plain messages, allowed hosts server,127.0.0.1,localhost, Nextcloud headless-install env, config/custom_apps volumes, the SSRF note. :? forcing verified: each missing knob names itself (measured exit 1). CI schema-checks grow 7→9 shapes (standalone + LAN-merged, RFC 5737 + synthetic paths). Contract suite: 14 new tests (map scopes/revocation/attribution, CORS inert/allowlisted/ composed-stack, embed surface pins, middleware install pins, end-to-end token map over the real API: search 200 with map tokens, index/estimate 403 for them, admin estimate 200, tokenless 401). Full suite in the container, all four mounts: 116 passed, 1 deselected, ZERO skips, 150.05 s — every pre-existing pin of the bearer/host rules passed unchanged over the refactor. Docs: stage-12-integrations.md (the contract, gates, multi-instance + remote-GPU honesty incl. the NFS prerequisite and 1-search-at-a-time concurrency), docs README entry, server/README Nextcloud section, API.md token-map rows + CORS section, .env.example knobs. Next: the pilot on this box (Nextcloud + server one stack, corpus2 public gallery as files, planted junk dirs, token/ACL/no-mutation gates), then the digup-nextcloud app repository.
…dit log gets a handler, 28 pilot gate numbers Two fixes measured on the Nextcloud 28 field pilot (gate lines in IMPLEMENTATION_STATUS.md): the estimate argv now carries the excludes the index job will honour — the forecast describes the tree the indexer walks (new suite test pins the argv shape); the principal audit logger carries its own stdout handler (uvicorn's config leaves application loggers handlerless, so the lines never appeared). Suite: 117 passed, 1 deselected, 0 skipped, 111.09 s.
…2 + FUSE + live propagation), per-user multi-root env, host-mount script, CI schema lines, pilot gates measured
…ng, trusted-domain rule measured (first-install-only env application), CI schema line
Reference an existing data tree without moving it: a read-only /extstorage bind on the Nextcloud service (the files_external local backend's datadir points inside it; the mount is read-only in Nextcloud's config, and the DigUp server binds the SAME host folder). group_add carries the tree's host group so uid 33 can read it (DIGUP_NEXTCLOUD_EXTSTORAGE_GID, default 1000). CI covers the new env in the compose config check.
…group Apache workers run as www-data and call initgroups() for the configured User, so the web process reads a group-restricted host tree only when www-data holds that group in the container's /etc/group (group_add alone cannot reach them: setuid drops supplementary groups). The official image only runs executable hook scripts, hence the committed mode 755.
…search get Measured 2026-11-08 on the Nextcloud pilot: index_argv omitted --exclude/ --exclude-ext while estimate_argv had them (§11.3). The job walked the tree's appdata folder — 1,470,439 preview files — against a 131,972-file forecast: 250 h of embedding nobody asked for, previews in results, and an estimate that timed out walking work the excludes had already cancelled. The engine's shared parser accepts both flags on index-service (verified: left_out reports 'folders excluded folder'). The test's title already promised 'the excludes the job will honour'; the suite asserts it for index_argv now.
UX ask measured on the Nextcloud pilot 2026-11-08: a blocking window.prompt fires per new tab, cannot be reopened, and is gone once dismissed. The main UI's 401 now points at the gear (a soft pulse) and says where to go: Settings -> Access, where the key is pasted, saved (localStorage), cleared. The key is sent as a bearer header only, never in a URL; a 401 with a stored key shows the server's own words, so a wrong key cannot loop the page. Per-client tokens for other applications stay env-issued (DIGUP_SERVER_READ_TOKENS) and are never answered over the API — the field is paste-from-deployment. The embed widget, which has no settings panel of its own, keeps its prompt (test_integrations asserts it). The suite asserts the new main-UI shape (field present, no window.prompt in app.js).
…gine picks GPU All measured: job-excludes gap (1,643,492 walk vs 131,973 forecast), recovery via graceful stop + interrupted record (<config>/jobs.jsonl survives index wipes by design), fresh walk 131,973, CUDA image built on the box from the current tree (the ghcr CUDA image predates the excludes fix), speed --model device gpu + agreement 0.99977, utilization gate PASS (duty 45.9%, peak 95%, 1,697 MiB VRAM), 1.93 files/s → 18.5 h forecast. Suite 117/1/0 twice.
…alone AND serving applications The compose server keeps running standalone over its own media index, and serves other applications: it MINTS their access keys and builds a SEPARATE index per application, never the media one. The engine changes nothing — it answers for any (roots, --index) pair; the server supervises each pair independently. - targets.py: registry (DIGUP_SERVER_APP_TARGETS entries name=roots:index, roots colon-separated with the index LAST, merged with <config>/targets.json, env wins per name) with strict validation: slug names, reserved 'media', absolute paths, and a target's index may not BE the main index. - api.py: every guarded endpoint addresses a target (?target=name, default media); main scope = the lifespan's (the injected-searcher seam the 503 contract pins, resolved at call time); app targets spawn LAZILY — own search-service child, own JobRunner (jobs-<name>.jsonl), own writer lock for free (the flock lives beside each index folder, so crawls of different indexes run in parallel). Media handles resolve their target by longest-root match: preview/text/file work across indexes. GET /api/v1/targets lists the registry — counts from a running target's live searcher (the authority on a store it has open: measured in-suite, a WAL pinned by the searcher's reads reads 0 to a fresh CLI open), from the CLI table for unspawned targets, null for an empty folder; no engine is spawned for a GET. The single status cache is keyed per index folder (no cross-target bleed). - security.py: mint_tokens — server-issued app:<name> keys + admin key when the deployment has none, stored 0600 <config>/tokens.json (recreation-safe, restart-stable, displayable from any boot); minting fires ONLY when targets exist (a target-less standalone deployment runs unguarded, byte-identical to before); env secrets never stored, never echoed. The bearer gate binds tokens to targets: admin answers all; read/plain-named tokens answer the main index (their pre-stage-13 contract, unchanged); an app:<name> token defaults to its OWN target (plain paths — the bridge needs no target parameter) and is 403 on any other; mutating routes accept an app token for its own target only. The audit line names the target too (principal=app:nc target=nc). GET /api/v1/tokens shows minted values to the admin principal only. - UI: gear → Settings → Access gained Application keys (copy buttons) and Indexes; both hide for non-admin keys. Docs: API.md 'Named indexes' + auth table, .env.example, deploy/docker-compose.app-targets.yml (opt-in -f; the target's index rides its own bind mount, recreation-safe like the media index's), CI validates the fragment with and without the nvidia profile. - Dockerfile: ARG SWIFT_TESTS=1 gates the engine swift test in the builder stage (CI never passes the arg, gate unchanged; local suite builds use 0 after the measured box fact — CodeTests gitListingNeverRunsRepositoryFsmonitor dies in this box's builder; the contract suite is the server's gate). Suite (this box, container, four mounts, -rs, zero skips): run 3 6F/123P → 4 2F/127P → 5 1F/128P (the preview poll's own text-doc pick: the engine's render refuses text docs permanently) → 7 + 8: 129P/1D/0F → final tree: 129 passed / 1 deselected / 0 failed / 0 skipped in 160.04 s. Live demo gate (compose stack, port 8792, CPU, public corpus photos): minted admin + app keys 0600; app token answers target nextcloud roots /media/app, 403 on ?target=media; its index 202, second 409 naming the live job, job done, 36 files / 36 searchable; media index untouched (0 files, 0 hits); app search 3 hits all under /media/app; targets counts media 0 / nextcloud 36; tokens visible to admin only, 403 to the app key. The Nextcloud bridge 0.4.0 (status line + Index now) is deployed on the pilot and gated: status 200 target media 131,973 files / 118,778 searchable live, index-now passthrough verbatim, non-admin 403.
…ronicle Suite runs 3→11 with each failure named and diagnosed (binding default, injected- searcher seam, tokens-file dir, env main-index clash, status-cache bleed, restart displayability, admin principal form, parser-shape crash — the shipped docs/compose shape won, parser moved to it — and the WAL-pinned store the CLI open reads empty: /targets counts now come from a running target's live searcher), the SWIFT_TESTS builder fact, the live demo gate numbers, and the refreshed 'What runs today'.
…ild path reaches the Dockerfile's engine-test gate; unset keeps it running (CI shape unchanged), a box with a measured builder-environment failure documents 0
…s, inert minting, contract unchanged) + the pilot job's 10 h ceiling → interrupted 118,206 → 202 resume + battery numbers
…ey (2026-11-08 UI brief)
The main page keeps no status nag: the "status unavailable: … paste your key" line
is deleted, and the engine's counts live in the panel's `The index` (per-target rows
with files/searchable/roots). Access/Application-keys/Indexes collapse into two
sections in the operator's reading order: `The index` and **`API key`** — the minted
keys as plain rows (`API key`, `API key — <target>`) each with **Copy**, a
**Regenerate** button behind a confirm ("the applications that have these keys lose
access"), and the small paste field for deployments whose keys come from the
environment (env keys are never shown and never rotated — their operator gives THIS
browser its key from the deployment's own config).
`POST /api/v1/tokens` (admin scope) is the rotation: every minted value (admin + app
keys) gets a fresh `secrets.token_urlsafe(24)`, persisted 0600 in the keys file, live
immediately — the old values answer 401 from that call on, which is exactly what the
confirm warns about; the browser's own key updates itself. Environment-configured keys
are kept untouched (a rotation must not silently override the deployment's own
config), and a deployment with nothing minted answers 400 honestly. `Auth` stays
frozen: the rotation swaps the bearer gate's auth reference by walking the built
middleware stack. The admin-scope check also accepts a direct admin-bearer match — an
admin-token-only deployment deliberately leaves reads open (principal `none`), and
this surface exists to answer the admin key's owner.
Measured on this box (ai-ubuntu-computer): suite run 13 — 133 passed / 1 deselected /
0 failed / 0 skipped, 224.61 s (three new rotation contracts, reworked UI contract);
demo flow gate green; rotate gate live: POST 200 rotated, old app key 401 and old
admin key 401 on the same call, new app key 200 target `nextcloud`, 403 cross-target,
tokens file 0600 carrying exactly the new values, media untouched, search after
rotate 3 hits all under the app's own roots; served UI markup: sections present,
main-page statusline absent.
…24.61 s), demo flow + rotate gates green, the pilot's app at 0.4.1 (one "API key" field) The paragraph tells the rework as measured: the rotation's live behavior on the demo stack (old keys 401 on the same call, new keys answering their own target, keys file 0600 carrying exactly the new values, media untouched), the served-UI markup check, and the "What runs today, exactly" table now carries the current stacks: the bench CUDA image awaiting the UI-rework rebuild, the pilot's media-resume job in flight with app 0.4.1, the demo stack with both gates green.
…the gear's API key section is Copy + Regenerate (2026-11-08 user brief, round 2) The user's model, made the contract: "the standalone app just uses the directory that was set in the environment" — main page searches with NO key. The gate now draws the line where the deployment's own typed secrets end: keys SET IN THE ENVIRONMENT guard every call (the existing shape, its contracts untouched — test_security pins it); keys MINTED by the server (applications registered) guard what keys are FOR — index jobs, estimates, the keys surface, and each application's OWN index (reads, previews, its own crawl) — while the main index's reads answer openly to the operator's browser: status, search, targets, embed_ahead. Media handles close the side door: a handle whose file lives under an application's roots answers 403 to a key-less request and 200 to that application's key. `POST /index`, `POST /estimate`, `GET /tokens` keep their key requirements exactly. UI per the same brief: the gear's API key section is ONLY the minted-key rows (label, value, Copy) and Regenerate — the paste field, Save, and Clear are deleted (the key is server-generated at runtime; the APPLICATIONS paste it into their settings; Regenerate adopts the browser's new admin key itself). Rows became a uniform grid — label | value | action, one row height, equal padding and borders, key text clips with an ellipsis and Copy carries the whole value. The 401 message names the server's shape honestly (a minted server lists its keys in the gear; a tokenless server asks for nothing). Measured on this box: suite run 14 — 135 passed / 1 deselected / 0 failed / 0 skipped, 164.25 s (two new shape contracts: open main reads + guarded mutations/keys/app target; open principal never previews an application's files); demo flow gate on the rebuilt stack: status no key 200 target media, index/estimate no key 401, tokens no key 403, app-target read no key 401, app key 200 own target + 403 cross, 202→409→done, 36/36, media untouched; rotate gate: rotated 200, old keys 401, new keys answer, file 0600, media untouched, search after rotate all under the app's roots. Bench :8787 (tokenless launch): wakeboarding un-keyed 10 hits 10 strong 0.72 s; 10/10 queries strong 0.73–0.92 s; index intact 131,972/131,758; served UI carries the grid rows, no paste field.
…ar API key = Copy + Regenerate, the app's preview-pane visibility fix; suite run 14 (135/1D/0F/0S, 164.25 s), demo gates 4 + rotate 2, bench tokenless numbers
…s the shape honestly; Estimate spins a wheel in the gear (2026-11-08 user brief, rounds 3–4) The user's live test of the panel: uneven rows, a lone Regenerate button where keys should render, and an estimate popup on the main page they never asked for. Every panel control is now exactly 2rem tall — appearance select, Index now, the prune row, Estimate, Regenerate, every key and index row — section titles aligned to the first row; key rows are `label | value | Copy` in the grid, key text ellipsis-clipped (Copy carries the whole value). The API-key section reports the shape truthfully: minted keys as rows; an environment-keyed deployment as the honest row "Set in the server's environment" (typed keys are never echoed — the panel says so); a tokenless deployment has NO keys and the section collapses entirely (minting fires when an application registers — there is nothing to show, so nothing shows). Estimate: a CSS spinning wheel beside the gear's button is the progress indicator while the engine walks; the main-page "estimating — the engine is walking the folders…" line is deleted; the forecast fills the panel on arrival, the failure line stays. Measured on this box: suite runs 15 and 16 — 135 passed / 1 deselected / 0 failed / 0 skipped (205.85 s; 164.51 s), module run 37 passed mid-round; served UI on both stacks (:8787 tokenless, :8792 minted): api-key-sec present, estimate-spin present, paste field absent, grid rows + 2rem rule + spin keyframe live; wakeboarding un-keyed 10 hits 10 strong.
…e; gear shows the key keylessly (stages 14 + round 6) - DIGUP_SERVER_MINT_KEYS=1: the STANDALONE box mints its admin key at boot with no application registered — mint_tokens(mint_admin); compose passes the knob. - POST /api/v1/targets (admin key): an application REGISTERS at the RUNNING server — startup validator at runtime (slug, absolute paths, not-the-main-index, media reserved, 409 dupes), server mints its PRIVATE key (returned once, persisted 0600), own index folder (default <config>/index-<name>), targets.json persists it; its key answers only its target (own crawl in the background, invisible to the main page). - GET /api/v1/tokens is key-free in a SERVER-MINTED deployment: the gear renders 'API key | value | Copy' with no key in the browser — the ticket model (main page searches keyless; the gear's Copy row is what applications paste). Typed secrets (env shape) stay guarded, never displayed (keyless 403 at the endpoint, app keys 403); security.py: env_secrets() module fn, _env_secrets delegates. - UI round 6: Regenerate rides the key row's right edge (grid label|key|Copy|Regen, built in app.js); Estimate's forecast sits beside the button (plain row column — no stray box); a finished crawl re-reads the index list (no stale target rows). - Contracts: minted keyless tokens 200-with-value, env-shape keyless 403, app-key 403 the surface, registration flow, UI pins (regen inline, grid4, job-finish reload). Measured (suite run 18): 137 passed / 1 deselected / 0 failed / 0 skipped, 197.37 s; live :8787: keyless tokens shows the minted value (= the 0600 file), wakeboarding un-keyed 10/10 strong, index/estimate un-keyed 401, app-connect flow verified (app row keyless-shown, app key 403 surface, own target 200), wiped to pristine standalone.
…no scroller, real Estimate switch
The API-key row was '9rem minmax(0,1fr) auto auto': label + gaps + two padded buttons
≈ 394 px minimum inside a 271 px control column (25 rem panel minus the label column),
so the row sat ~120 px past the panel's right edge, worse with the overflow-y:auto
scrollbar. Fixed at the grid level:
- .appkey-line: minmax(6rem,9rem) | minmax(0,1fr) | min-content min-content, with
min-width: 0 on every track; #settings .set-sec > * { min-width: 0 } — a row can
never exceed the column it lives in (label + key ellipsize, buttons keep their word).
- Panel: min(27rem, 100vw - 2rem), overflow-x: clip, NO scroller (overflow-y:auto made
a clipping box that cut controls, stole ~10 px, and let the modal drift past the
viewport); sections fold to one column under 720 px (label above its controls).
- Appearance select fills its control column (its native popup opens in the top layer,
never clipped); the two-column label/control grid is preserved and pinned by measure.
- Estimate is a real toggle switch (native checkbox + track/knob, 2.6rem x 1.5rem,
2rem row system) gating the Estimate button, its forecast rows stacked directly
above the control line (#estimate-ctl) — no dead space, no popup on the main page;
a finished crawl re-reads the index list (no stale target rows).
- The gear ADOPTS the minted key it is shown (its own controls are admin-scope): Index
now / Estimate / Regenerate work from the panel with zero typing; a stored key the
server no longer recognizes is replaced by the value shown now. Main-page reads stay
key-free; an env-keyed deployment shows nothing and adopts nothing.
- Restored <p id='engine-says'> — my round-6 line deletion removed it and renderList()
threw at first paint.
Measured: suite run 19b — 137 passed / 1 deselected / 0 failed / 0 skipped, 189.48 s;
module runs 50 green; geometry gate 34 PASS (minted flow on the :8793 testbed: adoption,
toggle gating, estimate rows, Copy, Regenerate + old-key 401, stale-key recovery);
round-7 layout gate 42 PASS at 1280/900/640; live :8787 27 PASS (panel right 1196/1280,
key row 1182.2 inside it, control column aligned in all five sections, select fills the
column, toggle 41.6x24 px, no inner scrollers, no page horizontal scroll, keyless search
10 hits 10 strong, index/estimate un-keyed 401).
The vanilla UI becomes a Lit + Vite app (server/ui/src/, stable asset names app.js/embed.js/style.css/embed.css; ES modules, CSS custom properties; no TypeScript, no framework/router/state lib, no Node at runtime; the server speaks its published interfaces unchanged). - Settings gear: four sections with icons, cards, semantic badges; the roots line and the estimates description removed; Estimate rides beside Index now. - Header full-bleed (title left, gear right), centered search axis, footer instructions at the right edge. - Kind row: the all button is a TOGGLE — all six on, click clears all, click restores all; aria-pressed marks the state. Engine semantics verbatim. - Copy buttons work over plain http (async clipboard when present, hidden textarea execCommand fallback). - Cleanup audit (post rounds 4-7): zero dead CSS selectors, zero unused exports; removed the roots slice write, the estimate-toggle listener, and the rule-less set-lines class. Contract-pinned remnants kept with reasons. - Contract suite + CI + compose + docs follow the new markup (137 tests). - Dev server (vite.config.js) is env-forced: loopback by default, LAN bind and host allowlist opt-in. Measured on ai-ubuntu-computer (x86_64, Ryzen 9 7950X): vitest 54/54; vite build 287 ms (app.js 65.53 kB / 20.36 gzip); contract suite 137 passed, 1 deselected, 0 skipped, 172.19 s on the cleaned tree. Nextcloud app 0.4.3 (media-type chips + live index counts) gate-verified on the pilot stack.
…ged shape - docker-compose.all-in-one.yml: the shipped chain merged into ONE file — Nextcloud + server (CUDA build target) + LAN binds on both web ports + a named app index target + the --profile dav mount container. Env names unchanged; the header prints the launch chain and how to peel every layer (CPU-only, loopback-only, data vs WebDAV shape, no app target). - deploy/README.md: the operator page — compose file map, every env var with its effect, launch chains (example values only), the measured rules (chown 990, ghost-row prune, single uvicorn process, 503-warming honesty, 409 one-writer lock, 143/137 stop semantics, GPU-is-measured honesty), and the API surface table with scopes, pointing at server/docs/API.md as the canonical contract. - server.yml: the compose-schema step validates the all-in-one chain with the full synthetic env chain (15th schema shape; RFC 5737 addresses, /srv paths). Measured on this box (x86_64, Ryzen 9 7950X): host schema checks full / no-profile / dav-only all OK; resolved config shows the CUDA target, the nvidia reservation, app=/media/app:/data/index-app, four port mappings, the ro + rslave binds; the exact CI line ran clean on the host. No server/ui code changes — the contract suite is untouched by this commit (docs + example compose + a CI validation line).
Full-surface audit (Python 2,776 L, UI 2,447 L, Dockerfile, compose, CI): the implementation was disciplined — zero unused imports/defs, zero dead CSS, all stages/vars referenced, no engine duplication. Applied changes: - engine: drop IndexService.sync_code (zero refs) and SearchService.folders() + _folders_fields respawn replay (the server's scope is argv at spawn) - jobs: counters ride the live job record only (_live_counts) — the per-event events list + _counts re-scan were a second source of the same numbers; a 1.8M-file job no longer buffers every event twice. Test pin adapted to the real paths, same assertions - api: make_runner — the JobRunner knobs (ceiling, rotation, log name, reload→prewarm) existed twice verbatim; now one source - CI: the byte-identical vendor-seeding step in both jobs is one YAML anchor (PyYAML-verified, run scripts md5-identical) - ui: 11 internal-only export keywords demoted (self-registering components, media.js helpers — zero importers) - docs: DIGUP_SERVER_STATUS_TTL_S documented in .env.example + deploy README Gates: py_compile OK; UI units 54/54 (host + in-image); vite build clean; contract suite on the final tree: 137 passed, 1 deselected, 0 skipped, 173.56 s. Architecture untouched: passthrough, engine source of truth.
Ocean572
added a commit
to Ocean572/DigUp
that referenced
this pull request
Nov 10, 2026
…r — FastAPI adapter, Lit+Vite UI, Docker/compose, contract suite, ghcr images 68 commits, additive-only: server/**, deploy/**, docs/implementation/**, .github/workflows/server.yml, .dockerignore, .gitignore appends. Engine untouched (Sources/**, Package.swift, scripts/**, shell/**, website/**, worker/**, evals/**, Tests/**).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add self-hosted Docker server for DigUp
This PR introduces a self-hosted Docker server for DigUp, enabling users to search and browse their media library via a lightweight web interface or REST API.
The server operates as an adapter around the existing DigUp engine rather than implementing a separate search system, ensuring consistent search behavior across desktop, CLI, and server environments.
Closes #183
Key Features
/api/v1.Dockerfileanddocker-composesetups. Data paths (index,models,config, job logs) use bind mounts to ensure container recreation never destroys index state.:latest) and CUDA (:latest-cuda) images to GHCR with pre-configured Compose override files. Opt-in NVIDIA GPU acceleration is measured and validated on physical hardware.Once running, the interface is accessible at:
http://127.0.0.1:8787/How It Works
The server delegates all indexing and search logic to the underlying DigUp engine via its published interfaces (
search-service/index-serviceJSON-lines protocols, standard CLI, and--jsonoutput).SearchServiceand the HTTP API on identical database snapshots.Testing & Performance Benchmarks
All performance metrics were gathered on physical Linux x86_64 hardware (CI hosted runners and a host running a Ryzen 9 7950X, 64 GB RAM, GTX 1070).
1. Automated Test Suite
2. CI Pipeline (
server.yml)swift test), CLI headless checks, contract tests, search evaluations (R@1 0.76 / R@5 0.88), Compose schema checks, and gracefulSIGTERMcontainer termination (exit 143).Sources/DigUpKit/ModelFiles.swift.3. Docker & GPU Validation
4. Large-Scale Library Benchmark (131,972 Indexable Files)
Validated against a 187,055-file real-world library on USB storage over three sequential jobs:
POST /api/v1/index). Single-writer locks correctly returned409on concurrent index requests.Continuous Integration & Delivery
The
server.ymlGitHub Actions workflow triggers on modifications to server code, deployment files, engine dependencies, or scripts::latest(CPU) and:latest-cudatags toghcr.ioformainbranch updates, as well as feature-branch tags (:feat-server/:feat-server-cuda). Unused interim tags are routinely cleaned up via registry hygiene steps.Known Limitations
docker-compose.lan.ymloverride. Public internet usage requires an authenticating reverse proxy.x86_64only. Other architectures (e.g., Apple Silicon) build natively viadocker compose up -d --build.cuda). AMD/Vulkan setups are left for future PRs.Quickstart Guide
Option 1: Local Build (Any Architecture)
cd deploy docker compose up -d --buildOption 2: Pre-built CPU Image (x86_64)
cd deploy DIGUP_INDEX_DIR=/path/to/index \ DIGUP_MODELS_DIR=/path/to/models \ DIGUP_MEDIA_DIR=/path/to/media \ DIGUP_IMAGE_TAG=latest \ docker compose -f docker-compose.yml -f docker-compose.ghcr.yml up -dOption 3: Pre-built NVIDIA GPU Image
Option 4: Home LAN Exposure (Unauthenticated)
Access the UI at
http://127.0.0.1:8787/. Trigger indexing manually via the UI orPOST /api/v1/index.PR Footprint
server/IMPLEMENTATION_STATUS.mdanddocs/implementation/.