Skip to content

v5.0: modular refactor, integrated recorder, automated multi-Pi rollout, SpecKit + personas - #20

Merged
bjoernhoefer merged 9 commits into
mainfrom
feature/refactor
Nov 12, 2026
Merged

bjoernhoefer merged 9 commits into
mainfrom
feature/refactor

Conversation

@bjoernhoefer

Copy link
Copy Markdown
Owner

Summary

e2proxy v5.0. The 14 k-line e2proxy.py becomes a modular package with an open extension interface. e2recorder is integrated as an optional extension, and rollouts to one or more Raspberry Pis are automated via the pi-ops standard. SpecKit is initialised, with a constitution and five reviewer personas.

1. SpecKit, constitution, personas

  • .specify/memory/constitution.md: principles Reliable first · Slim · Fast & light on the Pi · Modular with open interfaces · Every bug becomes a test · Durable operations.
  • .github/agents/: DVB & Streaming, Raspberry Pi, Python developer, UI developer (screenshot verification), Documentation & QA (regression tests, minor/major decision, version bump, Help-page changelog).
  • AGENTS.md, .github/copilot-instructions.md (pi-ops markers), specs/001-modular-refactor/ (spec/plan/tasks).

2. Modular refactor

  • e2proxy/ package with one concern per module, started via python -m e2proxy (--selfcheck still works). The split was mechanical, done with an AST-based tool. Page/API parity was verified against the old monolith.
  • e2proxy/extensions.py: @route(method, path, extension) + Extension(name, start, enabled). Routes of a disabled extension return 404. The handler dispatches to extensions first and falls back to the legacy routes.
  • Build id = VERSION+branch.sha7, injected at image build time.

3. Integrated recorder (optional, off by default)

  • e2proxy/recorder/ (engine with injectable I/O, pure matching, atomic JSON store, routes, UI): about 1.2 k lines, vs. 4.3 k in the standalone service.
  • EPG overview: marks planned/recording/done/skipped/matching airings. The detail popup offers record once / record all episodes / skip.
  • Settings → Recorder: options, remote e2proxies (shared tuner pool), series table (regex, keep-last, on/off), planned & recent recordings.
  • Ported behaviour includes:
    • dedupe, also when the EPG shifts start times
    • a skipped airing stays skipped
    • back-to-back lead time
    • late-start tolerance
    • keep-last cleanup (.ts + .nfo)
    • recovery after a restart
  • Migration: copy series.json / recordings.json to /data/recorder/.

Settings → Recorder tab
EPG overview with recorder marks and the detail popup

4. Automated rollout, multi-Pi ready

  • Dockerfile (arm64, ffmpeg installed at build time instead of apt-get at every start, HEALTHCHECK) and selftest.py (health, UI, WebUI self-check, libx264/aac transcode).
  • test.yml and build-deploy.yml follow the pi-ops templates. The deploy job is a matrix over deploy/targets.json; the per-host steps are unchanged and fail-fast: false keeps one failing Pi from blocking the others. workflow_dispatch takes hosts=all|none|a,b.
  • deploy/resolve_targets.py validates the targets file and sanitises the deploy user.
  • reload_e2proxy.sh is replaced by deploy/manual-deploy.sh, an emergency path that only swaps the service after the selftest passes.

Bugs found and fixed during review (each with a regression test)

  • EPG wiped at start-up: a run with the receiver off replaced a valid cache with an empty XMLTV. The previous EPG is now kept.
  • Recorder: a recording that ended early (ffmpeg or receiver died) was marked done and counted for keep-last, which could delete a good older episode. It is now failed, and the partial file is kept.
  • UI: the remote-proxy and OpenWebif UA textareas inherited the 380 px min-height of the JSON editor (seen in the screenshot review).

Tests (108)

  • Package: every module imports standalone; no import-time side effects; build id.
  • WebUI: built-in self-check plus node --check on every inline script of every page, with the recorder on and off.
  • HTTP smoke: pages, APIs, /api/health under 1.5 s, unknown routes 404, recorder toggle.
  • Changelog regressions:
    • v4.3 e2length MIN:SS
    • v4.6 tuner refcount and stale epoch
    • v4.9 probe requests
    • Wilson interval
    • EPG keep
  • Recorder engine with a fake backend and clock; deploy targets.
  • Verified locally: ruff, pytest, --selfcheck, actionlint, docker build + selftest.py (healthy).

Persona review

Persona Result
DVB & Streaming Tuner refcount/sharing is unchanged and covered by tests. The recorder uses the existing /api/record/* path (shared tuner, profiles). Transcode codec path is checked in the selftest.
Raspberry Pi arm64 image, no apt at start, nothing new on the request path, recorder loops sleep and cost nothing when off. Note: mem_limit is not enforced on Pi OS without cgroup memory, so pi-deploy watches RSS.
Python Stdlib only, atomic state writes, injectable I/O, no import cycles (tested), ruff clean.
UI Screenshots above. Found and fixed the textarea height. Question for you, see below.
Docs & QA Major → v5.0 (architecture, deployment and data layout change). Changelog entry on the Help page; README, AGENTS.md and spec updated.

⚠️ OFFEN

  • UI decision: the EPG popup now has two sections, "Recording" (existing: record this airing now) and "Recorder" (new: series rules). Merge them or keep them separate?
  • raspberrypi_management: set e2proxy to status: active, install the deploy runner (deploy-e2proxy) on wien-streamer, retire the e2recorder inventory entry after migration.
  • Upstream the multi-host deploy matrix into the pi-ops build-deploy template (this repo deviates from the template, as documented in the workflow header).
  • GHCR package e2proxy visibility/access for the Pi.
  • One-time migration on the Pi: new docker-compose.e2proxy.yml (image instead of mounted file) and copying the recorder JSON files (see README → Deployment).
  • Measure peak RSS while transcoding on the Pi (AGENTS.md resource profile has an estimate).
  • Follow-ups (spec 001): move legacy routes to extensions; remove dead e2probe is None branches; slimmer ffmpeg image (currently about 830 MB).

bjoernhoefer and others added 9 commits November 12, 2026 18:31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Behavior-preserving AST-driven split into domain modules (config, receivers,
streaming, epg, recordings, webui/*, http_proxy, app ...). Circular references
resolved via function-local imports; build id from environment.
Entry point: python -m e2proxy [--selfcheck].

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- e2proxy/extensions.py: route registry + extension lifecycle (open interface)
- e2proxy/recorder: optional e2recorder port (EPG marks, settings tab)
- tests: package/import guards, webui JS parse, HTTP smoke, regressions, recorder engine

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Dockerfile (ffmpeg baked in, build id from git sha), selftest.py
- test.yml / build-deploy.yml per pi-ops template; deploy as matrix over deploy/targets.json
- deploy/manual-deploy.sh replaces reload_e2proxy.sh (emergency path with selftest gate)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A start-up run with the receiver switched off replaced a valid cache with an
empty XMLTV; the guide stayed blank until the next successful run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- a backend recording that stops before the broadcast ends was reported done
  and counted for keep_last (could delete a good older episode)
- recorder remote-proxy and OpenWebif UA textareas inherited the 380px
  min-height of the JSON editor (found in UI screenshot review)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Pis pull without credentials; check the pushed digest right after the
build and point to the package settings. README: switch compose before the
first pi-deploy run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bjoernhoefer
bjoernhoefer merged commit d437247 into main Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant