Replace pull-config with TypeScript agent by cmyers-mieweb · Pull Request #407 · mieweb/opensource-server

cmyers-mieweb · 2026-07-15T14:45:20Z

Resolves: #357

Replaces the cron-driven bash pull-config system with a TypeScript Node.js agent that runs every 30 seconds via a systemd timer.

Key changes:

New agent/ package: TypeScript sources (config, state, api, system, apply), EJS templates moved from create-a-container/views/, systemd service+timer units
Manager: new Agents DB model/migration, POST /api/v1/agents check-in endpoint with ETag/304 support, GET /api/v1/agents admin list, utils/agent-config.js config snapshot builder
Client: AgentsListPage with service health badges, sidebar nav entry, query/types
Removes: pull-config bash scripts, EJS template rendering from manager (templates router, ejs dependency, views engine setup)
Updates docs, Dockerfiles, Makefiles, and OpenAPI spec

Resolves: #357 Replaces the cron-driven bash pull-config system with a TypeScript Node.js agent that runs every 30 seconds via a systemd timer. Key changes: - New `agent/` package: TypeScript sources (config, state, api, system, apply), EJS templates moved from create-a-container/views/, systemd service+timer units - Manager: new `Agents` DB model/migration, `POST /api/v1/agents` check-in endpoint with ETag/304 support, `GET /api/v1/agents` admin list, `utils/agent-config.js` config snapshot builder - Client: AgentsListPage with service health badges, sidebar nav entry, query/types - Removes: pull-config bash scripts, EJS template rendering from manager (templates router, ejs dependency, views engine setup) - Updates docs, Dockerfiles, Makefiles, and OpenAPI spec

Copilot

Pull request overview

Replaces the legacy cron-driven pull-config bash system with a TypeScript/Node-based site agent that checks in to the manager every 30 seconds (systemd timer), receives a JSON config snapshot (ETag/304), and applies nginx/dnsmasq configuration locally. The manager gains an Agents model + API endpoints, and the web client gains an admin Agents status page.

Changes:

Introduces new agent/ package (TypeScript agent, local EJS templates, systemd service/timer, packaging).
Adds manager-side agent check-in + status listing (/api/v1/agents), config snapshot builder + ETag, and Agents DB model/migration.
Removes manager-side EJS template rendering routes and deletes the old pull-config/ cron/bash implementation; updates docs and images accordingly.

Reviewed changes

Copilot reviewed 54 out of 62 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
README.md	Updates top-level component list to replace `pull-config/` with `agent/`.
pull-config/README.md	Removes legacy pull-config docs stub (component retired).
pull-config/etc/pull-config.d/nginx	Removes legacy nginx pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-servers	Removes legacy dnsmasq forwarders pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-hosts	Removes legacy dnsmasq hosts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-dhcp-opts	Removes legacy dnsmasq DHCP opts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-dhcp-hosts	Removes legacy dnsmasq DHCP hosts pull-config instance script.
pull-config/etc/pull-config.d/dnsmasq-conf	Removes legacy dnsmasq main config pull-config instance script.
pull-config/etc/cron.d/pull-config	Removes cron schedule for pull-config run-parts.
pull-config/bin/pull-config	Removes legacy bash pull-config engine.
pull-config/.gitignore	Removes pull-config packaging artifact ignores (component retired).
mie-opensource-landing/zensical.toml	Renames docs nav entry from Pull Config to Agent.
mie-opensource-landing/docs/developers/release-pipeline.md	Updates release pipeline docs to build/package `agent` instead of `pull-config`.
mie-opensource-landing/docs/developers/pull-config.md	Removes legacy developer reference for pull-config (replaced by agent docs).
mie-opensource-landing/docs/developers/docker-images.md	Updates agent image description to reflect check-in agent.
mie-opensource-landing/docs/developers/database-schema.md	Extends ER diagram and schema docs to include new `Agents` entity.
mie-opensource-landing/docs/developers/agent.md	Adds new developer reference for agent architecture and protocol.
mie-opensource-landing/docs/admins/deploying-agents.md	Updates admin deployment guide from pull-config verification to agent check-in verification.
Makefile	Swaps top-level component list from `pull-config` to `agent`.
images/manager/Dockerfile	Updates comments around default env to refer to the new agent behavior.
images/agent/Dockerfile	Updates comments for dependencies/config to refer to the new agent.
create-a-container/utils/agent-config.js	Adds manager-side config snapshot builder + strong ETag computation.
create-a-container/server.js	Removes view engine + templates router wiring; simplifies SPA fallback routing.
create-a-container/routers/templates.js	Removes legacy template-rendering routes for nginx/dnsmasq.
create-a-container/routers/api/v1/index.js	Mounts new `/agents` router before CSRF guard.
create-a-container/routers/api/v1/agents.js	Adds agent check-in endpoint + admin listing with “online” computed status.
create-a-container/package.json	Removes `ejs` dependency from manager (no longer renders templates).
create-a-container/package-lock.json	Removes `ejs` transitive deps and updates lock metadata.
create-a-container/openapi.v1.yaml	Documents new `/agents` endpoints under API v1.
create-a-container/models/agent.js	Adds Sequelize `Agent` model with `(siteId, hostname)` uniqueness.
create-a-container/migrations/20260714120000-create-agents.js	Adds `Agents` table and unique composite index migration.
create-a-container/middlewares/index.js	Extracts and exports `isLocalhostRequest` helper used by agent auth.
create-a-container/Makefile	Stops packaging `views/` since manager no longer renders templates.
create-a-container/client/src/pages/agents/AgentsListPage.tsx	Adds admin Agents list page with online/offline and last check-in display.
create-a-container/client/src/pages/agents/AgentServiceBadges.tsx	Adds per-service status badges for agent-reported service states.
create-a-container/client/src/lib/types.ts	Adds typed Agent/AgentServiceStatus models to match new API responses.
create-a-container/client/src/lib/queries.ts	Adds React Query key + `listAgents()` query to call `/api/v1/agents`.
create-a-container/client/src/app/Sidebar.tsx	Adds admin-only “Agents” navigation entry.
create-a-container/client/src/app/router.tsx	Registers `/agents` route to render Agents list page.
agent/tsconfig.json	Adds TypeScript compiler configuration for the agent package.
agent/templates/nginx.conf.ejs	Updates nginx template to consume the new JSON snapshot shape.
agent/templates/dnsmasq/servers.ejs	Adds agent-side dnsmasq forwarders template.
agent/templates/dnsmasq/hosts.ejs	Adds agent-side dnsmasq hosts template.
agent/templates/dnsmasq/dhcp-opts.ejs	Adds agent-side dnsmasq DHCP options template.
agent/templates/dnsmasq/dhcp-hosts.ejs	Adds agent-side dnsmasq DHCP hosts template.
agent/templates/dnsmasq/conf.ejs	Adds agent-side dnsmasq main config template.
agent/src/types.ts	Adds shared request/response/config snapshot types for agent logic.
agent/src/system.ts	Adds hostname/IP/systemd ActiveState collection helpers.
agent/src/state.ts	Adds persisted state (ETag + per-service lastApply) storage helpers.
agent/src/index.ts	Adds agent oneshot main loop: check-in, apply config, repeat until 304.
agent/src/config.ts	Adds environment-based agent config loader (SITE_ID/MANAGER_URL/etc).
agent/src/apply.ts	Adds apply/test/reload orchestration for nginx and dnsmasq configs.
agent/src/api.ts	Adds HTTP client for POST `/api/v1/agents` with If-None-Match + auth.
agent/README.md	Adds agent package overview and links to admin/dev docs.
agent/package.json	Adds agent package manifest with build/start scripts and dependencies.
agent/package-lock.json	Adds agent lockfile for reproducible builds.
agent/Makefile	Adds build/package install flow (npm ci, tsc build, stage systemd units).
agent/contrib/systemd/opensource-agent.timer	Adds 30-second systemd timer unit.
agent/contrib/systemd/opensource-agent.service	Adds oneshot systemd service unit for the agent.
agent/contrib/postinstall.sh	Enables timer on package install and restarts when systemd is running.
agent/.gitignore	Ignores node/TS build and packaging artifacts for agent component.
agent/.fpm	Updates packaging metadata/dependencies and adds postinstall hook for systemd timer.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

- Use systemd StateDirectory=/STATE_DIRECTORY instead of Makefile-created state dir - Rework state.ts into a State class; only catch ENOENT/SyntaxError - Replace systemctl subprocess calls with the systemd D-Bus API (@jellybrick/dbus-next); keep nginx -t / dnsmasq --test exec validation - Roll back and best-effort re-reload when a service reload fails - Skip dnsmasq management until all required site fields are configured; mark SiteInfo fields nullable to match the manager model - Check in once more after MAX_PASSES so final apply results are reported - Use requireLocalhostOrAdmin for agent check-in auth; document why manual ETag handling is needed on POST - Replace boolean online flag with server-computed secondsSinceCheckin; client decides the offline threshold and renders drift-free timestamps - Docs: systemctl start for manual runs, drop the file tree listing, STATE_DIRECTORY env var

runleveldev · 2026-07-21T19:24:56Z

So far unable to root cause. Bootstrapping in docker compose is failing because the agent fails to apply the initial config.

Jul 21 19:20:01 manager node[914]: nginx: reload failed, rolled back: fast_xml_builder.default is not a constructor

To reproduce, set MANAGER_TAG=sha-ca620d0 in .env, make -C agent build, then docker compose up -d

The jellybrick fork's CommonJS bundle has a broken ESM-interop shim: its MessageBus constructor calls 'new fast_xml_builder.default(...)' where __toESM() has set .default to the whole module namespace, so any bus operation fails with 'fast_xml_builder.default is not a constructor'. This broke the agent's initial config apply during docker compose bootstrap. @particle/dbus-next is the original dbus-next codebase (native CJS, no bundler interop) with the vulnerable usocket/node-gyp/request chain removed and xml2js patched; npm audit remains clean.

Copilot

Pull request overview

Copilot reviewed 54 out of 62 changed files in this pull request and generated 2 comments.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

cmyers-mieweb · 2026-07-22T15:01:28Z

So far unable to root cause. Bootstrapping in docker compose is failing because the agent fails to apply the initial config.
Jul 21 19:20:01 manager node[914]: nginx: reload failed, rolled back: fast_xml_builder.default is not a constructor
To reproduce, set MANAGER_TAG=sha-ca620d0 in .env, make -C agent build, then docker compose up -d

I pushed a new commit dd77c50 that fixes the issue. The root cause copilot found was related to the following:
its MessageBus constructor calls new fast_xml_builder.default(...) where __toESM() has set .default to the whole module namespace, so every bus operation failed with fast_xml_builder.default is not a constructor during compose bootstrap. Swapped to @particle/dbus-next (the original dbus-next codebase, native CJS, with the vulnerable usocket/node-gyp/request chain removed); same API, npm audit still clean. Verified end-to-end in docker compose: nginx + dnsmasq apply, D-Bus reload/restart/SIGHUP all work, both services active with lastApply: success.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 54 out of 62 changed files in this pull request and generated 3 comments.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

runleveldev

I've ran several QA tests locally in the compose setup, including bootstrapping, killing services, and updating an existing system via the DEB packages. Everything seems in order. @cmyers-mieweb can you address Copilot's most recent reviews then we can merge.

Copilot

Pull request overview

Copilot reviewed 55 out of 63 changed files in this pull request and generated 1 comment.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

Comments suppressed due to low confidence (2)

agent/Makefile:40

make build prunes devDependencies in-place (npm prune --omit=dev), which can break local development and CI steps that expect TypeScript/@types to still be present (e.g. compose.yml runs make -C agent build). Consider pruning only in the staged install directory so the working tree remains intact.
create-a-container/middlewares/index.js:107
For non-localhost requests, requireLocalhostOrAdmin funnels through requireAuth, which (on API-key auth) populates req.session. Since the agent checks in every 30s and does not persist cookies, this can create an unbounded number of persisted sessions in the Sequelize session store. Consider authenticating Bearer API keys here without mutating the session object.

- agent/api.ts: add a 30s AbortSignal timeout to the check-in fetch so the oneshot unit fails fast instead of hanging when the manager stalls - agent/apply.ts: write config files atomically (temp file + rename) so a crash mid-write can't leave truncated configs on disk - agents.js: replace requireLocalhostOrAdmin with a checkinAuth that uses apiAuth/apiAdmin for remote agents, keeping error responses in the v1 JSON envelope; localhost bootstrap bypass unchanged - middlewares: isLocalhostRequest now also checks the first hop of X-Forwarded-For; drop the now-unused requireLocalhostOrAdmin - client: extract OnlineBadge into its own file (one component per file)

cmyers-mieweb · 2026-07-23T16:47:03Z

Remaining copilot suggestions have been implemented in cfcfda5

Copilot

Pull request overview

Copilot reviewed 56 out of 64 changed files in this pull request and generated 2 comments.

Files not reviewed (2)

agent/package-lock.json: Generated file
create-a-container/package-lock.json: Generated file

+  // Domains needing a TLS server block: any domain referenced by an HTTP
+  // service plus the site's own external domains.
+  const usedDomainIds = new Set();
+  for (const node of site.nodes || []) {
+    for (const container of node.containers || []) {
+      for (const service of container.services || []) {
+        const id = service.httpService?.externalDomain?.id;
+        if (id) usedDomainIds.add(id);
+      }
+    }
+  }
+  for (const d of site.externalDomains || []) usedDomainIds.add(d.id);
+  const externalDomains = await ExternalDomain.findAll({
+    where: { id: [...usedDomainIds] },
+    order: [['id', 'ASC']],
+  });


+function run(cmd: string[]): void {
+  execFileSync(cmd[0], cmd.slice(1), { stdio: 'pipe' });
+}


runleveldev

Copilot reviews have been over aggressive. I'm merging this despite them open.

# Conflicts: # agent/Makefile # create-a-container/Makefile # create-a-container/package-lock.json # create-a-container/server.js

cmyers-mieweb requested a review from runleveldev July 15, 2026 14:45

cmyers-mieweb mentioned this pull request Jul 15, 2026

Agent check-in #357

Closed

runleveldev requested a review from Copilot July 16, 2026 13:51

Copilot started reviewing on behalf of runleveldev July 16, 2026 13:52 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread agent/src/apply.ts

Comment thread agent/src/types.ts Outdated

Comment thread agent/src/index.ts

Comment thread create-a-container/routers/api/v1/agents.js

runleveldev requested changes Jul 16, 2026

View reviewed changes

runleveldev mentioned this pull request Jul 21, 2026

[Bug]: Uploads are very slow through the agent reverse proxy (ModSecurity scanning request bodies) #395

Open

Copilot AI review requested due to automatic review settings July 22, 2026 14:54

Copilot started reviewing on behalf of cmyers-mieweb July 22, 2026 14:55 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread create-a-container/routers/api/v1/agents.js

Comment thread create-a-container/client/src/pages/agents/AgentsListPage.tsx Outdated

This was referenced Jul 22, 2026

Default TUS uploads back to a single remainder PATCH (chunking stays an opt-in knob) morepriyam/pulse#114

Merged

upload: re-land chunked TUS PATCH with 32–64 MB chunks + connection reuse (successor to #87) morepriyam/pulse#89

Closed

Potential fix for pull request finding

2bc4be0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 22, 2026 20:57

Copilot started reviewing on behalf of cmyers-mieweb July 22, 2026 20:57 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread create-a-container/routers/api/v1/agents.js Outdated

Comment thread create-a-container/middlewares/index.js Outdated

Comment thread agent/src/api.ts

fix: build agent in the docker compose up pipeline

4c32ec4

Copilot AI review requested due to automatic review settings July 23, 2026 15:14

Copilot started reviewing on behalf of runleveldev July 23, 2026 15:15 View session

runleveldev previously approved these changes Jul 23, 2026

View reviewed changes

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread agent/src/apply.ts

Copilot AI review requested due to automatic review settings July 23, 2026 16:44

cmyers-mieweb dismissed runleveldev’s stale review via cfcfda5 July 23, 2026 16:44

Copilot started reviewing on behalf of cmyers-mieweb July 23, 2026 16:44 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

runleveldev mentioned this pull request Jul 23, 2026

fix(agent): remove ModSecurity, raise HTTP/2 and HTTP/3 upload buffers #423

Open

runleveldev approved these changes Jul 23, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into cmyers_issue357

c7b1ba9

# Conflicts: # agent/Makefile # create-a-container/Makefile # create-a-container/package-lock.json # create-a-container/server.js

runleveldev merged commit 477018b into main Jul 23, 2026
8 checks passed

runleveldev deleted the cmyers_issue357 branch July 23, 2026 18:04

Uh oh!

Conversation

cmyers-mieweb commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

runleveldev commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

cmyers-mieweb commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

runleveldev left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

cmyers-mieweb commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

runleveldev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmyers-mieweb commented Jul 22, 2026 •

edited

Loading