feat(replication)!: replicate live, over a websocket on the HTTP port

[xiaodoudou]

Replication is now live. A node keeps a link open to each peer it receives a resource from. The peer first sends what the node lacks, then each change as soon as it is written. When a link drops, it connects again by itself after 1, 2, 5, 10, then 30 seconds. The peer pings every 15 seconds, and a link that hears nothing for 45 seconds reconnects. Files follow half a second after their records, over the REST API as before. Stores on MongoDB or PostgreSQL can't hold a link open, so a node pulls from them every replication.pollInterval (30 seconds by default).

Records no longer use a TCP port of their own. They go over a websocket on the HTTP port, at /_replicator next to the REST API, so an HTTPS site replicates over wss://. A small upgrade router in lib/util/upgrades.js lets that socket share the server with the record updates socket. Without replication.secret a node doesn't listen (/_replicator answers 404) and opens no link.

Each node pulls, and the node a peer connects to only sends. A connecting peer can't write anything, even one that knows the secret. The type of a resource now says what this node does with it: upstream is only sent from here, normal and downstream are also pulled from the peers. With strictTypes, a node opens links only to peers marked upstream or normal. The CMS resources (_users, _groups, _settings, _sync, _xlsx) are left out of the global replication.peers list and replicate only when peersByResource names them, because every node makes its own admins group and localAdmin.

This is a breaking change. Upgrading means removing netPort, giving each peer the url of its REST API, setting replication.secret on every node, and listing on each node the peers it receives from. settleDelay and POST /replicator/sync/:resource/:id are gone. REPLICATION.md has a "Moving from 3.x" section with these steps.

The Replicator page moved to the CMS menu and is drawn with the plugin kit. It starts nothing now. It shows each resource that replicates, its direction, the peers it receives from with the state of each link (live, or reconnecting with the error and the next try), and the peers that pull it from this node. It refreshes every two seconds. A link that fails on every address of a host shows the code and the port, ECONNREFUSED (port 9997), where it used to show an empty AggregateError.

A record made on another node used to open with "All changes saved" above fields that can't be edited. The editor now shows "Read-only, made on" and the mid of that node there, with a lock, so it is clear where to edit the record.

REPLICATION.md is rewritten around live links and the types and directions as the code applies them, with a central node and site example and a screenshot of the page. A new section explains the record id (time, mid, then a growing number) and what the API and the admin do with a foreign record. SECURITY.md, ARCHITECTURE.md, RESOURCES.md, CONFIG.md, TESTING.md, index.d.ts and the changelog follow.

The first CI run failed on Linux and showed two bugs of the live links. A record written while a link was connecting could stay unsent until the next write. The clock moves before a write is stored, and a write that began before the link listened sends no event when it ends, so the node now looks again for a moment when its clock shows a change its store does not show yet. A link still in its handshake also could not be stopped, so closing a node waited up to 30 seconds. The handshake can now be cancelled, and restart() connects again at once instead of after the 1 second retry delay. A new test stops a node whose peer never answers.

New tests: two nodes replicating live (catch-up, changes as they are written, reconnection after the peer stopped, files, status on both sides), which resources a node pulls, the link error text, the page component, and security tests where a fake peer sends crafted frames to a node that pulls from it. Two nodes were also run by hand with articles going one way, signups the other and notes both ways, each change arriving within two seconds. One peer with an unknown host and one refused port were added to check the page. On Linux (a clone of the branch in WSL, the steps of CI): ESLint is clean, the backend suite passes 1983 tests, the frontend suite 2873, the integration suite 59, and the build succeeds. The security and live replication tests ran 20 times in a row on Windows without a failure; before the fix about one run in ten failed.