libvncserver: add server-side support for the Open H.264 encoding

[CMGS]

Implements the server side of the Open H.264 encoding (50), as requested in #669. The TigerVNC viewer and noVNC already decode this encoding, and QEMU recently gained a server-side implementation.

Design

LibVNCServer does not encode. The application supplies complete, pre-encoded H.264 access units through a new screen hook:

rfbBool getH264FrameHook(rfbClientPtr cl, char **frame, size_t *frameSize);

This fits the cases where H.264 pays off most — hardware encoders, Android/scrcpy capture, media pipelines that already produce H.264 — without adding a codec dependency to the library. A software-encoder integration could be layered on top of the same hook later.

Key points:

• Non-blocking contract. The hook returns FALSE when it has no new access unit for the client. When a new frame becomes available, the application calls rfbNotifyH264FrameAvailable(), which wakes exactly the clients streaming H.264; damage tracking of clients on other encodings is not disturbed. Nothing blocks the update path, which the threaded loop runs while holding the client's sendMutex.
• Streaming subscription. A client whose preferred encoding is Open H.264 keeps its requestedRegion, so pushing a 30-60 fps stream does not cost one FramebufferUpdateRequest round trip per frame. The subscription ends when the client selects another preferred encoding via SetEncodings.
• Per-client streams. The hook is called per client; applications keep a per-client cursor in clientData, and every client's stream starts with SPS/PPS + IDR. Clients consume at different rates or resync to a keyframe independently.
• Clean fallback. Encoding 50 is only accepted when the hook is set; a viewer offering H.264 to an application without a frame source degrades to its next listed encoding.
• ABI. The new rfbScreenInfo/rfbClientRec fields are appended at the end of the structs.
• Access units routinely exceed UPDATE_BUF_SIZE, so the rectangle payload (big-endian u32 length, u32 reset flags, then the access unit) is written directly to the socket after flushing updateBuf.

The first commit fixes busy loops when deferUpdateTime is 0 — the natural setting for streaming applications, whose frames are already paced: the clientOutput wait/defer paths spun with zero-length sleeps, and rfbRunEventLoop's default select() timeout derived 0 from deferUpdateTime, pinning one core in the listener thread (or the rfbProcessEvents loop) even with no clients connected.

Testing

• cmake build and ctest pass on macOS (clang) and Ubuntu 22.04 (gcc) with no new warnings.
• A protocol-level harness (test server feeding synthetic access units through the hook, plus a raw-socket RFB client) verified on both the threaded and the select()-based event loops that:
  • after SetEncodings [50] and a single FramebufferUpdateRequest, every pushed access unit arrives as a full-frame rect with exact framing — x=0, y=0, w/h = screen size, encoding 50, big-endian length/flags, payload byte-identical and in order — with no further update requests;
  • a raw-only client connected to the same server keeps receiving ordinary Raw rects and never sees encoding 50;
  • a hostile client that selects encoding 50 but only ever requests a 1x1 region still receives all pushed frames and leaves the server at 0% CPU while connected and idle (the streaming path drops the whole modified region per update, so partial requests cannot pin the update loop);
  • with deferUpdateTime = 0 and no clients, the server idles at 0% CPU (was 100% in the listener thread before the first commit).
• The API is exercised by scrcpy-rfb, a bridge that passes Android scrcpy H.264 streams through to TigerVNC (vncviewer -PreferredEncoding=H.264) and decodes to Tight/JPEG for ordinary clients.

Happy to adjust naming or split things differently if you prefer.