Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

70 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

signals-protocol

The shared wire contracts of the zndx signals federation — the protocols by which the sibling projects' engines (Ægir, Atelier, Gaius) invoke each other's capabilities while remaining independently operable. Patterned after the KServe Open Inference Protocol repository: versioned proto packages + a specification document per protocol, evolved additively, adopted by reference (each project vendors this repo as a submodule and generates bindings from proto/).

Why this exists

Measured 2026-07-03, minutes after the first live cross-engine call: the per-project engines mirror each other's message shapes, but gRPC method paths embed package+service — aegir.engine.AegirEngine vs atelier.engine.AtelierEngine — so a foreign stub gets UNIMPLEMENTED against a wire-identical peer. "Identical wire contract" requires an identical service path, not just identical messages. Each engine therefore registers the shared zndx.engine.v1.Engine service additionally, beside its native service: the native service remains the project-internal surface; the shared service is the federation face. One stub, any engine.

Protocols

package spec status
inference (OIP) specification/protocol/oip_stream.md KServe messages plus ModelStreamInfer: the bi-directional model hop. A forwarder relays it; only the hosting engine dials vLLM
zndx.engine.v1 specification/protocol/engine_grpc.md v1 — Complete + Status + Remediate + Yield + ServerQuery + RecordLineage + WatchWorkload + Announce (TTL’d S2S join); operating profiles (instruct = same model, effort low — capabilities.md); warehouse tx_id is RFC 9562 UUIDv7 (tx_id.md); data products on shared RustFS (data_products.md) with SHACL Core Aspects (NodeShape / ProductSpec / AspectBinding) and Agenda attachments; primary UI discovery in surfaces.md
zndx.scheduler.v1 specification/protocol/scheduler_grpc.md v1 — federated scheduler capability (queues, policy, projection, queue-share requests). Lab backend: YuniKorn; not a vendor in the service name
zndx.supervision.v1 specification/operations/nautilus_supervision.md v1 — the supervision grammar: supervision tree, state machines with phase gates and horizons, objectives on the final surfaced result, the position vocabulary engines report and the ledger stamps, and the Nautilus supervisor service. Grammar here; a project ships an instance (no project is named in the protocol)
coordination Activities specification/protocol/coordination_activities.md inter-project intent with a lifetime as Signals Airflow runs: zndx.engine.v1.Activity + Scheduler/{Declare,Renew,Release,List,Watch}Activity + supervision ActivityEvent; two hops, never one — local processes → local engine → Signals engine → Airflow
zndx.agent.v1 specification/protocol/agent_grpc.md v1 draft (2026-09-09) — agents as the third capability class (Holland / JADE: the bounded assemblage of runtime + tools + workspace + the model capability it thinks with): ListAgents, Run (stream), Cancel, GetRun; Harbor --agent/--model alignment; runs are Activities; trajectories (ATIF) are data products
zndx.verify.v1 — RESERVED: verification artifacts on the wire (reasoner certificates, kvasir proof DAGs — the rase_types direction from Gaius)

Operations (identity, secrets, process coordination)

Wire protocols alone are not enough to use Signals core services or to coordinate multi-engine host work. Adopters must follow:

Document Purpose
specification/operations/kerberos_and_secretspec.md Binding Kerberos principal catalog, SecretSpec allowlists, kinit process wrappers, Ranger onboarding
specification/operations/minifi_sentinels.md Binding design — MiNiFi C++ sentinels (C2 + OTel); Knative Serving scale-to-zero + YuniKorn admission on RKE2
specification/operations/nautilus_supervision.md Binding design — Nautilus, the deterministic external supervisor (Erlang-style tree, phase gates, Brier-scored ledger, write-ahead bookkeeping); Overwatch stays engine-native reasoning. Each project ships a zndx.supervision.v1 instance

Reference implementation: weathership/signals is the first federation project on Kerberos + SecretSpec, and vendors MiNiFi C++ (components/minifi-cpp) and YuniKorn core (components/yunikorn-core) for sentinel coordination. Sibling engines should implement these procedures—do not invent a parallel long-term identity or process-control path.

Adopters

project native service shared service gRPC port
signals zndx.scheduler.v1.Scheduler (platform) zndx.engine.v1.Engine :50551 (lab lattice)
aegir aegir.engine.AegirEngine zndx.engine.v1.Engine :50151
atelier atelier.engine.AtelierEngine zndx.engine.v1.Engine :50251
gaius (see gaius FEDERATION.md) zndx.engine.v1.Engine :50051
hermes-agent ACP / plugins planned (client of core + engines) —

GPU co-tenancy rides the shared advisory lease dir /tmp/zndx-gpu-leases (per-GPU-set lock files, project-tagged owners) plus each engine's authoritative nvidia-smi probe. Cross-project admission is YuniKorn (sentinel is the Application). Yield is C2 HTTP → Engine/Yield gRPC. See specification/operations/minifi_sentinels.md.

Evolution rules

  • Additive-only within a version: new fields get new numbers; nothing is renamed, renumbered, or removed. Breaking changes mean a new versioned package (v2).
  • Capabilities, not models: requests name abilities ("instruct", "referee"); the serving engine chooses and reports the model.
  • Engine-private details stay private: internal serving ports, log paths, and process details do not cross this boundary.
  • Changes land here first, by PR/commit visible to all three projects, then propagate by submodule bump — a shared proto is only shared if there is exactly one of it.

Relationship to the Open Inference Protocol

OIP is the model hop. proto/inference/open_inference_grpc.proto is the KServe message set plus ModelStreamInfer, a bi-directional stream of those same messages. The engine that hosts a capability dials vLLM in private and emits OIP frames. An engine that does not host it forwards that RPC to a peer chosen by Engine/Status and ModelReady — pairwise, no gossip — and does not advertise the forwarded capability. Frame tensors, cancel, and progress are specification/protocol/oip_stream.md.

zndx.engine.v1.Complete remains the unary capability convenience over the same tensors. It is not a second streaming protocol.

Codegen

Python (the current adopters):

python -m grpc_tools.protoc -Iproto \
    --python_out=<dst> --grpc_python_out=<dst> \
    proto/zndx/engine/v1/engine.proto

Generated code is vendored per-project (committed beside the native stubs); the proto here is the single source of truth.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors