Let several healthy DNS servers share one address #729

Activity

1. privateip commented on Oct 25, 2026

   

   privateip

   on Oct 25, 2026

   Contributor

   More actions

   The Galactic side of this, datum-cloud/galactic#1086, is in review as a three-PR stack: datum-cloud/galactic#1092 (API), datum-cloud/galactic#1093 (datapath) and datum-cloud/galactic#1094 (router). It is not merged or released yet, so nothing here can use it today.

   Once it ships, network services opts each resolver entry in on the ServiceEndpoint it creates:

   spec:
     distribution: Spread    # default Single keeps today's behaviour
     healthCheck:
       port: 8053            # required for a udp endpoint: a TCP port every resolver listens on
       intervalSeconds: 5
       timeoutSeconds: 2
       unhealthyThreshold: 3
       healthyThreshold: 3

   What Galactic does with it:

   • Every ready resolver on a consumer's node, up to 16, serves the one address. Each UDP query picks a resolver at random, so one client socket spreads too, and each TCP connection stays on the resolver it started on.
   • Each node probes its own resolvers over TCP, from the host-side interface's link-local address to the resolver attachment's IPv6 address, and reports each one's decision in the endpoint's status.producers. Where interface traffic policy is enforced, the resolver's policy has to allow that probe from fe80::/10.
   • An Unhealthy resolver gets no new queries but finishes the ones in flight. If every resolver on a node is Unhealthy, all of them keep taking queries.
   • healthCheck without Spread, and Spread without healthCheck, are refused by the API server.

   For a rollout that loses no lookups, the resolver should fail its health check first, keep serving for at least intervalSeconds × unhealthyThreshold + timeoutSeconds plus a few seconds for the decision to propagate, and then stop.

   Rollout order on the Galactic side is galactic-cni first, then the CRD and router. Before rolling back either component, set endpoints back to Single. I will update this issue when the stack merges and when it is in a release.
2. scotwells

   added a parent issue on Oct 29, 2026

   • Connect private networks to Private DNS automatically #777
3. scotwells

   added the

   Enhancement

   issue type on Oct 29, 2026
4. scotwells commented on Oct 29, 2026

   

   scotwells

   on Oct 29, 2026

   ContributorAuthor

   More actions

   The description now explains the problem and expected outcome in plain language. The original report is preserved below for the observations, diagnostics, and implementation details behind this work. It describes the situation when the issue was reported; later comments and the issue state record subsequent progress.

   Original report and engineering evidence

   Summary

   Each Private DNS address can be served by only one resolver on a node. To run two resolvers per node, each one needs its own address, and clients decide how to use them.

   Why it matters

   Many clients send almost all lookups to the first address and use the second only after a timeout. One resolver does most of the work, and a rollout of that resolver interrupts lookups until clients time out. In one measured rollout, the first address failed lookups for about 75 seconds.

   What we want

   • One Private DNS address can be served by several resolvers on the same node.
   • Each resolver can declare a health check, which network services passes to Galactic so that Galactic sends lookups only to healthy resolvers.
   • Existing configurations, with one resolver per address, keep working unchanged.

   Acceptance criteria

   • A resolver entry can select several resolvers per node, and startup no longer refuses it.
   • A health check set on a resolver entry reaches the service endpoints that network services creates.
   • The runbook explains how to move from one resolver per address to several behind one address.

   Depends on Spread traffic across several producers of a private service on one node.

   Related: Deliver Private DNS for Galactic VPC
5. scotwells

   changed the title [-]Serve one Private DNS address from several resolvers on a node[/-] [+]Let several healthy DNS servers share one address[/+] on Oct 29, 2026