Repository navigation
Private service routes can fail silently after a roll or on IPv4 #986
Description
Private service routes from #851 have follow-ups that did not block the review. Each can leave a service unreachable with no error, or keep access longer than intended.
- A router binary that rolls before the agent on a node cannot program service routes until the agent reloads.
- Some services stay unreachable with no error and no metric, because a missing next hop falls back to the old behavior.
- A pod with any default route on another link loses its VPC default route.
- Persistent access has no limit on who may set it.
Done when each is fixed, or documented with its roll order or trust boundary. The detail is in the first comment.
Related to #851
Activity
ecv commentedon Oct 22, 2026
Detail, as of #851 at d67c997:
serviceroutemap.OpenPinnednow fails hard ifservice_peer_tableis not pinned, and the pin exists only after the new usid program loads. Document the roll order, or treat a missing pin as the old fallback. Nothing tests the missing-pin case.attachmentNextHopreturns nil on a family mismatch or a missingPodSubnet, andSetPeer(nil)falls back todest_addr. IPv4 virtual services and attachments with several subnets stay unreachable, and the new alert counts only programming errors. Return an error or count a metric.cni/netns.go, any default route on another link suppresses the VPC default, including blackhole, unreachable and ECMP defaults withLinkIndex0. Require a unicast default withLinkIndex > 0.ServiceRouteAuthorization.persistentdrops the node-side deadline backstop. Only RBAC on ServiceRoutePolicy limits who sets it. State that only the network control plane writes these.TestPersistentPathSurvivesControllerOutageAndWithdrawscovers only a running router.runbook_urlanchor#galacticservicerouteprogrammingfailingdoes not exist in the infra galactic runbook.ecv commentedon Oct 22, 2026
From #851 at a28836b: when a pod's veth disappears,
recoverableEntriesreturns empty entries,repairDesiredcloses the node's policy gate, and every service route on that node is off until the whole policy is rebuilt. Pod churn triggers this. Withdraw only the affected intent's entries. Separately,clearWithdrawnAdvertisementclears the attachment's node, interfaces and PodSubnet. A cache-lag NotFound after a fast delete and recreate makes that status flap until the next reconcile.