feat(kubernetes): retry disrupted Job pods

[steve-bako]

Problem

Tangle renders every Kubernetes container execution as an Indexed Job with backoffLimitPerIndex: 0 and maxFailedIndexes: 0. A pod deleted by scheduler preemption, the eviction API, or node loss therefore fails its index immediately and the Job never creates a replacement.

This is distinct from a component returning a non-zero exit code: infrastructure disruptions often have no useful application exit code and carry the Kubernetes DisruptionTarget=True condition.

Observed production examples:

• scheduler preemption: DisruptionTarget=True, reason PreemptionByScheduler; a higher-priority CoreDNS pod displaced a running Tangle task
• autoscaler eviction: DisruptionTarget=True, reason EvictionByEvictionAPI

See Shopify/oasis-backend#491.

Change

Add an opt-in task annotation:

annotations:
  tangleml.com/launchers/kubernetes/job/disruption_retries: "1"

For opted-in Kubernetes Jobs:

• set backoffLimitPerIndex to the requested bounded retry count (0–10)
• Count a pod with DisruptionTarget=True, allowing the Job controller to replace it until that budget is exhausted
• FailIndex immediately when the main container exits non-zero, so deterministic user-code failures are not rerun
• set podReplacementPolicy: Failed, as required with podFailurePolicy

The default remains exactly zero retries with no pod failure policy, preserving existing behavior for every task that does not opt in.

Safety

• Opt-in only; no global rerun behavior.
• Values are validated as canonical integers in [0, 10] and fail closed.
• Application failures remain fail-fast.
• This is a pod replacement within the same Kubernetes Job and Tangle execution, not a sibling pipeline run.
• Components opting in must be restart-safe because a disrupted attempt may have produced partial external side effects.

Validation

• PYTHONPATH=. uv run --frozen pytest -q — 475 passed
• Black check passed
• Focused tests verify default behavior, disruption classification, application fail-fast behavior, and invalid inputs

Remaining validation

Draft pending a real-GKE canary that deletes/evicts an opted-in test pod and confirms one replacement is created under the same Job before this is enabled on production workloads.

[steve-bako]

@Ark-kun When you have a chance, could you review the API/Job-policy shape? This is deliberately draft and opt-in pending a real-GKE disruption canary. The exact production DisruptionTarget evidence and safety constraints are in the PR body; full suite is green (475 tests).

[steve-bako]

Follow-up pushed: added the user-facing annotation/restart-safety contract to README and an assertion over the exact serialized Kubernetes Job spec (backoffLimitPerIndex, podFailurePolicy, podReplacementPolicy, and maxFailedIndexes). Full suite remains green: 475 passed.

[steve-bako]

Marking ready for maintainer review. The feature is opt-in and defaults to the existing zero-retry behavior, so upstream merge alone changes no workload. Proposed validation sequence: merge upstream -> bump Oasis staging submodule -> run a real GKE DisruptionTarget replacement canary -> only then add the annotation to restart-safe production tasks.

[steve-bako]

New real-GKE evidence from staging run https://oasis.shopify.io/runs/01a000936be1d56d65cc: the Embed pod ran successfully for 10h15m, then its node became NotReady. Kubernetes set DisruptionTarget=True, reason DeletionByTaintManager, message Taint manager: deleting due to NoExecute taint. The Job had backoffLimitPerIndex: 0, immediately set failedIndexes: "0", and failed MaxFailedIndexesExceeded; there was no application or BigQuery error. The pod also had cluster-autoscaler.kubernetes.io/safe-to-evict: "false", confirming anti-scale-down annotation cannot cover node failure. The opt-in policy in this PR matches this exact condition and would create the bounded replacement. The checkpointed candidate will not be retried.