feat(k8s-launcher): explain empty logs when a pod was deleted before capture

[morgan-wowk]

Stacked on #325.

What this closes

#325 stops a deleted pod from turning FAILED into SYSTEM_ERROR. But it trades a loud-wrong-status for a silent blank: with the pod gone, _get_all_logs() recovers nothing, upload_log() persists an empty log, and the runs v2 page renders the bare text "No logs available" — indistinguishable from a task that simply produced no output. The Job launcher also can't help: launcher_error_message is an unimplemented return None, so no orchestration_error_message is recorded either.

New rules

• A vanished pod is reported as deleted, not as "no log yet". _get_log_by_pod_key now returns a _POD_DELETED sentinel on a 404, distinct from None (pod exists, HTTP 400, still initializing). _get_all_logs returns (logs, deleted_pod_keys) so callers can tell the two apart.

• When a deleted pod is the only reason there are no logs, the persisted log is an explanatory notice instead of an empty string:

  Logs are unavailable: the pod was deleted before its logs could be captured (for example, autoscaler eviction or Kubernetes garbage collection). The task's final status still reflects what happened; only the logs were lost.

  This flows straight into the existing log pane (the runs v2 page reads the persisted artifact for terminal executions) with no frontend change.

• The notice never masks a real result. It is substituted only when the merged log is empty and at least one pod 404'd. A pod that still exists and printed nothing stays empty; any recovered logs (even partial, when only some pods vanished) are persisted verbatim.

Why a pod vanishes before capture

Two independent causes, both seen in practice: the cluster-autoscaler evicting the node mid-run, and Kubernetes garbage-collecting a finished pod before the orchestrator reads it. Both surface identically as a 404 on read_namespaced_pod_log.

Tests

• tests/test_kubernetes_launcher_error_classification.py:
  • _get_log_by_pod_key returns the _POD_DELETED sentinel on 404 (still raises on 403/500).
  • upload_log/get_log persist the notice when the pod was deleted, keep an existing pod's empty output empty, and leave recovered (partial) logs untouched.
• Full suite green (uv run pytest).

Downstream

Shopify/oasis-backend pins this repo as the backend submodule; picking this up needs a submodule pointer bump (together with #325).

[morgan-wowk]

• feat(k8s-launcher): explain empty logs when a pod was deleted before capture #326 👈 (View in Graphite)
• fix(orchestrator): losing pod logs must not fail an execution #325
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Ark-kun]

The distinction between "empty log" and "missing log" can/should be done on the UI side.
The APi already returns log_text: "" vs log_text: undefined. If get_log returns None, the log_text is undefined. See

tangle/cloud_pipelines_backend/api_server_sql.py

Line 898 in 1b3e614

log_text: str | None = None