Skip to content

Orchestrator - Retry the Kubernetes API server errors #50

Description

@Ark-kun
kubernetes.client.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7512bc49-8a1e-4c57-a488-9cc9088b88c2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'cb8e61d0-d4ac-4497-9215-e39742c07fc8', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e0167ddd-e4d7-449d-b305-9dce12e027ee', 'Date': 'Sat, 06 Dec 2025 02:13:12 GMT', 'Content-Length': '154'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Internal desc = Internal error encountered.","code":500}

This is non-trivial since we do not want to stall the queue.

Other common errors are Kubernetes API Server connection errors.

Anther error (only happened once):

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/http/client.py", line 603, in _read_chunked
    value.append(self._safe_read(chunk_left))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 644, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(8493 bytes read, 7057 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 903, in _error_catcher
    yield
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 1028, in _raw_read
    data = self._fp_read(amt, read1=read1) if not fp_closed else b""
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/urllib3/response.py", line 1011, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
                                                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 473, in read
    return self._read_chunked(amt)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 609, in _read_chunked
    raise IncompleteRead(b''.join(value)) from exc
http.client.IncompleteRead: IncompleteRead(0 bytes read)

The above exception was the direct cause of the following exception:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions