[fix] Do not log an expected concurrent-clear key miss as an error

[huniu20]

Motivation

kv_batch_get resolves metadata and then fetches data, holding no lease across
the two steps:

  batch_meta = tq_client.kv_retrieve_meta(keys=keys, partition_id=partition_id, create=False)
  ...
  data = tq_client.get_data(batch_meta)

A kv_clear from another client can land in between, so get_data reaches a
storage unit whose key is gone. Both are public APIs and TransferQueue is built
for "fine-grained, concurrent data read/write operations", so concurrent readers
and cleaners are the advertised use case.

Reproducing this on current main — one process clears the keys another already
resolved metadata for:

  meta = client.kv_retrieve_meta(keys=keys, partition_id=p, create=False)  # 8 keys, ready
  ray.get(clear_from_another_process.remote(p, keys))
  client.get_data(meta)

  ERROR ... _handle_get error, fields=['data'], global_indexes=[0, 2, 4, 6]:
        KeyError: "StorageUnitData get_data: key 0 not found in field 'data'"
  ERROR ... Unexpected error from storage unit TQ_STORAGE_UNIT_8156e4cb: RuntimeError: ...
  ERROR ... get_data failed. ... storage_units=['TQ_STORAGE_UNIT_8156e4cb', ...], ...

Three ERROR lines across two processes for one expected event. Two problems:

• Severity. The condition is expected under concurrency and retryable, but at
  ERROR it is indistinguishable from a real storage fault and trips log-based
  alerting. The controller already treats the analogous case as a warning:
  logger.warning(f"Partition {partition_id} were not found in controller!").
• Callers cannot classify it. It surfaces as a generic RuntimeError whose
  message is assembled across a ZMQ hop, so the only way to tolerate the race is
  substring matching on the message, which breaks when the wording changes.

The third line also logs the whole routing table. With many storage units that is
hundreds of ids on one line, burying the unit, key and field the error already
names.

Modification

• Add StorageKeyNotFoundError(KeyError), raised by StorageUnitData.get_data
  when a global index is absent.
• Tag the GET_ERROR reply so the manager can rebuild the type after the ZMQ hop
  rather than flattening it into RuntimeError.
• Log this condition at DEBUG in all three places. Every other failure keeps its
  ERROR level and message; an unknown field still raises ValueError at ERROR,
  since that is a caller bug rather than the race.
• Replace storage_units={list(routing.keys())} with num_storage_units={len(routing)}.
• Export StorageKeyNotFoundError from transfer_queue.

Test

tests/test_storage_key_race.py: a cleared key raises StorageKeyNotFoundError;
it is still a KeyError; an unknown field still raises ValueError; a surviving
key is unaffected; and _handle_get marks the GET_ERROR reply while emitting no
ERROR record.

[ascend-robot]

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 791f4488ae

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Use a structured key-miss discriminator

When any non-key GET_ERROR message contains TQKeyNotFound, this substring check reconstructs it as StorageKeyNotFoundError; for example, an unknown-field error includes both the requested and available field names, so a supported field named TQKeyNotFound causes a genuine schema error to be logged at DEBUG and potentially swallowed by callers tolerating concurrent clears. Put a dedicated error code in the reply body and inspect that field instead of matching human-readable message text.

AGENTS.md reference: AGENTS.md:L5-L9

Useful? React with 👍 / 👎.

[ascend-robot]

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍