Skip to content

Echo the first served name in response model fields - #317

Open
jedi95 wants to merge 1 commit into
NVIDIA:mainfrom
jedi95:fix/served-model-name-echo
Open

jedi95 wants to merge 1 commit into
NVIDIA:mainfrom
jedi95:fix/served-model-name-echo

Conversation

@jedi95

@jedi95 jedi95 commented Nov 17, 2026

Copy link
Copy Markdown

The OpenAI-compatible API accepts --served-model-name and advertises the alias from /v1/models, but response payloads carry the package's canonical model identifier. A relay that compares the response model against the requested one therefore rejects every request to an aliased deployment, even though the request itself was served correctly.

HttpState now carries the name echoed in responses, resolved once while the server is constructed: the first configured served name when --served-model-name is set, and the package's canonical identifier otherwise. The response encoder in helios/src/server/http.rs reads that field instead of the package configuration, so streaming chunks and non-streaming bodies emit the same name. helios/src/server.rs initializes the field and the test constructor in helios/src/server/runtime/phase_tests.rs carries it. Requests still accept both the alias and the canonical identifier, and the fallback leaves responses unchanged when no alias is configured.

An aliased deployment now presents one name consistently: /v1/models, the accepted request, and the echoed response agree, so a relay that compares them no longer rejects requests that were served correctly. No other response field, endpoint, or request path changes.

Evidence: the flare-cli release build succeeds, and cargo check --release --tests --manifest-path helios/Cargo.toml reports no errors. On a four-GPU RTX PRO 6000 host serving a TP4 prepared package with --served-model-name DeepSeek-V4.1-Flash-1017, a non-streaming completion returned that name with finish_reason stop, a streaming completion returned it in every chunk, and a relay that compares request and response model names forwarded the request successfully.

Limitations: per-request model aliases and non-chat endpoints were not tested, and the echoed name is fixed for the lifetime of the server.

The OpenAI-compatible API accepts --served-model-name and advertises the
alias from /v1/models, but response payloads carried the package's
canonical model identifier. A relay that compares the response model
against the requested one therefore rejected every request to an aliased
deployment, even though the request itself was served correctly.

HttpState now carries the name echoed in responses, resolved once while
the server is constructed: the first configured served name when
--served-model-name is set, and the package's canonical identifier
otherwise. The response encoder reads that field instead of the package
configuration, so streaming chunks and non-streaming bodies emit the
same name. Requests still accept both the alias and the canonical
identifier, and the fallback leaves responses unchanged when no alias
is configured.

An aliased deployment now presents one name consistently: /v1/models,
the accepted request, and the echoed response agree, so a relay that
compares them no longer rejects requests that were served correctly.
No other response field, endpoint, or request path changes.

Validation on base 5fd4070: the flare-cli release build succeeds, and
cargo check --release --tests --manifest-path helios/Cargo.toml reports
no errors. On a four-GPU RTX PRO 6000 host serving a TP4 prepared
package with --served-model-name DeepSeek-V4.1-Flash-1017, a
non-streaming completion returned that name with finish_reason stop, a
streaming completion returned it in every chunk, and a relay that
compares request and response model names forwarded the request
successfully. Per-request model aliases and non-chat endpoints were not
tested; the echoed name is fixed for the lifetime of the server.

Design-Signed-Off-By: Angela (DeepSeek-V4.1-Flash); design-review=ed22
Signed-off-by: jedi95 <jedi95@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant