Repository navigation
Conversation
The OpenAI-compatible API accepts --served-model-name and advertises the alias from /v1/models, but response payloads carried the package's canonical model identifier. A relay that compares the response model against the requested one therefore rejected every request to an aliased deployment, even though the request itself was served correctly. HttpState now carries the name echoed in responses, resolved once while the server is constructed: the first configured served name when --served-model-name is set, and the package's canonical identifier otherwise. The response encoder reads that field instead of the package configuration, so streaming chunks and non-streaming bodies emit the same name. Requests still accept both the alias and the canonical identifier, and the fallback leaves responses unchanged when no alias is configured. An aliased deployment now presents one name consistently: /v1/models, the accepted request, and the echoed response agree, so a relay that compares them no longer rejects requests that were served correctly. No other response field, endpoint, or request path changes. Validation on base 5fd4070: the flare-cli release build succeeds, and cargo check --release --tests --manifest-path helios/Cargo.toml reports no errors. On a four-GPU RTX PRO 6000 host serving a TP4 prepared package with --served-model-name DeepSeek-V4.1-Flash-1017, a non-streaming completion returned that name with finish_reason stop, a streaming completion returned it in every chunk, and a relay that compares request and response model names forwarded the request successfully. Per-request model aliases and non-chat endpoints were not tested; the echoed name is fixed for the lifetime of the server. Design-Signed-Off-By: Angela (DeepSeek-V4.1-Flash); design-review=ed22 Signed-off-by: jedi95 <jedi95@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OpenAI-compatible API accepts
--served-model-nameand advertises the alias from/v1/models, but response payloads carry the package's canonical model identifier. A relay that compares the response model against the requested one therefore rejects every request to an aliased deployment, even though the request itself was served correctly.HttpStatenow carries the name echoed in responses, resolved once while the server is constructed: the first configured served name when--served-model-nameis set, and the package's canonical identifier otherwise. The response encoder inhelios/src/server/http.rsreads that field instead of the package configuration, so streaming chunks and non-streaming bodies emit the same name.helios/src/server.rsinitializes the field and the test constructor inhelios/src/server/runtime/phase_tests.rscarries it. Requests still accept both the alias and the canonical identifier, and the fallback leaves responses unchanged when no alias is configured.An aliased deployment now presents one name consistently:
/v1/models, the accepted request, and the echoed response agree, so a relay that compares them no longer rejects requests that were served correctly. No other response field, endpoint, or request path changes.Evidence: the
flare-clirelease build succeeds, andcargo check --release --tests --manifest-path helios/Cargo.tomlreports no errors. On a four-GPU RTX PRO 6000 host serving a TP4 prepared package with--served-model-name DeepSeek-V4.1-Flash-1017, a non-streaming completion returned that name withfinish_reasonstop, a streaming completion returned it in every chunk, and a relay that compares request and response model names forwarded the request successfully.Limitations: per-request model aliases and non-chat endpoints were not tested, and the echoed name is fixed for the lifetime of the server.