bedrock closeout: aws-embed replaces EOL'd aws-cohere; mistral + embeddings live coverage - #39
Merged
Conversation
AWS answered live probes with end-of-life for the legacy Command models and legacy-gates Command R per account, so the aws-cohere chat protocol is dead weight. In its place, aws-embed serves /v1/embeddings on Bedrock: Titan invokes once per input (body inputTextTokenCount as the count fallback), Cohere embeds the batch in one call, both answered in the OpenAI list shape. Embed replies carry only the input-count header, never the output one, so usage reads that header directly instead of the pairwise bedrock_header_usage contract.
Mistral Large through aws-converse needed zero code; titan-embed-v2 and cohere.embed-english-v3 pin the new aws-embed wire.
The correction's json! interpolation deep-copied the input string; the body now takes it by move. Its header read duplicated the parse inside bedrock_header_usage, so both go through one token_header in the module that owns the Bedrock header contract, and the single-input rule reaches the live matrix case and the provider table.
A workspace scan of every json! interpolation found four sites handing the macro an owned value it then deep-copies: the config document, the model-status rows, the usage series, and the Bedrock Llama stream's per-frame text. Each dies right after, so each moves instead. Field order is unchanged — serde_json orders a BTreeMap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One-day Bedrock key round: close every remaining Bedrock gap so no future E2E depends on having a key.
What changed
aws-cohereremoved. AWS answered live probes with "This model version has reached the end of its life" forcohere.command-text-v14/amazon.titan-text-express-v1, and legacy-gates Command R per account ("marked by provider as Legacy… not actively using the model in the last 30 days"). The chat protocol targeted dead models; deleted engine, mock, tests, docs.aws-embedadded on the existing/v1/embeddingssurface. Titan takes exactly one{inputText}per call — the vendor has no batch form ({"inputText":["a","b"]}→ "expected type: String, found: JSONArray",inputTexts→ schema violation), so a multi-input request is refused with 400 rather than silently fanned out into N InvokeModel calls at client-controlled N. Cohere batches{texts, input_type: search_document}in one call. Both answered in the OpenAI list shape.x-amzn-bedrock-input-token-count(no output header), so the pairwisebedrock_header_usagecontract never matches them. Onetoken_headerparse now serves both forms, in the module that owns the Bedrock header contract.aws-converse(zero code — converse is family-agnostic) + both embed models; bedrock group is now 20 cases.Live evidence (real key, us-east-1)
503 "Bedrock is unable to process your request"on mistral) — AWS embed capacity for this fresh account was visibly flaky: directcurlranged 271ms → 26s → 60s-hang.400 validation_exceptionin 3.7ms, zero vendor calls.-D warnings0 / 547 tests, 0 failed.Perf
Interleaved A/B (both orders, 2 rounds) against
origin/main: serial p50 25µs → 25µs, p99 43 → 39µs, serial rps +1.7%, concurrent rps +0.3%, request-clone 3.45 → 3.44µs, big-payload p50 162 → 162.5µs. No regression; the chat hot path is untouched (leaf protocol swap). Production lines net -3.Post-merge status
Bedrock is closed out: InvokeModel (Claude/Llama), Converse (Claude/Nova/DeepSeek/Mistral), embeddings (Titan/Cohere), EventStream, prompt cache, signed thinking — all live-verified. Text-generation legacies (Cohere Command, Titan text) are EOL'd by AWS and documented as such. No future round needs a Bedrock key.