Skip to content

fix(sdk): serialize audio and video inputs#215

Draft
svonava wants to merge 3 commits into
mainfrom
agent/sdk-media-inputs
Draft

fix(sdk): serialize audio and video inputs#215
svonava wants to merge 3 commits into
mainfrom
agent/sdk-media-inputs

Conversation

@svonava

@svonava svonava commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Related: superlinked/sie-internal#1958

Summary

  • serialize audio and video byte/path inputs consistently across Python SDK encode, score, and extract
  • add matching TypeScript SDK audio/video Item types and msgpack serialization across encode, score, and extract
  • preserve generic structured extractor data in TypeScript ExtractResult (for document/media adapters)
  • make TypeScript extract labels optional and map instruction / outputSchema to the generic instruction / output_schema primitive parameters
  • document native audio/video Item fields in gateway and local-server OpenAPI
  • repair Pydantic nested-schema references when exporting request models

This adds no task-specific endpoint or model behavior. It aligns both public clients and schemas with the existing generic Item wire contract; model adapters and catalog rollout remain separate.

Verification

  • Python SDK media + sync/async client tests: 103 passed
  • TypeScript focused client/media/type tests: 105 passed
  • TypeScript SDK typecheck: passed
  • TypeScript SDK ESM/CJS/declaration build: passed
  • TypeScript changed-file Biome format/lint: passed
  • server OpenAPI export tests: 6 passed
  • Ruff format/check: passed
  • ty: no new errors (four pre-existing unused-ignore warnings)
  • cargo fmt: passed
  • gateway binary built and regenerated OpenAPI; generated audio/video refs validated
  • git diff --check: passed

The complete TypeScript suite reached 309 passing tests; its only failure is the existing public-checkout wireContract.test.ts dependency on the absent packages/wire-fixtures/model_state.json. The package-wide Biome command also reports the pre-existing package.json formatting drift and two existing encoding.ts non-null warnings; all files changed here pass Biome.

Review posture

Draft for human review. Do not merge as part of automated course work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant