Skip to content

feat(sync): Cloudflare AI Gateway model catalog sync#3317

Open
rekram1-node wants to merge 2 commits into
devfrom
fix/cloudflare-ai-gateway-sync
Open

feat(sync): Cloudflare AI Gateway model catalog sync#3317
rekram1-node wants to merge 2 commits into
devfrom
fix/cloudflare-ai-gateway-sync

Conversation

@rekram1-node

Copy link
Copy Markdown
Collaborator

Summary

  • Add central cloudflare-ai-gateway sync module (bun models:sync cloudflare-ai-gateway / cloudflare group).
  • Reuse existing Workers AI sync secrets (CLOUDFLARE_WORKERS_AI_SYNC_ACCOUNT_ID / CLOUDFLARE_WORKERS_AI_SYNC_API_TOKEN) against GET /compat/models on gateway default.
  • Hybrid sync: API is authoritative for membership + list prices; capabilities/limits/cache/reasoning_options come from base_model / existing TOMLs / source OpenAI·Anthropic·Workers AI files (endpoint is too thin for full authoring).
  • Catalog refresh: +25 models (gpt-5 family, gpt-4.1, o1-pro, deep-research, gemma-4, kimi-k2.7-code, …), −1 (nemotron-3-120b-a12b no longer served), cost/format updates on a few existing entries. 82 → 106 models.

Evidence

Test plan

  • bun models:sync cloudflare-ai-gateway (live)
  • Second dry-run: 0 created / 0 updated / 0 deleted
  • bun validate
  • bun models:sync --list-providers includes cloudflare-ai-gateway
  • CI hourly matrix picks up the new provider (uses existing CF secrets)

Reuse Workers AI sync secrets against /compat/models (gateway default).
Hybrid sync: API membership + list prices; capabilities via base_model.
Refresh catalog: add missing OpenAI/Workers AI models, drop nemotron.
factorBaseModel was called with context:0 when the gateway has no
limit override, which tripped baseModelOmit and stripped inherited
limit.input. Inherit the base metadata context instead.
@github-actions

Copy link
Copy Markdown
Contributor

Now I have all the information I need. Let me compile my findings.

Action items

  • [medium] [violation] providers/cloudflare-ai-gateway/models/openai/gpt-5.6.toml:1 - Check: New model files must resolve to a display name that matches their model ID when the source provider carries a name override. Why: The file uses base_model = "openai/gpt-5.6-sol" (whose metadata name = "GPT-5.6 Sol") with no name override, but the model ID from the filename is openai/gpt-5.6. The source providers/openai/models/gpt-5.6.toml declares name = "GPT-5.6" as an explicit override over the gpt-5.6-sol base, but buildNewModel only copies source?.base_model, cost, reasoning_options, interleaved, temperature, and status into the factored values — it never includes source?.name. The generated catalog entry will therefore display "GPT-5.6 Sol" for model ID openai/gpt-5.6, contradicting the OpenAI provider's own naming. Action: Add name: source?.name to the values object in buildNewModel (so source-level name overrides are preserved when source.base_model differs from the gateway model ID), or hand-add name = "GPT-5.6" to this TOML.

  • [medium] [possible mistake] providers/cloudflare-ai-gateway/models/anthropic/claude-3.5-sonnet.toml:14-15 - Check: API-driven cost changes for pass-through models should be consistent with the gateway's other entries for the same upstream provider. Why: Input/output changed from 3/15 to 6/30 — exactly 2× Anthropic's known direct price for Claude Sonnet 3.5 v2. Every other Anthropic model in this gateway (claude-sonnet-4, claude-sonnet-4-5, claude-opus-4-6, claude-haiku-4-5, etc.) retained Anthropic's direct prices and was not touched by the sync, making this the sole outlier. The PR body cites a live fetch of 2361 models but does not map this specific doubling to a provider pricing reference. Action: Confirm the gateway /compat/models endpoint actually returns cost_in/cost_out corresponding to $6/$30 per million for anthropic/claude-3.5-sonnet, and cite the provider reference that supports it; otherwise restore 3/15.

  • [low] [possible mistake] providers/cloudflare-ai-gateway/models/workers-ai/@cf/zai-org/glm-5.2.toml:8-9 - Check: API-driven cost changes for Workers AI models served through the gateway should be plausible relative to the Workers AI direct price. Why: Input/output dropped from 1.4/4.4 to 0.06/0.4 — a 23×/11× decrease from the Workers AI direct price for the same @cf/zai-org/glm-5.2 model. The newly added @cf/google/gemma-4-26b-a4b-it moves in the opposite direction (0.6/3 vs Workers AI's 0.1/0.3, a 6×/10× increase), so the two workers-ai cost deltas are inconsistent with a uniform gateway markup or discount. Action: Verify the gateway API's cost_in/cost_out for workers-ai/@cf/zai-org/glm-5.2 and confirm the per-token-to-per-million conversion is correct for this entry; cite the source if the drop is intentional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant