Skip to content

frontier_models seed contains invalid provider model IDs (openrouter llama-4-maverick, together DeepSeek-V3.1) #41

Description

@emooreatx

Two of the 13 seeded frontier_models rows failed 300/300 on 2026-02-16 with provider-side "not a valid model ID" errors:

model_id provider error
openrouter-llama-4-maverick OpenRouter HTTP 400 meta-llama/llama-4-maverick-instruct is not a valid model ID
together-deepseek-v3.1 Together HTTP 404 Unable to access model deepseek-ai/DeepSeek-V3.1-0324

Migrations 017_seed_frontier_models.sql, 019_fix_openrouter_model_id.sql, 020_fix_together_model_ids.sql touch these IDs — check whether 019/020 actually fixed the default_model_name these rows carry in production (the failures may predate 019/020, but the seed should be verified against the providers' current catalogues rather than assumed). Note the EU database later recorded 4 completed runs for both models, so a working ID existed at some point that evening — compare the target_model/default_model_name used by the later successful rows.

Asks:

  1. A startup or pre-sweep model-ID validation step: call each provider's /models (or a 1-token completion) for every active frontier_models row and mark invalid ones active=false with the provider error, before burning 300 scenarios each.
  2. Fix the two IDs (or retire the models) in a new migration; do not edit 017–020 in place.
  3. The 5 models below the publish threshold on US (gemini-2.5-flash 4, gemini-2.5-pro 2, o3 4, openrouter-llama-4-maverick 4, together-deepseek-v3.1 4) are all there because of provider errors or the 12 orphaned running rows (Frontier sweep cannot run: flag off, no beat scheduler exists, 15 evaluations stuck in status='running' since 2026-02-16 #34) — fixing IDs + the reaper + one clean sweep gets all 13 publishing.

Related: #34, #39, #40.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions