v0.3.0 feat: audio and video for Cloudflare's clef-omni

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.3.0 - 2026-10-12
+
+Cloudflare's `clef-omni` reads audio and video.
+
+- `Client#ask` takes `audio:` and `videos:`, arrays of base64 data URLs, for
+  `clef-omni`. Cloudflare reads WAV or MP3 audio and MP4 or WebM video. Added
+  `RubyDecisionModel::Media.data_url` and `.from_file`, which build data URLs
+  for images, audio, and video.
+- When the provider or model doesn't read audio or video, including `clef`
+  and `clef-flash`, the client raises `RequestError` before sending.
+- `cloudflare/clef-omni` and `@cf/cloudflare/clef-omni` resolve to
+  `clef-omni` on the Cloudflare provider. In 0.2.0 they raised
+  `ConfigurationError`. The bare `clef-omni` already worked.
+- New provider hooks `supports_audio?(model)` and `supports_videos?(model)`,
+  both false by default. `Base#request_body` accepts `audio:` and `videos:`.
+- `rake smoke` takes `SMOKE_AUDIO` and `SMOKE_VIDEO`.
+- Hosted `clef-flash` now has a 24,576-token context window. Cloudflare cuts
+  a text state that runs past the window to fit instead of rejecting it. The
+  gem is unchanged.
+
 ## 0.2.0 - 2026-10-06
 
 The decision models that shipped after Jev, behind the same `Client`.

README.md

@@ -127,7 +127,9 @@ client = RubyDecisionModel::Client.new(provider: provider)
 
 Requests go to
 `https://api.cloudflare.com/client/v4/accounts/<account_id>/ai/run/@cf/cloudflare/<model>`.
-The default model is `clef`; `clef-flash` is the smaller, faster one. The body
+The default model is `clef`; `clef-flash` is the smaller, faster one, and
+`clef-omni` also reads audio and video (see
+[Images, audio, and video](#images-audio-and-video)). The body
 is System One. The provider unwraps Cloudflare's `{"result": ...}` envelope,
 and a body with `"success": false` raises `InvalidResponse` carrying
 Cloudflare's error text. The account id and model end up in the URL path, so
@@ -138,6 +140,11 @@ raises `ConfigurationError`.
 Usage reports tokens with no cost, and `response.request_id` is the `cf-ray`
 header.
 
+Hosted `clef-flash` has a 24,576-token context window; `clef` and `clef-omni`
+have about 64k. Cloudflare cuts a text state that runs past the window to
+fit and returns no error. Images, audio, and video that don't fit fail the
+request.
+
 ### Perplexity (pplx-decider)
 
 ```ruby
@@ -206,7 +213,9 @@ Databricks, and System One servers default to 30, since Perplexity documents
 responses of up to 23 seconds on large inputs and a local server may load the
 model on the first request. Connecting gets 5 seconds either way. A number
 passed as `timeout:` sets both, as in 0.1.0. Through OpenRouter, pass a
-longer `timeout:` yourself for large inputs to slower models.
+longer `timeout:` yourself for large inputs to slower models. Audio and
+video take longer than text: Cloudflare reports about 1.5 seconds for a
+21-second video with sound. Pass a longer `timeout:` for long clips.
 
 ### Model aliases
 
@@ -225,26 +234,31 @@ whatever the provider returned.
 | `"luna"`, `"gpt-6-luna"` | `openai/gpt-6-luna-decisions` | `gpt-6-luna` (OpenAI) |
 | `"openai/gpt-6-luna-decisions"`, `"openai/gpt-6-luna"`, `"gpt-6-luna-decisions"` | as given | `gpt-6-luna` (OpenAI) |
 | `"clef"`, `"clef-flash"` | `cloudflare/clef`, `cloudflare/clef-flash` | as given (Cloudflare) |
-| `"cloudflare/clef"`, `"cloudflare/clef-flash"` | as given | `clef`, `clef-flash` (Cloudflare) |
-| `"@cf/cloudflare/clef"`, `"@cf/cloudflare/clef-flash"` | as given | `clef`, `clef-flash` (Cloudflare) |
+| `"cloudflare/clef"`, `"cloudflare/clef-flash"`, `"cloudflare/clef-omni"` | as given | `clef`, `clef-flash`, `clef-omni` (Cloudflare) |
+| `"@cf/cloudflare/clef"`, `"@cf/cloudflare/clef-flash"`, `"@cf/cloudflare/clef-omni"` | as given | `clef`, `clef-flash`, `clef-omni` (Cloudflare) |
 | `"pplx-decider"` | `perplexity/pplx-decider-v1-27b` | `pplx-decider-v1.1-27b` (Perplexity) |
 | `"pplx-decider-v1-27b"` | `perplexity/pplx-decider-v1-27b` | as given (Perplexity) |
 | `"perplexity/pplx-decider-v1-27b"` | as given | `pplx-decider-v1-27b` (Perplexity) |
 | anything else | as given | as given |
 
 OpenRouter carries pplx-decider v1 only, so `"pplx-decider"` means v1 there
-and v1.1 on Perplexity's own API.
+and v1.1 on Perplexity's own API. OpenRouter doesn't list `clef-omni` as of
+October 2026, so it has no alias there; on Cloudflare pass `"clef-omni"`.
 
 ### Writing a provider
 
 Subclass `RubyDecisionModel::Providers::Base` and define `name`, `env_var`,
 `default_base_url`, `endpoint_path`, and `default_model`. Optional hooks:
-`aliases`, `reports_cost?`, `supports_images?`, `request_id_header`,
+`aliases`, `reports_cost?`, `supports_images?`, `supports_audio?(model)`,
+`supports_videos?(model)`, `request_id_header`,
 `requires_api_key?`, `default_timeout`, `error_message`, and `validate!`. When the wire format differs from
 System One, override `request_body` to encode and `normalize_response` to turn
 the parsed body back into System One answers keyed by question id; the
 OpenAI provider shows both directions. Override `url(model)` when the model
-belongs in the path. Pass an instance as `provider:`.
+belongs in the path. A provider that answers true from `supports_audio?` or
+`supports_videos?` gets `audio:` or `videos:` passed to `request_body`, so
+an override of `request_body` has to accept them. Pass an instance as
+`provider:`.
 
 ## Questions and answers
 
@@ -268,7 +282,7 @@ approximately 1; treat them as calibrated, not normalized. Noul
 `probabilities` are always `{"true" => p, "false" => 1 - p}`. Most providers
 send only the probability, and the client fills in the split.
 
-## Images
+## Images, audio, and video
 
 OpenAI, Cloudflare, Perplexity, and System One servers read images. Pass them
 as base64 data URLs; no provider fetches a remote URL.
@@ -290,6 +304,39 @@ OpenAI, the `images` field for Cloudflare and System One servers, and
 Databricks don't take images, so the client raises `RequestError` before
 sending. Size and count limits vary by vendor and are enforced server side.
 
+Cloudflare's `clef-omni` also reads audio (WAV or MP3) and video (MP4 or
+WebM), passed as `audio:` and `videos:`. `RubyDecisionModel::Media.from_file`
+and `.data_url` build data URLs for all three kinds, and `Images` still works
+for images.
+
+```ruby
+client = RubyDecisionModel::Client.new(provider: :cloudflare, model: "clef-omni")
+
+client.ask(
+  state: "Customer sent a photo, a voice note, and a short video of the fault.",
+  questions: { "faulty" => RubyDecisionModel::Questions.noul("Is the device faulty?") },
+  images: [RubyDecisionModel::Media.from_file("device.jpg")],
+  audio: [RubyDecisionModel::Media.from_file("note.mp3")],
+  videos: [RubyDecisionModel::Media.from_file("fault.mp4")]
+)
+```
+
+On Cloudflare they go in the `images`, `audio`, and `videos` fields. A
+provider or model that doesn't read audio or video, including `clef` and
+`clef-flash`, makes the client raise `RequestError` before sending.
+Cloudflare documents up to 4 images, 4 audio clips, and 2 videos per request,
+enforced server side.
+
+The client checks only that each item is a data URL of the right kind
+(`data:image/`, `data:audio/`, or `data:video/`). It does not decode the
+payload or measure it, and `from_file` reads the whole file into memory.
+Format, size, and count are the provider's to enforce, so check uploads you
+don't control before you pass them on.
+
+A retry sends the whole body again, so with the default policy a request
+carrying large clips can upload up to three times. Build the client with
+`retry: { max_retries: 0 }` to turn retries off for those calls.
+
 ## Retries
 
 Retry behaviour follows the official Typesafe SDKs and lives in
@@ -338,7 +385,7 @@ return is still accepted and treated as having no headers, which means no
 | Error | Meaning |
 | --- | --- |
 | `ConfigurationError` | No provider could be resolved, a missing key or setting (api_key, account_id, base_url), unknown provider, a `model:` Databricks can't take, or bad `retry:` value |
-| `RequestError` | Questions hash was empty, images were not data URLs or went to a provider that doesn't read them, or state or questions could not be encoded as JSON (original error on `#cause`) |
+| `RequestError` | Questions hash was empty, images, audio, or videos were not data URLs or went to a provider or model that doesn't read them, or state or questions could not be encoded as JSON (original error on `#cause`) |
 | `TransportError` (`TimeoutError`) | Network or timeout failure after retries, carries `#cause_error` |
 | `ApiError` | Non-2xx response, carries `#status`, `#body`, and `#headers`. The message ends with the vendor's own reason when the body has one |
 | `Unauthorized` | 401 |
@@ -349,7 +396,7 @@ return is still accepted and treated as having no headers, which means no
 | `InvalidResponse` | Body wasn't JSON, wasn't a Hash, or an answer was malformed (including a noul outside 0..1 or an id answered twice) |
 | `MissingAnswers` | One or more question ids came back missing, wrong-typed, or refused. Carries `#missing`, `#refused` (the subset the provider declined), and `#answers` |
 
-Status: 0.2.0, API may change.
+Status: 0.3.0, API may change.
 
 The companion gem `decide` builds decisions and verdicts on top of this client.
 
@@ -364,6 +411,8 @@ anywhere. To ship a version:
    request per configured provider and prints the answers. CI never runs it.
    `TARGETS=open_router:clef,open_router:luna` picks provider and model
    pairs, and `SMOKE_IMAGE=photo.png` adds an image where supported.
+   `SMOKE_AUDIO=clip.mp3` and `SMOKE_VIDEO=clip.mp4` add audio and video
+   where the model reads them, such as `TARGETS=cloudflare:clef-omni`.
 4. Merge to `main`. The Release workflow runs the suite, builds the gem with
    `gem build --strict`, checks the built gem carries every file under
    `lib/`, and pushes it. A version already on RubyGems is skipped, so the

Rakefile

@@ -18,6 +18,8 @@ task default: :test
 #   rake smoke                                     every configured provider
 #   rake smoke TARGETS=open_router:clef,perplexity provider[:model] pairs
 #   rake smoke SMOKE_IMAGE=photo.png               also send an image where supported
+#   rake smoke TARGETS=cloudflare:clef-omni SMOKE_AUDIO=clip.mp3 SMOKE_VIDEO=clip.mp4
+#                                                  also send audio and video where supported
 desc "Live smoke test against each configured provider (needs real keys)"
 task :smoke do
   $LOAD_PATH.unshift File.expand_path("lib", __dir__)
@@ -47,18 +49,24 @@ task :smoke do
                                        criteria: ["No impact", "Minor", "Major", "Critical"])
   }
   image = ENV["SMOKE_IMAGE"].to_s.empty? ? nil : rdm::Images.from_file(ENV["SMOKE_IMAGE"])
+  audio = ENV["SMOKE_AUDIO"].to_s.empty? ? nil : rdm::Media.from_file(ENV["SMOKE_AUDIO"])
+  video = ENV["SMOKE_VIDEO"].to_s.empty? ? nil : rdm::Media.from_file(ENV["SMOKE_VIDEO"])
 
   failures = 0
   targets.each do |name, model|
     label = model ? "#{name}:#{model}" : name.to_s
     begin
       client = rdm::Client.new(provider: name, model: model)
       images = image && client.provider.supports_images? ? [image] : nil
+      clips = audio && client.provider.supports_audio?(client.model) ? [audio] : nil
+      videos = video && client.provider.supports_videos?(client.model) ? [video] : nil
       started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      response = client.ask(state: state, questions: questions, images: images)
+      response = client.ask(state: state, questions: questions, images: images, audio: clips, videos: videos)
       elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round
 
-      puts "ok    #{label}  model=#{response.model.inspect}  #{elapsed}ms#{images ? '  +image' : ''}"
+      media = { "image" => [image, images], "audio" => [audio, clips], "video" => [video, videos] }
+      marks = media.filter_map { |kind, (given, sent)| "  #{sent ? '+' : 'skipped '}#{kind}" if given }.join
+      puts "ok    #{label}  model=#{response.model.inspect}  #{elapsed}ms#{marks}"
       puts format("      outage=%.3f urgent=%.3f team=%s (%.2f) severity=%.2f (%.2f)",
                   response["outage"].noul, response["urgent"].noul, response["team"].choice,
                   response["team"].confidence, response["severity"].score, response["severity"].confidence)

lib/ruby_decision_model.rb

@@ -7,6 +7,7 @@
 require_relative "ruby_decision_model/errors"
 require_relative "ruby_decision_model/questions"
 require_relative "ruby_decision_model/images"
+require_relative "ruby_decision_model/media"
 require_relative "ruby_decision_model/answers"
 require_relative "ruby_decision_model/response"
 require_relative "ruby_decision_model/providers"

lib/ruby_decision_model/client.rb

@@ -60,12 +60,13 @@ def base_url
       @provider.base_url
     end
 
-    # images: data URLs (see Images) for providers that read images. Each
-    # provider places them where its API expects.
-    def ask(state:, questions:, images: nil)
+    # images:, audio:, videos: data URLs (see Images and Media) for providers
+    # and models that read them. Each provider places them where its API
+    # expects.
+    def ask(state:, questions:, images: nil, audio: nil, videos: nil)
       raise RequestError, "questions must not be empty" if questions.nil? || questions.empty?
 
-      body = build_body(state, questions, images)
+      body = build_body(state, questions, images: images, audio: audio, videos: videos)
       status, response_body, response_headers = perform_with_retry(
         url: request_url, headers: @provider.headers, body: body
       )
@@ -81,25 +82,44 @@ def request_url
       takes_model ? @provider.url(@model) : @provider.url
     end
 
-    def build_body(state, questions, images)
-      images = Array(images)
-      return @provider.request_body(model: @model, state: state, questions: questions) if images.empty?
+    MEDIA_PREFIXES = { images: "data:image/", audio: "data:audio/", videos: "data:video/" }.freeze
+    private_constant :MEDIA_PREFIXES
 
-      raise RequestError, "#{@provider.name} does not accept images" unless @provider.supports_images?
+    def build_body(state, questions, **media)
+      media = media.transform_values { |urls| Array(urls) }.reject { |_kind, urls| urls.empty? }
 
-      images.each do |image|
-        next if image.is_a?(String) && image.start_with?("data:image/")
+      media.each do |kind, urls|
+        raise RequestError, unsupported_message(kind) unless accepts?(kind)
 
-        raise RequestError, "images must be data URLs (data:image/...); see RubyDecisionModel::Images"
+        prefix = MEDIA_PREFIXES.fetch(kind)
+        next if urls.all? { |url| url.is_a?(String) && url.start_with?(prefix) }
+
+        raise RequestError, "#{kind} must be data URLs (#{prefix}...); see RubyDecisionModel::#{kind == :images ? 'Images' : 'Media'}"
       end
 
-      @provider.request_body(model: @model, state: state, questions: questions, images: images)
+      @provider.request_body(model: @model, state: state, questions: questions, **media)
     rescue JSON::GeneratorError, JSON::NestingError => e
       # The generator's message can quote the offending value, so it stays
       # on #cause rather than in a message that may be logged.
       raise RequestError, "state or questions could not be encoded as JSON (#{e.class})"
     end
 
+    # supports_images? takes no argument in 0.2.0 providers, so it is called
+    # bare; audio and videos can differ per model, so those get the model.
+    def accepts?(kind)
+      case kind
+      when :images then @provider.supports_images?
+      when :audio then @provider.supports_audio?(@model)
+      when :videos then @provider.supports_videos?(@model)
+      end
+    end
+
+    def unsupported_message(kind)
+      return "#{@provider.name} does not accept images" if kind == :images
+
+      "#{[@provider.name, @model].compact.join(' ')} does not accept #{kind}"
+    end
+
     def resolve_provider(provider, api_key:, base_url:)
       case provider
       when Providers::Base

lib/ruby_decision_model/media.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require_relative "images"
+
+module RubyDecisionModel
+  # Builds the base64 data URLs that Client#ask takes as images:, audio:, and
+  # videos:. Images has the image-only version of the same helpers.
+  module Media
+    CONTENT_TYPES = Images::CONTENT_TYPES.merge(
+      ".wav" => "audio/wav",
+      ".mp3" => "audio/mpeg",
+      ".mp4" => "video/mp4",
+      ".webm" => "video/webm"
+    ).freeze
+
+    module_function
+
+    def data_url(bytes, content_type:)
+      unless content_type.to_s.start_with?("image/", "audio/", "video/")
+        raise ArgumentError, "content_type must be an image/*, audio/*, or video/* type, got #{content_type.inspect}"
+      end
+
+      # pack("m0") is strict base64 without the base64 gem, which leaves the
+      # default gems in Ruby 3.4.
+      "data:#{content_type};base64,#{[bytes].pack('m0')}"
+    end
+
+    # Reads a file and infers its type from the extension.
+    def from_file(path, content_type: nil)
+      content_type ||= CONTENT_TYPES[File.extname(path.to_s).downcase]
+      raise ArgumentError, "cannot infer a media type for #{path}; pass content_type:" if content_type.nil?
+
+      data_url(File.binread(path), content_type: content_type)
+    end
+  end
+end

lib/ruby_decision_model/providers/base.rb

@@ -65,6 +65,18 @@ def supports_images?
         false
       end
 
+      # Whether requests may carry audio or videos. Client refuses audio: and
+      # videos: for a provider that says no, before anything goes over the
+      # wire. The model is passed because support can differ between one
+      # provider's models.
+      def supports_audio?(_model = nil)
+        false
+      end
+
+      def supports_videos?(_model = nil)
+        false
+      end
+
       # Response header carrying the provider's request id, or nil.
       # Typesafe's, which 0.1.0 read for every provider and System One
       # servers that follow Typesafe send too. Vendors with their own header
@@ -129,12 +141,14 @@ def headers
         headers
       end
 
-      # Client passes images: only when the caller supplied some, so a
-      # subclass that does not take images can keep a three-keyword
-      # signature.
-      def request_body(model:, state:, questions:, images: nil)
+      # Client passes images:, audio:, and videos: only when the caller
+      # supplied some and the provider said it supports them, so a subclass
+      # that takes none can keep a three-keyword signature.
+      def request_body(model:, state:, questions:, images: nil, audio: nil, videos: nil)
         body = { "model" => model, "state" => state, "questions" => questions }
         body["images"] = images if images
+        body["audio"] = audio if audio
+        body["videos"] = videos if videos
         JSON.generate(body)
       end