diff --git a/CHANGELOG.md b/CHANGELOG.md index 64a2858..b65ae83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +## 0.3.0 - 2026-10-12 + +Cloudflare's `clef-omni` reads audio and video. + +- `Client#ask` takes `audio:` and `videos:`, arrays of base64 data URLs, for + `clef-omni`. Cloudflare reads WAV or MP3 audio and MP4 or WebM video. Added + `RubyDecisionModel::Media.data_url` and `.from_file`, which build data URLs + for images, audio, and video. +- When the provider or model doesn't read audio or video, including `clef` + and `clef-flash`, the client raises `RequestError` before sending. +- `cloudflare/clef-omni` and `@cf/cloudflare/clef-omni` resolve to + `clef-omni` on the Cloudflare provider. In 0.2.0 they raised + `ConfigurationError`. The bare `clef-omni` already worked. +- New provider hooks `supports_audio?(model)` and `supports_videos?(model)`, + both false by default. `Base#request_body` accepts `audio:` and `videos:`. +- `rake smoke` takes `SMOKE_AUDIO` and `SMOKE_VIDEO`. +- Hosted `clef-flash` now has a 24,576-token context window. Cloudflare cuts + a text state that runs past the window to fit instead of rejecting it. The + gem is unchanged. + ## 0.2.0 - 2026-10-06 The decision models that shipped after Jev, behind the same `Client`. diff --git a/README.md b/README.md index 837949a..94106fa 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,9 @@ client = RubyDecisionModel::Client.new(provider: provider) Requests go to `https://api.cloudflare.com/client/v4/accounts//ai/run/@cf/cloudflare/`. -The default model is `clef`; `clef-flash` is the smaller, faster one. The body +The default model is `clef`; `clef-flash` is the smaller, faster one, and +`clef-omni` also reads audio and video (see +[Images, audio, and video](#images-audio-and-video)). The body is System One. The provider unwraps Cloudflare's `{"result": ...}` envelope, and a body with `"success": false` raises `InvalidResponse` carrying Cloudflare's error text. The account id and model end up in the URL path, so @@ -138,6 +140,11 @@ raises `ConfigurationError`. Usage reports tokens with no cost, and `response.request_id` is the `cf-ray` header. +Hosted `clef-flash` has a 24,576-token context window; `clef` and `clef-omni` +have about 64k. Cloudflare cuts a text state that runs past the window to +fit and returns no error. Images, audio, and video that don't fit fail the +request. + ### Perplexity (pplx-decider) ```ruby @@ -206,7 +213,9 @@ Databricks, and System One servers default to 30, since Perplexity documents responses of up to 23 seconds on large inputs and a local server may load the model on the first request. Connecting gets 5 seconds either way. A number passed as `timeout:` sets both, as in 0.1.0. Through OpenRouter, pass a -longer `timeout:` yourself for large inputs to slower models. +longer `timeout:` yourself for large inputs to slower models. Audio and +video take longer than text: Cloudflare reports about 1.5 seconds for a +21-second video with sound. Pass a longer `timeout:` for long clips. ### Model aliases @@ -225,26 +234,31 @@ whatever the provider returned. | `"luna"`, `"gpt-6-luna"` | `openai/gpt-6-luna-decisions` | `gpt-6-luna` (OpenAI) | | `"openai/gpt-6-luna-decisions"`, `"openai/gpt-6-luna"`, `"gpt-6-luna-decisions"` | as given | `gpt-6-luna` (OpenAI) | | `"clef"`, `"clef-flash"` | `cloudflare/clef`, `cloudflare/clef-flash` | as given (Cloudflare) | -| `"cloudflare/clef"`, `"cloudflare/clef-flash"` | as given | `clef`, `clef-flash` (Cloudflare) | -| `"@cf/cloudflare/clef"`, `"@cf/cloudflare/clef-flash"` | as given | `clef`, `clef-flash` (Cloudflare) | +| `"cloudflare/clef"`, `"cloudflare/clef-flash"`, `"cloudflare/clef-omni"` | as given | `clef`, `clef-flash`, `clef-omni` (Cloudflare) | +| `"@cf/cloudflare/clef"`, `"@cf/cloudflare/clef-flash"`, `"@cf/cloudflare/clef-omni"` | as given | `clef`, `clef-flash`, `clef-omni` (Cloudflare) | | `"pplx-decider"` | `perplexity/pplx-decider-v1-27b` | `pplx-decider-v1.1-27b` (Perplexity) | | `"pplx-decider-v1-27b"` | `perplexity/pplx-decider-v1-27b` | as given (Perplexity) | | `"perplexity/pplx-decider-v1-27b"` | as given | `pplx-decider-v1-27b` (Perplexity) | | anything else | as given | as given | OpenRouter carries pplx-decider v1 only, so `"pplx-decider"` means v1 there -and v1.1 on Perplexity's own API. +and v1.1 on Perplexity's own API. OpenRouter doesn't list `clef-omni` as of +October 2026, so it has no alias there; on Cloudflare pass `"clef-omni"`. ### Writing a provider Subclass `RubyDecisionModel::Providers::Base` and define `name`, `env_var`, `default_base_url`, `endpoint_path`, and `default_model`. Optional hooks: -`aliases`, `reports_cost?`, `supports_images?`, `request_id_header`, +`aliases`, `reports_cost?`, `supports_images?`, `supports_audio?(model)`, +`supports_videos?(model)`, `request_id_header`, `requires_api_key?`, `default_timeout`, `error_message`, and `validate!`. When the wire format differs from System One, override `request_body` to encode and `normalize_response` to turn the parsed body back into System One answers keyed by question id; the OpenAI provider shows both directions. Override `url(model)` when the model -belongs in the path. Pass an instance as `provider:`. +belongs in the path. A provider that answers true from `supports_audio?` or +`supports_videos?` gets `audio:` or `videos:` passed to `request_body`, so +an override of `request_body` has to accept them. Pass an instance as +`provider:`. ## Questions and answers @@ -268,7 +282,7 @@ approximately 1; treat them as calibrated, not normalized. Noul `probabilities` are always `{"true" => p, "false" => 1 - p}`. Most providers send only the probability, and the client fills in the split. -## Images +## Images, audio, and video OpenAI, Cloudflare, Perplexity, and System One servers read images. Pass them as base64 data URLs; no provider fetches a remote URL. @@ -290,6 +304,39 @@ OpenAI, the `images` field for Cloudflare and System One servers, and Databricks don't take images, so the client raises `RequestError` before sending. Size and count limits vary by vendor and are enforced server side. +Cloudflare's `clef-omni` also reads audio (WAV or MP3) and video (MP4 or +WebM), passed as `audio:` and `videos:`. `RubyDecisionModel::Media.from_file` +and `.data_url` build data URLs for all three kinds, and `Images` still works +for images. + +```ruby +client = RubyDecisionModel::Client.new(provider: :cloudflare, model: "clef-omni") + +client.ask( + state: "Customer sent a photo, a voice note, and a short video of the fault.", + questions: { "faulty" => RubyDecisionModel::Questions.noul("Is the device faulty?") }, + images: [RubyDecisionModel::Media.from_file("device.jpg")], + audio: [RubyDecisionModel::Media.from_file("note.mp3")], + videos: [RubyDecisionModel::Media.from_file("fault.mp4")] +) +``` + +On Cloudflare they go in the `images`, `audio`, and `videos` fields. A +provider or model that doesn't read audio or video, including `clef` and +`clef-flash`, makes the client raise `RequestError` before sending. +Cloudflare documents up to 4 images, 4 audio clips, and 2 videos per request, +enforced server side. + +The client checks only that each item is a data URL of the right kind +(`data:image/`, `data:audio/`, or `data:video/`). It does not decode the +payload or measure it, and `from_file` reads the whole file into memory. +Format, size, and count are the provider's to enforce, so check uploads you +don't control before you pass them on. + +A retry sends the whole body again, so with the default policy a request +carrying large clips can upload up to three times. Build the client with +`retry: { max_retries: 0 }` to turn retries off for those calls. + ## Retries Retry behaviour follows the official Typesafe SDKs and lives in @@ -338,7 +385,7 @@ return is still accepted and treated as having no headers, which means no | Error | Meaning | | --- | --- | | `ConfigurationError` | No provider could be resolved, a missing key or setting (api_key, account_id, base_url), unknown provider, a `model:` Databricks can't take, or bad `retry:` value | -| `RequestError` | Questions hash was empty, images were not data URLs or went to a provider that doesn't read them, or state or questions could not be encoded as JSON (original error on `#cause`) | +| `RequestError` | Questions hash was empty, images, audio, or videos were not data URLs or went to a provider or model that doesn't read them, or state or questions could not be encoded as JSON (original error on `#cause`) | | `TransportError` (`TimeoutError`) | Network or timeout failure after retries, carries `#cause_error` | | `ApiError` | Non-2xx response, carries `#status`, `#body`, and `#headers`. The message ends with the vendor's own reason when the body has one | | `Unauthorized` | 401 | @@ -349,7 +396,7 @@ return is still accepted and treated as having no headers, which means no | `InvalidResponse` | Body wasn't JSON, wasn't a Hash, or an answer was malformed (including a noul outside 0..1 or an id answered twice) | | `MissingAnswers` | One or more question ids came back missing, wrong-typed, or refused. Carries `#missing`, `#refused` (the subset the provider declined), and `#answers` | -Status: 0.2.0, API may change. +Status: 0.3.0, API may change. The companion gem `decide` builds decisions and verdicts on top of this client. @@ -364,6 +411,8 @@ anywhere. To ship a version: request per configured provider and prints the answers. CI never runs it. `TARGETS=open_router:clef,open_router:luna` picks provider and model pairs, and `SMOKE_IMAGE=photo.png` adds an image where supported. + `SMOKE_AUDIO=clip.mp3` and `SMOKE_VIDEO=clip.mp4` add audio and video + where the model reads them, such as `TARGETS=cloudflare:clef-omni`. 4. Merge to `main`. The Release workflow runs the suite, builds the gem with `gem build --strict`, checks the built gem carries every file under `lib/`, and pushes it. A version already on RubyGems is skipped, so the diff --git a/Rakefile b/Rakefile index 4ead84f..8dca031 100644 --- a/Rakefile +++ b/Rakefile @@ -18,6 +18,8 @@ task default: :test # rake smoke every configured provider # rake smoke TARGETS=open_router:clef,perplexity provider[:model] pairs # rake smoke SMOKE_IMAGE=photo.png also send an image where supported +# rake smoke TARGETS=cloudflare:clef-omni SMOKE_AUDIO=clip.mp3 SMOKE_VIDEO=clip.mp4 +# also send audio and video where supported desc "Live smoke test against each configured provider (needs real keys)" task :smoke do $LOAD_PATH.unshift File.expand_path("lib", __dir__) @@ -47,6 +49,8 @@ task :smoke do criteria: ["No impact", "Minor", "Major", "Critical"]) } image = ENV["SMOKE_IMAGE"].to_s.empty? ? nil : rdm::Images.from_file(ENV["SMOKE_IMAGE"]) + audio = ENV["SMOKE_AUDIO"].to_s.empty? ? nil : rdm::Media.from_file(ENV["SMOKE_AUDIO"]) + video = ENV["SMOKE_VIDEO"].to_s.empty? ? nil : rdm::Media.from_file(ENV["SMOKE_VIDEO"]) failures = 0 targets.each do |name, model| @@ -54,11 +58,15 @@ task :smoke do begin client = rdm::Client.new(provider: name, model: model) images = image && client.provider.supports_images? ? [image] : nil + clips = audio && client.provider.supports_audio?(client.model) ? [audio] : nil + videos = video && client.provider.supports_videos?(client.model) ? [video] : nil started = Process.clock_gettime(Process::CLOCK_MONOTONIC) - response = client.ask(state: state, questions: questions, images: images) + response = client.ask(state: state, questions: questions, images: images, audio: clips, videos: videos) elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round - puts "ok #{label} model=#{response.model.inspect} #{elapsed}ms#{images ? ' +image' : ''}" + media = { "image" => [image, images], "audio" => [audio, clips], "video" => [video, videos] } + marks = media.filter_map { |kind, (given, sent)| " #{sent ? '+' : 'skipped '}#{kind}" if given }.join + puts "ok #{label} model=#{response.model.inspect} #{elapsed}ms#{marks}" puts format(" outage=%.3f urgent=%.3f team=%s (%.2f) severity=%.2f (%.2f)", response["outage"].noul, response["urgent"].noul, response["team"].choice, response["team"].confidence, response["severity"].score, response["severity"].confidence) diff --git a/lib/ruby_decision_model.rb b/lib/ruby_decision_model.rb index 6cb462b..159862e 100644 --- a/lib/ruby_decision_model.rb +++ b/lib/ruby_decision_model.rb @@ -7,6 +7,7 @@ require_relative "ruby_decision_model/errors" require_relative "ruby_decision_model/questions" require_relative "ruby_decision_model/images" +require_relative "ruby_decision_model/media" require_relative "ruby_decision_model/answers" require_relative "ruby_decision_model/response" require_relative "ruby_decision_model/providers" diff --git a/lib/ruby_decision_model/client.rb b/lib/ruby_decision_model/client.rb index b3668d2..f54c002 100644 --- a/lib/ruby_decision_model/client.rb +++ b/lib/ruby_decision_model/client.rb @@ -60,12 +60,13 @@ def base_url @provider.base_url end - # images: data URLs (see Images) for providers that read images. Each - # provider places them where its API expects. - def ask(state:, questions:, images: nil) + # images:, audio:, videos: data URLs (see Images and Media) for providers + # and models that read them. Each provider places them where its API + # expects. + def ask(state:, questions:, images: nil, audio: nil, videos: nil) raise RequestError, "questions must not be empty" if questions.nil? || questions.empty? - body = build_body(state, questions, images) + body = build_body(state, questions, images: images, audio: audio, videos: videos) status, response_body, response_headers = perform_with_retry( url: request_url, headers: @provider.headers, body: body ) @@ -81,25 +82,44 @@ def request_url takes_model ? @provider.url(@model) : @provider.url end - def build_body(state, questions, images) - images = Array(images) - return @provider.request_body(model: @model, state: state, questions: questions) if images.empty? + MEDIA_PREFIXES = { images: "data:image/", audio: "data:audio/", videos: "data:video/" }.freeze + private_constant :MEDIA_PREFIXES - raise RequestError, "#{@provider.name} does not accept images" unless @provider.supports_images? + def build_body(state, questions, **media) + media = media.transform_values { |urls| Array(urls) }.reject { |_kind, urls| urls.empty? } - images.each do |image| - next if image.is_a?(String) && image.start_with?("data:image/") + media.each do |kind, urls| + raise RequestError, unsupported_message(kind) unless accepts?(kind) - raise RequestError, "images must be data URLs (data:image/...); see RubyDecisionModel::Images" + prefix = MEDIA_PREFIXES.fetch(kind) + next if urls.all? { |url| url.is_a?(String) && url.start_with?(prefix) } + + raise RequestError, "#{kind} must be data URLs (#{prefix}...); see RubyDecisionModel::#{kind == :images ? 'Images' : 'Media'}" end - @provider.request_body(model: @model, state: state, questions: questions, images: images) + @provider.request_body(model: @model, state: state, questions: questions, **media) rescue JSON::GeneratorError, JSON::NestingError => e # The generator's message can quote the offending value, so it stays # on #cause rather than in a message that may be logged. raise RequestError, "state or questions could not be encoded as JSON (#{e.class})" end + # supports_images? takes no argument in 0.2.0 providers, so it is called + # bare; audio and videos can differ per model, so those get the model. + def accepts?(kind) + case kind + when :images then @provider.supports_images? + when :audio then @provider.supports_audio?(@model) + when :videos then @provider.supports_videos?(@model) + end + end + + def unsupported_message(kind) + return "#{@provider.name} does not accept images" if kind == :images + + "#{[@provider.name, @model].compact.join(' ')} does not accept #{kind}" + end + def resolve_provider(provider, api_key:, base_url:) case provider when Providers::Base diff --git a/lib/ruby_decision_model/media.rb b/lib/ruby_decision_model/media.rb new file mode 100644 index 0000000..63c686c --- /dev/null +++ b/lib/ruby_decision_model/media.rb @@ -0,0 +1,36 @@ +# frozen_string_literal: true + +require_relative "images" + +module RubyDecisionModel + # Builds the base64 data URLs that Client#ask takes as images:, audio:, and + # videos:. Images has the image-only version of the same helpers. + module Media + CONTENT_TYPES = Images::CONTENT_TYPES.merge( + ".wav" => "audio/wav", + ".mp3" => "audio/mpeg", + ".mp4" => "video/mp4", + ".webm" => "video/webm" + ).freeze + + module_function + + def data_url(bytes, content_type:) + unless content_type.to_s.start_with?("image/", "audio/", "video/") + raise ArgumentError, "content_type must be an image/*, audio/*, or video/* type, got #{content_type.inspect}" + end + + # pack("m0") is strict base64 without the base64 gem, which leaves the + # default gems in Ruby 3.4. + "data:#{content_type};base64,#{[bytes].pack('m0')}" + end + + # Reads a file and infers its type from the extension. + def from_file(path, content_type: nil) + content_type ||= CONTENT_TYPES[File.extname(path.to_s).downcase] + raise ArgumentError, "cannot infer a media type for #{path}; pass content_type:" if content_type.nil? + + data_url(File.binread(path), content_type: content_type) + end + end +end diff --git a/lib/ruby_decision_model/providers/base.rb b/lib/ruby_decision_model/providers/base.rb index bcf643a..2e34cdc 100644 --- a/lib/ruby_decision_model/providers/base.rb +++ b/lib/ruby_decision_model/providers/base.rb @@ -65,6 +65,18 @@ def supports_images? false end + # Whether requests may carry audio or videos. Client refuses audio: and + # videos: for a provider that says no, before anything goes over the + # wire. The model is passed because support can differ between one + # provider's models. + def supports_audio?(_model = nil) + false + end + + def supports_videos?(_model = nil) + false + end + # Response header carrying the provider's request id, or nil. # Typesafe's, which 0.1.0 read for every provider and System One # servers that follow Typesafe send too. Vendors with their own header @@ -129,12 +141,14 @@ def headers headers end - # Client passes images: only when the caller supplied some, so a - # subclass that does not take images can keep a three-keyword - # signature. - def request_body(model:, state:, questions:, images: nil) + # Client passes images:, audio:, and videos: only when the caller + # supplied some and the provider said it supports them, so a subclass + # that takes none can keep a three-keyword signature. + def request_body(model:, state:, questions:, images: nil, audio: nil, videos: nil) body = { "model" => model, "state" => state, "questions" => questions } body["images"] = images if images + body["audio"] = audio if audio + body["videos"] = videos if videos JSON.generate(body) end diff --git a/lib/ruby_decision_model/providers/cloudflare.rb b/lib/ruby_decision_model/providers/cloudflare.rb index de384ee..83286cf 100644 --- a/lib/ruby_decision_model/providers/cloudflare.rb +++ b/lib/ruby_decision_model/providers/cloudflare.rb @@ -4,8 +4,9 @@ module RubyDecisionModel module Providers - # Cloudflare's Clef and Clef-flash on Workers AI. The body is System One - # with Cloudflare's images extension; the model also appears in the + # Cloudflare's Clef, Clef-flash, and Clef-omni on Workers AI. The body is + # System One with Cloudflare's images, audio, and videos extensions + # (audio and videos on Clef-omni only); the model also appears in the # path, and the account id is required. Responses arrive inside the # Workers AI envelope ({"result": ..., "success": true}), which is # unwrapped here. @@ -14,9 +15,14 @@ class Cloudflare < Base "cloudflare/clef" => "clef", "cloudflare/clef-flash" => "clef-flash", "@cf/cloudflare/clef" => "clef", - "@cf/cloudflare/clef-flash" => "clef-flash" + "@cf/cloudflare/clef-flash" => "clef-flash", + "cloudflare/clef-omni" => "clef-omni", + "@cf/cloudflare/clef-omni" => "clef-omni" }.freeze + # The models that read audio and videos as well as images. + OMNI_MODELS = %w[clef-omni].freeze + # Cloudflare's own tooling reads CLOUDFLARE_API_TOKEN; the Clef docs # use CLOUDFLARE_AUTH_TOKEN. Either works, in that order. FALLBACK_ENV_VAR = "CLOUDFLARE_AUTH_TOKEN" @@ -76,6 +82,14 @@ def supports_images? true end + def supports_audio?(model = nil) + OMNI_MODELS.include?(model.to_s) + end + + def supports_videos?(model = nil) + OMNI_MODELS.include?(model.to_s) + end + def request_id_header "cf-ray" end diff --git a/lib/ruby_decision_model/version.rb b/lib/ruby_decision_model/version.rb index 6a6e3b5..3c48667 100644 --- a/lib/ruby_decision_model/version.rb +++ b/lib/ruby_decision_model/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module RubyDecisionModel - VERSION = "0.2.0" + VERSION = "0.3.0" end diff --git a/ruby_decision_model.gemspec b/ruby_decision_model.gemspec index 0b32ef1..1c8b5f7 100644 --- a/ruby_decision_model.gemspec +++ b/ruby_decision_model.gemspec @@ -11,7 +11,7 @@ Gem::Specification.new do |spec| spec.summary = "The decision-model interface for Ruby: one client for Jev, gpt-6-luna, Clef, pplx-decider, and more" spec.description = "Decision models answer typed questions about a state instead of generating text. " \ "ruby_decision_model builds noul (yes/no probability), choice, and score questions, " \ - "posts them with a state and optional images to a decision-model provider " \ + "posts them with a state and optional images, audio, and video to a decision-model provider " \ "(OpenRouter, Typesafe, OpenAI, Cloudflare, Perplexity, Databricks, or any System One " \ "server such as Ollama), and returns the same normalized answers with probabilities, " \ "confidence, legends, and usage whichever vendor served them. Retries follow the " \ diff --git a/test/providers_test.rb b/test/providers_test.rb index 14ee9d4..bd1cfcb 100644 --- a/test/providers_test.rb +++ b/test/providers_test.rb @@ -371,6 +371,109 @@ def test_image_data_url_and_from_file end end + def test_audio_and_videos_are_refused_by_providers_that_do_not_read_them + transport = FakeTransport.new([[200, success_body, {}]]) + client = RubyDecisionModel::Client.new(provider: :typesafe, api_key: "k", transport: transport, sleeper: no_sleep) + audio = RubyDecisionModel::Media.data_url("x", content_type: "audio/mpeg") + video = RubyDecisionModel::Media.data_url("x", content_type: "video/mp4") + + assert_raises(RubyDecisionModel::RequestError) { client.ask(state: {}, questions: questions, audio: [audio]) } + assert_raises(RubyDecisionModel::RequestError) { client.ask(state: {}, questions: questions, videos: [video]) } + assert_empty transport.calls + end + + def test_calls_without_media_pass_exactly_the_three_keywords + klass = Class.new(legacy_provider_class) do + attr_reader :seen + + def request_body(**keywords) + @seen = keywords.keys + JSON.generate(keywords) + end + end + provider = klass.new(api_key: "k") + transport = FakeTransport.new([[200, success_body, {}], [200, success_body, {}]]) + client = RubyDecisionModel::Client.new(provider: provider, transport: transport, sleeper: no_sleep) + + client.ask(state: {}, questions: questions) + assert_equal %i[model state questions], provider.seen + client.ask(state: {}, questions: questions, images: [], audio: nil, videos: []) + assert_equal %i[model state questions], provider.seen + end + + def test_image_error_messages_are_unchanged + typesafe = RubyDecisionModel::Client.new(provider: :typesafe, api_key: "k", transport: FakeTransport.new([]), + sleeper: no_sleep) + image = RubyDecisionModel::Images.data_url("x", content_type: "image/png") + error = assert_raises(RubyDecisionModel::RequestError) { typesafe.ask(state: {}, questions: questions, images: [image]) } + assert_equal "typesafe does not accept images", error.message + + openai = RubyDecisionModel::Client.new(provider: :openai, api_key: "k", transport: FakeTransport.new([]), + sleeper: no_sleep) + error = assert_raises(RubyDecisionModel::RequestError) { openai.ask(state: {}, questions: questions, images: [nil]) } + assert_equal "images must be data URLs (data:image/...); see RubyDecisionModel::Images", error.message + end + + def test_provider_with_a_zero_argument_supports_images_and_images_only_request_body_still_works + klass = Class.new(legacy_provider_class) do + def supports_images? = true + + def request_body(model:, state:, questions:, images: nil) + JSON.generate("model" => model, "images" => images) + end + end + transport = FakeTransport.new([[200, success_body, {}]]) + client = RubyDecisionModel::Client.new(provider: klass.new(api_key: "k"), transport: transport, sleeper: no_sleep) + image = RubyDecisionModel::Images.data_url("x", content_type: "image/png") + audio = RubyDecisionModel::Media.data_url("x", content_type: "audio/mpeg") + + client.ask(state: {}, questions: questions, images: [image]) + assert_equal [image], JSON.parse(transport.calls.first[:body])["images"] + + error = assert_raises(RubyDecisionModel::RequestError) do + client.ask(state: {}, questions: questions, images: [image], audio: [audio]) + end + assert_includes error.message, "does not accept audio" + assert_equal 1, transport.calls.size + end + + def test_media_data_url_and_from_file_reject_bad_input + assert_raises(ArgumentError) { RubyDecisionModel::Media.data_url("hi", content_type: nil) } + + Dir.mktmpdir do |dir| + missing = File.join(dir, "missing.mp3") + assert_raises(Errno::ENOENT) { RubyDecisionModel::Media.from_file(missing) } + + other = File.join(dir, "clip.ogg") + error = assert_raises(ArgumentError) { RubyDecisionModel::Media.from_file(other) } + assert_includes error.message, other + end + end + + def test_media_data_url + assert_equal "data:audio/mpeg;base64,aGk=", RubyDecisionModel::Media.data_url("hi", content_type: "audio/mpeg") + assert_equal "data:video/mp4;base64,aGk=", RubyDecisionModel::Media.data_url("hi", content_type: "video/mp4") + assert_equal "data:image/png;base64,aGk=", RubyDecisionModel::Media.data_url("hi", content_type: "image/png") + assert_raises(ArgumentError) { RubyDecisionModel::Media.data_url("hi", content_type: "text/plain") } + end + + def test_media_from_file_infers_the_type_from_the_extension + types = { "a.wav" => "audio/wav", "a.MP3" => "audio/mpeg", "v.mp4" => "video/mp4", "v.webm" => "video/webm", + "p.png" => "image/png" } + Dir.mktmpdir do |dir| + types.each do |name, type| + path = File.join(dir, name) + File.binwrite(path, "hi") + assert_equal "data:#{type};base64,aGk=", RubyDecisionModel::Media.from_file(path) + end + + other = File.join(dir, "clip.ogg") + File.binwrite(other, "hi") + assert_raises(ArgumentError) { RubyDecisionModel::Media.from_file(other) } + assert_equal "data:audio/ogg;base64,aGk=", RubyDecisionModel::Media.from_file(other, content_type: "audio/ogg") + end + end + # --- timeouts --- def test_timeout_defaults_come_from_the_provider diff --git a/test/vendor_providers_test.rb b/test/vendor_providers_test.rb index 17935fb..cf10dd7 100644 --- a/test/vendor_providers_test.rb +++ b/test/vendor_providers_test.rb @@ -119,6 +119,135 @@ def test_cloudflare_sends_images_as_the_images_extension assert_equal [png], JSON.parse(transport.calls.first[:body])["images"] end + def test_cloudflare_clef_omni_sends_images_audio_and_videos + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + audio = RubyDecisionModel::Media.data_url("ID3", content_type: "audio/mpeg") + video = RubyDecisionModel::Media.data_url("ftyp", content_type: "video/mp4") + cloudflare(transport, model: "clef-omni").ask(state: "What happens?", questions: questions, + images: [png], audio: [audio], videos: [video]) + + call = transport.calls.first + assert call[:url].end_with?("/@cf/cloudflare/clef-omni") + body = JSON.parse(call[:body]) + assert_equal "clef-omni", body["model"] + assert_equal [png], body["images"] + assert_equal [audio], body["audio"] + assert_equal [video], body["videos"] + end + + def test_cloudflare_clef_omni_sends_only_the_media_given + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + audio = RubyDecisionModel::Media.data_url("RIFF", content_type: "audio/wav") + cloudflare(transport, model: "clef-omni").ask(state: "x", questions: questions, audio: [audio]) + + body = JSON.parse(transport.calls.first[:body]) + assert_equal [audio], body["audio"] + refute body.key?("images") + refute body.key?("videos") + end + + def test_cloudflare_clef_and_clef_flash_refuse_audio_and_videos + audio = RubyDecisionModel::Media.data_url("ID3", content_type: "audio/mpeg") + video = RubyDecisionModel::Media.data_url("ftyp", content_type: "video/mp4") + + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + error = assert_raises(RubyDecisionModel::RequestError) do + cloudflare(transport).ask(state: "x", questions: questions, audio: [audio]) + end + assert_includes error.message, "clef does not accept audio" + assert_empty transport.calls + + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + error = assert_raises(RubyDecisionModel::RequestError) do + cloudflare(transport, model: "clef-flash").ask(state: "x", questions: questions, videos: [video]) + end + assert_includes error.message, "does not accept videos" + assert_empty transport.calls + end + + def test_cloudflare_clef_omni_aliases_resolve + %w[cloudflare/clef-omni @cf/cloudflare/clef-omni].each do |name| + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + client = cloudflare(transport, model: name) + client.ask(state: "x", questions: questions) + + assert_equal "clef-omni", client.model + assert transport.calls.first[:url].end_with?("/clef-omni") + end + end + + def test_cloudflare_clef_omni_audio_and_videos_must_be_matching_data_urls + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + client = cloudflare(transport, model: "clef-omni") + + error = assert_raises(RubyDecisionModel::RequestError) do + client.ask(state: "x", questions: questions, audio: ["https://example.test/a.mp3"]) + end + assert_includes error.message, "data:audio/" + error = assert_raises(RubyDecisionModel::RequestError) do + client.ask(state: "x", questions: questions, videos: ["data:image/png;base64,AAAA"]) + end + assert_includes error.message, "data:video/" + assert_raises(RubyDecisionModel::RequestError) { client.ask(state: "x", questions: questions, audio: [png]) } + assert_empty transport.calls + end + + def test_cloudflare_clef_omni_ignores_empty_audio_and_videos + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + cloudflare(transport, model: "clef-omni").ask(state: "x", questions: questions, audio: [], videos: []) + + body = JSON.parse(transport.calls.first[:body]) + refute body.key?("audio") + refute body.key?("videos") + end + + def test_cloudflare_supports_audio_and_videos_only_for_clef_omni + provider = RubyDecisionModel::Providers::Cloudflare.new(api_key: "cf-token", account_id: "acc123") + + assert provider.supports_audio?("clef-omni") + assert provider.supports_videos?("clef-omni") + %w[clef clef-flash].each do |model| + refute provider.supports_audio?(model) + refute provider.supports_videos?(model) + end + refute provider.supports_audio? + refute provider.supports_videos?(nil) + end + + def test_cloudflare_clef_omni_refuses_any_element_that_is_not_a_matching_data_url + audio = RubyDecisionModel::Media.data_url("ID3", content_type: "audio/mpeg") + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + client = cloudflare(transport, model: "clef-omni") + + [[audio, "https://example.test/a.mp3"], [audio, nil], [nil], [audio, :clip], [audio, png]].each do |urls| + assert_raises(RubyDecisionModel::RequestError) { client.ask(state: "x", questions: questions, audio: urls) } + end + error = assert_raises(RubyDecisionModel::RequestError) do + client.ask(state: "x", questions: questions, videos: ["data:audio/mpeg;base64,AAAA"]) + end + assert_equal "videos must be data URLs (data:video/...); see RubyDecisionModel::Media", error.message + assert_empty transport.calls + end + + def test_cloudflare_clef_omni_wraps_a_single_data_url_string + audio = RubyDecisionModel::Media.data_url("ID3", content_type: "audio/mpeg") + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + cloudflare(transport, model: "clef-omni").ask(state: "x", questions: questions, audio: audio) + + assert_equal [audio], JSON.parse(transport.calls.first[:body])["audio"] + end + + def test_cloudflare_clef_refuses_the_whole_request_when_audio_comes_with_images + audio = RubyDecisionModel::Media.data_url("ID3", content_type: "audio/mpeg") + transport = FakeTransport.new([[200, cloudflare_body, {}]]) + + error = assert_raises(RubyDecisionModel::RequestError) do + cloudflare(transport).ask(state: "x", questions: questions, images: [png], audio: [audio]) + end + assert_equal "cloudflare clef does not accept audio", error.message + assert_empty transport.calls + end + def test_cloudflare_blank_api_token_falls_back_to_auth_token without_provider_env do with_env("CLOUDFLARE_API_TOKEN" => " ", "CLOUDFLARE_AUTH_TOKEN" => "auth-token", "CLOUDFLARE_ACCOUNT_ID" => "acc") do