Skip to content

[Feat] Add a served OneJev multimodal decision backend (System One + media) #88

Closed

Description

@Birfy

Why

system1-agents already has an in-process OmniJev path (--model omnijev, MSO1.system_one over a local clone) for screenshot + text choice/noul on the browser front. There is no served OmniJev/OneJev worker in System1-Omni today (capabilities matrix notes Omni vision support is Laya / Open-Jev text / Cua-S1 screenshots).

Separately, the OmniJev org published OneJev: open Apache-2.0 multimodal System One models (0.8B / 4B / 9B / 27B) that answer typed Choice / Noul / Score questions over screenshots (and text/video) in one forward pass, served by qev with a TypeSafe-compatible /v1/systemone API plus a media field.

This fits the existing served decision-model seam (laya-served, PR #86's served Cua-S1 client pattern): an HTTP client that posts observation + questions and maps answers into Decision, without cloning OmniJev into the Agents process.

Distinct from:

Proposed scope (one PR)

  1. Add --model onejev (name bikeshed OK) as a served client: env ONEJEV_URL (or reuse a documented System One URL), optional model id, timeout, and provenance fields (url, request_id, served_by) matching other served adapters.
  2. Map Agents choice / noul questions onto the OneJev/qev client; send the browser/desktop screenshot through the documented media / <image:N> state convention. Refuse cleanly when the observation has no image (same policy as OmniJev).
  3. Contract tests with a fake HTTP server (no GPU/weights in CI): happy path, missing image, transport error, malformed answers.
  4. Docs: docs/decision-models.md, docs/configuration.md, docs/capabilities.md — mark registration vs hardware validation separately; do not claim task success rates without a pinned run artifact.
  5. Optional smoke recipe: start qev serve --model OmniJev/OneJev-4B locally and run one browser tick or decide_many with a fixture screenshot (document only; do not gate CI on GPU).

Acceptance criteria

  • build_model("onejev") (or chosen name) works against a stub server in tests.
  • Env vars and failure modes documented; image-required path matches OmniJev semantics.
  • Capability matrix row added with validation marked unverified until a real run lands.
  • NOTICE / third-party attribution for OneJev / qev as needed.

Out of scope

  • Vendoring OneJev weights or training code.
  • Replacing in-process OmniJev in this PR.
  • Full Flights/OSWorld success-rate campaigns (follow-up eval issue after the adapter lands).

References

Activity

  1. linear3735 commented on Oct 28, 2026

    @linear3735

    I'd like to take this on. I plan to add --model onejev as a served client, reusing the existing DecisionModel validation and HTTP transport patterns for choice/noul, the documented media + image-marker mapping, and clean failure handling. The existing in-process OmniJev backend will remain available.

    I'll cover registration and the wire contract with offline tests, then establish a bounded local baseline on an Apple M5 (16 GiB) using OneJev1.6-0.8B GGUF and qev/llama.cpp Metal. I'll pin the model assets and runtime, reuse the existing benchmark/evidence tooling, and report actual latency, memory, failures, and validation limits separately from the adapter's CPU tests. Kernel changes, broad task-success claims, and automated optimization are outside this initial contribution.

    I'll prepare a focused change with independent review before proposing a PR. Please let me know if this overlaps work already in progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions