Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
name: CI

on: [push, pull_request]
on:
push:
branches: [main]
pull_request:

permissions:
contents: read
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "openai-api-dispatch"
version = "0.1.0"
version = "0.1.1"
authors = ["Victor Lopez <vhrlopes@gmail.com>"]
description = "OpenAI-compatible chat requests through memory or NATS queues."
edition = "2024"
Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,21 +59,32 @@ Environment settings are read when constructing producers, workers, and executor
| Variable | Default |
| --- | --- |
| `OPENAI_API_NATS_URL` | `nats://localhost:4222` |
| `OPENAI_API_NATS_WORKERS_GROUP` | `task_workers` |
| `OPENAI_API_NATS_PREFIX` | `openai-api-queue/` |
| `OPENAI_API_URL` | `http://127.0.0.1:8000/v1` |
| `OPENAI_API_DEFAULT_MODEL` | Unset; required by NATS workers |

- A task's explicit model overrides the default and selects its NATS subject. Prefix and model are concatenated verbatim. Workers share the `task_workers` queue group; this uses Core NATS without persistence or task retries. Constructors do not wait for server confirmation of subscriptions, so startup can race with publishing.
- Each worker handles one task at a time. Queue/API errors stop its loop without an error reply; supervise spawned workers. `send_and_wait` limits only the reply wait, not submission, and expiry does not cancel execution. Check `response.success` even when the call returns `Ok`.
- Only non-streaming chat is implemented. The current prompt is sent as a user message; system entries in input history are ignored (use `with_system`). Returned history currently duplicates the latest user prompt before the assistant reply. Schemas request strict JSON output without local validation. Keep `max_tokens` within `u32`; it is cast unchecked. `payload` is caller metadata, not model input.
- A task's explicit model overrides the default and selects its NATS subject.
- Constructors do not wait for server confirmation of subscriptions, so startup can race with publishing.
- Each worker handles one task at a time.
- Queue/API errors stop its loop without an error reply; supervise spawned workers.
- `send_and_wait` limits only the reply wait, not submission, and expiry does not cancel execution.
- Check `response.success` even when the call returns `Ok`.
- Only non-streaming chat is implemented.
- The current prompt is sent as a user message; system entries in input history are ignored (use `with_system`).
- Schemas request strict JSON output without local validation.
- `payload` is caller metadata, not model input.
- `no_std` generated IDs can repeat after restart or wraparound.

## Features

Default features are `std`, `memory-queue`, and `nats-queue`; NATS implies `std`. The API executor requires `std`.

For `no_std` memory queues, set `default-features = false, features = ["memory-queue"]`. An allocator and pointer/32-bit atomics are required. This backend polls once, has no timeout support, and evicts old items when bounded; the `std` backend uses bounded Tokio channels and backpressure. Use positive capacities and unique task IDs: `TaskBuilder::default()` uses ID zero, and `no_std` generated IDs can repeat after restart or wraparound.
#### no_std

Custom queues/executors implement `QueueProducer`, `QueueWorker`, and `Executor`. Only the NATS/API pairing has a public `Worker` constructor; other combinations need a caller-managed loop.
For memory queues, use the feature `memory-queue`. An allocator and pointer/32-bit atomics are required.

This backend polls once, has no timeout support, and evicts old items when bounded; the `std` backend uses bounded Tokio channels and backpressure.

## Development

Expand Down
1 change: 0 additions & 1 deletion src/executor/openai/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,6 @@ impl ExecutorAsyncOpenai {
None
}
})
.chain(iter::once(Interaction::User(prompt)))
.chain(iter::once(Interaction::Assistant(m.clone())))
.collect();

Expand Down
6 changes: 4 additions & 2 deletions src/queue/nats/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ pub struct NatsProducer {
}

#[derive(Debug, Clone)]
/// Consumes one model subject in the fixed `task_workers` queue group.
/// Consumes one model subject in the workers queue group.
///
/// Clones share one subscription. Replies require the incoming message's reply
/// subject; malformed JSON and missing reply subjects return errors.
Expand Down Expand Up @@ -103,8 +103,10 @@ impl NatsWorker {
.connect(url)
.await?;

let workers_group =
env::var("OPENAI_API_NATS_WORKERS_GROUP").unwrap_or_else(|_| "task_workers".into());
let subscriber = client
.queue_subscribe(subject.clone(), "task_workers".to_string())
.queue_subscribe(subject.clone(), workers_group)
.await?;
let subscriber = Arc::new(Mutex::new(subscriber));

Expand Down
16 changes: 14 additions & 2 deletions src/task.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use serde_json::Value;

use crate::{queue::QueueProducer, utils};

#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
/// Builds a chat task; only the prompt is required at build time.
///
/// Prefer [`Self::new`] for a generated ID: derived `Default` uses ID zero.
Expand All @@ -30,12 +30,24 @@ pub struct TaskBuilder {
pub prompt: Option<String>,
}

impl Default for TaskBuilder {
fn default() -> Self {
Self::new()
}
}

impl TaskBuilder {
/// Creates an empty builder with an ID from [`utils::id`].
pub fn new() -> Self {
Self {
id: utils::id(),
..Default::default()
model: None,
max_tokens: None,
payload: None,
system: None,
history: None,
schema: None,
prompt: None,
}
}

Expand Down