Skip to content

Select action response-header timeouts from the manifest - #93

Open
CryptoJones wants to merge 3 commits into
NVIDIA:developfrom
CryptoJones:fix/action-timeout-per-manifest
Open

CryptoJones wants to merge 3 commits into
NVIDIA:developfrom
CryptoJones:fix/action-timeout-per-manifest

Conversation

@CryptoJones

Copy link
Copy Markdown

Description

Closes #25.

nvpair-engine-manager defined two HTTP clients — a 30s response-header bound for everything and a 10m one reserved for Ollama's cold load — and actions.go chose between them with a literal engine == "ollama" && action == "run_model" comparison. Every other engine got 30 seconds for every action, no manifest field could change it, and the shipped lmstudio.json chat action (and any vLLM manifest) inherited the same ceiling: a long prefill against a large model failed with net/http: timeout awaiting response headers, which reads like a broken engine rather than a client-side limit.

This makes the response-header budget a per-action manifest field:

  • New optional timeout_s on http actions (seconds). Validation rejects negative values and placement on non-http actions at manifest load.
  • Default unchanged. An action that does not declare timeout_s gets exactly the same 30s budget as before; third-party manifests are unaffected.
  • No engine-name special case left in the executor. The ollamaLoadClient field and the name comparison are gone; the client is selected by declared value and cached per distinct value (all default-budget actions share one client).
  • Ollama's run_model declares 600s in the bundled manifest — behavior preserved, now data-driven.
  • LM Studio's chat declares 600s in the bundled manifest — the action [Bug]: Action HTTP response-header timeout is selected by a hardcoded engine name, so only Ollama gets the long client #25 was reported against now gets a budget that fits a cold model load or long prefill before the first byte. The total call stays bounded by the executor's per-action timeout (default 30 min), so a declared header budget can never unbound a call.

Scope

In: registry.go (field + validation); executor.go (client selection/caching); both bundled manifests; the README engine:action row; tests; versions.json.

Out (deliberately): JSON-RPC surface changes — timeout_s is manifest data, not a wire field, so no broker relay, desktop bridge, or service-contracts changes; per-engine defaults in code (the manifest is the single source of truth); an upper cap on timeout_s (every action call is already wrapped in the 30-min context deadline).

Validation

Go 1.26.0 on Linux (x86_64). Desktop tree untouched, so the npm gates were not run.

Gate Result
gofmt -l / go vet ./... in nvpair-engine-manager clean
Bundled manifest JSON validity (lmstudio.json, ollama.json) valid
go test ./... in services/nvpair-engine-manager (incl. 6 new tests) ok 46.756s
New behavior tests under -race ok 1.931s

New tests: TestEngineHTTPClientBoundsResponseHeaders, TestActionTimeoutSFollowsManifestNotEngineName, TestBundledLMStudioChatDeclaresResponseHeaderBudget, TestActionClientCachesPerDeclaredTimeout, TestValidateAcceptsActionTimeoutS, TestBundledOllamaRunModelTimeout. The two TestBundled* tests load the real go:embed manifests and pin that both declaring actions keep their long budgets, so a future manifest edit that silently reverts this fix fails CI.

Independent review. The diff was reviewed by a panel of fourteen independently trained models (Anthropic ×2, OpenAI, Google, Z.AI, NVIDIA Nemotron ×2, Mistral, DeepSeek, xAI, Moonshot AI, Nous Research, Amazon, and a local Qwen), one at a time, each blind to the others. Tally: 2 approve, 9 approve-with-nits, 3 request-changes — all three request-changes lanes on the same finding. Every finding was verified against the code before being acted on: (a) confirmed real and fixed in this branch — lmstudio's chat action never declared timeout_s, so the reported action would still have received the 30s default; it now declares 600s with a regression pin; (b) refuted — no upper cap is needed because every action call is wrapped in the 30-min context deadline; (c) confirmed minor and fixed — the README now documents the field; (d) refuted — the executor already guards non-positive/default values before building a client.

Risk

  • Behavior: lmstudio chat's response-header budget rises from 30s to 600s for the first byte only; body streaming is unchanged and the total call remains capped by the per-action timeout. Every other action keeps its existing behavior (30s default). Ollama run_model: same 600s, now declared in data instead of hardcoded.
  • Compatibility: additive optional manifest field. No JSON-RPC method or payload changes; desktop tree untouched.
  • Conflict note for maintainers: open PR Opt-in, API-key-authenticated LAN access to the compatibility endpoints #38 also bumps product/installer to 0.92.0 in versions.json; if it lands first this branch needs a one-line rebase (the values match).

Versions

nvpair-engine-manager 0.17.4 → 0.18.0 (MINOR: additive optional manifest field on the engine:action surface). Per services/VERSIONING.md, a MINOR component bump implies product/installer 0.91.7 → 0.92.0.

AI disclosure

This change was written with AI assistance and reviewed by a panel of AI models, under the direction of the human author, who tested it on his own hardware and takes responsibility for it.

  • Authored: Qwen 3.8 27B (qwen3.8-27b) running as a Hermes agent, operated by Aaron K. Clark (CryptoJones), who set the design decisions (per-action manifest field over an engine-name special case; 600s matching Ollama's existing cold-load budget; MINOR version bump) and ran all validation on his own hardware.
  • Quality review: the diff was put to fourteen independently trained models, one at a time, each blind to the others' answers (round 1 is blind by construction — a lane cannot see an answer that does not exist yet); every finding was verified against the code before being acted on, and the adopted ones are listed under Independent review above.
Reviewer Model
Anthropic Claude Fable (fable), Claude Opus (opus) — via the claude CLI
OpenAI gpt-5.6-sol
Google gemini-3.1-pro-high
Z.AI z-ai/glm-5.3-flash (via OpenRouter)
NVIDIA nvidia/nemotron-3-ultra-550b-a55b, nvidia/nemotron-3-super-120b-a12b (via OpenRouter, free tier)
Mistral mistralai/mistral-large-2512 (via OpenRouter)
DeepSeek deepseek/deepseek-v4-flash (via OpenRouter)
xAI x-ai/grok-4.6 (via OpenRouter)
Moonshot AI moonshotai/kimi-k2.6 (via OpenRouter)
Nous Research nousresearch/hermes-4-405b (via OpenRouter)
Amazon amazon/nova-pro-v1 (via OpenRouter)
Local (Ollama) qwen3.8-27b on the review host

Transcripts of every lane's answer are retained by the author and can be shared with maintainers on request.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

Proudly Made in Nebraska. Go Big Red! 🌽 https://xkcd.com/2347/

🤖 Authored in Hermes (qwen3.8-27b) under the direction of Aaron K. Clark

executor.go defined two HTTP clients — a 30s response-header bound for
everything and a 10m one reserved for Ollama's cold load — and actions.go
chose between them with a literal engine == "ollama" && action ==
"run_model" comparison. Every other engine got 30 seconds for every
action, no manifest field could change it, and the shipped lmstudio.json
chat action (and any vLLM manifest) inherited the same ceiling: a long
prefill against a large model failed with "net/http: timeout awaiting
response headers", which reads like a broken engine rather than a
client-side limit.

Make the budget a property of the action instead: an optional per-action
timeout_s in the manifest, defaulting to the existing 30s when unset (or
equal to it). The executor resolves one cached HTTP client per distinct
declared value and reuses the shared 30s client otherwise. Ollama's cold
run_model moves its 600s into manifests/ollama.json, and the engine-name
comparison and ollamaLoadClient field are deleted. Validation rejects a
negative timeout_s and a timeout_s on a non-http action so it can never
be a silent no-op. The total call stays bounded by the executor's action
timeout regardless.

Tests: the name-based selection test is replaced by one that pins the new
behavior (a declared timeout_s is honored for any engine name; an
undeclared action on an engine named ollama gets the ordinary budget), a
client-caching unit test, validation accept/reject cases, and a pin on
the bundled ollama run_model declaration so it cannot silently regress.

Fixes NVIDIA#25.

Signed-off-by: Aaron K. Clark <cryptojones@owasp.org>
The new optional per-action timeout_s manifest field is an additive
feature visible over the engine:action surface, so MINOR for the
component (0.17.4 -> 0.18.0). Product and installer follow the MINOR
component bump per VERSIONING.md (0.91.7 -> 0.92.0).

Signed-off-by: Aaron K. Clark <cryptojones@owasp.org>
Issue NVIDIA#25 was reported against the lmstudio chat action, but the
mechanism commit only turned the new timeout_s knob on for Ollama's
run_model. Without an opt-in, lmstudio chat still gets the 30s
response-header default and the reported timeout is unchanged. Declare
600s (matching Ollama's cold run_model) so a cold model load or long
prefill no longer cuts off the first byte; the total call stays bounded
by the executor's action timeout. Pin it with a regression test against
the bundled manifest and document the field on the engine:action row.

Signed-off-by: Aaron K. Clark <cryptojones@owasp.org>
@Noah-Tervalon-Nvidia
Noah-Tervalon-Nvidia changed the base branch from main to develop September 21, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Action HTTP response-header timeout is selected by a hardcoded engine name, so only Ollama gets the long client

1 participant