diff --git a/.agents/skills/nimbus-interface/SKILL.md b/.agents/skills/nimbus-interface/SKILL.md index 43965fd..6d0ee4f 100644 --- a/.agents/skills/nimbus-interface/SKILL.md +++ b/.agents/skills/nimbus-interface/SKILL.md @@ -1,6 +1,6 @@ --- name: "nimbus-interface" -description: "Runtime NimbusImage/Girder API reference for worker code. Use for image loading, annotation CRUD, property values, channel merging, coordinate conversions, local API tests, or infrastructure errors such as HTTP 500. Use nimbus-worker-scaffold to create a new worker and nimbus-worker-hardening to diagnose or sweep failures in existing workers." +description: "Runtime NimbusImage/Girder API reference for worker code. Use for image loading, annotation CRUD, property values, channel merging, coordinate conversions, API test scripts, or infrastructure errors such as HTTP 500. Use nimbus-run-worker to execute a built image against a real local dataset, nimbus-worker-scaffold to create a worker, and nimbus-worker-hardening to diagnose or sweep failures." --- # NimbusImage Worker Development @@ -73,6 +73,10 @@ Typical shapes: ## Local Testing +To execute a built production image end to end against a live local dataset, +including safe authentication and output verification, use the +`nimbus-run-worker` skill. This section covers lower-level API test scripts. + ### Avoid importing entrypoint.py Worker entrypoints import heavy ML libraries (torch, sam2) at module level. Copy helper functions locally instead of importing the entrypoint. diff --git a/.agents/skills/nimbus-run-worker/SKILL.md b/.agents/skills/nimbus-run-worker/SKILL.md new file mode 100644 index 0000000..8af4b5e --- /dev/null +++ b/.agents/skills/nimbus-run-worker/SKILL.md @@ -0,0 +1,175 @@ +--- +name: nimbus-run-worker +description: "Run a built NimbusImage Docker worker end to end against a real local Girder-backed dataset, including resolving dataset-view URLs, minting and revoking a short-lived token, constructing compute parameters, attaching to the Nimbus Docker network, and verifying API and frontend results. Use when asked to run, execute, smoke-test, reproduce, or validate a worker on local NimbusImage data. Use nimbus-worker-scaffold to create workers, nimbus-worker-hardening to fix failures, and nimbus-interface for worker API implementation details." +--- + +# Running a NimbusImage Worker Locally + +Run the production Docker image directly while using the live local Girder API +for all reads and writes. This exercises the image entrypoint, `compute()` code, +source downloads, result uploads, and metadata handling. It bypasses only +Celery dispatch and therefore does **not** prove CPU/GPU queue routing. + +A compute run can create annotations, property values, or image items. Run it +only when the user authorized a live result; use `--dry-run` for inspection. + +## Standard workflow + +### 1. Check the stack and image + +Run from the `ImageAnalysisProject` root. Confirm the Girder-backed Nimbus stack +and the worker image exist: + +```bash +curl -fsS http://localhost:8080/api/v1/system/version +docker image inspect :latest +docker network inspect nimbusimage_default +``` + +If Girder returns HTTP 500 beyond `/system/version`, confirm MongoDB and the +rest of the Nimbus stack are running. Read `nimbus-interface` for infrastructure +diagnostics. + +Inspect the image labels before a live run. Direct Docker execution ignores the +queue label, but production dispatch does not: + +```bash +docker image inspect --format \ + '{{ index .Config.Labels "interfaceName" }} GPU={{ index .Config.Labels "isGPUWorker" }}' \ + :latest +``` + +Pass `--gpus all` to the bundled runner only when the image and host require it. + +### 2. Resolve the dataset correctly + +Workers expect the **Girder dataset folder ID** in `--datasetId`. A Nimbus URL +such as `#/datasetView//view` contains a dataset-view ID, not that +folder ID. Resolve it with authenticated `GET /dataset_view/` and use +the response's `datasetId` value. + +The bundled runner accepts either `--dataset-id` or `--dataset-view-url` and +performs this resolution safely. Never pass the large-image item ID merely +because `/item//tiles` works; workers use the folder to select the active +large image and access sibling resources. + +### 3. Build the exact compute payload + +Read the worker's `interface()`, worker documentation, and compute tests. Do not +guess field types or assume interface defaults will fill omitted saved values. +Construct the complete `params` object the worker expects, including top-level +fields such as `tile`, `tags`, `channel`, or `scales` when that worker uses them. + +Write the payload to a private temporary JSON file. Keep credentials out of it: + +```json +{ + "workerInterface": { + "Example option": true, + "Output filename": "validated-output.tiff" + } +} +``` + +For a destructive or expensive worker, use a unique output name and inspect +existing results first. Never delete a prior item just to make a rerun easier +unless the user explicitly requested replacement. + +### 4. Authenticate with a short-lived token and run + +Keep `NIMBUS_API_KEY` in the repository's ignored `.env` or the process +environment. Never print it, enable shell tracing, or pass the long-lived API +key to `--token`. + +Use the bundled runner for the standard local stack: + +```bash +python3 .agents/skills/nimbus-run-worker/scripts/run_worker.py \ + --image :latest \ + --dataset-view-url 'http://localhost:5173/#/datasetView//view' \ + --parameters-file /tmp/worker-parameters.json +``` + +The runner: + +1. reads `NIMBUS_API_KEY` without echoing it; +2. mints a temporary Girder token with `POST /api_key/token`; +3. resolves and validates the dataset folder; +4. snapshots the folder's current items; +5. runs the image with `--rm` on `nimbusimage_default`; +6. reports newly created folder items; and +7. revokes the token with `DELETE /token/session`, including on failure. + +Preview the command shape without API or Docker calls: + +```bash +python3 .agents/skills/nimbus-run-worker/scripts/run_worker.py \ + --image :latest \ + --dataset-id \ + --parameters-file /tmp/worker-parameters.json \ + --dry-run +``` + +Check authentication, dataset-view resolution, the folder, image, and Docker +network without starting the worker: + +```bash +python3 .agents/skills/nimbus-run-worker/scripts/run_worker.py \ + --image :latest \ + --dataset-view-url 'http://localhost:5173/#/datasetView//view' \ + --parameters-file /tmp/worker-parameters.json \ + --preflight-only +``` + +For a differently named Compose project, override `--docker-network`. Override +both API URLs together when ports or service names differ: + +- `--host-api-url`: reachable from the host, used for auth and verification. +- `--container-api-url`: reachable from the worker container, passed as + `--apiUrl` (normally `http://girder:8080/api/v1`). + +The current worker contract necessarily exposes the short-lived token in the +local container process arguments while it runs. Limit this to a trusted local +machine and rely on immediate revocation; never substitute the API key. + +### 5. Verify the result at three boundaries + +Do not treat exit code zero as sufficient. + +1. **Worker report** — retain the final progress/output and check worker-specific + metrics, warnings, counts, and quality gates. +2. **Girder state** — inspect the new or modified resources through the API. + For an image output, check `GET /item/`, wait for + `GET /item//tiles`, confirm axes/frame count/levels, and fetch one or more + exact `GET /item//tiles/region` crops. For annotations or properties, + compare relevant counts and inspect representative records. +3. **Nimbus frontend** — load the dataset from a fresh page, select a new image + in the large-image dropdown when applicable, and verify that it renders. + +Compare the same coordinates and display style for before/after image crops. +For registration or restoration workers, inspect the known worst-case region, +not only an easy area. + +Report the image tag/ID, branch or commit, exact non-secret parameters, output +resource IDs, quantitative gates, and visual check. State explicitly that a +direct run bypassed Celery. If queue routing itself matters, perform a second +run through Nimbus's normal UI/job path and inspect the selected queue. + +## Manual command contract + +If the helper cannot cover a specialized container option, preserve the same +contract and token lifecycle: + +```bash +docker run --rm \ + --network nimbusimage_default \ + :latest \ + --datasetId \ + --apiUrl http://girder:8080/api/v1 \ + --token \ + --request compute \ + --parameters '' +``` + +Use argument arrays or carefully quoted JSON; never use `eval`. Arrange token +revocation in a `finally` block or shell trap before launching the container. diff --git a/.agents/skills/nimbus-run-worker/agents/openai.yaml b/.agents/skills/nimbus-run-worker/agents/openai.yaml new file mode 100644 index 0000000..c5ae61f --- /dev/null +++ b/.agents/skills/nimbus-run-worker/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Run Nimbus Worker" + short_description: "Run and verify Nimbus workers locally" + default_prompt: "Use $nimbus-run-worker to run a built worker against a local NimbusImage dataset and verify its output." diff --git a/.agents/skills/nimbus-run-worker/scripts/run_worker.py b/.agents/skills/nimbus-run-worker/scripts/run_worker.py new file mode 100755 index 0000000..4965752 --- /dev/null +++ b/.agents/skills/nimbus-run-worker/scripts/run_worker.py @@ -0,0 +1,421 @@ +#!/usr/bin/env python3 +"""Run a NimbusImage worker container with a short-lived Girder token.""" + +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +import re +import shlex +import subprocess +import sys +from typing import Any +from urllib.error import HTTPError, URLError +from urllib.parse import quote, urlencode, urlsplit +from urllib.request import Request, urlopen + + +OBJECT_ID_RE = re.compile(r"^[0-9a-fA-F]{24}$") +DATASET_VIEW_RE = re.compile(r"(?:^|/)datasetView/([^/?#]+)") + + +class RunError(RuntimeError): + """An expected, safely reportable worker-run failure.""" + + +def _api_url(api_root: str, path: str) -> str: + return f"{api_root.rstrip('/')}/{path.lstrip('/')}" + + +def _request_json( + api_root: str, + method: str, + path: str, + *, + token: str | None = None, + form: dict[str, str] | None = None, + timeout: float = 30.0, +) -> Any: + headers = {"Accept": "application/json"} + data = None + if token: + headers["Girder-Token"] = token + if form is not None: + data = urlencode(form).encode("utf-8") + headers["Content-Type"] = "application/x-www-form-urlencoded" + request = Request( + _api_url(api_root, path), data=data, headers=headers, method=method + ) + try: + with urlopen(request, timeout=timeout) as response: + body = response.read() + except HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace")[:500] + raise RunError( + f"{method} {path} returned HTTP {exc.code}: {detail}" + ) from None + except URLError as exc: + raise RunError(f"Could not reach Girder at {api_root}: {exc.reason}") from None + if not body: + return None + try: + return json.loads(body) + except json.JSONDecodeError as exc: + raise RunError(f"{method} {path} did not return JSON") from exc + + +def _read_api_key(env_file: Path) -> str: + value = os.environ.get("NIMBUS_API_KEY", "").strip() + if value: + return value + try: + lines = env_file.read_text(encoding="utf-8").splitlines() + except FileNotFoundError: + raise RunError( + f"NIMBUS_API_KEY is unset and environment file {env_file} is missing" + ) from None + for raw_line in lines: + line = raw_line.strip() + if not line or line.startswith("#"): + continue + if line.startswith("export "): + line = line[len("export ") :].lstrip() + key, separator, candidate = line.partition("=") + if separator and key.strip() == "NIMBUS_API_KEY": + candidate = candidate.strip() + if ( + len(candidate) >= 2 + and candidate[0] == candidate[-1] + and candidate[0] in {"'", '"'} + ): + candidate = candidate[1:-1] + if candidate: + return candidate + raise RunError(f"NIMBUS_API_KEY is not defined in {env_file}") + + +def _load_parameters(path: Path) -> tuple[dict[str, Any], str]: + try: + parameters = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + raise RunError(f"Parameters file {path} does not exist") from None + except json.JSONDecodeError as exc: + raise RunError(f"Parameters file {path} is invalid JSON: {exc}") from exc + if not isinstance(parameters, dict): + raise RunError("Worker parameters must be one JSON object") + return parameters, json.dumps(parameters, separators=(",", ":"), sort_keys=True) + + +def _dataset_view_id(value: str) -> str: + parsed = urlsplit(value) + candidates = (parsed.fragment, parsed.path, value) + for candidate in candidates: + match = DATASET_VIEW_RE.search(candidate) + if match: + view_id = match.group(1) + if OBJECT_ID_RE.fullmatch(view_id): + return view_id + raise RunError(f"Dataset-view URL contains an invalid ID: {view_id!r}") + raise RunError( + "Could not find '/datasetView/' in the supplied dataset-view URL" + ) + + +def _mint_session_token(api_root: str, api_key: str) -> str: + response = _request_json( + api_root, "POST", "api_key/token", form={"key": api_key} + ) + try: + token = response["authToken"]["token"] + except (KeyError, TypeError): + raise RunError("Girder did not return authToken.token") from None + if not isinstance(token, str) or not token: + raise RunError("Girder returned an empty session token") + return token + + +def _revoke_session_token(api_root: str, token: str) -> None: + _request_json(api_root, "DELETE", "token/session", token=token) + + +def _resolve_dataset_id(args: argparse.Namespace, token: str) -> str: + if args.dataset_id: + return args.dataset_id + view_id = args.dataset_view_id + if args.dataset_view_url: + view_id = _dataset_view_id(args.dataset_view_url) + response = _request_json( + args.host_api_url, + "GET", + f"dataset_view/{quote(view_id, safe='')}", + token=token, + ) + dataset_id = response.get("datasetId") if isinstance(response, dict) else None + if not isinstance(dataset_id, str) or not OBJECT_ID_RE.fullmatch(dataset_id): + raise RunError(f"Dataset view {view_id} has no valid datasetId") + return dataset_id + + +def _validate_dataset_folder(api_root: str, dataset_id: str, token: str) -> None: + response = _request_json( + api_root, "GET", f"folder/{quote(dataset_id, safe='')}", token=token + ) + if not isinstance(response, dict) or str(response.get("_id")) != dataset_id: + raise RunError(f"Girder did not resolve dataset folder {dataset_id}") + + +def _list_folder_items(api_root: str, dataset_id: str, token: str) -> list[dict]: + items: list[dict] = [] + limit = 50 + offset = 0 + while True: + query = urlencode( + {"folderId": dataset_id, "limit": limit, "offset": offset} + ) + page = _request_json(api_root, "GET", f"item?{query}", token=token) + if not isinstance(page, list): + raise RunError("Girder item listing did not return a list") + items.extend(item for item in page if isinstance(item, dict)) + if len(page) < limit: + return items + offset += len(page) + + +def _preflight(image: str, network: str) -> None: + checks = ( + (["docker", "image", "inspect", image], f"Docker image {image}"), + (["docker", "network", "inspect", network], f"Docker network {network}"), + ) + for command, label in checks: + try: + result = subprocess.run( + command, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + text=True, + check=False, + ) + except OSError as exc: + raise RunError(f"Could not execute Docker: {exc}") from None + if result.returncode: + detail = result.stderr.strip()[:500] + raise RunError(f"{label} is unavailable: {detail}") + + +def _container_name(image: str) -> str: + base = image.rsplit("/", 1)[-1].split(":", 1)[0] + base = re.sub(r"[^a-zA-Z0-9_.-]+", "-", base).strip("-.") or "worker" + return f"nimbus-run-{base}-{os.getpid()}" + + +def _docker_command( + args: argparse.Namespace, + dataset_id: str, + token: str, + parameters_json: str, +) -> list[str]: + command = [ + "docker", + "run", + "--rm", + "--name", + args.container_name or _container_name(args.image), + "--network", + args.docker_network, + ] + if args.gpus: + command.extend(["--gpus", args.gpus]) + command.extend( + [ + args.image, + "--datasetId", + dataset_id, + "--apiUrl", + args.container_api_url, + "--token", + token, + "--request", + "compute", + "--parameters", + parameters_json, + ] + ) + return command + + +def _redacted_command(command: list[str]) -> list[str]: + redacted = list(command) + try: + token_index = redacted.index("--token") + 1 + except ValueError: + return redacted + if token_index < len(redacted): + redacted[token_index] = "" + return redacted + + +def _summary_item(item: dict) -> dict[str, Any]: + return { + "id": item.get("_id"), + "name": item.get("name"), + "size": item.get("size"), + "large_image": bool(item.get("largeImage")), + } + + +def execute(args: argparse.Namespace) -> int: + _, parameters_json = _load_parameters(args.parameters_file) + dry_dataset_id = args.dataset_id or "" + if args.dry_run: + command = _docker_command( + args, + dry_dataset_id, + "", + parameters_json, + ) + print( + json.dumps( + { + "mode": "dry-run", + "api_calls_made": False, + "docker_calls_made": False, + "command": shlex.join(_redacted_command(command)), + }, + indent=2, + sort_keys=True, + ) + ) + return 0 + + api_key = _read_api_key(args.env_file) + token = "" + try: + token = _mint_session_token(args.host_api_url, api_key) + dataset_id = _resolve_dataset_id(args, token) + _validate_dataset_folder(args.host_api_url, dataset_id, token) + _preflight(args.image, args.docker_network) + before_items = _list_folder_items(args.host_api_url, dataset_id, token) + if args.preflight_only: + print( + json.dumps( + { + "mode": "preflight-only", + "dataset_id": dataset_id, + "image": args.image, + "existing_folder_items": len(before_items), + "worker_started": False, + }, + indent=2, + sort_keys=True, + ) + ) + return 0 + before_ids = {str(item.get("_id")) for item in before_items} + command = _docker_command(args, dataset_id, token, parameters_json) + try: + result = subprocess.run(command, check=False) + except OSError as exc: + raise RunError(f"Could not start the worker container: {exc}") from None + after_items = _list_folder_items(args.host_api_url, dataset_id, token) + new_items = [ + _summary_item(item) + for item in after_items + if str(item.get("_id")) not in before_ids + ] + print( + json.dumps( + { + "dataset_id": dataset_id, + "image": args.image, + "container_exit_code": result.returncode, + "new_folder_items": new_items, + }, + indent=2, + sort_keys=True, + ) + ) + if result.returncode: + raise RunError( + f"Worker container exited with status {result.returncode}" + ) + return 0 + finally: + if token: + try: + _revoke_session_token(args.host_api_url, token) + except RunError as exc: + print( + f"Warning: could not revoke the temporary Girder token: {exc}", + file=sys.stderr, + ) + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--image", required=True, help="Built worker image tag") + dataset = parser.add_mutually_exclusive_group(required=True) + dataset.add_argument("--dataset-id", help="Girder dataset folder ID") + dataset.add_argument("--dataset-view-id", help="Nimbus dataset-view ID") + dataset.add_argument("--dataset-view-url", help="Nimbus dataset-view URL") + parser.add_argument( + "--parameters-file", required=True, type=Path, help="Complete params JSON" + ) + parser.add_argument( + "--env-file", + type=Path, + default=Path(".env"), + help="File containing NIMBUS_API_KEY (default: .env)", + ) + parser.add_argument( + "--host-api-url", + default="http://localhost:8080/api/v1", + help="Girder API URL reachable from the host", + ) + parser.add_argument( + "--container-api-url", + default="http://girder:8080/api/v1", + help="Girder API URL reachable from the worker container", + ) + parser.add_argument( + "--docker-network", + default="nimbusimage_default", + help="Docker network shared with Girder", + ) + parser.add_argument("--container-name", help="Optional deterministic name") + parser.add_argument( + "--gpus", help="Value for Docker --gpus, for example 'all'" + ) + mode = parser.add_mutually_exclusive_group() + mode.add_argument( + "--dry-run", + action="store_true", + help="Validate JSON and print a redacted command without API/Docker calls", + ) + mode.add_argument( + "--preflight-only", + action="store_true", + help="Check auth, dataset, image, and network without starting the worker", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + try: + args = _parser().parse_args(argv) + for field in ("dataset_id", "dataset_view_id"): + value = getattr(args, field) + if value and not OBJECT_ID_RE.fullmatch(value): + raise RunError(f"--{field.replace('_', '-')} must be a 24-hex ID") + return execute(args) + except RunError as exc: + print(f"Error: {exc}", file=sys.stderr) + return 2 + except KeyboardInterrupt: + print("Interrupted", file=sys.stderr) + return 130 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/nimbus-run-worker/tests/test_run_worker.py b/.agents/skills/nimbus-run-worker/tests/test_run_worker.py new file mode 100644 index 0000000..6e49219 --- /dev/null +++ b/.agents/skills/nimbus-run-worker/tests/test_run_worker.py @@ -0,0 +1,170 @@ +import argparse +import contextlib +import importlib.util +import io +import json +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest import mock + + +SCRIPT = Path(__file__).parents[1] / "scripts" / "run_worker.py" +SPEC = importlib.util.spec_from_file_location("nimbus_run_worker", SCRIPT) +MODULE = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +SPEC.loader.exec_module(MODULE) + + +class RunWorkerTests(unittest.TestCase): + def _parameters_file(self, directory): + path = Path(directory) / "params.json" + path.write_text( + json.dumps({"workerInterface": {"Enabled": True}}), encoding="utf-8" + ) + return path + + def _args(self, parameters_file, **overrides): + values = { + "image": "annotations/example:latest", + "dataset_id": "a" * 24, + "dataset_view_id": None, + "dataset_view_url": None, + "parameters_file": parameters_file, + "env_file": Path(".env"), + "host_api_url": "http://localhost:8080/api/v1", + "container_api_url": "http://girder:8080/api/v1", + "docker_network": "nimbusimage_default", + "container_name": "test-worker-run", + "gpus": None, + "dry_run": False, + "preflight_only": False, + } + values.update(overrides) + return argparse.Namespace(**values) + + def test_extracts_dataset_view_id_from_hash_route(self): + view_id = "6a94208ad233ea7fbc4328f9" + url = f"http://localhost:5173/#/datasetView/{view_id}/view" + self.assertEqual(MODULE._dataset_view_id(url), view_id) + + def test_reads_quoted_api_key_without_printing_it(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / ".env" + path.write_text('NIMBUS_API_KEY="private-value"\n', encoding="utf-8") + with mock.patch.dict(MODULE.os.environ, {}, clear=True): + self.assertEqual(MODULE._read_api_key(path), "private-value") + + def test_dry_run_redacts_token_and_makes_no_external_calls(self): + with tempfile.TemporaryDirectory() as directory: + parameters_file = self._parameters_file(directory) + args = self._args(parameters_file, dry_run=True) + output = io.StringIO() + with contextlib.redirect_stdout(output), mock.patch.object( + MODULE, "_mint_session_token" + ) as mint, mock.patch.object(MODULE.subprocess, "run") as run: + self.assertEqual(MODULE.execute(args), 0) + report = output.getvalue() + self.assertIn("", report) + self.assertNotIn("private-value", report) + mint.assert_not_called() + run.assert_not_called() + + def test_live_run_revokes_token_and_reports_new_items(self): + with tempfile.TemporaryDirectory() as directory: + parameters_file = self._parameters_file(directory) + args = self._args(parameters_file) + old_item = {"_id": "1", "name": "source.tiff"} + new_item = { + "_id": "2", + "name": "corrected.tiff", + "size": 123, + "largeImage": {"fileId": "3"}, + } + output = io.StringIO() + with contextlib.redirect_stdout(output), mock.patch.object( + MODULE, "_read_api_key", return_value="api-key" + ), mock.patch.object( + MODULE, "_mint_session_token", return_value="temporary-token" + ), mock.patch.object( + MODULE, "_validate_dataset_folder" + ), mock.patch.object( + MODULE, "_preflight" + ), mock.patch.object( + MODULE, + "_list_folder_items", + side_effect=[[old_item], [old_item, new_item]], + ), mock.patch.object( + MODULE.subprocess, + "run", + return_value=subprocess.CompletedProcess([], 0), + ) as run, mock.patch.object( + MODULE, "_revoke_session_token" + ) as revoke: + self.assertEqual(MODULE.execute(args), 0) + command = run.call_args.args[0] + self.assertEqual(command[command.index("--token") + 1], "temporary-token") + self.assertEqual(command[command.index("--request") + 1], "compute") + self.assertNotIn("temporary-token", output.getvalue()) + report = json.loads(output.getvalue()) + self.assertEqual(report["new_folder_items"][0]["id"], "2") + revoke.assert_called_once_with(args.host_api_url, "temporary-token") + + def test_preflight_resolves_resources_without_starting_worker(self): + with tempfile.TemporaryDirectory() as directory: + parameters_file = self._parameters_file(directory) + args = self._args(parameters_file, preflight_only=True) + output = io.StringIO() + with contextlib.redirect_stdout(output), mock.patch.object( + MODULE, "_read_api_key", return_value="api-key" + ), mock.patch.object( + MODULE, "_mint_session_token", return_value="temporary-token" + ), mock.patch.object( + MODULE, "_validate_dataset_folder" + ), mock.patch.object( + MODULE, "_preflight" + ), mock.patch.object( + MODULE, "_list_folder_items", return_value=[{"_id": "1"}] + ), mock.patch.object( + MODULE.subprocess, "run" + ) as run, mock.patch.object( + MODULE, "_revoke_session_token" + ) as revoke: + self.assertEqual(MODULE.execute(args), 0) + report = json.loads(output.getvalue()) + self.assertEqual(report["mode"], "preflight-only") + self.assertFalse(report["worker_started"]) + run.assert_not_called() + revoke.assert_called_once_with(args.host_api_url, "temporary-token") + + def test_failed_worker_still_revokes_token(self): + with tempfile.TemporaryDirectory() as directory: + parameters_file = self._parameters_file(directory) + args = self._args(parameters_file) + output = io.StringIO() + with contextlib.redirect_stdout(output), mock.patch.object( + MODULE, "_read_api_key", return_value="api-key" + ), mock.patch.object( + MODULE, "_mint_session_token", return_value="temporary-token" + ), mock.patch.object( + MODULE, "_validate_dataset_folder" + ), mock.patch.object( + MODULE, "_preflight" + ), mock.patch.object( + MODULE, "_list_folder_items", side_effect=[[], []] + ), mock.patch.object( + MODULE.subprocess, + "run", + return_value=subprocess.CompletedProcess([], 7), + ), mock.patch.object( + MODULE, "_revoke_session_token" + ) as revoke: + with self.assertRaisesRegex(MODULE.RunError, "status 7"): + MODULE.execute(args) + self.assertNotIn("temporary-token", output.getvalue()) + revoke.assert_called_once_with(args.host_api_url, "temporary-token") + + +if __name__ == "__main__": + unittest.main() diff --git a/.agents/skills/nimbus-worker-scaffold/SKILL.md b/.agents/skills/nimbus-worker-scaffold/SKILL.md index 751290c..5f06b8d 100644 --- a/.agents/skills/nimbus-worker-scaffold/SKILL.md +++ b/.agents/skills/nimbus-worker-scaffold/SKILL.md @@ -1,6 +1,6 @@ --- name: "nimbus-worker-scaffold" -description: "Scaffold a new NimbusImage Docker worker end to end, including entrypoint, Dockerfiles and GPU labels, tests, Compose registration, WORKERNAME.md, and REGISTRY.md. Use when adding, creating, or porting a new annotation, property, image-processing, ML, or test worker. Use nimbus-worker-hardening for existing-worker fixes and nimbus-interface for runtime API details." +description: "Scaffold a new NimbusImage Docker worker end to end, including entrypoint, Dockerfiles and GPU labels, tests, Compose registration, WORKERNAME.md, and REGISTRY.md. Use when adding, creating, or porting a new annotation, property, image-processing, ML, or test worker. Use nimbus-run-worker for live local execution, nimbus-worker-hardening for existing-worker fixes, and nimbus-interface for runtime API details." --- # Scaffolding a NimbusImage Worker @@ -19,6 +19,10 @@ either annotations or property values. access, annotation CRUD, coordinate conventions, local testing), read the `nimbus-interface` skill — don't re-derive that here. +For a real end-to-end run of the built image against a local Girder dataset, +use `nimbus-run-worker`; direct execution has authentication, dataset-view ID, +Docker-network, mutation, and verification requirements beyond a test build. + ## Step 1 — Decide the worker's shape Three questions determine every downstream choice. Settle them first (ask the