feat: Improve inference CLI and update tests

.github/workflows/tests.yml

@@ -28,6 +28,14 @@ jobs:
       - name: Checkout tested commit
         uses: actions/checkout@v4
 
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install host CLI and test dependencies
+        run: pip install -e .
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
@@ -42,12 +50,6 @@ jobs:
           push: false
           tags: agentomics:ci
 
-      # Free GitHub-hosted runners have no GPU, so an empty CUDA_VISIBLE_DEVICES skips GPU tests.
+      # Free GitHub-hosted runners have no GPU.
       - name: Run CPU-compatible test suite
-        run: |
-          docker run --rm \
-            --env AGENT_ID=github-actions-ci \
-            --env CUDA_VISIBLE_DEVICES= \
-            --entrypoint /opt/conda/envs/agentomics-env/bin/python \
-            agentomics:ci \
-            -m test.run_all_tests --workspace-dir /workspace
+        run: python scripts/run_tests.py --image agentomics:ci --cpu-only

.gitignore

@@ -194,6 +194,8 @@ CLAUDE.md
 # AI agent instructions (Codex, Devin, etc.)
 AGENTS.md
 .agents/
+.scratch/
+CONTEXT.md
 
 # Folders
 .genomic_benchmarks

CONTRIBUTING.md

@@ -23,6 +23,44 @@ Agentomics versions use `MAJOR.MINOR.PATCH`, for example `2.4.1`:
 | `chore` | Repository maintenance | None | `chore: update maintainers` |
 | `refactor` | Internal changes without changed behavior | None | `refactor: share argument parsing` |
 
-Add `!` to `feat`, `fix`, or `perf` for Major impact, for example `feat!: remove legacy configuration`. Authors propose the impact; maintainers require or remove `!` after reviewing compatibility.
+Add `!` to `feat`, `fix`, or `perf` for Major impact, for example `feat!: remove legacy configuration`. Authors propose the impact. Maintainers require or remove `!` after reviewing compatibility.
 
 Pull requests are squash-merged using the reviewed PR title as the commit title.
+
+## Testing standard
+
+### What to test
+
+- For each change in your PR, identify what could break and test each such potential regression.
+- If a test already exists for that, update it. Delete tests for behavior that no longer exists.
+- Before fixing a bug, ensure a test exists that fails because of that bug.
+
+### How to test
+
+- Use pytest. All tests must be deterministic.
+- Name tests after the behavior being checked and assert that behavior. Do not rely on internal implementation details.
+- Prefer testing through the **lowest stable interface**: the lowest-level function expected to stay the same when codebase changes. Avoid testing functions like private utilities and unnecessarily high-level functions. For example, test metric calculation through `get_metrics`, not a private _get_auroc or the entire `agentomics-run` workflow.
+- Use production functions to create valid test state when possible. Do not duplicate production code or reimplement their ordering into fixtures or helpers (this can cause drift).
+- Name fixtures after the state they create. E.g. a configuration fixture must not initialize a run.
+- Keep scenario-specific setup local to its test module.
+
+### Test placement
+
+- **Unit** (`tests/unit/`): calls one lowest stable interface.
+- **Integration** (`tests/integration/`): checks multiple lowest stable interfaces working together, directly or through a higher-level production function.
+- **End-to-end** (`tests/end_to_end/`): runs outside the application container, invokes a user-facing CLI through Docker, and checks results. Scripted LLM behavior at the provider boundary is allowed.
+
+### Agent evaluations
+
+Assess real-LLM behavior through **agent evaluations**, not deterministic automated tests. Agent evaluations may be expensive, nondeterministic, and use LLM judges. (To be implemented)
+
+### How to run tests
+
+From the repository root, with Docker running:
+
+```bash
+pip install -e .
+python scripts/run_tests.py --cpu-only
+```
+
+This will use the current repository code (including unstaged and uncommitted changes) to run the tests. Omit `--cpu-only` to require GPU access and include GPU tests.

docs/configuration/cli-options.md

@@ -26,7 +26,6 @@ The run stops when either the iteration count is reached or the timeout expires.
 | Option | Description |
 |--------|-------------|
 | `--cpu-only` | Disable GPU acceleration |
-| `--test` | Run the integrated test suite |
 | `--image <name>` | Docker image to launch (default: `biogemt/agentomics:<installed-package-version>`; use this option for an explicit override) |
 | `--dev` | Build from the current local working tree before launching; cannot be combined with `--image` |
 | `--datasets-dir <path>` | Host directory to read datasets from (default: `./datasets`) |

docs/developer/development.md

@@ -1,14 +1,18 @@
 # Local Development
 
-## How to test changes locally
-
 Install Agentomics in editable mode (one time setup) from the repository root:
 
 ```bash
 pip install -e .
 ```
 
-Use `--dev` to test changes with any Docker-backed command.
+## Running tests
+
+See [How to run tests in CONTRIBUTING.md](https://github.com/BioGeMT/Agentomics-ML/blob/main/CONTRIBUTING.md#how-to-run-tests).
+
+## Running application changes locally
+
+Use `--dev` with any Docker-backed command.
 These must be run from the repository root.
 
 ```bash

docs/user-guide/inference.md

@@ -36,7 +36,8 @@ agentomics-inference \
 | Argument | Description |
 |----------|-------------|
 | `--label-col` | Label column in the input **CSV** — when set, metrics are computed against it and written to `<output>.metrics.json`. Ignored for a split folder; there, metrics run only if the folder has a `labels.csv` |
-| `--iteration-dir` | Iteration directory to use, relative to `--agent-dir` (default: `best_iteration_snapshot`) |
+| `--iteration-dir` | Iteration supplying the inference script and environment, relative to `--agent-dir` (default: `best_iteration_snapshot`) |
+| `--artifacts-dir` | Existing directory of model artifacts to use instead of the selected iteration's original artifacts; cannot be combined with `--all-iterations` |
 | `--all-iterations` | Run inference for every `run/iteration_N` against `--input` (see below) |
 | `--cpu-only` | Run without GPU |
 | `--image` | Docker image to use (default: `biogemt/agentomics:<installed-package-version>`; use this option for an explicit override) |
@@ -45,7 +46,8 @@ agentomics-inference \
 ## Docker Execution
 
 Docker is the default execution mode. The command mounts the agent directory,
-input, and output directory into the configured image automatically:
+input, and any supplied artifact directory read-only. The output directory is
+mounted writable:
 
 ```bash
 agentomics-inference \
@@ -135,6 +137,20 @@ Per-iteration predictions are written to `preds/<iteration>_predictions.csv`
 iteration's environment is rebuilt and removed in turn, and a failing iteration
 warns and continues.
 
+
+## Using Retrained Artifacts
+
+Use `--artifacts-dir` to run inference with [Retrained Artifacts](training.md) instead of the original artifacts.
+
+```bash
+agentomics-inference \
+  --agent-dir outputs/<agent_id> \
+  --artifacts-dir /path/to/output_artifacts \
+  --input /path/to/data.csv \
+  --output /path/to/predictions.csv
+```
+
+
 ## What's in best_iteration_snapshot
 
 ```
@@ -171,7 +187,9 @@ differ.
 
 ### "Model file not found"
 
-Check that `best_iteration_snapshot/` contains the model artifacts. If the agent run failed, there may be no trained model.
+Check the directory supplied with `--artifacts-dir`, or the selected iteration's
+`model_training/training_artifacts` directory when that option is omitted.
+If the agent run failed, there may be no trained model.
 
 ### GPU out of memory
 

envs/environment.yaml

@@ -20,3 +20,4 @@ dependencies:
     - uv==0.9.26
     - openai==2.16.0
     - anthropic==0.79.0
+    - pytest==9.1.1

pyproject.toml

@@ -17,6 +17,8 @@ dependencies = [
     "scikit-learn>=1.5",
     "scipy>=1.11",
     "rich>=14.0",
+    "pytest==9.1.1",
+    "pypdf>=6.0",
 ]
 license = { file = "LICENSE" }
 authors = [
@@ -38,6 +40,9 @@ agentomics-retrain = "agentomics.cli.retrain:main"
 agentomics-check-dataset = "agentomics.cli.check_dataset:main"
 agentomics-add-supplementary = "agentomics.cli.add_supplementary:main"
 
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
 [tool.setuptools]
 package-dir = {"" = "src"}
 packages = [

scripts/run_tests.py

@@ -0,0 +1,71 @@
+from __future__ import annotations
+
+import argparse
+import os
+from pathlib import Path
+import subprocess
+import sys
+from uuid import uuid4
+
+from agentomics.cli.docker_utils import (
+    build_development_image,
+    run_python_in_docker,
+    validate_docker_gpu_access,
+    resolve_proxy_env_vars,
+)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        description="Run unit/integration tests in Docker, then end-to-end tests on the host.",
+        allow_abbrev=False,
+    )
+    parser.add_argument("--image", help="Reuse an already-built Docker image")
+    parser.add_argument("--cpu-only", action="store_true", help="Disable GPU access and GPU tests")
+    arguments = parser.parse_args(argv)
+
+    image = arguments.image or build_development_image()
+    result = subprocess.run(
+        ["docker", "image", "inspect", image], capture_output=True, text=True
+    )
+    if result.returncode:
+        raise RuntimeError(f"Docker image {image!r} is unavailable: {result.stderr.strip()}")
+    if not arguments.cpu_only:
+        validate_docker_gpu_access(image)
+
+    container_name = f"agentomics-tests-{uuid4().hex}"
+    docker_arguments = ["--name", container_name, "--pull", "never"]
+    for name, value in resolve_proxy_env_vars().items():
+        docker_arguments.extend(["-e", f"{name}={value}"])
+    if arguments.cpu_only:
+        docker_arguments.extend(["-e", "CUDA_VISIBLE_DEVICES="])
+    try:
+        container_exit = run_python_in_docker(
+            image=image,
+            cpu_only=arguments.cpu_only,
+            mounts=[],
+            python_arguments=["-m", "pytest", "tests/unit", "tests/integration"],
+            docker_arguments=docker_arguments,
+            check=False,
+        )
+    finally:
+        # --rm does not cover interruption of the Docker client.
+        subprocess.run(
+            ["docker", "rm", "--force", container_name],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+
+    host_exit = subprocess.run(
+        [sys.executable, "-m", "pytest", "tests/end_to_end"],
+        cwd=Path(__file__).resolve().parents[1],
+        env=os.environ | {
+            "AGENTOMICS_TEST_IMAGE": image,
+            "AGENTOMICS_TEST_CPU_ONLY": "1" if arguments.cpu_only else "0",
+        },
+    ).returncode
+    return int(bool(container_exit or host_exit))
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/agentomics/agents/prompt_builder.py

@@ -1,6 +1,5 @@
 from agentomics.datasets.data_contract import SUPPLEMENTARY_DIR_NAME, TRAIN_SPLIT, VALIDATION_SPLIT
 
-from agentomics.runtime.conda_utils import get_shared_environment_path
 from agentomics.runtime.read_write_utils import load_dataset_metadata
 from agentomics.runtime.system_resources import check_gpu_availability, get_resources_summary
 from agentomics.utils.config import Config
@@ -42,7 +41,7 @@ def get_system_prompt(config: Config):
     You are provided with your own already activated environment
     Use this environment to install any packages you need (use non-verbose mode for installations, run conda installations with -y option).
     Don't delete this environment.
-    Your conda environment lives under {get_shared_environment_path(config)}.
+    Your conda environment lives under {config.shared_environment_path}.
     Write all your python scripts in files.
     Run all commands in a way that prints the least amount of tokens into the console.
     Always call tools with the right arguments, specifying each argument as separate key-value pair. 

src/agentomics/agents/steps/base.py

@@ -82,7 +82,7 @@ def on_step_start(self) -> None:
     def on_step_success(self, output: Any) -> None:
         save_step_output(self.config, self.step_id, output)
         export_shared_environment_descriptor(self.config)
-        self._archive_step_folder()
+        self.archive_step_folder()
         commit_step_checkpoint(self.config, iteration=load_current_iteration_index(self.config), step_id=self.step_id)
 
     def on_iteration_start(self, iteration: int) -> None:
@@ -94,7 +94,7 @@ def on_iteration_fail(self, iteration: int) -> None:
     def on_iteration_end(self, iteration: int) -> None:
         return None
 
-    def _archive_step_folder(self) -> None:
+    def archive_step_folder(self) -> None:
         step_dir = self.config.current_step_dir
         archived_dir = self.config.archived_step_dir(self.step_id)
         rewrite_symlinks_to_absolute(step_dir)

src/agentomics/agents/steps/data_split.py

@@ -57,10 +57,35 @@ class DataSplitStep(AgenticStep):
     step_id = "data_split"
     display_name = "SPLITTING"
     output_type = DataSplitOutput
-
     CLASSIFICATION_MINI_TRAIN_SAMPLES_PER_CLASS = 100
     REGRESSION_MINI_TRAIN_SAMPLE_COUNT = 100
 
+    @staticmethod
+    def validate_generated_split(split_path: Path, dataset_dir: Path) -> None:
+        try:
+            validate_symlinks_targets_in(split_path, dataset_dir)
+        except ValueError as e:
+            raise ModelRetry(str(e))
+
+        dataset_train_input = dataset_dir / TRAIN_SPLIT / INPUT_DIR_NAME
+        split_input = split_path / INPUT_DIR_NAME
+        copied_files = []
+        for path in split_input.rglob("*"):
+            if path.is_symlink() or path.is_dir():
+                continue
+            # Assumes split mirrors the original structure.
+            # If the agent reorganized files, original.is_file() is False and we skip — no false positives but could have some false negatives.
+            original = dataset_train_input / path.relative_to(split_input)
+            if original.is_file() and path.stat().st_size == original.stat().st_size:
+                copied_files.append(path)
+        if copied_files:
+            names = [str(c.relative_to(split_path)) for c in copied_files[:10]]
+            raise ModelRetry(
+                f"Files in {split_path.name}/input/ that exist unchanged in the source dataset "
+                f"must be symbolic links, not copies. Up to first 10: {names}"
+            )
+
+
     def _get_latest_split_strategy(self) -> str:
         iteration = get_last_successful_iteration(self.config)
         assert iteration is not None
@@ -111,25 +136,6 @@ def _validate_mini_train_regression(self, mini_train_df: pd.DataFrame, train_df:
                 f"got {len(mini_train_df)}."
             )
 
-    def _raise_on_unnecessary_copies(self, split_path: Path) -> None:
-        dataset_train_input = self.config.dataset_dir / TRAIN_SPLIT / INPUT_DIR_NAME
-        split_input = split_path / INPUT_DIR_NAME
-        copied_files = []
-        for path in split_input.rglob("*"):
-            if path.is_symlink() or path.is_dir():
-                continue
-            # Assumes split mirrors the original structure.
-            # If the agent reorganized files, original.is_file() is False and we skip — no false positives but could have some false negatives.
-            original = dataset_train_input / path.relative_to(split_input)
-            if original.is_file() and path.stat().st_size == original.stat().st_size:
-                copied_files.append(path)
-        if copied_files:
-            names = [str(c.relative_to(split_path)) for c in copied_files[:10]]
-            raise ModelRetry(
-                f"Files in {split_path.name}/input/ that exist unchanged in the source dataset "
-                f"must be symbolic links, not copies. Up to first 10: {names}"
-            )
-
     def _move_split_to_versioned_dir(self, result: DataSplitOutput, flag_as_changed: bool) -> DataSplitOutput:
         train_path = Path(result.train_path)
         val_path = Path(result.val_path)
@@ -209,11 +215,7 @@ async def validate_split_dataset(ctx: RunContext[dict], result: AgenticStepOutpu
             if not train_path.is_relative_to(self.config.splits_dir) or not val_path.is_relative_to(self.config.splits_dir) or not mini_train_path.is_relative_to(self.config.splits_dir):
                 split_paths_to_validate = [mini_train_path] if is_mini_train_only else [train_path, val_path, mini_train_path]
                 for split_path in split_paths_to_validate:
-                    try:
-                        validate_symlinks_targets_in(split_path, self.config.dataset_dir)
-                    except ValueError as e:
-                        raise ModelRetry(str(e))
-                    self._raise_on_unnecessary_copies(split_path)
+                    self.validate_generated_split(split_path, self.config.dataset_dir)
                     rewrite_symlinks_to_absolute(split_path)
                 result = self._move_split_to_versioned_dir(result, flag_as_changed=not is_mini_train_only)
             else:

src/agentomics/agents/steps/model_inference.py

@@ -20,7 +20,6 @@
 from agentomics.agents.steps.base import AgenticStep, AgenticStepOutput
 from agentomics.agents.steps.data_split import DataSplitStep
 from agentomics.agents.steps.model_training import ModelTrainingStep
-from agentomics.runtime.conda_utils import get_shared_environment_path
 from agentomics.runtime.filesystem import remove_path
 from agentomics.runtime.inference_runner import compute_metrics, run_inference_on_split
 from agentomics.runtime.read_write_utils import (
@@ -141,7 +140,7 @@ def _validate_dry_run(self) -> None:
         inference_result = run_inference_on_split(
             split_path=split_path,
             output_path=output_path,
-            conda_env_path=get_shared_environment_path(self.config),
+            conda_env_path=self.config.shared_environment_path,
             inference_script_path=self.config.current_step_dir / "inference.py",
             training_artifacts_dir=Path(model_training.path_to_artifacts_dir),
         )