Skip to content

DeveloperGuide.md

Latest commit

 

History

History
200 lines (159 loc) · 9.87 KB

DeveloperGuide.md

File metadata and controls

200 lines (159 loc) · 9.87 KB
Raw

Developer Guide: Creating a New Workflow

A session workflow lives in one directory, workflows/my-session/, and needs three things:

File Runs on Purpose
app/controller.sh Controller (login) node Install software, download dependencies
app/start-template.sh Controller or compute node Start the web service
yamls/<variant>.yaml (e.g. yamls/general.yaml) Platform Define the UI form, generate inputs.sh, orchestrate

app/ holds everything the run needs (scripts + support files) and is the only subtree the workflow sparse-checkouts; yamls/, thumbnails/, README, and build tooling stay outside it.

The controller node always has internet access. The compute node may not.

The platform docs are the source of truth — consult them when anything here is unclear or behaves differently: building workflows (YAML fields · inputs & expressions · actions) · endpoint sessions · pw endpoints CLI.

Working with an AI assistant? The repo ships a Claude Code skill — .claude/skills/activate-workflows/ — that encodes this process plus the platform reference. In Claude Code: "Using the activate-workflows skill, create a new interactive session workflow for [deployment] that [does X]."

1. The controller script

workflows/my-session/app/controller.sh runs before the service starts, on the login node. Use it for anything needing internet. All inputs.sh variables are available.

#!/usr/bin/env bash
set -o pipefail

if [ -z ${service_parent_install_dir} ]; then
    service_parent_install_dir=${HOME}/pw/software
fi

if ! [ -f "${service_parent_install_dir}/my-server" ]; then
    echo "Installing my-server..."
    mkdir -p ${service_parent_install_dir}
    wget https://example.com/my-server.tar.gz -O /tmp/my-server.tar.gz
    tar -xzf /tmp/my-server.tar.gz -C ${service_parent_install_dir}
fi

Keep it idempotent — check whether software exists before installing.

2. The start script

workflows/my-session/app/start-template.sh starts the web service. The platform provides service_port — your service must listen on it. All inputs.sh variables are available.

#!/bin/bash
if [ -z ${service_parent_install_dir} ]; then
    service_parent_install_dir=${HOME}/pw/software
fi

# cancel.sh lets the platform stop the service
echo '#!/bin/bash' > cancel.sh
chmod +x cancel.sh

${service_parent_install_dir}/my-server --port=${service_port} &
pid=$!
echo "kill ${pid}" >> cancel.sh

sleep inf

Requirements: listen on service_port, write a cancel.sh, end with sleep inf (or run the service in the foreground).

3. The workflow YAML

workflows/my-session/yamls/general.yaml. Its jobs:

  1. preprocessing — parallelworks/checkout of this repo (sparse: workflows/my-session/app, plus tools/... if the scripts use the shared tools), generate inputs.sh from the form values + PW_* environment, run inputs.sh + controller.sh inline, and assemble the start script (inputs.sh + a cleanup trap + start-template.sh).
  2. session_runner (the job name kept for history) — submit the start script via workflows/script_submitter/v3.6/<variant>.yaml (uses: github/parallelworks/workflows@canary).
  3. wait_for_endpoint — calls the wait_for_endpoint subworkflow (uses: github/parallelworks/workflows@canary, $yaml: workflows/wait_for_endpoint/general.yaml), then cancels the submitter. The subworkflow polls pw endpoints list for <service.name>-${PW_RUN_SLUG}, probes the URL with the run's key until the status matches healthy (within budget), and touches the SKIP_CLEANUP file so the service outlives the run. If the service never answers, the job fails without the skip file and the submitter's cleanup tears the job down. The workflow, not the test runner, owns this check. Set healthy (a web UI answers 2xx/3xx), path (an API server's health route) and budget (minutes for a container or a model load) per service and verify them in a test. Inputs: workflows/wait_for_endpoint/README.md.

On a Kubernetes cluster the same endpoint is registered by a pw-cli sidecar in the pod, the run stays alive streaming pod logs, and cancelling the run is the teardown; see .claude/skills/activate-workflows/references/k8s-workflows.md.

Copy a real one instead of writing from scratch — workflows/webshell/yamls/general.yaml is the smallest complete example; workflows/jupyterlab/yamls/general.yaml shows a conda install plus support files; workflows/streamlit/yamls/general.yaml + its scripts show a Singularity/SIF service (SIF pulled via oras, .def + build-container.sh alongside). Key parts to adapt:

  • the hidden service.name input (endpoint name prefix),
  • the healthy, budget and path inputs of the wait_for_endpoint call,
  • the sparse-checkout paths (workflows/my-session/app, tools/...),
  • the cat workflows/my-session/app/controller.sh / start-template.sh lines,
  • the service input group (your form fields → inputs.sh variables).

4. Platform variants

One YAML per deployment: yamls/general.yaml (standard SLURM/PBS clusters), plus emed.yaml / hsp.yaml / noaa.yaml where the workflow is offered there. Variants differ in scheduler directives, partitions, module loads, and defaults — copy the matching variant of a similar workflow (they pass their variant's workflows/script_submitter/v3.6/<variant>.yaml).

5. Testing

Every workflow is tested end-to-end at least once, and any end-to-end test is recorded under workflows/my-session/tests/<variant>/. The test layout, the test file keys, the pass criteria and the result columns are documented once, in tools/tests/README.md. Start from an existing test, e.g. workflows/webshell/tests/<variant>/<test-name>.json.

Push first — the YAML pulls this repo from GitHub at run time, so local edits to app/ are invisible until they are on the referenced branch. Then:

python3 tools/tests/run-workflow-test.py workflows/my-session/tests/<variant>/<test-name>.json

Commit the test and the rows the runner appends to its CSV with your change; never edit a CSV by hand. The runner does not probe the endpoint URL: a run that completes has passed the workflow's own health probe, and the wait job's output (pw workflows runs logs <slug> --job wait_for_endpoint) shows the status codes it saw.

Verify cleanup on cancel — part of testing, every time. Cancel a run mid-flight (pw workflows runs cancel <slug> while the service is starting or serving) and confirm the cleanup actually ran: no service processes left (ps -x), no scheduler job (squeue/qstat when scheduler:true), no container instances (singularity instance list, docker ps), no stray listeners. Apps that daemonize and re-parent to PID 1 (e.g. RStudio's rsession) can survive the tree kill and need handling in cancel.sh. Write cancel.sh at the very top of the start script so a cancel at any moment finds it.

While iterating, point the YAML's checkout branch: at a development branch and restore it to canary before the PR merges (canary only accepts pull requests).

6. Debugging

Everything a run did is in its job dir on the execution node: ~/pw/jobs/<run-slug>/ for CLI file runs, or ~/pw/jobs/<workflow-name>/<run-number, 5 digits>/ for registered workflows.

  • run.<JOBID>.out — the service's stdout/stderr
  • logs/<job>/step_N/step.out, step.exit — per-step trace and exit code
  • logs/<job>/step_N/script-unstable.sh — the rendered step: every ${{ input }} appears as the literal value the form sent. When a value seems ignored (a default not applied, an empty field), read this first.
  • From any machine: pw workflows runs errors <slug> and pw workflows runs logs <slug>.

Common pitfalls

  • Testing unpushed code — the checkout fetches GitHub, not your working tree.
  • Relative YAML path in pw workflows run — parsed as a git host; use absolute.
  • Composing checkout paths wrong — checked-out files materialize at ${PW_PARENT_JOB_DIR}/workflows/<name>/app/… (or …/<impl>/…), including paths built from variables ("${PW_PARENT_JOB_DIR}/${service_name}"-style bugs surface only at run time).
  • Globs copy everything in the directory they target. Some preprocessing steps collect support files with a glob — e.g. jupyter runs cp workflows/jupyter/app/*.yaml . to gather its conda-env files. That only works because app/ contains nothing but runtime files: the workflow's own variant YAMLs live in yamls/, outside app/. Keep it that way — a workflow YAML (or any unrelated file) placed inside app/ would be swept into every run.
  • Single-attempt ghcr pulls — ghcr intermittently rate-limits anonymous pulls; use tools/oras/libs.sh:oras_pull_file (it retries) and keep packages public.
  • A registered workflow ignoring your defaults — the registration pins one YAML path (pw workflows get <name> → remote.yaml); if it points at the wrong variant, the form (and its defaults) are the wrong variant's.
  • "Authentication has expired" — pw tokens lapse; re-run pw auth.

Appendix: converting an older workflow

To bring an older workflow onto the endpoint pattern, follow .claude/skills/activate-workflows/references/session-to-endpoint-upgrade.md; which workflows those are and where they live is in MIGRATION.md.