Skip to content

Log [om-metrics] resource usage from the docker entrypoint - #88

Closed
aethr wants to merge 2 commits into
mainfrom
om-metrics-entrypoint
Closed

aethr wants to merge 2 commits into
mainfrom
om-metrics-entrypoint

Conversation

@aethr

@aethr aethr commented Oct 8, 2026

Copy link
Copy Markdown

Closes #86.

wrf-run asks for 63 vCPU and 32000 MiB, and neither has been measured. This adds the same [om-metrics] start/finish lines the openmethane image emits (openmethane/openmethane#235), so one aust10km daily run is enough to size it:

just logs <log-stream-name> | grep om-metrics

Changes

  • scripts/docker-entrypoint-debug.sh: a port of the openmethane wrapper, with the same fields and format. Unlike openmethane, it runs the command directly and passes on its real exit code, because setup-wrf has no entrypoint of its own.
  • Dockerfile: installs tini and sets the wrapper as the ENTRYPOINT. Batch's command replaces only CMD, so wrf-run and wrf-download_geog are measured with no om-infra change.
  • Interactive docker run -it sessions run the command directly and are not measured. A backgrounded command would otherwise get /dev/null as stdin.
  • After forwarding a signal, the wrapper keeps waiting until the command has exited. Without this, a stopped job would end the container while run-wrf.sh was still copying partial output from /mnt/scratch to EFS.
  • Tests ported from openmethane, plus checks for exit-code passthrough and the signal handling.
  • docs/development.md: a short section on the metrics.

Scratch cleanup is deliberately left out of the entrypoint. run-wrf.sh already syncs and then removes its RUN_DIR on exit. A blanket rm -rf $CHK_PATH/* would risk wiping other jobs' data, since run-wrf.sh defaults CHK_PATH to the shared /mnt/scratch.

Reading the results for wrf-run

  • The figures cover the whole container: WPS, the single-process real.exe and wrf.exe. mean_parallelism will therefore sit below the rank count. The time output around mpirun in run.sh covers wrf.exe on its own.
  • safe_mpi_tasks() sizes ranks from os.cpu_count(). Batch uses CPU shares rather than a quota, so this is probably the host's 64 vCPUs, not the 63 requested. The running with N mpi ranks log line will confirm it.
  • Ranks are capped by the host, not the domain, so a larger domain needs no more vCPU. Memory scales roughly with cell count, on top of a fixed overhead per rank. A second run on aust25km would separate the two: it also gets 64 ranks, with about 1/6 of aust10km's cells.

Testing

  • make test: 38 passed.
  • Built the image locally and ran it under --cpus/--memory limits. The start and finish lines parse, and the exit code passes through.
  • docker stop on a container whose command has an exit trap: the trap now completes before the container exits. Before the fix it was cut off.

🤖 Generated with Claude Code

aethr and others added 2 commits October 8, 2026 17:18
Wraps every command the image runs to log the same start and finish
metrics lines as the openmethane image, so wrf-run and wrf-download_geog
can be sized from measured CPU and memory rather than guessed.

The wrapper keeps waiting for the command after forwarding a signal, so
run-wrf.sh can still copy partial output off the scratch disk when a job
is stopped.

Closes #86

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aethr

aethr commented Oct 9, 2026

Copy link
Copy Markdown
Author

Closing, as the perf testing this was made for is concluded. Results are in #89 .

@aethr aethr closed this Oct 9, 2026
@aethr
aethr deleted the om-metrics-entrypoint branch October 12, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add [om-metrics] instrumentation to the entrypoint so wrf-run can be right-sized

1 participant