[nicolasbisurgi]

Closes #124.

Summary

Each run phase can now have an optional time limit. A phase that goes past its limit is a hang: RushTI writes a TimedOut result and exits, so the TI process that called it stops waiting. Limits are off by default.

 main()
+  watchdog.phase("task_retrieval")          # --tm1-instance only
     connect_to_tm1_instance + read_taskfile_from_tm1
+  watchdog.start("connection")
     setup_tm1_services
     wait_for_exclusive_access
+      sleep=watchdog.paused_sleep            # sleeps between polls don't count; polls do
     build DAG, validate_tasks
     work_through_tasks_dag
+      on_first_task_start → watchdog.start("task_execution")
     logout(task instances)                   # still inside task_execution
+  watchdog.stop()
+  watchdog.phase("push_results")            # push_results = true only
     connect → upload → }rushti.load.results → logout

On a hang, the watchdog thread (HangHandler in the new rushti/timeouts.py):

log which phase limit was hit
write result file: Status = TimedOut, Overall Success = False, tasks finished so far
save the checkpoint (no cleanup) · flush execution logs · stats to local SQLite only
log out of every session in daemon threads, wait ≤ 5s, warn on any that didn't finish
logging.shutdown(); os._exit(1)

Once a hang is declared, the scheduler starts no new task and stops early session release. While the hang is being handled, watchdog.start/stop/paused block until the handler is done, so the main thread can't run its own end-of-run cleanup at the same time. CheckpointManager takes a lock, so the watchdog thread's save can't collide with the scheduler recording results.

Settings follow the max_workers precedence: --timeout-<phase> > taskfile settings.timeouts > settings.ini [timeouts] > no limit. rushti resume forwards the flags.

The --result file gets a trailing Status column on every outcome: Success, Failed, Aborted (including the exclusive-mode wait running out) or TimedOut.

Follow-ups from review: #192 (result file on every early exit), #193 and #194 (cleanups).

Also included:

• A failed }rushti.load.results no longer overwrites the run's own success flag.
• The docs had the exclusive-mode timeout as exit code 5; it's 1. Corrected.
• Per-task cancel_at_timeout doesn't cancel on TM1 v12, because TM1py blocks its own thread calls there. The docs now say so wherever it's documented.
• New docs page: docs/features/timeouts.md.

Evidence

• Before: a run stuck inside a TM1 call (e.g. a login blocked by a lock) never exits, and the calling TI waits forever.
  After: a real process with setup_tm1_services blocking and --timeout-connection 2 exits with code 1 after about 2s:

  ERROR - Run phase 'connection' went past its time limit of 2s. RushTI is terminating and logging out of TM1.
  PID|Process Runs|Process Fails|Start|End|Runtime|Overall Success|Status
  47468|0|0|…|…|0:00:02.004630|False|TimedOut
  

• Unit tests: 884 passed, up from 798. They cover each of the four limits firing (exit code 1, the phase named in the log, TimedOut, exit within limit + 5s); task_retrieval ignored for local taskfiles; the exclusive-mode wait not counted; the checkpoint kept and SQLite stats written; preserved connections handled; a logout that never returns not holding up the exit; Status for each outcome; precedence and validation in all three tiers; resume forwarding; a hang already being handled blocks stop()/start(); the checkpoint saved from another thread while results are recorded (this reproduced dictionary changed size during iteration before the lock); a final logout that blocks counting against task_execution; stop() racing the timer never returning during a hang; no new task starting after a hang; an exclusive-mode session poll that blocks counting against connection.
• Integration tests on tm1srv01 (v11.8) and tm1srv02 (v12.6): 104 passed, 1 failed. The failure, test_v11_chore_kind.py::test_mixed_dag_executes_in_order, fails the same way on master: }bedrock.server.wait with pWaitSec=0 aborts on tm1srv01.
• Checks on real servers (v11.8 and v12.6):
  • A TI reading the result file with the old 7 variables still runs with the new column, and an 8-variable TI reads Status = TimedOut. Passes on both; a negative control fails as it should.
  • A real rushti run with a TI that writes a cell and then sleeps, and --timeout-task-execution 8: exit code 1 after about 8–10s, TimedOut written, checkpoint kept, no RushTI session left behind.
  • The logout does not roll back the running TI. It runs to completion and commits, on both versions. This is accepted: the checkpoint shows the task as in progress, and the docs tell the developer to check how it really ended on the server before resuming.

Merge Danger

Door: one-way

The [timeouts] / settings.timeouts / --timeout-* settings and the Status column are new public surface. Once released they're hard to remove.

Blast Radius: result-file readers

• A TI that reads the --result file now sees an extra trailing column. TM1 ignores it with a fixed set of variables (checked on v11 and v12). A reader that parses the line some other way (e.g. counting delimiters) could break.
• With no limits set, the only behaviour change is the new column. setup_tm1_services gained two optional parameters, and work_through_tasks_dag gained an optional on_first_task_start callback; the defaults keep existing callers working as before.

🤖 Generated with Claude Code