Skip to content

Journal every backtest run: RunID directory with run.json and events.jsonl (part of #702) - #716

Merged
rustyeddy merged 2 commits into
mainfrom
feat/journal-in-process-run
Nov 21, 2026
Merged

rustyeddy merged 2 commits into
mainfrom
feat/journal-in-process-run

Conversation

@rustyeddy

Copy link
Copy Markdown
Owner

Part of #702: PR 1, the journal and the in-process path, as approved on the issue. #702 stays open for the remote EMAX run, examples/emax and the parity test (PR 2).

What

Every backtest run now leaves a directory under a results root, named by its RunID (the ported internal/id, #714), holding two files:

  • run.json: RunID, kind, strategy (emax, in-process), source, interval, instruments, range, wall-clock started_at / ended_at, status (running from the start, then completed / failed / cancelled), error, and for a completed run the engine's result.
  • events.jsonl: one JSON line per order transition, in the order it happened.

Example (scenario b2-stop-cross, an open, its broker stop, the stop's fill):

{"seq":1,"time":"2010-01-07T00:00:00Z","order":1,"state":"requested","instrument":"EURUSD","action":"open","direction":"long","type":"market","cause":"strategy","price":"1.103","stop_loss":"1.09197"}
{"seq":2,"time":"2010-01-07T00:00:00Z","order":1,"state":"sized","units":"9066"}
{"seq":3,"time":"2010-01-07T00:00:00Z","order":1,"id":"sim-1","state":"submitted","units":"9066"}
{"seq":4,"time":"2010-01-08T00:00:00Z","order":1,"id":"sim-1","state":"filled","instrument":"EURUSD","action":"open","direction":"long","cause":"strategy","units":"9066","fill_price":"1.10323"}
{"seq":5,"time":"2010-01-08T00:00:00Z","order":2,"id":"sim-2","state":"requested","instrument":"EURUSD","action":"close","direction":"long","type":"stop","cause":"stop loss","price":"1.09197"}
{"seq":7,"time":"2010-01-11T00:00:00Z","order":2,"id":"sim-2","state":"filled","instrument":"EURUSD","action":"close","direction":"long","cause":"stop loss","units":"9066","fill_price":"1.09197"}

Integration point (the one place transitions are written)

order.record is the single writer of every order transition, so it also tells the run's journal (journal.go, unexported), through one new field order.journal. A nil journal is a no-op, so engine tests and the EMAX md5 are unchanged. No polling, no second record of the run: it copies what the engine writes, when it writes it.

Owner points on the plan, each tested:

  • Write failures do not change the trade. record returns no error and its signature is unchanged. The journal keeps its first write error and stops writing; trader.onFrame checks it after each frame and RunSeriesWith after the last, and the run ends with journal: .... Test: a writer that fails after 3 writes ends the run with that error and nothing more is written.
  • The first requested line is never lost. The journal pointer is set before the first record for strategy orders (handleOrder) and for every broker-made order: protective stop and take profit (placeExit, which now takes the trail so the stop's requested line has it, and which an amend uses too) and margin closeouts (closePosition). Test: every order's first line is requested, including the broker's exits, and a target added by an amend.
  • Every run is journaled. service.RunBacktest always creates and finalizes the run, and ResultsRoot is required (the cmd resolves the default). Which failures come before a run exists: only an unsupported source, a missing or unusable results root, or a directory that cannot be made, since there is then nowhere to record it. Everything after is a run with a status: bars that cannot be loaded, a request the engine refuses (for example an interval it does not support), a cancellation. Tests for failed, cancelled, refused-by-engine, and for the three pre-run refusals (which create nothing).
  • run.json is minimal. The result is the engine's own summary (Result.Summary(): the numbers Write already prints, exact, read from the result, nothing calculated again; money and prices as decimal strings), and the orders are events.jsonl. No parallel order model.
  • RunID: run_<ULID> (Port RunID from the archive; add internal/clock (part of #702) #714). The directory is made with Mkdir, so a clash is an error, never an overwrite.

Layering

The engine still opens no file: RunSeriesWith(ctx, RunOptions{Journal: io.Writer}, ...). service may not import os (a layering test), so the files are opened by a new foundational package, internal/results (Start, Journal, Finish; run.json written to a temp file, synced and renamed, so a reader sees the old record or the new, never half; a journal that cannot be flushed turns a would-be completed run into a failed one). service.RunBacktest returns *BacktestRun (the engine's *Result embedded, so Write is unchanged, plus RunID and Dir); the error of a failed run says where the run is. cmd/trader stays transport only.

Results root

--results-root, then TRADER_RESULTS_ROOT, then results.root in etc/trader.yml, then /srv/trading/backtest/results: the same precedence as the data root. The command ends its output with run <RunID>: <dir>. Every test uses temporary directories and never touches /srv/trading.

Tests

  • journal_test.go (engine): every transition in order with contiguous seq; the open's four states with units, fill price and id where they belong; a protective stop's requested and fill lines (cause: stop loss); an order requested on the last frame is in the journal although no OnBars could ever tell the strategy; a sizing refusal is requested then rejected with no id; a target added by an amend; same run, same lines; journaling changes nothing the run trades, for all six scenarios (summary and Summary() equal with and without); a failing writer ends the run with its error.
  • internal/results: directory and files, running record, every refusal (no root, root that is a file, an existing run directory is left untouched), finish as completed / failed (partial journal kept) / cancelled, a flush failure, no temp file left.
  • service: reads the run from the files only, with no trader object: an opening request, sized, submitted, filled, a closing fill, status completed, result agreeing with the journal, RunID in the name and in run.json, its time near started_at; a failed load, an engine-refused request and a cancellation are journaled with their status and error; nothing runs without somewhere to record it; two runs get distinct sorted directories with the same events.
  • cmd/trader: the summary ends with the run's directory; root resolved flag, then environment, then file; a failed run's error names its directory; results.root parses.
  • make check passes; the EMAX md5 is unchanged (195e457a…).

Not in this PR (PR 2 and later)

The remote EMAX run (RunSeriesWith with a worker address, --worker), examples/emax and the parity test against the internal baseline, REST (#700), data provenance and build digests, crash-recovery beyond temp-file-and-rename, retention.

🤖 Generated with Claude Code

…jsonl (part of #702)

order.record tells a run-scoped journal of every order transition, the broker's
own exits included, so the journal holds the whole sequence in the order it
happened, the final frame's transitions too. internal/results makes the run
directory (named by the RunID) and finalizes run.json by temp file and rename;
service.RunBacktest journals every run, failed and cancelled ones too, under a
results root given by --results-root, TRADER_RESULTS_ROOT, results.root or
/srv/trading/backtest/results. A journal that cannot write ends the run with its
error without changing what is traded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 21:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copy link
Copy Markdown
Owner Author

Review of PR #716 against #702. Good architecture overall: order.record is reused instead of polling order histories; service owns run lifecycle; internal/results is a reasonable narrow filesystem boundary given the existing no-os service rule. Strong tests for protective exits, last-frame orders, stable execution results, and disk-only reading. I do have the following actionable findings.

Must fix before merge

  1. internal/results/run.go:Finish silently loses flush/close errors if the run already failed or was cancelled. Both checks only assign their error when runErr == nil. That means a disconnect + disk-full combination reports just the disconnect, with no indication that events.jsonl may be incomplete. Always attempt flush and close, and join their errors with the original run error (preserving errors.Is(..., context.Canceled) for status classification, perhaps classify the original error before joining). Test a failed/cancelled run whose journal also fails during flush.

  2. Check short writes on the journal boundary. In journal.go, j.w.Write can legally return n < len(line+newline), err == nil. This must count as io.ErrShortWrite, otherwise a truncated JSONL record can be treated as a successful run. bufio.Writer helps in the normal file-backed path, but RunSeriesWith accepts any io.Writer. Add a small targeted test for a writer that returns a short count with nil error.

Clarify scope (no extra framework)

  1. What does “every run” mean for the exported RunSeries? The current wrapper calls RunSeriesWith(..., RunOptions{}, ...) and produces no journal, while service.RunBacktest always journals. If direct engine invocations are intentionally a test/library primitive, document that exception explicitly; if they are supported production run entry points, Trader: journal every run with durable RunID results (first increment) #702's invariant should cover them too. We don't need another run manager to answer this.

Simplification / maintainability

  1. Result.Summary() adds 3 exported structs (Summary, OrderCount, Position) and re-expresses values already printed by Write. This is understandable for JSON, but please check whether an existing structured result representation can be reused (especially the internal result fields). If no existing serialization shape fits, keeping these small structs is acceptable; avoid further parallel result models or calculations.
  2. RunID creation and directories: the configured default/root and no-overwrite semantics look right. Keep the failure cleanup simple; no new recovery abstractions for this milestone.

Verdict: request the two concrete error-path fixes, document the RunSeries scope decision, then I expect this can merge as the in-process journal increment. Remote EMAX/parity remains the next PR; do not close #702.

…failure (part of #702)

Finish always flushes and closes and adds a journal failure to the run's own error
instead of dropping it, classifying the status from the run's error first. A
journal writer that writes less than a line and reports no error is a failed
journal (io.ErrShortWrite). RunOptions documents which entry points journal.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@rustyeddy

Copy link
Copy Markdown
Owner Author

Addressed in 8a69cff (not merged). make check passes; EMAX md5 unchanged.

  1. Flush/close errors are no longer lost. Finish always flushes and closes the journal. If that fails and the run had already failed or been cancelled, the journal error is added to the run's own (<run error> (and the journal is incomplete: journal: ...)), the status is classified from the run's own error first (so a cancelled run stays cancelled, and errors.Is still finds context.Canceled), and run.json says events.jsonl may be incomplete. A run that would have completed becomes failed and reports no result. Tests: failed + flush failure, cancelled + flush failure, completed + flush failure.
  2. Short writes. journal.record treats n < len(line) with a nil error as io.ErrShortWrite. Test: a writer that drops the last byte and reports no error ends the run with journal: short write and no result.
  3. Scope of RunSeries: documented on RunOptions and in the Developers Guide. Journaling is the application's rule: every run started through service.RunBacktest (what trader backtest calls) is journaled. RunSeries/RunSeriesWith are the engine's library entry points and write a journal only when handed a writer (tests, or any caller that does not want a run directory). The no-argument scenario regression (Run, the six EMAX scenarios) is the baseline the journaled runs are compared with, not a run with a directory. No run manager added.
  4. Summary: I looked for an existing structured result. result is unexported, and Write only renders text (money floored to cents for display), so nothing exists to serialize the exact values. The three small structs (Summary, OrderCount, Position) are plain copies of fields result already holds, with no calculation; I kept them and will add nothing parallel (the remote parity test will compare these same values).
  5. Cleanup: unchanged and simple (Start leaves nothing behind on a refusal; a clash is an error).

Copy link
Copy Markdown
Owner Author

Re-review at 8a69cff: ready to merge from my review. The two requested error-path fixes are present in code and covered by tests: Finish preserves flush/close errors alongside pre-existing failure/cancellation, and journal.record detects short writes (io.ErrShortWrite). The documentation now explicitly distinguishes journaled application runs through service.RunBacktest from direct engine/library RunSeries calls. I accept that boundary for this increment, and the small Summary data types are justified because the existing Write method only emits formatted text. No new framework or duplicate authoritative state was added. Claude reports make check passing and unchanged EMAX baseline; I verified the changed patches and tests but have not independently run the suite. Keep #702 open for remote EMAX and parity.

@rustyeddy
rustyeddy merged commit e575880 into main Nov 21, 2026
1 check passed
@rustyeddy
rustyeddy deleted the feat/journal-in-process-run branch November 21, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants