How the code is organized, how the layers fit and how to change them. The
decisions behind it are the ADRs (registry); the rules
every contributor follows are in AGENTS.md and
CONTRIBUTING.org. The guide for the earlier, larger runtime is
kept in _archive/docs/DevelopersGuide.md;
it does not describe this tree.
cmd/trader/ transport: commands, flags, environment, config file
service/ application services (marketdata.go)
marketdata/ public model: Bar, OHLC, MidOf, SourcePartition
service/ domain service: Build (classify, availability, publish)
internal/canonical/ canonical-v2 files, availability rule (private)
provider/oanda/ OANDA: raw store, REST client, raw-to-source mapping
internal/store/ the only data-file I/O (internal: not importable from outside the module); storetest for tests
internal/marketdata/layout/ the one data path resolver
internal/logging/ structured logging on log/slog
num/ instrument/ exact numbers; instrument identity
(root package trader) engine: orders, broker, account, strategy, scenarios
_archive/ earlier code and docs, a donor; never imported
cmd/trader -> service -> marketdata/service -> marketdata/internal/canonical -> store
|
+------> provider/oanda -> store
- Transport (
cmd/trader) parses flags, reads the environment and the config file, sets the store's root and calls an application service. It imports no provider: for a download it handsservice.Downloadthe base URL, the request gap and aservice.Credentialthat reads the token from the environment on each request, and the service builds the client (a test enforces the import rule). It contains no domain logic and is the only place that reads the environment (ADR-078). - Application service (
service) carries out one use case by coordinating a provider and a domain service. It reads no environment. Today there is one,BuildCanonical; a package per service waits until the services are not small. - Domain service (
marketdata/service) owns the domain's rules. Domain services never call each other.marketdata/internalmay be imported only frommarketdata/service(Go'sinternalrule, plus an architecture test). - Provider (
provider/oanda) is one external system: its HTTP client, its raw file format, its symbols. It parses and maps; it applies no domain rule. It imports only public models,store,layout,numandinstrument. - Store (
store) reads and writes bytes at paths relative to its root:Read,WriteandCreateandUpdate(atomic, under a per-path lock), andList. Nothing else opens, renames or walks data files; a test enforces it. - Public models (
marketdata,num,instrument) import no service, transport or provider.
Each boundary has an architecture test that fails when it is crossed; when you add a boundary, add its test and prove it fails (see AGENTS.md).
download: OANDA REST --provider/oanda--> raw partition (<root>/oanda/raw/...)
build: raw partition --ReadSourcePartition--> marketdata.SourcePartition
--marketdata/service.Build--> canonical file (<root>/oanda/canonical/...)
- A raw partition is the provider's own format (raw-v1). It stores only complete candles. A complete row is never changed; a forming row stored by an older download is replaced by the closed candle when the provider reports it (a forming candle is not a fact), and nothing else is rewritten.
SourcePartitioncarries what a bare[]Barwould lose: the raw row numbers, the provider's complete flag and the fingerprint of the raw bytes. Its bars are unvalidated: the provider sets no mid and noAvailableAt.Buildclassifies each row (incomplete, misaligned, invalid, crossed, duplicate, out of order), derives the mid and the availability time, keeps every rejected row as a bad record, and publishes.canonical.Publishwrites nothing when the same build (raw fingerprint, builder, validator, availability policy) is already there.- Reading is the other direction:
marketdata/service.Loadreads the canonical files of a range and returns amarketdata.Series(one instrument's bars at one interval, strictly ascending by time and by availability) with the state of every month:loaded,bad(has bad records),missing, orstale(another build in force, never used). Strict by default: a stale month always fails the load, and a missing or bad one fails it unless the request allows it.AvailableAtis the stored one, never recomputed.service.LoadSeriesis the application-service path;trader data statusis its first consumer. marketdata/service.Verify(trader data verify, ADR-090 section 8) checks a file against its raw partition independently ofBuild's code;service.VerifyCanonicalreads the raw (oanda.ReadSourcePartition), calls it per month, and--samplepicks months evenly (sampleMonths).rest(trader serve, REST API) is the HTTP transport: it normalizes the query parameters, callsservice.ReadBarsand writes JSON; reading, paging, the partial-by-default policy and the report of what is missing areReadBars' (which usesLoadwithSkipStale).restimports onlyserviceand the publicmarketdatamodel, and no provider (a test enforces it);cmd/trader/serve.goreads the environment (TRADER_API_TOKEN) and refuses a non-loopback address without it.service.MigrateLayout(trader data migrate, ADR-092) movesraw/<source>to<source>/rawwithstore.Move(one rename, never overwrites), the mapping inlayout.LegacyRawDirand verification bystore.Inventory; it is a dry run unless asked to apply, and file I/O stays instore(the architecture test coversservice).service.Audit(trader data audit, ADR-090 section 9) composesLoadSeries(months, gaps) andVerifyCanonicalinto oneAuditReportthat marshals to JSON; it adds no rule. The command only chooses the format and exits non-zero onErrAuditFailed.Loadalso checks the interior of the series with the market calendar (canonical.ClosureAtandGaps, ADR-090 section 7): the weekly close is not a gap, a year-end holiday is a reportedKnownClosure, a run on the reviewed listcanonical/known_holes.csv(bars the provider does not have) is a reportedKnownHole, any other missing bar isUnexplainedand fails the load unlessAllowMissing. The ends of the range are not judged (a forming candle is never stored).Loaddoes not callSeries.Validate: every file is validated as it is read and months are read in order, so the series is already valid; a consumer that needs the guarantee (runSeries) validates it itself.Bar.Instrumentnames a bar. Readers set it from the file's key; it is never a column of a row.
The root package runs on marketdata.Bar and replays marketdata.Series values
that a reader (Load) has already loaded; it has no series type of its own and
opens no file, resolves no path and knows no provider. Prices are read through
Bar.OHLC() (mid first). The scenario files in testdata/emax are read into a
Series (source synthetic) as mid.
backtest.runSeries replays one or more series on one trader with one shared
account and broker (replay.go, backtest.go):
- A run has one interval; series of different intervals are refused. Series are of different instruments, and an empty series contributes nothing.
mergedBarsorders bars by their storedAvailableAt, then by instrument name, so the order does not depend on the order of the series.AvailableAtis never recomputed from the opening time or the interval, and an opening time never authorises early delivery.framesgroups the bars with the sameAvailableAt: one group per distinct instant, holding only fresh bars. An instrument with no bar at that instant is absent; nothing is forward-filled or reused.- Per group the trader advances the broker through the whole group, then calls
strategy.onBars(ctx, bars)once with the whole fresh group (instrument order). The strategy sees every bar of the group before any of its orders is submitted; the trader then submits the returned orders in order.eachBaradapts a decision made one bar at a time (emax,emaxEach). A single-instrument run is the same path with one-bar groups.backtest.runkeeps the per-bar access the research tests and studies use. backtest.runSeriesvalidates every series (marketdata.Series.Validate) before replaying;mergedBarstrusts its input and checks only what the merge needs.trader.RunSeries(ctx, name, series...)is the engine's entry for series a reader loaded: it works on copies, normalizes instrument keys, refuses an instrument it has no market for, runs EMAX on each instrument with the scenarios' fixed settings and returns aResult(Writeprints the summary,Summaryhas the numbers). The engine opens no file;RunSeriesWithtakes anio.Writerfor the journal, whichorder.recordfeeds (journal.go; onlyservice.RunBacktestjournals every run, a directRunSeriescall writes one only if given a writer) so every order transition of every order, the broker's exits included, is written as it happens.service.RunBacktestis the application service that joins the two and journals every run: it makes the RunID and the run directory (internal/results, the one package that opens those files), thenLoadSeries(the market-data service's policy for missing, bad and stale months), thenRunSeriesWith, then finalizesrun.json;trader backtestonly parses flags and prints.- Mid prices and ticks (#667). A canonical mid is the exact midpoint of bid and
ask, so it can sit half a tick from a whole tick (1.100005 for ticks of 0.00001).
trader.RunSeriesrounds every bar price to the nearest tick of the market (a tie goes up) as the bars enter the engine (catalog.roundToTicks), on its own copy: the canonical data and the series passed in are unchanged. The series is validated as it was given first; the engine's bar then carries only the price series it trades (Bar.OHLC, asMid), since a rounded mid is no longer the midpoint of the stored bid and ask. The engine and its strategies only see whole ticks. The scenario path does not round: it still refuses a bar that is off the tick. - Warm-up.
trader.RunSeriestakestradeFrom: a bar that opens before it is a warm-up bar, shown to a strategy that implementswarmeronly to prefill its indicators (EMAX feeds its EMAs, takes no position, places no order), never replayed on the broker and not counted. After the warm-up EMAX finds its EMAs seeded, so it trades the trend as it stands.service.RunBacktestsupplies the bars of the UTC month beforeFromwhen they can be loaded: it uses the loader and the run'sAllowBad, treats a missing month as simply absent, and runs an instrument without warm-up (logged) when its earlier months are stale or bad. Only the traded range is strict.
- Every new or changed function has tests, including error and boundary cases.
- One test file per source file, named after it; cross-file tests and shared helpers
in
<package>_test.go(rules in ai-skills.org). - Tests never touch
/srv/data: each package that uses the store hasfunc TestMain(m *testing.M) { os.Exit(storetest.Main(m)) }, and tests that write callstoretest.Root(t)for a fresh temporary root. - Fixtures: shared ones in the root
testdata/, package ones in the package'stestdata/(testing architecture). make checkbefore every pull request, and the EMAX summaries unchanged (go run ./cmd/trader 2>/dev/null | md5sumprints195e457a21def1223e7181c86cb824a4).
- A command: a file in
cmd/traderthat resolves options and calls a service; import nothing but the service, the store and logging, and add the import-guard test (seebuild_test.go). - A provider capability: in
provider/<name>; map to the public model and add nothing domain-specific. Wire it in an application service. - A domain rule: in the domain service, with a test per rule.
- A file format or path: a new version name and a rebuild, never a silent
change; paths come from
layout. - An ADR: one decision per record; do not edit an accepted ADR silently, add an amendment or a new record.