Code quality

.clang-tidy

@@ -13,6 +13,7 @@ Checks: >
   -cppcoreguidelines-avoid-do-while,
   -readability-trailing-comma,
   -misc-non-private-member-variables-in-classes,
+  -cppcoreguidelines-non-private-member-variables-in-classes,
   -readability-implicit-bool-conversion,
   -misc-no-recursion,
   -readability-function-cognitive-complexity,
@@ -28,6 +29,24 @@ Checks: >
   -cppcoreguidelines-pro-type-union-access,
   -cppcoreguidelines-pro-type-vararg,
   -misc-use-internal-linkage
+# cppcoreguidelines-pro-bounds-avoid-unchecked-container-access and
+# readability-trailing-comma: unknown to the standalone clang-tidy-20/18 apt
+# packages CI uses (excluding them there is a harmless no-op), but real,
+# firing checks under clangd's bundled clang-tidy despite reporting the same
+# "20.1.2" version string -- same package version, different check registry.
+# Confirmed live in the editor: without these, operator[] bounds-access noise
+# floods every file, contradicting the already-excluded pro-bounds-* siblings.
+# Do not remove based on `clang-tidy-20 --list-checks` alone.
+# Without this, clang-tidy parses included headers but only reports
+# diagnostics located in the main file (the .cpp) being compiled, so issues
+# inside our own headers were silently dropped in CI even though every
+# header is pulled in by some TU. Match only our own headers, not
+# FetchContent'd third-party ones under build/.
+# NOTE: as of clang-tidy-20, setting this here is not sufficient on its own —
+# --dump-config shows it loaded, but it has no effect on `-p <builddir>`
+# invocations unless the identical value is ALSO passed via --header-filter
+# on the command line. CI's clang-tidy step does that; keep both in sync.
+HeaderFilterRegex: '^.*/include/ibex/.*\.hpp$'
 CheckOptions:
   # libstdc++ implementation headers are not portable include providers.
   # Prefer the public standard-library header for every standard declaration.

.clangd

@@ -23,6 +23,7 @@ Diagnostics:
       - cppcoreguidelines-avoid-do-while
       - readability-trailing-comma
       - misc-non-private-member-variables-in-classes
+      - cppcoreguidelines-non-private-member-variables-in-classes
       - readability-implicit-bool-conversion
       - misc-no-recursion
       - readability-function-cognitive-complexity

.github/workflows/ci.yml

@@ -242,15 +242,20 @@ jobs:
 
       - name: Run clang-tidy
         # Analyze every project translation unit and treat every diagnostic as
-        # an error. Headers are covered transitively by their including TUs.
+        # an error. clang-tidy parses every included header too, but only
+        # *reports* diagnostics located in the main file (the .cpp) unless
+        # --header-filter matches the header's path — and, at least as of
+        # clang-tidy-20, that filter is NOT picked up from .clang-tidy's
+        # HeaderFilterRegex on its own (--dump-config shows it loaded, but it
+        # has no effect unless the same value is also passed on the command
+        # line here). Keep this regex in sync with .clang-tidy.
         run: |
           if command -v clang-tidy-20 >/dev/null 2>&1; then TIDY=clang-tidy-20; else TIDY=clang-tidy-18; fi
-          # Run one clang-tidy process per core. Analyze only our .cpp TUs;
-          # headers are covered transitively via the TUs that include them, so
-          # tidying *.hpp standalone is redundant (and was a big chunk of the
-          # runtime). -n1 so each file is its own job for -P to parallelize.
+          # Run one clang-tidy process per core. -n1 so each file is its own
+          # job for -P to parallelize.
           find src -name '*.cpp' -print0 | \
-            xargs -0 -P "$(nproc)" -n1 "$TIDY" -p build --warnings-as-errors='*'
+            xargs -0 -P "$(nproc)" -n1 "$TIDY" -p build --warnings-as-errors='*' \
+              --header-filter='^.*/include/ibex/.*\.hpp$'
 
   format-check:
     runs-on: ubuntu-24.04

SPEC.md

@@ -556,6 +556,10 @@ require a `Series<T>` annotation outside `Table { ... }`; inside a table
 constructor, `col = []` produces a zero-row `Int64` series by default.
 Duration literals (`1m`, `30s`) are not valid series elements.
 
+`null` is also a valid element, but only inside a `Table { ... }` column: a
+standalone Series binding is a bare column with no validity bitmap, so it has
+nowhere to record the null. See Section 8.2.
+
 `TimeFrame<S>` is a `DataFrame<S>` with the additional invariant that exactly
 one column of type `Timestamp` is designated as the time index, and rows are
 sorted by that index in ascending order.
@@ -2305,6 +2309,9 @@ trades join quotes[order { ts asc }] on symbol take first
   row — that is a property of this implementation, not a promise.
 - A row with no match has nothing to choose between, so an outer join's padded
   rows are unaffected.
+- `take` chooses pairs; it never changes which rows a join preserves. On a
+  `right` or `outer` join, a right row whose every match was taken by other left
+  rows is emitted null-padded, exactly as an unmatched right row is.
 - `expect` describes how the inputs match, so it is checked **before** `take`
   drops anything. Otherwise `take first` would satisfy every `expect n:1` by
   construction.
@@ -2430,6 +2437,12 @@ must be an aggregate. A `filter` may hold *capture equalities* —
 `inner_column == outer(outer_column)`, in either order — alongside any number of
 ordinary local predicates.
 
+Captures need not name distinct columns. Two may share the outer column
+(`a == outer(x) && b == outer(x)`: inner rows where both columns equal that
+outer row's `x`) or the inner one (`a == outer(x) && a == outer(y)`: matching
+only where the two outer values agree). An exactly repeated capture is
+redundant and ignored.
+
 **Correlated or not.** A subquery that captures with `outer(...)` is evaluated
 per captured key. One that captures nothing is *uncorrelated*: a single value,
 the same for every row, evaluated once and broadcast.
@@ -2449,10 +2462,22 @@ single row. That is also why a source used by both the outer query and the
 subquery should be bound once (`let partsupp = read_parquet(...)`) and named
 twice — one binding is one read.
 
-**Nulls.** When no inner row matches an outer row's captured key, the subquery's
-value is null, and a comparison against null is never true: the row is dropped.
-An uncorrelated subquery whose input is empty likewise keeps no rows. This is
-SQL's scalar-subquery behaviour.
+**Empty inputs.** A subquery over no rows has the value its aggregate has over
+no rows (Section 7.1): `0` for `count()`, `count(col)` and `count_distinct(col)`,
+null for every other aggregate. That holds both for an uncorrelated subquery
+whose input is empty and for an outer row whose captured key matches no inner
+row. A comparison against null is never true, so such a row is dropped. A
+comparison against a count of `0` is an ordinary comparison:
+
+```ibex
+// Parts nobody supplies: kept, because their count is 0, not null.
+part[filter 0 == scalar(supply[filter ps_partkey == outer(p_partkey), select { n = count() }])]
+```
+
+This is SQL's scalar-subquery behaviour. A count must be the whole selected
+column: `select { n = count() + 1 }` is rejected, because a key with no inner
+rows reaches the decorrelated plan as a missing group, and only a bare count's
+value over no rows is known there.
 
 **Result schema.** The subquery's value is not a column of the result. A filter
 yields the rows it kept, never a wider table. Naming the columns that stay is
@@ -2475,11 +2500,9 @@ refers to the query enclosing it, and subqueries do not nest.
 `scalar` keeps its two-argument form — `scalar(table, column)` extracts a value
 from a one-row table (Section 12). Arity tells the two apart.
 
-An uncorrelated subquery over an empty input, or one whose aggregate is null (an
-empty or all-null group), yields a **null scalar** (Section 6.7); in
-`filter … == scalar(…)` the comparison is then null and the row is dropped
-(Section 3), matching SQL. For a correlated subquery, a captured key with no
-matching inner rows yields null for that key.
+A subquery whose aggregate is null (no rows, or only null values, for anything
+but a count) yields a **null scalar** (Section 6.7); in `filter … == scalar(…)`
+the comparison is then null and the row is dropped (Section 3), matching SQL.
 
 ---
 
@@ -2646,8 +2669,8 @@ The following built-in functions are **aggregate functions**. They consume a
 |-----------------------|--------------------|------------|-------|
 | `sum(col)`            | `Series<Numeric>`  | Same numeric type | Returns null for an all-null group. |
 | `mean(col)`           | `Series<Numeric>`  | `Float64`  | |
-| `min(col)`            | `Series<T>`        | `T`        | |
-| `max(col)`            | `Series<T>`        | `T`        | |
+| `min(col)`            | `Series<T>`        | `T`        | `T` is numeric or `String` (including categorical). Strings compare byte-wise, i.e. by UTF-8 code point, so `"Z" < "a"`. |
+| `max(col)`            | `Series<T>`        | `T`        | As `min`. |
 | `count()`             | (none)             | `Int64`    | Counts rows in the group, nulls included. |
 | `count(col)`          | `Series<T>`        | `Int64`    | Counts the **non-null** values of `col`; `0` (not null) for a group with none. The argument must be a column name. |
 | `count_distinct(col)` | `Series<T>`        | `Int64`    | Counts the number of **distinct non-null** values of `col` in the group; `0` (not null) for a group with none. `T` may be any scalar type. |
@@ -2659,6 +2682,18 @@ The following built-in functions are **aggregate functions**. They consume a
 
 `Numeric` denotes `Int32 | Int64 | Float32 | Float64`.
 
+**Empty input.** Without `by`, the whole input is one group, even when it has
+no rows: the result is always exactly one row. Over no rows `count()`,
+`count(col)` and `count_distinct(col)` are `0` and every other aggregate is
+null, and expressions over them follow from that (`mean(x) / 7.0` is null).
+With `by`, groups come from the rows that exist, so an empty input has no
+groups and the result has no rows. This is SQL's rule.
+
+```
+t[filter false, select { n = count(), m = mean(v) }]   // 1 row: n = 0, m = null
+t[filter false, select { n = count() }, by g]           // 0 rows
+```
+
 All other functions (user-defined externs, built-in scalars) are **scalar
 functions**.
 
@@ -2805,6 +2840,25 @@ All elements within one series literal must have the same literal kind; mixing t
 is a lowering error. Duration literals (`1m`, `30s`) are not valid array
 elements.
 
+`null` may appear as an element, producing a null cell — it clears that row's
+validity bit (Section 3.5) rather than storing a value:
+
+```
+Table {
+    symbol = ["AAPL", null, "MSFT"],   // String, row 1 null
+    price  = [150.0, 140.0, null],     // Float64, row 2 null
+}
+```
+
+`null` carries no type of its own, so the column's type comes from its first
+non-null element and a `null` in a column of any other kind is not a type
+mismatch. A column whose elements are *all* `null` has no type to take, and
+falls back to `Int64` — the same default `col = []` uses:
+
+```
+Table { a = [null, null] }   // Int64, both rows null
+```
+
 ### 8.3 Expression Columns
 
 Any expression that produces a **Series** or **Table** may be used as a column
@@ -3521,7 +3575,10 @@ let iris = read_csv("iris.csv");
 ```
 
 `read_csv` infers column types from the input file (Int64, Float64, or String
-per column). The resulting schema is implementation-defined.
+per column). The resulting schema is implementation-defined. A bare empty
+field (`a,,c`) reads as null rather than forcing an otherwise-numeric column
+to String; a column with no non-empty values, or one with a non-numeric
+value in any row, still infers as String, with empty fields kept as `""`.
 
 The standard CLI, REPL, and Python hosts link the first-party Parquet backend
 directly; an optional thin compatibility plugin delegates to that same backend.

docs/benchmarks.html

@@ -143,10 +143,12 @@ <h2>Per-query timings</h2>
     </p>
     <p class="bench-note">
       <strong>Caveats.</strong> Every engine now materialises its full result.
-      <code>tf rolling EWMA</code> is time-windowed in Ibex versus full-series in
-      Polars (both O(n), different maths). Each page is generated from a single
-      run; <code>SQLite</code> and the data.table <code>rolling median/std</code>
-      cells are omitted (they dominate wall-clock and add no competitive signal).
+      <code>tf rolling EWMA</code> uses the bounded window definition in Ibex,
+      pandas and Polars. ClickHouse statistical/OHLC cells <em>cannot compute
+      correctly under the shared semantics</em> and are excluded from claims.
+      Each page is generated from a single run; <code>SQLite</code> and the
+      data.table <code>rolling median/std</code> cells are omitted (they dominate
+      wall-clock and add no competitive signal).
     </p>
     <p class="bench-note">
       <strong>The code behind these numbers.</strong> Every query's exact code, in

docs/engine/.gitignore

@@ -0,0 +1,10 @@
+*.aux
+*.bbl
+*.blg
+*.fdb_latexmk
+*.fls
+*.log
+*.out
+*.pdf
+*.synctex.gz
+*.toc

docs/engine/README.md

@@ -0,0 +1,56 @@
+# Ibex analytics engine description
+
+`engine.tex` is an intentionally incomplete technical foundation, complementary
+to `SPEC.md`. It describes the inspected implementation and identifies questions
+for author review. `references.bib` contains the cited research literature.
+
+Build with a standard LaTeX installation, from the repository root:
+
+```sh
+bash docs/engine/build.sh
+```
+
+Equivalently, run these commands from this directory:
+
+```sh
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+bibtex engine
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+```
+
+The output is `engine.pdf`. Generated files are ignored by Git. Required packages
+are listed in the TeX preamble; no shell escape or external fonts are required.
+
+Interpreted and compiled execution share a functional-equivalence requirement.
+The draft's parity bug appendix records encountered divergences; an unsupported
+diagnostic alone does not establish an intentional exception.
+
+Local source paths are repository-relative. Each substantive section includes an
+implementation evidence paragraph; citations to papers supply context and do
+not establish that Ibex implements every technique in those papers.
+
+Agreed direction: address external database-systems readers with operational
+detail, using a time-series running workload. Preserve row-order guarantees;
+floating-point reproducibility may be relaxed, with the precise contract still
+to be defined. `operations.tex` now covers lazy decoding, map kernels, streaming
+equality joins, aggregation phases, Arrow ownership, and retained memory.
+`cost.tex` explains cost drivers using the September 3, 2026 benchmark snapshot
+in `docs/benchmarks.html`, checked against its archived source CSV. It records
+hardware, timing boundaries, and the limits of interpreting the measurements.
+No new benchmark was run. Detailed time-series semantics, numerical
+reproducibility rules, and new performance experiments remain deferred.
+The agreed-direction section records remaining investigations.
+
+`trade_summary.ibex` is the executable running example, included directly in
+the LaTeX document. From the repository root:
+
+```sh
+build-release/tools/ibex --no-history --report-planner docs/engine/trade_summary.ibex
+```
+
+This was checked using the existing release executable: two output rows with
+volume/notional totals of 30/3040 (AAPL) and 20/4030 (MSFT). The planner reported
+statement execution for this in-memory example. This is an example check, not
+a benchmark or a compiled/interpreted parity test. No engine rebuild or test
+suite run was needed for this documentation change.

docs/engine/build.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright (C) 2026 Bob Jansen
+
+# Build the Ibex engine document and resolve its bibliography and references.
+
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "${script_dir}"
+
+for command in pdflatex bibtex; do
+    if ! command -v "${command}" >/dev/null 2>&1; then
+        echo "error: ${command} is required to build engine.tex" >&2
+        exit 1
+    fi
+done
+
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+bibtex engine
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+pdflatex -interaction=nonstopmode -halt-on-error engine.tex
+
+echo "built ${script_dir}/engine.pdf"