mbel-tech · Oct 16, 2026 · Oct 15, 2026 · Oct 15, 2026 · Oct 15, 2026 · Oct 15, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+
+- **Three more 3-D metrics.** IL-18 *Vertical Speed* (mean, median and maximum of |dz/dt| in cm/s, plus body lengths per second), GL-18 *3-D Inter-Individual Distance* (mean over all pairs, the 3-D counterpart of GL-2) and GL-19 *3-D Group Cohesion* (reciprocal of the 3-D nearest-neighbour or inter-individual distance, chosen by `cohesion_source`). They need a fused or native 3-D session and a depth scale, skip frames with a missing animal as GL-16 does, and are greyed out otherwise. The registry now holds 57 metrics (decision D-052).
+- **Warning before a run when most animal slots are empty.** A multi-animal session in which at least half the (frame, animal) slots have no position, typically a Ctrax file with one slot per track fragment, now raises an advisory in `track2data validate`, the log and the run README, saying that group metrics skip any frame with a missing animal and pointing to the `top_n` option.
+- **Metadata: Excel workbooks with several sheets.** The metadata screen loads `.xlsx` files (it only took CSV before) and, when the workbook has more than one sheet, shows a **Sheet** menu; the choice is saved with the project. Projects without a chosen sheet are unchanged, including their project hash. The old `.xls` format is refused with a message that says to save as `.xlsx` or CSV.
+- **DeepLabCut `.h5` prediction files** are read (`deeplabcut_h5`). The numbers and the pickled column labels are decoded without PyTables, and the labels by an unpickler that refuses every global, so a crafted file cannot run code. A `.csv` and `.h5` pair of the same name is one session. The reader is marked synthetic-only until the real sample has been read in a run; the multi-animal `*_el.h5` tracklet files are refused with a remediation.
+- **`movement` / xarray adapter.** `track2data.interop.movement.to_movement_dataset(psess)` returns a preprocessed session as an `xarray.Dataset` in the `movement` layout (`position` over time, individuals, keypoints, space), with true video time across tracking gaps, a `z` coordinate for native 3-D sessions, and the tracker's identity probability as a separate `identity_confidence` variable. Needs the new optional `xarray` extra.
+- **Import: "Choose software…" when nothing was recognised.** The empty-state dialog lists the readers that can try again with looser rules, and `track2data add --reader NAME` does the same on its own. DeepLabCut accepts a prediction table saved as `.txt` or `.dat`, and SLEAP an analysis file without an `.h5` extension. The header must still match, a candidate still goes through the reader's own `read`, and it is offered at low confidence.
+- **Import: "Read from video…"** fills the frame rate and frame size for readers that need them (DeepLabCut, SLEAP) from a video you choose (needs the optional `av` package). It never overwrites a value you typed, and an unreadable video changes nothing.
+- **Zones: the video's first frame is the backdrop** when a session has no background image (and no panel). It is used only when the decoded frame is exactly the declared video size, so zones are never drawn in the wrong pixel space; `extract_frame_rgb` returns the frame with its decoded size.
+
+### Changed
+
+- **IL-3 and IL-14 are skipped on a declared side view.** The arena centre and the wall distance are plan-view quantities, so a project whose camera view is **Side view** now greys them out and skips them (decision D-045). A project that never set a view, or set Top-down, is unchanged, as are fused and native 3-D sessions.
+
+### Fixed
+
+- **idtracker.ai v5 sessions with no recorded frame rate can be given one.** The legacy v5 reader now takes an `fps` option (`--option fps=25`, or the Frame rate field on import), used only after `video_object.npy` and the session json have been tried. Without any of the three it still stops with `VIDEO_FPS_MISSING` rather than assuming 1 fps; the message now says how to supply it.
+- **Rescanning a folder that was split into panels** offers the whole-frame session again instead of reporting it "already in the project". The import dialog and the project store now share one duplicate rule (decision D-046); a true duplicate is still left out.
+- **idtracker.ai pickle reader could run code.** The allowlist for `list_of_blobs.pickle` admitted every class in any module whose name starts with `numpy`, including helpers that execute code. It is now an exact list of the array-reconstruction functions numpy pickles use.
+- **Z-5 no longer invents an exit at a tracking-interval gap.** A stay in a zone that continued across two tracking intervals was reported as enter, exit, enter, with the exit on the separator row between them. Entry/exit counts, Z-6 and Z-9 were inflated for sessions tracked in separate intervals.
+- **Cache key includes the identity-free override.** Changing a session's identity-free setting now invalidates its cached preprocessing instead of serving the old gap-bridging.
+- **Binned exports.** The long table keeps `bin_index`, `bin_start_s` and `bin_end_s` as identifiers rather than melting them into measurements, and merging a binned metric after a whole-session one no longer drops its bin boundaries.
+- **idtracker.ai v5 reader** raises `VIDEO_FPS_MISSING` when the frame rate is in neither `video_object.npy` nor the session json, instead of assuming 1 fps. A real 1 fps recording is no longer mistaken for "missing".
+- **DeepLabCut multi-animal files with positional names** (`ind1`, `ind2`, ...) are declared identity-free, as SLEAP files already were, so per-individual metrics are skipped for them.
+- **Calibration page** offers inches and keeps whatever unit label the project has instead of rewriting it to the first choice on Apply.
+- **`track2data run --workers N`** now exists; the Processing page's headless command used it before the option did, and adds `--cache-dir` when a cache is in use.
+- **Export page** asks before overwriting after each export, not only when the folder is first chosen; the default folder moves on to a fresh timestamp.
+- **Metadata join** matches numeric session or key columns (trial numbers) against folder names.
+- **Panels:** a panel that no animal is in is refused with a message instead of producing a session of zero animals.
+- **Zones CSV** keeps each ROI's `+`/`-` sign, same-named polygons and the source resolution through save and load (older four-column files still load); `flip_orientation` keeps them too.
+- **Removing a session** drops its located-video override, so a later session with the same name does not inherit it.
+
 ## [0.3.0] — 2026-10-13
 
 ### Added

diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 
 You tracked your animals. idtracker.ai gave you a folder of trajectories.
 Track2Data turns that into tables you can put straight into R or Python:
-54 behavioural metrics, every one with a citation, in documented units,
+57 behavioural metrics, every one with a citation, in documented units,
 with a record of exactly which frames were measured and which were
 reconstructed.
 
@@ -209,7 +209,7 @@ every export's `manifest.json`.
 
 A rendered site is published from these files at
 **<https://mbel-tech.github.io/Track2Data/>**, including a browsable
-catalogue of all 54 metrics with their formulas, units and DOIs.
+catalogue of all 57 metrics with their formulas, units and DOIs.
 
 The **[wiki](https://github.com/mbel-tech/Track2Data/wiki)** covers what sits
 around that reference material and is not pinned to a release:

diff --git a/docs/3d-movement/2026-10-08-3d-roadmap.md b/docs/3d-movement/2026-10-08-3d-roadmap.md
@@ -28,5 +28,5 @@ project-level camera-view setting that B reuses.
   a vertical axis for 3-D data.
 - The `has_pixel_frame` guard from native-units joined the IL-3 and IL-14 arena fallback in A (the data extent is used, with a warning).
 - Per-session camera views arrive with D (`Engine.camera_view_for(session)` is the seam).
-- Whether IL-3 and IL-14, which assume a top-down view, should be gated for side views is
-  left open.
+- IL-3 and IL-14, which assume a top-down view, are gated for side views (D-045); an unset view keeps
+  both.
diff --git a/docs/3d-movement/2026-10-08-side-view-depth-design.md b/docs/3d-movement/2026-10-08-side-view-depth-design.md
@@ -15,7 +15,8 @@ computed from a top-down y axis.
 - The metric is **IL-15**, not Z-10: exports bucket metrics by ID prefix and the UI by level.
   Z-10 is re-reserved for auto depth bands.
 - Camera view is project-level. A per-session override arrives with sub-project D.
-- Existing metrics are not gated by camera view in this cycle.
+- Existing metrics were not gated by camera view in this cycle. (IL-3 and IL-14 are gated to top-down and
+  unset views since D-045.)
 
 ## Design
 

diff --git a/docs/ENGINE_DESIGN.md b/docs/ENGINE_DESIGN.md
@@ -385,10 +385,14 @@ ALIASES = {"condition": "treatment", "date": "trial_date"}
 
 ### 6.2 Loader
 
-`metadata.loader.load(path)` returns a `pandas.DataFrame` with stripped
-whitespace, lowered string columns where appropriate, and ISO-8601
-dates. Excel files: read sheet 0; multi-sheet support is a future
-feature.
+`metadata.loader.load(path, sheet=None)` returns a `pandas.DataFrame` with
+stripped whitespace, lowered string columns where appropriate, and ISO-8601
+dates. For an Excel workbook `sheet` names the sheet to read; `None` reads the
+first. `list_sheets(path)` gives a workbook's sheet names (`[]` for a CSV) and
+an unknown name raises `ValueError` listing them. The chosen sheet is stored
+in `MetadataSource.sheet`, which is left out of the saved manifest while it is
+`None` so projects without a sheet keep their `project_hash`. The old `.xls`
+format is refused with a message (openpyxl cannot read it).
 
 ### 6.3 Mapping & join
 

diff --git a/docs/INTEROPERABILITY.md b/docs/INTEROPERABILITY.md
@@ -50,93 +50,68 @@ long.merge(book[["column", "unit", "doi"]].drop_duplicates(), on="column")
 trajectory libraries expect: one row per (frame, individual) with `x_px`,
 `y_px`, `time_s`, and reconstruction flags.
 
-## Planned: the `movement` package
+## Using the `movement` package
 
 [`movement`](https://movement.neuroinformatics.dev/) (Neuroinformatics Unit,
-UCL) is an xarray-based library for animal-movement data. It is a natural
-downstream consumer: it has plotting, filtering and kinematic utilities that
-Track2Data does not attempt, and it reads pose data from DeepLabCut, SLEAP
-and others.
+UCL) is an xarray-based library for animal-movement data, with plotting,
+filtering and kinematic utilities that Track2Data does not attempt. Track2Data
+hands it a preprocessed session as an `xarray.Dataset` in `movement`'s layout.
+Install the extra first: `pip install track2data[xarray]`.
 
-**Not implemented yet.** The mapping is documented here first so that the
-shape of the adapter is agreed before it is written, and so anyone who wants
-it in the meantime can do the conversion by hand.
+```python
+from track2data.api import Engine
+from track2data.core.manifest import read
+from track2data.interop.movement import to_movement_dataset
+
+engine = Engine(read("project.t2d.json"))
+psess = engine.preprocess(engine.import_session("session_folder"))
+ds = to_movement_dataset(psess)
+```
 
 ### The mapping
 
-`movement` centres on a `xarray.Dataset` with a `position` DataArray of
-dimensions `(time, individuals, keypoints, space)`.
+`position` has dimensions `(time, individuals, keypoints, space)`.
 
 | `movement` | Track2Data | Note |
 |---|---|---|
-| `position` | `PreprocessedSession.xy`, `(n_frames, n_animals, 2)` | Add a length-1 `keypoints` axis |
-| `time` | `frame / fps` | `time_s` in the per-frame table |
-| `individuals` | `Session.identities_labels`, else `0..n-1` | Labels come from the Validator when set |
-| `keypoints` | a single `"centroid"` | idtracker.ai tracks a centroid, not a skeleton |
-| `space` | `["x", "y"]` | Always 2-D |
-| `confidence` | `Session.id_probabilities` | Identity confidence, **not** pose confidence — see below |
-| `fps` (attr) | `Session.video.fps` | |
-| `source_software` (attr) | `"idtracker.ai"` | |
+| `position` | `PreprocessedSession.xy` (plus `z` for a native 3-D session) | A length-1 `keypoints` axis is added |
+| `time` | the session's timeline divided by `fps` | True video time, so a session tracked in separate intervals keeps its gaps |
+| `individuals` | `Session.identities_labels`, else `0..n-1` | |
+| `keypoints` | the keypoint that stands for the animal, else `"centroid"` | The rest of a skeleton is not exported |
+| `space` | `["x", "y"]`, or `["x", "y", "z"]` for a native 3-D session | |
+| `identity_confidence` | `Session.id_probabilities` | A separate variable, **not** `confidence` (see below); absent when the tracker gave none |
+| attrs | `fps`, `source_software`, `reader_verification`, `length_unit`, `px_per_cm` | `length_unit` is the unit of `position` (`px`, or the file's unit for a native 3-D session) |
+
+The single NaN row that stands for an unfilled gap between tracking intervals
+is dropped; pass `include_separator_rows=True` to keep it. Rows for unobserved
+video stay as NaN.
 
 ### Two things that do not map cleanly
 
 **`confidence` means something different.** In `movement` it is a pose
 estimator's per-keypoint confidence: how sure the network is about *where*
 the body part is. idtracker.ai's `id_probabilities` is how sure it is about
-*which animal* this is — the position may be exact while the identity is a
-coin flip. Writing one into the other's slot would silently invite a filter
-on the wrong quantity. Any adapter should carry it as a separately-named
-variable, or document the substitution loudly.
+*which animal* this is, so the position may be exact while the identity is a
+coin flip. Writing one into the other's slot would invite a filter on the wrong
+quantity, which is why it is named `identity_confidence`.
 
 **Zones and calibration have no home in the pose schema.** `main_zone`,
-`sec_zone`, `px_per_cm` and the body-length scale are Track2Data concepts.
-They would ride along as dataset attributes and extra data variables, which
-`movement` tolerates but does not interpret.
-
-### If you need it now
-
-```python
-import numpy as np
-import xarray as xr
-from track2data.api import Engine
-from track2data.core.manifest import read
-
-engine = Engine(read("project.t2d.json"))
-psess = engine.preprocess(engine.import_session("session_folder"))
-
-n_frames, n_animals, _ = psess.xy.shape
-labels = psess.session.identities_labels or [str(k) for k in range(n_animals)]
-
-ds = xr.Dataset(
-    {
-        "position": (
-            ("time", "individuals", "keypoints", "space"),
-            psess.xy[:, :, np.newaxis, :],
-        )
-    },
-    coords={
-        "time": np.arange(n_frames) / psess.fps,
-        "individuals": labels,
-        "keypoints": ["centroid"],
-        "space": ["x", "y"],
-    },
-    attrs={"fps": psess.fps, "source_software": "idtracker.ai"},
-)
-```
-
-## Other formats
+`sec_zone` and the body-length scale stay in Track2Data; only `px_per_cm`
+rides along as an attribute.
 
 | Target | Status | Notes |
 |---|---|---|
 | CSV (long and wide) | ✅ shipped | `csv_long`, `csv_wide` |
 | Excel | ✅ shipped | `excel` exporter, needs `openpyxl` |
 | Feather / Arrow | ✅ shipped | `feather` exporter, needs `pyarrow` |
-| `movement` xarray | 📋 documented above | Not implemented |
+| `movement` xarray | ✅ shipped | `track2data.interop.movement.to_movement_dataset`, needs `xarray` |
 | DeepLabCut CSV input (also Lightning Pose, EKS) | ✅ supported | One session per prediction file; the frame rate and frame size are options you give (the file records neither); one keypoint stands for the animal and the whole skeleton is stored beside it. See `docs/tracker-formats/README.md`. |
 | SLEAP analysis HDF5 input | ✅ supported | In SLEAP choose *File > Export Analysis HDF5*. One session per file; the frame rate and frame size are options you give; one skeleton node stands for the animal and the whole skeleton is stored beside it. `.slp` project files are not read. See `docs/tracker-formats/README.md`. |
 | Ctrax raw `.mat` input | ✅ supported | The file Ctrax's *Save Tracks as Matlab File* writes. Frame size is an option you give; the frame rate comes from the file. Ctrax track ids are fragments, so a session is identity-free and one slot per id (`top_n` keeps the longest). See `docs/tracker-formats/README.md`. |
 | `trx.mat` input (Ctrax / FlyTracker / JAABA) | 🧪 synthetic only | A classic (v5/v7) MATLAB file holding a `trx` struct, built from the documented layout with no real sample pinned yet. Frame size is an option; `fps` is read from the file. `pxpermm` (pixels per mm) becomes the session scale for Session calibration, pending the usual unit confirmation; a value of exactly 1 is ignored. v7.3 files are not read. |
-| DeepLabCut `.h5` input | 🕒 planned | Waiting on the decoder decision in the reader rollout (`docs/tracker-formats/`). Until then export DeepLabCut as CSV. A reader plug-in is also possible — see the `track2data.readers` entry point in `CONTRIBUTING.md`. |
+| DeepLabCut `.h5` input | 🧪 synthetic only | The `.h5` DeepLabCut writes beside the CSV (pandas table under `df_with_missing`). Read with h5py; the pickled column labels are decoded by an unpickler that refuses every global, and PyTables is not needed (D-050). The frame rate and frame size are options you give; the keypoint is typed, not picked from a list. A CSV of the same name beside it is the one session. Multi-animal tracklet files (`*_el.h5`) are refused. Becomes *supported* once the real EPM sample has been read in a run. |
+| Anipose pose-3d CSV input | 🧪 synthetic only | One animal per file, native 3-D (X, Y, Z in the file's unit). Built from the documented layout; no real sample pinned yet. Several animals in one file are not read. |
+| DeepLabCut-3D input | 🧪 synthetic only | The triangulated 3-D table DeepLabCut writes, read as native 3-D. Built from the documented layout; no real sample pinned yet. |
 | NWB | ❌ not planned | Would be worth revisiting if the behavioural-NWB extensions stabilise. |
 
 Writing a reader for another tracker does not require changing Track2Data: