Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,18 @@ fix flows to every page automatically; two hand-written HTML pages never will.
firmware's ASCII error payload on non-ok status — keep that for new ops.
New `.cmenu` popups: the document click-away closer ignores clicks inside
`.cmenu-pop`; one-shot `.cmenu-item`s (not in a `.cmenu-row`) auto-close.
- **Metadata / controlled-vocab sourcing (THE rule):** when a **course repo is
- **Data repos (v0.72):** the Studio/Pattern Designer/dashboard target ONE GitHub
data repo chosen in File ▾ → GitHub → Repo, from the registry in
`js/studio-data-repos.js` (classic dual-export; course + lab entries + `parseRepo`
+ label helpers). **No default repo** on a fresh browser. **Copy rule:** user-visible
wording goes through `Studio.dataRepoLabel()` / `Studio.idLabel()` /
`Studio.sharedLabel()` ("Bench id" for the course repo, "Rig id" otherwise);
internal identifiers (`courseSettings`, `refreshCourseMeta`, `fetchCourse*`,
`openFromCourseRepo`, `fmOpenCourse`, `studio_bench_id`) keep their historical
names — tests anchor on them, do not mass-rename. New repo = registry entry +
`scripts/seed-data-repo.sh`; never enable branch protection. Setup guide:
`docs/development/data-repo-setup.md`.
- **Metadata / controlled-vocab sourcing (THE rule):** when a **data repo is
configured AND signed in**, ALL metadata vocabularies load from that repo (its
root-level YAML) and their ↗ source links repoint there — the connected repo is
the source of truth. When not (offline / not signed in), fall back to the
Expand Down
320 changes: 221 additions & 99 deletions arena_studio.html

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion course/cshl-2026/docs/github-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ Everything lives in one repository: **`reiserlab/cshl-2026-course`** (public sin
August 2026 — anyone can read it; writing from Arena Studio still needs the bench to
be signed in once per browser).

Each of the 7 bench rigs has a **bench id** (`bench01` … `bench07`). The repo
Each of the 7 bench rigs has a **bench id** (`bench01` … `bench07`; the field is
labelled "Bench id" while the course repo is selected in File ▾ → GitHub → Repo). The repo
is organized so no two benches ever overwrite each other's files:

```
Expand Down
14 changes: 13 additions & 1 deletion dashboard/data-browser/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -399,7 +399,7 @@ function renderCatalog() {
const visible = new Set(visibleDescriptors().map((item) => item.key));
if (!state.catalog.length) {
els.runCatalog.innerHTML = '<div class="empty-state">No runlogs indexed</div>';
els.catalogStatus.textContent = 'Open files or connect to the course repository.';
els.catalogStatus.textContent = 'Open files or connect to a data repository.';
els.catalogSearchInput.disabled = true;
els.renderSelectionButton.disabled = true;
renderFocusOptions();
Expand Down Expand Up @@ -1724,6 +1724,18 @@ window.addEventListener('resize', renderScope);

async function initialize() {
els.githubRepoInput.value = G.currentRepo();
// Offer the known data repos (course + lab) as suggestions; free text still works.
const dl = document.getElementById('dataRepoList');
const registry = window.StudioDataRepos;
if (dl && registry && Array.isArray(registry.DATA_REPOS)) {
dl.textContent = '';
for (const r of registry.DATA_REPOS) {
const o = document.createElement('option');
o.value = r.full;
o.label = r.label;
dl.appendChild(o);
}
}
loadAnalysisAxes();
if (window.location.hostname.endsWith('github.io')) {
els.plotSourceLink.href =
Expand Down
17 changes: 12 additions & 5 deletions dashboard/data-browser/github-repo.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,14 @@
const REPO_KEY = 'studio_gh_repo';
const BENCH_KEY = 'studio_bench_id';
const FOLDERS_KEY_PREFIX = 'dashboard_runlog_folders:';
const DEFAULT_REPO = 'reiserlab/cshl-2026-course';
// The registry (js/studio-data-repos.js, loaded before this file) lists the
// known data repos; the dashboard's default stays the first entry (the
// course repo). Falls back to the literal if the registry failed to load.
const REGISTRY = global.StudioDataRepos || null;
const DEFAULT_REPO =
REGISTRY && REGISTRY.DATA_REPOS && REGISTRY.DATA_REPOS[0]
? REGISTRY.DATA_REPOS[0].full
: 'reiserlab/cshl-2026-course';

function currentToken() {
return sessionStorage.getItem(TOKEN_KEY) || localStorage.getItem(TOKEN_KEY) || '';
Expand Down Expand Up @@ -98,7 +105,7 @@

async function apiJson(url, options) {
const token = currentToken();
if (!token) throw new Error('Sign in with the course GitHub token first');
if (!token) throw new Error('Sign in with a GitHub token first');
const response = await fetch(url, {
method: (options && options.method) || 'GET',
headers: headers(token),
Expand All @@ -125,7 +132,7 @@
const repo = parseRepo(repoValue || currentRepo());
const pat = prompt(
`Paste a GitHub personal access token for ${repo.full} (fine-grained for org members; classic for the shared course account).\n` +
'The course token should have Contents read/write access.\n\n' +
'It needs Contents read access (read/write keeps it compatible with Arena Studio).\n\n' +
'It is stored in sessionStorage first. The next prompt can remember it on this browser.'
);
if (!pat) return null;
Expand All @@ -134,7 +141,7 @@
sessionStorage.setItem(TOKEN_KEY, token);
if (
confirm(
'Remember this token on THIS browser?\nYES for a course bench; NO on a shared personal machine.'
'Remember this token on THIS browser?\nYES on a dedicated rig/bench computer or your own laptop; NO on a shared machine.'
)
) {
localStorage.setItem(TOKEN_KEY, token);
Expand Down Expand Up @@ -169,7 +176,7 @@

async function fetchRaw(repoValue, path, ref, prefixBytes) {
const token = currentToken();
if (!token) throw new Error('Sign in with the course GitHub token first');
if (!token) throw new Error('Sign in with a GitHub token first');
const repo = parseRepo(repoValue || currentRepo());
const requestHeaders = headers(token, 'application/vnd.github.raw');
if (prefixBytes) requestHeaders.Range = `bytes=0-${Math.max(1023, prefixBytes - 1)}`;
Expand Down
9 changes: 5 additions & 4 deletions dashboard/data-browser/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
<script defer src="vendor/kinematics.js?v=20260710-1647"></script>
<script defer src="analysis-core.js?v=20260710-2236"></script>
<script defer src="plot-specs.js?v=20260710-2236"></script>
<script defer src="github-repo.js?v=20260710-1647"></script>
<script defer src="../../js/studio-data-repos.js?v=20260904"></script>
<script defer src="github-repo.js?v=20260904"></script>
<script defer src="app.js?v=20260710-1653"></script>
</head>
<body>
Expand All @@ -20,7 +21,7 @@
<h1>Course Data Dashboard</h1>
</div>
<div class="header-actions">
<a id="courseRepoLink" class="quiet-link" href="https://github.com/reiserlab/cshl-2026-course" target="_blank" rel="noreferrer">Course repo</a>
<a id="courseRepoLink" class="quiet-link" href="https://github.com/reiserlab/cshl-2026-course" target="_blank" rel="noreferrer">Data repo</a>
<div class="status-strip" id="statusStrip" aria-live="polite">
<span class="status-dot" aria-hidden="true"></span>
<span id="statusText">No data loaded</span>
Expand Down Expand Up @@ -49,7 +50,7 @@ <h2>Data</h2>
</div>

<div class="github-controls">
<input id="githubRepoInput" type="text" value="reiserlab/cshl-2026-course" aria-label="GitHub repository">
<input id="githubRepoInput" type="text" list="dataRepoList" value="reiserlab/cshl-2026-course" aria-label="GitHub repository"><datalist id="dataRepoList"></datalist>
<button id="githubSignInButton" type="button">Sign in</button>
<button id="githubSignOutButton" type="button" disabled>Sign out</button>
<button id="chooseRigsButton" type="button" title="Choose which rig folders are included in the data catalog" disabled>Select rigs</button>
Expand Down Expand Up @@ -107,7 +108,7 @@ <h2>Data</h2>
</div>

<div class="catalog-head">
<span id="catalogStatus">Open files or connect to the course repository.</span>
<span id="catalogStatus">Open files or connect to a data repository.</span>
<input id="catalogSearchInput" type="search" placeholder="Filter by rig, experimenter, run, fly, genotype, protocol, notes" disabled>
</div>
<div id="runCatalog" class="run-catalog" aria-label="Available runlogs">
Expand Down
18 changes: 18 additions & 0 deletions docs/development/arena-studio-release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,24 @@ The Studio's footer used to carry the full changelog inline; it now shows one li
history lives here. Newest first. (Per-session engineering detail stays in
`arena-studio-handover.md` and the design docs — this file is the user-facing what-changed list.)

## v0.72 (2026-09-04) · Data repos: lab repo picker, "Rig id", no default repo

- **Repo is now a picker.** File ▾ → GitHub → Repo offers the CSHL 2026 course repo,
the new private lab repo (`reiserlab/arena-experiments`) and "Other…" for any
owner/name. A never-configured browser has **no repo selected** — saves go to local
files until one is picked (before, the course repo was silently the default).
Course benches already have their repo set, so nothing changes for them.
- **"Bench id" becomes "Rig id"** outside the course repo, with a pick-list fed by the
repo's `roster.yaml` (people's `rig_id`s plus an optional `rigs:` list, which also
carries the controller MAC for the connect-time cross-check).
- **Wording follows the repo** everywhere — save labels ("Save → Reiser lab
experiments"), the Open-from-repo picker ("This rig", "Shared protocols
(lab-wide)"), promote, run-log push, quick links, help texts. The Pattern Designer
and the data-browser dashboard use the same registry.
- **Expired token handling**: a 401 during a run-log commit now clears the stored
token and says so, instead of only "saved locally". Setup guide:
`docs/development/data-repo-setup.md`.

## v0.71 (2026-09-04) · Closed-loop apply is reset at run start and abort

- **No more error floods at the start of a run.** If FicTrac closed-loop "apply"
Expand Down
6 changes: 4 additions & 2 deletions docs/development/cshl-pipeline-test-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,10 @@ while destination/write settings are **locked by default**:
closes — a classic footgun for a kiosk bench).
2. Click **🛡 Safe mode**, enter the instructor password, then click the **🔒**
in the GitHub block to unlock destination/write setup.
3. **Repo** = `reiserlab/cshl-2026-course`.
4. **Bench id** = `bench01`…`bench07` (must match `roster.yaml`).
3. **Repo** = pick *CSHL 2026 course* in the dropdown (Studio v0.72+: nothing is
pre-selected on a fresh browser; "Other…" takes a custom owner/name).
4. **Bench id** = `bench01`…`bench07` (must match `roster.yaml`; the field is
labelled "Bench id" while the course repo is selected, "Rig id" otherwise).
5. Check **"Commit directly to default branch"**.
6. Click the lock again to **re-lock** (🔒). It re-locks automatically on the
next page load, so students can't alter the token/repo/bench id.
Expand Down
74 changes: 74 additions & 0 deletions docs/development/data-repo-rig-test-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Data-repo rig test checklist (Studio v0.72, PR #178)

Validates the data-repo generalization on a real rig: repo picker, "Rig id",
roster-fed rig list, and the full save / open / promote / run-log path against the
private lab repo `reiserlab/arena-experiments`. About 30 minutes with a rig.

**Before you start.** Serve the PR branch (`git checkout feat/data-repo-registry`,
`pixi run serve`, open `http://127.0.0.1:8000/arena_studio.html`) or merge #178 and
hard-refresh the rig after the Pages deploy (Cmd+Shift+R). You need your own
fine-grained token for `arena-experiments` (`data-repo-token-runbook.md` §B).
Steps 1–5 and 13–14 need no token and no arena.

## A. Picker and labels (no token)

1. **Fresh state.** Private window → Studio. GitHub block: Repo "— none (local
files) —", label **Rig id**, Save button "Save → local file", footer v0.72.
2. **Lock.** Repo dropdown is greyed. 🛡 advanced → 🔓 enables it; 🔒 greys it again.
3. **Lab repo.** Pick *Reiser lab experiments*. Label stays **Rig id**, placeholder
`e.g. 3e229-g6a`, bottom-corner quick links read "Reiser lab experiments ↗" and
point at `reiserlab/arena-experiments`.
4. **Course repo.** Pick *CSHL 2026 course*. Label flips to **Bench id**, placeholder
`e.g. bench03`, quick links repoint. Switch back to the lab repo.
5. **Other.** Pick *Other…*, type `nonsense`, Enter → banner "Repo must be
owner/name", previous repo restored. Type a valid `owner/name` → it sticks.

## B. Signed in against the lab repo

6. **Sign in.** File ▾ → Sign in… → paste your fine-grained token → YES to remember
(rig computer). Block reads `✓ @<you> → reiserlab/arena-experiments`. Run-view log
shows five "… from the Reiser lab experiments" lines (roster, genotypes, ages,
sexes, fly_numbers). Experimenter dropdown = the Janelia names from the roster.
7. **Rig id.** Click the Rig id field: no suggestions yet (lab roster has no rigs).
Type e.g. `3e229-g6a`, tick **Commit directly to default branch**, 🔒.
8. **Save.** Open any protocol → Save. Button reads "Save → Reiser lab experiments";
the file appears under `protocols/<rig-id>/` on GitHub within seconds.
9. **Open from Repo.** Shows "This rig — <rig-id>" with your save and "Shared
protocols (lab-wide)" (empty).
10. **Promote.** File ▾ → Promote to shared… → file appears in `protocols/shared/`
and under the shared header of the picker.
11. **Roster rigs + MAC chip.** In the repo, edit `roster.yaml`:
```yaml
rigs:
- rig_id: 3e229-g6a
mac: "00:00:00:00:00:00"
```
Reload the Studio, connect the arena. The Rig id field now suggests that id; the
"⚠ rig ≠ roster" chip appears (wrong MAC). Put the real MAC from the connect log
into the roster → reload → chip gone.
12. **Run log.** Run a short protocol as an *experiment* (bridge connected). Expect
"✓ Run log committed" and a file under `runlogs/<rig-id>/`.

## C. Other pages

13. **Pattern Designer.** Open via the Studio's *Patterns ↗* link. ⇪ Save to Repo →
the protocol destination row reads "rig <rig-id>" (not "bench").
14. **Dashboard.** `dashboard/data-browser/`: the repo field suggests both repos; pick
the lab repo → it lists your `runlogs/<rig-id>` folder.

## D. Course-bench regression

15. On a bench that already had the course repo stored, load v0.72: Repo shows
*CSHL 2026 course* pre-selected, label **Bench id**, "Save → CSHL 2026 course"
works as before.

## Known limits to keep in mind

- **Run-log size.** The browser commits via the GitHub Contents API, which rejects
files above ~35 MiB (measured; HTTP 422). A 20 s-trial P3 full run is ~26 MB; a
40 s-trial run is ~51 MB and will NOT auto-commit — the Studio says "saved locally
on the bridge machine". Push those from a clone (`git push` allows up to 100 MB per
file) until the Studio gains a large-file path (see release notes / issues).
- Aborted or test runs never auto-commit (by design) — use ⇪ Push log.

If anything in B misbehaves, the Run-view log line is the fastest diagnostic.
61 changes: 61 additions & 0 deletions docs/development/data-repo-setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Data repos — setup for a lab rig (and how the course repo fits)

Arena Studio, the Pattern Designer and the data-browser dashboard read and write a
GitHub **data repo**: protocols (`protocols/<rig-id>/`), their colocated patterns
(`…_patterns/`), the shared pattern library (`patterns/`), promoted protocols
(`protocols/shared/`) and one run log per completed recorded run
(`runlogs/<rig-id>/`). The root YAMLs (`roster.yaml`, `genotypes.yaml`, `ages.yaml`,
`sexes.yaml`, `fly_numbers.yaml`) are the controlled pick-lists for the Run-details
panel while that repo is configured; otherwise the site library
(`configs/metadata/*.yaml`) is used.

Two repos are registered in `js/studio-data-repos.js` and offered in the Studio's
File ▾ → GitHub **Repo** picker:

| Repo | Use | Visibility | Id label |
|---|---|---|---|
| `reiserlab/cshl-2026-course` | CSHL 2026 course showcase (kept as-is) | public | Bench id |
| `reiserlab/arena-experiments` | the lab's day-to-day experiments | private | Rig id |

A fresh browser has **no repo selected** — saves go to local files until someone picks
one. "Other…" accepts any `owner/name`.

## Set up a rig computer (once)

1. Get a token — see `data-repo-token-runbook.md` §B (org members: fine-grained,
resource owner `reiserlab`, only the repo, Contents read/write).
2. Arena Studio → **File ▾** → GitHub **Sign in…** → paste → **YES** to remember on a
dedicated rig computer (NO on a shared laptop).
3. 🛡 advanced mode → click **🔓** in the GitHub block → **Repo** = *Reiser lab
experiments* → **Rig id** = this station's id (pick from the roster list or type a
new one, e.g. `3e229-g6a`; unique per rig) → tick **Commit directly to default
branch** → **🔒**.
4. Add the rig to the repo's `roster.yaml` under `rigs:` (with the controller MAC
from the Studio connect log) and give people `rig_id`s if they own a station.

The label next to the id field follows the repo: "Bench id" for the course repo,
"Rig id" otherwise. The localStorage key is the same (`studio_bench_id`); switching
repos does not reset it — check it when you switch.

## Sharing across the lab

Everyone with access sees every rig's folders. **File ▾ → Promote to shared…** copies
a protocol and its `_patterns/` into `protocols/shared/` (refuses to overwrite a
*different* same-named file). The Pattern Designer's **Save to Repo → library**
writes `patterns/`. Nothing else is needed.

## Adding another data repo

1. Create + seed it: `scripts/seed-data-repo.sh --repo owner/name --apply`
(dry run without `--apply`; private by default; idempotent).
2. Add an entry to `DATA_REPOS` in `js/studio-data-repos.js` (label, id label,
placeholder, shared label, visibility). Tests: `tests/test-studio-data-repos.js`.
3. Never enable branch protection — the Studio commits straight to the default branch.

## Migrating rig folders between repos

Planned for the week of 2026-09-08 (`scripts/migrate-data-repo-dirs.sh`, see the
plan): copy `protocols/<id>/` + `runlogs/<id>/` from the course repo into the lab
repo under a new rig id without rewriting file contents; record the mapping in
`MIGRATION.md`. Removing from the public repo hides the files but not their git
history.
2 changes: 1 addition & 1 deletion docs/development/studio-github-save-proposal.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Studio → GitHub save: lean proposal (for review)

**Status: COURSE SCOPE SHIPPED (2026-07-03, Session 2 — Arena Studio v0.5).**
**Status: COURSE SCOPE SHIPPED (2026-07-03, Session 2 — Arena Studio v0.5). PER-LAB SCOPE SHIPPED 2026-09-04 (Studio v0.72) as a repo *registry* (`js/studio-data-repos.js`, private `reiserlab/arena-experiments`, per-person tokens) rather than a template repo — see `data-repo-setup.md`.**
The course-pipeline slice of this proposal is implemented; the generic
per-user/template-repo phases below remain the unbuilt follow-on. Shipped:
repo owner/name + bench-id settings + "commit directly to default branch"
Expand Down
Loading