Skip to content

Add opt-in lossless TIFF compression for materialized outputs #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
trissim merged 6 commits into from
Sep 24, 2026
Merged

Add opt-in lossless TIFF compression for materialized outputs #129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bb96cdc
Expose lossless TIFF artifact compression in pipeline config
trissim Sep 24, 2026
550cc85
Document TIFF output configuration and refresh reviewed authorities
trissim Sep 24, 2026
52b6cdb
Format TIFF materialization changes for CI
trissim Sep 24, 2026
cd86043
docs: explain per-callable 8-bit output versus TIFF compression
trissim Sep 24, 2026
f2310eb
Require published PolyStore TIFF compression release
trissim Sep 24, 2026
612f593
Keep gallery type-only execution DTO out of website runtime
trissim Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions docs/audits/architecture_a_m.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@
},
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Declares AnalysisConsolidationConfig and its standalone consolidation policy."
},
{
Expand All @@ -178,7 +178,7 @@
},
{
"path": "tests/unit/test_function_artifact_materialization.py",
"sha256": "4ae063cac0f93a03b5b4aab7304c8d7e6bcfc04ec74d107ea14c94f76be755ca",
"sha256": "ba4f8f5f165ab87c3000d6d371a6f25cf4ec6a5d39b5047abaa392c827fe9ae8",
"role": "Proves consolidation consumes runtime CSV values, uses the compiled backend destination, ignores non-table artifacts, and propagates failures."
},
{
Expand All @@ -188,7 +188,7 @@
},
{
"path": "openhcs/processing/materialization/core.py",
"sha256": "1372625f3d7dc04cf6822a1887871dfa9a149b2313df7d7acefd452a6ad0f932",
"sha256": "1f5a89a1eb50766ef093df3fece16d6bb03422b92d062fe8ae6471656b1ebd45",
"role": "Owns writer runtime-path selection before rendering and the shared ROI target request."
}
],
Expand Down Expand Up @@ -675,7 +675,7 @@
},
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Declares OpenHCS domain configuration that consumes ObjectState mechanics."
}
],
Expand Down Expand Up @@ -841,7 +841,7 @@
},
{
"path": "openhcs/core/context/processing_context.py",
"sha256": "9384fcc4d5ed7ea8607257ec448958f85a88a5caa6ae06f22142cc51b58b479e",
"sha256": "c7b79f50661c528261ce8c577fd67c753c84a5e499aa87cd5e540b53417351a2",
"role": "Owns the compiled multiprocessing-axis identity for one processing context."
},
{
Expand Down Expand Up @@ -912,7 +912,7 @@
},
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Declares the OpenHCS global, pipeline, step, and standalone configuration hierarchy."
}
],
Expand Down Expand Up @@ -1142,7 +1142,7 @@
"authority": [
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Declares standalone ExperimentalAnalysisConfig options and all directory-workflow filenames."
},
{
Expand Down Expand Up @@ -1198,7 +1198,7 @@
"authority": [
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Pins the eight first-party package constraints and documents their installed integration surface."
},
{
Expand Down Expand Up @@ -1805,7 +1805,7 @@
},
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Declares the WellFilterConfig family preview policy and WellFilterMode member operators."
},
{
Expand Down Expand Up @@ -1955,7 +1955,7 @@
},
{
"path": "tests/unit/test_config_document.py",
"sha256": "c6b291080398669b8449cb6ddaa78350f6c5b09f4f0cc57ebe92ac5f32460915",
"sha256": "0ee12db4633ab39b34f630e4ee1b0c48b40d8226f91fddc504bb3311b9612cda",
"role": "Proves explicit, defaulted, and invalid typed field projections from canonical configuration source."
},
{
Expand Down
34 changes: 17 additions & 17 deletions docs/audits/architecture_n_z_development.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@
},
{
"path": "openhcs/core/orchestrator/orchestrator.py",
"sha256": "ca733be328ed5f1f033ee521fbf9bf0b088791bd19542f0071ddb01b4f2d3d9c",
"sha256": "843e994d7b16592f43a15ce36c33a51ead534cf22df792838ce2b800621114b7",
"role": "PipelineOrchestrator configuration ownership, resolution, compiler setup, and executor lifecycle."
},
{
Expand Down Expand Up @@ -250,7 +250,7 @@
},
{
"path": "openhcs/core/context/processing_context.py",
"sha256": "9384fcc4d5ed7ea8607257ec448958f85a88a5caa6ae06f22142cc51b58b479e",
"sha256": "c7b79f50661c528261ce8c577fd67c753c84a5e499aa87cd5e540b53417351a2",
"role": "Owns explicit release of the context-local image-stack cache without discarding observations."
},
{
Expand Down Expand Up @@ -290,7 +290,7 @@
"authority": [
{
"path": "openhcs/core/orchestrator/orchestrator.py",
"sha256": "ca733be328ed5f1f033ee521fbf9bf0b088791bd19542f0071ddb01b4f2d3d9c",
"sha256": "843e994d7b16592f43a15ce36c33a51ead534cf22df792838ce2b800621114b7",
"role": "PipelineOrchestrator configuration ownership, resolution, compiler setup, and executor lifecycle."
},
{
Expand Down Expand Up @@ -512,7 +512,7 @@
"authority": [
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Pipeline, global, step, processing, and source configuration declarations."
},
{
Expand Down Expand Up @@ -572,7 +572,7 @@
},
{
"path": "openhcs/core/context/processing_context.py",
"sha256": "9384fcc4d5ed7ea8607257ec448958f85a88a5caa6ae06f22142cc51b58b479e",
"sha256": "c7b79f50661c528261ce8c577fd67c753c84a5e499aa87cd5e540b53417351a2",
"role": "Owns the compiled multiprocessing-axis identity carried by each runtime context."
},
{
Expand Down Expand Up @@ -822,7 +822,7 @@
"authority": [
{
"path": "openhcs/core/config.py",
"sha256": "628b338097814f3d0297c28da8d016252214ff2253aee2e4946bad09f5890ea9",
"sha256": "ef81b50ee22b8da69513fd50d6e45a929117fa4bfaa62a20ef08a4c02531c043",
"role": "Pipeline, global, step, processing, and source configuration declarations."
},
{
Expand Down Expand Up @@ -1020,7 +1020,7 @@
},
{
"path": "openhcs/processing/materialization/core.py",
"sha256": "1372625f3d7dc04cf6822a1887871dfa9a149b2313df7d7acefd452a6ad0f932",
"sha256": "1f5a89a1eb50766ef093df3fece16d6bb03422b92d062fe8ae6471656b1ebd45",
"role": "Nominal artifact materialisation and ROI projection strategies."
},
{
Expand Down Expand Up @@ -1408,7 +1408,7 @@
},
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
}
],
Expand Down Expand Up @@ -1646,7 +1646,7 @@
},
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
},
{
Expand Down Expand Up @@ -1719,7 +1719,7 @@
"authority": [
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package-discovery, and installed entry-point authority."
},
{
Expand Down Expand Up @@ -2612,7 +2612,7 @@
},
{
"path": "scripts/gallery_catalog.py",
"sha256": "e6516488d7024771c35faf1a25d3ec3abc3b2f21830a8b318c9665d9469c5c0a",
"sha256": "bda42a36c1c510af7ac587dc2642548758d28c4a0061d581ebe38a5be4fb3743",
"role": "Owns nominal gallery scenario identity, publication targets, representative images, copy, accessibility text, dimensions, contextual and stable-window capture targets, and derivative contracts."
},
{
Expand Down Expand Up @@ -2659,7 +2659,7 @@
},
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
},
{
Expand Down Expand Up @@ -2763,7 +2763,7 @@
},
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
},
{
Expand Down Expand Up @@ -2811,7 +2811,7 @@
"authority": [
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
},
{
Expand Down Expand Up @@ -2869,7 +2869,7 @@
},
{
"path": "scripts/gallery_catalog.py",
"sha256": "e6516488d7024771c35faf1a25d3ec3abc3b2f21830a8b318c9665d9469c5c0a",
"sha256": "bda42a36c1c510af7ac587dc2642548758d28c4a0061d581ebe38a5be4fb3743",
"role": "Owns the nominal gallery scenario lattice, stable-window and contextual capture targets, publication targets, derivative inventory, source evidence, card rendering, and generated release record."
},
{
Expand Down Expand Up @@ -3521,7 +3521,7 @@
"authority": [
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
},
{
Expand Down Expand Up @@ -3712,7 +3712,7 @@
},
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Current Python support, dependency, package discovery, and entry-point authority."
}
],
Expand Down
55 changes: 55 additions & 0 deletions docs/audits/pr129_tiff_compression_review_20260924.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# PR #129 TIFF compression authority-delta review

Date: 2026-09-24

The opt-in disk-TIFF writer change invalidated 34 recorded authority edges
across 29 active documentation pages. I re-read each complete dependent page
against the changed OpenHCS declarations, runtime path, and tests, and the
PolyStore codec declaration. The generated configuration reference remains the
authority for exact fields, accepted values, and defaults; it is not duplicated
as a hand-maintained field table.

Two pages needed prose changes. `concepts/storage_system.rst` now explains the
separation of disk TIFF compression from Zarr compression and semantic
materialization. `guide_for_biologists/configuration_reference.rst` now names
the `TiffConfig` family and its scope in the practitioner mental model. Both
remain explanations rather than becoming exhaustive field references. The
other 27 pages retain their existing claims and Diátaxis purposes: the new
config is resolved from the existing pipeline declaration, carried in the
compiled processing context, and used only on disk TIFF writes, so their
descriptions of inheritance, runtime ownership, ROI, MCP schema projection,
analysis, and viewer behavior remain accurate.

This is an editorial receipt, not product authority. The authoritative files
are the declarations, implementations, and tests recorded in the JSON audit
entries. The changed page and authority digests were refreshed only after this
comparison. Documentation validation passed for 155 audited sources, the full
Sphinx HTML build succeeded with warnings treated as errors, and 15 focused
documentation, MCP, and TIFF-output tests passed. At this initial review,
the separate published-dependency readiness gate remained open because the
PolyStore change had not yet been released.

The first CI pass found three Black-only formatting differences. After running
the pinned formatter, the changed materialization owner files differ only in
line wrapping. I rechecked the complete analysis-consolidation, ROI, and storage
pages against those unchanged semantics before refreshing their three affected
authority digests. The reformatted test file is not a recorded audit authority.

## Published-dependency follow-up

PolyStore PR #11 was merged and released as v0.2.19. Its publish workflow
passed, and both the wheel and source distribution are present on PyPI.
OpenHCS now requires `polystore>=0.2.19,<0.3` and pins the PolyStore
submodule to release commit `0efe67fdd14985bf90cee6e0f0c4735d41d265d1`.
The 17 documentation audit entries that depend on `pyproject.toml` were
rechecked against the new dependency floor and their authority digests
refreshed. No affected explanatory page asserted the prior floor.

The subsequent website CI run exposed an eager import in the gallery catalog:
its source-capture request used `ExecutionConnectionSpec` only as a postponed
type annotation, but importing that DTO loaded the scientific runtime into
the deliberately minimal website environment. The import is now restricted to
type checking. The gallery's runtime declarations, capture behavior, and
published projections are unchanged. The 13 audited pages referencing the
gallery catalog were checked for claims about those surfaces, and their
authority digests were refreshed.
2 changes: 1 addition & 1 deletion docs/audits/project_surface.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"authority": [
{
"path": "pyproject.toml",
"sha256": "61621859d0f780d2d3e004782f242ef923d6cd1e83bb3bbc9761197ba33a4165",
"sha256": "6217b5f029796edcd83aa47fb8632d37aa7552344755c578fa4f181a6b23282d",
"role": "Declares the published README, Python range, package dependencies, optional surfaces, console commands, and project links."
},
{
Expand Down
Loading
Loading