Repository navigation
Conversation
…cided by position
A pipeline boundary's contract made executable and visible: fields maps a record
key to a registered item type name ('*' = present with any type); a violation
raises ContractError naming the boundary, record ordinal, offending entry and
what the record does carry. One class serves both boundary roles — first op in a
chain states what the host must feed, last what the pipeline delivers (pinned by
tests/test_contract.py::TestInputAndOutputPositions). group='contract', per-module
entry point, README section, architecture record §15, CHANGELOG. TASKS: filed the
to_stream chain-spelled-linear-doc paper cut.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lightning's ModelCheckpoint falls back to Trainer.default_root_dir -- the process CWD -- when no tracker supplies save_dir, writing a flat checkpoints/ into the project root. Add the ignore entries so that output can never become committable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- FilesSource (paths as records) + ReadImage (decode on the spine) — the decomposed file pipeline: the source delivers paths, an OP turns a file into a record; unreadable files pass through with their row, never fatal. - Normalize: the per-channel standardization node ((x/max - mean)/std, albumentations convention, ImageNet defaults) a drawn pipeline needs between a conversion and a model. - ModelPredict: predict_batch (one forward per chunk, per-kind debatch; __call__ is a batch of one), selectable device (auto > cuda > mps > cpu, absent devices refused naming what the machine has), classification confidence stamped beside the label, batch_size knob. New widget params are APPEND-ONLY: canvas widget values are positional, and a mid-signature insert shifted every later widget on stale-editor conversions (pinned). - RandomNumber (value producer, per-call draws) + PutField (stamp a wired value under a key) replace the fused RandomField. - Projection narrowing on HuggingFaceSource (select_columns before iterating) and class_names as a declared @output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ModelPredict: a 'frame' knob scales detection boxes back to a named entry's raster; solidify() is a BUILD hook, never a model swap; detection batches are LISTS of tensors (the torchvision convention) and move to the device element-wise - ops.image: ConvertToBoxes/ConvertFromBoxes (canonical pixel-xyxy <-> the source's own layout), box_iou/match_boxes/size_bucket backing the review metrics - Boxes carries 'classes' — the vocabulary travels with the encoded labels - HuggingFaceSource.class_names reads a DETECTION target's nested vocabulary (objects.category ClassLabel) one level down Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mpanions A name-level filter (no file reads) keeps a paired format's companion half (SigMF's .sigmf-data) out of the listing, so len()/ids stay honest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed files FileFormat protocol + file_formats() over the recordstream.formats entry-point group (domain packages register; per-entry failure tolerance; cached scan) with the prefix-tolerant sibling()/bare_name() pairing helpers, the ReadFile op (consumed companion -> dropped, matched -> decoded, unknown -> refused naming the installed formats), and a pair-aware FilesSource: the listing excludes exactly what some format consumes() — a name+stat rule a static glob cannot express (a lone self-describing half must stay listed). No formats ship in the engine; with none installed every listing is the plain file list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stays the rule Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ordering-only churn from the aisland scaffold's internal-dep closure change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Regenerated from the aisland templates after the stage was removed there. The Jenkinsfile, Jenkinsfile.local and .github/workflows/ci.yml no longer carry the npm ci / test / build steps guarded on frontend/package.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…op chain by location
Ops may declare `consumes` / `produces` ({record key: item type}, RecordContract's own
vocabulary), `reports` (their key in a report; "" = a transform), `flags` (the booleans
they may raise) and an instance `requires` (one flag, truthy). `check_chain(ops,
provided=…, where=…)` walks a converted chain once and refuses an unmet consumes, a flag
no earlier node declares (naming the node that produces it LATER, and where to move it),
a flag declared twice and a report name declared twice — each message names the op.
Opt-in: a chain declaring none of these passes untouched, so every existing graph loads
as before. `flag_producers(ops)` is the flag → producer map a fork converter needs.
Docs: README section, docs/architecture.md record #16.
Tests: tests/test_chain_contract.py.
…quires` back on export — the document form stays the one shown
The list form serves "someone handed me files"; the folder form serves "review this directory tree". `pattern` is an ordinary Path.glob relative to `root` and IS the scope control — "*/*" every immediate subdirectory, "**/*" everything below, "one_dir/*" a single named subdirectory — so no second filter knob is needed beside it. Three listing rules, each from a real failure: directories the glob matches are skipped, dotfiles are never listed (globbing a captures parent listed exactly one file, .DS_Store, which is not data and fails to decode), and a pattern matching nothing raises naming the folder and the pattern rather than handing the chain an empty listing that surfaces as a broken pipeline three ops later. Passing `files` AND `root` is refused — two answers to "which files" is a config bug, not a merge — and the refusal repeats at READ time, because a consuming workspace may assign `files` on an already-built source (filling it from a drag-and-drop) where a constructor check cannot see it; silently preferring one would make the drop look like it did nothing. The scan is lazy, in a property rather than __init__, so zero-arg construction and the no-work-in-constructor mandate hold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urce forwards a key-restricted walk - formats.py / ops/formats.py: the registry's optional `scan` / `scan_file` capability beside `matches` / `consumes` / `read` — a format answers a projection from its metadata without decoding the payload; a chain that reads files offers a cheaper variant of itself (`for_projection`) and the stream verifies that answer per record (`formats.answers`) - projection.py + the view sources (concat, range, split, files): every view source forwards a key-restricted walk (`project_indices`), so `project` takes a source's cheap path only through the object it was handed - sources/files.py: the served listing is resolved once per file list (`FilesSource._served`), never per access - AGENTS mandates, architecture records 17–19, docs/sources.md sections and tests for each Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…talls nothing Regenerated by `aisland jenkins scaffold`: the Initialize stage no longer re-installs the project and its internal dependencies (a re-install took 1430 s and held the venv's lock). It now only verifies the project is editable-installed from this checkout and points at `aisland setup` when it is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e release checks it against the tag
The version is written once, in pyproject.toml; __init__.py reads it back with importlib.metadata.version() and falls back to 0.0.0.dev0 in an uninstalled source tree. tests/test_version.py pins __version__ == the installed distribution's version. The release workflow's wheel smoke test now asserts recordstream.__version__ == version("recordstream") == the git tag, so a missing or stale version stops a release before PyPI.
Regenerated by aisland after confluid became a local source (config.toml): CI installs confluid from Gearlux/confluid@main with --no-deps before .[dev].
…nerated CI Regenerated by aisland: CI installs a released internal dependency from PyPI through .[dev] and an unreleased one from GitHub main, whatever the workspace source setting (aisland ruling 2026-09-26).
numpy's ndarray.__reduce__ carries only the array; unpickling rebuilds it without __new__ and __array_finalize__ sees no source object, so every declared _item_attrs value fell back to its class default. Records cross spawn workers by pickle (Stream.parallel, FlowGraph.parallel, DataLoader workers), so a CHW Image arrived as HWC and a dB spectrogram as scaling="none" - silently, in both directions. NDArrayItem.__reduce__/__setstate__ now append and restore the declared attrs, so every array item type (including domain ones) inherits the fix. Storage is unaffected: it serializes through the io codec, never pickle. - tests: TestPickle round-trips every array item type at every pickle protocol, a non-contiguous view, and both spawn routes in both directions - docs: record-model, architecture section 1 amendment, AGENTS mandate, CHANGELOG - TASKS: track the related deepcopy attribute-sharing issue
… prep - GraphContract (ops/contract.py): what one graph must deliver — its outputs and what each record of a delivered stream carries — declared as data, with ONE check() the editor and the workspace both call, so a refused graph reads the same in both. Pins: tests/test_graph_contract.py, tests/test_contract.py. - ToTensor converts and nothing else: an array becomes a CHW tensor of the same element type; `normalize` and `mode` are removed (no alias). The value changes are ops of their own, placed before it: Scale (range -> range, a blank source bound is the integer TYPE's range, a float's is refused), ToType (closed ElementType, refuses complex -> real and out-of-range values) and ConvertMode (channel layout through PIL). One rule picks their entries: a blank field means every Image, never a Mask. Pins: tests/test_scale_and_to_type.py, tests/test_convert_mode.py, tests/test_typed_target_ops.py. - Release prep: version 0.1.0a2, its CHANGELOG section, the release workflow's tag example. - Docs: architecture.md §20, image.md, augmentation.md, graph.md, README, AGENTS.md; TASKS.md drops the ToTensor normalize-heuristic item it fixes.
An op that reads named record entries and writes named entries now declares
three kinds of slot and writes compute():
class BackgroundLevel(Algorithm):
percentile: float = Param(default=25.0, doc="...")
image: Image = Input(doc="...")
background: float = Output(doc="...")
It never sees a record. The base class reads each input from the entry of the
same name, checks registered item types, runs compute() on a shallow copy (the
configured instance never holds a record's values) and writes every output back.
`keys` names other entries; Output(replaces="an_input") writes back where that
input was read. Inputs are found by NAME only, so check_chain knows before a run
what a pipeline needs.
Derived from the declarations, never written twice: the keyword-only constructor
(real __signature__, annotations and an Args: docstring, so to_pydantic,
parse_param_docs, confluid validation and dump see an ordinary constructor), one
read-only confluid @output property per output (bind: step.name works), the
chain checker's consumes / produces, and algorithm_spec(). The slots are field
specifiers under dataclass_transform, so mypy checks every constructor call and
refuses an input passed as a setting.
Docs: docs/algorithm.md (new), architecture.md §21, record-model.md, README,
AGENTS.md, CHANGELOG. Pins: tests/test_algorithm.py (50).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second public pre-release. Merging this PR, then pushing the tag
v0.1.0a2, publishesrecordstream 0.1.0a2to PyPI throughrelease.yml.Key features
Param), what it reads (Input) and what it computes (Output), and writescompute(). The record handling, the constructor, the settings schema, the output sockets and the chain contract are derived from those declarations. Guide:docs/algorithm.md.RecordContractasserts what records carry at a boundary;check_chainrefuses a mis-wired op chain before the first record;GraphContractstates what a whole graph must deliver, with onecheck().ToTensorconverts and nothing else. Value changes are ops of their own:Scale(range to range),ToType(element type) andConvertMode(channel layout).ToTensor'snormalizeandmodeare removed.FilesSourcelists explicit files or a folder (root+pattern); a format may answer from its metadata alone, and view sources forward a key-restricted walk.Image's layout or a spectrogram's scaling reaches spawn workers intact.recordstream.__version__reads the installed metadata, and the release workflow checks it against the tag.Detailed changes
Features
816ad9bfeat: Algorithm — declare params, inputs and outputs; the op is derivede0331d2feat: GraphContract, value ops split out of ToTensor, 0.1.0a2 release prepbdd10dafeat: expose recordstream.version from the installed metadata; the release checks it against the tag334d22efeat: a file format may answer from its metadata alone, and a view source forwards a key-restricted walke8346bcfeat: FilesSource lists a FOLDER — root + pattern glob2277905feat(ops): per-node chain contract — check_chain refuses a mis-wired op chain by locationf417f67feat: the file-format registry — extension-decided decoding for dropped filesd9ae04afeat(sources): FilesSource grows an exclude glob for paired-format companions057144dfeat: detection inference and review primitives0cef6befeat: file-list pipeline, prediction batching, and composable record ops168a0dffeat: RecordContract — the pass-through interface op; input/output decided by positionFixes
4835065fix(items): keep NDArrayItem attributes through picklingDocs
b9823dfdocs(readme): a visual editor may draw a gate as a wire and writerequiresback on export — the document form stays the one showne7bc281docs(formats): consumes() may peek a sidecar's metadata — name-level stays the ruleCI and chores
ec74da4chore(ci): Node 24 GitHub Actions in the generated CI and the release workflowd04d3d9chore(ci): install released confluid and liquifai from PyPI in the generated CI430cbd4chore(ci): pre-install confluid from GitHub main in the generated CI5e97dbfchore(ci): Jenkinsfile.local tests against the workspace venv and installs nothing362a454chore(ci): install liquifai from git before the editable install (aisland regenerate)06d10ebchore: drop the Verify Frontend stage from the generated CI991a0cachore: regenerate Jenkinsfile.local (internal-dep pre-install order)6c8fb25chore: ignore stray checkpoint output at the launch directoryQuality and coverage
Run locally on
816ad9b:isort --check-only,black --checkandflake8: clean.mypyfrom the workspace root: no issues in 126 source files.pytest: 1241 passed. New pins includetests/test_algorithm.py(50),tests/test_graph_contract.py,tests/test_scale_and_to_type.pyandtests/test_convert_mode.py; the packaging pins check version, classifier and changelog.mypy .that CI runs could not be reproduced exactly here, because the workspace's confluid, liquifai and loggair are editable installs mypy cannot follow. It reported only those missing imports and no other error. This PR's Quality Gates job is the authoritative run.🤖 Generated with Claude Code