From eed86266b3275dc2e63fb159b96d228f768c36bf Mon Sep 17 00:00:00 2001 From: Lance Date: Sun, 20 Sep 2026 17:05:35 -0700 Subject: [PATCH 1/7] add glossary template and block/chunk definitions --- doc/glossary/index.rst | 72 ++++++++++++++++++++++++++++++++++++++++++ doc/python-blosc2.rst | 1 + 2 files changed, 73 insertions(+) create mode 100644 doc/glossary/index.rst diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst new file mode 100644 index 000000000..f7b558ee7 --- /dev/null +++ b/doc/glossary/index.rst @@ -0,0 +1,72 @@ +Glossary +======== + +.. glossary:: + + block + The unit of decompression, stored within a chunk. Blocks are sized to + fit CPU caches, typically 32-512KB. + + BUCKET + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + chunk + The unit of storage and compression, stored within a SChunk. Chunks are + sized to fit disk/network I/O, typically 1-64MB. + + clevel + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + codec + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + compression parameters + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + cparams + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + CTable + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + filters + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + frame + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + FULL + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + NDArray + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + OPSI + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + PARTIAL + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + SChunk + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + subblock + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. + + SUMMARY + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. diff --git a/doc/python-blosc2.rst b/doc/python-blosc2.rst index e932d3a7c..b817b6d94 100644 --- a/doc/python-blosc2.rst +++ b/doc/python-blosc2.rst @@ -224,5 +224,6 @@ Tutorials Guides API Reference + Glossary Development Release Notes From 911de8e02826c9225fdf8620b82c8c3731f41359 Mon Sep 17 00:00:00 2001 From: Lance Date: Thu, 24 Sep 2026 21:14:56 -0700 Subject: [PATCH 2/7] add more container definitions to glossary --- doc/glossary/index.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index f7b558ee7..5c2d5d396 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -32,8 +32,9 @@ Glossary tempor incididunt ut labore et dolore magna aliqua. CTable - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + A columnar table for structured data. Columns are stored, compressed, and + queried independently, with SUMMARY indexes available by default. Use with + structured data that benefits from compression, speed, and persistence. filters Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod @@ -48,8 +49,10 @@ Glossary tempor incididunt ut labore et dolore magna aliqua. NDArray - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + A compressed, chunked multidimensional data array. Supports NumPy-like + slicing and broadcasting, and out-of-core computation. Backed by an SChunk. + Use for array workloads, especially when too large to fit in memory + uncompressed. OPSI Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod @@ -60,12 +63,12 @@ Glossary tempor incididunt ut labore et dolore magna aliqua. SChunk - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + The foundational container for managing a sequence of individual, + compressed chunks. NDArrays and CTable columns are built on top of SChunk. + Use when you want to directly manipulate raw compresesd data and metadata. subblock - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + An indexing segment within a block. One eighth the length of a block. SUMMARY Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod From 75ce7ec786f5dbf44352288718bf6c8adc5e0272 Mon Sep 17 00:00:00 2001 From: Lance Date: Mon, 28 Sep 2026 23:55:10 -0700 Subject: [PATCH 3/7] add API references to doc glossary --- doc/glossary/index.rst | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index 5c2d5d396..21742400f 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -19,24 +19,20 @@ Glossary Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. - codec + codec (:class:`API `) Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. - compression parameters + compression parameters (:class:`API `) Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. - cparams - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. - - CTable + CTable (:ref:`API `) A columnar table for structured data. Columns are stored, compressed, and queried independently, with SUMMARY indexes available by default. Use with structured data that benefits from compression, speed, and persistence. - filters + filters (:class:`API `) Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. @@ -48,7 +44,7 @@ Glossary Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. - NDArray + NDArray (:ref:`API `) A compressed, chunked multidimensional data array. Supports NumPy-like slicing and broadcasting, and out-of-core computation. Backed by an SChunk. Use for array workloads, especially when too large to fit in memory @@ -62,7 +58,7 @@ Glossary Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. - SChunk + SChunk (:ref:`API `) The foundational container for managing a sequence of individual, compressed chunks. NDArrays and CTable columns are built on top of SChunk. Use when you want to directly manipulate raw compresesd data and metadata. From aa2cec6c09d654ac6c881727dcc088c35d95ac3f Mon Sep 17 00:00:00 2001 From: Lance Date: Mon, 28 Sep 2026 23:56:55 -0700 Subject: [PATCH 4/7] add lazy array and frame defns to doc glossary --- doc/glossary/index.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index 21742400f..431019c78 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -37,13 +37,21 @@ Glossary tempor incididunt ut labore et dolore magna aliqua. frame - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + A serialized format for storing chunks along with a header and trailer for + metadata. Frames may be contiguous (CFrame) or sparse (SFrame). FULL Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. + LazyArray (:ref:`API `) + API to store an expression or function and delay computation until the + value is explicitly requested. Executes and stores results chunk-by-chunk. + + LaxyExpr (:ref:`API `) + Object that stores an expression consisting of at least one NDArray + object. Follows the LazyArray API for storage and deferred computation. + NDArray (:ref:`API `) A compressed, chunked multidimensional data array. Supports NumPy-like slicing and broadcasting, and out-of-core computation. Backed by an SChunk. From b51031a415038cdee426352f4730b770e6e724cf Mon Sep 17 00:00:00 2001 From: kunsingh Date: Wed, 30 Sep 2026 16:48:21 -0700 Subject: [PATCH 5/7] docs: divided glossary into readable section and wrote compression and index sections --- doc/glossary/index.rst | 103 ++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 48 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index 431019c78..b695b2c0a 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -1,49 +1,22 @@ Glossary ======== -.. glossary:: - - block - The unit of decompression, stored within a chunk. Blocks are sized to - fit CPU caches, typically 32-512KB. +Data containers +--------------- - BUCKET - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. - - chunk - The unit of storage and compression, stored within a SChunk. Chunks are - sized to fit disk/network I/O, typically 1-64MB. - - clevel - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. - - codec (:class:`API `) - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. +.. glossary:: - compression parameters (:class:`API `) - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + NDArray (:ref:`API `) + A compressed, chunked multidimensional data array. Supports NumPy-like + slicing and broadcasting, and out-of-core computation. Backed by an SChunk. + Use for array workloads, especially when too large to fit in memory + uncompressed. CTable (:ref:`API `) A columnar table for structured data. Columns are stored, compressed, and queried independently, with SUMMARY indexes available by default. Use with structured data that benefits from compression, speed, and persistence. - filters (:class:`API `) - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. - - frame - A serialized format for storing chunks along with a header and trailer for - metadata. Frames may be contiguous (CFrame) or sparse (SFrame). - - FULL - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. - LazyArray (:ref:`API `) API to store an expression or function and delay computation until the value is explicitly requested. Executes and stores results chunk-by-chunk. @@ -52,28 +25,62 @@ Glossary Object that stores an expression consisting of at least one NDArray object. Follows the LazyArray API for storage and deferred computation. - NDArray (:ref:`API `) - A compressed, chunked multidimensional data array. Supports NumPy-like - slicing and broadcasting, and out-of-core computation. Backed by an SChunk. - Use for array workloads, especially when too large to fit in memory - uncompressed. +Compression +----------- - OPSI - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. +.. glossary:: - PARTIAL - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + codec (:class:`API `) + A compressor and matching decompressor that implements a particular + compression method. Codec choice affects compression/decompression speed and size of the compressed data. + + clevel + An integer from 0 (no compression) to 9 (the highest compression level) that controls compression effort. Higher levels generally take longer to compress and may produce smaller results, depending on the codec and data. + + filters (:class:`API `) + Transformations applied to data before the codec compresses it in order to expose patterns that can improve compression. Some filters are reversible (such as SHUFFLE), while others deliberately discard precision (such as TRUNC_PREC). + +Low-level data structures +------------------------- + +.. glossary:: SChunk (:ref:`API `) The foundational container for managing a sequence of individual, compressed chunks. NDArrays and CTable columns are built on top of SChunk. Use when you want to directly manipulate raw compresesd data and metadata. + frame + A serialized format for storing chunks along with a header and trailer for + metadata. Frames may be contiguous (CFrame) or sparse (SFrame). + + chunk + The unit of storage and compression, stored within a SChunk. Chunks are + sized to fit disk/network I/O, typically 1-64MB. + + block + The unit of decompression, stored within a chunk. Blocks are sized to + fit CPU caches, typically 32-512KB. + subblock An indexing segment within a block. One eighth the length of a block. +Indexes +------- + +.. glossary:: + SUMMARY - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod - tempor incididunt ut labore et dolore magna aliqua. + Lightweight index that stores per-segment minimum and maximum values to skip segments that cannot match a query. + + BUCKET + Stores values sorted within each chunk and groups their positions into buckets to locate possible query matches. + + PARTIAL + Stores values sorted separately within each chunk, along with their exact positions, to find matches. + + FULL + Stores values sorted together across all chunks, along with their exact positions, to find matches and speed up sorting. + + OPSI + Uses repeated ordering cycles to improve filtering and provide exact matching positions for checking conditions on other columns, but is not intended to converge to a globally sorted index. From 6beecde52463b1f7ac43fd7c35eadc8a0b70a1d8 Mon Sep 17 00:00:00 2001 From: Lance Date: Thu, 1 Oct 2026 19:22:59 -0700 Subject: [PATCH 6/7] minor glossary edits + index definition --- doc/glossary/index.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index b695b2c0a..44d662ded 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -14,14 +14,14 @@ Data containers CTable (:ref:`API `) A columnar table for structured data. Columns are stored, compressed, and - queried independently, with SUMMARY indexes available by default. Use with + queried independently, with SUMMARY indexes available when eligible. Use with structured data that benefits from compression, speed, and persistence. LazyArray (:ref:`API `) API to store an expression or function and delay computation until the value is explicitly requested. Executes and stores results chunk-by-chunk. - LaxyExpr (:ref:`API `) + LazyExpr (:ref:`API `) Object that stores an expression consisting of at least one NDArray object. Follows the LazyArray API for storage and deferred computation. @@ -48,7 +48,7 @@ Low-level data structures SChunk (:ref:`API `) The foundational container for managing a sequence of individual, compressed chunks. NDArrays and CTable columns are built on top of SChunk. - Use when you want to directly manipulate raw compresesd data and metadata. + Use when you want to directly manipulate raw compressed data and metadata. frame A serialized format for storing chunks along with a header and trailer for @@ -70,6 +70,10 @@ Indexes .. glossary:: + index (:ref:`API `) + Auxiliary data attached to an NDArray or CTable to speed up queries. + Allows queries to skip chunks, blocks, or rows of data that do not match. + SUMMARY Lightweight index that stores per-segment minimum and maximum values to skip segments that cannot match a query. From bb615bc30af2c9c1ca1993e99528c4675d4a5930 Mon Sep 17 00:00:00 2001 From: Lance Date: Sat, 3 Oct 2026 19:41:29 -0700 Subject: [PATCH 7/7] add CFrame + SFrame defns in glossary --- doc/glossary/index.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/doc/glossary/index.rst b/doc/glossary/index.rst index 44d662ded..89e742eb9 100644 --- a/doc/glossary/index.rst +++ b/doc/glossary/index.rst @@ -50,10 +50,6 @@ Low-level data structures compressed chunks. NDArrays and CTable columns are built on top of SChunk. Use when you want to directly manipulate raw compressed data and metadata. - frame - A serialized format for storing chunks along with a header and trailer for - metadata. Frames may be contiguous (CFrame) or sparse (SFrame). - chunk The unit of storage and compression, stored within a SChunk. Chunks are sized to fit disk/network I/O, typically 1-64MB. @@ -65,6 +61,18 @@ Low-level data structures subblock An indexing segment within a block. One eighth the length of a block. + frame + A serialized format for storing chunks along with metadata. Frames may + be contiguous (CFrame) or sparse (SFrame). + + CFrame + Frame format for storing chunks contiguously either in-memory or on-disk. + Short for contiguous frame. See `CFrame format `_. + + SFrame + Frame format for storing chunks non-contiguously on-disk. Short for sparse + frame. See `SFrame format `_. + Indexes -------