Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,9 @@ cpp/third_party/zlib-1.3.1
build/*
cpp/third_party/zlib-1.3.1/treebuild.xml
cpp/third_party/zlib-1.3.1/zlib-1.3.1/treebuild.xml

# Claude Code
.claude/settings.local.json
.claude/todos/
.claude/worktrees/
.claude/scheduled_tasks.json
67 changes: 67 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Apache TsFile is a columnar storage file format designed for time series data. The project provides implementations in three languages, each in its own top-level directory. The main branch is `develop`.

TsFile uses a hierarchical storage model: **Page → Chunk → ChunkGroup → File**. Data is organized by device, with each device's measurements stored as individual time series. Two write models exist: **aligned** (all measurements share timestamps) and **non-aligned** (independent timestamps per measurement).

## Repository Structure

| Directory | Language | Build System | Details |
|-----------|----------|--------------|---------|
| `java/` | Java | Maven | [java/CLAUDE.md](java/CLAUDE.md) |
| `cpp/` | C++ | CMake | [cpp/CLAUDE.md](cpp/CLAUDE.md) |
| `python/` | Python | setuptools + Cython | [python/CLAUDE.md](python/CLAUDE.md) |

The root `pom.xml` orchestrates all three via Maven profiles:

```bash
./mvnw clean verify -P with-java # Java only
./mvnw clean verify -P with-cpp # C++ only
./mvnw clean verify -P with-python # Python (requires C++ built first)
```

## Cross-Module Dependencies

- **Python depends on C++**: The Python module uses Cython to bind to the C++ shared library (`libtsfile`). C++ must be built before Python (`-P with-python` implies C++ build).
- **Java is independent**: The Java implementation shares no build-time dependency with C++ or Python.
- All three implementations read/write the same TsFile binary format.

## Code Formatting

```bash
./mvnw spotless:apply # Format all languages (Java: Google Java Format, C++: clang-format, Python: Black)
./mvnw spotless:check # Check formatting without modifying
```

## License Header

Every new file must include the Apache License 2.0 header at the top. Use the comment style appropriate for the file type (e.g., `<!-- -->` for Markdown, `/* */` for Java/C++, `#` for Python). See any existing file for the exact wording.

## Git Commit

- Do NOT add `Co-Authored-By` trailer to commit messages.
108 changes: 108 additions & 0 deletions cpp/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# CLAUDE.md — C++ Module

This file provides guidance to Claude Code (claude.ai/code) when working with the C++ module.

## Build Commands

```bash
# Build (Release, default)
bash build.sh

# Build variants
bash build.sh -t=Debug
bash build.sh -t=RelWithDebInfo
bash build.sh -a=ON # Enable AddressSanitizer
bash build.sh -c=ON # Enable code coverage

# Disable optional compression libraries
bash build.sh --disable-snappy --disable-lz4 --disable-lzokay --disable-zlib

# Or use CMake directly
mkdir -p build/Release && cd build/Release
cmake ../.. -DCMAKE_BUILD_TYPE=Release -DBUILD_TEST=ON
make -j$(nproc)

# Via Maven from repo root
./mvnw clean verify -P with-cpp

# Run all tests
./build/Release/lib/TsFile_Test

# Run specific test suite
./build/Release/lib/TsFile_Test --gtest_filter=SnappyCompressorTest.*
```

## Build Options (CMake)

All `ON` by default unless noted:

| Option | Purpose |
|--------|---------|
| `BUILD_TEST` | Compile tests (GTest 1.12.1, auto-downloaded) |
| `ENABLE_ANTLR4` | ANTLR4 parser runtime |
| `ENABLE_SNAPPY` / `ENABLE_LZ4` / `ENABLE_LZOKAY` / `ENABLE_ZLIB` | Compression libraries |
| `ENABLE_THREADS` | Multi-threaded read/write via pthreads |
| `ENABLE_ASAN` | AddressSanitizer (`OFF` by default) |
| `ENABLE_SIMDE` | SIMD Everywhere (`OFF` by default) |

## Source Structure

```
cpp/src/
├── common/ # Core types: Schema, Tablet, DeviceId, TsBlock, allocators, config
├── compress/ # Compression: Snappy, LZ4, LZOKAY, Zlib (factory pattern)
├── encoding/ # Encoding: Plain, TS2Diff, Gorilla, Dictionary, RLE, Zigzag, SPRINTZ
├── file/ # File I/O: TsFileIOReader/Writer, RestorableTsFileIOWriter
├── reader/ # Read path: TsFileReader, QueryExecutor, filters, result sets
├── writer/ # Write path: TsFileWriter, TsFileTableWriter, ChunkWriter, PageWriter
├── parser/ # ANTLR4 path parser (grammars + generated code)
├── cwrapper/ # C language bindings (used by Python module)
└── utils/ # Utilities: error codes, date handling, fault injection
```

## Architecture Notes

- **C++11** standard, targets CMake 3.11+
- Dual data model: **tree-view** (`TsFileTreeWriter/Reader`) and **table-view** (`TsFileTableWriter`, `TableQueryExecutor`)
- Parallel column encoding in table write path, controlled by `ENABLE_THREADS`
- Third-party libraries are bundled under `third_party/` (ANTLR4, Snappy, LZ4, LZOKAY, Zlib, SIMDe)
- `cwrapper/` provides the C API that the Python module binds to via Cython

## Code Style

- **Formatter**: clang-format (Google style), configured in `.clang-format`

## Testing

- **Framework**: Google Test 1.12.1 (auto-downloaded during build, or supply `third_party/googletest-release-1.12.1.zip`)
- Tests in `cpp/test/`, mirroring `src/` structure
- Test discovery via `gtest_discover_tests()`

## License Header

Every new file must include the Apache License 2.0 header at the top. For C/C++ files, use the `/* */` block comment style. See any existing `.h` or `.cc` file for the exact wording.

## Git Commit

- Do NOT add `Co-Authored-By` trailer to commit messages.
101 changes: 101 additions & 0 deletions java/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# CLAUDE.md — Java Module

This file provides guidance to Claude Code (claude.ai/code) when working with the Java module.

## Build Commands

Maven with wrapper (`./mvnw` from repo root). All commands require `-P with-java`.

```bash
# Build (skip tests)
./mvnw clean package -P with-java -DskipTests

# Install to local Maven repo
./mvnw install -P with-java -DskipTests

# Run all tests (unit + integration)
./mvnw clean verify -P with-java

# Run unit tests only
./mvnw test -P with-java

# Run integration tests only
./mvnw verify -P with-java -DskipUTs=true

# Run a single test class (use -pl to target submodule)
./mvnw test -P with-java -pl java/tsfile -Dtest=TsFileWriterTest

# Run a single test method
./mvnw test -P with-java -pl java/tsfile -Dtest=TsFileWriterTest#testWrite

# Format code (requires Java 17+ runtime)
./mvnw spotless:apply

# Check formatting
./mvnw spotless:check
```

## Module Structure

Four submodules under `java/`:

- **common** — Shared types: `TSDataType`, `ColumnCategory`, `Binary`, `BitMap`, constants
- **tsfile** — Core implementation: read/write APIs, encoding, compression, file format
- **examples** — Reference implementations for read/write operations
- **tools** — CLI utilities for TsFile inspection

## Architecture

Key packages in `java/tsfile/src/main/java/org/apache/tsfile/`:

- `write/` — Write path: `TsFileWriter`, `Tablet` (batch API), `TSRecord` (row API)
- `read/` — Read path: `TsFileReader`, `TsFileSequenceReader`, `QueryExpression`
- `encoding/` — Encoding algorithms (RLE, TS_2DIFF, GORILLA, DICTIONARY)
- `compress/` — Compression codecs (Snappy, LZ4, ZSTD, XZ)
- `file/` — File format structures (metadata, headers, footers)
- `compatibility/` — Version compatibility handling
- `parser/` — ANTLR4-generated path parser (grammar in `src/main/antlr4/`)

Code generation: FreeMarker templates in `tsfile/src/main/codegen/` generate type-specific implementations (e.g., per-datatype readers/writers) into `target/generated-sources/`.

## Code Style

- **Formatter**: Spotless with Google Java Format 1.28.0 (Spotless requires Java 17+ to run but the project targets Java 8 bytecode)
- **Checkstyle**: Google style variant in root `checkstyle.xml`, 100 char line limit
- **Import order**: `org.apache.tsfile`, `javax`, `java`, static imports

## Testing Conventions

- **Framework**: JUnit 4
- Unit tests: `*Test.java` — run via `mvn test`
- Integration tests: `*IT.java` — run via `mvn verify`
- Tests run in random order with `reuseForks=false`

## License Header

Every new file must include the Apache License 2.0 header at the top. For Java files, use the `/* */` block comment style. See any existing `.java` file for the exact wording.

## Git Commit

- Do NOT add `Co-Authored-By` trailer to commit messages.
104 changes: 104 additions & 0 deletions python/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# CLAUDE.md — Python Module

This file provides guidance to Claude Code (claude.ai/code) when working with the Python module.

## Prerequisites

The Python module depends on the C++ shared library (`libtsfile`). **Build C++ first:**

```bash
# From repo root
./mvnw clean package -P with-cpp -DskipTests
```

## Build Commands

```bash
# Build everything via Maven (builds C++ then Python)
./mvnw clean verify -P with-python

# Or build directly (after C++ is built)
cd python
python setup.py build_ext --inplace

# Run tests
pytest tests/

# Run a single test
pytest tests/test_write_and_read.py -k "test_name"

# Format code
cd ../ && ./mvnw spotless:apply # Uses Black 26.3.1
```

## Source Structure

```
python/tsfile/
├── __init__.py # Public API exports
├── constants.py # Enums: TSDataType, TSEncoding, Compressor, ColumnCategory
├── schema.py # TimeseriesSchema, DeviceSchema, TableSchema, ColumnSchema
├── tablet.py # Tablet class for batch columnar data
├── field.py / row_record.py # Row-based data structures
├── tag_filter.py # Tag filtering DSL for queries
├── tsfile_table_writer.py # High-level table writer API
├── utils.py # to_dataframe(), dataframe_to_tsfile()
├── exceptions.py # Custom exception types
│
├── tsfile_py_cpp.pyx # Cython core: Python ↔ C++ type conversion
├── tsfile_reader.pyx # Cython reader wrapper
├── tsfile_writer.pyx # Cython writer wrapper
├── tsfile_cpp.pxd # C++ interface declarations (cwrapper)
│
└── dataset/ # High-level multi-file API
├── dataframe.py # TsFileDataFrame (pandas-like interface)
├── timeseries.py # Timeseries, AlignedTimeseries
├── reader.py # Multi-file reader coordination
└── merge.py # K-way merge for overlapping time ranges
```

## Architecture Notes

- **Layered design**: C++ core → Cython bridge (`*.pyx`) → Pure Python API
- `setup.py` copies pre-built C++ headers and shared libraries from `../cpp/target/build/`
- Cython binds to `cwrapper/tsfile_cwrapper.h` from the C++ module
- C error codes are mapped to Python exceptions via `check_error()` in `tsfile_py_cpp.pyx`
- Requires Python 3.9+, numpy >= 2.0, pandas >= 2.0, pyarrow >= 16.0

## Code Style

- **Formatter**: Black 26.3.1 (via Spotless Maven plugin)

## Testing

- **Framework**: pytest
- Tests in `python/tests/`, 19 test files covering write, read, filtering, DataFrame conversion, and Arrow format

## License Header

Every new file must include the Apache License 2.0 header at the top. For Python files, use `#` line comments. For `.pyx`/`.pxd` files, also use `#` comments. See any existing file for the exact wording.

## Git Commit

- Do NOT add `Co-Authored-By` trailer to commit messages.
Loading