diff --git a/.gitignore b/.gitignore index f3b2b0a90..d27277565 100644 --- a/.gitignore +++ b/.gitignore @@ -46,3 +46,9 @@ cpp/third_party/zlib-1.3.1 build/* cpp/third_party/zlib-1.3.1/treebuild.xml cpp/third_party/zlib-1.3.1/zlib-1.3.1/treebuild.xml + +# Claude Code +.claude/settings.local.json +.claude/todos/ +.claude/worktrees/ +.claude/scheduled_tasks.json diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..3076a77ee --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,67 @@ + + +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Apache TsFile is a columnar storage file format designed for time series data. The project provides implementations in three languages, each in its own top-level directory. The main branch is `develop`. + +TsFile uses a hierarchical storage model: **Page → Chunk → ChunkGroup → File**. Data is organized by device, with each device's measurements stored as individual time series. Two write models exist: **aligned** (all measurements share timestamps) and **non-aligned** (independent timestamps per measurement). + +## Repository Structure + +| Directory | Language | Build System | Details | +|-----------|----------|--------------|---------| +| `java/` | Java | Maven | [java/CLAUDE.md](java/CLAUDE.md) | +| `cpp/` | C++ | CMake | [cpp/CLAUDE.md](cpp/CLAUDE.md) | +| `python/` | Python | setuptools + Cython | [python/CLAUDE.md](python/CLAUDE.md) | + +The root `pom.xml` orchestrates all three via Maven profiles: + +```bash +./mvnw clean verify -P with-java # Java only +./mvnw clean verify -P with-cpp # C++ only +./mvnw clean verify -P with-python # Python (requires C++ built first) +``` + +## Cross-Module Dependencies + +- **Python depends on C++**: The Python module uses Cython to bind to the C++ shared library (`libtsfile`). C++ must be built before Python (`-P with-python` implies C++ build). +- **Java is independent**: The Java implementation shares no build-time dependency with C++ or Python. +- All three implementations read/write the same TsFile binary format. + +## Code Formatting + +```bash +./mvnw spotless:apply # Format all languages (Java: Google Java Format, C++: clang-format, Python: Black) +./mvnw spotless:check # Check formatting without modifying +``` + +## License Header + +Every new file must include the Apache License 2.0 header at the top. Use the comment style appropriate for the file type (e.g., `` for Markdown, `/* */` for Java/C++, `#` for Python). See any existing file for the exact wording. + +## Git Commit + +- Do NOT add `Co-Authored-By` trailer to commit messages. diff --git a/cpp/CLAUDE.md b/cpp/CLAUDE.md new file mode 100644 index 000000000..00157dd5a --- /dev/null +++ b/cpp/CLAUDE.md @@ -0,0 +1,108 @@ + + +# CLAUDE.md — C++ Module + +This file provides guidance to Claude Code (claude.ai/code) when working with the C++ module. + +## Build Commands + +```bash +# Build (Release, default) +bash build.sh + +# Build variants +bash build.sh -t=Debug +bash build.sh -t=RelWithDebInfo +bash build.sh -a=ON # Enable AddressSanitizer +bash build.sh -c=ON # Enable code coverage + +# Disable optional compression libraries +bash build.sh --disable-snappy --disable-lz4 --disable-lzokay --disable-zlib + +# Or use CMake directly +mkdir -p build/Release && cd build/Release +cmake ../.. -DCMAKE_BUILD_TYPE=Release -DBUILD_TEST=ON +make -j$(nproc) + +# Via Maven from repo root +./mvnw clean verify -P with-cpp + +# Run all tests +./build/Release/lib/TsFile_Test + +# Run specific test suite +./build/Release/lib/TsFile_Test --gtest_filter=SnappyCompressorTest.* +``` + +## Build Options (CMake) + +All `ON` by default unless noted: + +| Option | Purpose | +|--------|---------| +| `BUILD_TEST` | Compile tests (GTest 1.12.1, auto-downloaded) | +| `ENABLE_ANTLR4` | ANTLR4 parser runtime | +| `ENABLE_SNAPPY` / `ENABLE_LZ4` / `ENABLE_LZOKAY` / `ENABLE_ZLIB` | Compression libraries | +| `ENABLE_THREADS` | Multi-threaded read/write via pthreads | +| `ENABLE_ASAN` | AddressSanitizer (`OFF` by default) | +| `ENABLE_SIMDE` | SIMD Everywhere (`OFF` by default) | + +## Source Structure + +``` +cpp/src/ +├── common/ # Core types: Schema, Tablet, DeviceId, TsBlock, allocators, config +├── compress/ # Compression: Snappy, LZ4, LZOKAY, Zlib (factory pattern) +├── encoding/ # Encoding: Plain, TS2Diff, Gorilla, Dictionary, RLE, Zigzag, SPRINTZ +├── file/ # File I/O: TsFileIOReader/Writer, RestorableTsFileIOWriter +├── reader/ # Read path: TsFileReader, QueryExecutor, filters, result sets +├── writer/ # Write path: TsFileWriter, TsFileTableWriter, ChunkWriter, PageWriter +├── parser/ # ANTLR4 path parser (grammars + generated code) +├── cwrapper/ # C language bindings (used by Python module) +└── utils/ # Utilities: error codes, date handling, fault injection +``` + +## Architecture Notes + +- **C++11** standard, targets CMake 3.11+ +- Dual data model: **tree-view** (`TsFileTreeWriter/Reader`) and **table-view** (`TsFileTableWriter`, `TableQueryExecutor`) +- Parallel column encoding in table write path, controlled by `ENABLE_THREADS` +- Third-party libraries are bundled under `third_party/` (ANTLR4, Snappy, LZ4, LZOKAY, Zlib, SIMDe) +- `cwrapper/` provides the C API that the Python module binds to via Cython + +## Code Style + +- **Formatter**: clang-format (Google style), configured in `.clang-format` + +## Testing + +- **Framework**: Google Test 1.12.1 (auto-downloaded during build, or supply `third_party/googletest-release-1.12.1.zip`) +- Tests in `cpp/test/`, mirroring `src/` structure +- Test discovery via `gtest_discover_tests()` + +## License Header + +Every new file must include the Apache License 2.0 header at the top. For C/C++ files, use the `/* */` block comment style. See any existing `.h` or `.cc` file for the exact wording. + +## Git Commit + +- Do NOT add `Co-Authored-By` trailer to commit messages. diff --git a/java/CLAUDE.md b/java/CLAUDE.md new file mode 100644 index 000000000..243d9e109 --- /dev/null +++ b/java/CLAUDE.md @@ -0,0 +1,101 @@ + + +# CLAUDE.md — Java Module + +This file provides guidance to Claude Code (claude.ai/code) when working with the Java module. + +## Build Commands + +Maven with wrapper (`./mvnw` from repo root). All commands require `-P with-java`. + +```bash +# Build (skip tests) +./mvnw clean package -P with-java -DskipTests + +# Install to local Maven repo +./mvnw install -P with-java -DskipTests + +# Run all tests (unit + integration) +./mvnw clean verify -P with-java + +# Run unit tests only +./mvnw test -P with-java + +# Run integration tests only +./mvnw verify -P with-java -DskipUTs=true + +# Run a single test class (use -pl to target submodule) +./mvnw test -P with-java -pl java/tsfile -Dtest=TsFileWriterTest + +# Run a single test method +./mvnw test -P with-java -pl java/tsfile -Dtest=TsFileWriterTest#testWrite + +# Format code (requires Java 17+ runtime) +./mvnw spotless:apply + +# Check formatting +./mvnw spotless:check +``` + +## Module Structure + +Four submodules under `java/`: + +- **common** — Shared types: `TSDataType`, `ColumnCategory`, `Binary`, `BitMap`, constants +- **tsfile** — Core implementation: read/write APIs, encoding, compression, file format +- **examples** — Reference implementations for read/write operations +- **tools** — CLI utilities for TsFile inspection + +## Architecture + +Key packages in `java/tsfile/src/main/java/org/apache/tsfile/`: + +- `write/` — Write path: `TsFileWriter`, `Tablet` (batch API), `TSRecord` (row API) +- `read/` — Read path: `TsFileReader`, `TsFileSequenceReader`, `QueryExpression` +- `encoding/` — Encoding algorithms (RLE, TS_2DIFF, GORILLA, DICTIONARY) +- `compress/` — Compression codecs (Snappy, LZ4, ZSTD, XZ) +- `file/` — File format structures (metadata, headers, footers) +- `compatibility/` — Version compatibility handling +- `parser/` — ANTLR4-generated path parser (grammar in `src/main/antlr4/`) + +Code generation: FreeMarker templates in `tsfile/src/main/codegen/` generate type-specific implementations (e.g., per-datatype readers/writers) into `target/generated-sources/`. + +## Code Style + +- **Formatter**: Spotless with Google Java Format 1.28.0 (Spotless requires Java 17+ to run but the project targets Java 8 bytecode) +- **Checkstyle**: Google style variant in root `checkstyle.xml`, 100 char line limit +- **Import order**: `org.apache.tsfile`, `javax`, `java`, static imports + +## Testing Conventions + +- **Framework**: JUnit 4 +- Unit tests: `*Test.java` — run via `mvn test` +- Integration tests: `*IT.java` — run via `mvn verify` +- Tests run in random order with `reuseForks=false` + +## License Header + +Every new file must include the Apache License 2.0 header at the top. For Java files, use the `/* */` block comment style. See any existing `.java` file for the exact wording. + +## Git Commit + +- Do NOT add `Co-Authored-By` trailer to commit messages. diff --git a/python/CLAUDE.md b/python/CLAUDE.md new file mode 100644 index 000000000..2fc0ca88e --- /dev/null +++ b/python/CLAUDE.md @@ -0,0 +1,104 @@ + + +# CLAUDE.md — Python Module + +This file provides guidance to Claude Code (claude.ai/code) when working with the Python module. + +## Prerequisites + +The Python module depends on the C++ shared library (`libtsfile`). **Build C++ first:** + +```bash +# From repo root +./mvnw clean package -P with-cpp -DskipTests +``` + +## Build Commands + +```bash +# Build everything via Maven (builds C++ then Python) +./mvnw clean verify -P with-python + +# Or build directly (after C++ is built) +cd python +python setup.py build_ext --inplace + +# Run tests +pytest tests/ + +# Run a single test +pytest tests/test_write_and_read.py -k "test_name" + +# Format code +cd ../ && ./mvnw spotless:apply # Uses Black 26.3.1 +``` + +## Source Structure + +``` +python/tsfile/ +├── __init__.py # Public API exports +├── constants.py # Enums: TSDataType, TSEncoding, Compressor, ColumnCategory +├── schema.py # TimeseriesSchema, DeviceSchema, TableSchema, ColumnSchema +├── tablet.py # Tablet class for batch columnar data +├── field.py / row_record.py # Row-based data structures +├── tag_filter.py # Tag filtering DSL for queries +├── tsfile_table_writer.py # High-level table writer API +├── utils.py # to_dataframe(), dataframe_to_tsfile() +├── exceptions.py # Custom exception types +│ +├── tsfile_py_cpp.pyx # Cython core: Python ↔ C++ type conversion +├── tsfile_reader.pyx # Cython reader wrapper +├── tsfile_writer.pyx # Cython writer wrapper +├── tsfile_cpp.pxd # C++ interface declarations (cwrapper) +│ +└── dataset/ # High-level multi-file API + ├── dataframe.py # TsFileDataFrame (pandas-like interface) + ├── timeseries.py # Timeseries, AlignedTimeseries + ├── reader.py # Multi-file reader coordination + └── merge.py # K-way merge for overlapping time ranges +``` + +## Architecture Notes + +- **Layered design**: C++ core → Cython bridge (`*.pyx`) → Pure Python API +- `setup.py` copies pre-built C++ headers and shared libraries from `../cpp/target/build/` +- Cython binds to `cwrapper/tsfile_cwrapper.h` from the C++ module +- C error codes are mapped to Python exceptions via `check_error()` in `tsfile_py_cpp.pyx` +- Requires Python 3.9+, numpy >= 2.0, pandas >= 2.0, pyarrow >= 16.0 + +## Code Style + +- **Formatter**: Black 26.3.1 (via Spotless Maven plugin) + +## Testing + +- **Framework**: pytest +- Tests in `python/tests/`, 19 test files covering write, read, filtering, DataFrame conversion, and Arrow format + +## License Header + +Every new file must include the Apache License 2.0 header at the top. For Python files, use `#` line comments. For `.pyx`/`.pxd` files, also use `#` comments. See any existing file for the exact wording. + +## Git Commit + +- Do NOT add `Co-Authored-By` trailer to commit messages.