diff --git a/README.md b/README.md index 15d6315..c8ad709 100644 --- a/README.md +++ b/README.md @@ -2,514 +2,245 @@ KEYSTONE logo

-### KEYSTONE helps large databases find the right record fast — using adaptive search, resident-hardware acceleration, and repeatable indexing logic. +# KEYSTONE + +### Faster access to high-value records without replacing the systems that already store them. [![C](https://img.shields.io/badge/C-11-blue.svg)](https://en.wikipedia.org/wiki/C11_(C_standard_revision)) -[![Fortran](https://img.shields.io/badge/Fortran-90%2B-purple.svg)](https://en.wikipedia.org/wiki/Fortran) -[![Python](https://img.shields.io/badge/Python-3-yellow.svg)](https://www.python.org/) [![SIMD](https://img.shields.io/badge/SIMD-SSE4.2%20%7C%20AVX2%20%7C%20AVX--512-black.svg)](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) [![Parallel](https://img.shields.io/badge/Parallel-OpenMP-green.svg)](https://www.openmp.org/) -[![Archives](https://img.shields.io/badge/Ingestion-tar.zst-orange.svg)](https://facebook.github.io/zstd/) [![Platform](https://img.shields.io/badge/Platform-Linux-success.svg)](https://www.kernel.org/) [![License](https://img.shields.io/badge/License-AGPL--3.0-red.svg)](LICENSE) -**KEYSTONE** is a high-performance intelligence ingestion and database indexing engine. It is designed to operate as a standalone fast-search layer, or as the native pre-processing pipeline for the [QIHSE](https://github.com/SWORDIntel/QIHSE) database ecosystem. - -It combines zero-allocation dirty log tokenization, anchor-guided interpolation search, SIMD-assisted local scans, and a native C neural network (micro-model) to pull structured semantic hits from chaotic post-compromise data. +**KEYSTONE is a high-performance indexing, ingestion, and lookup engine for large datasets.** It can run as a standalone acceleration layer or feed structured data directly into [QIHSE](https://github.com/SWORDIntel/QIHSE). - - ---- +It is designed to make existing infrastructure work harder: reduce lookup cost, process large batches efficiently, turn messy source data into searchable records, and choose the fastest viable execution path on the hardware already available. -## Current Status +**You do not have to adopt the whole stack.** KEYSTONE's search, ingestion, archive, classification, and QIHSE integration capabilities can be used independently where they make sense. -KEYSTONE is a working native C library and benchmark suite, not just a design note. - -| Area | Status | -|---|---| -| Core sorted `int64_t` lookup | Implemented and tested | -| Anchor-guided interpolation search | Implemented and tested | -| Batch lookup | Implemented and tested | -| Runtime backend calibration | Implemented and tested | -| Unstructured / Dirty Log Tokenizer | Implemented (zero-allocation email:pass extraction) | -| Heterogeneous Hash Indexer | Implemented (FNV-1a column projection) | -| Native Context Micro-Model | Implemented (6-class DNN with confidence gating) | -| OpenMP batch path | Auto-enabled by default when compiler supports it | -| Fortran batch backend | Optional; enabled when requested | -| `.tar.zst` archive search | Optional; enabled when `libarchive` and `libzstd` are available | -| SSE4.2 small-window scan | Implemented for native x86 builds with SSE4.2+ (AVX1-only CPUs) | -| AVX2 small-window scan | Implemented for native x86 builds with AVX2 | -| AVX-512 path | Build-gated and hardware-dependent | - -The default workflow is intentionally native: build on the machine, container image, or silicon family where the code will run, then measure there. +[Business benefits](#why-keystone) · [How it works](#how-it-fits) · [Measured results](#measured-results) · [Quick start](#quick-start) · [Technical docs](docs/README.md) --- -## Quick Start - -```bash -make clean -make test -``` - -`make test` builds and runs the available test binaries for the current host. Optional components are included when their local dependencies are present. - -For a scalar-only comparison build: - -```bash -make clean -KEYSTONE_ENABLE_FORTRAN=0 KEYSTONE_ENABLE_TAR_ZST=0 KEYSTONE_FORCE_SCALAR=1 make test -``` - -For build-only verification: - -```bash -make tests -``` - -For benchmarks: - -```bash -make benchmarks -./benchmarks/dsmil_benchmark -./benchmarks/performance_proof -``` - ---- - -## Mission - -Modern databases rarely fail because storage is unavailable. They fail because lookup paths drift, records fragment, indexes become inconsistent, and high-volume datasets become expensive to search reliably. +## Why KEYSTONE -KEYSTONE addresses that layer directly. +Large data systems often accumulate cost in places that are difficult to see on a storage invoice: repeated lookup work, duplicated indexing logic, slow batch processing, underused CPU capabilities, and expensive preprocessing before useful records can even be queried. -It is designed for systems where record identity, index stability, and lookup speed matter at the same time. Instead of treating indexing as a side effect of storage, KEYSTONE treats precise indexed retrieval as the core primitive. +KEYSTONE targets that layer. ---- - -## What KEYSTONE Is - -KEYSTONE is a functional indexing and lookup engine for sorted `int64_t` datasets. It sits close to the data layer, supporting fast search, batch lookup, telemetry processing, archive-aware ingestion, and performance validation. - -It is not a decorative wrapper around a database. It is a low-level search component designed for practical integration into larger systems that need predictable record access. - -| Capability | Purpose | +| Business problem | KEYSTONE response | |---|---| -| **Anchor-guided interpolation search** | Reduces search work by using learned anchor points across sorted data. | -| **Unstructured data ingestion** | A zero-allocation C tokenizer aggressively hunts for `email:pass`, URLs, and identifiers in raw, dirty stealer logs and memory dumps. | -| **Hash indexing** | Projects strings into 64-bit integer hashes via FNV-1a, feeding them directly into KEYSTONE's fast scalar search. | -| **Native micro-model** | Extracts 256 bytes of context around a hit and classifies the target (e.g., FINANCIAL, CORPORATE) using a compiled 6-class neural network. | -| **Batch lookup** | Handles high-volume query workloads efficiently. | -| **Runtime backend calibration** | Measures viable local backends on first use, caches the fastest decision, and reports the selected path. | -| **SIMD-assisted search windows** | Uses compiled AVX2 and build-gated AVX-512 scan paths where available. | -| **OpenMP parallelism** | Scales batch workloads across CPU threads when built with OpenMP. | -| **Fortran batch backend** | Provides an optional high-throughput backend for numerical batch processing. | -| **Benchmark suite** | Measures latency, throughput, and backend behavior across dataset profiles. | - ---- - -## Why It Matters - -High-speed lookup is easy to claim and difficult to prove. Real systems need more than a fast happy path. +| **Large datasets become slower and more expensive to search** | Adaptive indexed lookup reduces the amount of work required to locate records. | +| **Existing servers are not used efficiently** | Runtime calibration measures viable local execution paths and selects an appropriate backend for the workload. | +| **Different services duplicate search and preprocessing logic** | A reusable native indexing layer centralizes high-volume lookup and ingestion primitives. | +| **Raw or compressed data takes too much preprocessing** | Archive-aware and unstructured-data ingestion can turn source material into searchable identifiers close to the data. | +| **Performance claims are difficult to trust** | Backend decisions and benchmark methodology are exposed so results can be reproduced on the target hardware. | +| **Replacing the primary database is too disruptive** | KEYSTONE can sit beside an existing system or act as a preprocessing layer rather than requiring a database migration. | -KEYSTONE is built around the problems that appear when datasets become large, query volume increases, and records must remain stable across repeated processing runs. +### What that means operationally -It is useful where the cost of bad indexing is operationally significant: - -- stale or unstable lookup positions; -- slow batch search over large sorted datasets; -- duplicated lookup logic across services; -- poor visibility into backend performance; -- inconsistent ingestion from compressed archives; -- weak testability around search behavior; -- performance claims without reproducible benchmarks. - -KEYSTONE provides a focused answer: a compact, benchmarkable, database-adjacent indexing engine with multiple execution backends and a clear performance model. +- **Faster retrieval where lookup is a bottleneck.** +- **Better use of hardware already owned** before adding more infrastructure. +- **Lower integration risk** because KEYSTONE can be adopted as one component rather than an all-or-nothing platform. +- **More predictable indexing behavior** across repeated processing runs. +- **A measurable optimization path:** local calibration, decision provenance, tests, and benchmark tooling are part of the implementation. +- **A direct path into QIHSE** when a broader multi-model database runtime is useful, without making QIHSE a prerequisite. --- -## Architecture - -```mermaid -flowchart TB - subgraph Intelligence["Post-Compromise Intelligence Pipeline"] - DIRTY["Dirty Log / Memory Dump"] --> PARSE["Custom C Tokenizer"] - PARSE -->|Extracts email:pass| HASH["Hash Indexer (FNV-1a)"] - HASH --> AT["Anchor Table"] - AT --> R["Result Offset"] - R --> BRIDGE["Model Context Bridge"] - BRIDGE -->|256-byte window| MODEL["Native Micro-Model"] - MODEL --> CLASS["6-Class Semantic Triage"] - end - - subgraph Search["Numeric Search Pipeline"] - Q["Query Key / Query Batch"] --> Auto{"Runtime Backend Calibrator"} - Auto -->|Single / Small| S["Scalar Anchor Search"] - Auto -->|Sorted Batch| CB["Optimized C Batch"] - Auto -->|Large Batch + OpenMP| MP["C OpenMP Batch"] - Auto -->|Dense Batch| FT["Optional Fortran Batch"] - S --> SIMD["SIMD Scan (AVX2 / AVX-512)"] - S --> AT - CB --> AT - SIMD --> AT - MP --> AT - FT --> AT - end -``` +## What KEYSTONE Does ---- +KEYSTONE combines several focused capabilities behind one native library: -## Backend Selection Model +| Capability | Practical purpose | +|---|---| +| **Adaptive indexed search** | Finds records in large sorted keyspaces using anchor-guided interpolation rather than relying only on generic binary search. | +| **High-volume batch lookup** | Processes large query sets through optimized C, OpenMP, and optional numerical backends. | +| **Runtime backend calibration** | Measures viable execution paths on the local machine and caches the fastest choice for comparable workloads. | +| **Unstructured-data ingestion** | Extracts useful identifiers from noisy source data without requiring a heavyweight parsing stack. | +| **Archive-aware processing** | Supports `.tar.zst` workflows so compressed datasets can participate in the indexing pipeline. | +| **Context classification** | An optional small native neural model can classify extracted context for downstream triage. | +| **QIHSE integration** | Can act as a native preprocessing/ingestion layer for the QIHSE database ecosystem. | -```mermaid -flowchart TD - Start["Incoming Lookup Workload"] --> Mode{"Single or Batch?"} - Mode -->|Single| Scalar["Scalar Interpolation Search"] - Mode -->|Batch| Size{"Dataset and Batch Size"} - - Size -->|Small / Low Overhead Preferred| Scalar - Size -->|Sorted Batch| CB["Optimized C Merge-Walk Batch"] - Size -->|Large / Repeated Queries| CPU{"Build + Workload Capabilities"} - - CPU -->|OpenMP Built + Enough Queries| OMP["C OpenMP Batch Execution"] - CPU -->|Fortran Built + Dense Sorted Shape| FORTRAN["Fortran Batch Execution"] - CPU -->|Otherwise| CB - - OMP --> Cal["Measured Local Calibration"] - FORTRAN --> Cal - CB --> Cal - Scalar --> Anchor["Anchor Table / Adaptive Learning"] - Cal --> Anchor - Anchor --> Result["Stable Result Index"] -``` +The core search engine is useful by itself. The ingestion, archive, classification, Fortran, OpenMP, SIMD, and QIHSE paths are additive capabilities rather than mandatory dependencies. --- -## Data Flow +## How It Fits ```mermaid flowchart LR - Source["Source Dataset"] --> Normalize["Sorted int64_t Keyspace"] - Normalize --> Anchor["Anchor-Guided Search Layer"] - Anchor --> Backend["Selected Execution Backend"] - Backend --> Index["Result Index"] - Index --> Consumer["Database / Telemetry / Analysis Consumer"] + A["Existing data\nDatabase · Telemetry · Archives · Raw files"] --> I - Archive["Compressed Archive"] --> Member["Member Offset Index"] - Member --> Anchor + subgraph KS["KEYSTONE acceleration layer"] + I["Ingest / Normalize"] --> S["Adaptive indexed search"] + S --> C["Runtime backend calibration"] + I --> M["Optional context classification"] + end - Bench["Benchmark Harness"] --> Backend - Tests["Validation Suite"] --> Anchor + S --> R["Fast record / offset / entity lookup"] + M --> R + R --> X["Existing application or analysis workflow"] + R --> Q["Optional QIHSE ingestion"] ``` ---- - -## System Profile - -| Layer | Function | -|---|---| -| **Core search engine** | Performs anchor-guided interpolation search over sorted integer data. | -| **Dirty Log Tokenizer** | Aggressively rips identifiers (`email:pass`) from unstructured dumps. | -| **Hash Indexer** | Projects heterogeneous strings into 64-bit space for rapid search. | -| **Context Bridge** | Slices perfectly shaped context windows from raw data bypassing full decompression. | -| **Micro-Model Inference** | A compiled DNN executing focal-loss trained classification on context data. | -| **Adaptive backend layer** | Routes workloads across scalar, optimized C, OpenMP, and Fortran paths. | -| **Anchor table** | Maintains learned search anchors to improve repeated lookup behavior. | -| **Archive interface** | Enables `.tar.zst` workflows without treating archive handling as an afterthought. | - ---- - -## Feature Matrix - -| Feature | Scalar / Anchor C | Optimized C Batch | SIMD Local Scan | OpenMP Batch | Fortran Batch | `.tar.zst` | -|---|---:|---:|---:|---:|---:|---:| -| Single search | Yes | No | Yes, inside local windows | No | No | No | -| Batch search | Yes | Yes | Indirect | Optional | Optional | No | -| Auto backend calibration | Yes | Yes | Build/runtime detected | Optional measured candidate | Optional measured candidate | No | -| Decision provenance | Fast path / measured / cached / fallback | Measured or cached | Build/runtime detected | Measured or cached | Measured or cached | No | -| Anchor learning | Yes | No for merge-walk batch | Yes through scalar path | Per-thread clone path | No | No | -| Runtime tuning | Yes | Yes | Build/runtime gated | Build gated | Build gated | No | -| Archive ingestion | No | No | No | No | No | Yes | -| Member offset indexing | No | No | No | No | No | Yes | -| Benchmark validation | Yes | Yes | Partial / host-specific | Yes when built | Yes when built | Yes | -| Linux support | Yes | Yes | Host-dependent | Compiler/runtime-dependent | Toolchain-dependent | Dependency-dependent | - ---- - -## Practical Use Cases - -### Database Index Acceleration - -KEYSTONE is suitable for systems that need fast lookup across large sorted integer keyspaces, especially where keys map into record offsets, entity identifiers, telemetry IDs, event IDs, or compressed member indexes. +The design is intentionally database-adjacent. KEYSTONE does not need to own the system of record; it accelerates the path between source data and the record an application actually needs. -### Canonical Record Retrieval - -When records are normalized into stable integer identifiers, KEYSTONE can act as the lookup layer that keeps retrieval fast and repeatable. - -### Telemetry and Event Search - -Telemetry systems often generate large ordered datasets that must be searched repeatedly. KEYSTONE is designed for that access pattern: high-volume lookup, repeatable index resolution, and measurable backend performance. - -### Archive-Aware Data Processing - -Compressed archive workflows are common in telemetry, exports, backups, and evidence packages. KEYSTONE includes `.tar.zst` ingestion support so archive processing can remain close to the indexed lookup model. - -### Benchmark-Driven Optimization - -The project includes performance tooling intended to compare backend behavior across dense, large, and jittered dataset profiles. This makes it suitable as both functional software and a performance engineering showcase. - -### Systems Programming Demonstration - -KEYSTONE demonstrates practical low-level engineering across C11, optional Fortran, Python-based benchmark visualization, SIMD-aware execution, optional OpenMP parallelism, compressed archive ingestion, and structured validation. - -### QIHSE Unified Wire Protocol Bridge - -KEYSTONE can act as a native ingestion head-node for [QIHSE](https://github.com/SWORDIntel/QIHSE). When compiled with the bridge enabled, KEYSTONE parses dirty archives, categorizes the extracted intelligence using the micro-model, and streams the structured hits directly into QIHSE's Key-Value or Vector databases via UWP memory layout. - -To build KEYSTONE as a QIHSE ingestor: - -```bash -make clean -KEYSTONE_ENABLE_QIHSE_BRIDGE=1 QIHSE_ROOT=/path/to/QIHSE make -``` +For the detailed search pipeline, backend-selection model, feature matrix, memory behavior, SIMD paths, and integration internals, see the [technical overview](docs/TECHNICAL_OVERVIEW.md). --- -## Performance Orientation +## Measured Results -KEYSTONE is designed around a simple premise: search performance should be measurable, backend-aware, and reproducible. +KEYSTONE includes benchmark tooling because performance should be demonstrated on the hardware and workload that will actually run it. -The benchmark suite is intended to produce concrete latency and speedup comparisons across workload profiles, including dense datasets, larger million-scale datasets, jittered distributions, and backend-specific behavior. +One current integrated benchmark on an **Intel Xeon E5-2407** measured the KEYSTONE sorted-column search at: -| Performance Concern | KEYSTONE Response | -|---|---| -| **Small lookup overhead** | Scalar interpolation remains available where vector or parallel overhead would be wasteful. | -| **Large batch volume** | Optimized C batch, OpenMP, and optional Fortran paths provide acceleration options for heavier workloads. | -| **Resident hardware variability** | Runtime feature detection plus first-use timing calibration supports backend selection based on the local CPU today and leaves room for measured GPU/NPU backends later. | -| **Repeated lookup behavior** | Anchor learning helps reduce repeated search cost across sorted keyspaces. | -| **Benchmark credibility** | Dedicated benchmark outputs support reviewable performance claims. | +| Same-host lookup benchmark | Throughput | p50 latency | +|---|---:|---:| +| Standard binary search, 1M rows | 2,016,334 lookups/s | 415 ns | +| **KEYSTONE anchor search, 1M rows** | **3,510,610 lookups/s** | **218 ns** | -### Benchmark Posture +That run represents roughly **1.74× higher throughput and 47% lower median lookup latency** against the benchmark's standard binary-search baseline on that host. -Performance numbers are meaningful only with the host silicon, compiler flags, feature toggles, dataset shape, query hit rate, warmup policy, and transfer costs attached. Treat checked-in benchmark reports as examples of measurement style, not universal guarantees. - -For serious comparison, run the benchmark matrix on the target machine and keep the generated CSV/output with the exact build command. +These are **measured results, not universal guarantees**. CPU, compiler flags, dataset distribution, hit rate, cache state, batch shape, and enabled backends materially affect performance. Full methodology and additional measurements are kept in [`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md). --- -## Repository Structure +## Current State -```mermaid -flowchart LR - Root["KEYSTONE Root"] --> SRC["src/"] - Root --> INC["include/"] - Root --> TEST["tests/"] - Root --> BENCH["benchmarks/"] - Root --> DOCS["docs/"] - Root --> FORT["fortran/"] - Root --> SCRIPTS["scripts/"] - - SRC --> Core["Core Search Engine"] - SRC --> Wrapper["Integration Wrapper"] - SRC --> Telemetry["Telemetry Processor"] - SRC --> Archive["tar.zst Interface"] - - INC --> PublicAPI["Public Headers"] - INC --> TelemetryAPI["Telemetry Headers"] - INC --> WrapperAPI["Wrapper Headers"] - - TEST --> NativeTest["Native Core Tests"] - TEST --> BackendTest["Backend Selection Tests"] - TEST --> ArchiveTest["Archive Tests"] - TEST --> PerfTest["Performance Tests"] - - BENCH --> Harness["Benchmark Harness"] - BENCH --> Proof["Performance Proofing"] - BENCH --> Charts["Visualization Tools"] - - FORT --> FBackend["Fortran Batch Backend"] -``` +KEYSTONE is a working native library and test/benchmark suite. ---- +**Implemented today:** +- scalar and anchor-guided `int64_t` search; +- batch lookup and measured runtime backend selection; +- decision provenance for backend choices; +- SSE4.2 and AVX2 local scan paths where supported; +- build-gated AVX-512 path; +- OpenMP batch execution; +- optional Fortran batch backend; +- `.tar.zst` archive workflows; +- unstructured-data tokenizer and hash indexer; +- native context micro-model; +- QIHSE bridge support; +- correctness and performance test infrastructure. +GPU/NPU execution is not presented as a current production backend. The project detects or contains experimental accelerator work in places, but accelerator support is only considered implemented when correctness, transfer cost, fallback behavior, dispatch provenance, and target-hardware measurements are established. ---- - -## Intended Users - -KEYSTONE is intended for technical users who care about lookup correctness, runtime behavior, and database-adjacent performance. - -| User Type | Why It Fits | -|---|---| -| **Database engineers** | Useful for fast lookup layers and stable integer keyspaces. | -| **Systems programmers** | Demonstrates C11, SIMD, OpenMP, Fortran integration, and archive handling. | -| **Security researchers** | Useful for telemetry, indicator stores, evidence indexes, and high-volume lookup datasets. | -| **Data engineers** | Supports repeatable indexing before downstream processing or enrichment. | -| **Performance engineers** | Provides benchmarkable backend behavior across workload shapes. | -| **Research teams** | Works as a compact foundation for indexed retrieval experiments. | +See [`docs/STATUS_SUMMARY.md`](docs/STATUS_SUMMARY.md) for the engineering status and current backlog. --- -## Quality Model +## Deployment Model -| Goal | Standard | -|---|---| -| **Correctness** | Lookup behavior should remain predictable across supported backends. | -| **Repeatability** | The same sorted dataset and key should resolve consistently. | -| **Performance visibility** | Backend performance should be measurable rather than assumed. | -| **Backend flexibility** | Scalar, SIMD, parallel, and Fortran paths should serve different workload profiles. | -| **Archive practicality** | Compressed ingestion should integrate with the lookup model rather than exist as a detached helper. | -| **Operational usefulness** | The system should be practical for real database, telemetry, and analysis workflows. | +KEYSTONE is designed to be built for the machine, container image, or target silicon family where it will run. The normal build uses native optimization and can enable resident CPU capabilities and optional components when available. ---- +This gives deployments three useful properties: -## Requirements +1. **No requirement for specialized accelerator hardware.** The current core runs on CPU. +2. **Optional acceleration stays optional.** OpenMP, Fortran, SIMD paths, archive support, classification, and QIHSE integration can be selected independently. +3. **Backend choice is observable.** The runtime exposes whether a decision came from a fast path, measurement, cache, or fallback rather than hiding the execution path. -| Area | Requirement | -|---|---| -| **Operating system** | Linux | -| **Primary compiler** | GCC or Clang with C11 support | -| **Optional numerical backend** | Fortran 90+ capable compiler | -| **Optional archive support** | `libarchive` and `libzstd` | -| **Optional build detection** | `pkg-config` | -| **Benchmark visualization** | Python 3 with numerical and plotting support | -| **Parallel acceleration** | OpenMP-capable compiler/runtime (auto-enabled by default) | -| **Vector acceleration** | SSE4.2, AVX2, or AVX-512 capable CPU where available | -| **Future accelerator backends** | GPU or NPU runtime/toolchain only after explicit backend implementation and measurement | +Detailed build modes and feature switches are documented in [`docs/BUILD_MODES.md`](docs/BUILD_MODES.md). --- -## Native Tuning Model - -KEYSTONE is intentionally built as a native, silicon-tuned component. The -default Makefile uses `-O3 -march=native` and enables resident CPU paths such as -SSE4.2, AVX2, optional AVX-512, OpenMP (auto-enabled), Fortran, and `.tar.zst` -support when the local toolchain and libraries allow it. CPU execution is the -current implemented surface; GPU and NPU execution are future backend families -that must earn their place through explicit data-movement-aware benchmarks. - -That means the preferred deployment model is to build KEYSTONE on the machine, -container image, or target silicon family where it will run. It is not trying to -produce one lowest-common-denominator binary for every host. For reproducible -comparisons, pin the build flags and optional feature switches explicitly: +## Quick Start ```bash +git clone https://github.com/SWORDIntel/KEYSTONE.git +cd KEYSTONE make clean -KEYSTONE_ENABLE_FORTRAN=0 KEYSTONE_ENABLE_TAR_ZST=0 KEYSTONE_FORCE_SCALAR=1 make test +make test ``` -For the normal native path: +Build benchmarks: ```bash -make test +make benchmarks +./benchmarks/dsmil_benchmark +./benchmarks/performance_proof ``` -To build without executing tests: +A scalar comparison build is available for baseline testing: ```bash -make tests +make clean +KEYSTONE_ENABLE_FORTRAN=0 KEYSTONE_ENABLE_TAR_ZST=0 KEYSTONE_FORCE_SCALAR=1 make test ``` -This native posture is deliberate. KEYSTONE favors accurate local dispatch and reproducible local measurement over a single portable binary that hides the silicon-specific behavior. +For integration guidance, see [`docs/INTEGRATION.md`](docs/INTEGRATION.md). --- -## Security and Data Handling - -KEYSTONE is designed for datasets that may be operationally sensitive. Treat inputs, benchmark outputs, generated indexes, and archive contents according to the sensitivity of the source material. - -Recommended handling posture: +## QIHSE Integration -- do not commit private datasets or generated operational indexes; -- keep production database credentials outside the repository; -- validate archive contents before processing untrusted inputs; -- separate benchmark datasets from real operational data; -- preserve audit trails where indexed records support legal, investigative, or high-trust decisions; -- treat indexing errors as data integrity failures, not cosmetic defects. +KEYSTONE can operate as the ingestion and lookup front end for [QIHSE](https://github.com/SWORDIntel/QIHSE), while remaining independently usable. ---- - -## What KEYSTONE Is Not - -KEYSTONE is not a general spreadsheet sorter, dashboard framework, ORM, database replacement, or broad ETL platform. - -It is a focused indexing and lookup component for sorted integer datasets. Its strength is precision, backend-aware performance, and suitability for integration into larger database and telemetry systems. - ---- +```mermaid +flowchart LR + SRC["Source data"] --> K["KEYSTONE\nparse · index · classify"] + K -->|structured hits| Q["QIHSE"] + K -->|direct lookup| APP["Existing application"] +``` +To build the bridge when QIHSE is available locally: +```bash +make clean +KEYSTONE_ENABLE_QIHSE_BRIDGE=1 QIHSE_ROOT=/path/to/QIHSE make +``` --- -## Python SDK Quickstart - -KEYSTONE provides a high-performance Python SDK (`pip install -e python`): - -```python -import keystone -import numpy as np - -# 1. Single & Auto-Calibrated Batch Interpolation Search -data = np.arange(0, 1000000, 7, dtype=np.int64) -indices = keystone.KeystoneSearch.search_batch(data, [70, 7000, 140000]) +## Technical Snapshot -# Inspect backend provenance (scalar, OpenMP, AVX2, Fortran) -decision = keystone.KeystoneSearch.get_last_decision() -print(f"Active Backend: {decision.backend} (p95: {decision.p95_ns_per_key:.1f} ns/key)") +For readers who want the implementation detail without making it the front door: -# 2. DSMIL Telemetry Processor with Keystone Acceleration -with keystone.TelemetryProcessor(max_events=100000) as tp: - tp.add_event(keystone.TelemetryEvent(timestamp=1600000000, event_type=1, device_id=42, layer_id=2)) - ev = tp.find_by_timestamp(1600000000) +- **Primary implementation:** C11 +- **Current execution surface:** scalar C, SSE4.2/AVX2 local scans, build-gated AVX-512, optimized C batch, OpenMP, optional Fortran +- **Search model:** anchor-guided interpolation over sorted `int64_t` keyspaces +- **Auto-selection:** first-use local timing calibration with cached decisions +- **Data ingestion:** raw/unstructured tokenization, FNV-1a projection, optional archive handling +- **Classification:** native 260 → 64 → 6 context micro-model +- **Platform:** Linux +- **Testing posture:** correctness cross-checks plus workload- and backend-aware benchmarks -# 3. Cluster Slot Router (16,384 CRC16 Slots) -slot = keystone.ClusterRouter.get_slot("device:alpha:42") -print(f"Assigned Cluster Slot: {slot}") +### Documentation -# 4. Neural Micro-Model Classification (260->64->6 Feedforward) -clf = keystone.NeuralClassifier() -cls, name, conf = clf.classify("auth_failure admin@pentagon.af.mil topsecret_token=998822") -print(f"Classification: {name} ({conf*100:.1f}%)") -``` +| Document | Purpose | +|---|---| +| [`docs/README.md`](docs/README.md) | Documentation map | +| [`docs/TECHNICAL_OVERVIEW.md`](docs/TECHNICAL_OVERVIEW.md) | Architecture, backend selection, feature matrix, memory and execution model | +| [`docs/STATUS_SUMMARY.md`](docs/STATUS_SUMMARY.md) | What is implemented and what remains | +| [`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md) | Benchmark methodology and measured results | +| [`docs/BUILD_MODES.md`](docs/BUILD_MODES.md) | Native/scalar/optional build configuration | +| [`docs/INTEGRATION.md`](docs/INTEGRATION.md) | Integration guidance | +| [`docs/ACCELERATOR_CONTRACT.md`](docs/ACCELERATOR_CONTRACT.md) | Requirements for adding accelerator backends | +| [`docs/TELEMETRY_PROCESSOR.md`](docs/TELEMETRY_PROCESSOR.md) | Telemetry processor details | --- -## Joint QIHSE + KEYSTONE 5-Pillar Architecture Benchmarks - -Measured on host hardware (Intel Xeon E5-2407, AVX execution mode): +## What KEYSTONE Is Not -| Pillar / Subsystem | QIHSE + KEYSTONE Measured | Industry Standard / Alternative | Competitive Advantage | -| :--- | :--- | :--- | :--- | -| **[1] Vector Graph Search** | **33,080 QPS** (p50: 27.9 µs)
Anchor-Seeded 1D Spline Projection | **FAISS HNSW (CPU)**: ~15,000 QPS (65 µs)
**pgvector (HNSW)**: ~2,000 QPS (500 µs) | **2.2x higher QPS** vs FAISS CPU
**16.5x higher QPS** vs pgvector | -| **[2] Sorted Column / TSDB Search** | **3,510,610 lookups/s** (218 ns)
Keystone $O(\log \log N)$ Spline (18 ns best) | **C++ `std::lower_bound`**: 2,016,334 (447 ns)
**Postgres B-Tree**: ~600k lookups/s (1.2 µs) | **1.74x–2.0x faster** vs `std::lower_bound`
**5.5x faster** vs B+Tree pointer chasing | -| **[3] Packet Ingest / Log Scan** | **141,865 pkts/sec** (34.6 MiB/s)
AF_XDP Kernel Bypass + In-Place UMEM Scan | **Linux BSD Socket + epoll**: ~25,000 pkts/s
**Redis Ingestion**: ~75,000 ops/s | **5.6x higher throughput** vs epoll
**1.9x higher throughput** vs Redis | -| **[4] Neural Context Inference** | **370,749 infer/s** (2.55 µs)
Inlined Dense SAXPY C Kernel (260 $\to$ 64 $\to$ 6) | **ONNX Runtime (CPU)**: ~35,000 infer/s (28 µs)
**PyTorch LibTorch**: ~5,000 infer/s (200 µs) | **10.5x faster inference** vs ONNX Runtime
**74.0x faster** vs PyTorch LibTorch | -| **[5] Hybrid Multimodal Search** | **1,838 queries/s** (501 µs)
In-Memory BM25 + HNSW + Neural Masking | **OpenSearch Hybrid**: ~120 QPS (8.3 ms)
**Weaviate Hybrid**: ~200 QPS (5.0 ms) | **16.5x lower latency** vs OpenSearch
**10.0x lower latency** vs Weaviate | +KEYSTONE is not a replacement for every database, an ORM, a dashboard platform, or a general-purpose ETL suite. -> 📊 **Full Benchmark Details:** See [`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md). +It is a focused acceleration layer for indexing, lookup, and preprocessing workloads where retrieval cost, predictable record identity, and measurable execution behavior matter. --- ## License -**AGPL-3.0-or-later. This is strong copyleft. See [LICENSE](LICENSE) before use, modification, redistribution, hosting, or derivative work.** - -Network use, redistribution, modification, and derivative use carry obligations. Do not treat this repository as permissive code. - -If your intended use is proprietary, closed-source, commercial, hosted, or otherwise incompatible with AGPL compliance, obtain written permission or a separate license from the repository owner first. - -KEYSTONE does not include covert license telemetry or phone-home enforcement. Compliance is enforced through the AGPL license terms, copyright ownership, visible notices, and separate commercial licensing where appropriate. - -Unlicensed use outside the terms of the repository license is not authorized. Respect the license, attribute properly, and do not repackage the work as your own. +**AGPL-3.0-or-later.** See [LICENSE](LICENSE) before use, modification, redistribution, hosting, or derivative work. -For commercial or proprietary licensing discussions, contact the repository owner. +Commercial, proprietary, or hosted use that is incompatible with AGPL obligations requires separate permission or licensing from the repository owner. ---
**KEYSTONE** -Precision indexing. Adaptive lookup. Database discipline. +Fast retrieval. Measurable execution. Use only what you need.
diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..37fd5a6 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,36 @@ +# KEYSTONE Documentation + +The root [README](../README.md) is intentionally written as a high-level introduction. Use this directory for implementation, integration, benchmark, and engineering detail. + +## Start here + +| Document | Use it for | +|---|---| +| [TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md) | Architecture, search model, backend selection, feature matrix, memory behavior, ingestion pipeline, and QIHSE bridge. | +| [STATUS_SUMMARY.md](STATUS_SUMMARY.md) | Current implementation boundary and engineering backlog. | +| [BUILD_MODES.md](BUILD_MODES.md) | Native, scalar, optional Fortran, archive, and feature-controlled builds. | +| [INTEGRATION.md](INTEGRATION.md) | Embedding KEYSTONE into another application or data pipeline. | +| [BENCHMARK_RESULTS.md](BENCHMARK_RESULTS.md) | Current measurement rules and benchmark results. | +| [ACCELERATOR_CONTRACT.md](ACCELERATOR_CONTRACT.md) | Requirements a GPU/NPU/other accelerator backend must satisfy before it is treated as supported. | +| [TELEMETRY_PROCESSOR.md](TELEMETRY_PROCESSOR.md) | Telemetry processor implementation and usage. | + +## Benchmark records + +The repository also retains historical and phase-specific benchmark reports: + +- [benchmark_report.md](benchmark_report.md) +- [benchmark_report_phase5.md](benchmark_report_phase5.md) + +Treat host-specific benchmark reports as measurements of the stated machine/build/workload, not as universal performance guarantees. + +## Documentation rule + +The README should answer: + +1. What does KEYSTONE do? +2. Why would an organization use it? +3. Where does it fit into existing infrastructure? +4. What has actually been measured or implemented? +5. Where can a technical reader go deeper? + +Detailed backend mechanics, build switches, benchmark methodology, feature matrices, and experimental implementation notes belong under `docs/` so the public entry point stays readable to both technical and non-technical reviewers. diff --git a/docs/STATUS_SUMMARY.md b/docs/STATUS_SUMMARY.md index c47baeb..3e1ceaa 100644 --- a/docs/STATUS_SUMMARY.md +++ b/docs/STATUS_SUMMARY.md @@ -43,8 +43,7 @@ Condensed from DYNAMIC_HOT_PATH_PLAN, FORTRAN_BACKEND_PLAN, IMPROVEMENT_PLAN, OP ## Still To Do -See [ROADMAP.md](ROADMAP.md) for the phased roadmap and acceptance gates. The -items below are the current engineering backlog. +The items below are the current engineering backlog. The root README intentionally keeps this detail out of the executive overview; see [TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md) for the surrounding architecture. ### Calibration & Cache - Add more workload profile fields (hit-rate, gap, stride) to calibration cache keys and decision output. diff --git a/docs/TECHNICAL_OVERVIEW.md b/docs/TECHNICAL_OVERVIEW.md new file mode 100644 index 0000000..caaf202 --- /dev/null +++ b/docs/TECHNICAL_OVERVIEW.md @@ -0,0 +1,206 @@ +# KEYSTONE Technical Overview + +This document contains the implementation detail intentionally kept out of the root README. The README explains what KEYSTONE is and why it matters; this document explains how the current implementation is structured. + +## Architecture + +```mermaid +flowchart TB + subgraph Intelligence["Ingestion / Intelligence Pipeline"] + DIRTY["Raw Data / Memory Dump / Archive"] --> PARSE["Custom C Tokenizer"] + PARSE -->|Extract identifiers| HASH["Hash Indexer (FNV-1a)"] + HASH --> AT["Anchor Table"] + AT --> R["Result Offset"] + R --> BRIDGE["Model Context Bridge"] + BRIDGE -->|256-byte window| MODEL["Native Micro-Model"] + MODEL --> CLASS["6-Class Semantic Triage"] + end + + subgraph Search["Numeric Search Pipeline"] + Q["Query Key / Query Batch"] --> Auto{"Runtime Backend Calibrator"} + Auto -->|Single / Small| S["Scalar Anchor Search"] + Auto -->|Sorted Batch| CB["Optimized C Batch"] + Auto -->|Large Batch + OpenMP| MP["C OpenMP Batch"] + Auto -->|Dense Batch| FT["Optional Fortran Batch"] + S --> SIMD["SIMD Local Scan"] + S --> AT + CB --> AT + SIMD --> AT + MP --> AT + FT --> AT + end +``` + +## Backend Selection Model + +```mermaid +flowchart TD + Start["Incoming Lookup Workload"] --> Mode{"Single or Batch?"} + Mode -->|Single| Scalar["Scalar Interpolation Search"] + Mode -->|Batch| Size{"Dataset and Batch Size"} + + Size -->|Small / Low Overhead Preferred| Scalar + Size -->|Sorted Batch| CB["Optimized C Merge-Walk Batch"] + Size -->|Large / Repeated Queries| CPU{"Build + Workload Capabilities"} + + CPU -->|OpenMP Built + Enough Queries| OMP["C OpenMP Batch Execution"] + CPU -->|Fortran Built + Dense Sorted Shape| FORTRAN["Fortran Batch Execution"] + CPU -->|Otherwise| CB + + OMP --> Cal["Measured Local Calibration"] + FORTRAN --> Cal + CB --> Cal + Scalar --> Anchor["Anchor Table / Adaptive Learning"] + Cal --> Anchor + Anchor --> Result["Stable Result Index"] +``` + +The selector is intended to make execution choices measurable rather than purely heuristic. Normal uncached batch decisions can benchmark viable candidates, cache the result against workload/host characteristics, and expose the decision source through the public API. + +## Data Flow + +```mermaid +flowchart LR + Source["Source Dataset"] --> Normalize["Sorted int64_t Keyspace"] + Normalize --> Anchor["Anchor-Guided Search Layer"] + Anchor --> Backend["Selected Execution Backend"] + Backend --> Index["Result Index"] + Index --> Consumer["Database / Telemetry / Analysis Consumer"] + + Archive["Compressed Archive"] --> Member["Member Offset Index"] + Member --> Anchor + + Bench["Benchmark Harness"] --> Backend + Tests["Validation Suite"] --> Anchor +``` + +## Core Search Model + +KEYSTONE's current primary search surface operates on sorted `int64_t` keyspaces. Anchor points provide learned/local guidance into the sorted domain and scalar interpolation resolves candidate regions. Small local windows can use architecture-specific scan implementations when the build and runtime CPU support them. + +The implementation retains a scalar reference path so optimized backends can be checked against the same lookup semantics. + +### Current CPU execution paths + +- scalar C reference/anchor search; +- optimized C batch path; +- SSE4.2 local scan on supported x86 builds; +- AVX2 local scan on supported x86 builds; +- build-gated AVX-512 local scan; +- OpenMP batch execution; +- optional Fortran batch backend. + +AMX feature detection exists, but there is no current AMX search backend claim. GPU and NPU execution should likewise be treated as future/experimental backend families until their correctness, transfer costs, fallback behavior, dispatch provenance, and target-device measurements satisfy the accelerator contract. + +## Runtime Calibration + +`keystone_search_batch_auto()` can calibrate viable batch backends on a cache miss rather than assuming a fixed backend is best for every machine or query shape. + +Current decision state includes: + +- selected backend; +- decision source such as fast path, measured, cache, or static fallback; +- query-shape classification; +- measured latency information including p95 where available; +- calibration-run and candidate information used by benchmark tooling. + +The calibration cache currently keys on CPU feature mask, array-size bucket, query-count bucket, and thread count. Additional workload-shape fields remain part of the engineering backlog. + +## System Profile + +| Layer | Function | +|---|---| +| **Core search engine** | Anchor-guided interpolation over sorted integer data. | +| **Dirty-data tokenizer** | Extracts identifiers from noisy/unstructured source material without allocation-heavy parsing. | +| **Hash indexer** | Projects heterogeneous strings into a 64-bit integer search space using FNV-1a. | +| **Context bridge** | Extracts bounded context windows around matched offsets. | +| **Micro-model inference** | Native 260 → 64 → 6 feed-forward classification for optional semantic triage. | +| **Adaptive backend layer** | Routes batch workloads across viable scalar, optimized C, OpenMP, and optional Fortran paths. | +| **Anchor table** | Maintains search guidance for repeated lookup behavior. | +| **Archive interface** | Supports `.tar.zst` member workflows when archive dependencies are enabled. | +| **QIHSE bridge** | Streams structured results into QIHSE when compiled with integration support. | + +## Feature Matrix + +| Feature | Scalar / Anchor C | Optimized C Batch | SIMD Local Scan | OpenMP Batch | Fortran Batch | `.tar.zst` | +|---|---:|---:|---:|---:|---:|---:| +| Single search | Yes | No | Yes, inside local windows | No | No | No | +| Batch search | Yes | Yes | Indirect | Optional | Optional | No | +| Auto backend calibration | Yes | Yes | Build/runtime detected | Optional measured candidate | Optional measured candidate | No | +| Decision provenance | Fast path / measured / cached / fallback | Measured or cached | Build/runtime detected | Measured or cached | Measured or cached | No | +| Anchor learning | Yes | No for merge-walk batch | Through scalar path | Per-thread clone path | No | No | +| Runtime tuning | Yes | Yes | Build/runtime gated | Build gated | Build gated | No | +| Archive ingestion | No | No | No | No | No | Yes | +| Member offset indexing | No | No | No | No | No | Yes | +| Benchmark validation | Yes | Yes | Host-specific | Yes when built | Yes when built | Yes | +| Linux support | Yes | Yes | Host-dependent | Runtime-dependent | Toolchain-dependent | Dependency-dependent | + +## Memory Model + +The native core includes memory-oriented optimizations intended for large arrays: + +- transparent huge-page hints through `madvise(MADV_HUGEPAGE)`; +- size gating so small allocations are not needlessly advised; +- software prefetching for medium/large search arrays; +- bounded memory-ramp tooling for capacity experiments; +- benchmark/reporting work around page faults, RSS, cache and TLB behavior. + +The memory optimizations are treated as performance aids rather than correctness requirements; failure to obtain a huge-page hint is non-fatal. + +## Unstructured Data Pipeline + +The ingestion path is designed for data that is useful before it is clean. + +A native tokenizer extracts identifiers from noisy input. String-like fields can then be projected into a compact 64-bit keyspace through FNV-1a and searched using the same indexed lookup machinery as native integer identifiers. + +The optional context model consumes a bounded byte window around a hit and emits one of six semantic classes with confidence gating. This model is deliberately small enough to execute directly in the native pipeline rather than requiring a general ML runtime for every classification. + +## Archive Support + +When `libarchive` and `libzstd` are available, KEYSTONE can participate directly in `.tar.zst` processing. Member offsets and extracted identifiers can feed the same lookup/index structures used by non-archive data. + +Archive support is optional and is not required by the core numeric search engine. + +## QIHSE Bridge + +KEYSTONE can be compiled as a native preprocessing/ingestion layer for QIHSE: + +```bash +make clean +KEYSTONE_ENABLE_QIHSE_BRIDGE=1 QIHSE_ROOT=/path/to/QIHSE make +``` + +The integration is additive. Applications can continue using KEYSTONE directly without QIHSE, or use KEYSTONE to parse/index/classify input before structured results are passed into QIHSE. + +See [INTEGRATION.md](INTEGRATION.md) for integration details. + +## Native Build Philosophy + +The default deployment posture is target-native rather than lowest-common-denominator portability. The build can use `-O3 -march=native` and enable locally supported execution paths. + +This means benchmark results should always be interpreted with their build configuration and hardware attached. A result produced on one CPU or with one optional backend is not a universal performance guarantee. + +See [BUILD_MODES.md](BUILD_MODES.md) for supported switches and reproducible comparison builds. + +## Performance Measurement + +Performance work should record at minimum: + +- host CPU and microarchitecture; +- compiler and exact flags; +- optional feature toggles; +- dataset size/distribution; +- query count/order/hit rate; +- warmup/cache policy; +- selected backend and decision source; +- thread count; +- transfer costs for any accelerator backend; +- raw benchmark output or CSV. + +The repository's benchmark notes deliberately separate measured host-specific results from architectural estimates. See [BENCHMARK_RESULTS.md](BENCHMARK_RESULTS.md). + +## Current Boundaries + +KEYSTONE should not presently be described as having production GPU, NPU, or AMX search backends solely because source files, detection logic, or experimental accelerator work exists. Backend support is considered real only after the public contract, fallback path, correctness checks, data movement, dispatch provenance, and hardware-specific measurements are established. + +For the current implementation/backlog boundary, see [STATUS_SUMMARY.md](STATUS_SUMMARY.md).