Storage Benchmark Kit (SBK)

SBK is a Java framework for measuring the throughput and latency of storage systems with one common workload engine. Its drivers cover object stores, message systems, databases, file systems, caches, and local queues. The harness controls concurrency, duration, rate, payloads, timestamps, and reporting; each driver only adapts those operations to a backend.

Repository: https://github.com/kmgowda/SBK

Start here

Choose the guide that matches your task:

Goal	Documentation
Build and run SBK	This README
Build driver-scoped Docker images	Docker build and operations guide
Understand modules and runtime flow	Architecture and code flow
Browse all documentation	Documentation index
View live graphs without Docker	WebLogger guide
Retain live metrics with Prometheus and Grafana	PrometheusLogger and SBK Dashboard guide
Add or modify a storage driver	Driver guide
Make a code contribution	Contributing guide
Follow a task-specific procedure	Engineering recipes
Work as a coding agent	Agent guide
Study measurement internals in depth	Internal design

Architecture in one minute

flowchart LR
    CLI[SBK CLI or YML] --> BOOT[Sbk bootstrap]
    BOOT --> DRIVER[Storage driver]
    BOOT --> BENCH[SbkBenchmark]
    BENCH --> WORKERS[Writer and reader workers]
    WORKERS --> DRIVER
    WORKERS --> CHANNEL[PerL channels]
    CHANNEL --> RECORDER[Latency recorder]
    RECORDER --> OUTPUT[Console / CSV / Local Web Console / Prometheus / gRPC]
    OUTPUT -->|gRPC mode| SBM[SBM aggregator]
    GEM[SBK-GEM] -->|SSH orchestration| CLI
    GEM --> SBM

Loading

The main runtime path is:

1. io.sbk.main.SbkMain delegates to io.sbk.api.impl.Sbk.
2. Sbk discovers the requested driver and logger by class name, builds the combined CLI, parses it, and constructs SbkBenchmark.
3. SbkBenchmark opens the storage, creates driver readers and writers, starts PerL recorders, and schedules worker execution.
4. Driver operations are timed by the Writer and Reader default methods and submitted to a PerlChannel.
5. PerL processes every latency record away from the worker threads and publishes periodic and total results through the selected logger.

See docs/ARCHITECTURE.md for source links, lifecycle details, concurrency boundaries, and distributed execution.

Requirements

• JDK 25, either preinstalled, downloaded automatically by the wrapper, or installed on demand by a packaged launcher
• Git
• The Gradle wrapper included in the repository; a separate Gradle installation is not required
• A real backend only when exercising a remote-storage driver

Confirm the active JVM before building:

java -version
./gradlew --version

The Gradle wrappers resolve a complete JDK 25 in this order:

1. SBK_JAVA_HOME;
2. the persisted SBK_JAVA_HOME selection from the first successful build;
3. JAVA_HOME;
4. a JDK 25 available through PATH; and
5. the SBK-managed JDK cache.

If none is available, gradlew downloads the pinned OpenJDK 25 release, verifies its SHA-256 checksum, and installs it without administrator access in ${XDG_CACHE_HOME:-$HOME/.cache}/sbk/jdks on Unix or %LOCALAPPDATA%\SBK\jdks on Windows. Later wrapper invocations reuse that selection through sbk-java-home.properties in the same directory. The wrapper and generated launchers export the resolved path as SBK_JAVA_HOME to Gradle and the Java application process. A child process cannot modify its parent shell environment, so this per-user file provides persistence between invocations. Set SBK_JAVA_CACHE_DIR to select a different cache directory. Automatic installation supports Linux x64/AArch64, macOS x64/AArch64, and Windows x64. Other platforms can use a manually installed JDK through SBK_JAVA_HOME. The pinned version, platform checksums, and bootstrap network/lock timeouts have one authoritative source: gradle/sbk-java-bootstrap.properties. An invalid explicit SBK_JAVA_HOME, or an invalid JAVA_HOME when no higher priority persisted selection exists, is treated as an error so a selected configuration mistake is never hidden by an automatic download.

Generated launchers for sbk, sbk-yal, sbm, sbk-gem, sbk-gem-yal, and sbk-web-console use the same persisted selection and export it as SBK_JAVA_HOME. By default they do not access the network: run the Gradle wrapper once to establish the persisted selection, or set SBK_JAVA_HOME/JAVA_HOME.

Self-bootstrap a packaged application

The standard distTar, distZip, and installDist outputs contain the Java bootstrap scripts and their pinned-version/checksum configuration, but they do not embed a JDK. On a supported machine without JDK 25, set SBK_JAVA_INSTALL=true for the first packaged-application invocation:

SBK_JAVA_INSTALL=true ./bin/sbk -class file -writers 1 -size 100 -seconds 60
SBK_JAVA_INSTALL=true ./bin/sbm -class file -action w
SBK_JAVA_INSTALL=true ./bin/sbk-gem -nodes "host1 host2" -class file \
  -writers 1 -size 100 -seconds 60

On Windows Command Prompt, use:

set SBK_JAVA_INSTALL=true
bin\sbk.bat -class file -writers 1 -size 100 -seconds 60

The launcher downloads the pinned JDK, verifies its SHA-256 checksum, installs it in the per-user SBK Java cache, persists the selection, and starts the requested application. The cache and persisted selection are shared by all packaged SBK applications, so later invocations normally require neither the environment variable nor another download. An explicit invalid SBK_JAVA_HOME or JAVA_HOME remains an error and is never silently replaced. Keep SBK_JAVA_INSTALL unset or set it to false to retain the default offline/no-download behavior.

This setting bootstraps Java for the local application process. After sbk-gem starts, its independent remote-deployment workflow probes each node and provisions the selected compact or full remote Java runtime when required.

Runtime JVM and garbage collector defaults

Every generated SBK launcher uses one consolidated JDK 25 runtime profile from gradle/java.gradle:

-XX:+UseZGC
-XX:+UseCompactObjectHeaders
-XX:MaxRAMPercentage=50.0
-XX:+DisableExplicitGC
-XX:+ExitOnOutOfMemoryError

• Generational ZGC performs expensive collection work concurrently and dynamically scales generation sizes and GC threads. This keeps GC pauses small as benchmark concurrency, CPU count, and live heap grow.
• Compact object headers reduce per-object heap overhead and improve cache density for queues and measurement records.
• 50% maximum RAM gives large benchmark processes more heap than the JVM's default server sizing while retaining half of detected memory for SDK native buffers, direct buffers, thread stacks, filesystem cache, and the operating system.
• Disabled explicit GC prevents a driver or dependency from injecting a benchmark-distorting System.gc() collection. Necessary collections still run normally.
• Exit on OOM prevents a memory-exhausted benchmark from continuing and publishing misleading results.

The same options are applied to sbk, sbk-yal, sbm, sbk-gem, sbk-gem-yal, module launchers, and the SbkWebConsoleMain process.

ZGC reserves address space using many memory mappings. On large-memory Linux hosts, check sysctl vm.max_map_count; if it is too small, JDK 25 prints the minimum required value at startup. Configure that operating-system limit before a production benchmark.

These are safe cross-host defaults, not a substitute for capacity planning. For a dedicated benchmark host, JAVA_OPTS can override heap sizing. For example, a fixed heap removes resizing and commit/uncommit variation:

export JAVA_OPTS='-Xms32g -Xmx32g -XX:+AlwaysPreTouch'

Use fixed pre-touched heaps only when that memory is reserved for the process. Large pages are not enabled automatically because they require operating-system provisioning. To compare G1 with ZGC, explicitly disable ZGC before selecting G1:

export JAVA_OPTS='-XX:-UseZGC -XX:+UseG1GC'

Build

Clone and build the project:

git clone https://github.com/kmgowda/SBK.git
cd SBK
./gradlew check
./gradlew :installDist

The installed launcher is created at:

build/install/sbk/bin/sbk

Useful development commands:

# Compile, check style, and run tests for the whole build
./gradlew check

# Iterate on one driver
./gradlew :drivers:minio:check

# Generate launch scripts and runtime libraries
./gradlew :installDist

# Rebuild the pathing JAR after dependency changes
./gradlew clean :pathingJar :installDist --rerun-tasks

The leading : selects only the root distribution task. Omitting it also matches every driver's installDist task and creates dozens of standalone driver distributions that are not needed for the complete SBK package. Gradle's build cache and configuration cache are enabled by default. For local iteration, prefer the narrowest module task; CI retains the full check gate. SBK keeps reusable build outputs in .gradle/sbk-build-cache. The root ./gradlew clean task removes that cache, the repository-local configuration cache, and every project build directory. It deliberately preserves shared dependency, wrapper, and toolchain caches under ~/.gradle, which may belong to unrelated projects. A separate ./gradlew check after clean therefore executes all standard test tasks that have test sources instead of restoring their previous outputs from the build cache.

ChromaDB, HaloDB, and Ignite are present in the source tree but are not enabled in the aggregate build. ChromaDB's Java client adds an approximately 829 MiB all-platform local-embedding dependency closure; enable it explicitly only for ChromaDB benchmarks. HaloDB depends on a GitHub Packages artifact that may require credentials. The sbktemplate directory is a scaffold, not a runtime driver.

Run a benchmark

List drivers and common options:

./build/install/sbk/bin/sbk -help

Write to a local file for 30 seconds:

./build/install/sbk/bin/sbk \
  -class file \
  -file /tmp/sbk.bin \
  -writers 1 \
  -size 1048576 \
  -seconds 30

Read the same file:

./build/install/sbk/bin/sbk \
  -class file \
  -file /tmp/sbk.bin \
  -readers 1 \
  -size 1048576 \
  -seconds 30

Driver-specific options are added after SBK discovers -class. Use the selected driver with -help to see the merged option set:

./build/install/sbk/bin/sbk -class minio -help

Common workload options

Option	Meaning
-class NAME	Driver simple class name, matched case-insensitively
-writers N	Number of writer workers
-readers N	Number of reader workers
-size BYTES	Payload size per record
-seconds N	Time-based run duration
-records N	Total records in count mode, or the per-second target in timed mode
-throughput MBPS	Throughput target; -1 requests maximum throughput. A positive target must provide at least one whole record/s per worker.
-idletimeoutseconds N	In fixed-record mode, fail after N continuous seconds without a performance event; must exceed the logger reporting interval; default: 600
-mpscqueue true|false	Select intrusive TimeStampMpscQueue or the JDK ConcurrentLinkedQueue fallback; default comes from sbk.properties
-sync N	Records per flush/sync or transaction
-wstep N, -rstep N	Workers added at each ramp step; both must be greater than zero
-ro true	With readers and writers configured, read without writing new records
-thread p|f|v	Platform, fork-join, or virtual worker executor; default: virtual (v)
-out NAME	Output logger, such as SystemLogger, CSVLogger, WebLogger, PrometheusLogger, or GrpcLogger

Boolean options accept only true or false; misspellings fail during startup instead of silently selecting false. Thread mode likewise accepts only p, f, or v.

Always treat -help as authoritative because drivers and loggers add their own options at runtime.

The idle timeout applies only to fixed-record runs selected with -records and without -seconds. Each positive performance event renews the full interval; the benchmark fails only after one uninterrupted idle stretch. Timed runs do not use this deadline. The configured timeout must be strictly greater than the active logger's reporting interval so the recorder can observe an empty reporting window before declaring the benchmark idle. PerL raises the terminal failure, SBK performs bounded cleanup, and SBM uses the same rule for missing gRPC performance batches. SBK-GEM forwards the value and fixed-record mode to its remote SBK processes and embedded SBM.

String payloads in write/read latency workloads use a fixed 16-character hexadecimal timestamp header. It preserves the complete signed long range and leaves the record size and suffix unchanged. This replaces the older decimal header: regenerate timestamped string records and use the same SBK version for writers and readers; old decimal headers cannot be mixed with this format. Decimal digits are valid hexadecimal digits, so an old header can silently decode as a different timestamp instead of failing to parse. This can produce nonsensical latencies, including a run with 100% invalid latency records. Other payload types retain their existing timestamp formats.

Async readers may complete with null when no data is available. Such a completion produces no latency record. For consistency with synchronous readers, it increments the existing read-timeout counter when request accounting is active (getMaxReaderIDs() > 0; standard loggers enable it with -rq true). This counter includes empty reads, not just elapsed-time deadlines; an empty completion by itself is not a benchmark error. Fixed-count async workloads count submitted requests, so empty completions can make the measured record count smaller than the requested count. When all workers finish, SBK closes/drains readers and explicitly stops PerL; it does not necessarily wait for the idle timeout to make up missing records. The idle timeout can still fail a fixed-count run if workers remain active without measurements. Readers assigned a zero share of a fixed-record workload do not start and log the reason at startup.

Modules

Path	Responsibility
perl/	Performance Logger library: queues, latency windows, histograms, percentiles, and metrics
sbk-web-console/	Independent Local Web Console server/client runtime, protocol DTOs, and browser resources
sbk-api/	Storage and logger SPIs, CLI parsing, payload types, workers, and benchmark lifecycle
drivers/<name>/	Backend-specific adapters
sbm/	gRPC service that aggregates measurements from SBK clients
sbk-gem/	SSH-based multi-host launcher that embeds SBM
sbk-yal/	YML-to-SBK argument adapter
sbk-gem-yal/	YML-to-SBK-GEM argument adapter

The dependency direction is perl <- sbk-api <- drivers and sbk-web-console <- sbk-api. SBM depends on sbk-api; SBK-GEM depends on SBM. The YML launchers wrap their corresponding programmatic APIs.

Drivers and output loggers

Enabled drivers are registered in both settings-drivers.gradle and build-drivers.gradle. The complete categorized inventory and the driver contract are in docs/DRIVER_GUIDE.md.

Use the synthetic PerlBench driver to compare the intrusive PerL timestamp queue with the JDK fallback under exact-count, timed-saturation, and rate-controlled workloads. It performs no storage I/O, so its results describe harness and measurement-pipeline behavior rather than a storage device. Do not confuse it with the Null driver: Null's default operation deliberately remains pending for idle, timeout, and shutdown tests, while PerlBench completes every operation immediately.

SBK currently ships these logger implementations:

• SystemLogger: human-readable periodic and final output.
• Sl4jLogger: SLF4J-backed output.
• CSVLogger: results written in CSV form.
• WebLogger: console/CSV output plus the dependency-free SBK Local Web Console.
• PrometheusLogger: CSV behavior plus Prometheus metrics exposure.
• GrpcLogger: forwards measurements to SBM for distributed aggregation.

Prometheus metrics include stable component, class, and action labels. component="sbk" identifies a direct SBK exporter, while component="sbm" identifies SBM aggregation, including benchmarks orchestrated by SBK-GEM. SBK-GEM is not a separate metrics component because SBM owns its metrics endpoint and aggregated measurements.

PrometheusLogger and SBK Dashboard

Use PrometheusLogger when Prometheus should retain periodic results and Grafana should display a full historical dashboard:

./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-prometheus.dat \
  -writers 1 -size 4096 -seconds 60 \
  -out PrometheusLogger

Direct SBK metrics use http://<sbk-host>:9718/metrics by default. SBM and SBK-GEM aggregate metrics use http://<coordinator-host>:9719/metrics. Startup prints copy-paste scrape URLs for localhost, loopback, hostname, and usable host IPv4 addresses. Change the exporter with -context PORT/PATH, for example -context 19718/sbk-metrics.

The separately deployed SBK Dashboard manages official Prometheus and Grafana processes, persistent time-series history, and one endpoint-scoped dashboard per registered host:port. Its management UI defaults to port 9721 and Grafana to port 3000. See the PrometheusLogger and SBK Dashboard guide for direct and distributed commands, Docker Compose and native installation, endpoint registration, metrics semantics, retention, security, and troubleshooting.

SBK Local Web Console

Use WebLogger when you want live graphs without Docker, Prometheus, or Grafana:

./build/install/sbk/bin/sbk -class file -file /tmp/sbk.bin \
  -writers 4 -size 4096 -seconds 60 -out WebLogger

For a filesystem read benchmark, first create an input file and then read it with WebLogger. The record size must match between preparation and reading:

# Prepare a 1 GiB file: 1,048,576 records x 1,024 bytes.
./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-weblogger.dat \
  -writers 1 -size 1024 -records 1048576

# Measure filesystem reads for 60 seconds and display live graphs.
./build/install/sbk/bin/sbk \
  -class file -file /tmp/sbk-weblogger.dat \
  -readers 1 -size 1024 -seconds 60 \
  -out WebLogger

SBK opens the run URL in the default browser. The lightweight Local Web Console accepts plain HTTP on all IPv4 interfaces at port 9720, retains the latest 180 minutes of snapshots in memory, and streams new summaries with server-sent events. At benchmark start and completion, SBK prints run-specific URLs for localhost, IPv4 loopback, the hostname, and every usable private or public IPv4 address discovered on the console host. Change the retention duration with -websnapshotminutes N. A later SBK process reuses a compatible server already on that port. Multiple SBK, SBM, and SBK-GEM WebLogger benchmarks can publish concurrently to the same server; each receives a unique run URL and remains independently selectable in the browser. Completed graphs remain available while a browser is connected; after all benchmarks have finished and no browser has been connected for one minute by default, the server exits automatically. WebLogger reports whether it starts a new console or uses the existing process. Automatically started consoles append lifecycle, WebLogger count, browser/client count, and exit diagnostics to $HOME/.sbk/logs/sbk-web-console-<port>.log. Change the idle grace period with -webtimeoutminutes N. Snapshots and 15-second logger heartbeats renew the active-run lease. If a benchmark is killed without completing, the configured idle timeout without either signal marks only that run abandoned; other active runs continue unaffected. Use -webopen false on headless hosts and -webport PORT to select another port. Boards default to <application> <storage>, such as SBK File, SBM MinIO, or SBK-GEM Kafka; use -boardname NAME to override that display name. The timeout supplied by the process that starts a web console remains in effect when later benchmarks reuse it. A remote browser can use a printed hostname or IP URL when routing and firewall rules permit. Because the console has no authentication or TLS, expose it only on a trusted benchmark network or use an SSH tunnel. Run sbk -out WebLogger -help for the complete option set. See the WebLogger guide for every option, Local Web Console lifecycle, distributed usage, security, and troubleshooting.

Distributed monitoring uses the same Local Web Console and data model:

# Standalone SBM aggregate Local Web Console
sbm -out SbmWebLogger -class file -action r

# SBK-GEM aggregate Local Web Console; remote nodes continue sending results through GrpcLogger
sbk-gem -out GemWebLogger -class file -nodes host1,host2 -writers 2 -size 4096 -seconds 60

The Local Web Console binds to 0.0.0.0, while benchmark clients discover and reuse it through 127.0.0.1.

Distributed execution

• SBM accepts SBP/gRPC measurements and aggregates them.
• SBK-GEM copies and launches SBK on remote hosts over SSH while running an embedded SBM instance. Its default -fullcopy false mode uses a Gradle-defined compact Java runtime and per-driver dependency metadata to reduce both remote Java and SBK payloads. Set it to true for complete-JDK and complete-SBK copying; complete Gradle distributions are unchanged.
• SBK-YAL loads single-node SBK arguments from YML.
• SBK-GEM-YAL loads distributed SBK-GEM arguments from YML.

The default SBM gRPC port is 9717; Prometheus/JMX endpoints use separate configured ports. Review the component help before exposing any port outside a trusted benchmark network.

Measurement model

SBK records operation latency without sampling. Worker threads submit (start, end, records, bytes) measurements through PerL channels to concurrent queues. Dedicated recorder logic calculates window and total statistics and invokes the logger. This keeps histogram work out of the driver operation path.

Results are meaningful only when the experiment is controlled. Record at least:

• SBK version and commit SHA.
• Driver and vendor-client versions.
• Full command line and non-default properties.
• JVM, CPU, memory, network, and operating-system details.
• Storage topology and durability settings.
• Warm-up policy and whether results are cold-cache or warm-cache.

See the reproducibility section for a longer checklist.

Contributing

Before creating a release candidate, run the one-command automated release gate documented in SBK Release Qualification. Use its local-docker profile for automatic two-node SBK-GEM and SBK-GEM-YAL functional testing without a permanent SSH host. The release profile verifies installed launchers, logger contracts, PerL concurrency and performance, YAL mapping, SBM/gRPC aggregation, remote SBK-GEM execution, generated documentation, publications, and archive checksums.

Before publishing, run the non-mutating publication assembly locally:

./gradlew clean releasePublicationDryRun \
  --no-daemon --rerun-tasks

The separate guarded release publication workflow does not execute or depend on releasecheck. It builds the current version's reproducible archives, validates native AMD64/ARM64 containers, and can publish the distributions, SBOMs, manifest, and checksums to a detailed GitHub Release, the Java modules to GitHub Packages, and the signed multi-architecture GHCR image plus the digest-identical Docker Hub image. The root publish task uses local Docker Hub environment credentials without sending them to GitHub, then dispatches the workflow with only the immutable public image digest. See the publication guide for the required confirmation, credentials, and recovery controls.

For authorized maintainers, the final publication entry point is:

VERSION=$(sed -n 's/^sbkVersion=//p' gradle.properties)
DOCKER_USERNAME="<docker-user>" \
DOCKER_PASSWORD="<docker-access-token>" \
GITHUB_TOKEN="<actions-workflow-token>" \
./gradlew publish "-PreleaseConfirm=RELEASE-${VERSION}" --no-daemon

Run it only from a clean, synchronized master after separately qualifying the same commit. The command publishes Docker Hub and dispatches the asynchronous GitHub workflow; monitor that workflow and complete the post-release checks in the publication guide.

Read CONTRIBUTING.md before changing code. The minimum verification sequence is normally:

./gradlew :<module>:check
./gradlew check
./gradlew :installDist

Driver changes also require a smoke test against the relevant backend. Pull requests target master. Do not re-enable HaloDB or upgrade the intentionally pinned MinIO SDK without discussing the compatibility implications.

License and support

SBK is licensed under the Apache License 2.0. Use GitHub Issues for bugs and feature requests and GitHub Discussions for usage and design questions.