ContainerBench

ContainerBench is an open-source container image benchmarking framework. It serves as both a reproducible academic research artifact for empirical container evaluation and a DevOps utility to automate container optimizations.

Methodology

ContainerBench automatically builds, verifies, and analyzes container images across various optimization strategies. For each benchmark run, the framework captures:

1. Build Performance: Measures raw build time (build_time_seconds) using high-resolution performance counters.
2. Storage Metrics:
   • Image Size (image_size_mb): Extracted directly from docker inspect (converting bytes to Megabytes).
   • Layer Count (layer_count): Determined by counting filesystem layers in the image metadata.
3. Runtime Metrics:
   • Startup Validation: Maps exposed ports dynamically to avoid localhost conflicts, performs dynamic HTTP validation, and records the container initialization duration (startup_time_seconds).
4. Security Vulnerabilities: Integrates with Trivy to fetch CVE counts programmatically by severity class (Critical, High, Medium, Low, Unknown).
5. System Context: Dumps system environment details (OS version, CPU, RAM, Python version, Docker version) once per session to maintain reproducible research.

---

Express Pilot Experiment Results

An automated pilot experiment was conducted on the Express workload, executing 5 repetitions for each of the 6 optimization strategies (total of 30 runs). The raw readings derived from express-pilot.csv yield the following aggregated statistics:

Strategy	Mean Build Time (s)	Image Size (MB)	Layer Count	Mean Startup Time (s)	Security Scan	Overall Status
baseline	1.87s	391.28 MB	12	1.13s	SKIPPED	PASS
alpine	1.60s	56.60 MB	8	1.14s	SKIPPED	PASS
slim	1.52s	78.03 MB	9	1.11s	SKIPPED	PASS
multistage	2.23s	390.08 MB	10	1.12s	SKIPPED	PASS
alpine-multistage	1.93s	55.39 MB	6	1.12s	SKIPPED	PASS
distroless	2.07s	50.75 MB	21	1.15s	SKIPPED	PASS

---

Django Pilot Experiment Results

An automated pilot experiment was conducted on the Django workload, executing 5 repetitions for each of the 6 optimization strategies (total of 30 runs). The raw readings derived from experiments.csv yield the following aggregated statistics:

Strategy	Mean Build Time (s)	Image Size (MB)	Layer Count	Mean Startup Time (s)	Security Scan	Overall Status
baseline	19.00s	398.00 MB	11	1.13s	SKIPPED	PASS
alpine	7.43s	27.17 MB	8	1.09s	SKIPPED	PASS
slim	5.52s	51.14 MB	8	1.10s	SKIPPED	PASS
multistage	7.28s	51.13 MB	7	1.09s	SKIPPED	PASS
alpine-multistage	7.48s	27.13 MB	7	1.11s	SKIPPED	PASS
distroless	10.11s	29.32 MB	46	1.08s	SKIPPED	PASS

---

Key Findings and Conclusions

1. Storage Optimization Winner:
   • For Express, Distroless produced the smallest overall image footprint (50.75 MB), closely followed by alpine-multistage (55.39 MB).
   • For Django, Alpine-Multistage yielded the smallest container footprint (27.13 MB), closely followed by alpine (27.17 MB) and distroless (29.32 MB).
2. Layer Count Trade-off:
   • In both workloads, distroless images introduce the highest layer complexity (e.g., 46 layers in Django, 21 layers in Express) due to package dependency chain imports.
   • Conversely, alpine-multistage combines optimized sizes with minimal layer overhead (6 layers for Express, 7 layers for Django), making it the most balanced strategy.
3. Speed & Efficiency:
   • Slim (Debian Slim) delivered the fastest build times across both workloads (1.52s average for Express, 5.52s average for Django) and lowest startup latency, making it highly suitable for rapid development loops.
4. Framework Overhead: Python/Django workloads experience significantly higher build-time overhead in baseline configurations compared to JavaScript/Express workloads, highlighting the necessity of multi-stage and slim image optimization strategies in Python-based microservices.