Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,102 @@ This rerun compares the same CLI workload across uv-managed CPython 3.14.6, CPyt
- **Go remained the fastest CLI path overall**, mainly because the conversion work dominates process startup once the payload gets large.
- These numbers are **end-to-end subprocess timings**, not isolated serializer throughput, so interpreter startup and environment activation costs are part of the result by design.

### Security Hardening Benchmark (August 12, 2026)

This comparison measures the public `Json2xml(...).to_xml()` path before and after conversion limits, lexical pretty printing, and compact output by default were added.

It compares pre-hardening commit `826439f` with hardened `master` at `48dfd38`. The direct `dicttoxml` serializer did not change between these revisions, so `benchmark_all.py` would not expose the wrapper cost.

#### Method

- Apple Silicon and macOS 26.6.1, with CPython 3.14.6 for the primary run.
- CPython 3.15.0rc1 follow-up downloaded and managed by uv 0.12.3.
- Deterministic small, 100-record, and 1,000-record nested payloads generated by `benchmark_security_hardening.py`.
- Default, explicit `pretty=False`, and explicit `pretty=True` calls.
- Five warmups per worker and 68 timed samples per cell across four fresh workers per revision.
- Revisions were interleaved in mirrored ABBA and BAAB orders to give each revision every worker position.

The payload generator is deterministic by construction: record fields are derived only from each record's zero-based index, so there is no pseudorandom generator or external seed to supply. The committed harness contains the exact payloads, loop counts, worker order, timing code, and UTF-8 output checks.

It resolves both revisions to full commit IDs, creates detached temporary worktrees, and launches a fresh Python process for every worker. Each ABBA/BAAB pass contributes two workers per revision; two passes × 17 samples produce the documented 68 samples per revision and workload/mode cell.

Every child process receives an argv list with shell parsing disabled. User-supplied revision text is placed after Git's `--end-of-options` marker, and only the resulting full commit ID is passed to `git worktree add`; shell escaping is neither needed nor used.

#### CPython Results

Each timing is the median public-API conversion time. “Faster” and “slower” compare hardened `master` with the pre-hardening revision.

| Workload | Mode | Pre-hardening | Hardened master | Change |
|----------|------|--------------:|----------------:|-------:|
| Small | Default | 24.1µs | 6.3µs | **74.0% faster** |
| Small | Compact | 4.4µs | 6.4µs | **44.9% slower** |
| Small | Pretty | 24.3µs | 15.1µs | **38.0% faster** |
| 100 records | Default | 7.07ms | 1.80ms | **74.6% faster** |
| 100 records | Compact | 1.14ms | 1.83ms | **60.6% slower** |
| 100 records | Pretty | 6.87ms | 5.04ms | **26.7% faster** |
| 1,000 records | Default | 86.79ms | 17.44ms | **79.9% faster** |
| 1,000 records | Compact | 10.92ms | 17.40ms | **59.3% slower** |
| 1,000 records | Pretty | 85.03ms | 49.26ms | **42.1% faster** |

#### CPython 3.15.0rc1 Results

The rc1 follow-up uses the same payloads, modes, warmups, sample counts, worker isolation, and mirrored ABBA/BAAB revision ordering as the CPython 3.14.6 run.

uv 0.11.24 did not yet list rc1, so it was updated to 0.12.3. That release downloaded the exact 26.2 MiB Apple Silicon build and verified its version before measurement.

```bash
uv self update
uv python list 3.15 --all-versions
uv run --isolated --managed-python \
--python cpython-3.15.0rc1-macos-aarch64-none \
--with defusedxml \
--with 'urllib3>=2.7.0' \
python benchmark_security_hardening.py \
--before 826439f \
--after 48dfd38 \
--output-json /tmp/json2xml-security-benchmark-python315rc1.json
```

The console output contains the median and interquartile range plus output type, UTF-8 byte count, and full SHA-256 for every cell. The optional JSON file preserves all 1,224 raw timings: 9 cells × 2 revisions × 68 samples.

Each result is `median [p25, p75]`. The narrow hardened ranges contrast with the wider old DOM-pretty ranges on larger payloads.

| Workload | Mode | Pre-hardening median [p25, p75] | Hardened median [p25, p75] | Change |
|----------|------|--------------------------------:|----------------------------:|-------:|
| Small | Default | 24.7µs [24.6, 25.0] | 6.2µs [6.2, 6.3] | **74.8% faster** |
| Small | Compact | 4.4µs [4.4, 4.4] | 6.4µs [6.3, 6.4] | **44.0% slower** |
| Small | Pretty | 24.7µs [24.5, 24.8] | 14.8µs [14.7, 14.9] | **40.0% faster** |
| 100 records | Default | 7.13ms [6.65, 8.50] | 1.75ms [1.74, 1.76] | **75.5% faster** |
| 100 records | Compact | 1.09ms [1.09, 1.11] | 1.75ms [1.74, 1.79] | **60.3% slower** |
| 100 records | Pretty | 6.85ms [6.61, 8.47] | 4.96ms [4.90, 5.03] | **27.5% faster** |
| 1,000 records | Default | 88.01ms [62.86, 100.53] | 17.24ms [17.17, 17.30] | **80.4% faster** |
| 1,000 records | Compact | 10.90ms [10.87, 10.98] | 17.21ms [17.15, 17.28] | **57.8% slower** |
| 1,000 records | Pretty | 88.83ms [63.04, 102.21] | 50.46ms [49.61, 51.15] | **43.2% faster** |

#### Output Checks

Compact output stayed byte-for-byte identical across revisions. Default and pretty output sizes differ because default now returns compact bytes and lexical pretty formatting replaces `minidom` formatting.

| Workload | Compact bytes, both revisions | Default type/bytes, before → after | Pretty type/bytes, before → after |
|----------|------------------------------:|----------------------------------:|---------------------------------:|
| Small | 134 | `str`/142 → `bytes`/134 | `str`/142 → `str`/146 |
| 100 records | 57,049 | `str`/64,551 → `bytes`/57,049 | `str`/64,551 → `str`/69,952 |
| 1,000 records | 571,005 | `str`/646,007 → `bytes`/571,005 | `str`/646,007 → `str`/700,008 |

The compact-output SHA-256 values are `013c28cd992bb5caa9a3c357a492ab47ccf00ad479ad3010e7ec376754cb06a8` (small), `05d79278d394c70a816fce23aa60e2fd0059233abf94c863a26e17af4dcdb090` (100 records), and `72de1be56cbc381c25606a3ca5c73b40d482ee08ef2d74f93d3b2798e531d760` (1,000 records).

#### Interpretation

Default calls are roughly 4-5x faster because they now return compact serializer bytes instead of building pretty output. This comparison includes the intentional default-output contract change.

Explicit pretty output is 27-42% faster because bounded lexical indentation replaces DOM parsing. Its formatting and encoded size differ from the old `minidom` output.

Explicit compact output is 45-61% slower. Compact bytes were identical across revisions, and the serializer was unchanged, isolating the regression mainly to the new full-input resource-budget scan.

CPython 3.15.0rc1 reproduces the same tradeoff: default and pretty output improve substantially, while explicit compact conversion pays for the security scan.

PyPy 3.10.16 corroborated the tradeoff: default calls were 73-87% faster, pretty calls were 42-72% faster, and compact calls were 31-35% slower.

## Key Observations

### 1. Rust Extension is the Best Choice for Python Users 🦀
Expand Down Expand Up @@ -235,6 +331,17 @@ and PyPy 3.11.15 under `JSON2XML_UV_PYTHON_DIR` (default:
python benchmark_multi_python.py
```

### Security Hardening Public-API Benchmark

Compares two commits through `Json2xml(...).to_xml()` in default, compact, and pretty modes. The script creates and removes detached temporary worktrees automatically; it does not alter the current checkout.

```bash
python benchmark_security_hardening.py \
--before 826439f \
--after 48dfd38 \
--output-json /tmp/json2xml-security-benchmark.json
```

### Interpreting CLI Numbers

The Go and Zig rows measure full process startup plus conversion because
Expand Down
Loading