[FBumann]

Prompt: Let's check how we can make specsolve faster for small models 100-10k variables

Prompt: DO it. Do it as stacked PRs

Note

The following content was generated by AI.

Stacked on #2020. The build collects each stage's tables for every declaration in one pass: sources, variable labels, bounds, constraint labels, right-hand sides and linear terms. Port models build in 0.65–0.77× the time of #2020; dispatch in 0.78–0.81×.

What changed, method, measurements, mutation table

Why. Each declaration's tables were collected one query at a time. A polars query has a fixed cost however few rows it holds, so a small model with many declarations paid it dozens of times.

What.

• labels.Domain names the coordinates a declaration spans: its dims, its mask, and the presences its rows must lie in. labels.frames(scope, domains, label) reads several domains together, each numbered from 0. labels.frame is its one-domain case.
  • Every path orders a domain's coordinates by row-major position, and a label is the row in that order. So the plans are collected in one pass, and in one pass before it for the masks _factored ranks.
• Assembly._build_variables labels every variable in one pass and moves each onto its columns by adding to the label Series (_moved). It reads every variable's bounds in a second pass.
• Assembly._build_constraints:
  • Compiles every constraint into _Sides: its terms, quadratic terms and constants, each signed, and its restrictions.
  • Reads all constraints' labels, right-hand sides and linear terms in three passes.
  • Finishes each _block on its own rows, which run() moves onto the model's as chore(engine): a constraint's rows are numbered when the model is frozen rather than while it is built #2020 does.
  • No row is numbered provisionally, so the shift the first version of this PR carried is gone.
• coverage.constant_side returns its plan, and refuse_uncovered checks the read.
• attaching.attach reads every source table in one collected_all.

Method. Engine.build from tidied sources, the median of 5–15 builds per process, three alternating process runs per arm, the median of those. 4-vCPU cloud container, started at a 1-minute load below 1.5. Base: #2020's head.

model	variables / constraints	#2020 ms	this ms	ratio
genx_piecewise_fuel	8 / 15	190.0	124.0	0.65
osemosys_utopia	2 / 6	103.8	71.9	0.69
pypsa_unit_commitment	4 / 7	78.6	55.3	0.70
pypsa_committable_extendable	3 / 4	55.1	40.6	0.74
pypsa_ac_dc	6 / 8	87.6	65.1	0.74
pypsa_storage	5 / 5	79.8	61.5	0.77
network/2xs, network/xs		51.7, 57.4	35.9, 43.9	0.69, 0.77
nodal/2xs, nodal/xs		20.3, 23.0	16.3, 18.6	0.80, 0.81
dispatch/3xs, dispatch/xs	1 / 1	16.5, 20.6	13.4, 16.2	0.81, 0.78
network/m (1M)		319.0	314.6	0.99
dispatch/m (1M)		100.9	108.9	1.08

dispatch/m ran 96–101 ms on #2020 and 105–125 ms here; with one constraint it has nothing to batch, so a small cost there may be real.

Mutation table, taken by hand: tools.mutate cannot express an edited expression. Clean tree, restored with git checkout --, __pycache__ dropped before and after each run.

mutation	result
the variables' column offset (_moved(labelled, 'var_label', self.n_cols) → labelled)	caught, 596 failed
the uncovered-row refusal (if gaps: → if False: in refuse_uncovered)	still green

The refusal moved here unchanged from constant_side. The same mutation on main's code, at #2020's head, also leaves the suite green (5593 passed). So no test reaches that error today. That is an existing gap and is not addressed here.

Coverage moved. test_the_bounds_are_collected_in_memory records pl.collect_all as well as LazyFrame.collect. bench/profile_build.py and bench/profile_phases.py name _build_variables and _build_constraints.

Behaviour. A model with errors in several constraints can report a different one first, because every constraint is compiled and labelled before any is checked. No test asserts that order.

Gates. pixi run lint, pyrefly check on src/, ruff format --check ., pixi run test: 5593 passed. pixi run test-bench: 411 passed, run with --i-know-another-is-running after the suite raised the load, not to take a measurement.

Not done.

• The objective, the quadratic shares, and labels.declared_height for each restricted constraint still collect one at a time.
• The two fast paths that compute the same result a second way, _factored and _dense_row, stay. Removing either needs its own measurement.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YXphRT4nkmgh8LgdYMgCj

[read-the-docs-community]

Documentation build overview

📚 specsolve | 🛠️ Build #35112657 | 📁 Comparing b1cb527 against latest (b646d61)

  🔍 Preview build  

1 file changed

± about/changelog/index.html

[codspeed]

Merging this PR will regress 4 benchmarks

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 6 improved benchmarks
❌ 4 regressed benchmarks
✅ 94 untouched benchmarks
⏩ 307 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_window[storage-s-specsolve-highs-values]	10.5 MB	11.7 MB	-10.03%
❌	test_window[storage-s-specsolve-highs-one]	10.5 MB	11.7 MB	-10.03%
❌	test_window[network-s-specsolve-highs-one]	13.9 MB	14.8 MB	-6.34%
❌	test_window[network-s-specsolve-highs-values]	13.9 MB	14.8 MB	-6.34%
⚡	test_window[fleet-s-specsolve-highs-values]	11.8 MB	10.4 MB	+13.3%
⚡	test_window[fleet-s-specsolve-highs-one]	11.8 MB	10.4 MB	+13.3%
⚡	test_emit[fleet-s-specsolve-highs]	21.1 MB	19.7 MB	+7.02%
⚡	test_emit[fleet-s-specsolve-in-memory-highs]	21.1 MB	19.7 MB	+7.01%
⚡	test_window[fleet-s-specsolve-highs-cold]	21.2 MB	19.8 MB	+6.98%
⚡	test_window[fleet-s-specsolve-highs-shape]	21.1 MB	19.8 MB	+6.98%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/batched-build (b1cb527) with claude/rows-numbered-at-freeze (77de082)}

Footnotes

1. 307 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[FBumann]

Note

The following content was generated by AI.

Replaced by #2022, which carries this branch and #2020's commits in one PR against main. GitHub would not change this PR's base while it was stacked on #2020.

---

Generated by Claude Code