Skip to content

perf: removing a stage takes stage1 analysis from 40 s to over 600 s #537

Description

@MarcusKainth

What kind

Cost grows where the model says it should not

Which benchmark

No make target. clickdoom native demo demo3 --from sim --no-window --times-out <path>, reading the first tic's stage1_us, which is the
statement's own analysis.

The numbers

ClickHouse 26.8.2.7, main at 89e73f9, one throwaway container, machine lock
held. Statement sizes are from the generator, not system.query_log, so they
are not truncated.

  statement                       stage1 bytes   stage1 analysis
  the tic's first statement          2,015,048          39.74 s
  the same, with the last stage
  of mobj::thinkers removed          1,850,587      did not finish

The second statement is 164,461 bytes smaller, 8.2% less text, and does
strictly less work: it is the first statement with the call to
mobj::thrown_thinks left out and nothing else changed. The driver gave up
on it at FIRST_TIC_TIMEOUT, and the server still listed it in
system.processes at 660 s.

  clickdoom: error: a statement failed: the simulation's own first
  statement: the server did not answer within 60s of the body closing
  (system.processes still lists it, 660.0s elapsed)

So removing one stage took the analysis from 39.74 s to over 600 s, a
factor of at least 15, while making the statement smaller.

For scale, the same experiment over eleven other cut points behaves the way
you would expect, monotonically, from 0.10 s for a statement cut after the
tic command to 39.74 s for the whole thing. Only this one cut inverts.

The machine, and how quiet it was

Apple M-series laptop, macOS 25.6. Machine lock held, one ClickHouse
container and nothing else running, load average 2.93 at the start. No other
lane's container was up.

What you think is happening

I do not know, and I am reporting it rather than explaining it.

NATIVE.md already says analysis cost is not a function of the statement's
size and that where it comes from is not settled, so a smaller statement
costing more is not by itself a contradiction of the model. What is worth
recording is the size of the effect: a factor of at least 15 from removing
one stage, where every other cut moved the cost smoothly.

The cut was produced by a local-only switch in bindings_stage1 that
returns the Tic before mobj::thrown_thinks is staged. Nothing else
differs, and the resulting statement passes the same binding-order and
parenthesis-balance checks the committed one does. It is possible the fault
is in that switch rather than in the server, and I could not rule that out
from the outside, which is part of why this is worth someone else looking
at.

This matters beyond curiosity because the projection is 64% of stage1's
analysis (25.5 s of 39.7 s) and is the obvious thing to restructure. If
analysis cost is not monotonic in what a statement contains, then a rewrite
that plainly does less work cannot be assumed to analyse faster, and every
such change needs measuring rather than reasoning about.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceThroughput below expectation, or a regression

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions