Skip to content

perf: skip guest tracing work while the log level is off - #1840

Merged
jprendes merged 2 commits into
hyperlight-dev:mainfrom
jprendes:guest-tracing-off-overhead
Sep 18, 2026
Merged

jprendes merged 2 commits into
hyperlight-dev:mainfrom
jprendes:guest-tracing-off-overhead

Conversation

@jprendes

Copy link
Copy Markdown
Contributor

Closes #1837.

A guest log level of OFF skips the per-call tracing work: TSC reads, span
setup, trace state locking, and flush plumbing.

Behavior

  • is_trace_enabled and accepts_trace_events derive from a single relaxed
    atomic holding the encoded filter, so neither upgrades a Weak.
  • register_callsite caches interest per level. A level change rebuilds the
    interest cache, so runtime level updates take effect.
  • generic_init and the dispatch path open their spans only while tracing is
    active.
  • end_trace, flush, new_call, reset, and serialized_data return early
    while tracing is off, which keeps out32 clear of trace state locking.

hyperlight_guest_tracing::is_trace_enabled reports whether the configured
level is above OFF. It is public, so CHANGELOG.md carries an entry.

Why trace_guest stays

The feature and the log level control different things, and both earn their
place.

trace_guest is a build-time choice about whether a guest carries the tracing
machinery at all. It gates the subscriber, the trace state, and the batch
encoder. A guest built without it pays no binary size for code it never runs,
which matters for small guests.

The log level is a runtime choice. MultiUseSandbox::log_level raises and
lowers it while the guest runs, so the call sites keep a check that survives
compilation. That check is one relaxed atomic load at the boundaries above,
plus the callsite interest tracing already caches. Dropping the feature would
compile the machinery into every guest and still leave that check in place, so
it trades binary size for nothing.

Fixing the level at build time with tracing/max_level_off is the only way to
erase the check, and it costs the runtime control that
MultiUseSandbox::log_level provides.

Testing

  • max_guest_log_level_can_be_disabled_and_reenabled covers TRACE to OFF to
    TRACE on a live sandbox.
  • max_guest_log_level_can_be_enabled_after_off_init covers a sandbox built at
    OFF and raised at runtime, and asserts it matches a sandbox built at TRACE.
  • callsite_interest_tracks_filter_updates covers the interest contract.
  • The existing log_message integration test asserts exact span enter and exit
    counts, which covers the span gating.

An off guest log level avoids TSC reads, span setup, trace state locking,
and flush plumbing on every guest call.

* `is_trace_enabled` and `accepts_trace_events` derive from one relaxed
  atomic holding the encoded filter.
* `register_callsite` caches per-level interest, and a level change
  rebuilds the interest cache.
* `generic_init` and the dispatch path open spans only while tracing is
  active.

Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Copilot AI lite review requested due to automatic review settings September 17, 2026 21:38
@jprendes jprendes added the kind/refactor For PRs that restructure or remove code without adding new functionality. label Sep 17, 2026
@jprendes jprendes added area/performance Addresses performance ready-for-review PR is ready for (re-)review labels Sep 17, 2026
ludfjig
ludfjig previously approved these changes Sep 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Callsite caches may remain disabled after tracing is re-enabled, and the new regression tests are omitted from coverage.

Pull request overview

Improves guest tracing performance when the log level is OFF while preserving runtime re-enablement.

Changes:

  • Adds atomic filter checks and early exits for tracing work.
  • Updates callsite interest handling and span gating.
  • Adds runtime tests, isolated test commands, and changelog documentation.
File summaries
File Description
src/hyperlight_host/src/sandbox/initialized_multi_use.rs Adds runtime tracing-level tests.
src/hyperlight_guest/src/guest_handle/host_comm.rs Gates trace event forwarding.
src/hyperlight_guest_tracing/src/subscriber.rs Updates dynamic callsite interest caching.
src/hyperlight_guest_tracing/src/lib.rs Adds atomic filter checks and tracing early exits.
src/hyperlight_guest_bin/src/lib.rs Gates initialization spans.
src/hyperlight_guest_bin/src/guest_function/call.rs Gates per-call tracing work.
Justfile Registers isolated test commands.
CHANGELOG.md Documents the public behavior change.
Review details

Suppressed comments (2)

Justfile:249

  • The coverage-run recipe has its own isolated-test list at Justfile:531-537, but it does not include either new regression test. The coverage workflow therefore skips these tests; add both commands there as well.
    {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_can_be_disabled_and_reenabled --exact --ignored
    {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_can_be_enabled_after_off_init --exact --ignored

src/hyperlight_guest_tracing/src/lib.rs:81

  • init_guest_tracing can be reached after a guest initialized at OFF has already registered instrumented callsites with the default dispatcher, which caches Interest::never(). This branch installs the subscriber but does not rebuild that cache, so those callsites remain disabled after raising the level to TRACE; rebuild the cache after set_global_default succeeds.
        if tracing_core::dispatcher::set_global_default(tracing_core::Dispatch::new(sub)).is_ok() {
            store_max_log_filter(max_log_level);
        }
  • Files reviewed: 8/8 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

`coverage-run` carries its own isolated test list, so it names the
disable and re-enable test and the off init test alongside the other
guest log level tests.

Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Comment thread src/hyperlight_host/src/sandbox/initialized_multi_use.rs
@jprendes
jprendes merged commit 2a4a2ba into hyperlight-dev:main Sep 18, 2026
52 checks passed
@github-actions github-actions Bot removed the ready-for-review PR is ready for (re-)review label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performance Addresses performance kind/refactor For PRs that restructure or remove code without adding new functionality.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

setting the tracing level to OFF should remove all the tracing overhead

3 participants