Skip to content

Distinguish TraceLogging events that share a name in the schema cache - #282

Merged
Matt Swann (swannman) merged 7 commits into
microsoft:masterfrom
chrdavis:fix/193-tracelogging-schema-key
Oct 5, 2026
Merged

Matt Swann (swannman) merged 7 commits into
microsoft:masterfrom
chrdavis:fix/193-tracelogging-schema-key

Conversation

@chrdavis

@chrdavis Chris Davis (chrdavis) commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Fixes #193.

TraceLogging events do not get a meaningful event id, so the schema cache key could not tell two events from the same provider apart. Adding the event name (#242) covered the common case, but a provider may log the same event name from several call sites with different fields. Those variants agree on provider, name, id, version, opcode, level and keyword, so they still collided and the first schema retrieved was used to decode all of them - silently shifting every field, and potentially reading past the end of the user data buffer.

The TraceLogging metadata is the schema, so hash it into the key. This keeps the cache effective for repeated events while separating variants, and costs no extra TdhGetEventInformation call. The hash is FNV-1a over a small blob; 0 is reserved to mean "no metadata".

Opt-in per provider

Hashing walks the whole metadata blob, so it costs time proportional to the size of the event. Only a handful of providers ever reuse an event name, so this is off by default and enabled per provider:

krabs::provider<> provider(guid);
provider.enable_trace_logging_full_schema_hashing();  // before trace.enable()
trace.enable(provider);
var provider = new Provider("Microsoft.Windows.AppLifeCycle.UI");
provider.EnableTraceLoggingFullSchemaHashing();
trace.Enable(provider);

provider<T> carries the flag, krabs::trace::enable() hands the provider's GUID to the trace's schema_locator, and get_event_schema_no_throw() only hashes the metadata for providers on that list. Providers that do not opt in take the same code path they did before this PR, so there is no added per-event cost for existing users. Manifest, MOF and WPP events are unaffected either way; they carry no TraceLogging metadata and their event id already identifies them uniquely. Kernel providers are excluded at compile time.

The opt-in list is a flat vector rather than a set: it is empty unless someone opts in, and a trace has a handful of providers at most, so scanning it beats hashing a GUID on every event. trace_context::schema_locator and trace::context_ lose their const qualifier so enable() can record the policy; callbacks still only ever see a const trace_context&.

Also fixed

Two defects in the metadata parsing this relies on:

  • The name was read with an unbounded string_view constructor. A record whose name is not nul-terminated would read past the end of the extended data item. The search is now bounded by the metadata extent.

  • The parser required the metadata Size field to exactly equal the extended data item size. Size describes the pseudo-structure and the item may be larger, so valid events lost their name and fell back to the ambiguous key. Size is now range-checked against the item instead.

And schema_key::operator=, which recursed until the stack was exhausted: the user-declared copy constructor and copy assignment suppress the implicit move operations, so the std::swap it used resolved back to operator= itself. It now assigns members directly and copies the incoming name into new storage before releasing the old, so the assigned key owns its name, as with the copy constructor. That holds even when the source borrows its name from the key being assigned to (key = schema_key(record, key.name)).

These three are unconditional - they are correctness fixes, independent of the opt-in.

Verification

  • All 214 krabstests pass. New tests cover the opt-in and its propagation from the provider to the trace's schema_locator (7 tests), and schema_key::operator= ownership, including assignment from a key that borrows its name from the target (2 tests).
  • examples/SchemaKeyCollisionRepro registers its own TraceLogging provider, emits two variants of one event name and consumes them with krabs. It exits 0 with the opt-in and 1 without it, so it doubles as a regression check for both halves of the behaviour.

Fixes microsoft#193.

TraceLogging events do not get a meaningful event id, so the schema cache
key could not tell two events from the same provider apart. Adding the
event name (microsoft#242) covered the common case, but a provider may log the same
event name from several call sites with different fields. Those variants
agree on provider, name, id, version, opcode, level and keyword, so they
still collided and the first schema retrieved was used to decode all of
them - silently shifting every field, and potentially reading past the end
of the user data buffer.

The TraceLogging metadata is the schema, so hash it into the key. This
keeps the cache effective for repeated events while separating variants,
and costs no extra TdhGetEventInformation call. The hash is FNV-1a over a
small blob; 0 is reserved to mean "no metadata" so manifest-based events,
which the existing fields already identify uniquely, are unaffected.

Also fixes two defects in the metadata parsing this relies on:

 * The name was read with an unbounded string_view constructor. A record
   whose name is not nul-terminated would read past the end of the
   extended data item. The search is now bounded by the metadata extent.

 * The parser required the metadata Size field to exactly equal the
   extended data item size. Size describes the pseudo-structure and the
   item may be larger, so valid events lost their name and fell back to
   the ambiguous key. Size is now range-checked against the item instead.

And fixes schema_key::operator=, which recursed until the stack was
exhausted: the user-declared copy constructor and copy assignment suppress
the implicit move operations, so the std::swap it used resolved back to
operator= itself. It now assigns members directly and re-internalizes the
name to preserve the copy constructor's ownership invariant.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds examples/SchemaKeyCollisionRepro, a minimal program that demonstrates
issue microsoft#193 without needing an external provider, an ETL file, or any
third-party tooling.

It registers its own TraceLogging provider and emits two variants of a
single event name -- one with a leading PartA_PrivTags field and one
without -- then consumes them with krabs and prints the field names
decoded for each variant. Real providers do this; for example,
Microsoft.Windows.AppLifeCycle.UI emits AppLaunch_UserClick both ways.

Because the variants agree on provider, name, id, version, opcode, level
and keyword, they collide in schema_locator's cache: the schema cached
for the first variant is reused for the second, so every field of the
second is shifted by the size of the missing field.

The program exits 0 on PASS, 1 on FAIL and 2 if it could not start a
trace session, so it doubles as a regression check. Verified against both
the pre-fix and post-fix schema_locator: it fails on the former and
passes on the latter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@swannman

Copy link
Copy Markdown
Member

Chris Davis (@chrdavis) thanks for submitting this! The problem you're solving makes sense - we have one concern about performance.

In the current model, the schema lookup cost for all TraceLogging events is the same; in the proposed change, the cost is proportional to the size of the event:

  ━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━━  ━━━━━━━━━━
   Manifest                13 ns      14 ns       ~1 ns
  ─────────────────────  ────────  ─────────  ──────────
   TraceLogging, 64 B      44 ns      89 ns      ~44 ns
  ─────────────────────  ────────  ─────────  ──────────
   TraceLogging, 256 B     46 ns     325 ns     ~279 ns
  ─────────────────────  ────────  ─────────  ──────────
   TraceLogging, 1 KiB     46 ns    1.27 µs    ~1.22 µs
  ─────────────────────  ────────  ─────────  ──────────
   TraceLogging, 4 KiB     46 ns    5.01 µs    ~4.97 µs

Would you rework this to be a per-provider policy that is opt-in?

@swannman Matt Swann (swannman) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good approach, but let's make this opt-in on a per-provider basis. Thanks!

Hashing the TraceLogging metadata into the schema cache key costs time
proportional to the size of the event, which is real per-event overhead
for every user of the library -- roughly 44ns -> 5.01us for a 4KiB event,
even though only a handful of providers ever reuse an event name.

Make it a per-provider policy that is off by default:

    krabs::provider<> provider(guid);
    provider.enable_trace_logging_schema_disambiguation();
    trace.enable(provider);

provider<T> carries the flag, krabs::trace::enable() hands the provider's
GUID to the trace's schema_locator, and get_event_schema_no_throw() only
hashes the metadata for providers on that list. Providers that don't opt
in take the same code path they did before this series, so the default
cost is unchanged. Manifest, MOF and WPP events are unaffected either
way; they carry no TraceLogging metadata.

The opt-in list is a flat vector because it is empty unless someone opts
in, and a trace has a handful of providers at most, so scanning it beats
hashing a GUID on every event. trace_context::schema_locator and
trace::context_ lose their const qualifier so enable() can record the
policy; callbacks still only ever see a const trace_context&.

Kernel providers are excluded at compile time -- they never emit
TraceLogging events.

Also exposes EnableTraceLoggingSchemaDisambiguation() on the managed
Provider, and updates examples/SchemaKeyCollisionRepro to call the opt-in
so it still passes.

Verified: all 212 krabstests pass, and the repro exits 0 with the opt-in
and 1 without it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@chrdavis

Copy link
Copy Markdown
Member Author

Matt Swann (@swannman) Thanks - reworked it into a per-provider opt-in in 636fbc9. It's off by default, so nothing changes for existing users:

krabs::provider<> provider(guid);
provider.enable_trace_logging_schema_disambiguation();  // before trace.enable()
trace.enable(provider);
var provider = new Provider("Microsoft.Windows.AppLifeCycle.UI");
provider.EnableTraceLoggingSchemaDisambiguation();
trace.Enable(provider);

provider<T> carries the flag, krabs::trace::enable() hands the provider's GUID to the trace's schema_locator, and get_event_schema_no_throw() only hashes the metadata for providers on that list. A provider that hasn't opted in takes exactly the code path it did before this PR, so the 44ns -> 5.01us you measured only applies to providers that ask for it. Kernel providers are excluded at compile time, and manifest/MOF/WPP events were never affected.

The opt-in list is a flat vector rather than a hash set deliberately: it's empty unless someone opts in, and a trace has a handful of providers at most, so the scan is cheaper than hashing a GUID on every event. The one visible consequence is that trace_context::schema_locator and trace::context_ are no longer const - callbacks still only ever see a const trace_context&.

The three correctness fixes in this PR (the unbounded name read, the over-strict Size check, and the infinite recursion in schema_key::operator=) stayed unconditional, since they're bugs rather than part of the feature.

All 212 krabstests pass, including 7 new ones for the opt-in, and examples/SchemaKeyCollisionRepro now exits 0 with the opt-in and 1 without it - so it covers both halves of the behaviour.

Happy to rename enable_trace_logging_schema_disambiguation if you'd prefer something shorter, and equally happy to drop the managed EnableTraceLoggingSchemaDisambiguation() wrapper from this PR if you'd rather keep it native-only for now.

@swannman Matt Swann (swannman) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Found a potential memory corruption issue and added a few naming nits.

Comment thread krabs/krabs/schema_locator.hpp Outdated
Comment thread krabs/krabs/provider.hpp Outdated
Comment thread krabs/krabs/provider.hpp Outdated
* provider.
* </summary>
*/
bool trace_logging_schema_disambiguation_enabled() const;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trace_logging_full_schema_hashing_enabled, similar changes elsewhere

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d9bf192 - trace_logging_full_schema_hashing_enabled(), on both provider<T> and schema_locator.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backing member now follows suit as well: trace_logging_full_schema_hashing_enabled_ in 4c8ab69.

Comment thread Microsoft.O365.Security.Native.ETW/Provider.hpp Outdated
schema_key::operator= pointed 'name' at rhs's text, released its own
backing_name, and only then called internalize_name() to take a copy.
But rhs can be borrowing its name from the very key being assigned to --
for example, a temporary key built from this key's name:

    key.internalize_name();
    key = krabs::schema_key(record, key.name);

In that case backing_name.reset() freed the text both keys pointed at,
and internalize_name() copied freed memory. The this != &rhs check does
not catch it, because the keys are different objects sharing one buffer.

Copy the incoming name into new storage first, then assign the other
members and take ownership of the copy. As a side effect the operator
now gives the strong exception guarantee: the only allocation happens
before anything is modified.

Adds a regression test for the aliasing case -- the debug heap fills
freed memory, so the Debug run in CI fails deterministically without the
fix -- and one for the operator's existing guarantee that the assigned
key owns its name after the source is gone.

Reported by @swannman in review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per review feedback, name the opt-in after what it does:

- provider<T>::enable_trace_logging_schema_disambiguation() ->
  enable_trace_logging_full_schema_hashing()
- provider<T>::trace_logging_schema_disambiguation_enabled() ->
  trace_logging_full_schema_hashing_enabled()
- Provider.EnableTraceLoggingSchemaDisambiguation() ->
  EnableTraceLoggingFullSchemaHashing()

The matching schema_locator methods, private members, docs, tests and
the SchemaKeyCollisionRepro example are renamed to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@swannman Matt Swann (swannman) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with the most minor of nit comments if you'd be willing to do the rename?

Comment thread krabs/krabs/provider.hpp Outdated
T level_;
T trace_flags_;
bool rundown_enabled_;
bool trace_logging_full_schema_hashing_;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: trace_logging_full_schema_hashing_enabled_, parallel to rundown_enabled_ above

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 4c8ab69 - renamed to trace_logging_full_schema_hashing_enabled_. Thanks for the review!

Per review feedback, name the provider<T> member
trace_logging_full_schema_hashing_enabled_, parallel to rundown_enabled_
and to the trace_logging_full_schema_hashing_enabled() accessor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@swannman
Matt Swann (swannman) merged commit d0f4b2f into microsoft:master Oct 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong schema retrieval with TraceLogging events

2 participants