Repository navigation
Conversation
…nd reports new variants Both windows are mined in one tree, baseline first, and compared by cluster, so a token that is constant in one window and varies in the other no longer splits a pattern. New messages that Drain merges into a baseline pattern are found from Drain's own row assignment with a token compare against the baseline template, and reported as new_variant. The baseline window excludes its end.
🦋 Changeset detectedLatest commit: 95a4ed5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@bcanvural is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #3556, #3544 (option 2). Replaces draft #3557, which was stacked on the pre-merge version of #3551; this is the same change on current
main.emerging_signalsmines the baseline and current windows in two separate Drain trees and matches their templates as strings. The two trees can generalize the same message differently: a token that is constant in one window stays literal there and becomes<*>in the other, so one unchanged pattern is reported as both "new" and "gone". Date masking (#3551) removed the most common cause; this removes the rest.This PR:
TemplateMiner, baseline first, and compares them by cluster id. The order is fixed, so the result doesn't depend on which query finishes first.user login failed: account lockedintouser login failed: invalid password). Baseline templates are copied after the baseline is mined. A current row whose cluster came from the baseline but doesn't match that baseline template (token compare, no regex) is reported inemergingwithstatus: "new_variant",baselinePattern,variantShare, and the new line as its sample. Baseline clusters with fewer than 3 sampled rows are skipped.dateRangeEndInclusive: false), so a row at the boundary is counted once.Supporting changes:
minePatternstakes an optionalminerand returnsrowClusterIds(the cluster of each input row);TemplateMiner.tokenizereturns the masked tokens Drain compares.mineWindowPatternsis split intofetchWindowSample+mineWindowSampleand kept as a wrapper, soevent_patternsis unchanged.Cost: on test data with 100 changed patterns, the variant step takes about 10 ms per 10K rows, and the whole call went from 496 ms to 221 ms (one tree instead of two).
How to reproduce
Starting point:
make dev-int.On
main:packages/api/src/mcp/__tests__/emergingSignalsOrder.test.tsfrom this PR.npx jest src/mcp/__tests__/emergingSignalsOrder.test.tsinpackages/api: it fails, becausemainhas nofetchWindowSample/mineWindowSampleand reports nonew_variant.On this branch:
emergingSignalsOrder.test.tspasses (17 tests): the login example →new_variant; dated lines on two days → nothing; identical windows → nothing; a token varying only in the current window; same result whichever query finishes first; small baseline clusters skipped; a variant never shows indisappeared.drain-masking.test.ts(shared miner,rowClusterIds,tokenize) passes; common-utils 3,412, MCP unit 654 and MCP integration (queryTool,emergingSignalsTool,eventPatterns,dashboard) 370 tests pass.How to test on Vercel preview
N/A — non-UI change.
References
emerging_signalsreports dated log lines as both "new" and "gone": Drain keeps the literal date, and each window is mined separately #3544