Repository navigation
The network intakes stat the manifest instead of parsing it every second - #198
Merged
Merged
Conversation
The tick shared by the OTLP, Forward and incus intakes read and parsed every open store's .bark several times a second to look for a retention or wal change: declared_retention, subject_of, and the wal declaration for all stores under one lock. Each store now has a LivePolicy, as the appender and the records sink do: the manifest is re-read only when its (mtime, len) changed, and the wal declaration is applied to that one store when it did. The per-store entry point replaces the walk over every store. A manifest that stops parsing keeps the last good policy with a warning, where the tick used to skip enforcement silently. Forward intake with a store open, steady state: 4.2 .bark opens a second before, 0.2 after plus one stat. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #192, which did this for the records sink. The tick shared by the OTLP, Forward and incus intakes still read and parsed every open store's
.barkseveral times a second to look for a retention or wal change (declared_retention,subject_of, and the wal declaration for all stores under one lock).Change
Each store gets a
LivePolicy, as the appender and the sink already have: the manifest is re-read only when its(mtime, len)changed, and the wal declaration is applied to that store when it did.Intake::sync_wal_declaration(name)replaces the walk over every store.Measured
forward-intakewith one store open and retention declared, steady state (strace):.barkopens.barkstatsI did not trace the residual opens. A live
set wal=falseon an open store prints the identical note on both binaries.Behaviour change
A manifest that stops parsing keeps the last good policy with one warning, as it does for the appender and the sink, where this tick used to skip enforcement silently.
Test
sync_wal_declarationapplies to the store named and to no other (an intake store starts with a write-ahead segment, so the test declareswal=falseon two and syncs one). Making it sync every store, as the old code did, fails it.Not here
index_declaredis still read when a store's chunk count changes (about once per flush). It's per flush, not per tick, so I left it.🤖 Generated with Claude Code