Repository navigation
S3 object sync keeps per-bucket totals, not every object - #5
Merged
Merged
Conversation
With aws.storage.s3object.sync on, sync turned every object of every bucket into a graph resource, plus a fresh copy of its bucket per object, and held them all until the end of the sync. Memory grew with the number of objects in the account; a few busy buckets took a 512 MiB container to its limit. The fetcher now adds up the count and size while paging ListObjectsV2 and sends the bucket once, with ObjectCount and Size (bytes). A listing that fails part way sends no totals. bucket_sizer reads the totals from the buckets, and the sync line counts objects from them instead of printing "0 s3object". The objects themselves are still sent for `list s3objects` (fetch by type) and when the new aws.storage.s3object.keep is true. Without it, `list s3objects --local` says where the sizes are and how to get the objects. Closes #4
…ocal note bucket_sizer said nothing about buckets with no count, so a sync whose listing failed for some buckets printed a total that read as the account's. It now says how many were left out. A graph synced by an earlier version (objects, no totals) gets advice to sync again rather than to turn on object sync, which is already on. The list s3objects --local note names aws.storage.s3object.sync when that is what is off, and is only printed when the list is empty and the storage service was synced.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4.
With object sync on,
syncturned every object in every bucket into a graph resource, sent a fresh copy of the bucket along with each one, and held all of it until the end. Memory grew with the number of objects, and a few busy buckets were enough to take a 512 MiB container to its limit.Now the fetcher adds up the count and the bytes while it pages through
ListObjectsV2and sends the bucket once, withObjectCountandSizeon it. If a listing fails part way, that bucket gets no totals, because a partial count would look like the bucket's size. API calls are the same as before: one paginated listing per bucket.What else changed:
bucket_sizerreads the totals from the buckets. If some buckets have no count, it says how many and that the total leaves them out. On a graph synced by an older version (objects but no totals) it tells you to sync again.0 s3object.aws.storage.s3object.keep, off by default, keeps the objects as before, forlist s3objects --localon small buckets.list s3objectswithout--localstill asks AWS and returns every object.list s3objects --localon an empty result says why: object sync is off, or only totals were kept.I ran
sync --storagewith object sync on against a real account: 6 buckets, 633,565 objects. Peak RSS was 57 MB, andstorage.ntcame out at 4.7 KB.inspect -i bucket_sizerprinted the per-bucket counts and sizes from it.Tests use a fake S3 client that pages, and check the totals, that no object resources are kept, that each bucket is sent once, the keep option, the live listing, a listing that fails half way, the sync line and the
--localnotes.make testpasses with-race.🤖 Generated with Claude Code