Enhance manifest cache and CLI catalog worker functionality - #136
Merged
Merged
Conversation
- Introduced a new `summary_under` method in `ManifestCache` to aggregate cached manifest observations for a given key prefix, improving data retrieval efficiency. - Updated the `refresh_mount` method to log the number of refreshed entries, enhancing traceability during cache updates. - Refactored CLI to run the catalog worker before the async runtime, ensuring proper environment setup for catalog operations. - Enhanced request logging in the CLI server to include metrics for catalog operations, improving performance insights. - Updated documentation and tests to reflect changes in catalog handling and manifest caching mechanisms. This commit aims to improve the efficiency and reliability of dataset management and querying in the application.
- Introduced a HashSet to track seen datasets during grant operations, improving efficiency by preventing duplicates in granted and remaining datasets. - Updated error messages for clarity in the revoke_datasets function, enhancing user feedback. - Renamed dataset references in tests from "public" to "shared" to reflect changes in dataset naming conventions. - Added a new RFC document outlining the pChronicle catalog resolution and manifest cache, detailing the caching mechanisms and their impact on dataset management. This commit aims to improve the performance and clarity of dataset grant handling while providing comprehensive documentation for future reference.
- Added logging for server stderr to capture errors during catalog worker initialization, improving debugging capabilities. - Refactored the test for anonymous access to ensure proper scoping and validation of access rights, enhancing security checks. - Introduced a new asynchronous function to assert anonymous access is scoped correctly, improving test clarity and maintainability. This commit aims to strengthen the reliability of catalog worker tests and enhance error reporting for better troubleshooting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary_undermethod inManifestCacheto aggregate cached manifest observations for a given key prefix, improving data retrieval efficiency.refresh_mountmethod to log the number of refreshed entries, enhancing traceability during cache updates.This commit aims to improve the efficiency and reliability of dataset management and querying in the application.