CASSSIDECAR-434: Add failure policy handling for ConfigurationProvider unavailability - #378
Open
pauloricardomg wants to merge 1 commit into
Open
CASSSIDECAR-434: Add failure policy handling for ConfigurationProvider unavailability#378pauloricardomg wants to merge 1 commit into
pauloricardomg wants to merge 1 commit into
Conversation
…r unavailability Implement a FailurePolicyWrapper that decorates a downstream ConfigurationProvider with configurable fallback behavior when the provider is unavailable. The wrapper caches overlays locally via a FileBasedConfigurationProvider (cached_overlay.json) and applies one of three policies on failure: - CACHED_READ_WRITE: reads and writes fall back to the local cache - CACHED_READ_ONLY: reads fall back to cache, writes are rejected - FAIL: all operations fail with the original exception The wrapper is a no-op when the downstream provider is already file-based. ConfigurationManager now requires a configurationStore path and FailurePolicy, wrapping the provider transparently at construction time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
ConfigurationManagerreads and writes cassandra.yaml overlays through a pluggableConfigurationProvider. Remote implementations can beunreachable, and today any provider exception surfaces as a generic
ConfigurationManagerException- HTTP 500, with no way to keep serving configurationduring an outage.
This adds
FailurePolicyWrapper, a decorator that mirrors successful provider resultsto a local file cache and applies a configurable policy when the provider is down:
CACHED_READ_WRITE- reads and writes fall back to the cacheCACHED_READ_ONLY- reads fall back to the cache, writes are rejectedFAIL- every operation throwsFailures are classified by
isUnavailable(). Domain errors and provider bugs(
ConfigurationManagerException,IllegalArgumentException,NullPointerException,IllegalStateException,UnsupportedOperationException) propagate unchanged soclients still get the right error; everything else - IOException, socket and timeout
failures, provider client exceptions - is treated as an outage and hits the policy.
The cache is a
FileBasedConfigurationProviderwritingcached_overlay.json. Wrapping aFileBasedConfigurationProvideris a no-op - a local provider cannot be unreachablein the network sense, and double-caching it would serve no purpose.
FailurePolicyis threaded through the constructor but not yet read from sidecar.yamlor wired via DI. That plumbing is a follow-up.
Known limitation
Writes accepted by the cache under
CACHED_READ_WRITEare local-only. Once theprovider recovers, the next read returns the delegate's value and overwrites the cache,
so writes made during the outage are lost. There is no reconciliation. This is
documented on the enum constant.
Review guide
Start with
FailurePolicyfor the three behaviors, thenFailurePolicyWrapper. Theparts worth attention:
isUnavailable()- exception classification is by exclusion, so an unrecognized providerclient exception is considered unavailability and uses the cache rather than failing.
getOverlay()null handling - anullfrom the delegate provider means the overlay wasdeleted upstream, and the wrapper caches an empty snapshot rather than leaving the
stale entry, so a deleted overlay is not resurrected during a later outage. A
nullfrom the cache means nothing was ever cached, and throws instead.
updateCache()- best-effort. It hash-compares to skip redundant disk writes andswallows every cache exception with a log, because a cache failure must not fail an
operation the delegate already committed. Writes only refresh the cache when
storeOverlayreturnedtrue.FileBasedConfigurationProvideris a mechanical change: the filename constant became aconstructor parameter with
overlay.jsonas the default.Testing
FailurePolicyWrapperTestexercises each policy against a delegate that is healthy,down, and down with nothing cached, plus the cases that are easy to get wrong:
wrap()bypassing file-based providers, no cache update on CAS conflict, deleted overlays not
resurrected from cache, and domain rejections propagating instead of falling back.
ConfigurationManagerTestadds four end-to-end tests for read fallback,FAIL, andpatching under both cached policies. Existing tests moved to the new constructor with
FailurePolicy.FAIL, which preserves their prior behavior.