Skip to content

CASSSIDECAR-434: Add failure policy handling for ConfigurationProvider unavailability - #378

Open
pauloricardomg wants to merge 1 commit into
apache:trunkfrom
pauloricardomg:CASSSIDECAR-434
Open

CASSSIDECAR-434: Add failure policy handling for ConfigurationProvider unavailability#378
pauloricardomg wants to merge 1 commit into
apache:trunkfrom
pauloricardomg:CASSSIDECAR-434

Conversation

@pauloricardomg

Copy link
Copy Markdown
Contributor

Overview

ConfigurationManager reads and writes cassandra.yaml overlays through a pluggable
ConfigurationProvider. Remote implementations can be
unreachable, and today any provider exception surfaces as a generic
ConfigurationManagerException - HTTP 500, with no way to keep serving configuration
during an outage.

This adds FailurePolicyWrapper, a decorator that mirrors successful provider results
to a local file cache and applies a configurable policy when the provider is down:

  • CACHED_READ_WRITE - reads and writes fall back to the cache
  • CACHED_READ_ONLY - reads fall back to the cache, writes are rejected
  • FAIL - every operation throws

Failures are classified by isUnavailable(). Domain errors and provider bugs
(ConfigurationManagerException, IllegalArgumentException, NullPointerException,
IllegalStateException, UnsupportedOperationException) propagate unchanged so
clients still get the right error; everything else - IOException, socket and timeout
failures, provider client exceptions - is treated as an outage and hits the policy.

The cache is a FileBasedConfigurationProvider writing cached_overlay.json. Wrapping a
FileBasedConfigurationProvider is a no-op - a local provider cannot be unreachable
in the network sense, and double-caching it would serve no purpose.

FailurePolicy is threaded through the constructor but not yet read from sidecar.yaml
or wired via DI. That plumbing is a follow-up.

Known limitation

Writes accepted by the cache under CACHED_READ_WRITE are local-only. Once the
provider recovers, the next read returns the delegate's value and overwrites the cache,
so writes made during the outage are lost. There is no reconciliation. This is
documented on the enum constant.

Review guide

Start with FailurePolicy for the three behaviors, then FailurePolicyWrapper. The
parts worth attention:

  • isUnavailable() - exception classification is by exclusion, so an unrecognized provider
    client exception is considered unavailability and uses the cache rather than failing.
  • getOverlay() null handling - a null from the delegate provider means the overlay was
    deleted upstream, and the wrapper caches an empty snapshot rather than leaving the
    stale entry, so a deleted overlay is not resurrected during a later outage. A null
    from the cache means nothing was ever cached, and throws instead.
  • updateCache() - best-effort. It hash-compares to skip redundant disk writes and
    swallows every cache exception with a log, because a cache failure must not fail an
    operation the delegate already committed. Writes only refresh the cache when
    storeOverlay returned true.

FileBasedConfigurationProvider is a mechanical change: the filename constant became a
constructor parameter with overlay.json as the default.

Testing

FailurePolicyWrapperTest exercises each policy against a delegate that is healthy,
down, and down with nothing cached, plus the cases that are easy to get wrong: wrap()
bypassing file-based providers, no cache update on CAS conflict, deleted overlays not
resurrected from cache, and domain rejections propagating instead of falling back.

ConfigurationManagerTest adds four end-to-end tests for read fallback, FAIL, and
patching under both cached policies. Existing tests moved to the new constructor with
FailurePolicy.FAIL, which preserves their prior behavior.

…r unavailability

Implement a FailurePolicyWrapper that decorates a downstream ConfigurationProvider
with configurable fallback behavior when the provider is unavailable. The wrapper
caches overlays locally via a FileBasedConfigurationProvider (cached_overlay.json)
and applies one of three policies on failure:

- CACHED_READ_WRITE: reads and writes fall back to the local cache
- CACHED_READ_ONLY: reads fall back to cache, writes are rejected
- FAIL: all operations fail with the original exception

The wrapper is a no-op when the downstream provider is already file-based.
ConfigurationManager now requires a configurationStore path and FailurePolicy,
wrapping the provider transparently at construction time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant