Add rate limiting to xapi#7175
Open
cplaursen wants to merge 24 commits into
Open
Conversation
Token buckets form the core of the XAPI rate limiter. Token buckets hold tokens, which are refilled over time according to their refill rate up to a maximum. Each client making calls to XAPI will have one token bucket associated; each call takes tokens out of the bucket, and delays will be enforced such that any requests made to an empty bucket have to wait until the bucket has been refilled. The token bucket library implements functions for token bucket creation and thread-safe token consumption. Refills are calculated only when the bucket is observed by tracking a last_refill timestamp. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Token buckets form the basis of the rate limiting, but on their own they lack a way of enabling fairness when waiting. For this, we add a separate queuing mechanism which registers callbacks for requests that are waiting on a token bucket to refill. A worker thread is tasked with awakening requests as enough tokens are placed back in the bucket. We expose an interface for submitting requests to a queue, both in synchronous and asynchronous mode. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
We need an in-memory key-value store to catalogue callers for the purposes of rate limiting and usage tracking. We want to support the specification of callers via prefixes, which means that we cannot use a regular hashtable. Instead, we can use an association list with an LRU cache, which should amortise usage as we expect most calls to come from the same small group of callers. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
This commit adds a thread-safe counter that tracks API call usage, with the objective of exposing a counter to an RRD. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
The throttling feature branch is out of date, and we need to keep it up to date in order for it to build with the latest dependencies.
The rate limit library is currently called xapi_rate_limit, which will clash with the xapi file associated to the rate limit datamodel. Renaming to rate_limit_lib. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
As per the rate_limit design, this commit adds two new datamodels to xapi: - Caller: tracks usage statistics on clients calling into xapi. Clients are currently identified by their ip address and HTTP user agent - Rate_limit: lets xapi admins apply rate limits to groups of callers. Rate limiters allow a fixed number of "tokens" (number correlated with call cost) to be used per second; if the token count is exceeded, the calls get throttled. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@cloud.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Contributor
Author
|
Will be ready to merge once #7171 has been merged. I've thoroughly tested the feature with unit tests, quicktests, and XenRT, and it is gated behind the rate_limit flag in xapi.conf. |
There is an issue where tokens with a low maximum capacity hang forever when requests that cost more than the capacity come in, as they must wait for the bucket to refill past its capacity. We fix this by letting buckets go negative only when these conditions are met, in order to avoid deadlocks. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Currently, the caller table is being initialised in xapi.ml, but this means that additions to the caller datamodel are not tracked until the in-memory structure is initialised, which leads to tests failing. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
An IPv4 connection can surface at the socket as an IPv4-mapped IPv6 address (e.g. ::ffff:1.2.3.4) depending on how the listening socket is bound. Ipaddr.to_string preserves the IPv6 form, which then fails to match string comparisons against the plain IPv4 form of the same address, breaking the per-caller rate-limit table lookups. Add Http_svr.canonical_ip_string and route the client IP through it in string_of_client, Context.get_client_ip, and the caller-id assembled in Xapi_http.add_handler. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Caller_table uses Atomic for lock-free reads, but its writers are non-CAS Atomic.get/set pairs and refresh_caller_rate_limit's delete-then-insert is not atomic. Concurrent refreshes for the same caller could both read the pre-mutation state, and the later insert would then be silently refused as a duplicate, leaving the table with stale state. Replace the create-path-only mutex with caller_table_mutex, held around create, destroy, refresh_caller_rate_limit, and the auto-create re-check. Split create into create/create_locked so the auto-create path can reuse the body under the same lock. register keeps bypassing the mutex - it runs single-threaded at start of day. Also preserve the caller's stats across a rate_limit refresh, so attaching or detaching a rate_limit no longer resets the calls/tokens counters accumulated since xapi startup. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
add_caller was clearing the caller's rate_limit to Ref.null before setting the new value, which briefly left the caller unlimited and fired two full caller_table refreshes (each an extra DB read via the callback). The datamodel's reverse-relation machinery already removes the caller from any previous Rate_limit's callers set on assignment, so set the target directly and notify once. set_burst_size and set_fill_rate no longer notify attached callers: the caller_table entry still points at the same rate_limit_ref and install_bucket has already swapped the in-memory bucket, so the next find_bucket on the dispatch path picks up the new parameters. The old notifications were pure overhead. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
cplaursen
marked this pull request as ready for review
July 16, 2026 09:48
GabrielBuica
approved these changes
Jul 16, 2026
Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.