diff --git a/README.md b/README.md index af6d922aa..e945c7770 100644 --- a/README.md +++ b/README.md @@ -385,6 +385,106 @@ try (QuestDB db = QuestDB.connect("wss::addr=localhost:9000;tls_verify=unsafe_of } ``` +### OIDC Sign-In (Device Flow) + +For QuestDB Enterprise instances secured with OIDC, `OidcDeviceAuth` signs a user in interactively using the [OAuth 2.0 Device Authorization Grant](https://www.rfc-editor.org/rfc/rfc8628). It works from environments that have no local browser — a remote notebook kernel, a container, a headless job — because the user authorizes on any device (laptop or phone) while the process only makes outbound calls to the identity provider. + +On first use it prints a verification URL and a short code, and opens the URL in your default browser when one is available; authorize there (or open the URL on any device, such as your phone), enter the code, and the token is cached in memory and refreshed silently on later calls. + +```java +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; + +// Discover the client id, scope and endpoints from the QuestDB server's /settings: +try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB("https://questdb.example.com:9000")) { + auth.signIn(); // sign in once: prompts on first use, then caches and refreshes + + // The provider is shared by the ingest and query pools. It is queried for + // every initial WebSocket upgrade and reconnect, so both pools follow token + // rotation without putting a credential in the configuration string. + try (QuestDB db = QuestDB.connect( + "wss::addr=questdb.example.com:9000;", + auth::getToken)) { + try (Sender sender = db.borrowSender()) { + sender.table("trades") + .symbol("symbol", "ETH-USD") + .doubleColumn("price", 2615.54) + .atNow(); + } + // db.borrowQuery() uses the same provider for query connections. + } +} +``` + +For a standalone sender, use `httpTokenProvider(auth::getToken)` for the same rotating-token behavior. A fixed `httpToken(token)` or `token=` connect-string value captures the token once, so a client that reconnects after that token expires starts failing authentication. Hand rotating credentials to the provider API, not a `Sender.fromConfig(...)` string or the `QDB_CLIENT_CONF` environment variable, which are easily logged, persisted, or left in shell history. + +`getToken()` sits on a hot path — it is called once per ILP flush and once per WebSocket upgrade or reconnect — so a credential failure is rate-limited rather than retried on every call. When a silent refresh fails, `getToken()` does not attempt another one for 5 seconds: calls inside that window fail immediately, asking for an interactive `signIn()`, instead of hitting the identity provider again. Without that guard a producer retrying its rows would drive one token-endpoint round trip per flush, blocking the producer thread for each one and hitting the provider hard enough to trip its rate limits and lengthen the very outage being retried. Only a real refresh attempt arms the guard, and an explicit `signIn()` or `clearCache()` clears it outright. It is deliberately short — a stampede guard, not a circuit breaker — so a credential that comes back within seconds is picked up on the first call after the window rather than on the first call after it recovers. + +By default the prompt prints the verification URL and code to `System.out` **and** tries to open the URL in your default browser. The browser open is best-effort: it only opens an `http(s)` URL, is skipped on a headless host or a JVM without the `java.desktop` module, and never blocks sign-in (the client declares `requires static java.desktop`, so the module is optional at run time and its absence can never break module resolution; a modular application therefore gets the browser launch only when `java.desktop` is in its own module graph) — the URL and code are always printed too, so a remote or browserless process still works. To disable the browser launch for a whole process (a server, automation, CI), set the system property `-Dquestdb.client.oidc.open.browser=false`. To print only (no browser) for a single client, pass `DeviceCodePrompt.SYSTEM_OUT`; to render the challenge yourself (a clickable link or QR code in a notebook), pass any `DeviceCodePrompt`: + +```java +// print only, do not open a browser: +try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB( + "https://questdb.example.com:9000", + new OidcDeviceAuth.DiscoveryOptions().prompt(DeviceCodePrompt.SYSTEM_OUT))) { + auth.signIn(); +} +``` + +The same token can be presented to QuestDB over any auth path the server already validates: + +- **REST API:** send it as an `Authorization: Bearer ` header (`auth.getAuthorizationHeaderValue()` returns the full value). +- **PG-wire:** connect as user `_sso` with the token as the password (requires `acl.oidc.pg.token.as.password.enabled=true` on the server). + +To configure the identity provider explicitly instead of discovering it from the server: + +```java +OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint("https://idp.example.com/as/device_authz.oauth2") + .tokenEndpoint("https://idp.example.com/as/token.oauth2") + .scope("openid groups") + .groupsInToken(true) // matches acl.oidc.groups.encoded.in.token on the server + .build(); +``` + +Discovery via `fromQuestDB(...)` reads the OIDC client id, scope, audience and endpoints from the server's `/settings`, and the identity provider's client must have the device authorization grant enabled. When the server does not advertise its device authorization endpoint (today's servers), pin the identity provider by its issuer so the client can discover the endpoint from the issuer's `.well-known/openid-configuration` document: + +```java +try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB( + "https://questdb.example.com:9000", + new OidcDeviceAuth.DiscoveryOptions().issuer("https://idp.example.com"))) { + auth.signIn(); +} +``` + +The identity provider's device authorization and token endpoints must use `https` — a loopback endpoint (`localhost` or `127.0.0.0/8`) may use `http`, since the request never leaves the host — so the device code and refresh token are never sent in cleartext. That is a rule about the *scheme*: the trust anchor is `tlsConfig`, and one `OidcDeviceAuth` carries a single one, so `ClientTlsConfiguration.INSECURE_NO_VALIDATION` — reached for to talk to a QuestDB server with a self-signed certificate — also stops the client validating the *identity provider's* certificate on the legs that carry the device code and the refresh token. Point `tlsConfig` at a trust store rather than disabling validation whenever an identity provider is involved. `allowInsecureTransport(true)` relaxes only the QuestDB `/settings` link (for local development against an `http` QuestDB server), e.g. `OidcDeviceAuth.fromQuestDB(url, new OidcDeviceAuth.DiscoveryOptions().allowInsecureTransport(true))`; it never relaxes the identity provider endpoints, matching the Python client. + +`fromQuestDB(...)` takes the identity provider endpoints from the server's unauthenticated `/settings`, so it trusts that server to designate where you sign in: a spoofed, compromised, or man-in-the-middled server could redirect the sign-in to an attacker-controlled identity provider. Only use it against a server you trust, reached over `https`. Passing an issuer hardens this: the token and device authorization endpoints are then pinned to the issuer's origin (and, when the issuer has a path, an endpoint advertised by `/settings` must also be under that path — so a tampered `/settings` cannot redirect to a different tenant on a path-based provider such as Keycloak `…/realms/{realm}`), and an endpoint outside it is rejected; the issuer itself comes from you out of band, so a tampered `/settings` cannot move it. When `.well-known` discovery is needed, the document must also return the exact issuer prefix used to retrieve it before any discovered endpoint is accepted. When the server is not trusted, configure the identity provider explicitly with `OidcDeviceAuth.builder()` (optionally with `.issuer(...)`) instead of discovering it. + +#### Persisting the Token Across Restarts + +By default the token lives in memory only, so a process that restarts has to run the device flow again. Pass a `TokenStore` to persist it; the restarted process then resumes from the saved refresh token — a silent call to the token endpoint — instead of prompting the user again: + +```java +import io.questdb.client.cutlass.auth.FileTokenStore; + +try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB( + "https://questdb.example.com:9000", + new OidcDeviceAuth.DiscoveryOptions().tokenStore(FileTokenStore.atDefaultLocation()))) { + auth.signIn(); // prompts the first time; after a restart it refreshes silently from the saved token +} +``` + +`FileTokenStore.atDefaultLocation()` writes one file per OIDC configuration under `${user.home}/.questdb/oidc-tokens/` (override the directory with the `questdb.client.oidc.token.store.dir` system property). The file name is a hash of the endpoints, client id, scope, audience and groups-in-token mode — the *configuration*, not the person who signed in through it, since none of those fields names a subject. Entries for different servers, providers or client configurations therefore stay separate, but **two people signing in through the same configuration share one entry**: whoever signs in last overwrites the previous token, so a store represents a single active login. If more than one application user has to be signed in at the same time, give each their own store — `FileTokenStore.at(dir)` on a per-user directory, or a per-user `questdb.client.oidc.token.store.dir` — rather than relying on the file name to separate them. The default location is already per OS user, so this only arises when one OS user (a shared service account, a multi-tenant process) signs in as several people. After a restart, `getToken()` also works as the first call — no explicit `signIn()` needed — which suits a long-lived `Sender` built with `httpTokenProvider(auth::getToken)`. `clearCache()` removes the persisted entry and forces a fresh sign-in next time. + +A store read that *throws* — an unreadable file after a `chmod` or a uid change in a container, `EIO`/`ESTALE` on an NFS home — is not fatal and does not disable persistence for the life of the process, but it is not retried on every call either, since `getToken()` would otherwise pay a blocking file open and a `WARN` line per ILP flush, forever. The first failure is retried immediately, so a one-shot fault (notably a carried interrupt flag, which makes the channel underneath `FileTokenStore` throw on a thread that merely carries it) recovers on the next call; each consecutive failure after that backs off 5 seconds, doubling to a 60 second cap. A store that simply has nothing to return is unaffected — `load` reports that by returning `null` rather than by throwing, and the client stops asking. + +The token is stored as **plaintext JSON protected by file permissions** — `0600` file, `0700` directory on POSIX systems (Linux, macOS), the same approach `gcloud`, `aws` and `gh` take. On Windows these POSIX permissions cannot be enforced, so the file currently relies on the user-profile directory's default ACL (owner-only ACL hardening is a follow-up); the client logs a one-line warning through SLF4J at `WARN` the first time it cannot enforce them (the library ships `slf4j-api` only, so this - and every other client warning - is discarded unless your application supplies an SLF4J binding). Enabling persistence therefore writes a long-lived refresh token to disk: anyone who can read the file holds a credential until it expires or is revoked. To encrypt it at rest, supply your own `TokenStore` (backed by an OS keychain or a secrets manager) instead of `FileTokenStore`. A persisted file is treated as untrusted input on load, but it is **not cryptographically authenticated** — there is no MAC or signature over its contents. Anyone who can write the file can therefore substitute a well-formed entry of their own, and the client will adopt it and present those tokens: the file permissions, not the file format, are what protect it. What the load path rejects is corruption and mix-ups rather than forgery — an oversized, malformed or unparseable file; an entry whose recorded client id, endpoints, scope, audience or groups-in-token mode does not match the identity being loaded; an entry carrying no usable token; and a token with control or non-ASCII characters, which is never placed on the wire — with the recorded expiry and lifetime clamped rather than trusted. In each case the client falls back to a refresh or an interactive sign-in. On POSIX the container is checked too: if the directory was writable by other local users, it is tightened back to `0700` and **every** entry in it is discarded — not just the one being read, since the tightening is what destroys the evidence, so anything left behind would look protected to the next load. Each identity then signs in again. Inside those permissions, though, the client cannot tell a planted credential from its own. + +`FileTokenStore` is safe to share between processes that sign in as the same identity: each update is written atomically (so a concurrent reader never sees a half-written credential), and when the identity provider rotates the refresh token on each refresh, the read-refresh-write is serialized across processes with a lock file so they do not race each other into an unnecessary re-prompt. The lock file's staleness is judged by its modification time, so this coordination assumes the processes share a clock — a single machine, or machines with synchronized clocks; under significant clock skew (for example a store directory on NFS shared across hosts) a live lock can be mis-judged stale or a dead one never expire. `clearCache()` removes the persisted entry under the same lock, but across processes it is best-effort: a peer that still holds a live in-memory token may legitimately re-persist afterwards (it always forces a fresh sign-in for the calling process). + ### Explicit Timestamps ```java diff --git a/core/src/main/java/io/questdb/client/HttpTokenProvider.java b/core/src/main/java/io/questdb/client/HttpTokenProvider.java new file mode 100644 index 000000000..cf28bbf50 --- /dev/null +++ b/core/src/main/java/io/questdb/client/HttpTokenProvider.java @@ -0,0 +1,109 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client; + +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.std.Chars; + +/** + * Supplies an HTTP authentication token to a {@link Sender} or pooled {@link QuestDB} connection on + * demand, so a provider returning a freshly refreshed token - e.g. {@code OidcDeviceAuth::getToken} + * - keeps long-lived ingest and query connections authenticated as the token rotates, without + * rebuilding them. An HTTP sender calls {@link #getToken()} as it builds each request; WebSocket + * ingest and query clients call it once per connection handshake, on the initial connect and again + * on every reconnect. + *

+ * {@link #getToken()} runs on HTTP flush and pooled connection/reconnection paths. Different pooled + * connections may call it concurrently, so implementations must be thread-safe. It must return + * promptly and must not block on interactive input. A quick silent token refresh is fine, but it must + * not start an interactive sign-in; a provider that coordinates a shared token store across processes + * (for example {@code OidcDeviceAuth} with a {@code FileTokenStore}) may add a brief, bounded wait to + * acquire that store's cross-process lock before such a refresh, which still counts as a quick silent + * refresh. Note that "quick" bounds the interactive wait, not the network: the silent refresh is a + * synchronous HTTP round-trip to the token endpoint, and its connection phase (DNS, TCP connect, TLS) + * is bounded by the client timeout as well for the bundled {@code OidcDeviceAuth}, leaving only DNS + * resolution to the OS. A provider that builds its own HTTP client should bound its connect and TLS + * handshake likewise, or a black-holed token endpoint stalls a flush for the OS connect timeout. An exception from {@link #getToken()} fails the + * in-flight flush (HTTP) or the connection attempt (WebSocket). + * + * @see QuestDB#connect(CharSequence, HttpTokenProvider) + * @see QuestDBBuilder#httpTokenProvider(HttpTokenProvider) + * @see Sender.LineSenderBuilder#httpTokenProvider(HttpTokenProvider) + */ +@FunctionalInterface +public interface HttpTokenProvider { + /** + * Validates a token returned by {@link #getToken()} before the client writes it into an + * {@code Authorization: Bearer} header. + *

+ * Callers must pass a value that cannot change between this check and the write that follows it. + * {@code getToken()} may return a reused buffer, so validating the provider's sequence and then + * re-reading it to build the header reads it twice: a mutation in between passes the check and + * splices the mutated bytes - a CR/LF among them - into the header. Snapshot with + * {@link Object#toString()} first, then validate and send the snapshot. Every call site in this + * library does. + *

+ * Rejects a null, empty or blank token, and any token + * carrying a control or non-ASCII character (outside {@code 0x20}-{@code 0x7e}): a real bearer + * token is printable ASCII, so a stray CR/LF (which would inject into the HTTP request line) or a + * non-ASCII byte (silently truncated to one byte by the ASCII header writer, yielding a corrupt + * credential the server only answers with 401) is refused rather than sent. The token itself is + * never placed in the exception message - it is the secret this guards. + * + * @param token the token returned by a provider + * @throws LineSenderException if the token is null, empty, blank, or carries a control or + * non-ASCII character + */ + static void validateToken(CharSequence token) { + if (Chars.isBlank(token)) { + throw new LineSenderException("token provider returned a null or empty token"); + } + for (int i = 0, n = token.length(); i < n; i++) { + char c = token.charAt(i); + if (c < 0x20 || c > 0x7e) { + throw new LineSenderException("token provider returned a token containing a control or non-ASCII character; refusing to send it as a credential"); + } + } + } + + /** + * Returns the current HTTP authentication token, without the {@code "Bearer "} prefix (the client + * adds it). Must not return null or empty, and must contain only printable ASCII (no control or + * non-ASCII characters) - the client splices the value verbatim into an {@code Authorization: + * Bearer} header and rejects a token that violates this (see {@link #validateToken(CharSequence)}). + *

+ * Returning a reused, mutable {@link CharSequence} - the idiomatic zero-allocation style - is + * supported and expected: the client re-validates every pulled token rather than trusting instance + * identity, so a buffer whose contents changed since the last call is checked again. What an + * implementation must not do is mutate a sequence it has already returned while the client is + * still reading it. The client snapshots each returned value before validating it, so a + * concurrent mutation cannot slip past the check into the header; an implementation that mutates + * mid-call is nonetheless racing with a reader and may see its own token dropped for the one the + * snapshot captured. Mutate between calls, not during one. + * + * @return the current HTTP authentication token + */ + CharSequence getToken(); +} diff --git a/core/src/main/java/io/questdb/client/QuestDB.java b/core/src/main/java/io/questdb/client/QuestDB.java index a4c7cb03f..cf669a166 100644 --- a/core/src/main/java/io/questdb/client/QuestDB.java +++ b/core/src/main/java/io/questdb/client/QuestDB.java @@ -78,6 +78,37 @@ static QuestDB connect(CharSequence configurationString) { return builder().fromConfig(configurationString).build(); } + /** + * Connects with a token supplied on demand for every initial WebSocket + * upgrade and reconnect. Use this overload for rotating credentials such + * as an OIDC device-flow token ({@code auth::getToken}); unlike a fixed + * {@code token=} value in the configuration string, the provider is queried + * again whenever either the ingest or query pool establishes a connection. + *

+ * The caller owns the provider and anything it captures. In particular, + * this handle does not close an {@code OidcDeviceAuth}; declare/close the + * {@code QuestDB} handle before closing the auth object. The provider may be + * called concurrently by different pooled connections and must be + * thread-safe. Interactive sign-in must happen before this call when the + * pools connect eagerly because token providers run on connection paths and + * must not prompt. + *

+ * The configuration must not contain {@code token}, {@code username}, or + * {@code password}; those fixed credentials are mutually exclusive with a + * token provider. + * + * @param configurationString a {@code ws}/{@code wss} config string + * @param tokenProvider supplies the current bearer token without the + * {@code "Bearer "} prefix + * @return a connected QuestDB handle + */ + static QuestDB connect(CharSequence configurationString, HttpTokenProvider tokenProvider) { + return builder() + .fromConfig(configurationString) + .httpTokenProvider(tokenProvider) + .build(); + } + /** * Borrows a {@link Query} handle from the pool. The caller MUST call * {@link Query#close()} on the returned instance to release it back to the diff --git a/core/src/main/java/io/questdb/client/QuestDBBuilder.java b/core/src/main/java/io/questdb/client/QuestDBBuilder.java index be18bfbec..e846ad129 100644 --- a/core/src/main/java/io/questdb/client/QuestDBBuilder.java +++ b/core/src/main/java/io/questdb/client/QuestDBBuilder.java @@ -73,6 +73,7 @@ public final class QuestDBBuilder { private BackgroundDrainerListener drainerListener; private SenderErrorHandler errorHandler; private long housekeeperIntervalMillis = UNSET; + private HttpTokenProvider httpTokenProvider; private String config; private long idleTimeoutMillis = UNSET; private long maxLifetimeMillis = UNSET; @@ -186,6 +187,11 @@ public QuestDB build() { } ConfigString cs = ConfigString.parse(config); ConfigView view = new ConfigView(cs); + if (httpTokenProvider != null + && (view.has("token") || view.has("username") || view.has("password"))) { + throw new IllegalArgumentException( + "httpTokenProvider cannot be combined with token, username, or password in the configuration"); + } // Validate the single cluster config exactly as both pools will, but // without connecting: the full Sender parse plus validateParameters // (ingress value keys are registry-STRING, so only the real parse @@ -229,6 +235,7 @@ public QuestDB build() { maxLifetimeMillis, housekeeperIntervalMillis, queryCloseTimeoutMillis, + httpTokenProvider, errorHandler, connectionListener, drainerListener @@ -300,6 +307,34 @@ public QuestDBBuilder housekeeperIntervalMillis(long millis) { return this; } + /** + * Supplies the bearer token on demand to every pooled ingest and query + * connection. The provider is queried for each initial WebSocket upgrade + * and reconnect, so a rotating token such as + * {@code OidcDeviceAuth::getToken} remains usable for the lifetime of this + * handle. Different pooled connections may call it concurrently, so it + * must be thread-safe. + *

+ * The provider runs on connection/reconnection paths and must not perform + * interactive sign-in. Call {@code OidcDeviceAuth.signIn()} before + * {@link #build()} when no persisted token is available. The builder and + * resulting {@link QuestDB} handle do not own or close the provider. + *

+ * Mutually exclusive with {@code token}, {@code username}, and + * {@code password} in the configuration string. + * + * @param tokenProvider supplies the current bearer token without the + * {@code "Bearer "} prefix + * @return this instance for method chaining + */ + public QuestDBBuilder httpTokenProvider(HttpTokenProvider tokenProvider) { + if (tokenProvider == null) { + throw new IllegalArgumentException("httpTokenProvider must not be null"); + } + this.httpTokenProvider = tokenProvider; + return this; + } + /** * How long a connection may remain idle in the pool before the * housekeeper closes it. {@code minSize} is always respected -- the pool diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index bfef51898..645d7b254 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -77,6 +77,7 @@ import java.util.Base64; import java.util.List; import java.util.concurrent.TimeUnit; +import java.util.function.Supplier; /** * Influx Line Protocol client to feed data to a remote QuestDB instance. @@ -1091,6 +1092,7 @@ final class LineSenderBuilder { private String httpSettingsPath; private int httpTimeout = PARAMETER_NOT_SET_EXPLICITLY; private String httpToken; + private HttpTokenProvider httpTokenProvider; // Drives the initial-connect strategy. null means "not set // explicitly", which build() resolves to SYNC when any reconnect_* // knob was tuned by the user, otherwise OFF. SYNC retries on the @@ -1449,7 +1451,7 @@ public Sender build() { tlsConfig = new ClientTlsConfiguration(trustStorePath, trustStorePassword, tlsValidationMode == TlsValidationMode.DEFAULT ? ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL : ClientTlsConfiguration.TLS_VALIDATION_MODE_NONE); } return AbstractLineHttpSender.createLineSender(hosts, ports, httpPath, httpClientConfiguration, tlsConfig, actualAutoFlushRows, httpToken, - username, password, maxNameLength, actualMaxRetriesNanos, maxBackoffMillis, actualMinRequestThroughput, actualAutoFlushIntervalMillis, protocolVersion); + username, password, maxNameLength, actualMaxRetriesNanos, maxBackoffMillis, actualMinRequestThroughput, actualAutoFlushIntervalMillis, protocolVersion, httpTokenProvider); } if (protocol == PROTOCOL_WEBSOCKET) { @@ -1463,7 +1465,7 @@ public Sender build() { ? DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS : TimeUnit.MILLISECONDS.toNanos(autoFlushIntervalMillis); - String wsAuthHeader = buildWebSocketAuthHeader(); + Supplier wsAuthHeader = buildWebSocketAuthHeader(); ClientTlsConfiguration wsTlsConfig = null; if (tlsEnabled) { @@ -1689,7 +1691,7 @@ public Sender build() { // still rescue, so build() waits for that verdict rather than pre-judging it. while (connected == null) { try { - connected = QwpWebSocketSender.connect( + connected = QwpWebSocketSender.connectWithCredentialSupplier( wsEndpoints, wsTlsConfig, actualAutoFlushRows, @@ -2287,6 +2289,9 @@ public LineSenderBuilder httpToken(String token) { if (this.httpToken != null) { throw new LineSenderException("token was already configured"); } + if (this.httpTokenProvider != null) { + throw new LineSenderException("token provider was already configured"); + } if (Chars.isBlank(token)) { throw new LineSenderException("token cannot be empty nor null"); } @@ -2294,6 +2299,65 @@ public LineSenderBuilder httpToken(String token) { return this; } + /** + * Supplies the HTTP authentication token from a provider queried as the sender builds each request, + * instead of a fixed {@link #httpToken(String) token} captured once, so a long-lived sender follows + * token refreshes - e.g. an OIDC device-flow token: {@code .httpTokenProvider(auth::getToken)}. + *
+ * Over HTTP the provider is not called at build time: the first call happens when the first row is + * started, then once per flush. Over WebSocket it depends on whether the initial connect is eager. + * With an EAGER initial connect (the default, and any {@code initial_connect_retry} other than + * {@code async}) the handshake runs during {@code build()} and queries the provider once for it; + * under {@code lazy_connect=true} - or {@code initial_connect_retry=async} - the ingest side connects + * asynchronously, so {@code build()} pulls nothing and a provider failure surfaces through the error + * inbox rather than from {@code build()}. Either way the provider is queried again once per reconnect + * handshake, so a refreshed token is presented each time the link is (re)established; an + * already-established WebSocket is not re-authenticated mid-stream. The two transports differ in + * mechanism but both keep the producer alive across a sustained token outage: over HTTP a failed pull + * leaves the request token-pending and is retried on the next row; over WebSocket an EAGER initial + * handshake fails fast when no token can be obtained, after which a pull that keeps failing on later + * reconnects is retried indefinitely, with the buffered rows held in store-and-forward, until a token + * is available again. A token outage does not terminate a running WebSocket sender, just as a + * persistent transport reconnect failure does not (store-and-forward Invariant B). + *
+ * Over HTTP the token is pulled once per request and written into the request buffer ahead of the + * buffered rows, so a token refresh is picked up on the next new batch after a successful flush. A + * failed flush preserves the buffer - token included - for a later retry and re-sends it verbatim + * rather than re-pulling the token, so a flush that keeps failing until the already-pulled token + * expires is then rejected (for example a {@code 401}); recover by discarding the buffered rows (close + * and rebuild the sender) so the next request pulls a fresh token. + *
+ * A lazily-signing-in provider can therefore be wired before the interactive sign-in completes over HTTP, + * where the first pull is deferred to the first row, and over WebSocket under {@code lazy_connect=true} + * (or {@code initial_connect_retry=async}), where nothing is pulled at build time either. What does + * require a token up front is an EAGER WebSocket connect: its initial handshake pulls one during + * {@code build()}, so that {@code build()} (or, over HTTP, the first row) fails when none can be + * obtained. Running on the send/flush and reconnect paths, the provider must + * return promptly and must not block on interactive input (see {@link HttpTokenProvider}). Supported + * over HTTP and WebSocket transport, and mutually exclusive with {@link #httpToken(String)} and + * {@link #httpUsernamePassword(String, String)}. + * + * @param httpTokenProvider supplies the current HTTP authentication token + * @return this instance for method chaining + */ + public LineSenderBuilder httpTokenProvider(HttpTokenProvider httpTokenProvider) { + if (this.username != null) { + throw new LineSenderException("authentication username was already configured ") + .put("[username=").put(this.username).put("]"); + } + if (this.httpToken != null) { + throw new LineSenderException("token was already configured"); + } + if (this.httpTokenProvider != null) { + throw new LineSenderException("token provider was already configured"); + } + if (httpTokenProvider == null) { + throw new LineSenderException("token provider cannot be null"); + } + this.httpTokenProvider = httpTokenProvider; + return this; + } + /** * Use username and password for authentication when communicating over HTTP or WebSocket protocol. *
@@ -2319,6 +2383,9 @@ public LineSenderBuilder httpUsernamePassword(String username, String password) if (httpToken != null) { throw new LineSenderException("token authentication is already configured"); } + if (httpTokenProvider != null) { + throw new LineSenderException("token provider authentication is already configured"); + } this.username = username; this.password = password; return this; @@ -3357,13 +3424,34 @@ private void appendAddress(String host, int port) { ports.add(port); } - private String buildWebSocketAuthHeader() { + private Supplier buildWebSocketAuthHeader() { + // A constant credential goes through fixedAuthHeader, not a bare lambda: the tag is what lets + // the store-and-forward drainer tell a permanently-wrong password from a rotating token that a + // fresh pull can repair, and so decide whether a 401 may quarantine an orphan slot for good. if (username != null && password != null) { String credentials = username + ":" + password; - return "Basic " + Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8)); + String header = "Basic " + Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8)); + return QwpWebSocketSender.fixedAuthHeader(header); } if (httpToken != null) { - return "Bearer " + httpToken; + String header = "Bearer " + httpToken; + return QwpWebSocketSender.fixedAuthHeader(header); + } + if (httpTokenProvider != null) { + // pull a fresh token at each (re)handshake so a long-lived WebSocket follows token + // refreshes; validateToken rejects a null/empty/blank return, or a token carrying a + // control or non-ASCII char (both forbidden by the HttpTokenProvider contract), rather + // than send a malformed or CR/LF-injected "Bearer " header + final HttpTokenProvider provider = httpTokenProvider; + return () -> { + // snapshot before validating: the concatenation below re-reads the sequence, and a + // provider is free to reuse a mutable buffer, so validating the live sequence checks + // bytes the header need not carry. See HttpTokenProvider.validateToken. + CharSequence pulled = provider.getToken(); + CharSequence token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); + return "Bearer " + token; + }; } return null; } @@ -4343,6 +4431,9 @@ private void validateParameters() { if (httpToken != null) { throw new LineSenderException("HTTP token authentication is not supported for TCP protocol"); } + if (httpTokenProvider != null) { + throw new LineSenderException("HTTP token provider authentication is not supported for TCP protocol"); + } if (retryTimeoutMillis != PARAMETER_NOT_SET_EXPLICITLY) { throw new LineSenderException("retrying is not supported for TCP protocol"); } @@ -4368,6 +4459,9 @@ private void validateParameters() { if (httpToken != null) { throw new LineSenderException("HTTP token authentication is not supported for UDP transport"); } + if (httpTokenProvider != null) { + throw new LineSenderException("HTTP token provider authentication is not supported for UDP transport"); + } if (username != null || password != null) { throw new LineSenderException("username/password authentication is not supported for UDP transport"); } diff --git a/core/src/main/java/io/questdb/client/SenderError.java b/core/src/main/java/io/questdb/client/SenderError.java index bcc6250d1..3d11995e8 100644 --- a/core/src/main/java/io/questdb/client/SenderError.java +++ b/core/src/main/java/io/questdb/client/SenderError.java @@ -42,7 +42,9 @@ * *

The {@code [fromFsn, toFsn]} span is the load-bearing correlation key — join it to * whatever the producer thread logged alongside the published-sequence value returned by - * the sender to identify the rejected data. + * the sender to identify the rejected data. Background orphan-drainer reports use + * {@link #NO_MESSAGE_SEQUENCE} for both bounds because those FSNs belong to another sender + * engine and must not be joined to the live producer's rows. * * @see SenderErrorHandler * @see LineSenderServerException @@ -151,7 +153,8 @@ public long getDetectedAtNanos() { /** * @return inclusive lower bound of the FSN span for the rejected batch — correlation key for producer-side logs. - * For {@link Category#DATA_LOSS} this is {@link #NO_MESSAGE_SEQUENCE} — the abandoned span is unknown at quarantine time. + * For {@link Category#DATA_LOSS} and background orphan-drainer reports this is + * {@link #NO_MESSAGE_SEQUENCE} — the span is unknown or does not belong to the live sender. */ public long getFromFsn() { return fromFsn; @@ -202,7 +205,8 @@ public int getServerStatusByte() { /** * @return inclusive upper bound of the FSN span for the rejected batch. - * For {@link Category#DATA_LOSS} this is {@link #NO_MESSAGE_SEQUENCE} — the abandoned span is unknown at quarantine time. + * For {@link Category#DATA_LOSS} and background orphan-drainer reports this is + * {@link #NO_MESSAGE_SEQUENCE} — the span is unknown or does not belong to the live sender. */ public long getToFsn() { return toFsn; diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/BrowserLauncher.java b/core/src/main/java/io/questdb/client/cutlass/auth/BrowserLauncher.java new file mode 100644 index 000000000..e41bfd509 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/BrowserLauncher.java @@ -0,0 +1,117 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.awt.Desktop; +import java.net.URI; +import java.net.URISyntaxException; + +/** + * Opens a verification URL in the local default browser, best-effort. Kept separate from + * {@link DeviceCodePrompt} so a runtime without the {@code java.desktop} module fails only when + * {@link DeviceCodePrompt#openBrowser()} is actually used, not when the interface loads. + */ +final class BrowserLauncher { + + // System property to disable the automatic browser launch (default: enabled). Set to "false" on a + // host that must never pop a browser - a server, automation, CI - or to keep a test run headless. + private static final String OPEN_BROWSER_PROPERTY = "questdb.client.oidc.open.browser"; + + private BrowserLauncher() { + } + + /** + * Whether the automatic browser launch is enabled - the {@code questdb.client.oidc.open.browser} + * kill-switch (default enabled; set to {@code false} to disable). Package-private so a test can assert + * the kill-switch without triggering a real browser launch, which is otherwise unobservable. + */ + static boolean isBrowserOpenEnabled() { + return Boolean.parseBoolean(System.getProperty(OPEN_BROWSER_PROPERTY, "true")); + } + + /** + * Opens {@code url} in the default browser when it is an http(s) URL and a desktop browser is + * available. Does nothing on a headless JVM, for a non-http(s) URL, on a launch failure, or when the + * {@code questdb.client.oidc.open.browser} system property is set to {@code false}. May throw a + * {@link LinkageError} when the {@code java.desktop} module is absent from the runtime; the caller + * treats that as "no browser available". + *

+ * Every OTHER failure of the desktop stack is swallowed, {@link Error}s included. Initialising the AWT + * toolkit raises {@link java.awt.AWTError} - a bare {@code Error}, not a {@code LinkageError} and not an + * {@code Exception} - when {@code assistive_technologies} in {@code $JAVA_HOME/conf/accessibility.properties} + * (or the matching system property) names a class the runtime cannot load, which is the stock configuration + * on several Linux distributions, and again when a set {@code DISPLAY} points at no X server. Catching only + * {@code Exception} let that escape {@code signIn()} on such a host - aborting a sign-in the human could + * have completed from the URL already printed, and as a type the caller's documented + * {@code catch (OidcAuthException)} does not handle. + */ + static void open(String url) { + if (!isBrowserOpenEnabled()) { + return; + } + URI uri = safeHttpUri(url); + if (uri == null) { + return; + } + try { + if (Desktop.isDesktopSupported()) { + Desktop desktop = Desktop.getDesktop(); + if (desktop.isSupported(Desktop.Action.BROWSE)) { + desktop.browse(uri); + } + } + } catch (LinkageError e) { + // The java.desktop module is absent from this runtime. Rethrow rather than swallow: the + // degrade belongs to DeviceCodePrompt.openBrowser(), whose catch is what + // DesktopFreeModulePathTest and DesktopFreeKillSwitchMain pin. + throw e; + } catch (Throwable ignore) { + // A headless display, a missing default browser, a security restriction or a desktop stack + // that cannot initialise at all must never break sign-in: the verification URL and code are + // already shown to the user. Throwable, not Exception - see the javadoc on AWTError. + } + } + + /** + * Returns {@code url} as a {@link URI} only when it parses and uses an http(s) scheme, else + * {@code null}. The verification URL is an untrusted identity-provider response field; the + * allowlist stops a javascript:, data: or file: scheme from reaching the OS browser handler. + */ + static URI safeHttpUri(String url) { + if (url == null) { + return null; + } + try { + URI uri = new URI(url); + String scheme = uri.getScheme(); + if (scheme != null && (scheme.equalsIgnoreCase("http") || scheme.equalsIgnoreCase("https"))) { + return uri; + } + return null; + } catch (URISyntaxException e) { + return null; + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/DeviceAuthorizationChallenge.java b/core/src/main/java/io/questdb/client/cutlass/auth/DeviceAuthorizationChallenge.java new file mode 100644 index 000000000..f398ebfd1 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/DeviceAuthorizationChallenge.java @@ -0,0 +1,90 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +/** + * The user-facing part of an RFC 8628 device authorization response: the code to type and the URL + * to type it at. A {@link DeviceCodePrompt} receives this object and shows it to the user. + *

+ * The {@code device_code} secret is deliberately not exposed here; it stays inside + * {@link OidcDeviceAuth} and is never shown to the user. + */ +public class DeviceAuthorizationChallenge { + private final int expiresInSeconds; + private final int intervalSeconds; + private final String userCode; + private final String verificationUri; + private final String verificationUriComplete; + + public DeviceAuthorizationChallenge( + String userCode, + String verificationUri, + String verificationUriComplete, + int expiresInSeconds, + int intervalSeconds + ) { + this.userCode = userCode; + this.verificationUri = verificationUri; + this.verificationUriComplete = verificationUriComplete; + this.expiresInSeconds = expiresInSeconds; + this.intervalSeconds = intervalSeconds; + } + + /** + * @return seconds the {@link #getUserCode() user code} stays valid. + */ + public int getExpiresInSeconds() { + return expiresInSeconds; + } + + /** + * @return minimum seconds the client must wait between polls. + */ + public int getIntervalSeconds() { + return intervalSeconds; + } + + /** + * @return the code the user enters at the {@link #getVerificationUri() verification URL}. + */ + public String getUserCode() { + return userCode; + } + + /** + * @return the URL the user opens to authorize the device. + */ + public String getVerificationUri() { + return verificationUri; + } + + /** + * @return a URL with the user code already embedded, so the user need not type it, or + * {@code null} when the identity provider does not supply one. + */ + public String getVerificationUriComplete() { + return verificationUriComplete; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/DeviceCodePrompt.java b/core/src/main/java/io/questdb/client/cutlass/auth/DeviceCodePrompt.java new file mode 100644 index 000000000..314ddb748 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/DeviceCodePrompt.java @@ -0,0 +1,112 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.str.StringSink; + +/** + * Shows an RFC 8628 device authorization challenge to the user, who then opens the verification URL + * in any browser (same machine or phone) and enters the code. {@link OidcDeviceAuth} calls this once + * per interactive sign-in, just before polling the token endpoint. + *

+ * The default is {@link #openBrowser()}: it prints instructions to {@code System.out} and also tries + * to open the verification URL in the local default browser when one is available. Use + * {@link #SYSTEM_OUT} to print only, or supply your own to render the challenge elsewhere, e.g. a + * clickable link or a QR code in a notebook. + */ +@FunctionalInterface +public interface DeviceCodePrompt { + + /** + * Prints the sign-in instructions to {@code System.out} as display-safe text, without opening a browser. + * The default prompt is {@link #openBrowser()}; use this to opt out of the browser launch. + */ + DeviceCodePrompt SYSTEM_OUT = challenge -> { + String newLine = System.lineSeparator(); + StringSink sb = new StringSink(); + sb.put(newLine); + sb.put("=== QuestDB OIDC sign-in ===").put(newLine); + sb.put("To sign in, open this URL in a browser:").put(newLine); + sb.put(" ").put(challenge.getVerificationUri()).put(newLine); + sb.put("and enter the code: ").put(challenge.getUserCode()).put(newLine); + if (challenge.getVerificationUriComplete() != null) { + sb.put("(or open this URL, the code is already filled in:").put(newLine); + sb.put(" ").put(challenge.getVerificationUriComplete()).put(')').put(newLine); + } + sb.put("Waiting for authorization, up to ").put(challenge.getExpiresInSeconds()).put(" seconds..."); + System.out.println(sb); + }; + + /** + * Returns a prompt that prints the challenge like {@link #SYSTEM_OUT} and then also tries to open + * the verification URL in the local default browser. The browser open is best-effort: it is + * skipped on a headless JVM, on a runtime without the {@code java.desktop} module, or for a + * non-http(s) URL, and never prevents sign-in. Intended for a local terminal; on a remote or + * headless host the printed URL and code remain the way in. This is the default prompt when none + * is configured. + *

+ * {@code io.questdb.client} declares {@code requires static java.desktop}, so the module is a + * compile-time dependency only and its absence at run time is just another reason to skip the + * browser. One consequence is worth knowing: a static requires is not followed during module + * resolution, so an application that runs this client as an EXPLICIT module gets the browser launch + * only when {@code java.desktop} is in its module graph anyway - because it requires it, or because + * the launch adds it ({@code --add-modules java.desktop}). Class-path applications are unaffected, + * {@code java.desktop} being resolved by default there. + * + * @return a prompt that prints the challenge and opens the verification URL in a browser + */ + static DeviceCodePrompt openBrowser() { + return openBrowser(SYSTEM_OUT); + } + + /** + * Like {@link #openBrowser()}, but renders the challenge with {@code delegate} before opening the + * browser, instead of the built-in {@code System.out} printer. + * + * @param delegate the prompt that shows the challenge to the user + * @return a prompt that runs {@code delegate} and then opens the verification URL in a browser + */ + static DeviceCodePrompt openBrowser(DeviceCodePrompt delegate) { + return challenge -> { + delegate.promptUser(challenge); + String url = challenge.getVerificationUriComplete() != null + ? challenge.getVerificationUriComplete() + : challenge.getVerificationUri(); + try { + BrowserLauncher.open(url); + } catch (LinkageError ignore) { + // the java.desktop module is absent from this runtime; the printed URL and code remain + } + }; + } + + /** + * Shows the challenge to the user. Must return quickly; waiting for the user happens afterwards + * while {@link OidcDeviceAuth} polls the token endpoint. + * + * @param challenge the user code, verification URL and timing parameters to show + */ + void promptUser(DeviceAuthorizationChallenge challenge); +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/FileTokenStore.java b/core/src/main/java/io/questdb/client/cutlass/auth/FileTokenStore.java new file mode 100644 index 000000000..489ef2139 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/FileTokenStore.java @@ -0,0 +1,2193 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.cutlass.json.JsonException; +import io.questdb.client.cutlass.json.JsonLexer; +import io.questdb.client.cutlass.json.JsonParser; +import io.questdb.client.std.Chars; +import io.questdb.client.std.Numbers; +import io.questdb.client.std.NumericException; +import io.questdb.client.std.str.DirectUtf8Sink; +import io.questdb.client.std.str.StringSink; + +import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.AccessDeniedException; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.DirectoryStream; +import java.nio.file.FileAlreadyExistsException; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.FileAttribute; +import java.nio.file.attribute.FileTime; +import java.nio.file.attribute.PosixFilePermission; +import java.nio.file.attribute.PosixFilePermissions; +import java.util.Arrays; +import java.util.EnumSet; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.locks.ReentrantLock; + +/** + * The default {@link TokenStore}: one plaintext JSON file per OIDC configuration under a directory, with + * the refresh token protected at rest by file permissions (0600 file, 0700 directory) rather than by + * encryption. This matches what {@code gcloud}, {@code aws} and {@code gh} do; for encryption at rest, + * supply a {@link TokenStore} backed by an OS keychain or a secrets manager instead. + *

+ * The default location is {@code ${user.home}/.questdb/oidc-tokens/}, overridable with the + * {@code questdb.client.oidc.token.store.dir} system property. The file name is + * {@code .json}, so several configurations coexist and the name leaks neither the + * endpoint nor the client id. The on-disk format (file name, JSON schema, write protocol, lock-file + * protocol) is a deliberately language-neutral contract so other QuestDB clients can share the file. + *

+ * One store, one active login. {@link TokenStoreKey} names a CONFIGURATION - client id, endpoints, + * scope, audience, groups-in-token mode - and no field of it names a subject, so two people signing in + * through the same configuration address the same file and the later sign-in overwrites the earlier one. + * Separate application users need separate stores ({@link #at(Path)} on a per-user directory, or a per-user + * {@code questdb.client.oidc.token.store.dir}), not a reliance on the key to tell them apart. The default + * location is per OS user already, so this only arises inside one OS user - a shared service account, or a + * process signing in on behalf of several people. + *

+ * Not authenticated. The file carries no MAC or signature, so {@link #load} cannot distinguish a + * planted credential from its own: anyone able to WRITE the file can substitute a well-formed entry that + * this store will adopt and the caller will present. Permissions are the control, not the format. What load + * itself rejects is corruption and mix-ups - an oversized, malformed or unparseable file, an entry whose + * recorded identity fields do not match the key being loaded, and (on POSIX) an entry discarded outright + * when the directory is writable by other local users. The token-shape checks that guard what actually gets + * served - rejecting an entry with no usable token or a token carrying control/non-ASCII characters, and + * clamping the recorded expiry and lifetime rather than trusting them - do not run here; they live one level + * up in {@code OidcDeviceAuth.adopt()}, the choke point every {@link TokenStore} (including a caller's own + * SPI implementation) passes through. + *

+ * Integrity (always). {@link #save} writes a sibling temp file then atomically renames it over the + * target, so a crash or an overlapping reader - in any process or language - sees the whole old or whole + * new file, never a torn credential. A required directory-wide {@code .store.lock} serialises trust recovery, + * its full-directory discard, and token reads/writes, so an earlier discard cannot delete a different + * identity's completed save. It is held only for bounded filesystem work, never for an IdP request. + *

+ * Rotating refresh tokens (Layer 2). {@link #inLock} serialises the read-refresh-write of a token + * refresh across processes with an {@code O_CREAT|O_EXCL} lock file ({@code .lock}) - not an OS + * advisory lock, which a Java {@code FileLock} and a Python {@code flock} cannot reliably share. It steals + * a stale lock left by a crashed holder, and degrades to running without the lock (Layer 1 still protects + * integrity) rather than stall a sign-in if it cannot acquire one. + *

+ * That degrade has a residual worth understanding: if a peer's refresh genuinely outlasts the acquire + * budget (a slow or stalled IdP), or its lock is judged stale and stolen mid-refresh, two processes can + * POST the same parent refresh token concurrently. On an IdP that does not detect refresh-token reuse this + * costs only a redundant refresh; on one that DOES (for example Auth0's default), reusing one parent token + * twice can revoke the whole token family, forcing every process to re-run the interactive device flow - + * which, for a headless {@code getToken()} consumer with no interactive fallback, is a hard failure until a + * human re-signs in. If that matters, widen the acquire budget / staleness window, or back the store with a + * keychain or secrets manager instead. + *

+ * The store never writes a token value into a log or an exception message; only file paths and IO error + * kinds may surface. + */ +public final class FileTokenStore implements TokenStore { + public static final String TOKEN_STORE_DIR_PROPERTY = "questdb.client.oidc.token.store.dir"; + // wait this long for the per-identity lock before giving up and running without it (Layer 1 still + // guards integrity). Kept short because getToken() can take this lock on the latency-sensitive flush + // path: a real refresh round-trip is sub-second, so a peer not done within this budget is treated as + // too slow and we degrade to a lock-free refresh rather than stall the caller + private static final long DEFAULT_LOCK_ACQUIRE_BUDGET_MILLIS = 3_000L; + // treat a lock older than this as abandoned by a crashed holder and steal it. Must stay comfortably + // above the longest a live holder can hold it (one refresh under the lock) so a live holder is never + // stolen from. That hold runs, in order, the TCP connect, the TLS handshake, send, await, parse, and a + // body drain on a parse failure - six phases, each separately bounded by the client's HTTP timeout, + // because OidcDeviceAuth.httpConfig() derives BOTH the connect timeout and the request timeout from + // httpTimeoutMillis and HttpClient spends them separately (so up to LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE = 6x + // it; OidcDeviceAuth caps that timeout at 120s, hence up to ~720s). Only DNS resolution is left to the OS. + // At the 30s default httpTimeoutMillis the worst case is ~180s, so this 10-minute window leaves ample + // headroom; build() enforces a 6x floor, so a caller who raises httpTimeoutMillis toward the cap must + // raise this window to match. A pathological DNS hang beyond the headroom can still let a peer steal a + // live holder's lock mid-refresh, degrading to a concurrent refresh of the same parent refresh token: a + // redundant refresh on most IdPs, but on a reuse-detecting one (e.g. Auth0 default) a possible + // token-family revocation and re-prompt / headless hard-failure (see the class javadoc residual note) + private static final long DEFAULT_LOCK_STALE_MILLIS = 600_000L; + private static final FileAttribute> DIR_ATTRS = + PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rwx------")); + // Cross-process lock covering directory trust recovery and token-file operations. Unlike the per-identity + // refresh lock, this lock is REQUIRED: running a save without it could let a concurrent directory-wide + // untrusted-content sweep delete the file after save() has returned successfully. + private static final String DIRECTORY_LOCK_FILE_NAME = ".store.lock"; + // Once the directory is owner-only and has no pending distrust sweep, the directory lock protects only + // bounded atomic filesystem work, never the IdP round-trip covered by lockStaleMillis. Give that trusted + // state a short renewable lease so a process killed during writeAndFlush cannot strand every load/save for + // the refresh lock's 10-minute default. A live holder refreshes the mtime well inside this window; a dead + // one stops refreshing and is reclaimable within the default 3-second acquire budget. An untrusted or + // indeterminate directory retains the conservative configured staleness window: a paused sweep must never + // resume after a peer has displaced its lock and delete that peer's completed save. + private static final long DIRECTORY_LOCK_EMPTY_STEAL_GRACE_MILLIS = 2_000L; + private static final long DIRECTORY_LOCK_HEARTBEAT_MILLIS = 500L; + private static final long DIRECTORY_LOCK_STALE_MILLIS = 2_000L; + // the same owner-only directory permissions as DIR_ATTRS, in the form setPosixFilePermissions wants, so + // restrictToOwner can re-assert them on a directory that already exists with looser permissions + private static final Set DIR_PERMS = PosixFilePermissions.fromString("rwx------"); + // steal an empty/unstamped lock once it has existed at least this long. A validly held lock always + // carries an owner stamp (acquireLock stamps it immediately after the exclusive create); an empty lock is + // therefore either a peer momentarily between its create and its stamp - microseconds, far below this + // grace - or one a holder abandoned by crashing in that tiny window. Stealing on this short grace instead + // of the full staleness window keeps a post-crash empty lock from wedging peers (into lock-free refreshes) + // for the whole staleness window, while the grace stays well above the create->stamp gap so a peer + // mid-stamp is never pre-empted (which would force the rightful holder to degrade) + private static final long EMPTY_LOCK_STEAL_GRACE_MILLIS = 5_000L; + private static final FileAttribute> FILE_ATTRS = + PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rw-------")); + // Length of the identity fingerprint every file this store writes is named after: TokenStoreKey.hash() + // is a SHA-256 rendered as lowercase hex, so 64 characters. Used to tell this store's own files apart + // from whatever else shares the directory - see discardUntrustedDirectoryContents(). + private static final int HASH_NAME_LENGTH = 64; + private static final char[] HEX = "0123456789abcdef".toCharArray(); + private static final int JSON_LEXER_CACHE_SIZE = 1024; + private static final int JSON_LEXER_MAX_VALUE_BYTES = 1 << 20; + private static final long LOCK_POLL_SLICE_MILLIS = 50L; + private static final Logger LOG = LoggerFactory.getLogger(FileTokenStore.class); + // reject a token file larger than this; a real entry is a few KB even with a group-laden id token, so + // anything past this is corrupt or hostile and is not read into memory + private static final long MAX_FILE_BYTES = 1 << 20; + // upper bound on the configurable lock acquire budget. getToken() can take this lock on the + // latency-sensitive flush path, so a caller-supplied budget is kept short: a real peer refresh is + // sub-second, and bounding the wait stops a misconfigured budget from stalling a flush before it degrades + // to a lock-free refresh (Layer 1 still guards integrity). Stays well below DEFAULT_LOCK_STALE_MILLIS so a + // waiter degrades long before it could begin stealing live locks. + private static final long MAX_LOCK_ACQUIRE_BUDGET_MILLIS = 30_000L; + // reject a lock file larger than this before reading it: the .lock file sits in the same + // attacker-writable directory as the token file, and a real owner stamp (millis + UUID) is a few dozen + // bytes, so anything past this cap is corrupt or hostile and is not read into memory + private static final int MAX_LOCK_FILE_BYTES = 1 << 12; + // Serializes both same-identity refresh critical sections and same-directory token-file operations WITHIN + // this JVM. Two OidcDeviceAuth instances for one identity in a single process (e.g. an ILP Sender and a + // QwpQueryClient) have separate instance locks, so only the identity entry stops them running the + // read-refresh-write concurrently and double-POSTing the same parent refresh token - which a + // reuse-detecting IdP revokes the whole token family for. The directory entry stops a distrust sweep for + // one identity from deleting a different identity's concurrent save. Neither in-process lock degrades. + // + // Identity entries are keyed on directory + fingerprint; directory entries use the normalized directory + // alone. Each exists only while it has a holder or waiter and nothing is left behind once the last one + // leaves. TokenStoreKey is public and inLock() is public API, so how many distinct identities a process + // mints is the caller's business - one per end user in a multi-tenant service is a perfectly ordinary + // shape - and an entry per identity EVER SEEN, which an unpruned map gives, roots a 64-char hash plus a + // lock for the life of the JVM. Retiring on the last release bounds the map by concurrent operations. + // + // A fixed stripe table also bounds it, and was tried, but over-serializing is not the free trade it + // looks: this lock is held across a whole token-endpoint round trip while the caller also holds its + // OidcDeviceAuth instance lock, and the acquire has no budget. Two unrelated identities landing on one + // stripe therefore do not merely "wait for each other" - one tenant's ILP flush blocks on another + // tenant's stalled refresh for that holder's entire worst case, which getToken() sizes at six times + // httpTimeoutMillis plus an OS connect stall, and every other caller on the blocked instance fails + // meanwhile. That also made OidcDeviceAuth.getToken()'s "two instances sharing ONE IDENTITY" contract + // untrue. Per-identity entries serialize exactly the same-identity pairs the double-POST rule needs and + // nothing else, so the contract holds as written. + // + // compute()/computeIfPresent() apply their function atomically under the bin lock, so `users` needs no + // synchronization of its own and retirement has no race: an arriving thread cannot observe an entry + // that a departing one is removing. + private static final ConcurrentHashMap PROCESS_LOCKS = new ConcurrentHashMap<>(); + // Windows can fail the atomic token-file rename with a transient AccessDeniedException (a sharing violation) + // when a concurrent reader in any process holds the target open; retry the rename this many times on a short + // backoff before giving up, so a routine read/write overlap does not needlessly degrade persistence. Kept + // small - persistence is best-effort and the in-memory token is valid regardless. + private static final int REPLACE_MAX_ATTEMPTS = 5; + private static final long REPLACE_RETRY_SLEEP_MILLIS = 20L; + private static final int SCHEMA_VERSION = 1; + // Name of the "this directory is untrusted" sentinel, dropped into the store directory the instant + // restrictToOwner finds it writable by other local users - BEFORE it tightens the permissions to 0700. + // The chmod that publishes owner-only to every other process also erases the evidence the directory was + // ever exposed, so a concurrent caller reading the permissions in the window between that chmod and the + // discard sweep would see an owner-only directory, judge it trusted, and adopt an entry a local attacker + // planted while it stood open. The sentinel outlives the chmod: discardUntrustedDirectoryContents removes + // it only after it has swept every untrusted entry, and every trust check treats its presence as + // untrusted whatever the permission bits say. Its name has no 64-hex store prefix, so the sweep never + // mistakes it for an entry to delete, and it carries no secret. Part of the frozen cross-language + // contract (design/oidc-token-persistence.md), so the Python client marks and clears the same name. + private static final String UNTRUSTED_SENTINEL_NAME = ".untrusted"; + // set once if the platform cannot enforce owner-only POSIX permissions on the token files (e.g. Windows), + // so the at-rest protection falls back to the directory's inherited ACL; warns the user exactly once + // (compareAndSet, so a race between two threads still prints a single warning) + private static final AtomicBoolean warnedNoPosixPerms = new AtomicBoolean(); + private static final AtomicBoolean warnedStoreDirIoFailure = new AtomicBoolean(); + private static final AtomicBoolean warnedStuckUntrustedSentinel = new AtomicBoolean(); + private static final AtomicBoolean warnedTightenedStoreDir = new AtomicBoolean(); + private static final AtomicBoolean warnedUnprotectedStoreDir = new AtomicBoolean(); + // Test seam, null in production: runs in the gap between judging a lock stale and capturing it with + // the ATOMIC_MOVE. That gap IS the interleaving stealIfStale's capture-verify defends -- a peer + // replacing the abandoned lock with a live one -- and nothing else can force it, so without this the + // whole capture/verify/restore could be collapsed back into the bare deleteIfExists its own comment + // forbids and no test would go red. Not final: the test installs it reflectively, as it already does + // to reach stealIfStale from the separate test module. + @TestOnly + private volatile Runnable beforeCaptureHook; + // Test seam, null in production: runs after restrictToOwner has read a loose directory's permissions but + // before its chmod. A test removes the directory here so the real setPosixFilePermissions call fails, + // pinning that the success warning and its once-per-JVM latch happen only after a completed syscall. + @TestOnly + private volatile Runnable beforeDirectoryTightenHook; + // Test seam, null in production: runs at the top of discardUntrustedDirectoryContents, in the window + // AFTER restrictToOwner has tightened the directory to 0700 and dropped the untrusted sentinel but + // BEFORE the sweep clears it. The directory recovery lock is held while this fires, so a concurrent load + // or save must wait until the sweep has completed rather than act on a stale distrust verdict. Installed + // reflectively, like beforeCaptureHook. + @TestOnly + private volatile Runnable beforeUntrustedDiscardHook; + private final Path directory; + private final long lockAcquireBudgetMillis; + // Namespaces this store's entries in PROCESS_LOCKS, so two stores over DIFFERENT directories never + // contend even when they share one OIDC configuration. Normalized once here rather than per acquire: + // "a" and "./a" must not mint two locks over one directory, which would be the dangerous direction. + // toAbsolutePath().normalize() rather than toRealPath(): the directory may not exist yet (the store + // creates it lazily), and a key that changed once it did would be worse than one that ignores symlinks. + // Two stores reaching one directory through different symlinks therefore still get separate in-process + // locks; the cross-process lock file, which they DO share, remains the guard for that shape. + private final String lockNamespace; + private final long lockStaleMillis; + + public FileTokenStore(Path directory) { + this(directory, DEFAULT_LOCK_ACQUIRE_BUDGET_MILLIS, DEFAULT_LOCK_STALE_MILLIS); + } + + /** + * Advanced constructor exposing the cross-process lock-file timings used by + * {@link #inLock(TokenStoreKey, CriticalSection)}. Most callers should use {@link #FileTokenStore(Path)} + * or the factories, which apply sensible defaults. + * + * @param directory the directory to hold the token files + * @param lockAcquireBudgetMillis how long {@code inLock} waits to acquire a peer's lock before degrading + * to a lock-free refresh rather than stalling a sign-in. Must be positive + * and at most 30_000 (30s): {@code getToken()} can wait it out on the + * latency-sensitive flush path, so it is kept short + * @param lockStaleMillis an identity lock older than this is treated as abandoned by a crashed + * holder and stolen; it is also the conservative directory-lock window + * while a distrust recovery is pending. It MUST exceed the longest a live + * identity-lock holder can hold the lock: the + * under-lock refresh runs the TCP connect, the TLS handshake, send, await, + * parse, and a body drain on a parse failure - six phases, each bounded by + * the {@code OidcDeviceAuth} httpTimeoutMillis, because httpConfig() derives + * BOTH the connect timeout and the request timeout from it and HttpClient + * spends them separately (so up to ~6x it, ~720s at the 120s timeout cap). + * Only DNS resolution is left to the OS. Size this window above 6x + * httpTimeoutMillis plus a small DNS allowance, or a peer can judge a live + * holder stale and steal its lock mid-refresh, reopening the cross-process + * refresh race this lock exists to prevent. The store cannot see the + * client's timeout, so sizing this correctly is the caller's responsibility; + * the default is 600_000, ample for the 30s default httpTimeoutMillis (~180s + * worst case) - and {@code build()} enforces the 6x floor so a raised + * httpTimeoutMillis forces a matching lockStaleMillis. + */ + public FileTokenStore(Path directory, long lockAcquireBudgetMillis, long lockStaleMillis) { + if (directory == null) { + throw new OidcAuthException("the token store directory is required"); + } + if (lockAcquireBudgetMillis <= 0) { + throw new OidcAuthException("the token store lockAcquireBudgetMillis must be positive"); + } + if (lockAcquireBudgetMillis > MAX_LOCK_ACQUIRE_BUDGET_MILLIS) { + // getToken() can wait out this budget on the latency-sensitive flush path, so an unbounded value + // would let a misconfiguration stall a flush; keep it short - it degrades to a lock-free refresh + throw new OidcAuthException() + .put("the token store lockAcquireBudgetMillis must not exceed ").put(MAX_LOCK_ACQUIRE_BUDGET_MILLIS); + } + // a non-positive staleness window makes every freshly created lock look abandoned, so acquirers would + // steal each other's live locks; keep it well above one refresh round-trip (see the default) + if (lockStaleMillis <= 0) { + throw new OidcAuthException("the token store lockStaleMillis must be positive"); + } + this.directory = directory; + this.lockNamespace = directory.toAbsolutePath().normalize().toString(); + this.lockAcquireBudgetMillis = lockAcquireBudgetMillis; + this.lockStaleMillis = lockStaleMillis; + } + + /** + * @param directory the directory to hold the token files; created on first write with owner-only + * permissions + * @return a store rooted at the given directory + */ + public static FileTokenStore at(Path directory) { + return new FileTokenStore(directory); + } + + /** + * @return a store at {@code ${questdb.client.oidc.token.store.dir}} if that system property is set, + * otherwise at {@code ${user.home}/.questdb/oidc-tokens/} + */ + public static FileTokenStore atDefaultLocation() { + String override = System.getProperty(TOKEN_STORE_DIR_PROPERTY); + Path dir = override != null && !override.isEmpty() + ? Paths.get(override) + : Paths.get(System.getProperty("user.home"), ".questdb", "oidc-tokens"); + return new FileTokenStore(dir); + } + + @Override + public void clear(TokenStoreKey key) { + if (!Files.isDirectory(directory)) { + return; // nothing is persisted yet; do not create the directory just to clear it + } + // Interrupt-neutral, for the reason load() and save() are, and more sharply. Those two abandon file + // I/O; this one is a local DELETE whose entire purpose is to erase a secret, so there is nothing to + // abandon on a cancellation and "we were cancelled" is not a reason to leave a plaintext refresh token + // behind. Routed through inLock, a merely CARRIED interrupt flag - the standard state of a cancelled + // or shutting-down thread, which is exactly where a sign-out runs - made inLock skip the action and + // return false, which this method discarded: clear() returned normally, clearCache() reported success, + // the file stayed on disk, and the next process start silently resumed the old identity. + final boolean wasInterrupted = Thread.interrupted(); + try { + // delete under the cross-process lock, like the read-refresh-write, so a peer's in-flight refresh + // cannot resurrect the entry by atomically renaming a fresh file in just after we delete. inLock + // cleans up its own lock file and degrades to lock-free if it cannot acquire one. Cross-process + // clear is still best-effort: a peer holding a live in-memory token may legitimately re-persist + // later - clearing forces a fresh sign-in for THIS process regardless, since the caller resets its + // in-memory token state. + final CriticalSection delete = () -> { + try { + Files.deleteIfExists(tokenFile(key)); + } catch (IOException e) { + throw new OidcAuthException(e).put("could not remove the OIDC token store file"); + } + // Also remove any write temp for this identity. A crash between createTempFile and the atomic + // rename orphans a .tmp holding the FULL serialized entry - access, id and + // refresh tokens in plaintext - and until now nothing here reclaimed it: sweepStaleTempFiles + // runs only from save(), so a caller that clears and never signs in again left a live refresh + // token on disk indefinitely, contradicting this method's contract. Sweep at ANY age, unlike + // save()'s staleness-bounded sweep: clear() is an explicit "forget this credential", and a + // temp a concurrent save is mid-rename on is a benign loser - its rename fails, persistence + // is best-effort, and the caller is discarding the credential anyway. + sweepTempFiles(key.hash(), 0L); + return true; + }; + if (wasInterrupted || !inLock(key, delete)) { + // Run without coordination if inLock declined to RUN the action - a live cancellation, or it + // could not coordinate at all. Do the same for a carried interrupt consumed above, before inLock + // could mistake it for a live cancellation, so a close() interrupt delivered just before this + // call cannot be hidden for the duration of a peer refresh. This action always returns true, so + // the fallback cannot double-delete. The cross-process lock only orders us against a peer's + // in-flight refresh, and losing that ordering costs at worst a peer re-persisting later, which + // this method already documents as best-effort. Leaving the secret behind is not a trade this + // call may make. + delete.run(); + } + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + + @Override + public boolean inLock(TokenStoreKey key, CriticalSection action) { + // First serialize other threads of THIS JVM sharing the same identity: the cross-process file lock below + // degrades to lock-free after lockAcquireBudgetMillis, which is a fine cross-process fallback but must + // not let two threads of one process run the critical section at once (they would double-POST the same + // rotating refresh token and get the whole family revoked on a reuse-detecting IdP). This lock is not + // subject to the file lock's degrade. ReentrantLock is safe even though inLock's contract forbids + // nesting - a mistaken re-entry cannot self-deadlock. + // An interrupt CARRIED ON ENTRY is the caller's own state, not a signal aimed at any wait below: + // preserve it and abort before touching a lock. This has to be tested BEFORE the acquire, not after + // it: ReentrantLock.lockInterruptibly() begins with Thread.interrupted(), so it throws even on a FREE, + // UNCONTENDED lock and CLEARS the flag - a carried interrupt was therefore misread as a live + // cancellation by the catch below, which does not re-assert, so the caller's cancellation signal was + // destroyed and the critical section skipped on a lock nobody held. Aborting here also avoids + // acquiring a lock for a critical section we should not start, which would only delay the caller and + // risk stranding a lock file for its whole staleness window. + if (Thread.interrupted()) { + Thread.currentThread().interrupt(); + return false; + } + // Retain before the acquire and release in the outermost finally, so every exit - the interrupted + // acquire below included - gives the claim back exactly once. + // Namespaced by DIRECTORY as well as identity. TokenStoreKey names a CONFIGURATION and carries no + // directory, so keying on it alone made two stores over per-user directories - the multi-user recipe + // this class and the README both prescribe - queue on one lock while touching different files. That + // lock is held across a whole token-endpoint round trip and its acquire has no budget, so one user's + // stalled refresh blocked another's getToken() on the flush path, for exactly the reason the stripe + // table considered above was rejected. + final String lockIdentity = processLockIdentity(key); + final ProcessLock processLock = retainProcessLock(lockIdentity); + // lockInterruptibly, never lock(): a peer thread on this identity holds this for a whole refresh round + // trip, and an interrupt is the ONLY lever that reaches a caller stuck behind it. QWP's + // ConnectCancellation.cancel() interrupts a thread inside a credential pull precisely so close() can + // unstick it; an uninterruptible acquire here sleeps through that, outlives close()'s shutdown budget, + // and leaves the native client, the cursor engine and the slot lock to a delegated teardown. + try { + processLock.lock.lockInterruptibly(); + } catch (InterruptedException e) { + // Interrupted WAITING for the process lock: a live cancellation, acted on by abandoning the + // refresh. RE-ASSERT the flag before returning. OidcDeviceAuth records whether action.run() was + // entered to distinguish this no-attempt result from a refresh that ran and failed; it cannot use + // the flag alone because an unrelated cancellation can arrive during a real refresh. The flag must + // still survive as the caller's cancellation signal and as the reason this no-action return is + // reported as an abandoned wait. load() and save() already preserve it; only this method consumed + // the signal. + // + // Safe to restore here: nothing below this point performs interruptible I/O - + // releaseProcessLock is a ConcurrentHashMap update, and no lock file was ever opened. + releaseProcessLock(lockIdentity); // the acquire never happened, so give the claim straight back + Thread.currentThread().interrupt(); + return false; + } + try { + // A LIVE interrupt that landed between the acquire above and here - the carried case is already + // handled before the acquire. Same answer either way: preserve it and abort before touching any + // lock file, rather than clear it to push the FileChannel I/O through (a set flag turns that into + // ClosedByInterruptException) and run the critical section anyway. + if (Thread.interrupted()) { + Thread.currentThread().interrupt(); + return false; + } + Path lock = null; + // the unique owner nonce stamped into the lock when we acquired it, or null if we did not (or could + // not) acquire one and are running lock-free; releaseLock deletes the lock only when it still carries + // this nonce, so we never delete a lock a peer has since stolen + String nonce = null; + // set when an interrupt arrives while we poll for the cross-process lock; see acquireLock + boolean cancelled = false; + try { + // Prepare/recover the directory under the required directory-wide lock, then release it + // BEFORE acquiring the per-identity refresh lock. The action below commonly calls load() and + // save(), which briefly reacquire the directory lock while the identity lock is held; never + // holding the two file locks while acquiring each other keeps the cross-process order acyclic. + withDirectoryLock(isDirectoryTrusted -> null); + lock = lockFile(key); + nonce = acquireLock(lock, false); + } catch (InterruptedException e) { + // Arrived DURING the poll, so it is a live cancellation rather than carried state. Consumed + // for the same reason as the process-lock wait above. + cancelled = true; + } catch (IOException | RuntimeException e) { + // could not prepare the lock directory or file; run without the cross-process lock. Layer-1 + // atomic replacement still keeps every reader consistent - only a rotating-refresh-token race + // across processes is left unguarded for this one refresh. + // + // RuntimeException as well as IOException: this is lock BOOKKEEPING, and none of it is a + // reason to fail a sign-in the caller could otherwise complete. A SecurityManager denying + // the directory or the lock file throws SecurityException, and a filesystem that cannot + // carry POSIX permissions throws UnsupportedOperationException - both unchecked, both + // previously escaping past the caller's degrade path and aborting signIn()/getToken() + // outright, which is the opposite of what a best-effort store should do. + nonce = null; + } + try { + // The critical section is a fresh HTTP round trip - exactly the work a cancellation is trying + // to stop - so never start it once an interrupt has been observed. isInterrupted() rather than + // interrupted() for the late arrival: we did not catch that one, so it is not ours to clear. + if (cancelled || Thread.currentThread().isInterrupted()) { + if (cancelled) { + // acquireLock's poll consumed the flag; put it back for the same reason the + // process-lock wait above does, so the caller retains its cancellation signal and can + // report why action was not entered. The late-arrival case needs nothing - that flag + // is still set. + // Nothing below performs interruptible I/O: cancelled implies acquireLock threw, + // so nonce is null and the finally below skips releaseLock. + Thread.currentThread().interrupt(); + } + return false; + } + return action.run(); + } finally { + if (nonce != null) { + // Release under the same shield, and re-read the flag here rather than reusing the value + // above: the interrupt that matters usually arrives DURING action.run() (close() breaking + // a stuck credential pull). Without this, releaseLock's channel read throws + // ClosedByInterruptException, the lock file survives its whole staleness window, and every + // peer degrades to an unserialized refresh meanwhile. + boolean wasInterruptedInSection = Thread.interrupted(); + try { + releaseLock(lock, nonce); + } catch (RuntimeException e) { + // This runs in a finally, AFTER the critical section returned. A throw here would + // replace the caller's completed refresh with an exception - the refresh happened, + // the token is live, and the caller would be told the sign-in failed. releaseLock + // already absorbs IOException; a SecurityManager denying the delete throws + // SecurityException, which is unchecked and was escaping. Same operator-visible + // warning, same degrade: peers run unserialized until the lock goes stale. + // sanitized: an IO error message embeds the operator-supplied store path, which is + // the one untrusted string these warnings put in front of a terminal + LOG.warn("could not release the OIDC token store lock; peer operations may be delayed " + + "until it goes stale [error={}]", + OidcDeviceAuth.sanitizeForDisplay(e.getMessage())); + } finally { + if (wasInterruptedInSection) { + Thread.currentThread().interrupt(); + } + } + } + } + } finally { + processLock.lock.unlock(); + releaseProcessLock(lockIdentity); + } + } + + @Override + public PersistedToken load(TokenStoreKey key) { + // Every file operation below goes through FileChannel, an InterruptibleChannel: a thread that + // merely CARRIES a set interrupt flag makes the first read throw ClosedByInterruptException and + // closes the channel, and the flag survives. Two callers routinely arrive here with it set - an + // ILP producer on a pooled or managed thread, where interrupt is the standard cancellation + // signal, and the sender's own I/O thread, which close() interrupts to break a stuck credential + // pull. Neither means "abandon the token store", so clear the flag for the duration of the file + // I/O and restore it on the way out: the caller's cancellation signal survives intact, while the + // store's reads stop being collateral damage. + final boolean wasInterrupted = Thread.interrupted(); + try { + // Assert the directory on the READ path too, not only on the write paths. adopt() rejects an + // entry carrying only a refresh token, but a COMPLETE planted entry - a dummy access token, the + // attacker's refresh token, and an expiry already in the past - takes the normal path and the + // next silent refresh presents their credential. Closing that needs the container checked as + // well as the artefact: a store directory another local user can write is one whose contents + // were never ours to trust. Fail closed - a null return is the documented outcome for any + // unusable entry and degrades to a refresh or an interactive sign-in. + try { + return withDirectoryLock(isDirectoryTrusted -> { + if (!isDirectoryTrusted) { + // The directory was WRITABLE by other local users before the recovery lock tightened + // it, so this caller must not adopt anything from that exposure even though the sweep + // has now completed. A later call may trust freshly written entries after the sentinel + // is gone. + return null; + } + Path file = tokenFile(key); + byte[] bytes; + try { + bytes = readBounded(file); + } catch (NoSuchFileException e) { + return null; + } catch (IOException e) { + throw new OidcAuthException(e).put("could not read the OIDC token store file"); + } + if (bytes == null) { + return null; + } + return parseAndVerify(key, bytes); + }); + } catch (IOException e) { + // THROW, do not return null. load()'s contract makes the two mean opposite things: null is + // the definitive "there is nothing here", which latches storeLoadAttempted and ends the + // reads for the life of the OidcDeviceAuth, while a throw reads as a transient fault and is + // retried under the store-load back-off. What directory preparation reports here is squarely + // transient - Files.createDirectories failing because a home directory is not mounted yet, + // EIO/ESTALE on an NFS home, a momentarily read-only or full filesystem - so answering null + // told every later call that a store holding a perfectly good refresh token was empty. The + // process then re-runs the interactive device flow, and for the headless getToken() + // consumer this persistence exists to serve, that is a hard failure with no recovery short + // of a restart. The sibling arm below already throws for readBounded's IOException, and + // save() lets this very exception propagate; only this path disagreed. + warnStoreDirIoFailureOnce(e); + throw new OidcAuthException(e).put("could not prepare the OIDC token store directory"); + } + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + + @Override + public void save(TokenStoreKey key, PersistedToken token) { + // interrupt-neutral for the same reason as load(): a carried interrupt flag would otherwise abort + // the write or the atomic rename half-way and leave the rotated refresh token unpersisted + final boolean wasInterrupted = Thread.interrupted(); + try { + byte[] content = serialize(key, token); + try { + withDirectoryLock(isDirectoryTrusted -> { + // withDirectoryLock has already discarded every untrusted entry when this verdict is + // false. Write the fresh token only afterwards, while the same directory-wide lock is + // still held, so a peer cannot arrive with the old verdict and sweep this completed save. + sweepStaleTempFiles(key.hash()); + Path target = tokenFile(key); + Path tmp = createTempFile(key.hash()); + boolean moved = false; + try { + writeAndFlush(tmp, content); + replaceTarget(tmp, target); + moved = true; + } finally { + if (!moved) { + try { + Files.deleteIfExists(tmp); + } catch (IOException ignore) { + // best-effort: never let the cleanup failure replace the write/rename failure + // that is unwinding; sweepStaleTempFiles reclaims the orphan on a later save + } + } + } + return null; + }); + } catch (IOException e) { + throw new OidcAuthException(e).put("could not persist the OIDC token to the token store"); + } + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + + long getLockStaleMillis() { + // exposed package-private so OidcDeviceAuth.build() can verify this window dominates the worst-case time + // a coordinated refresh holds the lock, before a peer could otherwise judge a live lock stale and steal it + return lockStaleMillis; + } + + private static void createLockFile(Path lock, String nonce) throws IOException { + // Exclusively create the lock (O_CREAT|O_EXCL via CREATE_NEW), then write the owner nonce into that same + // open channel before closing it. The file exists empty only for the tiny window between the create and + // the stamp; a GC/safepoint pause (or a cross-machine clock skew) CAN land in that window, so what keeps + // our freshly-created lock from being stolen as empty-and-stale is the applicable grace (5 seconds for + // an identity lock; 2 seconds for a trusted directory lock) sitting well above it, not the absence of + // the window. FileAlreadyExists means a peer already holds it. + // releaseLock and stealIfStale verify this nonce before deleting. Keep the owner-only perms (and the + // non-POSIX fallback) to match the store's other files. + final byte[] bytes = nonce.getBytes(StandardCharsets.UTF_8); + try { + writeNewFile(lock, bytes, FILE_ATTRS); + } catch (UnsupportedOperationException e) { + warnNoPosixPermsOnce(); + writeNewFile(lock, bytes); + } + } + + private static void deleteCapturedLock(Path captured) { + // best-effort cleanup of a lock we atomically captured during a steal; a leftover .tmp is reclaimed by + // sweepStaleTempFiles on a later save + try { + Files.deleteIfExists(captured); + } catch (IOException ignore) { + // reclaimed by sweepStaleTempFiles later + } + } + + /** + * Whether {@code name} starts with the 64-character lowercase-hex identity fingerprint every file this + * store writes is named after. + *

+ * This is the test for "we could have written this", used where a sweep has no single + * {@link TokenStoreKey} to scope itself by and must therefore recognise the store's files by shape + * rather than by an exact name. It is deliberately a prefix test: the entry is + * {@code .json} but a write temp is {@code .tmp}, so only the leading fingerprint + * is common to both. + *

+ * Case matters. {@code TokenStoreKey} renders the digest through {@link #HEX}, which is lowercase, so + * an uppercase-hex name is not one of ours and is left alone. + */ + private static boolean hasStoreHashPrefix(String name) { + if (name.length() < HASH_NAME_LENGTH) { + return false; + } + for (int i = 0; i < HASH_NAME_LENGTH; i++) { + final char c = name.charAt(i); + if ((c < '0' || c > '9') && (c < 'a' || c > 'f')) { + return false; + } + } + return true; + } + + private static String newLockNonce() { + // A per-acquisition owner stamp: the acquire time is a human-readable debugging aid, and the random + // UUID guarantees two acquisitions never share a stamp even within one pid and one millisecond, so + // releaseLock's ownership check is exact rather than probabilistic. + // + // NO pid@host, deliberately. The obvious way to get one on Java 8 is + // ManagementFactory.getRuntimeMXBean().getName(), and that RESOLVES THE LOCAL HOSTNAME: + // VMManagementImpl.getVmId() calls InetAddress.getLocalHost(), a full resolver round trip. Measured + // at 3162ms on a macOS dev box whose mDNS cache was cold - inside an acquire whose entire budget was + // 200ms, on the producer thread, holding this store's in-process lock and the caller's + // OidcDeviceAuth lock. That is the documented bound (inLock degrades to a lock-free refresh after + // lockAcquireBudgetMillis) broken by a debugging aid, and it is a once-per-JVM surprise: the value is + // cached inside the MXBean afterwards, so the very first credential refresh in a process paid for it + // and nothing later did. Java has no cheap hostname - unlike Python's socket.gethostname(), which is + // gethostname(2) and does not resolve - so the field goes rather than the bound. Nothing reads it: + // releaseLock and stealIfStale compare the stamp byte-wise against their own, and the cross-language + // contract has each implementation check only its own stamp (design/oidc-token-persistence.md). + // + // Not even Compat.currentPid(), which SlotLock uses for exactly this kind of diagnostic: its Java 9+ + // variant is a free ProcessHandle.current().pid(), but its Java 8 variant IS the getName() call above, + // parsed for the part before the '@'. The bound has to hold on every runtime this artifact supports, + // not only on modern ones, so the stamp carries no process identity at all. + return System.currentTimeMillis() + " " + UUID.randomUUID(); + } + + + private static boolean nullableEquals(String keyValue, StringSink fileValue) { + boolean fileHasValue = fileValue.length() > 0; + if (keyValue == null) { + return !fileHasValue; + } + return fileHasValue && Chars.equals(keyValue, fileValue); + } + + private static PersistedToken parseAndVerify(TokenStoreKey key, byte[] bytes) { + if (bytes.length == 0) { + return null; + } + TokenFileParser parser = new TokenFileParser(); + try (DirectUtf8Sink mem = new DirectUtf8Sink(bytes.length); + JsonLexer lexer = new JsonLexer(JSON_LEXER_CACHE_SIZE, JSON_LEXER_MAX_VALUE_BYTES)) { + // bulk-copy the file bytes into native memory in one go rather than byte by byte + mem.put(bytes, 0, bytes.length); + long lo = mem.ptr(); + lexer.parse(lo, lo + mem.size(), parser); + lexer.parseLast(); // reject a truncated document + } catch (JsonException e) { + // corrupt or truncated file: treat as no usable entry, fall back to refresh / interactive + return null; + } + // schema and fingerprint must match the live identity; a mismatch is a hash collision or a file + // copied from a different identity, so ignore it rather than serve the wrong identity's token. A + // malformed shape (an array anywhere - the schema is a single flat object) is likewise rejected. + if (parser.malformed || parser.version != SCHEMA_VERSION) { + return null; + } + if (!Chars.equals(key.getClientId(), parser.clientId) + || !Chars.equals(key.getTokenEndpoint(), parser.tokenEndpoint) + || !Chars.equals(key.getDeviceAuthorizationEndpoint(), parser.deviceAuthorizationEndpoint) + || !Chars.equals(key.getScope(), parser.scope) + || !nullableEquals(key.getAudience(), parser.audience) + || key.isGroupsInToken() != parser.groupsInToken) { + return null; + } + String accessToken = parser.accessToken.length() > 0 ? parser.accessToken.toString() : null; + String idToken = parser.idToken.length() > 0 ? parser.idToken.toString() : null; + String refreshToken = parser.refreshToken.length() > 0 ? parser.refreshToken.toString() : null; + return new PersistedToken(accessToken, idToken, refreshToken, parser.expiresAtMillis, parser.tokenTtlMillis); + } + + private static long parseLongOrZero(CharSequence value) { + // The frozen on-disk contract stores these as JSON numbers: an optional '-' followed by bare digits. + // Numbers.parseLong is more permissive than that - it accepts '_' thousands separators and an 'L'/'l' + // suffix - so "5L" and "1_000" would parse here and fail in every other language client reading the + // same file, which is exactly the kind of silent divergence a frozen cross-language format exists to + // prevent. Screen the value first so this client accepts only what the format actually allows; an + // out-of-contract value falls back to 0 like any other unusable field. + final int n = value.length(); + int i = n > 0 && value.charAt(0) == '-' ? 1 : 0; + if (i == n) { + return 0; // empty, or a bare "-" + } + for (; i < n; i++) { + char c = value.charAt(i); + if (c < '0' || c > '9') { + return 0; + } + } + try { + return Numbers.parseLong(value); + } catch (NumericException e) { + return 0; + } + } + + private static void putBooleanMember(StringSink sink, String name, boolean value) { + sink.put(','); + putName(sink, name); + sink.put(Boolean.toString(value)); + } + + private static void putLongMember(StringSink sink, String name, long value) { + sink.put(','); + putName(sink, name); + // write the digits unconditionally. sink.put(long) routes through Numbers.append(..., checkNaN=true), + // which renders Long.MIN_VALUE as the literal JSON null - a bare null for a present, non-nullable + // integer field would break the frozen cross-language contract (serialize() OMITS absent fields rather + // than writing null, so a null here is indistinguishable from absent) and round-trips back to 0 via + // parseLongOrZero. checkNaN=false emits the full number, so every long value round-trips verbatim. + Numbers.append(sink, value, false); + } + + private static void putName(StringSink sink, String name) { + sink.put('"').put(name).put('"').put(':'); + } + + private static void putNullableStringMember(StringSink sink, String name, String value) { + // omit the member entirely when the value is null, rather than write a JSON null - see serialize() + if (value != null) { + putStringMember(sink, name, value); + } + } + + private static void putString(StringSink sink, CharSequence value) { + // a refresh token is an opaque IdP string, so escape the JSON string properly; the JsonLexer + // decodes these on read. Non-ASCII passes through and is encoded as UTF-8 by getBytes below. + sink.put('"'); + for (int i = 0, n = value.length(); i < n; i++) { + char c = value.charAt(i); + switch (c) { + case '"': + sink.put("\\\""); + break; + case '\\': + sink.put("\\\\"); + break; + case '\b': + sink.put("\\b"); + break; + case '\f': + sink.put("\\f"); + break; + case '\n': + sink.put("\\n"); + break; + case '\r': + sink.put("\\r"); + break; + case '\t': + sink.put("\\t"); + break; + default: + if (c < 0x20) { + sink.put("\\u00").put(HEX[(c >> 4) & 0x0f]).put(HEX[c & 0x0f]); + } else { + sink.put(c); + } + } + } + sink.put('"'); + } + + private static void putStringMember(StringSink sink, String name, CharSequence value) { + sink.put(','); + putName(sink, name); + putString(sink, value); + } + + private static byte[] readBounded(Path file) throws IOException { + // read with a hard cap instead of Files.readAllBytes after a separate Files.size: the file is + // attacker-writable, so a size-check-then-read races a concurrent grow - a file enlarged past the cap + // between the two would make readAllBytes allocate gigabytes and throw OutOfMemoryError (an Error, which + // the best-effort RuntimeException guard in OidcDeviceAuth.maybeLoadFromStore would not catch, so a bad + // file would abort sign-in instead of degrading). Cap the buffer at the reported size (already bounded + // by MAX_FILE_BYTES) plus one byte, so a file that grew past its reported size is rejected, not allocated. + try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ)) { + long size = channel.size(); + if (size <= 0 || size > MAX_FILE_BYTES) { + // an empty or implausibly large file is not a usable entry; ignore it rather than read it in + return null; + } + ByteBuffer buffer = ByteBuffer.allocate((int) size + 1); + while (buffer.hasRemaining() && channel.read(buffer) >= 0) { + // read until EOF or the (size + 1)-byte buffer fills + } + int read = buffer.position(); + if (read == 0 || read > size) { + // empty, or grew past its reported size between the stat and the read: treat as corrupt/hostile + return null; + } + byte[] bytes = new byte[read]; + buffer.flip(); + buffer.get(bytes); + return bytes; + } + } + private static byte[] readLockHolder(Path lock) throws IOException { + // read the lock's owner stamp with a hard cap rather than Files.readAllBytes: the .lock file + // sits in the same attacker-writable directory as the token file, so an inflated lock would otherwise + // make readAllBytes allocate without bound and throw OutOfMemoryError - an Error the best-effort + // RuntimeException guards on the getToken()/signIn() refresh path would not catch, aborting the + // sign-in (the same reason readBounded caps the token file). A real owner stamp is a few hundred + // bytes; anything past the cap is corrupt or hostile, so report it as unreadable (null) rather than + // read it into memory. Returns the exact bytes present, or null for an empty/oversized lock. + try (FileChannel channel = FileChannel.open(lock, StandardOpenOption.READ)) { + long size = channel.size(); + if (size <= 0 || size > MAX_LOCK_FILE_BYTES) { + return null; + } + ByteBuffer buffer = ByteBuffer.allocate((int) size); + while (buffer.hasRemaining() && channel.read(buffer) >= 0) { + // read until EOF or the buffer fills + } + int read = buffer.position(); + if (read == 0) { + return null; + } + byte[] bytes = new byte[read]; + buffer.flip(); + buffer.get(bytes); + return bytes; + } + } + + private static boolean isLockOwner(Path lock, String nonce) throws IOException { + final byte[] content = readLockHolder(lock); + return content != null && nonce.equals(new String(content, StandardCharsets.UTF_8)); + } + + private static void releaseLock(Path lock, String nonce) { + // release our own lock only: re-read it (bounded - see readLockHolder) and delete it solely when it + // still carries our nonce. A hold that outran lockStaleMillis may have been judged stale and stolen + // (captured and recreated) by a peer; deleting by bare path would then remove the peer's live lock and + // admit a third acquirer alongside it, defeating the mutual exclusion this lock exists to provide. A + // microscopic window remains if a steal lands between the read and the delete, but that is bounded to + // one syscall gap rather than the whole hold, so a misconfigured staleness window degrades to at most + // the documented double-refresh rather than corrupting a peer's lock state. + try { + if (isLockOwner(lock, nonce)) { + Files.deleteIfExists(lock); + } + // otherwise a peer now owns this lock file, or it is unreadable/oversized; leave it for that owner + // (or the staleness steal) to reclaim + } catch (NoSuchFileException e) { + // already gone (stolen and not yet recreated, or removed elsewhere); nothing to release + } catch (IOException e) { + // Best-effort release, but no longer silent. A lock we could not delete blocks every peer's + // coordinated refresh until it goes stale (lockStaleMillis - 10 minutes by default), and each + // peer degrades to an unserialized refresh meanwhile: the rotating-refresh-token race this lock + // exists to prevent. That is worth a line an operator can find, rather than surfacing later as + // unexplained repeated sign-ins. + // sanitized: see the sibling warning in inLock - the message embeds the store path + LOG.warn("could not release the OIDC token store lock; peer operations may be delayed until it " + + "goes stale [error={}]", OidcDeviceAuth.sanitizeForDisplay(e.getMessage())); + } + } + // Drops this caller's claim on the operation's lock, retiring the entry when it was the last one, so the + // map never outgrows the identities actually in flight. Pairs with retainProcessLock in a finally. + private static void releaseProcessLock(String identity) { + PROCESS_LOCKS.computeIfPresent(identity, (k, held) -> --held.users == 0 ? null : held); + } + + + + + + private static void replaceTarget(Path tmp, Path target) throws IOException { + // atomically rename tmp over target. On Windows a concurrent reader in any process holding target open + // can make the rename fail transiently with AccessDeniedException (a sharing violation); retry a few + // times on a short backoff before giving up, so a routine read/write overlap does not needlessly degrade + // persistence (best-effort - the in-memory token is still valid). POSIX rename over an open file never + // hits this. AtomicMoveNotSupported (a rare filesystem) falls back to a plain replace, which still beats + // leaving a partial write. + AccessDeniedException lastDenied = null; + for (int attempt = 0; attempt < REPLACE_MAX_ATTEMPTS; attempt++) { + if (attempt > 0) { + try { + // Thread.sleep, not Os.sleep, for the reason acquireLock's poll gives: Os.sleep catches + // InterruptedException and keeps sleeping to its deadline WITHOUT re-asserting the flag, so + // a cancellation aimed at this backoff was destroyed outright. save() only parks and restores + // the flag it saw on ENTRY, so an interrupt arriving mid-save -- which is exactly what + // PoolHousekeeper.stop()'s escalation delivers to break a credential pull -- vanished here and + // the stop signal was lost. Re-assert it and abandon the retry: persistence is best-effort, + // and lastDenied is non-null on every pass that reaches this (attempt > 0 only follows an + // AccessDeniedException), so the throw below reports the denial as it would have anyway. + Thread.sleep(REPLACE_RETRY_SLEEP_MILLIS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + break; + } + } + try { + Files.move(tmp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + return; + } catch (AtomicMoveNotSupportedException e) { + Files.move(tmp, target, StandardCopyOption.REPLACE_EXISTING); + return; + } catch (AccessDeniedException e) { + lastDenied = e; + } + } + throw lastDenied; + } + + // Claims the lock for this operation identity, creating the entry if this caller is the first to arrive. + // Registers the claim BEFORE the acquire, so an entry cannot be retired out from under a thread that is + // queued on it - which is what makes the retirement in releaseProcessLock safe. + private static ProcessLock retainProcessLock(String identity) { + return PROCESS_LOCKS.compute(identity, (k, existing) -> { + final ProcessLock held = existing != null ? existing : new ProcessLock(); + held.users++; + return held; + }); + } + private static byte[] serialize(TokenStoreKey key, PersistedToken token) { + // a null value (an absent audience, or a token kind the grant did not return) is omitted rather than + // written as JSON null: the JsonLexer reports a bare null and a quoted "null" identically, so omitting + // absent fields is the only encoding under which a present value - a token equal to "null" included - + // round-trips back verbatim. "v" is the first member; every later member prepends its own comma. + StringSink sink = new StringSink(); + sink.put('{'); + putName(sink, "v"); + sink.put(SCHEMA_VERSION); + putStringMember(sink, "client_id", key.getClientId()); + putStringMember(sink, "token_endpoint", key.getTokenEndpoint()); + putStringMember(sink, "device_authorization_endpoint", key.getDeviceAuthorizationEndpoint()); + putStringMember(sink, "scope", key.getScope()); + putNullableStringMember(sink, "audience", key.getAudience()); + putBooleanMember(sink, "groups_in_token", key.isGroupsInToken()); + putNullableStringMember(sink, "access_token", token.getAccessToken()); + putNullableStringMember(sink, "id_token", token.getIdToken()); + putNullableStringMember(sink, "refresh_token", token.getRefreshToken()); + putLongMember(sink, "expires_at_millis", token.getExpiresAtMillis()); + putLongMember(sink, "token_ttl_millis", token.getTokenTtlMillis()); + sink.put('}'); + return sink.toString().getBytes(StandardCharsets.UTF_8); + } + + private static void warnNoPosixPermsOnce() { + // best-effort, once per JVM: the token store could not enforce 0600/0700, so the persisted refresh + // token is protected only by the directory's inherited ACL. ASCII-only, and never includes a path or + // token byte (a path could itself carry terminal-spoofing characters) + if (!warnedNoPosixPerms.compareAndSet(false, true)) { + return; + } + LOG.warn("the OIDC token store could not enforce owner-only (0600/0700) permissions on this " + + "filesystem; the persisted refresh token is protected only by the directory's default ACL. " + + "Back the store with an OS keychain for at-rest encryption."); + } + + private static void warnStuckUntrustedSentinelOnce(String reason) { + // The untrusted sentinel is meant to be transient: a caller marks the directory, sweeps it, and + // clears the mark. When either the sweep or the clear keeps failing, the mark stands and + // restrictToOwner distrusts the directory on every later call - so load() returns null over an entry + // save() has just written, and token persistence is silently and permanently off for this directory. + // Nothing self-heals it, because the sweep skips the sentinel by design and markUntrusted only runs + // while the directory is still other-writable. + // + // Once per JVM, ASCII-only and without the path, for the same reasons as warnUnprotectedStoreDirOnce + // beside it: the condition is a property of the directory every identity in this process shares, + // load() sits on the flush path, and an operator-supplied path can carry terminal-spoofing bytes. + if (!warnedStuckUntrustedSentinel.compareAndSet(false, true)) { + return; + } + LOG.warn("the OIDC token store directory is marked untrusted and the mark cannot be lifted, so no " + + "token will be persisted or read there until it is: {}. Remove the '.untrusted' entry and " + + "anything left beside it in questdb.client.oidc.token.store.dir, or point that setting at " + + "a fresh directory only this user can write.", reason); + } + + private static void warnStoreDirIoFailureOnce(IOException cause) { + // This is an operational I/O fault, not evidence that the directory permissions are unsafe. Keep its + // once-per-JVM budget separate from warnUnprotectedStoreDirOnce: a transient mount or filesystem failure + // must not suppress the later warning that explains why exposed token contents were discarded. + if (!warnedStoreDirIoFailure.compareAndSet(false, true)) { + return; + } + LOG.warn("could not prepare the OIDC token store directory; persisted credentials are temporarily " + + "unavailable and the load will be retried [error={}]", + OidcDeviceAuth.sanitizeForDisplay(cause.getMessage())); + } + + private static void warnUnprotectedStoreDirOnce(String reason) { + // once per JVM, like warnNoPosixPermsOnce: the condition is a property of the directory, which every + // identity in this process shares, and load() sits on the flush path via OidcDeviceAuth.getToken(). + // ASCII-only and never the path itself - an operator-supplied path can carry terminal-spoofing + // characters, which is why warnNoPosixPermsOnce omits it too. + if (!warnedUnprotectedStoreDir.compareAndSet(false, true)) { + return; + } + LOG.warn("the OIDC token store directory is not owner-only, so a persisted token there cannot be " + + "trusted: {}. Point questdb.client.oidc.token.store.dir at a directory only this user " + + "can write, or supply a TokenStore backed by an OS keychain.", reason); + } + + /** + * Runs one short token-store filesystem operation under the directory-wide in-process and cross-process + * locks. This lock covers the trust check, a required directory-wide distrust sweep, and the operation + * itself as one unit. In particular, no caller may retain a pre-lock "untrusted" verdict and sweep after a + * peer has already completed recovery and saved a fresh token. + *

+ * The lock file has to be created before {@link #restrictToOwner()} changes a pre-existing directory: + * otherwise two processes can both consume the old permission verdict before either has a common lock. + * Because another local user can still remove or replace that lock while the directory is writable, the + * owner nonce is re-read after the chmod. A lost lock aborts the operation; when the directory was + * untrusted, the sentinel is reasserted after the chmod so the eventual owner must sweep before trusting. + */ + private T withDirectoryLock(DirectoryAction action) throws IOException { + createDirectory(); + + // Directory identities are normalized paths without NUL; per-identity keys append NUL + a 64-hex + // fingerprint, so the two domains cannot collide in PROCESS_LOCKS. + final String lockIdentity = lockNamespace; + final ProcessLock processLock = retainProcessLock(lockIdentity); + try { + processLock.lock.lockInterruptibly(); + } catch (InterruptedException e) { + releaseProcessLock(lockIdentity); + Thread.currentThread().interrupt(); + throw new IOException("interrupted while waiting for the OIDC token store directory lock", e); + } + try { + final Path lock = directoryLockFile(); + DirectoryLockHeartbeat heartbeat = null; + String nonce = null; + try { + try { + nonce = acquireLock(lock, true); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("interrupted while waiting for the OIDC token store directory lock", e); + } + + final boolean isDirectoryTrusted = restrictToOwner(); + final boolean isOwner; + try { + isOwner = isLockOwner(lock, nonce); + } catch (IOException e) { + // The directory may have been writable when the lock was created, so another local user + // could remove it before the chmod. An absent or unreadable stamp is loss of ownership, + // never successful acquisition. + if (!isDirectoryTrusted) { + markUntrusted(); + } + throw new IOException("lost ownership of the OIDC token store directory lock", e); + } + if (!isOwner) { + if (!isDirectoryTrusted) { + // restrictToOwner has already tightened the directory, so this mark cannot now be + // removed by the other local user who could write it before the chmod. + markUntrusted(); + } + throw new IOException("lost ownership of the OIDC token store directory lock"); + } + + // The required lock can cover a flush or a distrust sweep. Renew its timestamp while either + // runs. A trusted-directory waiter uses the short lease; one that still sees pending recovery + // retains the conservative configured window so a paused sweep cannot be displaced. + heartbeat = new DirectoryLockHeartbeat(lock, nonce); + + if (!isDirectoryTrusted) { + discardUntrustedDirectoryContents(); + } + return action.run(isDirectoryTrusted); + } finally { + if (heartbeat != null) { + heartbeat.close(); + } + if (nonce != null) { + // A live interrupt during the action must not strand the lock until its staleness window. + final boolean wasInterrupted = Thread.interrupted(); + try { + releaseLock(lock, nonce); + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + } + } finally { + processLock.lock.unlock(); + releaseProcessLock(lockIdentity); + } + } + + private static void writeAndFlush(Path file, byte[] content) throws IOException { + // write the payload and force it to disk before the rename, so a crash between the write and the + // atomic rename cannot leave the target pointing at unflushed (zero/partial) bytes - the temp file + // is the durability point of the write-temp / flush / atomic-rename protocol + try (FileChannel channel = FileChannel.open(file, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) { + ByteBuffer buffer = ByteBuffer.wrap(content); + while (buffer.hasRemaining()) { + channel.write(buffer); + } + channel.force(true); + } + } + + private static void writeNewFile(Path file, byte[] content, FileAttribute... attrs) throws IOException { + // exclusive-create (CREATE_NEW = O_CREAT|O_EXCL) with the given perms and write the content in one + // open; FileAlreadyExistsException is raised when the file already exists + try (FileChannel channel = FileChannel.open(file, EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE), attrs)) { + ByteBuffer buffer = ByteBuffer.wrap(content); + while (buffer.hasRemaining()) { + channel.write(buffer); + } + } + } + + private String acquireLock(Path lock, boolean isRequired) throws IOException, InterruptedException { + // returns the unique owner nonce stamped into the lock on success, or null if it could not be acquired + // within the budget when isRequired is false. A required directory lock throws instead: proceeding + // without it would let an untrusted-directory sweep delete another identity's completed save. + // releaseLock uses the nonce to verify ownership before deleting, so a hold that outran its staleness + // window (and was stolen by a peer) never deletes the peer's lock on release + final String nonce = newLockNonce(); + // nanoTime, not currentTimeMillis: this is an elapsed budget, and the wall clock is adjustable. An + // NTP step or an operator setting the date back stretches a millis-based deadline by the size of the + // jump, so a caller that documents a bounded degrade - inLock() promises to fall back to a lock-free + // refresh after lockAcquireBudgetMillis - would sit here for however long the clock moved instead. + // nanoTime is monotonic and immune to that. Compare by DIFFERENCE rather than by ordering, so the + // arithmetic stays correct across nanoTime's wraparound. + final long deadlineNanos = System.nanoTime() + lockAcquireBudgetMillis * 1_000_000L; + while (true) { + try { + // exclusive-create then stamp on the same open channel: the empty-file window between the two + // is tiny and covered by the applicable empty-lock grace, so a GC/safepoint pause + // mid-acquisition cannot get our freshly-created lock stolen as empty-and-stale + createLockFile(lock, nonce); + return nonce; + } catch (FileAlreadyExistsException e) { + // the lock exists; if a crashed holder abandoned it, steal it - atomically and stamp-verified, + // so a stealer never removes a peer's freshly-created live lock (see stealIfStale). Then fall + // through to the bounded wait below rather than retry immediately: a steal contest between + // several acquirers (or a misconfigured tiny lockStaleMillis) must not hot-spin. + if (isRequired && canUseShortDirectoryLockLease()) { + stealIfStale( + lock, + DIRECTORY_LOCK_STALE_MILLIS, + DIRECTORY_LOCK_EMPTY_STEAL_GRACE_MILLIS + ); + } else { + stealIfStale(lock); + } + if (System.nanoTime() - deadlineNanos >= 0) { + if (isRequired) { + throw new IOException("timed out acquiring the OIDC token store directory lock"); + } + return null; // give up and run without the refresh lock rather than stall a sign-in + } + // Thread.sleep, not Os.sleep: Os.sleep catches InterruptedException and keeps sleeping to its + // deadline WITHOUT re-asserting the flag, so a cancellation aimed at this poll was swallowed + // outright and the whole budget elapsed regardless. Propagate it and let inLock abandon the + // refresh - the budget can be tens of seconds, far past a QWP close()'s shutdown window. + Thread.sleep(LOCK_POLL_SLICE_MILLIS); + } catch (IOException e) { + // Windows reports CREATE_NEW against a DIRECTORY at this name as AccessDeniedException rather + // than FileAlreadyExistsException, so the squatter recovery above is otherwise unreachable on + // that platform. Recover only a shape this class cannot have written. The helper captures the + // name atomically and restores it if a peer replaced the squatter with a regular lock between + // this shape check and the capture; a successful displacement leaves the canonical name free, + // so retry the exclusive create. + if (!Files.isRegularFile(lock, LinkOption.NOFOLLOW_LINKS) && displaceLockSquatter(lock)) { + continue; + } + + // Do NOT delete any remaining lock here. That used to be justified by "the exclusive create + // succeeded and only the nonce write failed, so the file is ours" - true for one of the + // failures this arm catches, but not the others. From the second loop iteration onward a + // PEER's live lock occupies the path, and plenty of "cannot create" failures are not + // FileAlreadyExistsException: fd exhaustion (EMFILE/ENFILE), EACCES, EROFS, ENOSPC and a + // Windows sharing violation all arrive as a plain IOException. deleteIfExists cannot tell + // the two cases apart, and removing a peer's live lock admits a second holder - the + // double-POST of one rotating refresh token this lock exists to prevent, which a + // reuse-detecting identity provider answers by revoking the whole token family. + // + // A lock we genuinely did leave half-created is EMPTY, and stealIfStale already reclaims an + // empty lock on its dedicated grace, so leaving it behind costs at most that grace once it is + // safe to reclaim. A refresh lock degrades to a lock-free refresh; a required directory lock + // propagates the failure because running a sweep or write without it is unsafe. + if (isRequired) { + throw e; + } + // sanitized: see the sibling warning in inLock - the message embeds the store path + LOG.warn("could not acquire the OIDC token store lock; running this refresh without " + + "cross-process coordination [error={}]", + OidcDeviceAuth.sanitizeForDisplay(e.getMessage())); + return null; + } + } + } + + private void clearUntrusted() { + // Remove the untrusted sentinel once discardUntrustedDirectoryContents has swept every untrusted + // entry, so a later caller may trust the owner-only directory again. Called ONLY after a complete + // sweep: while any untrusted entry might remain, the sentinel must stay so the next caller re-sweeps + // rather than trusting a directory that still holds a plant. Best-effort - a failure here just leaves + // the directory marked untrusted, so the next caller sweeps again; it never fails a sign-in. + // + // "The next caller re-sweeps" assumes the failure is TRANSIENT. When it is not, this method is the + // only thing that can ever lift the distrust - the sweep skips the sentinel by design (no hash + // prefix), and markUntrusted only runs while the directory is still other-writable - so a sentinel + // that cannot be deleted latches the directory untrusted for good: every later load returns null + // over an entry save() just wrote, and persistence is silently dead until someone removes the file + // by hand. A non-empty directory squatting the name reaches that state with one mkdir, and it + // survives the chmod that ends the attacker's write access. + // + // So: report it. The distrust itself stays - it is the fail-closed direction and it is cheap to be + // wrong about - but an operator gets a line naming the condition and the fix instead of persistence + // that merely stops working. + try { + Files.deleteIfExists(untrustedSentinel()); + } catch (IOException e) { + // The exception KIND, not its message: a message can carry the path, and an operator-supplied + // path can carry terminal-spoofing bytes, which is why every warning in this class omits it. + // The kind is the part that tells them what to look for - DirectoryNotEmptyException means + // something is squatting the name, AccessDeniedException means the permissions are. + warnStuckUntrustedSentinelOnce("the '.untrusted' entry itself could not be removed (" + + e.getClass().getSimpleName() + ")"); + } + } + + private Path createTempFile(String prefix) throws IOException { + try { + return Files.createTempFile(directory, prefix, ".tmp", FILE_ATTRS); + } catch (UnsupportedOperationException e) { + // non-POSIX filesystem (e.g. Windows): rely on the owner-only directory ACL instead + warnNoPosixPermsOnce(); + return Files.createTempFile(directory, prefix, ".tmp"); + } + } + + /** + * Discards EVERY entry in the store directory, warns once, and lifts the untrusted sentinel on a clean + * sweep - run after {@link #restrictToOwner()} has reported the directory was writable by other local + * users, or has found the sentinel still standing from an earlier such report. + *

+ * Discarding only the caller's own {@code .json} is not enough, because the verdict is destroyed by + * the act of reporting it: restrictToOwner chmods the directory to 0700 as it returns, so whichever + * caller touches the store first consumes the one permission-based observation. Every later load - in + * this process or the next - would then see an owner-only directory and adopt whatever entry is sitting + * there, including one planted while the directory stood open. Two defences combine: this method discards + * ALL entries, not just the key being loaded (one store directory holds one file per configuration and is + * documented as the store's alone, so once other local users could write it nothing in it can be told + * apart from a plant); and restrictToOwner drops the {@code .untrusted} sentinel before the chmod so the + * verdict survives it, which this method clears only after a COMPLETE sweep. A concurrent caller reading + * the tightened permissions before that clear still finds the sentinel and distrusts - the gap the + * directory-wide sweep alone leaves open. Each identity then re-signs in. + *

+ * Best-effort by design. This runs on the flush path through {@code OidcDeviceAuth.getToken()}, so a + * delete that fails must degrade to a refresh or an interactive sign-in rather than throw - and the + * caller that triggered it fails closed either way, whether or not the file goes. A failed or partial + * sweep leaves the sentinel in place, so the next caller re-sweeps before anything trusts the directory. + */ + private void discardUntrustedDirectoryContents() { + warnUnprotectedStoreDirOnce("it was writable by other local users; every entry found in it was " + + "discarded rather than trusted, and a fresh sign-in is required"); + final Runnable hook = beforeUntrustedDiscardHook; + if (hook != null) { + // test seam: fires with the directory already tightened to 0700 and marked untrusted, but + // before the sweep below - the exact window a concurrent caller must still distrust through + hook.run(); + } + // Track a COMPLETE sweep. The untrusted sentinel is lifted only when every untrusted entry is gone; + // if the directory could not be listed or any delete failed, an entry may remain, so the sentinel + // stays and the next caller re-sweeps before anything trusts the now owner-only directory. + boolean sweptClean = true; + try (DirectoryStream stream = Files.newDirectoryStream(directory)) { + for (Path entry : stream) { + final String name = entry.getFileName().toString(); + // Only files THIS STORE could have written. Every one of them is named after a 64-hex + // identity fingerprint - tokenFile() builds ".json" and writeTemp() asks + // createTempFile() for ".tmp" - so a name without that prefix belongs to + // whatever else shares the directory (the untrusted sentinel included, whose name has no such + // prefix, so it is never swept as an entry - clearUntrusted removes it below). Without this + // test the filter below reads as "any .json file", and the operator who pointed + // questdb.client.oidc.token.store.dir at a directory holding their own config loses it on the + // first load(). The directory being group-writable is what brought us here; it is not a + // licence to delete files we did not write. sweepTempFiles() already scopes itself this way, + // by hash prefix. + if (!hasStoreHashPrefix(name)) { + continue; + } + // The entry is exactly ".json"; anything longer that merely starts with a hash is + // not ours either. A write temp carries a random infix, so only its suffix is fixed. + final boolean isEntry = name.length() == HASH_NAME_LENGTH + 5 && name.endsWith(".json"); + // A steal-captured lock (.lock..tmp) is a cross-process handover in flight, not + // an orphaned write temp - sweepTempFiles skips it for the same reason. The .lock files + // themselves stay too: they carry no token, and acquireLock already treats a hostile or + // stale one as stealable. + final boolean isWriteTemp = name.endsWith(".tmp") && !name.contains(".lock."); + if (!isEntry && !isWriteTemp) { + continue; + } + try { + Files.deleteIfExists(entry); + } catch (IOException ignore) { + // best-effort; skip this entry, the caller still fails closed. Leave the directory marked + // untrusted (below) so a later call retries the delete before anything adopts what is left. + sweptClean = false; + } + } + } catch (IOException ignore) { + // best-effort; an unreadable directory must not turn a fail-closed load into a thrown one. The + // sweep is incomplete, so the sentinel stays and the next caller re-sweeps. + sweptClean = false; + } + if (sweptClean) { + // Every untrusted entry is gone, so lift the distrust: a later caller may trust the owner-only + // directory again. Until this point the sentinel is the only thing keeping a concurrent caller + // from adopting an entry over a directory restrictToOwner's chmod already made look owner-only. + clearUntrusted(); + } else { + // The sweep is what lifts the distrust, so a sweep that keeps failing latches it: the sentinel + // stays, every later load returns null over an entry save() just wrote, and persistence is dead + // for this directory until someone intervenes. One undeletable entry is enough - an entry owned + // by another UID under a sticky-bit parent, or a persistent EPERM/EIO/ESTALE. Retrying is right; + // doing it forever in silence is not. + warnStuckUntrustedSentinelOnce("the directory could not be swept completely"); + } + } + + /** + * Creates the store directory owner-only when absent. A pre-existing directory is deliberately left as-is + * until the caller owns {@code .store.lock}; {@link #restrictToOwner()} both changes its permissions and + * emits the trust verdict whose directory-wide sweep must be serialized with every token write. + * + * @throws IOException if the directory cannot be created + */ + private void createDirectory() throws IOException { + if (!Files.isDirectory(directory)) { + try { + Files.createDirectories(directory, DIR_ATTRS); + } catch (UnsupportedOperationException e) { + warnNoPosixPermsOnce(); + Files.createDirectories(directory); + } + } + } + + private Path directoryLockFile() { + return directory.resolve(DIRECTORY_LOCK_FILE_NAME); + } + + /** + * Removes a shape occupying a lock name that this class cannot have written - a directory, or a symlink + * (dangling or not). Only a regular file is a lock: {@link #createLockFile} produces nothing else, and + * {@link #releaseLock} and the steal path both reclaim nothing else, so a squatter left standing wedges + * every later acquire on that name for good. + *

+ * Capture-then-decide rather than a bare delete, for the reason {@link #stealIfStale} captures: a peer + * that creates a genuine lock in the gap between the shape test and the removal would otherwise have it + * deleted out from under it, admitting two holders. The rename is atomic, so among racing callers exactly + * one captures the name; whatever it captured is then inspected, and a regular file - a peer's real lock, + * created in that gap - is put back untouched rather than stolen. + *

+ * Best-effort throughout: a squatter that resists removal (a non-empty directory) leaves a capture temp + * behind for {@code sweepStaleTempFiles}, but the lock NAME is free either way, which is what unwedges the + * store. Silent, like every other steal on this path - the condition is self-healing once the name is + * reclaimed, and an operator has nothing to act on. + * + * @return {@code true} when this call captured a non-regular squatter and freed the canonical lock name; + * {@code false} when capture failed or the captured name was a peer's regular lock and was restored + */ + private boolean displaceLockSquatter(Path lock) { + final Path captured = lock.resolveSibling(lock.getFileName().toString() + '.' + UUID.randomUUID() + ".tmp"); + try { + Files.move(lock, captured, StandardCopyOption.ATOMIC_MOVE); + } catch (IOException e) { + // already gone, a peer captured it first, or the filesystem cannot rename atomically; the next + // acquire poll re-tests the shape rather than risk a non-atomic removal + return false; + } + if (Files.isRegularFile(captured, LinkOption.NOFOLLOW_LINKS)) { + // a peer replaced the squatter with a real lock between the shape test and the rename; restore it + restoreCapturedLock(lock, captured); + return false; + } + deleteCapturedLock(captured); + return true; + } + + private Path lockFile(TokenStoreKey key) { + return directory.resolve(key.hash() + ".lock"); + } + + /** + * Whether the untrusted sentinel's name currently carries a mark this class could have written and can + * later clear - a regular file, following no symlink. + *

+ * Only a regular file qualifies. {@link #clearUntrusted()} removes the mark with a plain delete after a + * complete sweep, so any shape that delete cannot remove would stand at the name forever and latch the + * directory untrusted; and a symlink at the name is not a mark at all, since a dangling one reports + * absent to a link-following test. Both are squatters rather than a peer's mark, which is what + * {@link #markUntrusted()} uses this to decide. + * + * @return {@code true} when the name holds a regular file, {@code false} for a symlink, a directory, + * an absent name, or a stat this process cannot make + */ + private boolean isUsableSentinel() { + return Files.isRegularFile(untrustedSentinel(), LinkOption.NOFOLLOW_LINKS); + } + + private void markUntrusted() { + // Drop the untrusted sentinel. Reached only from restrictToOwner's POSIX path (getPosixFilePermissions + // has already succeeded), so FILE_ATTRS is supported here. Already-present - a concurrent caller, or a + // previous run - is success. Any other failure is swallowed rather than propagated: the caller that + // triggered it still returns untrusted and sweeps (restrictToOwner sees wasOtherWritable), so a failed + // mark only degrades a CONCURRENT caller back to the pre-sentinel race, and it must never flip the + // verdict (a RuntimeException escaping to restrictToOwner's UnsupportedOperationException catch would + // return "trusted") nor fail a sign-in. + final Path sentinel = untrustedSentinel(); + try { + writeNewFile(sentinel, new byte[0], FILE_ATTRS); + } catch (FileAlreadyExistsException e) { + // A peer's mark, or a squatter. CREATE_NEW is O_CREAT|O_EXCL, which reports EEXIST for a symlink + // (dangling or not) and for a directory just as it does for a peer's regular file, so this + // exception alone does not say the name carries a usable mark. Only a regular file is one: + // clearUntrusted deletes exactly that after a complete sweep, so a shape it cannot remove would + // otherwise stand at this name forever, and every later mark would land here and report success. + // Displace the squatter and mark again. Nothing but this class writes this name, so there is no + // peer state to lose; a plain symlink is unlinked by delete, and a directory only yields when + // empty, which is the fail-closed direction (restrictToOwner keeps distrusting it either way). + if (!isUsableSentinel()) { + try { + Files.deleteIfExists(sentinel); + writeNewFile(sentinel, new byte[0], FILE_ATTRS); + } catch (IOException | RuntimeException ignore) { + // best-effort; the squatter stands, and restrictToOwner's NOFOLLOW test still distrusts + // through it, so the directory fails closed rather than open + } + } + } catch (IOException | RuntimeException ignore) { + // best-effort; see the method contract above + } + } + + /** + * The key this store's entry for {@code key} takes in {@link #PROCESS_LOCKS}: the normalized store + * directory and the identity fingerprint, NUL-separated. + *

+ * Both halves are load-bearing. The fingerprint alone over-serializes, because it names a + * configuration and says nothing about where the entry lives - so two stores following the + * documented per-user-directory recipe would queue on one lock while touching different files, and that + * lock is held across a whole token-endpoint round trip with no acquire budget. The directory alone + * under-serializes, letting two identities in one directory run their read-refresh-write concurrently. + *

+ * NUL is the separator for the reason {@code TokenStoreKey} uses it: a path can contain almost anything + * else, and two different (directory, identity) pairs must never render to one string. + */ + private String processLockIdentity(TokenStoreKey key) { + return lockNamespace + '\0' + key.hash(); + } + + /** + * Puts a captured lock back at {@code lock} after the capture-verify decided it is NOT the abandoned lock + * we judged stale - a peer recreated it in the gap, or we could not re-read what we captured. + *

+ * Restores by hard-LINKING the capture back to the lock path, not by renaming it. {@code Files.move} + * without {@code REPLACE_EXISTING} looks atomic but is not: it stats the target, then renames, and + * {@code rename(2)} silently replaces. A third party that claims the freed path between those two steps + * therefore had its live lock destroyed by the very call whose comment promised to leave it intact. + * {@code link(2)} has no such gap - it fails outright when the target exists - and it preserves the peer's + * exact bytes, which matters because {@link #releaseLock} verifies the stamp before deleting. + *

+ * A residual remains and is not closeable with a lock file: if a third party did claim the path, we drop + * our copy, so the recreating peer's lock file is gone while that peer still believes it holds the lock, + * and for that one refresh two holders can run concurrently. A filesystem offers no atomic "delete or + * rename only if the content is still X", so the capture-verify narrows the window to this multi-actor + * race - our steal, a peer recreating, AND a third party claiming the freed path, all overlapping - + * without eliminating it. Best-effort by design: it degrades to one extra refresh, a re-prompt on a + * rotating-refresh-token identity provider, never a torn or forged credential (Layer 1's atomic rename + * still holds). + *

+ * Split out of {@link #stealIfStale} so it can be driven directly. Reaching it through {@code stealIfStale} + * needs a peer to replace the lock file between the staleness read and the capture rename, which no test + * can force without a production seam - so the whole restore path, the part that keeps a stealer from + * destroying a peer's live lock, otherwise ran only in production. + * + * @param lock the lock path to restore to + * @param captured the private capture name the steal renamed the lock to + */ + private void restoreCapturedLock(Path lock, Path captured) { + try { + Files.createLink(lock, captured); + deleteCapturedLock(captured); + } catch (FileAlreadyExistsException e) { + // a third party owns the path now; leave their lock untouched and drop our copy + deleteCapturedLock(captured); + } catch (IOException | UnsupportedOperationException e) { + // the filesystem does not support hard links, or the link failed for another reason. Fall back to + // the plain move: it preserves the bytes but reopens the stat-then-rename window described above, + // which is still better than abandoning the peer's lock outright. + try { + Files.move(captured, lock); + } catch (IOException moveFailure) { + deleteCapturedLock(captured); + } + } + } + + /** + * Re-asserts owner-only permissions on the store directory. The at-rest protection of the plaintext token + * files is exactly these permissions, so a pre-existing directory another tool or a permissive umask left + * loose is tightened rather than trusted as it stands. withDirectoryLock runs this on every load, save and + * inLock preparation, so it chmods only on detected drift - the common case costs one stat and no write + * syscall. + *

+ * NOT best-effort on the failure that matters. An {@code IOException} here means the directory is not ours + * to chmod, which is precisely the state in which the documented {@code 0700} protection does not hold and + * another local user can create, replace or delete entries in it. Swallowing it left every caller believing + * the protection applied. Callers degrade on the throw, each in the way that suits it: {@code save} refuses + * to write a plaintext refresh token into a directory it cannot protect, {@code inLock} runs lock-free. + * On a non-POSIX filesystem (Windows) the check is unavailable rather than failed, so it falls back to the + * inherited ACL as before (owner-only hardening there, via AclFileAttributeView, is a separate follow-up). + * + * @return {@code true} when the directory's content may be trusted, {@code false} when it was writable + * by group or other (so another local user could have planted an entry before this call tightened + * it), or when an un-swept {@code .untrusted} sentinel shows an earlier such tightening has not yet + * finished discarding + * @throws IOException if the directory exists but its permissions cannot be read or set + */ + private boolean restrictToOwner() throws IOException { + try { + final Set perms = Files.getPosixFilePermissions(directory); + // Writable by group or other is the state that decides TRUST, and it is narrower than "not + // owner-only": only write permission on a directory lets another local user create or replace an + // entry in it, which is what load() would then adopt. The 0755 a default umask produces exposes + // no token - the files themselves are 0600 - and everything in it was still put there by us, so + // it is tightened for defence in depth but its content stays trusted. + final boolean wasOtherWritable = perms.contains(PosixFilePermission.GROUP_WRITE) + || perms.contains(PosixFilePermission.OTHERS_WRITE); + if (wasOtherWritable) { + // Drop the untrusted sentinel BEFORE the chmod below. The chmod publishes owner-only to every + // other process the instant it lands, but the entries planted while the directory stood open + // are not swept until discardUntrustedDirectoryContents runs after we return - so a concurrent + // caller reading the permissions in that gap would otherwise compute wasOtherWritable == false + // and adopt a plant. The sentinel carries this "untrusted" verdict across the chmod that would + // otherwise erase it. Best-effort: if the mark fails we still return untrusted for THIS caller + // (wasOtherWritable is true below), so only a concurrent caller degrades to the pre-sentinel + // race. Residual: an attacker who both planted an entry and actively deletes the sentinel in + // the sub-syscall window between this mark and the chmod can still race a concurrent trust - a + // far narrower window than the chmod-to-sweep gap this closes, and Layer 1's atomic replace + // still bars a torn or forged credential. + markUntrusted(); + } + // Any group/other bit is looser than owner-only and must be removed. Test containment rather than + // exact equality: 0500, 0600 and other owner-only subsets are already at least as strict as 0700; + // assigning DIR_PERMS to them would silently WIDEN access by adding missing owner permissions. + if (!DIR_PERMS.containsAll(perms)) { + // Tightening is load-bearing - it IS the at-rest protection of the plaintext token files, so + // it stays unconditional - but it changes a directory the operator chose and may share with + // something else, so it must not be silent. Announce only AFTER chmod succeeds: consuming the + // once-per-JVM latch first would claim a change that did not happen and suppress the warning + // for the next directory that is genuinely tightened. Never name the path. + final Runnable hook = beforeDirectoryTightenHook; + if (hook != null) { + hook.run(); + } + Files.setPosixFilePermissions(directory, DIR_PERMS); + if (warnedTightenedStoreDir.compareAndSet(false, true)) { + LOG.warn("the OIDC token store directory was not owner-only and has been tightened to " + + "0700; it holds plaintext refresh tokens, so it must not be shared with " + + "anything else. Point questdb.client.oidc.token.store.dir at a directory of " + + "its own if another tool needs access to that path."); + } + } + // Trusted only when the directory was never other-writable AND no un-swept untrusted sentinel + // remains - the one just dropped above, one a concurrent caller is mid-sweep on, or one a previous + // run tightened but left behind after an incomplete sweep. This test is what carries the verdict + // across the chmod above. + // + // NOFOLLOW_LINKS, and notExists rather than !exists, because both defaults fail OPEN and the + // attacker this sentinel defends against is the one who can write this directory. A bare + // Files.exists follows symlinks, so a dangling link planted at the sentinel's name reports + // "absent" and the directory reads as trusted - the sentinel silently disabled, with no race to + // win, for as long as the link stands (markUntrusted's exclusive create cannot replace it + // either; see the squatter branch there). And !exists is true both for "absent" and for "cannot + // tell", so an unreadable stat also trusted. notExists(NOFOLLOW_LINKS) is positive evidence of + // absence: a link, a directory or an indeterminate stat all leave the directory distrusted, + // which costs at most one extra sweep and a re-sign-in. + return !wasOtherWritable && Files.notExists(untrustedSentinel(), LinkOption.NOFOLLOW_LINKS); + } catch (UnsupportedOperationException e) { + // non-POSIX FS (e.g. Windows): cannot enforce owner-only perms; keep the inherited ACL. + // + // The permission half of the verdict is unavailable here, but the SENTINEL half is not, and it is + // the half that carries a peer's verdict across a filesystem this client cannot read mode bits on. + // The sentinel is deliberately permission-INDEPENDENT - design/oidc-token-persistence.md requires + // honouring it "whatever the permission bits say", precisely so a client on one platform can act on + // a distrust another platform's client published into the shared directory the frozen on-disk + // contract exists to allow. Returning a bare true ignored it: a directory a POSIX peer marked + // untrusted and had not finished sweeping (its sweep latches whenever a single entry resists + // deletion - see discardUntrustedDirectoryContents) read as trusted here, so this client adopted + // entries out of it and presented their tokens, and never swept or cleared the mark either. + // canUseShortDirectoryLockLease already evaluates the sentinel on this same catch, for the same + // reason it gives there. + // + // NOFOLLOW_LINKS and notExists for the reason the POSIX return above spells out: both defaults fail + // OPEN, and a link, a directory or an indeterminate stat must all leave the directory distrusted. + warnNoPosixPermsOnce(); + return Files.notExists(untrustedSentinel(), LinkOption.NOFOLLOW_LINKS); + } + } + + private boolean canUseShortDirectoryLockLease() { + // Read permissions BEFORE the sentinel. Recovery publishes the sentinel before tightening the + // directory, and clears it only after a complete sweep. That order means we cannot combine a stale + // "no sentinel" observation with the owner-only permissions recovery publishes later and mistake an + // in-progress sweep for a trusted directory. + try { + try { + final Set perms = Files.getPosixFilePermissions(directory); + if (perms.contains(PosixFilePermission.GROUP_WRITE) + || perms.contains(PosixFilePermission.OTHERS_WRITE)) { + return false; + } + } catch (UnsupportedOperationException e) { + // On a non-POSIX filesystem restrictToOwner cannot derive an untrusted verdict from mode bits. + // A sentinel still blocks the short lease below if another implementation left one behind. + } + return Files.notExists(untrustedSentinel(), LinkOption.NOFOLLOW_LINKS); + } catch (IOException | SecurityException e) { + // Uncertainty is the fail-closed direction: retain the long configured window rather than risk + // admitting a peer while a distrust sweep can still resume. + return false; + } + } + + private void stealIfStale(Path lock) { + stealIfStale(lock, lockStaleMillis, EMPTY_LOCK_STEAL_GRACE_MILLIS); + } + + private void stealIfStale(Path lock, long staleMillis, long emptyLockStealGraceMillis) { + // Steal a lock abandoned by a crashed holder, but never remove a peer's freshly-created LIVE lock. A + // bare deleteIfExists(lock) removes whatever sits at the path at that instant - including a fresh lock + // a peer created in the gap since we judged the old one stale - and would admit two holders at once. + // Instead: read the current owner stamp, confirm the lock is stale, then capture it atomically into a + // private name (rename is atomic, so among racing stealers exactly one captures it; the losers get + // NoSuchFileException and fall back to the wait), then verify what we captured carries the same stamp + // we judged stale. If a peer had already replaced it with a live lock we grabbed that instead, so we + // put it back rather than steal it. This mirrors releaseLock's own-stamp check and shrinks the + // residual race from the whole age-check->delete gap to the gap between the two renames. + // + // A name this class could not have written is a SQUATTER, not a lock, and no amount of waiting turns + // it into one: createLockFile only ever produces a regular file. Displace it at once, exactly as + // markUntrusted displaces a squatted sentinel name and for the reason it gives there - "Nothing but + // this class writes this name, so there is no peer state to lose". + // + // It has to be a shape test rather than the stamp read below, because neither the age check nor the + // stamp read can settle these. createLockFile's O_CREAT|O_EXCL reports EEXIST for a directory and for + // a symlink just as it does for a peer's lock, so acquireLock polled its whole budget and threw, and + // threw again on every later call - permanently, since no sweep reclaims this name either + // (discardUntrustedDirectoryContents skips it for want of a hash prefix, and sweepTempFiles globs + // *.tmp). For the required directory lock that means persistence is silently dead: every + // load()/save() fails, OidcDeviceAuth degrades to "continuing without persistence", and every process + // start re-runs the interactive device flow. Ageing them instead would merely delay it by a staleness + // window, and a dangling symlink would never age at all - NOFOLLOW stats the link, and the link is as + // young as the ln -s that planted it. + // + // This is also what this file already claims, where discardUntrustedDirectoryContents deliberately + // leaves .lock names in place: "acquireLock already treats a hostile or stale one as stealable". + if (!Files.isRegularFile(lock, LinkOption.NOFOLLOW_LINKS)) { + displaceLockSquatter(lock); + return; + } + // THREE states, not two. readLockHolder hands back the stamp bytes, hands back null for a + // legitimately empty or oversized lock, and THROWS when a regular file that IS shaped like a lock + // cannot be opened for reading - its mode denies this uid. That throw used to return outright, on the + // reasoning that "the create attempt or a peer settles it"; nothing settles it, for the same reason + // the shapes above are not settled. It needs no attacker either: a run under a different uid (a + // sudo -E start, a re-mapped container uid) killed while holding the lock leaves a 0600 file this uid + // cannot read. + // + // Unlike a squatter this one is genuinely ambiguous - it may be another uid's LIVE lock - so it is not + // displaced. Carry "the stamp could not be read" as its own state and let the age check and the + // capture-verify below settle it, exactly as they settle an empty one. It is aged on the FULL + // staleness window, never the short empty-lock grace: that grace exists for a creator paused inside + // createLockFile's own create->stamp window, a state this protocol can genuinely be in for a moment, + // and an unreadable file is not. + byte[] before = null; + boolean beforeStampReadable = false; + try { + before = readLockHolder(lock); + beforeStampReadable = true; + } catch (IOException e) { + // leave beforeStampReadable false; the age check and the capture-verify below still apply + } + // Read the stamp first, then the mtime: if a peer replaces the lock in between, the fresh mtime keeps + // us from proceeding. Preserve the exact mtime for the capture verification too. The directory-lock + // heartbeat changes metadata rather than the owner stamp, so comparing only the stamp after capture + // could discard a live lock that renewed between this age check and the atomic move. + // + // NOFOLLOW_LINKS: stat the NAME, not whatever it points at. A dangling symlink squatting the lock name + // has no target to stat, so the link-following default threw here and returned - the second route into + // the same permanent wedge. It also keeps the before/after mtimes describing one object across the + // capture, since the rename moves the link itself rather than its target. + final FileTime beforeModified; + try { + beforeModified = Files.getLastModifiedTime(lock, LinkOption.NOFOLLOW_LINKS); + } catch (IOException e) { + return; // the name is gone, or the directory is unreadable; nothing to age or steal + } + final long staleThresholdMillis = beforeStampReadable && before == null + ? emptyLockStealGraceMillis + : staleMillis; + if (System.currentTimeMillis() - beforeModified.toMillis() <= staleThresholdMillis) { + return; + } + if (beforeStampReadable && before == null) { + // an empty/unreadable lock is almost never a validly-held lock: acquireLock creates the lock and + // stamps the owner nonce onto the same open channel (createLockFile via CREATE_NEW), so a live lock + // carries its stamp within the tiny create->stamp window. An empty lock therefore means either a + // crash mid-write (the exclusive create succeeded but the nonce write did not) or a peer momentarily + // caught in that narrow window - a GC/safepoint pause CAN land there, which is exactly why the grace + // exists. Steal it on the short empty-lock grace rather than the full staleness window, so a crash + // orphan stops wedging peers for the whole window; the capture-verify below still confirms the lock + // is unchanged before completing the steal. (A cross-machine clock skew wider than the grace could + // still pre-empt such a partial lock, but that never forges or tears a credential - Layer-1's + // atomic rename holds - it degrades to at most a concurrent refresh, the best-effort residual + // inLock already accepts.) + // + // The per-identity grace is used verbatim, never clamped down to a smaller lockStaleMillis. It is + // the one thing standing between a refresh peer caught mid-stamp and having its live lock stolen, + // so the frozen cross-language contract states that clients MUST NOT shorten it. The required + // directory lock uses its own shorter grace: withDirectoryLock verifies the owner stamp again after + // tightening the directory, so a paused creator whose empty file was reclaimed aborts before doing + // any trust recovery or token I/O. That extra ownership check makes prompt crash recovery safe there. + } + final Runnable hook = beforeCaptureHook; + if (hook != null) { + hook.run(); + } + final Path captured = lock.resolveSibling(lock.getFileName().toString() + '.' + UUID.randomUUID() + ".tmp"); + try { + Files.move(lock, captured, StandardCopyOption.ATOMIC_MOVE); + } catch (IOException e) { + // NoSuchFile: a peer already stole/removed it; AtomicMoveNotSupported or other IO: degrade and + // leave the lock for the staleness path. Either way we have not removed a peer's live lock. + return; + } + // Read the stamp and the mtime in SEPARATE try blocks. Folding them into one (as this did) meant a + // stamp read that throws skipped the mtime read with it, leaving afterModified null - harmless while + // an unreadable stamp returned before ever reaching the capture, but with that state now carried here + // it would make every such capture unconfirmable and restore the very shape we are reclaiming. + byte[] after = null; + boolean afterStampReadable = false; + try { + after = readLockHolder(captured); + afterStampReadable = true; + } catch (IOException ignore) { + // still unreadable after the capture; matched against beforeStampReadable below + } + FileTime afterModified = null; + try { + afterModified = Files.getLastModifiedTime(captured, LinkOption.NOFOLLOW_LINKS); + } catch (IOException ignore) { + // captured but cannot re-stat it; treated as a non-match below, so we restore + } + // Confirm we captured the same stamp AND mtime we judged stale (or the same empty/oversized junk, or + // the same unreadable shape), not a peer's replacement or a live directory lock renewed in the gap. + // + // Readability is part of that identity, which is why beforeStampReadable is compared rather than + // folded into "before == null": readLockHolder returns null for a legitimately empty or oversized lock + // but THROWS on an IO error, and treating those alike would complete a steal on the strength of an IO + // error rather than on evidence the name was unchanged. It cuts both ways here - a name that could not + // be read before the capture but yields a stamp after it is not what we judged stale, so we restore. + final boolean confirmedStale = afterModified != null + && beforeStampReadable == afterStampReadable + // The heartbeat and age calculation both operate at millisecond precision. Compare at that + // same precision so a filesystem that normalizes sub-millisecond metadata during the atomic + // rename does not make every genuinely abandoned lock look renewed. + && beforeModified.toMillis() == afterModified.toMillis() + && (!beforeStampReadable + || (before == null ? after == null : Arrays.equals(before, after))); + if (confirmedStale) { + // genuinely the abandoned lock: drop it, so the next createLockFile can claim a fresh one + deleteCapturedLock(captured); + return; + } + // We captured a live lock a peer recreated in the gap (or could not re-read what we captured): put it + // back rather than steal it. + restoreCapturedLock(lock, captured); + } + + + private void sweepStaleTempFiles(String hashPrefix) { + // a crash between createTempFile and the atomic rename orphans a *.tmp holding a + // valid-at-the-time refresh token; unlike the lock file nothing ever steals it, so it would accumulate + // across crashes. Best-effort sweep on save: delete only temps older than the lock-staleness window, so + // a temp a concurrent writer is actively using (its mtime is seconds old) is never removed. A separate + // random suffix per writer keeps concurrent saves from colliding, which is why temps are not a fixed name + sweepTempFiles(hashPrefix, lockStaleMillis); + } + + private void sweepTempFiles(String hashPrefix, long minAgeMillis) { + // shared by save()'s staleness-bounded sweep and clear()'s unconditional one (minAgeMillis 0), which + // must reclaim even a freshly orphaned temp because it holds a plaintext refresh token the caller has + // just asked to forget + try (DirectoryStream stream = Files.newDirectoryStream(directory, hashPrefix + "*.tmp")) { + final long now = System.currentTimeMillis(); + for (Path tmp : stream) { + // never sweep a steal-captured lock (.lock..tmp): it is a cross-process steal in + // progress, not an orphaned write temp, and ATOMIC_MOVE preserves the stale lock's old mtime + // onto it, so the age guard below would judge an in-flight capture sweepable and delete it - + // destroying a lock the stealer may be about to restore to its live owner. A save write temp is + // .tmp and never contains ".lock.", so this only excludes captures. A capture + // orphaned by a crash mid-steal is rare and harmless (the canonical lock path is left free), so + // it is deliberately not reclaimed here. + if (tmp.getFileName().toString().contains(".lock.")) { + continue; + } + try { + // minAgeMillis <= 0 is clear()'s "at ANY age" sweep and must not consult the clock at + // all. Going through the comparison made it conditional on one: a temp whose recorded + // mtime is AHEAD of now - a network home whose server clock leads the client's, or a + // wall-clock step back from an NTP correction, a snapshot restore, a container starting + // before time sync - yields a negative left-hand side, which is not >= 0, so the sweep + // skipped it. save()'s sweep skips it too, for the same reason against a larger + // threshold, so nothing in this class would ever reclaim it: clear() would report + // success while a temp holding the full serialized entry - access, id and refresh + // tokens in plaintext - stayed on disk after the caller asked to forget the credential. + if (minAgeMillis <= 0 + || now - Files.getLastModifiedTime(tmp).toMillis() >= minAgeMillis) { + Files.deleteIfExists(tmp); + } + } catch (IOException ignore) { + // best-effort; skip this entry and let a later sweep retry + } + } + } catch (IOException ignore) { + // best-effort; a sweep failure must never fail a save + } + } + + private Path tokenFile(TokenStoreKey key) { + return directory.resolve(key.hash() + ".json"); + } + + private Path untrustedSentinel() { + return directory.resolve(UNTRUSTED_SENTINEL_NAME); + } + + @FunctionalInterface + private interface DirectoryAction { + T run(boolean isDirectoryTrusted) throws IOException; + } + + private static final class DirectoryLockHeartbeat implements AutoCloseable { + private final Path lock; + private final String nonce; + private final Thread thread; + private volatile boolean closed; + + private DirectoryLockHeartbeat(Path lock, String nonce) { + this.lock = lock; + this.nonce = nonce; + this.thread = new Thread(this::run, "questdb-oidc-store-lock-heartbeat"); + this.thread.setDaemon(true); + this.thread.start(); + } + + @Override + public void close() { + closed = true; + thread.interrupt(); + boolean interrupted = false; + try { + // Bound teardown too: a filesystem call stuck in the heartbeat must not turn a completed token + // operation into an unbounded close. The daemon checks closed again before renewing. + thread.join(DIRECTORY_LOCK_HEARTBEAT_MILLIS); + } catch (InterruptedException e) { + interrupted = true; + } + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + + private void run() { + while (!closed) { + try { + Thread.sleep(DIRECTORY_LOCK_HEARTBEAT_MILLIS); + } catch (InterruptedException e) { + if (closed) { + return; + } + } + if (closed) { + return; + } + try { + // Do not renew a lock another process stole or replaced. The ownership check and timestamp + // update retain the same one-syscall residual as releaseLock's checked delete: a replacement + // in that tiny gap can receive one harmless extra lease interval, never altered contents. + if (!isLockOwner(lock, nonce)) { + return; + } + if (closed) { + return; + } + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis())); + } catch (IOException | RuntimeException e) { + // Losing the ability to renew falls back to the lease: peers may reclaim after the stale + // interval rather than wait forever. The foreground operation still verifies ownership + // before entering this heartbeat and releaseLock never deletes a replacement. + return; + } + } + } + } + + /** + * One operation identity's in-process lock plus the number of callers currently holding or queued on it. + *

+ * {@code users} is read and written only inside {@link ConcurrentHashMap#compute} / + * {@link ConcurrentHashMap#computeIfPresent} remapping functions, which run under the bin lock, so it + * needs no volatility or atomics of its own. + */ + private static final class ProcessLock { + final ReentrantLock lock = new ReentrantLock(); + int users; + } + + private static final class TokenFileParser implements JsonParser { + private static final int FIELD_ACCESS_TOKEN = 8; + private static final int FIELD_AUDIENCE = 6; + private static final int FIELD_CLIENT_ID = 2; + private static final int FIELD_DEVICE_AUTHORIZATION_ENDPOINT = 4; + private static final int FIELD_EXPIRES_AT_MILLIS = 11; + private static final int FIELD_GROUPS_IN_TOKEN = 7; + private static final int FIELD_ID_TOKEN = 9; + private static final int FIELD_NONE = 0; + private static final int FIELD_REFRESH_TOKEN = 10; + private static final int FIELD_SCOPE = 5; + private static final int FIELD_TOKEN_ENDPOINT = 3; + private static final int FIELD_TOKEN_TTL_MILLIS = 12; + private static final int FIELD_VERSION = 1; + final StringSink accessToken = new StringSink(); + final StringSink audience = new StringSink(); + final StringSink clientId = new StringSink(); + final StringSink deviceAuthorizationEndpoint = new StringSink(); + final StringSink idToken = new StringSink(); + final StringSink refreshToken = new StringSink(); + final StringSink scope = new StringSink(); + final StringSink tokenEndpoint = new StringSink(); + long expiresAtMillis; + boolean groupsInToken; + long tokenTtlMillis; + long version; + private int depth; + private int field = FIELD_NONE; + private boolean malformed; + + @Override + public void onEvent(int code, CharSequence tag, int position) { + switch (code) { + case JsonLexer.EVT_ARRAY_START: + // the on-disk schema is a single flat JSON object; an array anywhere (for example a + // top-level [ {..} ] wrapper) is a malformed or hostile shape - mark the document invalid + // rather than extract fields from it through the object-depth gate + malformed = true; + break; + case JsonLexer.EVT_OBJ_START: + depth++; + break; + case JsonLexer.EVT_OBJ_END: + depth--; + break; + case JsonLexer.EVT_NAME: + if (depth == 1) { + if (Chars.equals("v", tag)) { + field = FIELD_VERSION; + } else if (Chars.equals("client_id", tag)) { + field = FIELD_CLIENT_ID; + } else if (Chars.equals("token_endpoint", tag)) { + field = FIELD_TOKEN_ENDPOINT; + } else if (Chars.equals("device_authorization_endpoint", tag)) { + field = FIELD_DEVICE_AUTHORIZATION_ENDPOINT; + } else if (Chars.equals("scope", tag)) { + field = FIELD_SCOPE; + } else if (Chars.equals("audience", tag)) { + field = FIELD_AUDIENCE; + } else if (Chars.equals("groups_in_token", tag)) { + field = FIELD_GROUPS_IN_TOKEN; + } else if (Chars.equals("access_token", tag)) { + field = FIELD_ACCESS_TOKEN; + } else if (Chars.equals("id_token", tag)) { + field = FIELD_ID_TOKEN; + } else if (Chars.equals("refresh_token", tag)) { + field = FIELD_REFRESH_TOKEN; + } else if (Chars.equals("expires_at_millis", tag)) { + field = FIELD_EXPIRES_AT_MILLIS; + } else if (Chars.equals("token_ttl_millis", tag)) { + field = FIELD_TOKEN_TTL_MILLIS; + } else { + field = FIELD_NONE; + } + } + break; + case JsonLexer.EVT_VALUE: + if (depth == 1) { + switch (field) { + case FIELD_VERSION: + // keep the full long: an over-32-bit value (e.g. 1 + 2^32) must not narrow to + // SCHEMA_VERSION and slip through the schema gate, so compare it as a long + version = parseLongOrZero(tag); + break; + case FIELD_CLIENT_ID: + putValue(clientId, tag); + break; + case FIELD_TOKEN_ENDPOINT: + putValue(tokenEndpoint, tag); + break; + case FIELD_DEVICE_AUTHORIZATION_ENDPOINT: + putValue(deviceAuthorizationEndpoint, tag); + break; + case FIELD_SCOPE: + putValue(scope, tag); + break; + case FIELD_AUDIENCE: + putValue(audience, tag); + break; + case FIELD_GROUPS_IN_TOKEN: + groupsInToken = Chars.equals("true", tag); + break; + case FIELD_ACCESS_TOKEN: + putValue(accessToken, tag); + break; + case FIELD_ID_TOKEN: + putValue(idToken, tag); + break; + case FIELD_REFRESH_TOKEN: + putValue(refreshToken, tag); + break; + case FIELD_EXPIRES_AT_MILLIS: + expiresAtMillis = parseLongOrZero(tag); + break; + case FIELD_TOKEN_TTL_MILLIS: + tokenTtlMillis = parseLongOrZero(tag); + break; + default: + break; + } + } + field = FIELD_NONE; + break; + default: + break; + } + } + + private static void putValue(StringSink sink, CharSequence tag) { + // The writer omits a null/absent field entirely, so a value event means the field was present + // with a real string: store it verbatim, including a value that is literally "null". A bare JSON + // null in a hand-edited or non-conforming file lands here as "null" too, because JsonLexer + // reports the two identically - which is exactly why the frozen format forbids a writer from + // emitting one (design/oidc-token-persistence.md). + // + // Faithfully round-tripping whatever is on disk is this parser's job; deciding whether a value is + // fit to be a credential is not. OidcDeviceAuth.adopt() makes that call, and refuses a served + // token of "null" along with the blank and control-character ones, so a non-conforming writer + // degrades to an interactive sign-in rather than to a "Bearer null" header the server answers + // with 401. Nothing here rejects it: the fingerprint covers client_id, the endpoints, scope, + // audience and groups_in_token - never the token - and "null" is four printable ASCII characters, + // so the char check passes it too. + sink.clear(); + sink.put(tag); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/OidcAuthException.java b/core/src/main/java/io/questdb/client/cutlass/auth/OidcAuthException.java new file mode 100644 index 000000000..ab3c64cbb --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/OidcAuthException.java @@ -0,0 +1,143 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.str.DisplaySafe; +import io.questdb.client.std.str.StringSink; + +/** + * Thrown when the OIDC device authorization flow cannot obtain a token. The message is built via + * the fluent {@link #put(CharSequence)} family, backed by a {@link StringSink}. + *

+ * For an OAuth error response (RFC 6749 / RFC 8628), {@link #getOauthError()} returns the + * machine-readable error code (e.g. {@code access_denied}, {@code expired_token}), stripped of any + * display-unsafe characters so a caller may log it as-is; else {@code null}. + */ +public class OidcAuthException extends RuntimeException { + private final StringSink message = new StringSink(); + private String oauthError; + + public OidcAuthException() { + } + + public OidcAuthException(CharSequence message) { + this.message.put(message); + } + + public OidcAuthException(Throwable cause) { + super(cause); + } + + /** + * Builds an exception from an OAuth error response. + * + * @param error the OAuth {@code error} code, never null + * @param description the optional {@code error_description}, may be null or empty + * @return a new exception carrying the error code + */ + public static OidcAuthException oauthError(CharSequence error, CharSequence description) { + OidcAuthException e = new OidcAuthException(); + // Sanitize the IdP error code once and reuse the result for both the message and the field. + // getOauthError() is public, so it must not hand a logging caller the raw provider string: JsonLexer + // now decodes JSON escapes, so an "error" such as access_deniedESC[2J would otherwise deliver a + // real ESC (ANSI/CR-LF/bidi injection) straight out of getOauthError(). + e.oauthError = sanitize(error); + e.put("the identity provider returned an error [error="); + if (e.oauthError != null) { + e.put(e.oauthError); + } + if (description != null && description.length() > 0) { + e.put(", description=").putSanitized(description); + } + e.put(']'); + return e; + } + + // Whether a character must never reach a terminal or log line, delegated to the shared DisplaySafe + // classifier so the auth layer and Utf16Sink.putAsPrintable judge display safety identically. The + // argument is a code point, not a UTF-16 unit: appendSanitized scans with codePointAt, which joins a + // surrogate pair into one code point, so a supplementary-plane format/control char is judged whole + // rather than as two harmless-looking halves (the gap that once let an invisible U+E00xx "tag" char + // through). A lone unpaired surrogate surfaces as a SURROGATE code point and is stripped too. + static boolean isUnsafeForDisplay(int c) { + return DisplaySafe.isUnsafeForDisplay(c); + } + + // strips display-unsafe chars from cs into sink; the shared primitive behind putSanitized and sanitize + private static void appendSanitized(StringSink sink, CharSequence cs) { + if (cs != null) { + for (int i = 0, n = cs.length(); i < n; ) { + final int cp = Character.codePointAt(cs, i); + final int count = Character.charCount(cp); + if (!isUnsafeForDisplay(cp)) { + sink.put(cs, i, i + count); + } + i += count; + } + } + } + + // returns cs with display-unsafe chars stripped, or null when cs is null. Backs the oauthError field, + // which getOauthError() exposes to callers that may render it raw. + private static String sanitize(CharSequence cs) { + if (cs == null) { + return null; + } + StringSink sink = new StringSink(); + appendSanitized(sink, cs); + return sink.toString(); + } + + @Override + public String getMessage() { + return message.toString(); + } + + public String getOauthError() { + return oauthError; + } + + public OidcAuthException put(char ch) { + message.put(ch); + return this; + } + + public OidcAuthException put(CharSequence cs) { + message.put(cs); + return this; + } + + public OidcAuthException put(long value) { + message.put(value); + return this; + } + + // appends untrusted text with display-unsafe chars stripped, so an attacker-influenced IdP error + // string cannot inject ANSI escapes, forge log lines, or smuggle bidi/zero-width formatting when + // the exception message is rendered + private void putSanitized(CharSequence cs) { + appendSanitized(message, cs); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/OidcDeviceAuth.java b/core/src/main/java/io/questdb/client/cutlass/auth/OidcDeviceAuth.java new file mode 100644 index 000000000..9e1785df9 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/OidcDeviceAuth.java @@ -0,0 +1,3239 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.DefaultHttpClientConfiguration; +import io.questdb.client.HttpClientConfiguration; +import io.questdb.client.cutlass.http.HttpException; +import io.questdb.client.cutlass.http.client.Fragment; +import io.questdb.client.cutlass.http.client.HttpClient; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.http.client.HttpClientFactory; +import io.questdb.client.cutlass.http.client.Response; +import io.questdb.client.cutlass.json.JsonException; +import io.questdb.client.cutlass.json.JsonLexer; +import io.questdb.client.cutlass.json.JsonParser; +import io.questdb.client.std.Chars; +import io.questdb.client.std.Misc; +import io.questdb.client.std.Mutable; +import io.questdb.client.std.Numbers; +import io.questdb.client.std.NumericException; +import io.questdb.client.std.Os; +import io.questdb.client.std.QuietCloseable; +import io.questdb.client.std.str.DirectUtf8Sequence; +import io.questdb.client.std.str.StringSink; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.net.InetAddress; +import java.net.URLEncoder; +import java.net.UnknownHostException; +import java.util.Locale; +import java.util.Objects; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.ReentrantLock; + +/** + * Obtains an OIDC access or id token via the OAuth 2.0 Device Authorization Grant + * (RFC 8628), so a browserless process (remote notebook kernel, container, headless job) + * can sign a human in: the user authorizes on any device while the token request travels + * outbound only. + *

+ * The token works on any auth path the server validates: + *

+ * Typical use, discovering everything from the QuestDB server: + * 
{@code
+ * try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB("https://questdb.example.com:9000")) {
+ *     String token = auth.signIn(); // signs in on first use, then caches and refreshes
+ *     // ... use token as an HTTP Bearer header or a PG-wire _sso password ...
+ * }
+ * }
+ * Or configuring the identity provider explicitly: + * 
{@code
+ * OidcDeviceAuth auth = OidcDeviceAuth.builder()
+ *         .clientId("questdb")
+ *         .deviceAuthorizationEndpoint("https://idp.example.com/as/device_authz.oauth2")
+ *         .tokenEndpoint("https://idp.example.com/as/token.oauth2")
+ *         .scope("openid groups")
+ *         .groupsInToken(true)
+ *         .build();
+ * }
+ * {@link #signIn()} serves a cached token while valid, silently refreshes when a refresh token + * exists, otherwise re-runs the interactive flow. An instance lock serializes calls, so two + * sign-ins never start at once. A sign-in waiting for the user holds that lock for the device code + * lifetime (up to 30 minutes), so a concurrent {@link #signIn()} or {@link #clearCache()} blocks + * behind it - but {@link #getToken()} never waits behind an interactive sign-in: it fails fast with an + * {@link OidcAuthException} rather than stall a request/flush path (a needed silent refresh still runs, + * each HTTP round-trip phase bounded by {@link Builder#httpTimeoutMillis(int)}, and the TCP connect and TLS + * handshake bounded by it too - though DNS resolution is still the OS's to bound - plus, with a + * coordinating {@link TokenStore}, a brief + * cross-process lock wait; see {@link #getToken()}). To abort a waiting sign-in, call + * {@link #close()} from another thread; it signals the flow to stop, which then fails with an + * {@link OidcAuthException} rather than polling until the device code expires. Cancellation is seen + * between polls (within ~100ms while waiting out an interval); a token-store lock waiter is interrupted to + * abandon that interruptible wait, but a poll already in flight is not cancelled at the transport, so the + * abort - and {@link #close()} - can take up to one HTTP request timeout (see + * {@link Builder#httpTimeoutMillis(int)}), still far short of the device-code lifetime (a + * {@link DeviceCodePrompt} that blocks in {@code promptUser}, such as the default browser launch, can + * extend that wait by however long it runs). + *

+ * Instances are interactive and hold a network connection; close them when done. Token state is + * in-memory only by default; pass a {@link TokenStore} (via {@link Builder#tokenStore(TokenStore)} or + * {@link DiscoveryOptions#tokenStore(TokenStore)}) to persist it across process restarts, so a restarted + * process resumes from a saved refresh token instead of running the interactive flow again. + */ +public class OidcDeviceAuth implements QuietCloseable { + public static final String DEFAULT_SCOPE = "openid"; + static final String GRANT_TYPE_DEVICE_CODE = "urn:ietf:params:oauth:grant-type:device_code"; + static final String GRANT_TYPE_REFRESH_TOKEN = "refresh_token"; + // fixed clock-skew margin (matches the Python client questdb.auth): getToken() treats a cached token as + // expired this many millis before its real exp, to absorb clock drift and request latency. Not + // configurable; effectiveSkewMillis() caps it at half the token lifetime so a short-lived token is not + // reported expired the instant it is issued. + private static final long CLOCK_SKEW_MILLIS = 30_000L; + // device code TTL when the device authorization response omits (or zeroes) expires_in; matches Python + private static final int DEFAULT_DEVICE_CODE_TTL_SECONDS = 600; + private static final int DEFAULT_HTTP_TIMEOUT_MILLIS = 30_000; + private static final int DEFAULT_POLL_INTERVAL_SECONDS = 5; + // token cache TTL when the token response omits expires_in + private static final int DEFAULT_TOKEN_TTL_SECONDS = 300; + // The config the DISCOVERY clients take. Discovery runs off DEFAULT_HTTP_TIMEOUT_MILLIS rather than a + // builder value, because it happens before there is an instance to carry one. See httpConfig(). + private static final HttpClientConfiguration DISCOVERY_HTTP_CONFIG = httpConfig(DEFAULT_HTTP_TIMEOUT_MILLIS); + private static final String ERROR_AUTHORIZATION_PENDING = "authorization_pending"; + private static final String ERROR_SLOW_DOWN = "slow_down"; + // getToken() polls for the instance lock in slices this small so it observes an interactive sign-in that + // starts while it waits (and close()) promptly, rather than blocking a whole refresh behind a single + // acquire; see acquireForGetToken() + private static final long GET_TOKEN_LOCK_POLL_SLICE_MILLIS = 50; + // the grant_type values are constants, so url-encode them once at class load rather than on every + // device-code poll and token refresh + private static final String GRANT_TYPE_DEVICE_CODE_ENCODED = urlEncode(GRANT_TYPE_DEVICE_CODE); + private static final String GRANT_TYPE_REFRESH_TOKEN_ENCODED = urlEncode(GRANT_TYPE_REFRESH_TOKEN); + // a rate-limited identity provider answers 429; the token poll treats it as a transient backoff + private static final String HTTP_STATUS_TOO_MANY_REQUESTS = "429"; + // Token responses carry JWTs (an id token with group claims can be several KB), and a single + // value may arrive split across HTTP fragments. The lexer stashes a split value and rejects it + // past JSON_LEXER_MAX_VALUE_BYTES, so the limit must comfortably exceed any real token or large + // tokens fail to parse with "String is too long". + private static final int JSON_LEXER_CACHE_SIZE = 1024; + private static final int JSON_LEXER_MAX_VALUE_BYTES = 1 << 20; + // the I/O portion of a coordinated refresh, as a multiple of httpTimeoutMillis. The refresh under the + // lock runs, in order: TCP connect, the TLS handshake, send, await, parse, and a body drain on a parse + // failure. httpConfig() derives BOTH getConnectTimeout() and getTimeout() from httpTimeoutMillis, and + // HttpClient spends them separately - it grants the handshake a fresh budget anchored at its own start + // (see the tlsHandshakeStartNanos block), rather than continuing the connect's. So the connection phase + // alone is worth two of these, not one, and six is the count of independently bounded phases. It was 4, + // which enumerated only send/await/parse/drain and left connect and TLS out; the floor it feeds was then + // 480s at the 120s httpTimeoutMillis cap while a hold could reach 720s, so a peer could judge a live + // holder's lock stale and steal it mid-refresh - exactly the race the floor exists to prevent. The only + // part of a hold this still does not account for is DNS resolution, which the OS bounds. build() requires + // the FileTokenStore staleness window to exceed this multiple as a floor (see build()) + private static final int LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE = 6; + private static final Logger LOG = LoggerFactory.getLogger(OidcDeviceAuth.class); + // upper bound on the device code lifetime (the device authorization response's expires_in), so a + // hostile or buggy provider cannot make the client poll for an absurd duration; matches the Python client + private static final int MAX_DEVICE_CODE_TTL_SECONDS = 1800; + // upper bound on the token cache lifetime (the token response's expires_in), so an absurd or hostile + // value cannot overflow the timing arithmetic or make the client trust a token for absurdly long + private static final int MAX_EXPIRES_IN_SECONDS = 3600; + // upper bound on the configurable HTTP request timeout. A token-endpoint round-trip never needs longer, and + // bounding it keeps the I/O portion of a refresh held under the FileTokenStore cross-process lock (send + + // await + parse, plus a body drain on a parse failure - each separately bounded by this, so up to ~4x this) + // a known, bounded multiple, so the store's staleness window can be sized to dominate it. The connection + // phase is bounded by this too - httpConfig() derives the connect timeout and the TLS handshake budget + // from the same figure - leaving only DNS resolution to the OS (see Builder.build()) + private static final int MAX_HTTP_TIMEOUT_MILLIS = 120_000; + // upper bound on the poll interval, both the initial value and the growth after a slow_down or 429, so + // a hostile or buggy provider cannot stall the poll loop; matches the Python client + private static final int MAX_POLL_INTERVAL_SECONDS = 60; + // cap bytes drained per response so a hostile/MITM'd server cannot stream an endless body and + // wedge the thread; far above any real OIDC JSON response + private static final int MAX_RESPONSE_BODY_BYTES = 4 * 1024 * 1024; + // Ceiling on the back-off between token store reads. maybeLoadFromStore() runs on the getToken() path, + // which an ILP producer calls once per flush, so a store that is permanently unreadable - a chmod or uid + // mismatch in a container, EIO/ESTALE on an NFS home - otherwise cost a blocking file open, two exception + // fills and a WARN line on EVERY flush, on the producer thread and under this instance's lock. A store + // that simply has nothing to return is unaffected: load() reports that by returning null rather than by + // throwing, and that latches storeLoadAttempted outright. + private static final long MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS = 60_000L; + /** + * Floor on how often {@link #getToken()} will re-attempt a silent refresh after one failed. Without it a + * revoked refresh token, or an IdP outage, cost a full token-endpoint round trip on EVERY call - and + * getToken() is called once per ILP flush and once per WebSocket (re)connect, so a producer retrying its + * rows drove a sustained request flood at the identity provider (enough to trip its rate limits and + * lengthen the very outage being retried) while each call blocked the producer for the round trip, up to + * httpTimeoutMillis against a black-holed endpoint. + *

+ * Deliberately short: this is a stampede guard, not a circuit breaker. A credential that comes back + * within seconds is picked up on the next call, and any explicit {@link #signIn()} or + * {@link #clearCache()} clears the latch outright. + */ + private static final long MIN_REFRESH_RETRY_INTERVAL_MILLIS = 5_000L; + // First non-zero back-off between token store reads; it doubles per consecutive failure, up to + // MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS. Same floor as the refresh back-off above, and the same kind of + // stampede guard. The FIRST failure arms a ZERO-length back-off, so the very next call still re-reads the + // store: a one-shot fault - notably a carried interrupt flag, which makes the InterruptibleChannel under + // FileTokenStore throw on a thread that merely carries it - must recover on the next call rather than wait + // this out. Anything that survives that free retry needs an operator (a chmod, a remount), so waiting is + // no longer costing a recovery that was about to happen anyway. + private static final long MIN_STORE_LOAD_RETRY_INTERVAL_MILLIS = 5_000L; + private static final int POLL_PENDING = 1; + private static final long POLL_SLEEP_SLICE_MILLIS = 100; + private static final int POLL_SLOW_DOWN = 2; + private static final int POLL_SUCCESS = 0; + private static final int POLL_TRANSIENT_ERROR = 3; + private static final int REFRESH_FAILED = 0; + private static final int REFRESH_NOT_ATTEMPTED = 2; + private static final int REFRESH_SUCCEEDED = 1; + private static final int SLOW_DOWN_INCREMENT_SECONDS = 5; + private static final String USER_AGENT = "questdb/java-client-oidc"; + private static final String WELL_KNOWN_OPENID_CONFIGURATION_PATH = "/.well-known/openid-configuration"; + private final String audienceEncoded; + // This instance's HTTP transport budgets, derived from httpTimeoutMillis. See httpConfig(). + private final HttpClientConfiguration clientConfig; + private final String clientIdEncoded; + private final DeviceAuthorizationResponseParser deviceAuthParser = new DeviceAuthorizationResponseParser(); + private final Endpoint deviceAuthorizationEndpoint; + private final StringSink formSink = new StringSink(); + private final boolean groupsInToken; + private final int httpTimeoutMillis; + // serializes signIn()/getToken()/clearCache()/close(); signIn() holds it for the whole + // interactive flow, getToken() uses tryLock so the flush path never stalls behind a sign-in + private final ReentrantLock lock = new ReentrantLock(); + private final Object tokenStoreWaiterGuard = new Object(); + // The thread currently entering or waiting in TokenStore.inLock(), or running the bundled + // FileTokenStore.clear(). Deliberately narrower than the instance lock hold: close() may interrupt a + // coordinating store operation, but must not leave an interrupt on a thread doing an ordinary load/save or + // about to return a cached token. The refresh action clears the marker as soon as inLock() acquires its lock; + // FileTokenStore.clear() keeps it for the whole call because that implementation is interrupt-neutral. + private Thread tokenStoreWaiterThread; + private final DeviceCodePrompt prompt; + private final StringSink responseStatus = new StringSink(); + private final String scopeEncoded; + private final TokenStoreKey storeKey; + private final ClientTlsConfiguration tlsConfig; + private final Endpoint tokenEndpoint; + private final TokenResponseParser tokenParser = new TokenResponseParser(); + private final TokenStore tokenStore; + private String accessToken; + private volatile boolean closed; + private long expiresAtMillis; + private String idToken; + // set only while signIn() runs the interactive device flow (holding the lock for up to the device-code + // lifetime). getToken() reads it lock-free to fail fast behind an interactive sign-in while still waiting + // briefly behind a peer's quick silent refresh; volatile for that cross-thread read. See acquireForGetToken() + private volatile boolean interactiveSignInInProgress; + private JsonLexer jsonLexer; + private String lastPersistedRefreshToken; + // earliest wall-clock millis at which maybeLoadFromStore() may re-read the store after a read threw; 0 + // until the first failure. See isStoreLoadBackedOff() + private long nextStoreLoadAttemptMillis; + private HttpClient plainClient; + private long refreshFailedAtMillis; + private String refreshToken; + private boolean storeLoadAttempted; + // back-off applied to the NEXT failed store read, doubling from MIN_ to MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS; + // 0 while no read has failed yet, which is what makes the first retry immediate + private long storeLoadRetryIntervalMillis; + private HttpClient tlsClient; + // lifetime in millis of the currently cached token (its clamped TTL); effectiveSkewMillis() caps the + // clock skew at half of this so a short-lived token is not treated as expired the instant it is issued + private long tokenTtlMillis; + + private OidcDeviceAuth(Builder builder, ClientTlsConfiguration tlsConfig, Endpoint deviceAuthorizationEndpoint, Endpoint tokenEndpoint) { + String clientId = builder.clientId; + // pre-encode the invariant form params once here, so the poll loop and silent refresh do not + // re-run URLEncoder on every request (mirrors the pre-encoded GRANT_TYPE_* constants) + this.clientIdEncoded = urlEncode(clientId); + // build() already parsed and validated these endpoints; reuse them rather than re-parse the raw strings + this.deviceAuthorizationEndpoint = deviceAuthorizationEndpoint; + this.tokenEndpoint = tokenEndpoint; + String scope = builder.scope; + this.scopeEncoded = urlEncode(scope); + String audience = builder.audience; + this.audienceEncoded = audience != null ? urlEncode(audience) : null; + this.groupsInToken = builder.groupsInToken; + this.httpTimeoutMillis = builder.httpTimeoutMillis; + // Derive the transport budgets from the SAME figure the rest of the class quotes, so the connection + // phase is bounded by it too rather than by the 600s HttpClientConfiguration default (TLS) and the + // OS (TCP connect). See httpConfig(). + this.clientConfig = httpConfig(this.httpTimeoutMillis); + this.prompt = builder.prompt; + this.tlsConfig = tlsConfig; + this.tokenStore = builder.tokenStore; + // key any persisted token by the identity it belongs to, built before the native lexer alloc so a + // throw here cannot leak it. Canonicalise the endpoints (lower-case scheme/host, explicit port) and + // normalise an empty audience to null, so the hash matches across processes and language clients. + this.storeKey = tokenStore == null ? null : new TokenStoreKey( + clientId, + canonicalEndpoint(this.tokenEndpoint), + canonicalEndpoint(this.deviceAuthorizationEndpoint), + scope, + audience != null && !audience.isEmpty() ? audience : null, + this.groupsInToken + ); + // allocate the native lexer last: urlEncode and the TokenStoreKey construction above can throw, and + // the half-built instance is never returned, so close() could not free an earlier alloc + this.jsonLexer = new JsonLexer(JSON_LEXER_CACHE_SIZE, JSON_LEXER_MAX_VALUE_BYTES); + } + + public static Builder builder() { + return new Builder(); + } + + /** + * Discovers the OIDC configuration from a running QuestDB server and builds an instance. + * Reads the public {@code /settings} endpoint (no auth) for the client id, scope, token + * endpoint, device authorization endpoint and groups-in-token mode. + *

+ * Trust model: the endpoints the user signs in against come from the server's + * unauthenticated {@code /settings} response, so a spoofed, compromised, or MITM'd server can + * redirect the whole sign-in to an attacker-controlled identity provider and harvest the + * authorization. Only call {@code fromQuestDB} against a trusted server reached over {@code https} + * (required by default; {@link Builder#allowInsecureTransport(boolean)} removes that protection). + * For an untrusted server, configure the identity provider explicitly with {@link #builder()}, or + * pin it via {@link #fromQuestDB(String, DiscoveryOptions)} and {@link DiscoveryOptions#issuer(String)}. + * + * @param questdbUrl the QuestDB HTTP base URL, for example {@code https://questdb.example.com:9000} + * @return a configured, ready-to-use instance + * @throws OidcAuthException if the server has OIDC disabled, or does not advertise a device + * authorization endpoint and no issuer was pinned to discover it + */ + public static OidcDeviceAuth fromQuestDB(String questdbUrl) { + return fromQuestDB(questdbUrl, new DiscoveryOptions()); + } + + /** + * Discovers the OIDC configuration from a running QuestDB server, like {@link #fromQuestDB(String)}, + * but with explicit {@link DiscoveryOptions}: an identity provider pin (issuer), a TLS configuration, an + * insecure-transport opt-in, and the device code prompt - for example + * {@link DeviceCodePrompt#openBrowser()} to also open the verification URL in a browser. + * + * @param questdbUrl the QuestDB HTTP base URL, for example {@code https://questdb.example.com:9000} + * @param options how to pin the identity provider, configure TLS, permit insecure transport, and + * show the device code challenge; see {@link DiscoveryOptions} + * @return a configured, ready-to-use instance + * @throws OidcAuthException if the server has OIDC disabled, or does not advertise a device + * authorization endpoint and no issuer was pinned + */ + public static OidcDeviceAuth fromQuestDB(String questdbUrl, DiscoveryOptions options) { + String issuer = options.issuer; + ClientTlsConfiguration tlsConfig = options.tlsConfig != null ? options.tlsConfig : defaultTlsConfig(); + boolean allowInsecureTransport = options.allowInsecureTransport; + Endpoint server = Endpoint.parse(questdbUrl); + if (!allowInsecureTransport) { + requireSecureTransport(server.isTls, "QuestDB server url", questdbUrl); + } + SettingsDiscoveryParser parser = new SettingsDiscoveryParser(); + discoverSettings(server, tlsConfig, parser); + if (!parser.isOidcEnabled) { + throw new OidcAuthException().put("OIDC is not enabled on the QuestDB server [url=").put(questdbUrl).put(']'); + } + if (parser.clientId.length() == 0) { + throw new OidcAuthException().put("the QuestDB server does not advertise an OIDC client id [url=").put(questdbUrl).put(']'); + } + String tokenEndpoint = parser.tokenEndpoint.length() > 0 ? parser.tokenEndpoint.toString() : null; + String deviceAuthorizationEndpoint = parser.deviceAuthorizationEndpoint.length() > 0 ? parser.deviceAuthorizationEndpoint.toString() : null; + String resolvedIssuer = issuer != null && !issuer.isEmpty() ? issuer : null; + // capture each endpoint's provenance before discovery may fill a missing one: only an endpoint the + // untrusted /settings response advertised is origin-pinned to the issuer below. An endpoint discovered + // from the provider's own .well-known is authoritative for wherever the pinned issuer hosts it, after + // the document's returned issuer has exactly matched the issuer used for the discovery request. + final boolean tokenEndpointFromSettings = tokenEndpoint != null; + final boolean deviceEndpointFromSettings = deviceAuthorizationEndpoint != null; + + // Over a plaintext, MITM-able http /settings channel (only reachable with allowInsecureTransport; + // the default rejects it), advertised endpoints can be tampered in transit to route the device + // code and long-lived refresh token to an attacker. The missing-endpoint discovery path below + // already demands an out-of-band pin, but a tampered /settings advertising BOTH endpoints at one + // attacker origin skips that path - the co-location check passes trivially and there is no issuer + // to pin against - so require the same pin before trusting /settings endpoints over such a channel. + boolean settingsSuppliedCredentials = tokenEndpoint != null || deviceAuthorizationEndpoint != null; + if (settingsSuppliedCredentials && resolvedIssuer == null && settingsChannelIsPlaintext(server)) { + throw new OidcAuthException() + .put("the QuestDB server was reached over insecure http, so its /settings response - and the OIDC ") + .put("endpoints it advertises - can be tampered in transit and used to redirect the device-code and ") + .put("refresh-token requests to an attacker; pin the identity provider with an issuer (its origin, for ") + .put("example https://your-idp), configure the endpoints explicitly with OidcDeviceAuth.builder(), or ") + .put("connect to QuestDB over https [url=").put(questdbUrl).put(']'); + } + + // For /settings-supplied endpoints with an out-of-band issuer, require each under the issuer's PATH, + // not just its origin (validateEndpointOrigins): a path-based identity provider shares one origin per + // tenant (Keycloak issuers are https://host/realms/), so the origin check alone cannot stop a + // tampered /settings from steering credentials to a different realm. The issuer is supplied out of + // band and cannot be forged. Endpoints discovered from the identity provider below are not scoped this + // way - some providers (for example Azure AD) place their endpoints outside the issuer path. + if (issuer != null && !issuer.isEmpty()) { + if (tokenEndpoint != null && !isEndpointUnderIssuerPath(tokenEndpoint, issuer)) { + throw endpointNotUnderIssuer("token endpoint", tokenEndpoint, issuer); + } + if (deviceAuthorizationEndpoint != null && !isEndpointUnderIssuerPath(deviceAuthorizationEndpoint, issuer)) { + throw endpointNotUnderIssuer("device authorization endpoint", deviceAuthorizationEndpoint, issuer); + } + } + + // Fall back to identity provider discovery when the server omits the device authorization endpoint + // (and/or the token endpoint). The provider's origin must be pinned out of band: the discovery + // target is never derived from a server-supplied value, else a tampered or intercepted /settings + // could steer discovery - and the credential POSTs - to an attacker while the co-location and + // issuer checks pass trivially. + if (deviceAuthorizationEndpoint == null || tokenEndpoint == null) { + if (resolvedIssuer == null) { + throw new OidcAuthException() + .put("the QuestDB server did not advertise the OIDC device authorization endpoint (and/or the token ") + .put("endpoint), so it must be discovered from the identity provider, but the identity provider is not ") + .put("pinned; pass an issuer (its origin, for example https://your-idp) to OidcDeviceAuth.fromQuestDB so ") + .put("a tampered or intercepted /settings response cannot redirect the device-code and refresh-token ") + .put("requests to an attacker, or configure the endpoints explicitly with OidcDeviceAuth.builder() [url=") + .put(questdbUrl).put(']'); + } + WellKnownDiscoveryParser doc = new WellKnownDiscoveryParser(); + discoverFromIdp(resolvedIssuer, tlsConfig, allowInsecureTransport, doc); + if (deviceAuthorizationEndpoint == null && doc.deviceAuthorizationEndpoint.length() > 0) { + deviceAuthorizationEndpoint = doc.deviceAuthorizationEndpoint.toString(); + } + if (tokenEndpoint == null && doc.tokenEndpoint.length() > 0) { + tokenEndpoint = doc.tokenEndpoint.toString(); + } + } + + // Pin the ORIGIN of any endpoint the untrusted /settings response advertised to the pinned issuer + // origin, so a tampered /settings cannot redirect the device code and refresh token to + // an attacker. An endpoint discovered from the identity provider's own .well-known is deliberately NOT + // origin-pinned: that document is fetched from the pinned origin and is authoritative for wherever the + // issuer hosts its endpoints - some providers (for example Google) serve the token and device endpoints + // from a different origin than the issuer. The co-location check (token and device share one origin) + // still applies to every endpoint, enforced by validateEndpointOrigins in build(). + if (resolvedIssuer != null) { + Endpoint pin = Endpoint.parse(resolvedIssuer); + if (tokenEndpointFromSettings && !isSameOrigin(Endpoint.parse(tokenEndpoint), pin)) { + throw endpointOriginNotPinned("token endpoint", tokenEndpoint, originOf(pin)); + } + if (deviceEndpointFromSettings && !isSameOrigin(Endpoint.parse(deviceAuthorizationEndpoint), pin)) { + throw endpointOriginNotPinned("device authorization endpoint", deviceAuthorizationEndpoint, originOf(pin)); + } + } + + if (tokenEndpoint == null) { + throw new OidcAuthException() + .put("could not resolve the OIDC token endpoint from the QuestDB /settings response or the identity ") + .put("provider discovery document; configure it explicitly with OidcDeviceAuth.builder() [url=").put(questdbUrl).put(']'); + } + if (deviceAuthorizationEndpoint == null) { + throw new OidcAuthException() + .put("could not resolve the device authorization endpoint; the identity provider discovery document did ") + .put("not advertise \"device_authorization_endpoint\". Ensure the identity provider supports the device ") + .put("grant, or configure the endpoint explicitly with OidcDeviceAuth.builder() [url=").put(questdbUrl).put(']'); + } + return builder() + .clientId(parser.clientId.toString()) + .deviceAuthorizationEndpoint(deviceAuthorizationEndpoint) + .tokenEndpoint(tokenEndpoint) + .scope(parser.scope.length() > 0 ? parser.scope.toString() : DEFAULT_SCOPE) + .audience(parser.audience.length() > 0 ? parser.audience.toString() : null) + .groupsInToken(parser.groupsInToken) + .allowInsecureTransport(allowInsecureTransport) + .tlsConfig(tlsConfig) + .prompt(options.prompt) + .tokenStore(options.tokenStore) + .build(); + } + + /** + * Drops any cached token so the next {@link #signIn()} starts a fresh interactive sign-in. + */ + public void clearCache() { + lock.lock(); + try { + throwIfClosed(); + // the same sweep close() runs: nulling the served token is not enough on its own, since the raw + // response and request bytes are still legible in the reusable sinks that carried them + wipeCredentialState(); + expiresAtMillis = 0; + tokenTtlMillis = 0; + refreshFailedAtMillis = 0; + if (tokenStore != null) { + // FileTokenStore.clear() coordinates with a peer refresh through the same interruptible + // per-identity lock as tryRefreshCoordinated(). Publish this caller too, otherwise close() + // sees no waiter and blocks on this instance lock for the peer's whole refresh. The bundled + // store makes its whole clear interrupt-neutral: even when cancellation abandons coordination, + // it still erases the entry before restoring the signal. Do NOT publish an arbitrary TokenStore + // clear here. The public SPI also covers keychains and vaults whose interruptible backend call + // may abort deletion; close() must wait for those implementations rather than return while the + // credential this explicit sign-out was meant to remove remains reloadable. + final boolean interruptibleClear = tokenStore instanceof FileTokenStore; + if (interruptibleClear) { + publishTokenStoreWaiter(); + } + try { + try { + tokenStore.clear(storeKey); + } catch (RuntimeException e) { + warnPersistence("clear", e); + } + } finally { + if (interruptibleClear) { + clearTokenStoreWaiter(); + } + } + } + // do not reload the entry we just removed on the next signIn()/getToken() + storeLoadAttempted = true; + } finally { + lock.unlock(); + } + } + + /** + * Frees the network connections and native buffers this instance holds. If a {@link #signIn()} + * sign-in is in flight on another thread, signals it to stop so it fails with an + * {@link OidcAuthException} instead of polling until the device code expires. The signal is observed + * between polls (within ~100ms while waiting out a poll interval). Only while an operation is in a + * coordinating token-store call is its thread interrupted: a refresh publishes only its wait for a peer + * instance's shared lock, while a clear publishes its whole best-effort call because the store SPI does not + * expose that wait separately. An HTTP request already in flight is not cancelled at the transport, so + * {@code close()} acquires the lock - and returns - only once that request finishes or times out, not the full + * device-code lifetime. That bound is + * the in-flight operation's own worst case, which is NOT a single HTTP request timeout: a silent refresh + * under the lock runs a send, an await and a body parse (each bounded by + * {@link Builder#httpTimeoutMillis(int)}), and its connection phase - + * TCP connect, TLS handshake - is bounded by that timeout as well, leaving only DNS resolution to the + * OS, so a black-holed token endpoint can hold the lock, and this {@code close()}, for roughly that (on + * Linux) rather than a single httpTimeoutMillis. The exception is a + * {@link DeviceCodePrompt} that blocks in {@code promptUser} - for example the default + * {@link DeviceCodePrompt#openBrowser()} prompt while it hands the verification URL to the OS browser, + * which is not bounded by the HTTP timeout: the flow holds the lock across that one-off prompt, so a + * racing {@code close()} waits it out too. Idempotent. After close, {@link #signIn()}, + * {@link #getToken()} and {@link #clearCache()} throw. + */ + @Override + public void close() { + // Flag cancellation before taking the lock, then interrupt only a thread published for a coordinating + // store-lock operation. Refresh publishes the exact TokenStore.inLock() acquisition boundary; the + // bundled FileTokenStore covers its whole interrupt-neutral clear call. Custom clear implementations + // are waited out because interrupting a keychain/vault delete can leave the credential reloadable. + // Publishing every instance-lock holder would also interrupt ordinary store reads and cached-token + // returns, leaking close()'s cancellation signal into unrelated caller work after a successful + // getToken(). The lock acquire must still succeed before native resources are freed, so other in-flight + // work is waited out as before. + closed = true; + synchronized (tokenStoreWaiterGuard) { + Thread waiter = tokenStoreWaiterThread; + if (waiter != null && waiter != Thread.currentThread()) { + waiter.interrupt(); + } + } + lock.lock(); + try { + // Drop the credential material FIRST. Every token operation is already refused by the closed flag + // above, so nothing here can be needed again - and nulling a String or overwriting a sink cannot + // throw, whereas an HttpClient close conceivably can, so doing it first means a failing free + // cannot leave a refresh token legible in this instance for the rest of the JVM's life. + wipeCredentialState(); + // free the native lexer first: its close() is a bare Unsafe.free that cannot throw, whereas an + // HttpClient close conceivably could - freeing the native buffer first means such a throw cannot + // strand it (Misc.free nulls each field, so a second close() is still a safe no-op) + jsonLexer = Misc.free(jsonLexer); + plainClient = Misc.free(plainClient); + tlsClient = Misc.free(tlsClient); + } finally { + lock.unlock(); + } + } + + /** + * @return {@code "Bearer " + signIn()}, ready to use as the value of an HTTP + * {@code Authorization} header. + */ + public String getAuthorizationHeaderValue() { + return "Bearer " + signIn(); + } + + /** + * Like {@link #signIn()} but never starts the interactive device flow, never prompts, and never waits + * on interactive input: returns the cached token while valid, silently refreshes when a refresh token is + * available, otherwise throws. Designed for the request/flush path of a long-lived client, for example + * {@code Sender.builder(...).httpTokenProvider(auth::getToken)}, where an interactive prompt is + * inappropriate. Call {@link #signIn()} once to sign in first. + *

+ * It does not wait behind an interactive {@link #signIn()} running on another thread (which would stall + * the flush for the whole device-code lifetime): if such a sign-in holds the lock it fails fast, and the + * caller should retry once the sign-in completes. It does, however, wait briefly behind another thread's + * quick cached read or silent refresh rather than fail every concurrent caller sharing this instance on + * each token refresh - the {@code HttpTokenProvider} contract permits that bounded wait, capped here at + * SIX times {@link Builder#httpTimeoutMillis(int)} (three minutes at the 30s default), which is the + * holder's own worst case: a silent refresh under the lock runs a TCP connect, a TLS handshake, a send, + * an await and a body parse, each separately bounded by that timeout, so a peer waiting only one would + * fail every concurrent caller behind a refresh that was going to succeed. Size flush backpressure + * against the six-times figure, not + * against {@code httpTimeoutMillis} itself. It still fails fast the moment an interactive sign-in or + * {@link #close()} begins meanwhile. It is not, otherwise, instantaneous - when the cached + * token has expired it makes one synchronous refresh round-trip to the token endpoint (and, with a + * coordinating {@link TokenStore}, may first wait to acquire the store's per-identity lock before that + * round-trip). For {@link FileTokenStore} the CROSS-process file lock is bounded to a few seconds and then + * proceeds without it; but the IN-process lock that serializes two instances sharing one identity in the + * same JVM (an ILP {@code Sender} and a {@code QwpQueryClient}, say) is not time-bounded, so such a + * concurrent caller instead waits out the peer's whole refresh - itself bounded only by the OS connect + * stall described next, not by a few seconds. + * The send, response wait and body parse of that round-trip are each bounded by + * {@link Builder#httpTimeoutMillis(int)} (30s by default); the connection phase that precedes them - DNS + * the TCP connect and the TLS handshake - is bounded by httpTimeoutMillis as well, since httpConfig() + * derives the connect timeout and the handshake budget from it. Only DNS resolution is left to the OS, so + * an unreachable (black-holed) token endpoint stalls this refresh for about the configured timeout rather + * than the OS TCP-connect timeout. That is the "quick silent refresh" the {@code HttpTokenProvider} + * contract permits on the flush path, not an unbounded interactive wait. + * + * @return a non-null, non-empty token + * @throws OidcAuthException if no token has been obtained yet, if the cached token expired and could + * not be refreshed without an interactive sign-in, if an interactive sign-in is + * in progress on another thread, or if a concurrent refresh did not complete in time + */ + public String getToken() { + throwIfClosed(); + acquireForGetToken(); + try { + throwIfClosed(); + maybeLoadFromStore(); + final String cachedToken = groupsInToken ? idToken : accessToken; + if (cachedToken != null && System.currentTimeMillis() < expiresAtMillis - effectiveSkewMillis()) { + return cachedToken; + } + // The served-kind token is absent or expired. Try a silent refresh whenever a refresh token is + // available - including the case where a prior grant returned only the OTHER kind, leaving the served + // kind null: a refresh may yield the served kind and avoid forcing an interactive sign-in. selectToken() + // reports a clear error if the refresh still did not produce the kind the server expects. + // Back off after a failed refresh instead of re-attempting on every call. getToken() runs once + // per ILP flush and once per (re)connect, and a producer retrying rows calls it in a tight loop, + // so a revoked token or an unreachable IdP otherwise meant one full token-endpoint round trip per + // attempt - a request flood at the provider, and a producer blocked for each round trip. The + // failure is still reported on every call; only the network attempt is rate-limited. + // A caller whose thread is already interrupted is cancelled, and a silent refresh is a network + // round trip - exactly the work a cancellation is trying to stop. Decline it here, once, for + // three reasons the old shape got wrong: + // + // - the only interrupt guard was inside FileTokenStore.inLock, so this held ONLY when a token + // store was configured. Without one, tryRefreshCoordinated() went straight to tryRefresh() + // and POSTed to the token endpoint on a cancelled thread. + // - when the store's guard did decline, the fall-through reported "the cached token expired + // and could not be refreshed without an interactive sign-in; call signIn()". The endpoint was + // reachable and the lock free; the caller's own interrupt was the reason. That sends a user + // to re-authenticate over a credential that is fine. + // - it then latched the refresh back-off below, so one interrupt-carrying caller suppressed + // the next five seconds of legitimate refreshes for every thread sharing this instance. + // + // isInterrupted(), never interrupted(): the flag is the caller's cancellation signal and must + // survive this call, exactly as FileTokenStore.load()/save() preserve it. + if (refreshToken != null && Thread.currentThread().isInterrupted()) { + throw new OidcAuthException("the calling thread is interrupted, so no silent token refresh was attempted; retry on an uninterrupted thread"); + } + // Arm the latch ONLY when a refresh was actually attempted and failed. Stamping it on a call + // that the back-off itself skipped slides the window forward by one call every time, so it + // never expires for a caller that returns faster than MIN_REFRESH_RETRY_INTERVAL_MILLIS - and + // getToken() runs once per ILP flush, at a default auto-flush interval of one second. One + // transient refresh failure then wedges the sender for the life of the process, long after the + // identity provider recovered, which is a circuit breaker rather than the stampede guard this + // is documented to be. maybeLoadFromStore() arms its sibling back-off inside the catch for the + // same reason: only a real attempt may re-arm. + if (refreshToken != null && !isRefreshBackedOff()) { + final int refreshResult = tryRefreshCoordinated(); + if (refreshResult == REFRESH_SUCCEEDED) { + refreshFailedAtMillis = 0; + return selectToken(); + } + // Whether the action ran is the only sound discriminator here. The interrupt flag is not: + // Future.cancel(true), PoolHousekeeper.stop(), or any application cancellation can set it + // WHILE a refresh that really reached the IdP is failing. Reading that case as an abandoned + // store wait produced a false "no attempt" diagnostic and skipped the stampede latch; with no + // store configured it even named a lock that did not exist. The action-entry marker inside + // tryRefreshCoordinated() tells the two outcomes apart without consuming the caller's flag. + if (refreshResult == REFRESH_NOT_ATTEMPTED) { + if (Thread.currentThread().isInterrupted()) { + throw new OidcAuthException("the calling thread was interrupted while waiting for the token store lock, so no silent token refresh was attempted; retry on an uninterrupted thread"); + } + // A conforming store only declines without running action on an interrupt and preserves + // that flag. Keep the no-attempt outcome accurate even for a custom store that violates + // the flag half of the contract, and do not arm a back-off for work that never happened. + throw new OidcAuthException("the token store returned without running the silent token refresh; retry shortly"); + } + // REFRESH_FAILED means the endpoint was actually attempted, so arm the stampede guard even + // when an unrelated cancellation set this thread's interrupt flag during that round trip. + refreshFailedAtMillis = System.currentTimeMillis(); + } + if (cachedToken != null) { + throw new OidcAuthException("the cached token expired and could not be refreshed without an interactive sign-in; call signIn() to sign in again"); + } + throw new OidcAuthException("no token has been obtained yet; call signIn() to sign in before using getToken()"); + } finally { + lock.unlock(); + } + } + + /** + * Returns a valid token to present to QuestDB: the cached token while still valid, otherwise a + * silent refresh when possible, otherwise the interactive device flow. The token is the id token + * when the server expects groups encoded in the token, the access token otherwise. + * + * @return a non-null, non-empty token + * @throws OidcAuthException if the interactive flow fails, times out, the identity provider does not + * return the expected token, or the calling thread carries an interrupt and + * no valid cached token is available - a cancelled caller is not sent through + * a silent refresh or a device flow, both of which are network work + */ + public String signIn() { + lock.lock(); + try { + throwIfClosed(); + // signIn() is an explicit user action, and it is about to spend a whole interactive device flow - + // so it is never the caller the store-read back-off exists to throttle. Clear that back-off, for + // the same reason the refresh back-off is cleared further down: a store that has become readable + // again must be re-read here, rather than have a human sent through the device flow over a refresh + // token that is sitting on disk. + nextStoreLoadAttemptMillis = 0; + storeLoadRetryIntervalMillis = 0; + maybeLoadFromStore(); + // only the kind of token signIn() actually serves counts as a cache hit; a grant that + // returned the other kind (access token when the server wants the id token, or vice versa) + // leaves the served token null, so fall through rather than report the unusable grant as valid + // and have selectToken() throw on this and every later call + final String cachedToken = groupsInToken ? idToken : accessToken; + if (cachedToken != null && System.currentTimeMillis() < expiresAtMillis - effectiveSkewMillis()) { + return cachedToken; + } + // A cached token needs no network and was served above whatever the caller's state; everything + // from here on is a network round trip, which is the work a cancellation is trying to stop. So + // decline it, for the same reason and with the same test getToken() applies further up. + // + // Checked HERE rather than left to the store, because leaving it there made the outcome depend + // on whether a TokenStore was configured, and the two branches were wrong in OPPOSITE + // directions. With no store, tryRefreshCoordinated() went straight to tryRefresh() and POSTed to + // the token endpoint on a cancelled thread. With a FileTokenStore, inLock declined the carried + // interrupt by returning false - which reads here as "the refresh failed", so signIn() skipped a + // refresh it could have completed and started the DEVICE FLOW instead: a human prompt and a poll + // loop that is far more work than the round trip just declined, and that runs to the device-code + // lifetime (up to MAX_DEVICE_CODE_TTL_SECONDS) because sleepBetweenPolls uses Os.sleep, which + // ignores interrupts. A caller who cancelled got a browser prompt and a thread parked for half + // an hour. + // + // isInterrupted(), never interrupted(): the flag is the caller's cancellation signal and must + // survive this call, exactly as getToken() and FileTokenStore.load()/save() preserve it. + if (Thread.currentThread().isInterrupted()) { + throw new OidcAuthException("the calling thread is interrupted, so no sign-in was attempted; retry on an uninterrupted thread"); + } + // Spend a silent refresh before prompting, whenever a refresh token is available - the same rule + // getToken() applies. That deliberately includes a null served kind: a restored entry whose grant + // only ever produced the OTHER kind still carries a usable refresh token, and one round-trip + // beats sending a human back through the device flow. Gating this on cachedToken != null, as it + // used to, meant such an entry always re-prompted. A refresh that does not yield the served kind + // returns false and falls straight through to the flow below, so this costs at most one wasted + // request and cannot loop. + // No back-off here, and the latch is cleared either way: signIn() is an explicit user action, it + // falls through to the interactive flow when the refresh fails, and it is exactly the call a user + // makes to recover from the failure getToken() is backing off from. + refreshFailedAtMillis = 0; + if (refreshToken != null && tryRefreshCoordinated() == REFRESH_SUCCEEDED) { + return selectToken(); + } + // Re-check the flag, because the guard on entry only covers an interrupt the caller ARRIVED with. + // tryRefreshCoordinated() above is a network round trip - up to six times httpTimeoutMillis plus + // an OS connect stall - and a cancellation landing inside it is the common case, not a narrow + // race: it is precisely when a refresh is failing that a caller gives up. Proceeding would then + // launch a browser and park for the device-code lifetime on a thread whose owner has already + // asked it to stop. + throwIfInterrupted("the calling thread was interrupted before the interactive sign-in started"); + // flag the interactive phase so a concurrent getToken() fails fast (rather than waiting behind this + // for the whole device-code lifetime); it still waits behind the cheap cache/refresh work above + interactiveSignInInProgress = true; + try { + runDeviceFlow(); + } finally { + interactiveSignInInProgress = false; + } + return selectToken(); + } finally { + lock.unlock(); + } + } + + private static String appendSettingsPath(String basePath) { + String trimmed = basePath; + while (trimmed.length() > 1 && trimmed.charAt(trimmed.length() - 1) == '/') { + trimmed = trimmed.substring(0, trimmed.length() - 1); + } + return "/".equals(trimmed) ? "/settings" : trimmed + "/settings"; + } + + private static int boundedSeconds(int value, int defaultValue, int maxValue) { + if (value <= 0) { + return defaultValue; + } + return Math.min(value, maxValue); + } + + private static String canonicalEndpoint(Endpoint endpoint) { + // scheme and host lower-cased, port explicit, path verbatim: a stable rendering that hashes to the + // same TokenStoreKey across processes and language clients sharing this identity + return (endpoint.isTls ? "https://" : "http://") + + endpoint.host.toLowerCase(Locale.ROOT) + ':' + endpoint.port + endpoint.path; + } + + private static String[] decodePathSegments(String path) { + // Repeatedly percent-decode (a server or proxy may unescape more than once, so %252e%252e -> .. ) + // and fold backslash to slash (some proxies do), then split into segments. Comparing these decoded + // segments, not the raw wire string, means an encoding the server later undoes cannot hide a "..". + String decoded = path; + for (int i = 0; i < 10; i++) { // bounded; a real path needs 0-1 passes + String next = percentDecodeOnce(decoded); + if (next.equals(decoded)) { + break; + } + decoded = next; + } + return decoded.replace('\\', '/').split("/", -1); + } + + private static ClientTlsConfiguration defaultTlsConfig() { + return new ClientTlsConfiguration(null, null, ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL); + } + + private static boolean discardBody(Response body, int timeoutMillis) { + // best-effort drain after a parse failure to keep the keep-alive connection usable; bounded like + // parseBody so a hostile server cannot wedge the thread here either. Returns true only when the body + // was fully drained (so the connection can be reused); returns false when the drain stopped early - + // on the deadline, the byte cap, or a transport error - leaving unconsumed bytes, so the caller must + // drop the connection rather than parse this response's leftovers on the next request. + final long deadlineNanos = System.nanoTime() + timeoutMillis * 1_000_000L; + long totalBytes = 0; + try { + while (true) { + final long remainingNanos = deadlineNanos - System.nanoTime(); + if (remainingNanos <= 0) { + return false; + } + Fragment fragment = body.recv((int) Math.max(1, Math.min(remainingNanos / 1_000_000L, Integer.MAX_VALUE))); + if (fragment == null) { + return true; + } + totalBytes += fragment.hi() - fragment.lo(); + if (totalBytes > MAX_RESPONSE_BODY_BYTES) { + return false; + } + } + } catch (HttpClientException ignore) { + return false; + } + } + + private static void discoverFromIdp(String issuer, ClientTlsConfiguration tlsConfig, boolean allowInsecureTransport, WellKnownDiscoveryParser parser) { + // the issuer is pinned out of band (the caller guarantees it is non-null), so the server cannot choose + // where discovery - and the credential POSTs it resolves - are aimed + // A trailing slash remains part of the issuer identifier and must survive the exact comparison below. + // Remove it only from the base used to construct the well-known request URL, as required by OIDC + // Discovery section 4.1. + String discoveryBase = issuer; + while (discoveryBase.length() > 1 && discoveryBase.charAt(discoveryBase.length() - 1) == '/') { + discoveryBase = discoveryBase.substring(0, discoveryBase.length() - 1); + } + String url = discoveryBase + WELL_KNOWN_OPENID_CONFIGURATION_PATH; + Endpoint endpoint = Endpoint.parse(url); + requireSecureIdpEndpoint(endpoint, "OIDC issuer", url, allowInsecureTransport); + fetchJson(endpoint, endpoint.path, tlsConfig, parser, + "could not reach the identity provider to discover OIDC settings", + "could not parse the identity provider discovery document", + "the identity provider did not return an OIDC discovery document"); + // OpenID Connect Discovery requires code-point-for-code-point equality with the pinned issuer whose + // configuration this request retrieves: it is an identifier comparison, not an origin comparison. Do + // not case-fold the host, normalize Unicode, remove a default port, or otherwise turn a wrong-tenant + // document into a match. Validate before fromQuestDB copies either discovered endpoint out of the parser. + if (parser.issuer.length() == 0) { + throw new OidcAuthException() + .put("the identity provider discovery document does not contain the required issuer; ") + .put("refusing to use its endpoints"); + } + if (!Chars.equals(issuer, parser.issuer)) { + // Do not echo parser.issuer: it came from an untrusted response and may contain display-control + // characters. The caller already knows which issuer it pinned. + throw new OidcAuthException() + .put("the identity provider discovery document issuer does not exactly match the pinned issuer; ") + .put("refusing to use its endpoints"); + } + } + + private static void discoverSettings(Endpoint server, ClientTlsConfiguration tlsConfig, SettingsDiscoveryParser parser) { + fetchJson(server, appendSettingsPath(server.path), tlsConfig, parser, + "could not reach the QuestDB server to discover OIDC settings", + "could not parse the QuestDB /settings response", + "the QuestDB server did not return its settings"); + } + + private static OidcAuthException endpointNotUnderIssuer(String label, String url, String issuer) { + return new OidcAuthException() + .put("the OIDC ").put(label).put(" advertised by the QuestDB /settings response (").put(url) + .put(") is not under the pinned issuer (").put(issuer).put("); refusing to send credentials to ") + .put("an endpoint outside the trusted issuer, for example a different realm on the same host; ") + .put("if the identity provider places its endpoints outside the issuer path, configure them ") + .put("explicitly with OidcDeviceAuth.builder()"); + } + + private static OidcAuthException endpointOriginNotPinned(String label, String url, String pinOrigin) { + return new OidcAuthException() + .put("the OIDC ").put(label).put(" advertised by the QuestDB /settings response (").put(url) + .put(") is not on the pinned identity-provider origin (").put(pinOrigin).put("); refusing to send ") + .put("credentials to an endpoint outside the trusted issuer. If the identity provider hosts its ") + .put("endpoints on a different origin than its issuer, configure them explicitly with ") + .put("OidcDeviceAuth.builder()"); + } + + private static boolean endpointPathHasEncodedSeparator(String rawEndpointPath) { + // A real OIDC endpoint path is plain ASCII with no percent-encoding and no backslash, so reject either + // outright rather than try to out-decode the server. Percent-encoding is exactly where a tampered + // /settings hides a path separator ('/', '\') or a '..' that only surfaces once the server unescapes - + // and not only the forms this client's byte-oriented percentDecodeOnce resolves (%2f, %5c, %25, %252f, + // %2%66), but ones it deliberately does NOT: an overlong-UTF-8 %c0%ae or %e0%80%ae, or an IIS-style + // %u002e, which a permissive server decodes to '/' or '.' yet a single-byte decode leaves as high bytes + // or literal text - so they would sail past the segment scan in isEndpointUnderIssuerPath and, sitting + // past the issuer prefix, slip the scope. A literal backslash likewise folds to '/' on some proxies. + // Failing closed on any '%' or '\' keeps the issuer-path scope airtight against every encoding trick; a + // provider that genuinely percent-encodes its endpoint path must be configured explicitly with + // OidcDeviceAuth.builder(), which pins the origin only. + for (int i = 0, n = rawEndpointPath.length(); i < n; i++) { + char c = rawEndpointPath.charAt(i); + if (c == '%' || c == '\\') { + return true; + } + } + return false; + } + + private static void fetchJson(Endpoint endpoint, String path, ClientTlsConfiguration tlsConfig, JsonParser parser, String reachError, String parseError, String statusError) { + HttpClient client = endpoint.isTls + ? HttpClientFactory.newTlsInstance(DISCOVERY_HTTP_CONFIG, tlsConfig) + : HttpClientFactory.newPlainTextInstance(DISCOVERY_HTTP_CONFIG); + // allocate the native lexer inside the try: new JsonLexer mallocs and can throw (native OOM), and + // the client is already allocated, so a throw before the try is entered would skip the finally and + // leak the client's native buffers + JsonLexer lexer = null; + try { + lexer = new JsonLexer(JSON_LEXER_CACHE_SIZE, JSON_LEXER_MAX_VALUE_BYTES); + HttpClient.Request request = client.newRequest(endpoint.host, endpoint.port) + .GET() + .url(path) + .header("Accept", "application/json") + .header("User-Agent", USER_AGENT); + HttpClient.ResponseHeaders response = request.send(DEFAULT_HTTP_TIMEOUT_MILLIS); + response.await(DEFAULT_HTTP_TIMEOUT_MILLIS); + Response body = response.getResponse(); + // A discovery document decides WHERE the user signs in and where the refresh token is POSTed, + // so it must be read only out of a response that actually claims to carry one. Parsing + // regardless of status let an error page supply that configuration: a 500 from /settings or a + // 404 from .well-known whose body happens to hold the right keys - an error envelope, a proxy's + // branded page, a captive portal, a tenant-not-found stub - constructed a working instance + // pointed wherever those keys said. The token and device-authorization paths already gate on + // status; this one did not. + requireSuccessStatus(client, response, body, statusError); + // parseBody enforces an elapsed-time deadline and a byte cap so an untrusted server cannot wedge + // discovery, and its parseLast rejects a truncated document + parseBody(body, lexer, parser, DEFAULT_HTTP_TIMEOUT_MILLIS); + } catch (HttpClientException | HttpException e) { + // HttpException covers a malformed or oversized RESPONSE HEAD rejected by HttpHeaderParser (see + // postForm). It is a sibling of HttpClientException, not a subclass, so it escaped both catches + // here and left fromQuestDB throwing a type its own javadoc does not name - past every caller's + // catch (OidcAuthException) degrade handler. Discovery reaching an unusable response is the same + // outcome either way, so report it the same way. + throw new OidcAuthException(e).put(reachError); + } catch (JsonException e) { + throw new OidcAuthException(e).put(parseError); + } finally { + Misc.free(lexer); + Misc.free(client); + } + } + + private static boolean hasOnlyTokenChars(CharSequence token) { + for (int i = 0, n = token.length(); i < n; i++) { + char c = token.charAt(i); + if (c < 0x20 || c > 0x7e) { + return false; + } + } + return true; + } + + private static int hexValue(char c) { + if (c >= '0' && c <= '9') { + return c - '0'; + } + if (c >= 'a' && c <= 'f') { + return c - 'a' + 10; + } + if (c >= 'A' && c <= 'F') { + return c - 'A' + 10; + } + return -1; + } + + /** + * The HTTP transport budgets for a client this class owns, all derived from one timeout. + *

+ * DefaultHttpClientConfiguration answers 0 for the connect timeout and 600s for the request timeout, and + * HttpClient reads BOTH of those on the connection path: it leaves the TCP connect to the OS when the + * connect timeout is 0, and it sizes the TLS handshake as {@code connectTimeout > 0 ? connectTimeout : + * defaultTimeout}. Taking the defaults therefore gave the handshake alone a 600s budget -- the whole of + * FileTokenStore's DEFAULT_LOCK_STALE_MILLIS -- derived from nothing the caller set, so neither + * MAX_HTTP_TIMEOUT_MILLIS nor Builder.build()'s lockStaleMillis floor constrained it. + *

+ * That matters beyond a slow request. A refresh runs inside the store's cross-process lock, whose file + * is stamped once at creation and never re-stamped, so a hold that outruns the staleness window is + * judged abandoned and stolen by a peer. Two holders then POST the same rotating refresh token, and an + * identity provider with reuse detection answers by revoking the whole family. Deriving both budgets + * from httpTimeoutMillis is what makes the "up to LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE x httpTimeoutMillis" + * figure -- which the lock-stale floor, acquireForGetToken's wait cap and getToken()'s javadoc all + * quote -- actually true of the code. + */ + private static HttpClientConfiguration httpConfig(final int timeoutMillis) { + return new DefaultHttpClientConfiguration() { + @Override + public int getConnectTimeout() { + return timeoutMillis; + } + + @Override + public int getTimeout() { + return timeoutMillis; + } + }; + } + + private static boolean isDottedIpv4(String host) { + // validate a dotted IPv4 literal (four 0-255 octets) without DNS, so a hostname merely starting + // with "127." is not mistaken for the loopback block + int octets = 1; + int value = 0; + int digits = 0; + for (int i = 0, n = host.length(); i < n; i++) { + char c = host.charAt(i); + if (c == '.') { + if (digits == 0 || value > 255) { + return false; + } + octets++; + value = 0; + digits = 0; + } else if (c >= '0' && c <= '9') { + value = value * 10 + (c - '0'); + if (++digits > 3) { + return false; + } + } else { + return false; + } + } + return octets == 4 && digits > 0 && value <= 255; + } + + private static boolean isEndpointUnderIssuerPath(String endpointUrl, String issuer) { + // The endpoint's path must be the issuer's path or a sub-path of it, compared segment by segment (so + // /realms/prod does not match /realms/production). A root issuer (no path) constrains the origin only. + // This stops a tampered /settings from redirecting credentials to a different tenant on a path-based + // multi-tenant identity provider (Keycloak issuers are https://host/realms/), which the origin + // check alone cannot catch. Mirrors the Python client. + String basePath = pathOnly(issuer); + int baseEnd = basePath.length(); + while (baseEnd > 0 && basePath.charAt(baseEnd - 1) == '/') { + baseEnd--; // trailing slashes do not add a path segment + } + if (baseEnd == 0) { + return true; // root issuer: origin-only, every path is under it + } + String[] baseSegs = decodePathSegments(basePath.substring(0, baseEnd)); + String rawEndpointPath = pathOnly(endpointUrl); + // A real OIDC endpoint path never encodes a path separator or uses a backslash; reject either before + // the segment comparison, since decodePathSegments resolves them and would split one path segment in + // two, letting .../realms/acme%2fevil/token (or its split/backslash forms) slip the issuer-path scope. + if (endpointPathHasEncodedSeparator(rawEndpointPath)) { + return false; + } + String[] endpointSegs = decodePathSegments(rawEndpointPath); + // a "." or ".." segment is rejected outright: the server normalizes it away, so a naive prefix test + // would pass /realms/acme/../evil/token yet it resolves to a different realm + for (int i = 0; i < endpointSegs.length; i++) { + // strip an RFC 3986 ";matrix" parameter suffix before the dot-segment test: a server or proxy that + // drops matrix params resolves "..;" (or "..;x") to "..", so /realms/acme/..;/evil/token would + // otherwise slip the issuer-path pin to a sibling realm. decodePathSegments already percent-decoded, + // so a "%3b"-encoded ";" is a literal ";" here too. + String seg = endpointSegs[i]; + int semi = seg.indexOf(';'); + if (semi >= 0) { + seg = seg.substring(0, semi); + } + if (".".equals(seg) || "..".equals(seg)) { + return false; + } + } + if (endpointSegs.length < baseSegs.length) { + return false; + } + for (int i = 0; i < baseSegs.length; i++) { + if (!baseSegs[i].equals(endpointSegs[i])) { + return false; + } + } + return true; + } + + private static boolean isLoopbackHost(String host) { + // loopback traffic never leaves the host, so a plaintext fetch to it has no network interception + // risk; match the whole IPv4 127.0.0.0/8 block and the name "localhost" + if (host == null) { + return false; + } + // an address literal needs no resolution: it IS the address, and nobody can point it elsewhere + if (host.startsWith("127.") && isDottedIpv4(host)) { + return true; + } + if (!host.equalsIgnoreCase("localhost")) { + return false; + } + // "localhost" is a NAME. RFC 6761 says it must resolve to loopback, and every normal host honours + // that - but a minimal image with no /etc/hosts entry leaves it to DNS, and this exemption is + // precisely what allows the device code and the refresh token to travel in cleartext. Confirm what + // it actually resolves to rather than trusting the spelling, and require EVERY answer to be + // loopback: one non-loopback address is enough to send the credential off the machine. + try { + final InetAddress[] resolved = InetAddress.getAllByName(host); + if (resolved.length == 0) { + return false; + } + for (int i = 0; i < resolved.length; i++) { + if (!resolved[i].isLoopbackAddress()) { + return false; + } + } + return true; + } catch (UnknownHostException e) { + // fail CLOSED: the caller then requires https, which is never the less safe answer + return false; + } + } + + private static boolean isSameOrigin(Endpoint a, Endpoint b) { + // scheme (via isTls), host and port - the security origin; path is deliberately not compared, the + // token and device endpoints legitimately differ in path on one authorization server. Endpoint.parse + // rejects a non-ASCII host, so this equalsIgnoreCase host compare only ever folds ASCII case - no + // non-ASCII homoglyph can fold onto a pinned issuer host here. + return a.isTls == b.isTls && a.port == b.port && a.host.equalsIgnoreCase(b.host); + } + + private static String originOf(Endpoint endpoint) { + return (endpoint.isTls ? "https://" : "http://") + endpoint.host + ':' + endpoint.port; + } + + private static void parseBody(Response body, JsonLexer lexer, JsonParser parser, int timeoutMillis) throws JsonException { + // read and parse the whole body, bounded by an elapsed-time deadline and a cumulative byte cap, so a + // hostile or stalled server cannot wedge the thread by dribbling or endlessly streaming. nanoTime, + // not the wall clock: an NTP step or an operator setting the date back must not stretch this bound, + // which is the only thing standing between a dribbling identity provider and a wedged caller. + final long deadlineNanos = System.nanoTime() + timeoutMillis * 1_000_000L; + long totalBytes = 0; + while (true) { + final long remainingNanos = deadlineNanos - System.nanoTime(); + if (remainingNanos <= 0) { + throw new HttpClientException("timed out reading the identity provider response body"); + } + Fragment fragment = body.recv((int) Math.max(1, Math.min(remainingNanos / 1_000_000L, Integer.MAX_VALUE))); + if (fragment == null) { + break; + } + totalBytes += fragment.hi() - fragment.lo(); + if (totalBytes > MAX_RESPONSE_BODY_BYTES) { + throw new HttpClientException("the identity provider response body exceeded the size limit"); + } + lexer.parse(fragment.lo(), fragment.hi(), parser); + } + lexer.parseLast(); // reject a truncated body (unterminated string/object) + } + + private static int parseIntOrZero(CharSequence value) { + try { + return Numbers.parseInt(value); + } catch (NumericException e) { + return 0; + } + } + + private static String pathOnly(String url) { + // the path component only; Endpoint.parse rejects a url carrying a ?query or #fragment up front, so the + // returned path never contains one. A ;matrix parameter, by contrast, stays part of the path, so a + // traversal hidden in it (.../token;..%2f..) is still scanned by the issuer-path check. + return Endpoint.parse(url).path; + } + + private static String percentDecodeOnce(String s) { + int pct = s.indexOf('%'); + if (pct < 0) { + return s; // nothing encoded + } + StringSink sink = new StringSink(); + sink.put(s, 0, pct); + for (int i = pct, n = s.length(); i < n; ) { + char c = s.charAt(i); + if (c == '%' && i + 2 < n) { + int hi = hexValue(s.charAt(i + 1)); + int lo = hexValue(s.charAt(i + 2)); + if (hi >= 0 && lo >= 0) { + sink.put((char) ((hi << 4) | lo)); + i += 3; + continue; + } + } + sink.put(c); + i++; + } + return sink.toString(); + } + + private static void putNonNull(StringSink sink, CharSequence tag) { + // clear before storing so a repeated key replaces, not concatenates onto, the previous value; a + // JSON null arrives from the lexer as the literal "null", so treat it as absent rather than store + // the 4-char string "null" as a token, error code, endpoint or user code + sink.clear(); + if (!Chars.equals("null", tag)) { + sink.put(tag); + } + } + + private static void requireSecureIdpEndpoint(Endpoint endpoint, String label, String url, boolean allowInsecureTransport) { + // https is always fine; plaintext http is allowed only to a loopback host, where the request never + // leaves the machine. allowInsecureTransport relaxes the QuestDB link but never the identity + // provider: the device code and refresh token must not cross the network in cleartext (matching + // the Python client) + if (endpoint.isTls || isLoopbackHost(endpoint.host)) { + return; + } + OidcAuthException ex = new OidcAuthException() + .put("the ").put(label).put(" uses insecure http, which would send the device code and ") + .put("refresh token across the network in cleartext; use an https url"); + if (allowInsecureTransport) { + ex.put(" (allowInsecureTransport relaxes only the QuestDB connection, not the identity provider endpoints)"); + } + throw ex.put(" [url=").put(url).put(']'); + } + private static void requireSecureTransport(boolean isTls, String label, String url) { + if (!isTls) { + throw new OidcAuthException() + .put("the ").put(label).put(" uses insecure http, which exposes the OIDC sign-in to network ") + .put("attackers; use an https url, or call allowInsecureTransport(true) to override [url=").put(url).put(']'); + } + } + /** + * Requires a well-formed 2xx status before a discovery body is trusted as configuration. + *

+ * The status is validated to be exactly three bare digits BEFORE any of it is echoed: the header parser + * copies the status-line token verbatim apart from SP/CR/LF, so a non-digit byte means a malformed or + * hostile status line that must not splice ESC or other control bytes into a message, a log or a + * terminal. A short all-digit status ({@code 2}, {@code 5}) is malformed too, and must not be read as a + * 2xx class by its leading digit. Mirrors the check {@code readResponse} applies on the token path. + *

+ * On rejection the body is drained within the usual bound so the keep-alive connection stays usable; a + * body too large or too slow to drain leaves unconsumed bytes, so the connection is dropped instead of + * mis-framing the next request's response. + */ + private static void requireSuccessStatus( + HttpClient client, + HttpClient.ResponseHeaders response, + Response body, + String statusError + ) { + DirectUtf8Sequence statusCode = response.getStatusCode(); + StringSink status = new StringSink(); + boolean malformed = statusCode == null; + if (!malformed) { + CharSequence raw = statusCode.asAsciiCharSequence(); + for (int i = 0, n = raw.length(); i < n; i++) { + char c = raw.charAt(i); + if (c < '0' || c > '9') { + malformed = true; + break; + } + status.put(c); + } + malformed |= status.length() != 3; + } + if (malformed) { + if (!discardBody(body, DEFAULT_HTTP_TIMEOUT_MILLIS)) { + client.disconnect(); + } + throw new OidcAuthException().put(statusError) + .put("; the response carried a malformed HTTP status code"); + } + if (status.charAt(0) != '2') { + if (!discardBody(body, DEFAULT_HTTP_TIMEOUT_MILLIS)) { + client.disconnect(); + } + // the status is proven to be bare digits, so echoing it cannot smuggle control bytes + throw new OidcAuthException().put(statusError) + .put(" [httpStatus=").put(status).put(']'); + } + } + + + + // package-private, not private: FileTokenStore needs the same treatment for the operator-supplied path + // its IO errors embed, and a second copy of this walk in the same package would be the thing to avoid. + static String sanitizeForDisplay(String value) { + if (value == null) { + return null; + } + final int n = value.length(); + int firstUnsafe = -1; + for (int i = 0; i < n; ) { + final int cp = value.codePointAt(i); + if (OidcAuthException.isUnsafeForDisplay(cp)) { + firstUnsafe = i; + break; + } + i += Character.charCount(cp); + } + if (firstUnsafe < 0) { + return value; // common case: nothing to strip + } + // an attacker-influenced device-auth field can smuggle in terminal-spoofing characters - ANSI + // escapes, CR/LF, or bidi/zero-width formatting (including supplementary-plane "tag" chars that + // arrive as surrogate pairs) - that reorder or hide text, so strip them per code point; else a + // right-to-left override could make the verification URL a human reads differ from the one their + // browser opens + StringSink sink = new StringSink(); + sink.put(value, 0, firstUnsafe); + for (int i = firstUnsafe; i < n; ) { + final int cp = value.codePointAt(i); + final int count = Character.charCount(cp); + if (!OidcAuthException.isUnsafeForDisplay(cp)) { + sink.put(value, i, i + count); + } + i += count; + } + return sink.toString(); + } + + private static boolean settingsChannelIsPlaintext(Endpoint server) { + // /settings over plaintext http to a non-loopback host is MITM-able (only possible with + // allowInsecureTransport; the default rejects it), so its advertised endpoints must not be trusted + // to route credentials without an out-of-band pin + return !server.isTls && !isLoopbackHost(server.host); + } + + private static String urlEncode(String value) { + try { + // the Charset overload is Java 10; the client targets Java 8, so use the String-charset form + return URLEncoder.encode(value, "UTF-8"); + } catch (UnsupportedEncodingException e) { + // UTF-8 is guaranteed present on every JVM, so this is unreachable; rethrow defensively + throw new OidcAuthException(e).put("UTF-8 encoding is not supported"); + } + } + + private static void validateEndpointOrigins(Endpoint tokenEndpoint, Endpoint deviceAuthorizationEndpoint, Endpoint issuer) { + // the device code and long-lived refresh token are POSTed to the device authorization and token + // endpoints. RFC 8628 co-locates them on one authorization server, so reject a config that splits + // them across origins (a tampered /settings or discovery document siphoning one off) on every + // construction path. The issuer-origin pin here is the explicit builder().issuer() opt-in - a sanity + // check that user-supplied endpoints sit on the pinned origin; a provider hosting its endpoints off + // the issuer origin must then be configured without an issuer. fromQuestDB pins differently: it + // origin-pins only the /settings-advertised endpoints itself (a discovered endpoint is trusted), so + // it passes no issuer here and relies on this method only for the co-location check. + if (!isSameOrigin(tokenEndpoint, deviceAuthorizationEndpoint)) { + throw new OidcAuthException() + .put("the OIDC token and device authorization endpoints are on different origins (") + .put(originOf(tokenEndpoint)).put(" vs ").put(originOf(deviceAuthorizationEndpoint)) + .put("); refusing to send credentials. This indicates a misconfigured or tampered OIDC configuration"); + } + if (issuer != null) { + if (!isSameOrigin(tokenEndpoint, issuer)) { + throw new OidcAuthException() + .put("the OIDC token endpoint origin (").put(originOf(tokenEndpoint)) + .put(") does not match the issuer origin (").put(originOf(issuer)) + .put("); refusing to send credentials to an endpoint outside the trusted issuer"); + } + if (!isSameOrigin(deviceAuthorizationEndpoint, issuer)) { + throw new OidcAuthException() + .put("the OIDC device authorization endpoint origin (").put(originOf(deviceAuthorizationEndpoint)) + .put(") does not match the issuer origin (").put(originOf(issuer)) + .put("); refusing to send credentials to an endpoint outside the trusted issuer"); + } + } + } + + private static void validateTokenChars(CharSequence token, String tokenName) { + // The selected token goes verbatim into the "Authorization: Bearer " header sent to the + // trusted QuestDB server and into the PG-wire _sso password. A CR/LF or other control char would + // break out of the header into the request line (the lexer now decodes a \r or \n escape in the + // provider's response into a real control byte), and a non-ASCII char is silently truncated to one + // byte by the ASCII header writer. A real OAuth token is printable ASCII, so reject anything else + // rather than route a tampered or corrupt credential onto the wire. Token bytes are never embedded + // in the message: they are the secret this class protects. (A blank/whitespace-only served token is + // handled by storeTokens, which caches it as absent so it is never served, rather than rejected here - + // an EMPTY served kind is the legitimate "the grant returned the other kind" case selectToken handles.) + if (!hasOnlyTokenChars(token)) { + throw new OidcAuthException() + .put("the identity provider returned an ").put(tokenName) + .put(" containing a disallowed control or non-ASCII character; refusing to use it as a credential"); + } + } + + private void acquireForGetToken() { + throwIfClosed(); + // Uncontended fast path: a plain CAS. It deliberately bypasses the interruptible timed tryLock in the + // loop below, which throws InterruptedException the moment the calling thread merely carries a set + // interrupt flag - even on a FREE lock - and then re-arms that flag, so every later getToken() on the + // same thread would fail with a valid token sitting in the cache. An ILP producer on a pooled or + // managed thread, where interrupt is the standard cancellation signal, is the common case. An + // uncontended acquire cannot be behind an interactive sign-in (which holds the lock), so it is correct. + if (lock.tryLock()) { + return; + } + // Contended - a peer holds the lock. Never wait behind an interactive signIn(): it holds the lock for + // the whole device-code lifetime (up to 30 min) with no token to serve until it completes, so fail fast + // and let the caller retry. A peer holding the lock for a quick cached read or a silent refresh + // (bounded, usually well under a second) is different - the HttpTokenProvider contract permits a brief + // wait behind such a refresh - so poll for the lock in short slices rather than fail every concurrent + // caller sharing this instance on each token refresh (the old unconditional tryLock() did exactly that). + // Polling, not one blocking acquire, lets us still fail fast the moment an interactive sign-in - or + // close() - begins while we wait. Bound the total wait so a stuck or pathologically slow holder degrades + // to a retryable failure instead of stalling the flush path without bound - but size the bound to the + // holder's OWN worst-case hold, not a single httpTimeoutMillis. A legitimate silent refresh under the + // lock runs a send, an await and a body parse, each bounded by httpTimeoutMillis + // (LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE x in total, the same figure the FileTokenStore lock-stale floor is + // derived from), so a peer that waited only one httpTimeoutMillis would fail every concurrent caller + // behind a refresh that is going to succeed. + // nanoTime, not currentTimeMillis: this bound is an ELAPSED budget on the producer thread, and the + // wall clock is adjustable. An NTP step or an operator setting the date back stretches a millis-based + // deadline by the size of the jump, so the flush path this exists to protect would stall for however + // long the clock moved rather than the documented multiple of httpTimeoutMillis. The body reads + // (discardBody, parseBody) and the device-code poll already bound themselves this way. Compare by + // DIFFERENCE rather than by ordering, so the arithmetic stays correct across nanoTime's wraparound. + final long deadlineNanos = System.nanoTime() + + (long) LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE * httpTimeoutMillis * 1_000_000L; + while (true) { + throwIfClosed(); + if (interactiveSignInInProgress) { + throw new OidcAuthException("an interactive sign-in is in progress on another thread; no token is available without blocking - retry once it completes"); + } + final long remainingNanos = deadlineNanos - System.nanoTime(); + if (remainingNanos <= 0) { + throw new OidcAuthException("a token refresh is already in progress on another thread and no token became available in time; retry shortly"); + } + try { + if (lock.tryLock(Math.min(remainingNanos, GET_TOKEN_LOCK_POLL_SLICE_MILLIS * 1_000_000L), TimeUnit.NANOSECONDS)) { + return; + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + // tryAcquireNanos checks interruption before it attempts the CAS, including when the lock was + // released between the fast-path miss above and this timed acquire. Give that now-free lock one + // untimed attempt, which (like the fast path) ignores the carried flag and lets getToken() reach + // its cache check. Only report an interrupted WAIT when the lock is genuinely still held. + if (lock.tryLock()) { + return; + } + throw new OidcAuthException("interrupted while waiting to acquire the OIDC token"); + } + } + } + + private boolean adopt(PersistedToken token) { + if (token == null) { + return false; + } + // The file is attacker-writable, so the served token - the one getToken() puts verbatim into an + // Authorization header or a PG-wire password - is untrusted input. Two failure shapes look similar + // here and must be told apart, because the safe answer to each is the opposite of the other. + String servedToken = groupsInToken ? token.getIdToken() : token.getAccessToken(); + String fileRefreshToken = token.getRefreshToken(); + if (servedToken == null) { + // ABSENT, which is a legitimate shape rather than evidence of anything. Under + // groupsInToken=false a grant that returned only an id_token has storeTokens null the access + // token, and persistIfRotated writes the entry regardless; FileTokenStore also maps an empty + // on-disk value to null. Discarding such an entry throws away the refresh token, which is the + // one thing persistence exists to preserve, and sends a human back through the device flow + // where a single silent refresh would have done - a hard failure for a headless getToken() + // consumer. So keep the refresh token and leave the cache empty and expired, which puts + // getToken()/signIn() on the refresh path they already have for a null served kind. The refresh + // token needs no character check of its own: tryRefresh url-encodes it into the form body + // (appendParam -> urlEncode), unlike the served token, which reaches a header verbatim. + if (fileRefreshToken == null) { + return false; // nothing usable in this entry at all + } + if (token.getAccessToken() == null && token.getIdToken() == null) { + // NEITHER kind present, only a refresh token. That is not the legitimate shape above - it is + // positive evidence the entry was not written by this client, so reject the whole thing for + // the same reason the tampered-served-token branch below does. + // + // No grant this client stores can produce it. The device path reaches storeTokens only behind + // "accessToken.length() > 0 || idToken.length() > 0", and the refresh path only behind a + // non-blank served kind. persistIfRotated() is also reached from adoptRotatedRefreshToken(), + // where both kinds CAN be null - the branch below nulls them - so it refuses to write that + // shape rather than leave this rejection resting on a callsite count. A file with a refresh + // token and nothing else came from somewhere else. + // + // Left adopted, it is the cheapest credential swap there is: an attacker who can WRITE the + // store directory - never needing to read our 0600 file - drops in a file whose fingerprint + // fields are all derivable from public config, and the next silent refresh presents THEIR + // refresh token. The client then ingests and queries as them, with no prompt, no error, and + // nothing in any log recording that the identity changed. Unlike a directory-permission check + // this holds on every filesystem, Windows included, where owner-only permissions cannot be + // enforced at all. + // + // The cost when it fires on an honest file is one interactive sign-in. design/oidc-token- + // persistence.md states the rule for cross-language writers. + return false; + } + accessToken = null; + idToken = null; + refreshToken = fileRefreshToken; + expiresAtMillis = 0; + tokenTtlMillis = 0; + lastPersistedRefreshToken = fileRefreshToken; + return true; + } + if (Chars.isBlank(servedToken) || !hasOnlyTokenChars(servedToken) || Chars.equals("null", servedToken)) { + // PRESENT but unusable: whitespace-only (which passes hasOnlyTokenChars vacuously, space being + // 0x20, yet would be served as a blank "Bearer " header the server only answers with 401), + // carrying a control or non-ASCII character, or the four characters "null". Unlike an absent + // token this is positive evidence that something else wrote this file, so reject the WHOLE entry + // - refresh token included. Adopting the refresh token of a file we know was tampered with would + // let an attacker who can write the store swap in their own, and this client would silently sign + // in as them. + // + // On "null" specifically: JsonLexer reports a bare JSON null and a quoted "null" identically, so + // design/oidc-token-persistence.md forbids a writer from emitting a bare null at all and requires + // an absent value to be OMITTED. A served token that reads as "null" is therefore either a writer + // violating that rule - json.dumps({"access_token": None}) is the natural way to get there from + // Python, and cross-language sharing is the whole point of freezing the format - or a token + // pathological enough to be indistinguishable from one. Neither may become "Bearer null": the + // server answers that with 401, and because the persisted expiry is honoured getToken() would go + // on serving it rather than refreshing, so the producer 401s with nothing naming the cause until + // the clamped expiry lapses. Refusing costs one interactive sign-in, and only to a caller whose + // real bearer token is four characters long. + // + // Checked HERE rather than in FileTokenStore because adopt() is the choke point every TokenStore + // goes through, including a caller's own implementation of the SPI. The refresh token needs no + // equivalent arm: it is url-encoded into a form body rather than spliced into a header, so a + // "null" there is simply rejected by the token endpoint and degrades to an interactive sign-in. + return false; + } + accessToken = token.getAccessToken(); + idToken = token.getIdToken(); + // keep the current refresh token when the file carries none, mirroring the REFRESH branch of + // storeTokens() -- a stored entry is the same authorization read back, never a new one, so the + // grant-specific clearing that a fresh device grant does has no counterpart here. A file with a + // valid served token but no refresh_token - a cross-language peer that never received one, or a + // tampered file - must not null a live in-memory refresh token: doing so would make a later + // tryRefresh() urlEncode(null) and throw an uncaught NPE (aborting the sign-in) instead of refreshing + // with the token we still hold or degrading to an interactive sign-in. + if (fileRefreshToken != null) { + refreshToken = fileRefreshToken; + } + // the file is attacker-writable (and may have been written under a skewed clock), so bound how long + // the loaded token is trusted exactly as storeTokens() bounds a token from the wire: never past + // MAX_EXPIRES_IN_SECONDS from now. Clamp the expiry to [0, now + maxLife]: the ceiling stops a tampered + // far-future expiry from being trusted for decades, and the floor of 0 keeps a tampered far-past expiry + // in the past (1970, well before now) while keeping the validity check (now < expiresAtMillis - skew) + // underflow-safe - a near-Long.MIN_VALUE expiry would otherwise wrap that subtraction to a huge + // positive and serve a garbage-expiry token as valid forever. An already-expired entry still reads as + // expired and falls through to a refresh rather than being served. + long maxTokenLifeMillis = MAX_EXPIRES_IN_SECONDS * 1000L; + long now = System.currentTimeMillis(); + expiresAtMillis = Math.max(0L, Math.min(token.getExpiresAtMillis(), now + maxTokenLifeMillis)); + // Trust the file's stored ISSUED lifetime (bounded to [0, maxLife] against a tampered value), NOT the + // remaining span expiresAtMillis - now. effectiveSkewMillis() caps the clock-skew margin at half the + // lifetime - a guard meant only for a genuinely short-issued (< 60s) token - so deriving it from the + // shrinking remaining span would collapse the 30s skew toward zero as a normal token nears expiry and + // let getToken() serve a near-expired token on the flush path instead of refreshing. A tampered ttl can + // only shrink the skew (never inflate it past CLOCK_SKEW_MILLIS), exactly as the remaining-span form + // could, so trusting the stored value is no less safe; a file that carries no ttl (0) yields the full + // skew via effectiveSkewMillis()'s <= 0 branch. storeTokens() stores this same full issued lifetime, so + // both paths now give tokenTtlMillis one meaning. + tokenTtlMillis = Math.max(0L, Math.min(token.getTokenTtlMillis(), maxTokenLifeMillis)); + // track what the file actually carried (which is null when it had no refresh_token but we kept a live + // one above), so a later non-rotating refresh does not rewrite an unchanged on-disk token, yet a token + // we kept that the file did not carry is not mistaken for already-persisted and can be re-saved + lastPersistedRefreshToken = fileRefreshToken; + return true; + } + + /** + * Adopts a rotated {@code refresh_token} from a clean-2xx refresh response this client cannot otherwise + * use, so the rotation is not lost with the rest of the response. + *

+ * Two shapes reach here, and what they have in common is the only thing that matters: the provider + * accepted the refresh token we presented and answered 2xx with no OAuth error. The served kind may be + * ABSENT - RFC 6749 6 makes {@code id_token} optional, and OIDC Core 12.2 says the refresh response is + * the token response "except that it might not contain an id_token", which is exactly the shape a + * {@code groupsInToken} client meets against a provider that only mints an id token at authorization + * time. Or it may be PRESENT and unusable, rejected by {@code validateTokenChars} for a control or + * non-ASCII character. Either way the {@code refresh_token} in that body is authoritative: a rotating + * provider has already invalidated the one we presented. Keeping the old token would replay a spent + * credential on every later refresh, which a reuse-detecting provider answers by revoking the whole + * token family - so the caller loses the credential entirely rather than merely failing to refresh it + * once, and with a {@link TokenStore} that revocation reaches every process sharing the identity. + *

+ * Only the refresh token is taken. No usable served token arrived, so the cached tokens and the expiry + * stay as they were: the entry reads as expired, {@code tryRefresh()} still reports failure, and the + * caller falls back to the interactive flow exactly as before - now holding a refresh token that is + * still live, so the NEXT refresh can succeed on its own. + */ + private void adoptRotatedRefreshToken() { + if (tokenParser.refreshToken.length() == 0) { + return; + } + refreshToken = tokenParser.refreshToken.toString(); + // Persist it, for the same reason it is adopted at all. Without this the on-disk entry keeps the + // token the provider just burned, so the next process start adopts a dead credential and re-prompts + // a human who did not need to be asked. persistIfRotated() writes the snapshot's stale served token + // and past expiry alongside it, which adopt() reads back as expired and refreshes - one silent + // round trip, against a refresh token that works. + persistIfRotated(); + } + + private void appendEncodedParam(StringSink sink, String name, String encodedValue) { + sink.putAscii('&').putAscii(name).putAscii('=').putAscii(encodedValue); + } + + private void appendParam(StringSink sink, String name, String value) { + sink.putAscii('&').putAscii(name).putAscii('=').putAscii(urlEncode(value)); + } + + private long effectiveSkewMillis() { + // mirror the Python client's TokenSet.is_valid: cap the fixed 30s skew at half the token lifetime, so + // a short-lived (< 60s) token is not treated as expired the instant it is issued. With an unknown + // lifetime (no token cached yet), fall back to the full skew. + if (tokenTtlMillis <= 0) { + return CLOCK_SKEW_MILLIS; + } + return Math.min(CLOCK_SKEW_MILLIS, tokenTtlMillis / 2); + } + + private HttpClient httpClient(boolean isTls) { + if (isTls) { + if (tlsClient == null) { + tlsClient = HttpClientFactory.newTlsInstance(clientConfig, tlsConfig); + } + return tlsClient; + } + if (plainClient == null) { + plainClient = HttpClientFactory.newPlainTextInstance(clientConfig); + } + return plainClient; + } + + private boolean isHttpStatusSuccess() { + // responseStatus is the bare-digit HTTP status captured by readResponse. A real status is exactly 3 + // digits, so require that before reading the leading digit: a malformed short status such as "2" must + // not be mistaken for a 2xx success and accepted as a grant. + return responseStatus.length() == 3 && responseStatus.charAt(0) == '2'; + } + + private boolean isHttpStatusTerminal4xx() { + // a 4xx other than 429 is a terminal client-error rejection (429 is a transient rate-limit); require a + // full 3-digit status so a malformed short "4" is not classified as a terminal 4xx + return responseStatus.length() == 3 && responseStatus.charAt(0) == '4' && !Chars.equals(HTTP_STATUS_TOO_MANY_REQUESTS, responseStatus); + } + + private boolean isHttpStatusTransient() { + // a 5xx server error or a 429 rate-limit is transient - keep polling; any other non-2xx (a 4xx + // rejection) is terminal. Mirrors the Python client's _http_status_is_transient. Require a full + // 3-digit status so a malformed short "5" is not classified as a transient 5xx. + return responseStatus.length() == 3 && (responseStatus.charAt(0) == '5' || Chars.equals(HTTP_STATUS_TOO_MANY_REQUESTS, responseStatus)); + } + + private boolean isRefreshBackedOff() { + if (refreshFailedAtMillis == 0) { + return false; + } + // elapsed == 0 is the COMMON case, not an edge one: a producer retrying rows calls getToken() many + // times within the same millisecond, and that is exactly the flood this exists to stop - so zero + // counts as backed off. Only a NEGATIVE span, which means the clock jumped backwards, releases the + // latch early rather than pinning it until the clock catches up. + final long elapsed = System.currentTimeMillis() - refreshFailedAtMillis; + return elapsed >= 0 && elapsed < MIN_REFRESH_RETRY_INTERVAL_MILLIS; + } + + private boolean isStoreLoadBackedOff() { + final long remaining = nextStoreLoadAttemptMillis - System.currentTimeMillis(); + // Unlike isRefreshBackedOff(), a zero remaining span does NOT count as backed off: the first failure + // arms a zero-length back-off on purpose, so a same-millisecond retry still re-reads the store. + // A span longer than the cap cannot have been armed here, so it means the clock jumped BACKWARDS - + // release the latch rather than pin the store unreadable until the clock catches up. + return remaining > 0 && remaining <= MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS; + } + + private void maybeLoadFromStore() { + if (tokenStore == null || storeLoadAttempted || isStoreLoadBackedOff()) { + return; + } + PersistedToken token; + try { + token = tokenStore.load(storeKey); + } catch (RuntimeException e) { + // Best-effort: a store read failure must not break sign-in. Leave storeLoadAttempted UNSET so a + // transient failure is retried on a later call. Latching it here instead would make one failed + // read disable persistence for the whole life of this instance - so a process that owns a + // perfectly good refresh token on disk would re-run the interactive device flow, which for a + // headless getToken() consumer is a hard failure rather than a degraded one. + // + // Retried, but not on EVERY call: this runs on the getToken() path ahead of the cache check, so + // without a back-off a store that never becomes readable costs a blocking file open, two stack + // trace fills and a WARN line per ILP flush, forever, on the producer thread and under the lock. + // The first failure arms a zero-length back-off (an immediate retry, for the one-shot faults + // above), then each consecutive failure doubles it up to MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS. + final long backOffMillis = storeLoadRetryIntervalMillis; + storeLoadRetryIntervalMillis = backOffMillis == 0 + ? MIN_STORE_LOAD_RETRY_INTERVAL_MILLIS + : Math.min(backOffMillis * 2, MAX_STORE_LOAD_RETRY_INTERVAL_MILLIS); + nextStoreLoadAttemptMillis = System.currentTimeMillis() + backOffMillis; + warnPersistence("load", e); + return; + } + // the read COMPLETED, so its answer is definitive: a missing, corrupt or foreign-identity file yields + // null without throwing, and re-reading it on every later call would buy nothing + storeLoadAttempted = true; + adopt(token); + } + + private void persistIfRotated() { + if (tokenStore == null) { + return; + } + // This instance has now produced tokens of its own, so the on-disk entry is no longer authoritative + // for it and must never be read back over them. maybeLoadFromStore() deliberately leaves the latch + // UNSET when a read THROWS, so a transient fault is retried - but it runs at the top of getToken(), + // ahead of the cache check, and adopt() assigns the served kind, the expiry and the ttl + // unconditionally. Without this line: a store directory that is unavailable during signIn() (an + // unmounted home, a container started before its volume attaches) fails the read, the device flow + // completes, the save fails the same way and is swallowed, and then the directory recovers - so the + // next getToken(), one per ILP flush, re-reads and installs the PREVIOUS entry over the grant a + // human just authorized. Latched here rather than in storeTokens() so the refresh-only path through + // adoptRotatedRefreshToken() is covered by the same line, and before the rotation check below + // because it is true whether or not this call writes anything. + storeLoadAttempted = true; + // Never write an entry adopt() will refuse. It rejects a refresh token carried with NEITHER token + // kind as positive evidence of a foreign writer, and that reasoning is only sound while this client + // cannot produce the shape. It can: adopt()'s own served-kind-absent branch nulls BOTH kinds while + // keeping the refresh token, so a later refresh that rotates the refresh token but still returns no + // served kind reaches adoptRotatedRefreshToken() -> here with both null. Writing it would leave a + // file this client rejects for the life of the entry - a headless getToken() consumer re-running + // the device flow on every restart over a refresh token sitting on disk. + // + // Skipping the write leaves the previous entry in place, which is the better of the two: its + // refresh token is the one the provider just burned, so the next start spends one silent round + // trip and falls back to an interactive sign-in - the same end state, without a file that can + // never be read back. + if (accessToken == null && idToken == null) { + return; + } + // persist on a new or rotated refresh token (the interactive sign-in, or a provider that rotates the + // refresh token on every refresh); skip when it is unchanged, so the hot getToken() refresh path does + // not rewrite the file every few minutes. The on-disk access token then goes stale, which costs only + // one silent refresh on the next restart. With no refresh token there is nothing worth persisting. + if (Objects.equals(refreshToken, lastPersistedRefreshToken)) { + return; + } + try { + tokenStore.save(storeKey, snapshot()); + lastPersistedRefreshToken = refreshToken; + } catch (RuntimeException e) { + // best-effort: a save failure never fails an otherwise-valid sign-in; the token is valid in memory + warnPersistence("save", e); + } + } + + private void pollForToken(String deviceCode, int expiresInSeconds, int intervalSeconds) { + // url-encode the opaque device code once here, not on every poll: it is invariant for the whole + // poll loop (the grant_type and client_id are likewise pre-encoded) + final String deviceCodeEncoded = urlEncode(deviceCode); + final long deadlineNanos = System.nanoTime() + expiresInSeconds * 1_000_000_000L; + long intervalMillis = (long) intervalSeconds * 1000L; + while (true) { + throwIfClosed(); + // check the deadline before polling so an expiry that elapsed during the previous sleep aborts + // here, not after one more wasted poll round-trip + if (System.nanoTime() >= deadlineNanos) { + throw new OidcAuthException("timed out waiting for authorization, the device code expired; please retry"); + } + try { + int result = pollOnce(deviceCodeEncoded); + if (result == POLL_SUCCESS) { + return; + } + if (result == POLL_SLOW_DOWN) { + // grow the interval per RFC 8628, capped at the same bound as the initial value so + // repeated slow_down / 429 responses cannot inflate the wait without bound + intervalMillis = Math.min(intervalMillis + SLOW_DOWN_INCREMENT_SECONDS * 1000L, MAX_POLL_INTERVAL_SECONDS * 1000L); + } + // POLL_PENDING and POLL_TRANSIENT_ERROR (a transient 5xx) just poll again + } catch (HttpClientException e) { + // a transport failure (dropped connection, DNS blip, timeout) is transient: the user may + // already have authorized, and RFC 8628 expects polling to continue until the device code + // expires, so poll again rather than discard the sign-in (the deadline bounds the total + // wait). Matches the Python client. + } catch (OidcAuthException e) { + // a garbled / non-JSON body (a JsonException cause) is transient too, UNLESS its HTTP status + // is a terminal rejection (a non-JSON 4xx from a WAF or proxy); a well-formed terminal answer + // - an OAuth error, a terminal 4xx, a malformed status line - always aborts + if (!(e.getCause() instanceof JsonException) || isHttpStatusTerminal4xx()) { + throw e; + } + } + // wait for the next poll, never past the device-code deadline, so the timeout check at the top + // of the loop fires promptly at expiry instead of up to one poll interval late + sleepBetweenPolls(Math.min(intervalMillis, (deadlineNanos - System.nanoTime()) / 1_000_000L)); + } + } + + private int pollOnce(String deviceCodeEncoded) { + formSink.clear(); + formSink.putAscii("grant_type=").putAscii(GRANT_TYPE_DEVICE_CODE_ENCODED); + appendEncodedParam(formSink, "device_code", deviceCodeEncoded); + appendEncodedParam(formSink, "client_id", clientIdEncoded); + + tokenParser.clear(); + // a transport failure here propagates to pollForToken, which keeps polling (a transient blip) until + // the device-code deadline rather than swallowing it as a pending authorization + postForm(tokenEndpoint, tokenParser); + + // RFC 6749 5.2: an error response is an error even if the body also carries a token, or the status is + // 429 - so handle the OAuth error first. A terminal error (e.g. access_denied) must abort even when + // the identity provider also rate-limits, and a token smuggled alongside an error must never count as + // a grant. + if (tokenParser.error.length() > 0) { + if (Chars.equals(ERROR_AUTHORIZATION_PENDING, tokenParser.error)) { + return POLL_PENDING; + } + if (Chars.equals(ERROR_SLOW_DOWN, tokenParser.error)) { + return POLL_SLOW_DOWN; + } + throw OidcAuthException.oauthError(tokenParser.error, tokenParser.errorDescription); + } + + // A rate-limited identity provider answers 429 with no OAuth error; RFC 8628 does not define it, but + // the Python client and common practice treat it as "poll slower". Back off and keep polling (like + // slow_down) rather than treating it as a terminal error, so transient rate limiting does not fail + // the sign-in. + if (Chars.equals(HTTP_STATUS_TOO_MANY_REQUESTS, responseStatus)) { + return POLL_SLOW_DOWN; + } + // RFC 6749 5.1: a grant is a 2xx response carrying a token; a token under a non-2xx is malformed and + // is not trusted (the non-2xx is classified below instead) + if (isHttpStatusSuccess()) { + if (tokenParser.accessToken.length() > 0 || tokenParser.idToken.length() > 0) { + storeTokens(tokenParser, false); + return POLL_SUCCESS; + } + // a 2xx with neither a token nor an OAuth error is a definitive but malformed answer + throw new OidcAuthException().put("unexpected response from the token endpoint [httpStatus=").put(responseStatus).put(']'); + } + // a non-2xx with no recognized OAuth error: a 5xx (or 429, handled above) is a transient server or + // gateway condition - keep polling to the deadline; any other status is a terminal rejection (a 4xx + // from the identity provider, a WAF or a proxy) that aborts immediately rather than polling on to a + // misleading "device code expired". Matches the Python client. + if (isHttpStatusTransient()) { + return POLL_TRANSIENT_ERROR; + } + throw new OidcAuthException().put("the token endpoint rejected the request [httpStatus=").put(responseStatus).put("]; refusing to keep polling"); + } + + private void postForm(Endpoint endpoint, JsonParser parser) { + HttpClient client = httpClient(endpoint.isTls); + HttpClient.Request request = client.newRequest(endpoint.host, endpoint.port) + .POST() + .url(endpoint.path) + .header("Content-Type", "application/x-www-form-urlencoded") + .header("Accept", "application/json") + .header("User-Agent", USER_AGENT); + request.withContent(); + request.putAscii(formSink); + try { + HttpClient.ResponseHeaders response = request.send(httpTimeoutMillis); + response.await(httpTimeoutMillis); + readResponse(client, response, parser); + } catch (HttpClientException e) { + // a transport failure, or a bounded-read abort in parseBody (its elapsed-time deadline or the + // MAX_RESPONSE_BODY_BYTES cap), leaves the response half-read with unconsumed bytes in this + // cached keep-alive connection. Drop it so the next poll or refresh reconnects with a clean + // socket instead of parsing the previous response's leftovers - which pollForToken would + // otherwise keep doing, on a corrupted connection, until the device code expires. Mirrors the + // disconnect-on-failure handling in AbstractLineHttpSender.flush0. + client.disconnect(); + throw e; + } catch (HttpException e) { + // The RESPONSE HEAD was malformed or oversized, so HttpHeaderParser rejected it: a header block + // past the fixed 4096-byte parse buffer (a WAF or proxy stacking Set-Cookie/CSP), a malformed + // Content-Length, or a status line that is not HTTP/1.x. HttpException is a SIBLING of + // HttpClientException, not a subclass, so it missed the catch above - and with it the disconnect, + // leaving this CACHED keep-alive connection holding a half-read response for the next poll to + // parse as its own, exactly the corruption that catch exists to prevent. It also missed every + // classification downstream, aborting an interactive sign-in outright on a condition the same + // code rides out when it arrives as a transport error, and surfacing a type fromQuestDB/signIn + // do not document. The identity provider is untrusted here, so this is a response shape it can + // choose at will. + // + // Both are "the response is unusable", so answer identically: drop the connection and re-report + // as the transport-class failure every caller already handles. The message is a parser constant, + // never response bytes, so it carries no untrusted text. Copying it out also detaches the + // thread-local flyweight HttpException.instance() hands back, whose message the next + // HttpException on this thread would overwrite. + client.disconnect(); + throw new HttpClientException("malformed response from the identity provider: " + e.getMessage()); + } + } + + private void readResponse(HttpClient client, HttpClient.ResponseHeaders response, JsonParser parser) { + // capture only the HTTP status for diagnostics; the body is never retained or surfaced in a + // message - it carries access, id and refresh tokens that must not reach logs or exceptions + responseStatus.clear(); + DirectUtf8Sequence statusCode = response.getStatusCode(); + Response body = response.getResponse(); + if (statusCode != null) { + // a well-formed HTTP status code is bare digits, but the header parser copies the status-line + // token verbatim apart from SP/CR/LF, so a non-digit byte means a malformed or hostile status + // line. Reject it rather than echo any byte (which could smuggle ESC or other control sequences + // into a log or terminal when responseStatus is surfaced in a message below) or trust its + // leading digit as a success gate. Drain the body first to keep the keep-alive connection usable; + // if it could not be fully drained, drop the connection so the next request does not read this + // body's leftovers. + CharSequence raw = statusCode.asAsciiCharSequence(); + for (int i = 0, n = raw.length(); i < n; i++) { + char c = raw.charAt(i); + if (c < '0' || c > '9') { + if (!discardBody(body, httpTimeoutMillis)) { + client.disconnect(); + } + throw new OidcAuthException("the identity provider returned a malformed HTTP status code"); + } + responseStatus.put(c); + } + } + jsonLexer.clear(); + try { + parseBody(body, jsonLexer, parser, httpTimeoutMillis); + } catch (JsonException e) { + // drain the rest to keep the keep-alive connection usable; never embed the body, it may carry + // tokens. A body too large to drain within the cap (e.g. a multi-MB malformed response) leaves + // unconsumed bytes, so drop the connection rather than mis-frame the next request's response. + if (!discardBody(body, httpTimeoutMillis)) { + client.disconnect(); + } + throw new OidcAuthException(e) + .put("could not parse the identity provider response [httpStatus=").put(responseStatus).put(']'); + } + } + + private boolean refreshUnderLock() { + // runs inside the store's cross-process lock: re-read first, since another process sharing this + // identity may have refreshed (and rotated the refresh token) since our last load. Adopt a fresher + // entry and skip the network when it already yields a valid token; otherwise refresh with the freshest + // known refresh token (the one just adopted, so a rotated token is not replayed). + // + // Only re-read when the in-memory refresh token still matches what we last persisted. A mismatch no + // longer strictly means "in-memory is a newer unsaved token": it covers two cases, and re-adopting + // would regress in both, so keep the in-memory token and refresh with it. (1) A previous save failed + // (persistence is best-effort), so the in-memory token is genuinely newer than the on-disk one; + // re-adopting would regress it to the stale - and, on a rotating identity provider, already-revoked - + // on-disk token and force a needless re-prompt. (2) adopt() kept a live in-memory token that the loaded + // file did not carry (a cross-language peer that never received a refresh_token), leaving + // lastPersistedRefreshToken null; here the trade-off is that if a rotating-IdP peer has since revoked + // our token and written a fresher one, we skip that fresher on-disk token this round and fall back to an + // interactive re-prompt. Both are benign (never a stale/wrong served token; the pre-fix alternative in + // case 2 was an uncaught urlEncode(null) NPE) and cross-process-only. + if (Objects.equals(refreshToken, lastPersistedRefreshToken)) { + PersistedToken fresh; + try { + fresh = tokenStore.load(storeKey); + } catch (RuntimeException e) { + warnPersistence("load", e); + fresh = null; + } + if (adopt(fresh)) { + final String servedToken = groupsInToken ? idToken : accessToken; + if (servedToken != null && System.currentTimeMillis() < expiresAtMillis - effectiveSkewMillis()) { + return true; + } + } + } + return tryRefresh(); + } + + private void runDeviceFlow() { + formSink.clear(); + formSink.putAscii("client_id=").putAscii(clientIdEncoded); + appendEncodedParam(formSink, "scope", scopeEncoded); + if (audienceEncoded != null) { + appendEncodedParam(formSink, "audience", audienceEncoded); + } + + deviceAuthParser.clear(); + try { + postForm(deviceAuthorizationEndpoint, deviceAuthParser); + } catch (HttpClientException e) { + throw new OidcAuthException(e).put("could not reach the device authorization endpoint"); + } + + if (deviceAuthParser.error.length() > 0) { + throw OidcAuthException.oauthError(deviceAuthParser.error, deviceAuthParser.errorDescription); + } + // RFC 8628 3.2: a device authorization grant is a 2xx response. A non-2xx body with no OAuth error + // (handled above) is malformed or hostile; reject it rather than prompt the user and poll on it - + // the same 2xx gate pollOnce and tryRefresh apply before trusting a token + if (!isHttpStatusSuccess()) { + throw new OidcAuthException().put("unexpected response from the device authorization endpoint [httpStatus=").put(responseStatus).put(']'); + } + // the device code is sent in the poll requests, not shown, so check it on the wire; the user code and + // verification URL are shown to the user, so sanitize them first and require them non-empty after + // sanitizing - a value made entirely of control/format chars is non-empty on the wire but would + // otherwise display as a blank code or URL + final String deviceCode = deviceAuthParser.deviceCode.toString(); + final String userCode = sanitizeForDisplay(deviceAuthParser.userCode.toString()); + final String verificationUri = sanitizeForDisplay(deviceAuthParser.verificationUri.toString()); + if (deviceCode.isEmpty() || userCode.isEmpty() || verificationUri.isEmpty()) { + throw new OidcAuthException().put("incomplete device authorization response from the identity provider [httpStatus=").put(responseStatus).put(']'); + } + // a verification_uri_complete that is non-empty on the wire but sanitizes to empty is treated as + // absent (null), so the prompt prints no blank "(or open this URL ...)" line and the browser launcher + // is never handed an empty string + String verificationUriComplete = deviceAuthParser.verificationUriComplete.length() > 0 + ? sanitizeForDisplay(deviceAuthParser.verificationUriComplete.toString()) + : null; + if (verificationUriComplete != null && verificationUriComplete.isEmpty()) { + verificationUriComplete = null; + } + + final int expiresInSeconds = boundedSeconds(deviceAuthParser.expiresIn, DEFAULT_DEVICE_CODE_TTL_SECONDS, MAX_DEVICE_CODE_TTL_SECONDS); + final int intervalSeconds = boundedSeconds(deviceAuthParser.interval, DEFAULT_POLL_INTERVAL_SECONDS, MAX_POLL_INTERVAL_SECONDS); + final DeviceAuthorizationChallenge challenge = new DeviceAuthorizationChallenge( + userCode, + verificationUri, + verificationUriComplete, + expiresInSeconds, + intervalSeconds + ); + + throwIfClosed(); + prompt.promptUser(challenge); + pollForToken(deviceCode, expiresInSeconds, intervalSeconds); + } + + private String selectToken() { + if (groupsInToken) { + if (idToken != null) { + return idToken; + } + throw new OidcAuthException() + .put("the server expects groups encoded in the token (acl.oidc.groups.encoded.in.token=true) but the ") + .put("identity provider returned no id_token; ensure the requested scope includes 'openid'"); + } + if (accessToken != null) { + return accessToken; + } + throw new OidcAuthException("the identity provider returned no access_token"); + } + + private void sleepBetweenPolls(long millis) { + // Sleep in short slices so close() can abort an in-flight sign-in within ~POLL_SLEEP_SLICE_MILLIS + // instead of after a full (possibly slow_down-inflated) interval. + // + // Thread.sleep, not Os.sleep: Os.sleep catches InterruptedException, recomputes its deadline and + // keeps sleeping, and Thread.sleep CLEARS the flag when it throws - so the caller's interrupt was + // not merely ignored here, it was destroyed. A caller who cancelled then returned from signIn() with + // Thread.interrupted() reading false, its own cancellation bookkeeping none the wiser, having waited + // out a poll loop that runs to the device-code lifetime. This class states the opposite invariant + // twice ("the flag is the caller's cancellation signal and must survive this call"), and getToken() + // and FileTokenStore.load()/save() honour it. + long remaining = millis; + while (remaining > 0) { + throwIfClosed(); + throwIfInterrupted("the calling thread was interrupted while waiting for authorization"); + long slice = Math.min(POLL_SLEEP_SLICE_MILLIS, remaining); + try { + Thread.sleep(slice); + } catch (InterruptedException e) { + // Thread.sleep cleared the flag; put it back and let the caller see the cancellation both + // ways - as the exception below and as the flag their own shutdown path is waiting on. + Thread.currentThread().interrupt(); + throwIfInterrupted("the calling thread was interrupted while waiting for authorization"); + } + remaining -= slice; + } + } + + private PersistedToken snapshot() { + return new PersistedToken(accessToken, idToken, refreshToken, expiresAtMillis, tokenTtlMillis); + } + + /** + * @param isRefreshGrant true for a refresh_token grant, false for a fresh device grant. Decides what an + * OMITTED refresh_token means, which is not the same question for the two grants. + */ + private void storeTokens(TokenResponseParser parser, boolean isRefreshGrant) { + // reject a token with control or non-ASCII chars before caching: getToken() serves it verbatim as an + // HTTP Authorization header value and a PG-wire password, where a decoded CR/LF would inject into the + // request line sent to the trusted QuestDB server. Validate only the kind getToken() actually serves + // (the one that reaches the wire); the other kind is cached but never sent, so a stray char in it must + // not abort an otherwise-usable grant. + if (groupsInToken) { + validateTokenChars(parser.idToken, "id_token"); + } else { + validateTokenChars(parser.accessToken, "access_token"); + } + // treat a blank (empty OR whitespace-only) token as absent (null), not as a usable credential: a + // whitespace-only served token passes the char check vacuously (space is 0x20) but would be served as + // a blank "Bearer " header the server only answers with 401, so cache it as null and let selectToken / + // the wrong-token-kind fallback handle a missing served kind rather than serve it. An empty string was + // already treated as absent here; this only additionally folds in whitespace-only, matching adopt() and + // the sender's own HttpTokenProvider.validateToken (Chars.isBlank). + accessToken = Chars.isBlank(parser.accessToken) ? null : parser.accessToken.toString(); + idToken = Chars.isBlank(parser.idToken) ? null : parser.idToken.toString(); + // What an omitted refresh_token means depends on the grant, so the two must not share a policy. + // A refresh response usually omits one (RFC 6749 6 makes it optional) and is the SAME authorization + // continuing, so the current token stays valid and is kept -- dropping it would send a human back + // through the device flow every time a non-rotating provider answers. + // A device grant is a NEW authorization and may be a DIFFERENT human. Keeping the previous user's + // refresh token across it is cross-account confusion: user A's refresh fails, user B completes the + // device flow without a refresh token, B's access token expires, and the next silent refresh presents + // A's retained token and resumes as A -- no prompt, no error, nothing in any log to say the identity + // changed. So an omission here clears it: this authorization has no refresh token, and the honest + // outcome is that getToken() asks for an interactive sign-in. + if (parser.refreshToken.length() > 0) { + refreshToken = parser.refreshToken.toString(); + } else if (!isRefreshGrant) { + refreshToken = null; + } + // clamp like the device-side expires_in: default for a non-positive value, cap an absurd one, so a + // hostile or buggy token TTL cannot cache the token for decades (the server still enforces the real + // expiry; this only bounds how long the client trusts its cached copy) + int ttlSeconds = boundedSeconds(parser.expiresIn, DEFAULT_TOKEN_TTL_SECONDS, MAX_EXPIRES_IN_SECONDS); + tokenTtlMillis = ttlSeconds * 1000L; + expiresAtMillis = System.currentTimeMillis() + tokenTtlMillis; + persistIfRotated(); + } + + private void throwIfClosed() { + if (closed) { + throw new OidcAuthException("the OidcDeviceAuth instance is closed"); + } + } + + /** + * Abandons the current step when the calling thread carries an interrupt, LEAVING THE FLAG SET. + *

+ * isInterrupted(), never interrupted(): the flag is the caller's cancellation signal and has to outlive + * this call, so the shutdown path that raised it - an ExecutorService.shutdownNow(), a Future.cancel, + * QWP's ConnectCancellation - still sees it. Clearing it here would leave the caller believing it was + * never cancelled, which is the failure this guard exists to stop rather than one more of its causes. + * + * @param message what the caller was doing when the cancellation was noticed + */ + private void throwIfInterrupted(String message) { + if (Thread.currentThread().isInterrupted()) { + throw new OidcAuthException(message); + } + } + + private boolean tryRefresh() { + if (refreshToken == null) { + // nothing to present: degrade to the interactive flow rather than urlEncode(null) and throw. + // adopt() keeps a live refresh token, so this only fires if a caller reaches here with none. + return false; + } + formSink.clear(); + formSink.putAscii("grant_type=").putAscii(GRANT_TYPE_REFRESH_TOKEN_ENCODED); + appendParam(formSink, "refresh_token", refreshToken); + appendEncodedParam(formSink, "client_id", clientIdEncoded); + appendEncodedParam(formSink, "scope", scopeEncoded); + if (audienceEncoded != null) { + appendEncodedParam(formSink, "audience", audienceEncoded); + } + + tokenParser.clear(); + // A coordinated refresh may have spent time waiting for a peer instance's process lock and re-reading + // the store. close() publishes cancellation without taking this instance lock, so check again at the + // last point before network I/O and do not issue a fresh token POST for an instance being torn down. + throwIfClosed(); + try { + postForm(tokenEndpoint, tokenParser); + } catch (HttpClientException e) { + // could not reach the token endpoint; fall back to the interactive flow + return false; + } catch (OidcAuthException e) { + // postForm throws OidcAuthException only on a parse failure (a garbled / unparseable refresh + // response), never an OAuth error: a genuine OAuth error arrives in tokenParser.error, handled + // by hasRequiredToken below. So treat this as a transient blip and fall back to the interactive + // flow rather than fail the whole getToken() call + return false; + } + // succeed only on a clean 2xx (no OAuth error) returning the token getToken() actually serves (the + // id token when groups are encoded in it, the access token otherwise). A refresh that omits the served + // kind - which RFC 6749 permits and many providers do - or returns it blank/whitespace-only, or carries + // an error or a non-2xx status, must fall back to the interactive flow rather than be cached. Test the + // served kind with Chars.isBlank, the SAME contract storeTokens/adopt use to fold a blank token to null: + // gating on length() > 0 here would pass a whitespace-only token, which storeTokens then nulls, so + // tryRefresh would report success while selectToken() throws "no token" instead of falling back. + // + // Split the "this response is a clean grant" half out: it is what decides whether a rotated + // refresh_token in the SAME body is authoritative, and that question outlives the served-kind test + // below. See adoptRotatedRefreshToken(). + final boolean isCleanGrant = isHttpStatusSuccess() && tokenParser.error.length() == 0; + boolean hasRequiredToken = (groupsInToken + ? !Chars.isBlank(tokenParser.idToken) + : !Chars.isBlank(tokenParser.accessToken)) + && isCleanGrant; + if (hasRequiredToken) { + try { + storeTokens(tokenParser, true); + } catch (OidcAuthException e) { + // storeTokens -> validateTokenChars rejects a refreshed served token carrying a control or + // non-ASCII char (reachable now that JsonLexer decodes an escaped \r/\n in the response into a + // real byte). Fall back to the interactive flow like the transport/parse-failure arms above, + // rather than let the rejection propagate out of getToken()/signIn() past the runDeviceFlow() + // fallback the caller expects. validateTokenChars runs before any state mutation, so the cached + // token and refresh token are left intact for that fallback. + // + // Intact is exactly what the refresh token must NOT be left. This is still a clean 2xx, so a + // rotating provider has already invalidated the one we presented and the refresh_token in + // this body is the live one - the same reasoning adoptRotatedRefreshToken() states for the + // sibling branch below, which reaches it because the served kind was ABSENT rather than + // unusable. Dropping the rotation here leaves getToken() replaying a spent credential on + // every later refresh, and a reuse-detecting provider answers a replay by revoking the whole + // family - so the caller loses the credential outright instead of failing this one refresh, + // and with a TokenStore that revocation reaches every process sharing the identity. + adoptRotatedRefreshToken(); + return false; + } + return true; + } + if (isCleanGrant) { + // The provider accepted our refresh token and answered 2xx; it simply did not return the kind + // getToken() serves. Take the rotated refresh_token before dropping the rest of the response - + // see adoptRotatedRefreshToken() for why keeping the old one is worse than failing this refresh. + adoptRotatedRefreshToken(); + } + // the refresh token expired or was revoked, or did not return the token we need; fall back to the + // interactive flow + return false; + } + + private void clearTokenStoreWaiter() { + synchronized (tokenStoreWaiterGuard) { + if (tokenStoreWaiterThread == Thread.currentThread()) { + tokenStoreWaiterThread = null; + } + } + } + + private void publishTokenStoreWaiter() { + synchronized (tokenStoreWaiterGuard) { + // Pair publish-then-closed with close()'s closed-then-read handshake. If close won the race, do not + // enter a potentially blocking store call; if this publication won, close sees and interrupts it. + throwIfClosed(); + tokenStoreWaiterThread = Thread.currentThread(); + } + } + + private int tryRefreshCoordinated() { + if (tokenStore == null) { + return tryRefresh() ? REFRESH_SUCCEEDED : REFRESH_FAILED; + } + // Serialise the read-refresh-write across processes (and adopt a peer's just-rotated refresh token) + // through the store's per-identity lock; a store that does not coordinate just runs the refresh. + // + // TokenStore is a user-implemented SPI and persistence is documented best-effort, so a store that + // throws must not take the sign-in down with it - it did, because inLock was called bare. What the + // right degrade is depends entirely on whether the refresh already ran, which only the action + // itself can report: + // - the store threw BEFORE the action ran: nothing was refreshed, so run ONE uncoordinated + // refresh. Exactly one: the point of the lock is that a rotating refresh token must not be + // POSTed twice, and a reuse-detecting provider answers a replay by revoking the whole family. + // - the store threw AFTER the action completed (releasing a lock, closing a handle): the refresh + // HAPPENED and the token is live. Report what the action returned; re-running it would be that + // same double-POST, and throwing would tell the caller a completed sign-in failed. + // - the action itself threw: that is the refresh's own failure, not the store's. Never swallow + // it and never replay it - let it propagate exactly as it did before. + // Error is deliberately not caught: an OutOfMemoryError is not a store fault to degrade around. + final boolean[] actionEntered = new boolean[1]; + final boolean[] actionCompleted = new boolean[1]; + final boolean[] actionResult = new boolean[1]; + publishTokenStoreWaiter(); + try { + final boolean storeResult; + try { + storeResult = tokenStore.inLock(storeKey, () -> { + actionEntered[0] = true; + // inLock() acquired (or deliberately declined to acquire) its coordination lock. Clear the + // interrupt target before doing store I/O or returning a cached peer token, then re-check + // close so an interrupt delivered just before this hand-off cannot accompany a successful + // return. + clearTokenStoreWaiter(); + throwIfClosed(); + boolean refreshed = refreshUnderLock(); + actionResult[0] = refreshed; + actionCompleted[0] = true; + return refreshed; + }); + } finally { + // Covers an interrupted/declined wait and a store that throws or returns without invoking action. + // This inner finally runs before the catch below degrades to an uncoordinated refresh. + clearTokenStoreWaiter(); + } + // inLock() can return false either because action ran and the refresh failed, or because an + // interrupt abandoned its wait before action. Do not infer which from the thread flag: the flag + // may have arrived during a refresh that did run. The callback entry is the exact discriminator. + if (!actionEntered[0]) { + return REFRESH_NOT_ATTEMPTED; + } + return storeResult ? REFRESH_SUCCEEDED : REFRESH_FAILED; + } catch (RuntimeException e) { + if (actionCompleted[0]) { + warnPersistence("lock release", e); + return actionResult[0] ? REFRESH_SUCCEEDED : REFRESH_FAILED; + } + if (actionEntered[0]) { + throw e; + } + warnPersistence("lock", e); + return tryRefresh() ? REFRESH_SUCCEEDED : REFRESH_FAILED; + } + } + + private void warnPersistence(String operation, Throwable cause) { + // best-effort persistence: warn through SLF4J and carry on with the in-memory token. The store never + // puts token bytes in its messages, but an IO error can carry the operator-supplied store path, which + // could itself hold terminal-spoofing characters - sanitize the detail before printing, as every other + // untrusted display string is sanitized (sanitizeForDisplay is null-safe). + String detail = sanitizeForDisplay(cause.getMessage()); + LOG.warn("OIDC token store {} failed; continuing without persistence{}", + operation, detail != null ? " [" + detail + ']' : ""); + } + private void wipeCredentialState() { + // Best effort, and worth being precise about what that means. + // + // Nulling the four String fields is all Java offers for a String - the characters live on until the + // GC reclaims them - but it does stop this instance from handing them back. + // + // The sinks are the part a plain null misses. formSink carries the request body, which on the refresh + // path is literally "refresh_token="; the two parsers hold every field of the last + // response, tokens and device code included. All of them are REUSED, and clear() only rewinds the + // write position, so a long secret followed by a short write stays legible in the tail. wipe() + // overwrites the whole backing array instead. + // + // jsonLexer is in that set too, and wiping the parsers alone missed it: the lexer ASSEMBLES every + // name and value in its own decode sinks before a listener ever sees one, so the parsers' copies + // are the second copy, not the first. It is a field reused for every token response, and neither + // JsonLexer.clear() (parse state only) nor close() (frees the native cache without zeroing it) + // touches those sinks, so the whole token stayed legible on the heap for the life of this + // instance - through clearCache(), which is exactly when a caller expects it gone. + // + // What it cannot reach: any String already handed to a caller, and the HTTP client's native receive + // buffers, where the raw token bytes also passed. Freeing those returns the pages to the allocator + // without zeroing them. A caller who needs more than this should not be persisting tokens in this + // process at all. + accessToken = null; + idToken = null; + refreshToken = null; + lastPersistedRefreshToken = null; + formSink.wipe(); + responseStatus.wipe(); + deviceAuthParser.wipe(); + tokenParser.wipe(); + if (jsonLexer != null) { + // null on a second close(): the first one wiped it and then freed the field. close() is + // documented idempotent, so the guard keeps it so - there is nothing left to wipe by then + // anyway, and the object is already unreachable. + jsonLexer.wipe(); + } + } + + + /** + * Fluent builder for an {@link OidcDeviceAuth} configured against a known identity provider. + * The client id, device authorization endpoint and token endpoint are required. + */ + public static final class Builder { + private boolean allowInsecureTransport; + private String audience; + private String clientId; + private String deviceAuthorizationEndpoint; + private boolean groupsInToken; + private int httpTimeoutMillis = DEFAULT_HTTP_TIMEOUT_MILLIS; + private String issuer; + private DeviceCodePrompt prompt = DeviceCodePrompt.openBrowser(); + private String scope = DEFAULT_SCOPE; + private ClientTlsConfiguration tlsConfig; + private String tokenEndpoint; + private TokenStore tokenStore; + + private Builder() { + } + + /** + * Opts into insecure {@code http} for the QuestDB {@code /settings} link (only meaningful via + * {@link #fromQuestDB}). It does not relax the identity provider endpoints configured here: + * the device authorization and token endpoints always require {@code https} unless they are + * loopback, so the device code and refresh token never cross the network in cleartext (matching + * the Python client). Defaults to {@code false}. + *

+ * It bounds the SCHEME, not the trust anchor. {@code tlsConfig} is what decides whether the client + * validates the identity provider's certificate, and one instance carries a single one, so passing + * {@link ClientTlsConfiguration#INSECURE_NO_VALIDATION} to reach a self-signed QuestDB also turns + * validation off on the device-authorization and token requests - the legs that carry the device code + * and the refresh token. Point {@code tlsConfig} at a trust store instead when an identity provider + * is in play. + * + * @see #tlsConfig(ClientTlsConfiguration) + */ + public Builder allowInsecureTransport(boolean allowInsecureTransport) { + this.allowInsecureTransport = allowInsecureTransport; + return this; + } + + /** + * Sets the {@code audience} (or {@code resource}) request parameter, sent on the device + * authorization and refresh requests. Some identity providers require it so the issued token + * carries the {@code aud} claim QuestDB expects. {@link #fromQuestDB} discovers it from + * {@code acl.oidc.audience}. Optional. + */ + public Builder audience(String audience) { + this.audience = audience; + return this; + } + + public OidcDeviceAuth build() { + if (clientId == null || clientId.isEmpty()) { + throw new OidcAuthException("clientId is required"); + } + if (deviceAuthorizationEndpoint == null || deviceAuthorizationEndpoint.isEmpty()) { + throw new OidcAuthException("deviceAuthorizationEndpoint is required"); + } + if (tokenEndpoint == null || tokenEndpoint.isEmpty()) { + throw new OidcAuthException("tokenEndpoint is required"); + } + if (scope == null || scope.isEmpty()) { + scope = DEFAULT_SCOPE; + } + Endpoint deviceEndpoint = Endpoint.parse(deviceAuthorizationEndpoint); + Endpoint parsedTokenEndpoint = Endpoint.parse(tokenEndpoint); + Endpoint issuerEndpoint = issuer != null && !issuer.isEmpty() ? Endpoint.parse(issuer) : null; + requireSecureIdpEndpoint(deviceEndpoint, "device authorization endpoint", deviceAuthorizationEndpoint, allowInsecureTransport); + requireSecureIdpEndpoint(parsedTokenEndpoint, "token endpoint", tokenEndpoint, allowInsecureTransport); + // enforce the credential-endpoint co-location / issuer pin on every construction path, not just + // discovery, so the documented guarantee holds for the explicit builder too + validateEndpointOrigins(parsedTokenEndpoint, deviceEndpoint, issuerEndpoint); + ClientTlsConfiguration tls = tlsConfig != null ? tlsConfig : defaultTlsConfig(); + // a FileTokenStore steals a lock older than its staleness window, presuming a crashed holder; that + // window must exceed the worst-case time a live refresh holds the lock, or a peer could steal a live + // holder's lock mid-refresh and reopen the rotating-refresh-token race the lock prevents. Enforce the + // bounded part of that worst case here, where both values are known: the refresh I/O under the lock is + // up to LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE x httpTimeoutMillis, which counts the TCP connect and the + // TLS handshake as the two separate budgets HttpClient actually spends on them (DNS resolution + // remains the OS's), so this floor covers the hold rather than only part of it. The default 600s + // window leaves ample headroom at the DEFAULT 30s timeout (180s), but NOT at the 120s cap, where a + // hold can reach 720s: a caller raising httpTimeoutMillis must raise lockStaleMillis to match, and + // this rejects the combination rather than shipping the race. A non-coordinating TokenStore is + // exempt - it takes no lock. + if (tokenStore instanceof FileTokenStore) { + long minStaleMillis = (long) LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE * httpTimeoutMillis; + long staleMillis = ((FileTokenStore) tokenStore).getLockStaleMillis(); + if (staleMillis < minStaleMillis) { + throw new OidcAuthException() + .put("the FileTokenStore lockStaleMillis (").put(staleMillis) + .put(") must be at least ").put(LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE) + .put("x httpTimeoutMillis (").put(minStaleMillis) + .put("), otherwise a slow refresh's live cross-process lock could be stolen by a peer mid-refresh"); + } + } + return new OidcDeviceAuth(this, tls, deviceEndpoint, parsedTokenEndpoint); + } + + public Builder clientId(String clientId) { + this.clientId = clientId; + return this; + } + + public Builder deviceAuthorizationEndpoint(String deviceAuthorizationEndpoint) { + this.deviceAuthorizationEndpoint = deviceAuthorizationEndpoint; + return this; + } + + /** + * Selects which token {@link #getToken()} returns. Set to {@code true} when the server has + * {@code acl.oidc.groups.encoded.in.token=true} (the id token is returned), {@code false} + * otherwise (the access token is returned). Defaults to {@code false}. + */ + public Builder groupsInToken(boolean groupsInToken) { + this.groupsInToken = groupsInToken; + return this; + } + + public Builder httpTimeoutMillis(int httpTimeoutMillis) { + if (httpTimeoutMillis <= 0) { + throw new OidcAuthException("httpTimeoutMillis must be positive"); + } + if (httpTimeoutMillis > MAX_HTTP_TIMEOUT_MILLIS) { + throw new OidcAuthException() + .put("httpTimeoutMillis must not exceed ").put(MAX_HTTP_TIMEOUT_MILLIS) + .put("; a token-endpoint round-trip never needs longer, and a larger value could let a ") + .put("slow refresh outlast the token store's cross-process lock staleness window"); + } + this.httpTimeoutMillis = httpTimeoutMillis; + return this; + } + + /** + * Pins the identity provider by its {@code issuer} origin (for example + * {@code https://idp.example.com}). When set, {@link #build()} rejects the explicitly configured token + * or device authorization endpoint if it is not on this origin - a sanity check that the endpoints you + * supplied belong to the issuer you intended. A provider hosting its endpoints on a different origin + * than its issuer (for example Google) is rejected when pinned this way; for such a provider, configure + * the endpoints without an issuer. Optional. + *

+ * {@link #fromQuestDB(String, DiscoveryOptions)} pins differently: it constrains only the endpoints the + * untrusted {@code /settings} response advertised (to the issuer's origin, and under its path when the + * issuer has one), while endpoints discovered from the provider's own {@code .well-known} are trusted + * wherever the issuer hosts them - so discovery against an off-origin provider like Google works. + */ + public Builder issuer(String issuer) { + this.issuer = issuer; + return this; + } + + /** + * Sets how the device code challenge is shown to the user. Defaults to + * {@link DeviceCodePrompt#openBrowser()} - prints to {@code System.out} and also opens the + * verification URL in a browser when one is available; pass {@link DeviceCodePrompt#SYSTEM_OUT} + * to print only. + */ + public Builder prompt(DeviceCodePrompt prompt) { + this.prompt = prompt != null ? prompt : DeviceCodePrompt.openBrowser(); + return this; + } + + public Builder scope(String scope) { + this.scope = scope; + return this; + } + + /** + * Sets the TLS configuration for every {@code https} request this instance makes: the QuestDB + * {@code /settings} discovery request, any identity provider discovery document, and the + * device-authorization and token requests of the sign-in itself. Defaults to full validation. + *

+ * One configuration covers all of them, so it is also what decides whether the client validates the + * IDENTITY PROVIDER's certificate. {@link ClientTlsConfiguration#INSECURE_NO_VALIDATION}, reached for + * to talk to a QuestDB server presenting a self-signed certificate, therefore also stops the client + * authenticating the token endpoint - which carries the refresh token - leaving that leg encrypted + * but open to an on-path attacker. {@link #allowInsecureTransport(boolean)} is the knob scoped to the + * QuestDB link alone, and it only relaxes the scheme; there is no per-leg trust anchor. Prefer a + * trust store over disabling validation whenever an identity provider is involved. + * + * @param tlsConfig the TLS configuration, or {@code null} for full validation + * @return this instance for method chaining + */ + public Builder tlsConfig(ClientTlsConfiguration tlsConfig) { + this.tlsConfig = tlsConfig; + return this; + } + + public Builder tokenEndpoint(String tokenEndpoint) { + this.tokenEndpoint = tokenEndpoint; + return this; + } + + /** + * Persists the obtained token through the given {@link TokenStore}, so a restarted process can resume + * from the saved refresh token instead of running the device flow again. Defaults to {@code null} + * (in-memory only). Use {@link FileTokenStore#atDefaultLocation()} for the default file-backed store, + * or supply your own to back persistence with an OS keychain or a secrets manager. Optional. + */ + public Builder tokenStore(TokenStore tokenStore) { + this.tokenStore = tokenStore; + return this; + } + } + + /** + * Options for {@link #fromQuestDB(String, DiscoveryOptions)}: how to pin the identity provider + * (issuer), the TLS configuration for discovery and sign-in, whether to permit insecure {@code http}, + * and how to show the device code challenge. Every option is optional; an instance with nothing set + * behaves like {@link #fromQuestDB(String)}. + */ + public static final class DiscoveryOptions { + private boolean allowInsecureTransport; + private String issuer; + private DeviceCodePrompt prompt = DeviceCodePrompt.openBrowser(); + private ClientTlsConfiguration tlsConfig; + private TokenStore tokenStore; + + /** + * Permits insecure {@code http} for the QuestDB server link only (the {@code /settings} discovery + * request). It does not relax the identity provider endpoints, which always require + * {@code https} unless they are loopback, so the device code and refresh token are never sent in + * cleartext. Enable only for local development on a trusted network. Defaults to {@code false}. + *

+ * It bounds the SCHEME, not the trust anchor. {@code tlsConfig} is what decides whether the client + * validates the identity provider's certificate, and one instance carries a single one, so passing + * {@link ClientTlsConfiguration#INSECURE_NO_VALIDATION} to reach a self-signed QuestDB also turns + * validation off on the device-authorization and token requests - the legs that carry the device code + * and the refresh token. Point {@code tlsConfig} at a trust store instead when an identity provider + * is in play. + * + * @see #tlsConfig(ClientTlsConfiguration) + */ + public DiscoveryOptions allowInsecureTransport(boolean allowInsecureTransport) { + this.allowInsecureTransport = allowInsecureTransport; + return this; + } + + /** + * Pins the identity provider by its {@code issuer} origin (for example + * {@code https://idp.example.com}). It plays two roles: when the server does not advertise the + * device authorization endpoint, it is discovered from the issuer's + * {@code .well-known/openid-configuration} (the discovery origin comes only from this out-of-band + * issuer, never from {@code /settings}); and it constrains the endpoints the untrusted + * {@code /settings} response advertised - they must be on the issuer's origin, and under its path when + * the issuer has one, so a tampered {@code /settings} cannot redirect credentials to a different origin + * or to a different tenant on a path-based provider (for example a Keycloak realm path like + * {@code /realms/acme}). Endpoints discovered from the provider's own {@code .well-known} are trusted + * wherever the issuer hosts them, so an identity provider that serves its endpoints from a different + * origin than its issuer (for example Google) works through discovery. Optional. + */ + public DiscoveryOptions issuer(String issuer) { + this.issuer = issuer; + return this; + } + + /** + * Sets how the device code challenge is shown to the user. Defaults to + * {@link DeviceCodePrompt#openBrowser()} - prints to {@code System.out} and also opens the + * verification URL in a browser when one is available; pass {@link DeviceCodePrompt#SYSTEM_OUT} + * to print only. + */ + public DiscoveryOptions prompt(DeviceCodePrompt prompt) { + this.prompt = prompt != null ? prompt : DeviceCodePrompt.openBrowser(); + return this; + } + + /** + * Sets the TLS configuration used for the {@code /settings} discovery request, any identity + * provider discovery document, and the later sign-in requests. Defaults to full validation. + *

+ * One configuration covers all of them, so it is also what decides whether the client validates the + * IDENTITY PROVIDER's certificate - see {@link Builder#tlsConfig(ClientTlsConfiguration)} for what + * {@link ClientTlsConfiguration#INSECURE_NO_VALIDATION} costs on the leg that carries the refresh + * token. + */ + public DiscoveryOptions tlsConfig(ClientTlsConfiguration tlsConfig) { + this.tlsConfig = tlsConfig; + return this; + } + + /** + * Persists the obtained token through the given {@link TokenStore}, so a restarted process can resume + * from the saved refresh token instead of running the device flow again. Defaults to {@code null} + * (in-memory only). See {@link FileTokenStore#atDefaultLocation()} for the default file-backed store. + */ + public DiscoveryOptions tokenStore(TokenStore tokenStore) { + this.tokenStore = tokenStore; + return this; + } + } + + private static final class DeviceAuthorizationResponseParser implements JsonParser, Mutable { + private static final int FIELD_DEVICE_CODE = 1; + private static final int FIELD_ERROR = 7; + private static final int FIELD_ERROR_DESCRIPTION = 8; + private static final int FIELD_EXPIRES_IN = 5; + private static final int FIELD_INTERVAL = 6; + private static final int FIELD_NONE = 0; + private static final int FIELD_USER_CODE = 2; + private static final int FIELD_VERIFICATION_URI = 3; + private static final int FIELD_VERIFICATION_URI_COMPLETE = 4; + final StringSink deviceCode = new StringSink(); + final StringSink error = new StringSink(); + final StringSink errorDescription = new StringSink(); + final StringSink userCode = new StringSink(); + final StringSink verificationUri = new StringSink(); + final StringSink verificationUriComplete = new StringSink(); + int expiresIn; + int interval; + // objects nested inside a JSON array are never trusted: an array-wrapped response must not surface its + // element object's fields at the top-level depth. arrayDepth gates every name/value read on being 0. + private int arrayDepth; + private int depth; + private int field = FIELD_NONE; + + void wipe() { + // clear() rewinds; this overwrites. The device code is a credential until it expires, and the + // verification URIs carry the user code, so none of it should outlive the instance that read it. + deviceCode.wipe(); + error.wipe(); + errorDescription.wipe(); + userCode.wipe(); + verificationUri.wipe(); + verificationUriComplete.wipe(); + clear(); + } + + @Override + public void clear() { + deviceCode.clear(); + error.clear(); + errorDescription.clear(); + userCode.clear(); + verificationUri.clear(); + verificationUriComplete.clear(); + expiresIn = 0; + interval = 0; + arrayDepth = 0; + depth = 0; + field = FIELD_NONE; + } + + @Override + public void onEvent(int code, CharSequence tag, int position) { + switch (code) { + case JsonLexer.EVT_ARRAY_START: + arrayDepth++; + break; + case JsonLexer.EVT_ARRAY_END: + arrayDepth--; + break; + case JsonLexer.EVT_OBJ_START: + depth++; + break; + case JsonLexer.EVT_OBJ_END: + depth--; + break; + case JsonLexer.EVT_NAME: + if (arrayDepth == 0 && depth == 1) { + if (Chars.equals("device_code", tag)) { + field = FIELD_DEVICE_CODE; + } else if (Chars.equals("user_code", tag)) { + field = FIELD_USER_CODE; + } else if (Chars.equals("verification_uri", tag) || Chars.equals("verification_url", tag)) { + field = FIELD_VERIFICATION_URI; + } else if (Chars.equals("verification_uri_complete", tag) || Chars.equals("verification_url_complete", tag)) { + field = FIELD_VERIFICATION_URI_COMPLETE; + } else if (Chars.equals("expires_in", tag)) { + field = FIELD_EXPIRES_IN; + } else if (Chars.equals("interval", tag)) { + field = FIELD_INTERVAL; + } else if (Chars.equals("error", tag)) { + field = FIELD_ERROR; + } else if (Chars.equals("error_description", tag)) { + field = FIELD_ERROR_DESCRIPTION; + } else { + field = FIELD_NONE; + } + } + break; + case JsonLexer.EVT_VALUE: + if (arrayDepth == 0 && depth == 1) { + switch (field) { + case FIELD_DEVICE_CODE: + putNonNull(deviceCode, tag); + break; + case FIELD_USER_CODE: + putNonNull(userCode, tag); + break; + case FIELD_VERIFICATION_URI: + putNonNull(verificationUri, tag); + break; + case FIELD_VERIFICATION_URI_COMPLETE: + putNonNull(verificationUriComplete, tag); + break; + case FIELD_EXPIRES_IN: + expiresIn = parseIntOrZero(tag); + break; + case FIELD_INTERVAL: + interval = parseIntOrZero(tag); + break; + case FIELD_ERROR: + putNonNull(error, tag); + break; + case FIELD_ERROR_DESCRIPTION: + putNonNull(errorDescription, tag); + break; + default: + break; + } + } + field = FIELD_NONE; + break; + default: + break; + } + } + } + + private static final class Endpoint { + final String host; + final boolean isTls; + final String path; + final int port; + + private Endpoint(String host, int port, String path, boolean isTls) { + this.host = host; + this.port = port; + this.path = path; + this.isTls = isTls; + } + + static Endpoint parse(String url) { + if (url == null) { + throw new OidcAuthException("url is required"); + } + // Reject control characters, whitespace and display-unsafe code points anywhere in the url + // before it is split or used. A smuggled CR/LF (or other control char) in the host corrupts the + // outbound Host header; in the path or query it injects into the HTTP request line (postForm + // sends the path verbatim via .url(endpoint.path)) - a request-smuggling / header-injection + // vector when the url comes from a tampered /settings or discovery document. A bidi, zero-width + // or other format char (isUnsafeForDisplay, scanned per code point so a supplementary-plane one + // is not missed) reorders, hides or forges text when the url is echoed into a log line or the + // parse errors below. Rejecting up front keeps the raw url safe on the wire and on screen. + for (int i = 0, n = url.length(); i < n; ) { + final int cp = url.codePointAt(i); + if (cp <= ' ' || OidcAuthException.isUnsafeForDisplay(cp)) { + throw new OidcAuthException().put("invalid url, it contains an illegal character [url=").put(sanitizeForDisplay(url)).put(']'); + } + i += Character.charCount(cp); + } + // reject a fragment (#...): it has no meaning in an endpoint url (RFC 3986 fragments are client-side + // only, never sent to a server), and folding it into the path opens a pin-bypass - pathOnly() strips + // it before the issuer-path check while postForm sends endpoint.path verbatim on the wire, so a + // lenient server that normalizes a '..' hidden past the '#' (POST /realms/acme#/../other/token) could + // resolve the request-target to a path the issuer-path pin never validated. Fail closed instead. + if (url.indexOf('#') >= 0) { + throw new OidcAuthException().put("invalid url, a fragment (#) is not supported [url=").put(url).put(']'); + } + // reject a query (?...) for the same pin-bypass reason as the fragment above: pathOnly() strips it + // before the issuer-path check, yet postForm sends endpoint.path - query included - verbatim on the + // wire, so a tampered /settings could advertise an endpoint carrying a query the issuer-path pin + // never validated (and a lenient server could even normalize a '..' hidden past the '?'). An OIDC + // device/token endpoint carries its parameters in the request body (application/x-www-form-urlencoded), + // never the url query - RFC 6749 3.2 permits a query component but no real provider uses one here - so + // fail closed. The user-facing verification url, which legitimately carries the user code as a query, + // is parsed by BrowserLauncher (java.net.URI), not this method, so it is unaffected. + if (url.indexOf('?') >= 0) { + throw new OidcAuthException().put("invalid url, a query (?) is not supported [url=").put(url).put(']'); + } + int schemeEnd = url.indexOf("://"); + if (schemeEnd < 0) { + throw new OidcAuthException().put("invalid url, expected a scheme [url=").put(url).put(']'); + } + boolean isTls; + // lower-case the scheme before matching: RFC 3986 schemes are case-insensitive, so HTTPS/Http are + // valid. toLowerCase(Locale.ROOT) folds only ASCII case, so a homoglyph scheme (a long-s for the s, + // say) does NOT fold onto http/https and still falls through to the reject below. + String scheme = url.substring(0, schemeEnd).toLowerCase(Locale.ROOT); + if ("https".equals(scheme)) { + isTls = true; + } else if ("http".equals(scheme)) { + isTls = false; + } else { + throw new OidcAuthException().put("invalid url, expected http or https [url=").put(url).put(']'); + } + int hostStart = schemeEnd + 3; + // the authority ([userinfo@]host[:port]) ends at the first '/', or at the end of the url for a + // path-less endpoint. A ?query or #fragment was already rejected above, so neither can fold into the + // host or the path here (this used to also split on '?'/'#' to guard that, now handled up front). + int authorityEnd = url.length(); + for (int i = hostStart, n = url.length(); i < n; i++) { + if (url.charAt(i) == '/') { + authorityEnd = i; + break; + } + } + String hostPort = url.substring(hostStart, authorityEnd); + // a path-less url uses '/'; otherwise the authority is '/'-terminated and the path starts at that + // slash, which already carries its own leading slash + String path = authorityEnd == url.length() ? "/" : url.substring(authorityEnd); + if (hostPort.indexOf('@') >= 0) { + // userinfo (user[:pass]@host) is unsupported: the HTTP layer would connect to the literal + // "user@host". Reject it clearly rather than mis-resolve it or surface a misleading port error + throw new OidcAuthException().put("invalid url, userinfo (user@host) is not supported [url=").put(url).put(']'); + } + if (hostPort.startsWith("[")) { + // bracketed IPv6 literal: the client's HTTP layer does not bracket the Host header, so + // reject it clearly rather than mis-parse it on a ':' inside the address + throw new OidcAuthException().put("invalid url, IPv6 literal hosts are not supported [url=").put(url).put(']'); + } + int colon = hostPort.indexOf(':'); + String host; + int port; + if (colon >= 0) { + host = hostPort.substring(0, colon); + String portStr = hostPort.substring(colon + 1); + // reject a leading '+': Integer.parseInt would read ":+443" as 443 and slip the range check, + // but a real authority port is bare digits. A leading '-' or any non-digit still flows to + // parseInt below, which rejects it (a negative fails the 1..65535 range check, a non-number + // throws NumberFormatException) - so only the '+' that parseInt silently accepts is caught here + if (portStr.isEmpty() || portStr.charAt(0) == '+') { + throw new OidcAuthException().put("invalid url, could not parse the port [url=").put(url).put(']'); + } + try { + port = Integer.parseInt(portStr); + } catch (NumberFormatException e) { + throw new OidcAuthException().put("invalid url, could not parse the port [url=").put(url).put(']'); + } + if (port < 1 || port > 65535) { + throw new OidcAuthException().put("invalid url, the port must be between 1 and 65535 [url=").put(url).put(']'); + } + } else { + host = hostPort; + port = isTls ? 443 : 80; + } + if (host.isEmpty()) { + throw new OidcAuthException().put("invalid url, the host is empty [url=").put(url).put(']'); + } + // reject a non-ASCII host. The HTTP layer hands the host to the OS resolver as raw UTF-8 with no + // IDNA, so a non-ASCII name would not resolve anyway; and a non-ASCII code point makes the origin-pin + // host compare (isSameOrigin -> String.equalsIgnoreCase) unsafe, because equalsIgnoreCase folds + // several non-ASCII letters (U+0130, U+0131, U+017F, U+212A, ...) onto ASCII - so a homoglyph host + // advertised by a tampered /settings could otherwise pass the pin against the issuer. LDH ASCII + // hosts, punycode (xn--...) and dotted IPv4 are all ASCII and unaffected. + for (int i = 0, n = host.length(); i < n; i++) { + char hc = host.charAt(i); + if (hc > 0x7f) { + throw new OidcAuthException().put("invalid url, the host contains a non-ASCII character [url=").put(url).put(']'); + } + // reject a backslash in the host: the WHATWG URL spec folds '\' to '/', so a host like + // good.com\.evil.com could be re-split by a lenient consumer into a different authority. The OS + // resolver this client hands the host to never resolves such a name anyway, so fail closed. + if (hc == '\\') { + throw new OidcAuthException().put("invalid url, the host contains a backslash [url=").put(url).put(']'); + } + } + return new Endpoint(host, port, path, isTls); + } + } + + private static final class SettingsDiscoveryParser implements JsonParser { + private static final int FIELD_AUDIENCE = 7; + private static final int FIELD_CLIENT_ID = 2; + private static final int FIELD_DEVICE_AUTHORIZATION_ENDPOINT = 5; + private static final int FIELD_ENABLED = 1; + private static final int FIELD_GROUPS_IN_TOKEN = 6; + private static final int FIELD_NONE = 0; + private static final int FIELD_SCOPE = 3; + private static final int FIELD_TOKEN_ENDPOINT = 4; + final StringSink audience = new StringSink(); + final StringSink clientId = new StringSink(); + final StringSink deviceAuthorizationEndpoint = new StringSink(); + final StringSink scope = new StringSink(); + final StringSink tokenEndpoint = new StringSink(); + boolean groupsInToken; + boolean isOidcEnabled; + // objects nested inside a JSON array are never trusted config: track array depth and require it 0 for + // every name/value/config-arming decision, so a tampered {"config":[{...}]} (or a top-level array + // wrapper) cannot surface the array element's object at the config depth. Array VALUES are ignored + // regardless; legitimate array-valued config keys (never read here) are harmlessly skipped. + private int arrayDepth; + private int depth; + private int field = FIELD_NONE; + private boolean isConfigNext; + private boolean isInConfig; + + @Override + public void onEvent(int code, CharSequence tag, int position) { + switch (code) { + case JsonLexer.EVT_ARRAY_START: + arrayDepth++; + break; + case JsonLexer.EVT_ARRAY_END: + arrayDepth--; + break; + case JsonLexer.EVT_OBJ_START: + depth++; + if (arrayDepth == 0 && depth == 2 && isConfigNext) { + isInConfig = true; + } + isConfigNext = false; + break; + case JsonLexer.EVT_OBJ_END: + if (depth == 2) { + isInConfig = false; + } + depth--; + break; + case JsonLexer.EVT_NAME: + if (arrayDepth == 0 && depth == 1) { + // only the top-level "config" object is trusted; the sibling "preferences" object + // holds arbitrary user-written keys and must not feed OIDC discovery + isConfigNext = Chars.equals("config", tag); + field = FIELD_NONE; + } else if (arrayDepth == 0 && depth == 2 && isInConfig) { + if (Chars.equals("acl.oidc.enabled", tag)) { + field = FIELD_ENABLED; + } else if (Chars.equals("acl.oidc.client.id", tag)) { + field = FIELD_CLIENT_ID; + } else if (Chars.equals("acl.oidc.scope", tag)) { + field = FIELD_SCOPE; + } else if (Chars.equals("acl.oidc.token.endpoint", tag)) { + field = FIELD_TOKEN_ENDPOINT; + } else if (Chars.equals("acl.oidc.device.authorization.endpoint", tag)) { + field = FIELD_DEVICE_AUTHORIZATION_ENDPOINT; + } else if (Chars.equals("acl.oidc.groups.encoded.in.token", tag)) { + field = FIELD_GROUPS_IN_TOKEN; + } else if (Chars.equals("acl.oidc.audience", tag)) { + field = FIELD_AUDIENCE; + } else { + field = FIELD_NONE; + } + } else { + field = FIELD_NONE; + } + break; + case JsonLexer.EVT_VALUE: + if (arrayDepth == 0 && depth == 2 && isInConfig) { + switch (field) { + case FIELD_ENABLED: + isOidcEnabled = Chars.equals("true", tag); + break; + case FIELD_CLIENT_ID: + putNonNull(clientId, tag); + break; + case FIELD_SCOPE: + putNonNull(scope, tag); + break; + case FIELD_TOKEN_ENDPOINT: + putNonNull(tokenEndpoint, tag); + break; + case FIELD_DEVICE_AUTHORIZATION_ENDPOINT: + putNonNull(deviceAuthorizationEndpoint, tag); + break; + case FIELD_GROUPS_IN_TOKEN: + groupsInToken = Chars.equals("true", tag); + break; + case FIELD_AUDIENCE: + putNonNull(audience, tag); + break; + default: + break; + } + } + field = FIELD_NONE; + break; + default: + break; + } + } + } + + private static final class TokenResponseParser implements JsonParser, Mutable { + private static final int FIELD_ACCESS_TOKEN = 1; + private static final int FIELD_ERROR = 6; + private static final int FIELD_ERROR_DESCRIPTION = 7; + private static final int FIELD_EXPIRES_IN = 4; + private static final int FIELD_ID_TOKEN = 2; + private static final int FIELD_NONE = 0; + private static final int FIELD_REFRESH_TOKEN = 3; + final StringSink accessToken = new StringSink(); + final StringSink error = new StringSink(); + final StringSink errorDescription = new StringSink(); + final StringSink idToken = new StringSink(); + final StringSink refreshToken = new StringSink(); + int expiresIn; + // objects nested inside a JSON array are never trusted: an array-wrapped response must not surface its + // element object's fields at the top-level depth. arrayDepth gates every name/value read on being 0. + private int arrayDepth; + private int depth; + private int field = FIELD_NONE; + + void wipe() { + // clear() rewinds; this overwrites. These five sinks hold the raw grant: the access token, the id + // token and the refresh token, exactly as the identity provider sent them. + accessToken.wipe(); + error.wipe(); + errorDescription.wipe(); + idToken.wipe(); + refreshToken.wipe(); + clear(); + } + + @Override + public void clear() { + accessToken.clear(); + error.clear(); + errorDescription.clear(); + idToken.clear(); + refreshToken.clear(); + expiresIn = 0; + arrayDepth = 0; + depth = 0; + field = FIELD_NONE; + } + + @Override + public void onEvent(int code, CharSequence tag, int position) { + switch (code) { + case JsonLexer.EVT_ARRAY_START: + arrayDepth++; + break; + case JsonLexer.EVT_ARRAY_END: + arrayDepth--; + break; + case JsonLexer.EVT_OBJ_START: + depth++; + break; + case JsonLexer.EVT_OBJ_END: + depth--; + break; + case JsonLexer.EVT_NAME: + if (arrayDepth == 0 && depth == 1) { + if (Chars.equals("access_token", tag)) { + field = FIELD_ACCESS_TOKEN; + } else if (Chars.equals("id_token", tag)) { + field = FIELD_ID_TOKEN; + } else if (Chars.equals("refresh_token", tag)) { + field = FIELD_REFRESH_TOKEN; + } else if (Chars.equals("expires_in", tag)) { + field = FIELD_EXPIRES_IN; + } else if (Chars.equals("error", tag)) { + field = FIELD_ERROR; + } else if (Chars.equals("error_description", tag)) { + field = FIELD_ERROR_DESCRIPTION; + } else { + field = FIELD_NONE; + } + } + break; + case JsonLexer.EVT_VALUE: + if (arrayDepth == 0 && depth == 1) { + switch (field) { + case FIELD_ACCESS_TOKEN: + putNonNull(accessToken, tag); + break; + case FIELD_ID_TOKEN: + putNonNull(idToken, tag); + break; + case FIELD_REFRESH_TOKEN: + putNonNull(refreshToken, tag); + break; + case FIELD_EXPIRES_IN: + expiresIn = parseIntOrZero(tag); + break; + case FIELD_ERROR: + putNonNull(error, tag); + break; + case FIELD_ERROR_DESCRIPTION: + putNonNull(errorDescription, tag); + break; + default: + break; + } + } + field = FIELD_NONE; + break; + default: + break; + } + } + } + + private static final class WellKnownDiscoveryParser implements JsonParser { + private static final int FIELD_DEVICE_AUTHORIZATION_ENDPOINT = 1; + private static final int FIELD_ISSUER = 2; + private static final int FIELD_NONE = 0; + private static final int FIELD_TOKEN_ENDPOINT = 3; + final StringSink deviceAuthorizationEndpoint = new StringSink(); + final StringSink issuer = new StringSink(); + final StringSink tokenEndpoint = new StringSink(); + // objects nested inside a JSON array are never trusted: an array-wrapped document must not surface its + // element object's fields at the top-level depth. arrayDepth gates every name/value read on being 0. + private int arrayDepth; + private int depth; + private int field = FIELD_NONE; + + @Override + public void onEvent(int code, CharSequence tag, int position) { + switch (code) { + case JsonLexer.EVT_ARRAY_START: + arrayDepth++; + break; + case JsonLexer.EVT_ARRAY_END: + arrayDepth--; + break; + case JsonLexer.EVT_OBJ_START: + depth++; + break; + case JsonLexer.EVT_OBJ_END: + depth--; + break; + case JsonLexer.EVT_NAME: + // the OIDC discovery document is a flat top-level object; only read top-level keys so a + // nested value cannot be mistaken for an endpoint + if (arrayDepth == 0 && depth == 1) { + if (Chars.equals("device_authorization_endpoint", tag)) { + field = FIELD_DEVICE_AUTHORIZATION_ENDPOINT; + } else if (Chars.equals("issuer", tag)) { + field = FIELD_ISSUER; + } else if (Chars.equals("token_endpoint", tag)) { + field = FIELD_TOKEN_ENDPOINT; + } else { + field = FIELD_NONE; + } + } + break; + case JsonLexer.EVT_VALUE: + if (arrayDepth == 0 && depth == 1) { + switch (field) { + case FIELD_DEVICE_AUTHORIZATION_ENDPOINT: + putNonNull(deviceAuthorizationEndpoint, tag); + break; + case FIELD_ISSUER: + putNonNull(issuer, tag); + break; + case FIELD_TOKEN_ENDPOINT: + putNonNull(tokenEndpoint, tag); + break; + default: + break; + } + } + field = FIELD_NONE; + break; + default: + break; + } + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/PersistedToken.java b/core/src/main/java/io/questdb/client/cutlass/auth/PersistedToken.java new file mode 100644 index 000000000..91c73770f --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/PersistedToken.java @@ -0,0 +1,71 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +/** + * An immutable snapshot of the token state an {@link OidcDeviceAuth} holds, passed to and from a + * {@link TokenStore} so the device flow does not have to be re-run after a process restart. Mirrors + * the in-memory fields: the access token, the id token, the refresh token, the absolute wall-clock + * expiry of the access/id token, and the (clamped) lifetime that expiry was derived from. + *

+ * Any of the three token strings may be {@code null}. {@link #getExpiresAtMillis()} is an absolute + * {@code System.currentTimeMillis()} value, so it remains meaningful across a restart (unlike a + * monotonic clock reading). + */ +public final class PersistedToken { + private final String accessToken; + private final long expiresAtMillis; + private final String idToken; + private final String refreshToken; + private final long tokenTtlMillis; + + public PersistedToken(String accessToken, String idToken, String refreshToken, long expiresAtMillis, long tokenTtlMillis) { + this.accessToken = accessToken; + this.idToken = idToken; + this.refreshToken = refreshToken; + this.expiresAtMillis = expiresAtMillis; + this.tokenTtlMillis = tokenTtlMillis; + } + + public String getAccessToken() { + return accessToken; + } + + public long getExpiresAtMillis() { + return expiresAtMillis; + } + + public String getIdToken() { + return idToken; + } + + public String getRefreshToken() { + return refreshToken; + } + + public long getTokenTtlMillis() { + return tokenTtlMillis; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenStore.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenStore.java new file mode 100644 index 000000000..69b5880cf --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenStore.java @@ -0,0 +1,145 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +/** + * Persists the token state of an {@link OidcDeviceAuth} so a restarted process can resume from a saved + * refresh token instead of running the interactive device flow again. Persistence is opt-in: an + * {@code OidcDeviceAuth} with no store keeps its tokens in memory only (the previous behaviour). + *

+ * The default implementation is {@link FileTokenStore} (a strict-permissions file under the user's home + * directory). Supply your own to back persistence with an OS keychain, a secrets manager, or a vault - + * for example to encrypt the refresh token at rest, which the file store does not do. + *

+ * Entries are keyed by {@link TokenStoreKey} (the non-secret identity: endpoints, client id, scope, + * audience, groups-in-token mode), so a token minted for one identity is never returned for another. + * {@code TokenStoreKey} is a value type - it implements {@code equals}/{@code hashCode} over that + * identity - so an implementation may hold its entries in a {@code Map} keyed by it directly. A store + * that needs a stable name instead (a file, a keychain entry, a row id) should use + * {@link TokenStoreKey#hash()}, which is the same identity as an opaque hex string and is stable across + * processes and across QuestDB's client implementations in other languages. + * Calls are made while {@code OidcDeviceAuth} holds its own instance lock, so an implementation does not + * need to be thread-safe against concurrent calls from one {@code OidcDeviceAuth} instance; it does, + * however, share its backing storage with other processes (and other language clients), so it must keep a + * concurrent reader from observing a half-written entry - see {@link FileTokenStore} for how the file + * store does that and coordinates a rotating refresh token across processes. + *

+ * A store reports a failure by throwing; {@code OidcDeviceAuth} treats persistence as best-effort and a + * thrown failure as non-fatal - it logs a warning through SLF4J at WARN and continues with the in-memory + * token, which is valid regardless of whether it could be saved. + */ +public interface TokenStore { + /** + * Removes any persisted entry for this identity. Called from {@link OidcDeviceAuth#clearCache()}. + * A no-op when nothing is stored. + * + * @param key the identity whose entry to remove + */ + void clear(TokenStoreKey key); + + /** + * Runs {@code action} while holding a cross-process lock scoped to {@code key}, so a refresh by + * another process sharing this identity is observed rather than raced. The action re-reads the store + * inside the lock and refreshes only if still needed, which keeps a rotating refresh token consistent + * across processes. + *

+ * The default runs {@code action} with no locking, which is correct for a single process or a + * non-rotating refresh token; {@link FileTokenStore} overrides it with a lock-file protocol. An + * implementation that cannot acquire the lock should run {@code action} anyway (degrade) rather than + * fail a sign-in. + *

+ * {@code OidcDeviceAuth} calls this while holding its own instance lock, and {@code action} runs + * synchronously on the calling thread. An implementation therefore must not call back into the owning + * {@code OidcDeviceAuth} (for example {@code signIn()}/{@code getToken()}) from {@code inLock}, + * {@code load}, or {@code save}, and must not block waiting on another thread that could need that + * instance lock - either would re-enter or deadlock. Do the store I/O only. + *

+ * Like {@code load} and {@code save}, this is BEST-EFFORT: an implementation that throws must not be + * able to fail a sign-in the caller could otherwise complete. {@code OidcDeviceAuth} therefore degrades + * on a throw rather than propagating it, and what it does depends on whether {@code action} ran: a + * throw before the action runs falls back to a single uncoordinated refresh, while a throw after the + * action completed - releasing a lock, closing a handle - keeps the action's result, because the + * refresh already happened and re-running it is the duplicate POST of a rotating refresh token this + * lock exists to prevent. An exception from {@code action} itself is the caller's own and propagates + * untouched. An implementation should still absorb its own bookkeeping failures and degrade to running + * {@code action} unlocked, rather than lean on that fallback. + *

+ * An implementation that waits for its lock must make that wait INTERRUPTIBLE and, on an interrupt, + * return {@code false} without running {@code action} AND leave the thread's interrupt flag SET. The + * wait can outlast the caller's own shutdown budget - QWP's connect cancellation interrupts a thread + * stuck in a credential pull precisely so {@code close()} can reclaim its native resources - and an + * uninterruptible wait defeats that, leaving the client, the cursor engine and the store-and-forward + * slot lock to a delegated teardown. + *

+ * Restoring the flag is not optional politeness. {@code OidcDeviceAuth} distinguishes "the refresh ran + * and failed" from "the wait was abandoned" by recording whether {@code action} was entered, because an + * interrupt may independently arrive while a real refresh is running. It still needs the preserved flag + * to report why a no-action return occurred and, more importantly, the caller owns that cancellation + * signal. An implementation that consumes the interrupt (as {@code InterruptedException} does) must + * re-assert it with {@code Thread.currentThread().interrupt()} before returning, once it is past any + * interruptible I/O of its own. + * + * @param key the identity to lock + * @param action the critical section; its boolean result is returned unchanged + * @return whatever {@code action} returned, or {@code false} if an interrupt abandoned the wait before + * {@code action} could run + */ + default boolean inLock(TokenStoreKey key, CriticalSection action) { + return action.run(); + } + + /** + * Loads the persisted token for this identity, or returns {@code null} if there is none usable (no + * entry, or an entry that does not match {@code key}, or one that cannot be read as a valid token). + * A {@code null} return makes {@code OidcDeviceAuth} fall back to a refresh or an interactive sign-in, + * so an unreadable or stale entry is recoverable rather than fatal. + *

+ * Returning {@code null} is the definitive answer, and ends the reads for the life of that + * {@code OidcDeviceAuth}. Throwing is not: it reads as a transient fault and is retried - immediately + * once, then behind a back-off that grows to a minute, so an implementation that can never succeed is + * not re-entered on every {@code getToken()} call (which an ILP producer makes once per flush). + * + * @param key the identity to load + * @return the persisted token, or {@code null} + */ + PersistedToken load(TokenStoreKey key); + + /** + * Persists (atomically replaces) the token for this identity. + * + * @param key the identity to store under + * @param token the token state to persist + */ + void save(TokenStoreKey key, PersistedToken token); + + /** + * A unit of work {@link #inLock(TokenStoreKey, CriticalSection)} runs while holding the per-identity + * lock. Returns whether a valid token resulted. + */ + @FunctionalInterface + interface CriticalSection { + boolean run(); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenStoreKey.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenStoreKey.java new file mode 100644 index 000000000..b30489e8c --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenStoreKey.java @@ -0,0 +1,198 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; + +/** + * The non-secret identity a persisted token belongs to: the client id, the (canonicalised) token and + * device-authorization endpoints, the scope, the optional audience, and whether the server expects + * groups encoded in the token. A {@link TokenStore} keys its entries by this so a token minted for one + * server / identity provider / scope / audience is never served to a process configured for another. + *

+ * {@link #hash()} is a stable, lowercase-hex SHA-256 over a canonical, NUL-separated rendering of the + * fields - intended as a file name (or any opaque key) that is identical across client implementations + * (the Python client mirrors this), so several processes - and languages - sharing one identity address + * the same persisted entry. The fields themselves are exposed (they are not secret) so a store can also + * record them and re-check them on load as a defence against a hash collision or a copied file. + */ +public final class TokenStoreKey { + // the canonical-string prefix doubles as a domain tag and a schema version, so a future format change + // produces a different hash (and hence a different file) rather than silently colliding with v1 entries + private static final String CANONICAL_PREFIX = "questdb-oidc-token-v1"; + private static final char[] HEX = "0123456789abcdef".toCharArray(); + private final String audience; + private final String clientId; + private final String deviceAuthorizationEndpoint; + private final boolean groupsInToken; + private final String hash; + private final String scope; + private final String tokenEndpoint; + + /** + * @param clientId the OIDC client id + * @param tokenEndpoint the canonical token endpoint ({@code scheme://host:port/path}, + * scheme and host lower-cased, port explicit) + * @param deviceAuthorizationEndpoint the canonical device-authorization endpoint, same form + * @param scope the requested scope + * @param audience the audience, or {@code null} if none + * @param groupsInToken whether the id token (rather than the access token) is served + */ + public TokenStoreKey( + String clientId, + String tokenEndpoint, + String deviceAuthorizationEndpoint, + String scope, + String audience, + boolean groupsInToken + ) { + // the identity fields are required; reject a null up front with a clear error rather than letting it + // surface later as a raw NullPointerException deep inside a TokenStore's serialize/fingerprint path + if (clientId == null || tokenEndpoint == null || deviceAuthorizationEndpoint == null || scope == null) { + throw new OidcAuthException( + "clientId, tokenEndpoint, deviceAuthorizationEndpoint and scope are required for a token store key"); + } + this.clientId = clientId; + this.tokenEndpoint = tokenEndpoint; + this.deviceAuthorizationEndpoint = deviceAuthorizationEndpoint; + this.scope = scope; + // normalise an empty audience to null so getAudience(), hash() (which already folds null and "" together + // via nullToEmpty), and a TokenStore's save/load round-trip all agree that an absent audience is null - + // matching how OidcDeviceAuth builds the key + this.audience = audience != null && !audience.isEmpty() ? audience : null; + this.groupsInToken = groupsInToken; + this.hash = computeHash(clientId, tokenEndpoint, deviceAuthorizationEndpoint, scope, this.audience, groupsInToken); + } + + /** + * Value equality over the identity this key names, so a {@link TokenStore} may hold its entries in a + * {@code Map} keyed by this type - which the {@link TokenStore} contract ("entries are keyed by + * {@link TokenStoreKey}") invites, and which identity equality would silently defeat: {@code + * OidcDeviceAuth} builds its key once per instance, so a Map-backed store appears to work until a + * second instance or a restart rebuilds an equal key and misses, sending the user back through the + * device flow on every refresh. + *

+ * Compares {@link #hash()} rather than the fields one by one, so equality means exactly "the same + * store entry": the hash folds every identity field through the same null-vs-empty normalization the + * constructor applies, so two keys that address one entry are equal here even when their raw + * arguments differed in that respect. + */ + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof TokenStoreKey)) { + return false; + } + return hash.equals(((TokenStoreKey) o).hash); + } + + public String getAudience() { + return audience; + } + + public String getClientId() { + return clientId; + } + + public String getDeviceAuthorizationEndpoint() { + return deviceAuthorizationEndpoint; + } + + public String getScope() { + return scope; + } + + public String getTokenEndpoint() { + return tokenEndpoint; + } + + /** + * @return a stable lowercase-hex SHA-256 of the canonical identity string; suitable as an opaque file + * name. Identical inputs (across processes and language implementations) yield an identical hash. + */ + public String hash() { + return hash; + } + + /** + * Consistent with {@link #equals(Object)}: both derive from {@link #hash()}, which is a pure function + * of the identity fields. + */ + @Override + public int hashCode() { + return hash.hashCode(); + } + + public boolean isGroupsInToken() { + return groupsInToken; + } + + private static String computeHash( + String clientId, + String tokenEndpoint, + String deviceAuthorizationEndpoint, + String scope, + String audience, + boolean groupsInToken + ) { + // NUL-separate the fields so no field value can be confused with a separator; an OAuth client id, + // url, scope or audience never contains a NUL. The prefix tags the domain and schema version. + StringBuilder canonical = new StringBuilder(); + canonical.append(CANONICAL_PREFIX).append('\0') + .append(nullToEmpty(clientId)).append('\0') + .append(nullToEmpty(tokenEndpoint)).append('\0') + .append(nullToEmpty(deviceAuthorizationEndpoint)).append('\0') + .append(nullToEmpty(scope)).append('\0') + .append(nullToEmpty(audience)).append('\0') + .append(groupsInToken ? '1' : '0'); + try { + MessageDigest digest = MessageDigest.getInstance("SHA-256"); + byte[] bytes = digest.digest(canonical.toString().getBytes(StandardCharsets.UTF_8)); + return toHex(bytes); + } catch (NoSuchAlgorithmException e) { + // SHA-256 is mandated on every JVM, so this is unreachable; rethrow defensively rather than + // declare a checked exception across the whole construction path + throw new OidcAuthException(e).put("SHA-256 is not available to key the OIDC token store"); + } + } + + private static String nullToEmpty(String s) { + return s != null ? s : ""; + } + + private static String toHex(byte[] bytes) { + char[] out = new char[bytes.length * 2]; + for (int i = 0; i < bytes.length; i++) { + int v = bytes[i] & 0xff; + out[i * 2] = HEX[v >>> 4]; + out[i * 2 + 1] = HEX[v & 0x0f]; + } + return new String(out); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractChunkedResponse.java b/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractChunkedResponse.java index ee9995596..0e29dcad6 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractChunkedResponse.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractChunkedResponse.java @@ -35,6 +35,11 @@ */ public abstract class AbstractChunkedResponse implements Response, Fragment { private final static int CRLF_LEN = 2; + // Most hex digits a chunk-size line may carry once leading zeros are stripped. 16^15 - 1 is about + // 1.15e18 and always fits a long; a 16th digit can push it past Long.MAX_VALUE and wrap. The + // smallest thing this turns away is a 2^60-byte (1 EiB) chunk, so it costs no real server + // anything, and it rejects absurd-but-representable sizes that a mere overflow check admits. + private static final int MAX_CHUNK_SIZE_HEX_DIGITS = 15; private static final int STATE_CHUNK_DATA = 1; private static final int STATE_CHUNK_DATA_END = 2; private static final int STATE_CHUNK_SIZE = 0; @@ -90,11 +95,32 @@ public long lo() { return dataAddr; } + @Override public Fragment recv(int timeout) { + // A positive timeout bounds the whole call, not each socket read. This loop re-reads while a + // chunk-size line (or the chunk-data-end CRLF) is incomplete, so without one shared deadline a server + // dribbling those bytes - one per timeout window - would run a single recv() for (line length) x + // timeout and defeat a caller's elapsed-time bound (e.g. OidcDeviceAuth.parseBody). A non-positive + // timeout keeps the legacy "no bound" behaviour. + final boolean bounded = timeout > 0; + final long startNanos = bounded ? System.nanoTime() : 0L; while (true) { + // Consult the deadline on EVERY pass, not only on the passes that read. A pass that neither + // reads nor advances the state machine re-enters the loop with receive == false and + // dataLo < dataHi, which skips the read gate below - so a deadline checked only inside that + // gate is never reached, and the loop spins without the bound this method promises. Keeping + // the check above the gate makes the bound hold for every pass, however the state machine got + // there. + int callTimeout = timeout; + if (bounded) { + callTimeout = timeout - (int) ((System.nanoTime() - startNanos) / 1_000_000L); + if (callTimeout <= 0) { + throw new HttpClientException("timed out reading the chunked response body"); + } + } if (receive || dataLo == dataHi) { compactBuffer(); - dataHi += recvOrDie(dataHi, bufHi, timeout); + dataHi += recvOrDie(dataHi, bufHi, callTimeout); } long p; // moving data pointer for scanning buffer switch (state) { @@ -126,8 +152,26 @@ public Fragment recv(int timeout) { if (res != -1) { // at this stage we consumed the chunk size end (CRLF) chunkSize.of(dataLo, res + 1); + final CharSequence chunkSizeHex = chunkSize.asAsciiCharSequence(); + // Bound the SIZE LINE here, before parsing it. Numbers.parseHexLong wraps on + // overflow like every other hex-word parser in that class, which is right for a + // general-purpose utility and wrong for a count the peer chose - and the size line + // is chosen by the server, which for an OIDC discovery or token response is + // untrusted. Each residue breaks framing its own way: a negative one + // (8000000000000000 is the smallest) matches neither the "size > 0" data branch nor + // the "size == 0" terminator below, so the state machine loops on it forever; zero + // (10000000000000000) reads as the TERMINAL chunk, truncating the response and + // losing framing for the next keep-alive response on the connection; a positive + // residue frames a short data chunk and mis-reads everything after it. + // + // It has to happen BEFORE the parse, not after: the zero residue is + // indistinguishable from a genuine 0 once the high bits are gone, so no check on + // the returned value can catch the worst of the three. + if (isChunkSizeTooLong(chunkSizeHex)) { + throw new HttpClientException("malformed chunk size"); + } try { - size = Numbers.parseHexLong(chunkSize.asAsciiCharSequence()); + size = Numbers.parseHexLong(chunkSizeHex); consumed = 0; // consume data buffer ignoring chunk size value and its furniture state = STATE_CHUNK_DATA; @@ -230,6 +274,22 @@ private byte getByte(long addr) { return Unsafe.getUnsafe().getByte(addr); } + /** + * Whether a chunk-size line carries more significant hex digits than a long can hold. + *

+ * Counts SIGNIFICANT digits, skipping leading zeros: {@code 0000000000000001} is sixteen characters + * and a perfectly ordinary size, so a raw length check would reject legitimate input from a server + * that pads. + */ + private static boolean isChunkSizeTooLong(CharSequence hex) { + final int n = hex.length(); + int i = 0; + while (i < n && hex.charAt(i) == '0') { + i++; + } + return n - i > MAX_CHUNK_SIZE_HEX_DIGITS; + } + /** * Receives data into the buffer or throws an exception. * diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractResponse.java b/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractResponse.java index b3b521daf..589b2e405 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractResponse.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/AbstractResponse.java @@ -58,6 +58,7 @@ public long lo() { return dataLo; } + @Override public Fragment recv(int timeout) { if (bytesReceived >= contentLength) { return null; @@ -65,9 +66,24 @@ public Fragment recv(int timeout) { if (receive) { dataLo = bufLo; dataHi = bufLo; + // A positive timeout bounds the whole call, not each socket read. recvOrDie can return 0 + // without consuming the full timeout - e.g. an incomplete TLS record that decrypts to no + // application bytes - so without one shared deadline a server dribbling such reads would + // re-arm the full timeout on every iteration and keep this recv() running without bound, + // defeating a caller's elapsed-time bound (e.g. OidcDeviceAuth.parseBody). A non-positive + // timeout keeps the legacy "no bound" behaviour. + final boolean bounded = timeout > 0; + final long startNanos = bounded ? System.nanoTime() : 0L; int len = 0; while (len == 0) { - len = recvOrDie(dataHi, bufHi, timeout); + int callTimeout = timeout; + if (bounded) { + callTimeout = timeout - (int) ((System.nanoTime() - startNanos) / 1_000_000L); + if (callTimeout <= 0) { + throw new HttpClientException("timed out reading the response body"); + } + } + len = recvOrDie(dataHi, bufHi, callTimeout); } dataHi += len; } diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClient.java b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClient.java index 0175ad6c9..941dc58ff 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClient.java @@ -83,16 +83,40 @@ public abstract class HttpClient implements QuietCloseable { public HttpClient(HttpClientConfiguration configuration, SocketFactory socketFactory) { this.nf = configuration.getNetworkFacade(); - this.socket = socketFactory.newInstance(nf, LOG); this.defaultTimeout = configuration.getTimeout(); this.connectTimeout = configuration.getConnectTimeout(); this.bufferSize = configuration.getInitialRequestBufferSize(); this.maxBufferSize = configuration.getMaximumRequestBufferSize(); this.responseParserBufSize = configuration.getResponseBufferSize(); this.fixBrokenConnection = configuration.fixBrokenConnection(); - this.bufLo = Unsafe.malloc(bufferSize, MemoryTag.NATIVE_DEFAULT); - this.responseParserBufLo = Unsafe.malloc(responseParserBufSize, MemoryTag.NATIVE_DEFAULT); - this.responseHeaders = new ResponseHeaders(responseParserBufLo, responseParserBufSize, defaultTimeout, 4096, csPool); + // Stage every acquisition and roll the lot back on any throw. A constructor that fails partway + // leaves an object nobody can close: it never reaches the caller, so no finally, no try-with-resources + // and no close() ever runs on it, and whatever it had already taken is lost for the life of the + // process. The two mallocs and ResponseHeaders' own buffer are native, so the loss is native memory, + // and the trigger is the same condition that makes these fail in the first place - memory pressure, or + // fd exhaustion in the socket factory. Retrying then compounds it. Kqueue already guards its + // constructor this way; this one did not. + Socket stagedSocket = null; + long stagedBufLo = 0; + long stagedResponseParserBufLo = 0; + try { + stagedSocket = socketFactory.newInstance(nf, LOG); + stagedBufLo = Unsafe.malloc(bufferSize, MemoryTag.NATIVE_DEFAULT); + stagedResponseParserBufLo = Unsafe.malloc(responseParserBufSize, MemoryTag.NATIVE_DEFAULT); + this.responseHeaders = new ResponseHeaders(stagedResponseParserBufLo, responseParserBufSize, defaultTimeout, 4096, csPool); + } catch (Throwable t) { + if (stagedResponseParserBufLo != 0) { + Unsafe.free(stagedResponseParserBufLo, responseParserBufSize, MemoryTag.NATIVE_DEFAULT); + } + if (stagedBufLo != 0) { + Unsafe.free(stagedBufLo, bufferSize, MemoryTag.NATIVE_DEFAULT); + } + Misc.free(stagedSocket); + throw t; + } + this.socket = stagedSocket; + this.bufLo = stagedBufLo; + this.responseParserBufLo = stagedResponseParserBufLo; } @Override @@ -329,8 +353,22 @@ public int getContentLength() { } } + /** + * The address of the request's content section, or {@code 0} when no content section has been + * started - deliberately NOT the {@code -1} sentinel the field carries in that state. + *

+ * Callers pair this with {@link #getContentLength()}, which already reports 0 for the same state, so + * handing back {@code -1} here produced a view that is empty by length but whose base address is a + * non-zero, unusable pointer: a {@code ptr() != 0} test reads as true, and pointer arithmetic on it + * is nonsense. That state became reachable when withContent() started being deferred - an ILP + * request with an httpTokenProvider sits at the header stage until the first row stamps the + * Authorization header - so {@code Sender.bufferView()} returned it between every flush and the next + * row. {@code trimContentToLen} was guarded against the same sentinel; this accessor was not. + * + * @return the content-section address, or 0 when there is no content section + */ public long getContentStart() { - return contentStart; + return contentStart < 0 ? 0 : contentStart; } public long getPtr() { @@ -547,7 +585,27 @@ public String toString() { return ss.toString(); } + /** + * Rewinds the write pointer to {@code contentLen} bytes into the content section, discarding + * whatever was written past it. + *

+ * A request that has not reached {@code withContent()} yet has no content section to rewind, and + * the sentinel guard below is the only thing standing between that state and a SIGSEGV: without it + * the pointer becomes {@code -1 + contentLen} and the next write to the buffer takes the process + * down. That state is ordinary, not exotic - an ILP request with an {@code httpTokenProvider} sits + * at the header stage between every flush and the next row - and {@code Request} is exported, so an + * external caller can reach it too. {@code HttpClientRequestTrimTest} pins it. + * + * @param contentLen the content length to rewind to + */ public void trimContentToLen(int contentLen) { + if (contentStart < 0) { + // withContent() has not started a content section yet, so contentStart is the -1 sentinel + // and contentStart + contentLen would be a negative, invalid write pointer that the next + // write would segfault on. Nothing has been written into a content section, so there is + // nothing to trim. + return; + } ptr = contentStart + contentLen; } @@ -853,9 +911,20 @@ public class ResponseHeaders extends HttpHeaderParser { public ResponseHeaders(long respParserBufLo, int respParserBufSize, int defaultTimeout, int headerBufSize, ObjectPool pool) { super(headerBufSize, pool); - this.defaultTimeout = defaultTimeout; - this.response = new ResponseImpl(respParserBufLo, respParserBufLo + respParserBufSize, defaultTimeout); - this.chunkedResponse = new ChunkedResponseImpl(respParserBufLo, respParserBufLo + respParserBufSize, defaultTimeout); + // super() mallocs the header parse buffer as its FIRST statement, so from here on this object owns + // native memory while still being unreachable by anyone who could free it. A heap OOM in either + // allocation below would strand those bytes past the enclosing constructor's catch (Throwable), + // which frees only what IT staged - it never holds a reference to a ResponseHeaders that failed + // to finish constructing. Same rule as out there: whoever took it frees it when construction + // cannot complete. + try { + this.defaultTimeout = defaultTimeout; + this.response = new ResponseImpl(respParserBufLo, respParserBufLo + respParserBufSize, defaultTimeout); + this.chunkedResponse = new ChunkedResponseImpl(respParserBufLo, respParserBufLo + respParserBufSize, defaultTimeout); + } catch (Throwable t) { + super.close(); // gated on headerPtr != 0, so it is safe and idempotent + throw t; + } } public void await() { @@ -865,8 +934,26 @@ public void await() { public void await(int timeout) { int totalBytesReceived = 0; long unprocessedLo = responseParserBufLo; + // A positive timeout bounds the whole call, not each socket read - the same rule + // AbstractResponse.recv and AbstractChunkedResponse.recv apply to the BODY, and for the same + // reason. recvOrDie returns 0 whenever a read produced no application bytes (an incomplete TLS + // record that decrypts to nothing is the common case, and the IDP endpoints are required to be + // https), and a 0 leaves totalBytesReceived unmoved, so the loop neither advances the header + // parser nor fills its buffer: without one shared deadline it re-arms the full timeout forever + // and never reaches the "header is too large" escape either. That put no bound at all on + // OidcDeviceAuth's postForm/fetchJson, which read this head from an untrusted identity provider + // on the getToken() flush path. A non-positive timeout keeps the legacy "no bound" behaviour. + final boolean bounded = timeout > 0; + final long startNanos = bounded ? System.nanoTime() : 0L; while (isIncomplete()) { - final int len = recvOrDie(responseParserBufLo + totalBytesReceived, timeout); + int callTimeout = timeout; + if (bounded) { + callTimeout = timeout - (int) ((System.nanoTime() - startNanos) / 1_000_000L); + if (callTimeout <= 0) { + throw new HttpClientException("timed out reading the response head"); + } + } + final int len = recvOrDie(responseParserBufLo + totalBytesReceived, callTimeout); if (len > 0) { totalBytesReceived += len; unprocessedLo = parse(unprocessedLo, responseParserBufLo + totalBytesReceived, false, true); diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientLinux.java b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientLinux.java index 472b5257a..198cbf965 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientLinux.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientLinux.java @@ -36,10 +36,18 @@ public class HttpClientLinux extends HttpClient { public HttpClientLinux(HttpClientConfiguration configuration, SocketFactory socketFactory) { super(configuration, socketFactory); - epoll = new Epoll( - configuration.getEpollFacade(), - configuration.getWaitQueueCapacity() - ); + // The base constructor already took a socket and two native buffers. If epoll_create fails here - + // fd exhaustion is exactly when it does - this object never reaches the caller, so nothing ever + // closes it and those stay lost. Roll the base back before rethrowing. + try { + epoll = new Epoll( + configuration.getEpollFacade(), + configuration.getWaitQueueCapacity() + ); + } catch (Throwable t) { + super.close(); + throw t; + } } @Override diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientOsx.java b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientOsx.java index aae49dc3e..980fe9da5 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientOsx.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientOsx.java @@ -35,10 +35,17 @@ public class HttpClientOsx extends HttpClient { public HttpClientOsx(HttpClientConfiguration configuration, SocketFactory socketFactory) { super(configuration, socketFactory); - this.kqueue = new Kqueue( - configuration.getKQueueFacade(), - configuration.getWaitQueueCapacity() - ); + // See HttpClientLinux: a kqueue() failure here would strand the socket and native buffers the base + // constructor already took, on an object nobody can close. + try { + this.kqueue = new Kqueue( + configuration.getKQueueFacade(), + configuration.getWaitQueueCapacity() + ); + } catch (Throwable t) { + super.close(); + throw t; + } } @Override diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientWindows.java b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientWindows.java index 62ee43fe3..e1153f552 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientWindows.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientWindows.java @@ -37,8 +37,20 @@ public class HttpClientWindows extends HttpClient { public HttpClientWindows(HttpClientConfiguration configuration, SocketFactory socketFactory) { super(configuration, socketFactory); - this.fdSet = new FDSet(configuration.getWaitQueueCapacity()); - this.sf = configuration.getSelectFacade(); + // See HttpClientLinux: an allocation failure here would strand the socket and native buffers the + // base constructor already took, on an object nobody can close. + // getSelectFacade() is inside the guard, not after it: the shipped default cannot throw, but this + // takes a caller-supplied HttpClientConfiguration, and an override that does would have stranded the + // FDSet as well as everything the base constructor took. Linux and Osx evaluate every configuration + // getter inside their guard already; this was the odd one out. + try { + this.fdSet = new FDSet(configuration.getWaitQueueCapacity()); + this.sf = configuration.getSelectFacade(); + } catch (Throwable t) { + this.fdSet = Misc.free(fdSet); // null when FDSet itself threw; Misc.free tolerates that + super.close(); + throw t; + } } @Override diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/Response.java b/core/src/main/java/io/questdb/client/cutlass/http/client/Response.java index 166a7a28c..0c7845fc8 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/Response.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/Response.java @@ -29,9 +29,48 @@ */ public interface Response { /** - * Receives the next fragment of response data using the default timeout. + * Receives the next fragment of response data, bounded by this response's default timeout - the + * {@link io.questdb.client.HttpClientConfiguration#getTimeout()} of the client that produced it. + *

+ * The bound {@link #recv(int)} describes applies here too, because the implementations in this library + * implement this method as {@code recv(defaultTimeout)}: it caps the WHOLE call rather than each socket + * read, so a server dribbling the body cannot keep one call running past it. Every configuration this + * library builds supplies a positive timeout - {@code request_timeout} is rejected below 1 on both the + * builder and the configuration-string paths - so the bound is live unless a caller supplies its own + * {@code HttpClientConfiguration} returning a non-positive value, which disables it. + *

+ * Size that timeout against a SINGLE fragment read rather than the whole body: each call starts its own + * budget, so a large body spread over many calls is unaffected, and only one call that cannot complete + * within the timeout aborts. + *

+ * Note the two methods delegate in OPPOSITE directions, which is what decides whether the bound exists + * at all. The {@link #recv(int)} default defers down to this method and discards its argument; the + * implementations here do the reverse. So an implementation overriding only this method is unbounded on + * both, while one extending {@code AbstractResponse} or {@code AbstractChunkedResponse} is bounded on + * both. * - * @return the received fragment + * @return the received fragment, or null once the body has been fully read */ Fragment recv(); + + /** + * Receives the next fragment of response data. A positive {@code timeout} bounds the whole call to that + * many milliseconds in total (not per socket read), so a server dribbling the body one byte at a time + * cannot keep a single call running past it; a non-positive {@code timeout} disables the bound. + *

+ * Defaulted rather than abstract for compatibility: this interface is exported, ships with a javadoc + * jar, and gained {@code recv(int)} after {@link #recv()}, so an implementation written against the + * earlier interface must keep both compiling and linking. The default ignores the bound and defers to + * {@link #recv()} -- precisely what such an implementation did before this overload existed. + *

+ * Every implementation in this library overrides it, and any implementation that wants the bound + * honoured must do the same. An overriding implementation must not then implement {@link #recv()} by + * calling back into this default, which would recurse. + * + * @param timeout the receive timeout in milliseconds + * @return the received fragment, or null once the body has been fully read + */ + default Fragment recv(int timeout) { + return recv(); + } } diff --git a/core/src/main/java/io/questdb/client/cutlass/json/JsonLexer.java b/core/src/main/java/io/questdb/client/cutlass/json/JsonLexer.java index 565a02344..5802e7167 100644 --- a/core/src/main/java/io/questdb/client/cutlass/json/JsonLexer.java +++ b/core/src/main/java/io/questdb/client/cutlass/json/JsonLexer.java @@ -55,10 +55,12 @@ public class JsonLexer implements Mutable, Closeable { private final int cacheSizeLimit; private final IntStack objDepthStack = new IntStack(64); private final StringSink sink = new StringSink(); + private final StringSink unescapeSink = new StringSink(); private int arrayDepth = 0; private long cache; private int cacheCapacity; private int cacheSize = 0; + private boolean hasEscape = false; private boolean ignoreNext = false; private int objDepth = 0; private int position = 0; @@ -85,6 +87,7 @@ public void clear() { arrayDepth = 0; ignoreNext = false; quoted = false; + hasEscape = false; cacheSize = 0; useCache = false; position = 0; @@ -93,6 +96,9 @@ public void clear() { @Override public void close() { if (cacheCapacity > 0 && cache != 0) { + // The stash may contain raw credential bytes from a value split across parse() calls. Do not hand + // those bytes back to the native allocator, where they remain readable until the block is reused. + Vect.memset(cache, cacheCapacity, 0); Unsafe.free(cache, cacheCapacity, MemoryTag.NATIVE_TEXT_PARSER_RSS); cache = 0; } @@ -109,6 +115,7 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { int state = this.state; boolean quoted = this.quoted; boolean ignoreNext = this.ignoreNext; + boolean hasEscape = this.hasEscape; boolean useCache = this.useCache; int objDepth = this.objDepth; int arrayDepth = this.arrayDepth; @@ -125,6 +132,7 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { if (quoted) { if (c == '\\') { ignoreNext = true; + hasEscape = true; continue; } @@ -137,10 +145,10 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { int vp = (int) (posAtStart + valueStart - lo + 1 - cacheSize); if (state == S_EXPECT_NAME || state == S_EXPECT_FIRST_NAME) { - listener.onEvent(EVT_NAME, getCharSequence(valueStart, p, vp), vp); + listener.onEvent(EVT_NAME, getCharSequence(valueStart, p, vp, hasEscape), vp); state = S_EXPECT_COLON; } else { - listener.onEvent(arrayDepth > 0 ? EVT_ARRAY_VALUE : EVT_VALUE, getCharSequence(valueStart, p, vp), vp); + listener.onEvent(arrayDepth > 0 ? EVT_ARRAY_VALUE : EVT_VALUE, getCharSequence(valueStart, p, vp, hasEscape), vp); state = S_EXPECT_COMMA; } @@ -240,6 +248,7 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { } valueStart = p; quoted = true; + hasEscape = false; break; default: if (state != S_EXPECT_VALUE) { @@ -248,6 +257,7 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { // this isn't a quote, include this character valueStart = p - 1; quoted = false; + hasEscape = false; break; } } @@ -257,6 +267,7 @@ public void parse(long lo, long hi, JsonParser listener) throws JsonException { this.state = state; this.quoted = quoted; this.ignoreNext = ignoreNext; + this.hasEscape = hasEscape; this.objDepth = objDepth; this.arrayDepth = arrayDepth; @@ -282,10 +293,51 @@ public void parseLast() throws JsonException { } } + /** + * Overwrites the decode buffers, so a secret this lexer parsed is no longer legible through them. + *

+ * Every name and value the lexer emits is assembled in {@link #sink} first, and an escaped one is + * then resolved into {@link #unescapeSink}; a listener that copies the value out leaves the lexer's + * own copy behind. When a value spans parse calls, its raw bytes are also assembled in the native + * {@link #cache}. {@link #clear()} does not help - it rewinds the parse state and never touches these + * buffers, and {@link StringSink#clear()} would only rewind the write position anyway, leaving a long + * secret legible in the tail past a shorter later write. None is reachable from outside this class. + *

+ * Callers that parse credentials should wipe rather than clear between documents - {@code + * OidcDeviceAuth} parses the token endpoint's response with a long-lived lexer, so its access, id + * and refresh tokens would otherwise stay on the heap for the life of that instance. Like + * {@link StringSink#wipe()} this is best effort: it reaches this lexer's own storage, not a copy a + * listener has already taken. + */ + public void wipe() { + sink.wipe(); + unescapeSink.wipe(); + if (cacheCapacity > 0 && cache != 0) { + // Wipe the whole allocation, not cacheSize: a completed value resets cacheSize to zero, and a + // shorter later split value can leave the tail of an earlier credential beyond the current size. + Vect.memset(cache, cacheCapacity, 0); + } + } + private static boolean isNotATerminator(char c) { return unquotedTerminators.excludes(c); } + private static int parseHex4(CharSequence value, int offset) { + int result = 0; + for (int j = 0; j < 4; j++) { + final char c = value.charAt(offset + j); + // shared hex table lookup (-1 for non-hex), cheaper than Character.digit; the table is + // ASCII-sized, so a code point above 127 is never a hex digit + final int digit = c < 128 ? Numbers.hexNumbers[c] : -1; + if (digit < 0) { + return -1; + } + result = (result << 4) | digit; + } + return result; + } + private static JsonException unsupportedEncoding(int position) { return JsonException.$(position, "Unsupported encoding"); } @@ -313,13 +365,17 @@ private void extendCache(int n) throws JsonException { long ptr = Unsafe.malloc(n, MemoryTag.NATIVE_TEXT_PARSER_RSS); if (cacheCapacity > 0) { Vect.memcpy(ptr, cache, cacheSize); + // Growth replaces the allocation before a credential owner has an opportunity to call wipe(). + // Zero the old block before returning it to the allocator so a copied split token is not retained + // in freed native memory. + Vect.memset(cache, cacheCapacity, 0); Unsafe.free(cache, cacheCapacity, MemoryTag.NATIVE_TEXT_PARSER_RSS); } cacheCapacity = n; cache = ptr; } - private CharSequence getCharSequence(long lo, long hi, int position) throws JsonException { + private CharSequence getCharSequence(long lo, long hi, int position, boolean hasEscape) throws JsonException { sink.clear(); if (cacheSize == 0) { if (!Utf8s.utf8ToUtf16(lo, hi - 1, sink)) { @@ -328,7 +384,81 @@ private CharSequence getCharSequence(long lo, long hi, int position) throws Json } else { utf8DecodeCacheAndBuffer(lo, hi - 1, position); } - return sink; + // the decode above assembled the raw bytes verbatim; resolve JSON escapes only when the scan saw a + // backslash, so escape-free values and names skip unescape() and return the assembled sink directly. + return hasEscape ? unescape(sink) : sink; + } + + private CharSequence unescape(CharSequence raw) { + // called only when the scan saw a backslash, so at least one escape is present; walk the value once, + // copying plain chars and resolving each escape - no leading scan to re-find the first backslash. + final int n = raw.length(); + unescapeSink.clear(); + int i = 0; + while (i < n) { + char c = raw.charAt(i); + if (c != '\\' || i + 1 >= n) { + unescapeSink.put(c); + i++; + continue; + } + char esc = raw.charAt(i + 1); + switch (esc) { + case '"': + unescapeSink.put('"'); + i += 2; + break; + case '\\': + unescapeSink.put('\\'); + i += 2; + break; + case '/': + unescapeSink.put('/'); + i += 2; + break; + case 'b': + unescapeSink.put('\b'); + i += 2; + break; + case 'f': + unescapeSink.put('\f'); + i += 2; + break; + case 'n': + unescapeSink.put('\n'); + i += 2; + break; + case 'r': + unescapeSink.put('\r'); + i += 2; + break; + case 't': + unescapeSink.put('\t'); + i += 2; + break; + case 'u': + int cp = i + 6 <= n ? parseHex4(raw, i + 2) : -1; + if (cp >= 0) { + unescapeSink.put((char) cp); + i += 6; + } else { + // malformed unicode escape: keep the backslash and the 'u' verbatim (lenient), so a + // non-conformant server's literal text survives rather than silently losing a byte + unescapeSink.put('\\').put(esc); + i += 2; + } + break; + default: + // an unrecognized escape letter: keep the backslash and the char verbatim (lenient), so a + // stray '\' before a non-escape char in non-conformant input survives rather than being + // dropped. A '\' before a RECOGNIZED escape letter (" \ / b f n r t u) is still decoded by + // the cases above - standard JSON unescape - so only genuinely unknown sequences reach here. + unescapeSink.put('\\').put(esc); + i += 2; + break; + } + } + return unescapeSink; } private void utf8DecodeCacheAndBuffer(long lo, long hi, int position) throws JsonException { diff --git a/core/src/main/java/io/questdb/client/cutlass/json/JsonParser.java b/core/src/main/java/io/questdb/client/cutlass/json/JsonParser.java index a4d6c45da..174b78ef0 100644 --- a/core/src/main/java/io/questdb/client/cutlass/json/JsonParser.java +++ b/core/src/main/java/io/questdb/client/cutlass/json/JsonParser.java @@ -24,7 +24,39 @@ package io.questdb.client.cutlass.json; +/** + * Receives the events {@link JsonLexer} emits as it parses. Implementations assemble whatever they need + * from the event stream; the lexer keeps no document. + */ @FunctionalInterface public interface JsonParser { + /** + * Called once per parse event, on the thread driving {@link JsonLexer#parse}. + * + *

{@code tag} is JSON-UNESCAPED. A value written {@code "a\\nb"} in the document arrives as + * the four characters {@code a \ n b}, not as the five raw ones. An implementation must NOT unescape it + * again: doing so decodes the {@code \n} a second time and yields {@code a}, LF, {@code b}. Earlier + * releases handed back the raw bytes and left the decoding to the listener, so a parser carried over + * from one of those has exactly that second decode to remove. + * + *

{@code tag} is a reused buffer, and not necessarily the same instance twice. It is valid + * only for the duration of this call - copy it to keep it. The lexer assembles an escape-free value in + * one sink and an escaped one in another, so which object arrives depends on whether that particular + * value contained a backslash. An implementation must therefore never compare {@code tag} by identity + * or cache the reference: either works across escape-free input and then fails on the first value that + * carries an escape. + * + *

{@code tag} is {@code null} for the structural events - {@link JsonLexer#EVT_OBJ_START}, + * {@link JsonLexer#EVT_OBJ_END}, {@link JsonLexer#EVT_ARRAY_START} and {@link JsonLexer#EVT_ARRAY_END} + * - and non-null only for {@link JsonLexer#EVT_NAME}, {@link JsonLexer#EVT_VALUE} and + * {@link JsonLexer#EVT_ARRAY_VALUE}. + * + * @param code the event, one of {@code JsonLexer.EVT_*} + * @param tag the name or value the event carries, unescaped, or {@code null} for a structural + * event; borrowed for the duration of the call only + * @param position byte offset of the event within the whole parsed stream, accumulated across + * {@link JsonLexer#parse} calls rather than an index into {@code tag} + * @throws JsonException to abort the parse + */ void onEvent(int code, CharSequence tag, int position) throws JsonException; } \ No newline at end of file diff --git a/core/src/main/java/io/questdb/client/cutlass/line/LineSenderException.java b/core/src/main/java/io/questdb/client/cutlass/line/LineSenderException.java index b599efcdb..507597985 100644 --- a/core/src/main/java/io/questdb/client/cutlass/line/LineSenderException.java +++ b/core/src/main/java/io/questdb/client/cutlass/line/LineSenderException.java @@ -48,26 +48,32 @@ public class LineSenderException extends RuntimeException { private final StringSink message = new StringSink(); + private final boolean retryable; private int errno = Integer.MIN_VALUE; public LineSenderException(CharSequence message) { this.message.put(message); + this.retryable = false; } public LineSenderException(CharSequence message, boolean retryable) { this.message.put(message); + this.retryable = retryable; } public LineSenderException(Throwable t) { super(t); + this.retryable = false; } public LineSenderException(String message, Throwable cause) { super(message, cause); this.message.put(message); + this.retryable = false; } + public LineSenderException appendIPv4(int ip) { Net.appendIP4(message, ip); return this; @@ -90,6 +96,24 @@ public String getMessage() { return errNoRender + " " + message; } + /** + * Whether the sender classified this failure as worth retrying - a 5xx, a 429, a transport error - as + * opposed to one that will keep failing, such as a 401 or a malformed request. + *

+ * This is the flag the class documentation above tells a caller to act on: a transient error means call + * {@code flush()} again on the same sender, a permanent one means close it or {@code reset()}. It was + * accepted by the {@link #LineSenderException(CharSequence, boolean)} constructor and then discarded, + * so the sender computed the answer and no caller could read it. + *

+ * {@code false} means "not classified as retryable", not "proven permanent": the constructors that carry + * no classification - a bare message, a wrapped cause - report {@code false}, which is the conservative + * direction for a caller that retries only on {@code true}. + * + * @return true when the sender classified this failure as retryable + */ + public boolean isRetryable() { + return retryable; + } public LineSenderException put(char ch) { message.put(ch); return this; diff --git a/core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java b/core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java index 398aa70a1..49204fa8e 100644 --- a/core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java @@ -27,9 +27,11 @@ import io.questdb.client.BuildInformationHolder; import io.questdb.client.ClientTlsConfiguration; import io.questdb.client.HttpClientConfiguration; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.cairo.TableUtils; import io.questdb.client.cutlass.http.HttpConstants; +import io.questdb.client.cutlass.http.HttpException; import io.questdb.client.cutlass.http.HttpKeywords; import io.questdb.client.cutlass.http.client.Fragment; import io.questdb.client.cutlass.http.client.HttpClient; @@ -57,10 +59,13 @@ import io.questdb.client.std.str.Utf8Sequence; import io.questdb.client.std.str.Utf8s; import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.io.Closeable; public abstract class AbstractLineHttpSender implements Sender { + private static final Logger LOG = LoggerFactory.getLogger(AbstractLineHttpSender.class); private static final String PATH = "/write?precision=n"; private static final int RETRY_BACKOFF_MULTIPLIER = 2; private static final int RETRY_INITIAL_BACKOFF_MS = 10; @@ -82,12 +87,15 @@ public abstract class AbstractLineHttpSender implements Sender { private final CharSequence questDBVersion; private final Rnd rnd; private final StringSink sink = new StringSink(); + private final String userAgent; private final String username; protected HttpClient.Request request; private HttpClient client; private boolean closed; private int currentAddressIndex; private long flushAfterNanos = Long.MAX_VALUE; + private HttpTokenProvider httpTokenProvider; + private boolean isTokenPending; private JsonErrorParser jsonErrorParser; private boolean lastFlushFailed; private long pendingRows; @@ -200,6 +208,9 @@ protected AbstractLineHttpSender( : HttpClientFactory.newPlainTextInstance(clientConfiguration); } this.questDBVersion = new BuildInformationHolder().getSwVersion(); + // precompute the User-Agent header value once: newRequest() runs on every flush, so concatenating it + // there would allocate a String each time + this.userAgent = "QuestDB/java/" + questDBVersion; this.request = newRequest(); this.maxNameLength = maxNameLength; this.rnd = rnd; @@ -225,10 +236,17 @@ public static AbstractLineHttpSender createLineSender( ) { return createLineSender(new ObjList<>(host), IntList.createWithValues(port), path, clientConfiguration, tlsConfig, autoFlushRows, authToken, username, password, maxNameLength, maxRetriesNanos, maxBackoffMillis, minRequestThroughput, flushIntervalNanos, - protocolVersion + protocolVersion, + null ); } + /** + * Provider-less form of the overload below, kept so callers compiled against the pre-{@code + * httpTokenProvider} signature keep linking. Mirrors the single-host overload above, which delegates + * with the same {@code null} provider. + */ + @SuppressWarnings("unused") public static AbstractLineHttpSender createLineSender( ObjList hosts, IntList ports, @@ -245,6 +263,29 @@ public static AbstractLineHttpSender createLineSender( long minRequestThroughput, long flushIntervalNanos, int protocolVersion + ) { + return createLineSender(hosts, ports, path, clientConfiguration, tlsConfig, autoFlushRows, + authToken, username, password, maxNameLength, maxRetriesNanos, maxBackoffMillis, + minRequestThroughput, flushIntervalNanos, protocolVersion, null); + } + + public static AbstractLineHttpSender createLineSender( + ObjList hosts, + IntList ports, + String path, + HttpClientConfiguration clientConfiguration, + ClientTlsConfiguration tlsConfig, + int autoFlushRows, + String authToken, + String username, + String password, + int maxNameLength, + long maxRetriesNanos, + int maxBackoffMillis, + long minRequestThroughput, + long flushIntervalNanos, + int protocolVersion, + HttpTokenProvider httpTokenProvider ) { HttpClient cli = null; Rnd rnd = new Rnd(NanosecondClockImpl.INSTANCE.getTicks(), MicrosecondClockImpl.INSTANCE.getTicks()); @@ -297,7 +338,16 @@ public static AbstractLineHttpSender createLineSender( } else { lastErrorSink.clear(); } - chunkedResponseToSink(response, lastErrorSink); + // This error-body read is bounded at request_timeout. The construct-time probe does + // catch a read abort in the retry loop below, but it does NOT absorb it into extra + // retries: retry_timeout (maxRetriesNanos, 10s default) is smaller than a single + // attempt's bound (request_timeout, 30s default). The probe's no-arg reads - + // response.await() above and the recv() inside parser.parse() - each bound the WHOLE + // call on elapsed time at request_timeout, so after the first attempt aborts nowNanos + // has already passed the retry deadline (armed at first-failure nowNanos + + // maxRetriesNanos). The probe therefore runs at most two attempts (initial + one retry) + // before build() throws "Failed to detect server line protocol version". + chunkedResponseToSink(response, lastErrorSink, clientConfiguration.getTimeout()); } catch (HttpClientException e) { if (lastErrorSink == null) { lastErrorSink = new StringSink(); @@ -329,14 +379,17 @@ public static AbstractLineHttpSender createLineSender( if (protocolVersion == PROTOCOL_VERSION_NOT_SET_EXPLICIT) { Misc.free(cli); if (lastErrorSink != null) { - throw new LineSenderException("Failed to detect server line protocol version: " + lastErrorSink); + // sanitize the raw server body before it reaches the exception message (and any log/terminal): + // a hostile or proxied endpoint must not splice control, ANSI or bidi chars into the render + throw new LineSenderException("Failed to detect server line protocol version: ").putAsPrintable(lastErrorSink); } throw new LineSenderException("Failed to detect server line protocol version"); } + final AbstractLineHttpSender sender; switch (protocolVersion) { case PROTOCOL_VERSION_V1: - return new LineHttpSenderV1( + sender = new LineHttpSenderV1( hosts, ports, path, @@ -355,8 +408,9 @@ public static AbstractLineHttpSender createLineSender( currentAddressIndex, rnd ); + break; case PROTOCOL_VERSION_V2: - return new LineHttpSenderV2( + sender = new LineHttpSenderV2( hosts, ports, path, @@ -375,8 +429,9 @@ public static AbstractLineHttpSender createLineSender( currentAddressIndex, rnd ); + break; case PROTOCOL_VERSION_V3: - return new LineHttpSenderV3( + sender = new LineHttpSenderV3( hosts, ports, path, @@ -395,9 +450,22 @@ public static AbstractLineHttpSender createLineSender( currentAddressIndex, rnd ); + break; default: throw new LineSenderException("Unsupported protocol version: " + protocolVersion); } + if (httpTokenProvider != null) { + // The constructor built the initial request before the provider was wired (httpTokenProvider was + // still null, so it took the no-auth path with withContent). Rebuild it via the deferred path now + // that the provider is set: this leaves the request at the header stage with the token pending, + // matching the reset() path, so the first row's stampTokenIfPending() finishes it (appends the auth + // header + withContent()) without a second client.newRequest(). Deferring the first getToken() off + // the build path also lets a lazily-signing-in provider (e.g. OidcDeviceAuth::getToken) be wired + // before sign-in completes, keeping the token pull on the use/flush path the provider documents. + sender.httpTokenProvider = httpTokenProvider; + sender.request = sender.newRequest(); + } + return sender; } public static boolean isNotFound(DirectUtf8Sequence statusCode) { @@ -409,20 +477,10 @@ public static boolean isNotFound(DirectUtf8Sequence statusCode) { @Override public void atNow() { - switch (state) { - case EMPTY: - throw new LineSenderException("no table name was provided"); - case TABLE_NAME_SET: - throw new LineSenderException("no symbols or columns were provided"); - case ADDING_SYMBOLS: - case ADDING_COLUMNS: - request.put('\n'); - state = RequestState.EMPTY; - break; - } - if (rowAdded()) { - flush(); - } + // validateRowStarted() rejects EMPTY and TABLE_NAME_SET, so only ADDING_SYMBOLS and ADDING_COLUMNS + // reach the terminator write + validateRowStarted(); + terminateRow(); } @Override @@ -439,6 +497,12 @@ public DirectByteSlice bufferView() { @Override public void cancelRow() { validateNotClosed(); + // While isTokenPending, newRequest() has left the request at the header stage: withContent() has not + // run, contentStart is still the -1 sentinel, and no row bytes exist to trim. trimContentToLen is + // guarded against exactly that state and no-ops, so this needs no second guard of its own - one that + // could never be observed to be missing, since the other one masks it. The guard that survives is + // the one that protects every caller of an exported method, not just this one; it is pinned by + // HttpClientRequestTrimTest. Do not re-add a check here: add coverage there instead. request.trimContentToLen(rowBookmark); state = RequestState.EMPTY; } @@ -455,7 +519,7 @@ public void close() { flush0(true); } } finally { - Misc.free(jsonErrorParser); + jsonErrorParser = Misc.free(jsonErrorParser); closed = true; client = Misc.free(client); } @@ -483,6 +547,9 @@ public Sender longColumn(CharSequence name, long value) { @TestOnly public void putRawMessage(Utf8Sequence msg) { + // stamp the deferred provider token (like table() does) so a raw message sent as the first row + // carries it; a no-op when no provider is configured + stampTokenIfPending(); request.put(msg); // message must include trailing \n state = RequestState.EMPTY; if (rowAdded()) { @@ -539,6 +606,9 @@ public Sender table(CharSequence table) { if (table.length() == 0) { throw new LineSenderException("table name cannot be empty"); } + // stamp the deferred provider token before the first row of this request, so the send carries it; + // a no-op once the token has been stamped or when no provider is configured + stampTokenIfPending(); // set bookmark at start of the line. rowBookmark = request.getContentLength(); state = RequestState.TABLE_NAME_SET; @@ -553,13 +623,13 @@ private static int backoff(Rnd rnd, int retryBackoff, int retryMaxBackoffMs) { return Math.min(retryMaxBackoffMs, backoff * RETRY_BACKOFF_MULTIPLIER); } - private static void chunkedResponseToSink(HttpClient.ResponseHeaders response, StringSink sink) { + private static void chunkedResponseToSink(HttpClient.ResponseHeaders response, StringSink sink, int timeoutMillis) { if (!response.isChunked()) { return; } Response chunkedRsp = response.getResponse(); Fragment fragment; - while ((fragment = chunkedRsp.recv()) != null) { + while ((fragment = chunkedRsp.recv(timeoutMillis)) != null) { sink.putNonAscii(fragment.lo(), fragment.hi()); } } @@ -600,13 +670,13 @@ private static boolean keepAliveDisabled(HttpClient.ResponseHeaders response) { return HttpKeywords.isClose(connectionHeader); } - private void consumeChunkedResponse(HttpClient.ResponseHeaders response) { + private void consumeChunkedResponse(HttpClient.ResponseHeaders response, int timeoutMillis) { if (!response.isChunked()) { return; } Response chunkedRsp = response.getResponse(); //noinspection StatementWithEmptyBody - while ((chunkedRsp.recv()) != null) { + while ((chunkedRsp.recv(timeoutMillis)) != null) { // we don't care about the response, just consume it, so it won't stay in the socket receive buffer } } @@ -668,12 +738,54 @@ private void flush0(boolean closing) { throw new HttpClientException("Request timed out"); } + // Bounded on ELAPSED time, not per socket read - see HttpClient.ResponseHeaders.await(int). + // That makes a head which dribbles but keeps making progress abort here, where base ran on + // with it, and this is the third body-read-shaped change on this path that an existing + // non-OIDC sender can observe (the other two are consumeChunkedResponse below and + // throwOnHttpErrorResponse). + // + // It differs from both in what it can conclude: NO STATUS HAS BEEN READ yet, so unlike the + // 2xx drain - which knows the server committed and reports success - and unlike the error + // arm - which has a verdict to surface - this abort says nothing about whether the batch + // landed. It falls to the HttpClientException arm and retries, which is the only available + // answer, and the retry spends the pre-existing ILP-over-HTTP at-least-once window: against + // a table without DEDUP keys a peer that dribbles a head past the budget duplicates rows. + // Reaching it needs an intermediary; QuestDB's own /write answers 204 with a small head. + // Pinned by LineHttpSenderErrorResponseTest#testDribbledResponseHeadFailsTheFlushWithinTheRetryBudget. response.await(remainingMillis); DirectUtf8Sequence statusCode = response.getStatusCode(); if (isSuccessResponse(statusCode)) { - consumeChunkedResponse(response); // if any - if (keepAliveDisabled(response)) { - // Server has HTTP keep-alive disabled, and it's closing this TCP connection. + // pass the whole per-flush budget (base + throughput extension) as EACH recv() read's + // timeout, NOT the raw request_timeout: recv() otherwise inherits defaultTimeout, so a + // tuned-low request_timeout paired with request_min_throughput would abort a large, + // still-progressing chunked body. This bounds each read, not the whole body cumulatively - + // fine here because the ILP server is trusted (unlike OidcDeviceAuth.parseBody, which also + // caps total bytes and elapsed time against an untrusted identity provider). + // A 2xx IS the commit: the server already has these rows. Draining its response body + // afterwards is only bookkeeping to keep the connection reusable, so a failure there must + // not escape into the catch below, which treats HttpClientException as a transport error + // and re-sends the whole batch -- duplicate rows on data the server accepted. Base could + // not reach this, because recv() re-armed its timeout on every socket read and a + // dribbling-but-progressing body never aborted; bounding the whole call means it now can. + // On abort the body is left unconsumed, which would mis-frame the next response on this + // connection, so drop the connection and report the flush as what it was: a success. + boolean drained = true; + try { + consumeChunkedResponse(response, actualTimeoutMillis); // if any + } catch (HttpClientException e) { + // The flush already SUCCEEDED - a 2xx IS the commit - so this changes no outcome, + // only the connection: unconsumed bytes would mis-frame the next response on it, so + // it is dropped below and the next flush reconnects. That cost is otherwise + // invisible - a server or intermediary that dribbles every response turns into one + // reconnect per flush, and the only symptom is churn nothing explains. DEBUG, not + // WARN: the handling is correct and a legitimately slow body is not a fault. + drained = false; + LOG.debug("could not drain the response body after a successful flush; dropping the " + + "connection so the next response cannot be mis-framed [reason={}]", + e.getMessage()); + } + // Server has HTTP keep-alive disabled, and it's closing this TCP connection. + if (!drained || keepAliveDisabled(response)) { client.disconnect(); } lastFlushFailed = false; @@ -692,15 +804,46 @@ private void flush0(boolean closing) { : retryingDeadlineNanos; if (nowNanos >= retryingDeadlineNanos) { // throw, but do not reset - a caller can try to flush later - throwOnHttpErrorResponse(statusCode, response, true); + throwOnHttpErrorResponse(statusCode, response, true, actualTimeoutMillis); } client.disconnect(); // forces reconnect, just in case retryBackoff = backoff(rnd, retryBackoff, maxBackoffMillis); continue; } - throwOnHttpErrorResponse(statusCode, response, false); + throwOnHttpErrorResponse(statusCode, response, false, actualTimeoutMillis); + } catch (HttpException e) { + // An unparseable response head: response.await() above hands it to HttpHeaderParser, which + // rejects a header block past its fixed 4096-byte buffer (an intermediary stacking + // Set-Cookie/CSP), a malformed Content-Length, or a status line that is not HTTP/1.x. + // HttpException is a SIBLING of HttpClientException, not a subclass, so it used to escape + // both arms - taking with it the client.disconnect() that keeps the next flush off a + // connection holding a half-read response, and leaving flush() throwing a raw + // HttpException rather than the LineSenderException its contract promises. + // + // Handled here rather than in the retry arm below, because it is not a transport failure + // and must not be retried. HttpHeaderParser only runs on bytes that ARRIVED, so this is + // positive evidence the server answered - the same evidence the 2xx drain arm above treats + // as decisive - and the head is chosen by an intermediary, not by chance: the next attempt + // parses the same block and fails identically. Retrying spent the whole budget re-sending a + // batch the server had already taken (measured: 16 sends over ~11s per flush at the default + // budget, against 1 before HttpException reached the arm), which for a table without DEDUP + // keys is 15 extra copies of every row. + // + // Disconnect anyway: the half-read response would mis-frame the next one on this + // connection. lastFlushFailed suppresses the close-time re-flush for the same reason - the + // server already has these rows. + lastFlushFailed = true; + client.disconnect(); + LineSenderException headEx = new LineSenderException("Could not flush buffer: http"); + if (isTls) { + headEx.put('s'); + } + headEx.put("://"); + headEx.put(currentHost()).put(':').put(currentPort()).put(this.path); + headEx.put(" Malformed HTTP response head").put(": ").put(e.getMessage()); + throw headEx; } catch (HttpClientException e) { - // this is a network error, we can retry + // this is a network error, we can retry. lastFlushFailed = true; client.disconnect(); // forces reconnect long nowNanos = System.nanoTime(); @@ -730,9 +873,21 @@ private HttpClient.Request newRequest() { HttpClient.Request r = client.newRequest(currentHost(), currentPort()) .POST() .url(path) - .header("User-Agent", "QuestDB/java/" + questDBVersion); + .header("User-Agent", userAgent); if (username != null) { r.authBasic(username, password); + } else if (httpTokenProvider != null) { + // Do NOT pull the token here (the construct / flush-completion path): getToken() can throw (not + // signed in yet, or a failed silent refresh), and a throw after client.newRequest() reset the + // shared request would corrupt the sender, turning an already-successful flush into an exception. + // Leave the request at the header stage (no withContent() yet) with the token pending, so the first + // row's stampTokenIfPending() appends the Authorization header + withContent() on THIS request + // WITHOUT a second client.newRequest() - the request line and headers are written once per flush, + // not twice. bufferView() reads empty meanwhile (contentStart is -1, so getContentLength() is 0). + isTokenPending = true; + rowBookmark = r.getContentLength(); + state = RequestState.EMPTY; + return r; } else if (authToken != null) { r.authToken(authToken); } @@ -766,21 +921,98 @@ private boolean rowAdded() { return pendingRows == autoFlushRows; } - private void throwOnHttpErrorResponse(DirectUtf8Sequence statusCode, HttpClient.ResponseHeaders response, boolean retryable) { + private void stampTokenIfPending() { + if (isTokenPending) { + // The construct/flush path deferred the token so a lazily-signing-in provider (e.g. + // OidcDeviceAuth::getToken) could be wired before sign-in completed, and so a provider failure + // never strikes after a successful send. The caller is now starting the first row, so finish the + // request newRequest() left at the header stage: pull a fresh token (so a long-lived sender + // follows token rotation), then append the Authorization header + withContent() on THIS request - + // no second client.newRequest(), so the request line and headers are written once, not twice. + // + // The throwing operations run BEFORE the request is mutated: a getToken()/validateToken() throw + // (not signed in yet, a failed refresh, or a rejected token) leaves isTokenPending set and the + // request untouched at the header stage, so the next row retries cleanly - the sender is never left + // corrupted. Validate EVERY pulled token, not just a changed instance: HttpTokenProvider.getToken() + // makes no immutability promise, so a provider that reuses one CharSequence buffer (the idiomatic + // zero-alloc style) and mutates its content between flushes must be re-checked, or a mutated token + // could splice a CR/LF into the "Authorization: Bearer" header (request.authToken writes it verbatim, + // with no CR/LF filtering). The scan is O(token length) and is dwarfed by the flush's network + // round-trip; the WebSocket auth path validates on every pull for the same reason. + CharSequence pulled; + try { + pulled = httpTokenProvider.getToken(); + } catch (LineSenderException e) { + throw e; + } catch (RuntimeException e) { + throw new LineSenderException( + e.getMessage() == null + ? "token provider failed to supply a credential" + : e.getMessage(), + e); + } + // Snapshot BEFORE validating, so the bytes that are checked are the bytes that are sent. Without + // it validateToken scans the provider's sequence and authToken then re-reads it - two reads of a + // buffer the provider owns and, per the paragraph above, is invited to reuse. A mutation landing + // between them passes the check and splices the mutated content, CR/LF included, into the + // Authorization header. One String per FLUSH (not per row), dwarfed by the round-trip that + // follows. Null-safe: a null pull must still reach validateToken's "null or empty" message + // rather than NPE here. + CharSequence token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); + request.authToken(token); + request.withContent(); + rowBookmark = request.getContentLength(); + state = RequestState.EMPTY; + isTokenPending = false; + } + } + + private void throwOnHttpErrorResponse(DirectUtf8Sequence statusCode, HttpClient.ResponseHeaders response, boolean retryable, int timeoutMillis) { + // The STATUS is the verdict; the body is detail for the message. A body read that aborts must not + // escape into flush0's catch, which treats HttpClientException as a transport failure: a definitive + // 401/403/405 would be reclassified as a network error, retried for the whole retry budget, and + // finally surfaced as "Connection Failed: timed out reading the chunked response body" with the real + // status nowhere in it. Report the status we already have instead, and say the body was unreadable + // rather than inventing detail. LineSenderException is a sibling of HttpClientException, not a + // subclass, so the intended throw passes through this catch untouched. + try { + throwOnHttpErrorResponse0(statusCode, response, retryable, timeoutMillis); + } catch (HttpClientException e) { + client.disconnect(); + // Carry the reason across. The status is the verdict, but WHY the body could not be read is the + // actionable half, and the three shapes call for different responses: "timed out reading the + // chunked response body" points at the flush timeout, "peer disconnect [errno=54]" at the + // connection, "malformed chunk size" at an intermediary mangling the framing. Binding e and + // dropping it left an operator a status and no way to tell those apart, on a path that has + // already disconnected. Plain put, not putAsPrintable: HttpClientException's messages are + // client-authored constants plus an errno, so unlike the status beside them they carry no + // server-supplied bytes. + final String reason = e.getMessage(); + throw new LineSenderException("Could not flush buffer: could not read the error response body", retryable) + .put(" [http-status=").putAsPrintable(statusCode.asAsciiCharSequence()) + .put(", reason=").put(reason != null ? reason : "") + .put(']'); + } + } + + private void throwOnHttpErrorResponse0(DirectUtf8Sequence statusCode, HttpClient.ResponseHeaders response, boolean retryable, int timeoutMillis) { CharSequence statusAscii = statusCode.asAsciiCharSequence(); if (Chars.equals("405", statusAscii)) { - consumeChunkedResponse(response); + consumeChunkedResponse(response, timeoutMillis); client.disconnect(); throw new LineSenderException("Could not flush buffer: HTTP endpoint does not support ILP. [http-status=405]", retryable); } if (Chars.equals("401", statusAscii) || Chars.equals("403", statusAscii)) { sink.clear(); - chunkedResponseToSink(response, sink); + chunkedResponseToSink(response, sink, timeoutMillis); LineSenderException ex = new LineSenderException("Could not flush buffer: HTTP endpoint authentication error", retryable); if (sink.length() > 0) { - ex = ex.put(": ").put(sink); + // sanitize the raw server body before it reaches the exception message (and any log/terminal): + // an untrusted or proxied endpoint must not splice control, ANSI or bidi chars into the render + ex = ex.put(": ").putAsPrintable(sink); } - ex.put(" [http-status=").put(statusAscii).put(']'); + ex.put(" [http-status=").putAsPrintable(statusAscii).put(']'); client.disconnect(); throw ex; } @@ -790,17 +1022,20 @@ private void throwOnHttpErrorResponse(DirectUtf8Sequence statusCode, HttpClient. jsonErrorParser = new JsonErrorParser(); } jsonErrorParser.reset(); - LineSenderException ex = jsonErrorParser.toException(response.getResponse(), statusCode, retryable); + LineSenderException ex = jsonErrorParser.toException(response.getResponse(), statusCode, retryable, timeoutMillis); client.disconnect(); throw ex; } // ok, no JSON, let's do something more generic sink.clear(); - sink.put("Could not flush buffer: "); - chunkedResponseToSink(response, sink); - sink.put(" [http-status=").put(statusCode).put(']'); + chunkedResponseToSink(response, sink, timeoutMillis); + // sanitize the raw server body before it reaches the exception message (and any log/terminal): + // an untrusted or proxied endpoint must not splice control, ANSI or bidi chars into the render + LineSenderException ex = new LineSenderException("Could not flush buffer: ", retryable) + .putAsPrintable(sink) + .put(" [http-status=").putAsPrintable(statusCode.asAsciiCharSequence()).put(']'); client.disconnect(); - throw new LineSenderException(sink, retryable); + throw ex; } private void validateNotClosed() { @@ -858,6 +1093,45 @@ protected void validateColumnName(CharSequence name) { } } + /** + * Writes the row terminator and closes the row, WITHOUT re-checking that a row was started - the caller + * has already done it. + *

+ * {@link #at(long, java.time.temporal.ChronoUnit)} and {@link #at(java.time.Instant)} must validate + * before they write the timestamp, not after: a rejected row would otherwise leave a stray timestamp in + * the request buffer for the next row to inherit. They used to follow that write with {@code atNow()}, + * which validated the very same state a second time - nothing between the two calls can change it, since + * only {@code table()}, a column write and this method touch {@code state} - so every explicit-timestamp + * row paid for a second switch on the hot ingestion path. They call this instead. + */ + protected void terminateRow() { + request.put('\n'); + state = RequestState.EMPTY; + if (rowAdded()) { + flush(); + } + } + + /** + * Rejects a row terminator that no row precedes. Subclasses MUST call this before writing the first byte + * of a terminator, not after: with an httpTokenProvider configured, newRequest() leaves the request at the + * header stage (withContent() deferred until the first row stamps the Authorization header), so a write + * that lands here while the state is EMPTY goes into the HTTP HEADER block, not the request body. Those + * bytes then start a line that folds the following "Authorization: Bearer ..." into the previous header + * (RFC 7230 obs-fold), and the request ships with no credential at all. cancelRow() cannot undo it either: + * trimContentToLen only rewinds within the content section. + */ + protected void validateRowStarted() { + switch (state) { + case EMPTY: + throw new LineSenderException("no table name was provided"); + case TABLE_NAME_SET: + throw new LineSenderException("no symbols or columns were provided"); + default: + break; + } + } + protected HttpClient.Request writeFieldName(CharSequence name) { validateColumnName(name); switch (state) { @@ -968,16 +1242,16 @@ public void onEvent(int code, CharSequence tag, int position) throws JsonExcepti private void drainAndReset(LineSenderException sink, DirectUtf8Sequence httpStatus) { assert state == State.INIT; - sink.put(messageSink).put(" [http-status=").put(httpStatus.asAsciiCharSequence()); + sink.putAsPrintable(messageSink).put(" [http-status=").putAsPrintable(httpStatus.asAsciiCharSequence()); if (codeSink.length() != 0 || errorIdSink.length() != 0 || lineSink.length() != 0) { if (errorIdSink.length() != 0) { - sink.put(", id: ").put(errorIdSink); + sink.put(", id: ").putAsPrintable(errorIdSink); } if (codeSink.length() != 0) { - sink.put(", code: ").put(codeSink); + sink.put(", code: ").putAsPrintable(codeSink); } if (lineSink.length() != 0) { - sink.put(", line: ").put(lineSink); + sink.put(", line: ").putAsPrintable(lineSink); } } sink.put(']'); @@ -994,10 +1268,10 @@ private void reset() { jsonSink.clear(); } - LineSenderException toException(Response chunkedRsp, DirectUtf8Sequence httpStatus, boolean retryable) { + LineSenderException toException(Response chunkedRsp, DirectUtf8Sequence httpStatus, boolean retryable, int timeoutMillis) { Fragment fragment; LineSenderException exception = new LineSenderException("Could not flush buffer: ", retryable); - while ((fragment = chunkedRsp.recv()) != null) { + while ((fragment = chunkedRsp.recv(timeoutMillis)) != null) { try { jsonSink.putNonAscii(fragment.lo(), fragment.hi()); lexer.parse(fragment.lo(), fragment.hi(), this); @@ -1005,10 +1279,12 @@ LineSenderException toException(Response chunkedRsp, DirectUtf8Sequence httpStat // we failed to parse JSON, but we still want to show the error message. // if we cannot parse it then we show the whole response as is. // let's make sure we have the whole message - there might be more chunks - while ((fragment = chunkedRsp.recv()) != null) { + while ((fragment = chunkedRsp.recv(timeoutMillis)) != null) { jsonSink.putNonAscii(fragment.lo(), fragment.hi()); } - exception.put(jsonSink).put(" [http-status=").put(httpStatus.asAsciiCharSequence()).put(']'); + // sanitize the raw server body before it reaches the exception message (and any log/terminal): + // an untrusted or proxied endpoint must not splice control, ANSI or bidi chars into the render + exception.putAsPrintable(jsonSink).put(" [http-status=").putAsPrintable(httpStatus.asAsciiCharSequence()).put(']'); reset(); return exception; } diff --git a/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV1.java b/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV1.java index 7b2ff47fb..9ecba1ca0 100644 --- a/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV1.java +++ b/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV1.java @@ -116,14 +116,20 @@ protected LineHttpSenderV1(ObjList hosts, @Override public void at(long timestamp, ChronoUnit unit) { + // validate BEFORE writing the timestamp: a rejected row must not leave a stray timestamp in + // the request buffer for the next row to inherit + validateRowStarted(); request.putAscii(' ').put(NanosTimestampDriver.INSTANCE.from(timestamp, unit)); - atNow(); + terminateRow(); // atNow() without the re-validation; see its javadoc } @Override public void at(Instant timestamp) { + // validate BEFORE writing the timestamp: a rejected row must not leave a stray timestamp in + // the request buffer for the next row to inherit + validateRowStarted(); request.putAscii(' ').put(NanosTimestampDriver.INSTANCE.from(timestamp)); - atNow(); + terminateRow(); // atNow() without the re-validation; see its javadoc } @Override diff --git a/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV2.java b/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV2.java index 00649f36f..a69b99bfa 100644 --- a/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV2.java +++ b/core/src/main/java/io/questdb/client/cutlass/line/http/LineHttpSenderV2.java @@ -163,16 +163,22 @@ protected LineHttpSenderV2( @Override public void at(long timestamp, ChronoUnit unit) { + // validate BEFORE writing the timestamp: a rejected row must not leave a stray timestamp in + // the request buffer for the next row to inherit + validateRowStarted(); request.putAscii(' '); putTimestamp(timestamp, unit); - atNow(); + terminateRow(); // atNow() without the re-validation; see its javadoc } @Override public void at(Instant timestamp) { + // validate BEFORE writing the timestamp: a rejected row must not leave a stray timestamp in + // the request buffer for the next row to inherit + validateRowStarted(); request.putAscii(' '); putTimestamp(timestamp); - atNow(); + terminateRow(); // atNow() without the re-validation; see its javadoc } @Override diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java new file mode 100644 index 000000000..2652a7f79 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java @@ -0,0 +1,80 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client; + +import io.questdb.client.cutlass.line.LineSenderException; + +/** + * Signals that the client could not OBTAIN an Authorization credential for a + * (re)connect handshake: the configured {@code httpTokenProvider} threw instead of + * returning a token -- a failed silent refresh, or no sign-in yet. + *

+ * Distinct from {@link QwpAuthFailedException}, which means the server rejected a + * credential the client did present (a terminal auth failure). A credential the client + * cannot ACQUIRE is instead handled by connection phase, exactly like a transport outage: + * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under + * Invariant B -- the IdP becomes reachable again, or the user completes an interactive + * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by + * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer + * store-and-forward promised to keep alive). Only the foreground/SYNC initial connect + * fails fast, because a connectivity error is the caller's to see during initialization, + * not after the drainer is running. + *

+ * It exists so the send loop can tell "the provider failed" apart from "the network + * failed", and it carries the provider's own exception so a handler can surface that + * instead of this wrapper. + *

+ * Where a caller can meet it. Not from the ordinary sender API: no path out of + * {@code build()}, {@code flush()} or any row call delivers this type. The foreground + * connects - SYNC in {@code CursorWebSocketSendLoop.connectWithRetry}, and the OFF-mode + * connect in {@code QwpWebSocketSender} - both catch it and rethrow + * {@link #providerFailure()}, so a token-provider failure reaches the caller as the + * provider's own exception; the running background drainer catches it and retries under + * the invariant above. It is public because both of those packages handle it, and + * because {@code QwpWebSocketSender.newReconnectFactory()} is public: a caller that + * drives {@code ReconnectFactory.reconnect()} itself runs the endpoint walk directly and + * so can receive this type unwrapped. Such a caller should treat it as the provider + * having failed rather than the cluster, and unwrap it with {@link #providerFailure()} + * the way the two foreground paths do. + */ +public class QwpCredentialUnavailableException extends LineSenderException { + private final RuntimeException providerFailure; + + public QwpCredentialUnavailableException(RuntimeException providerFailure) { + super(providerFailure.getMessage() == null + ? "token provider failed to supply a credential" + : providerFailure.getMessage(), providerFailure); + this.providerFailure = providerFailure; + } + + /** + * The exception the token provider threw, for a caller that must surface the + * provider's own error rather than this wrapper. Never null: the wrapper is only + * ever constructed around a provider failure. + */ + public RuntimeException providerFailure() { + return providerFailure; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 1bc478dc4..bfc924e52 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -25,10 +25,12 @@ package io.questdb.client.cutlass.qwp.client; import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.http.client.WebSocketClientFactory; import io.questdb.client.cutlass.http.client.WebSocketFrameHandler; +import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.impl.ConfigString; import io.questdb.client.impl.ConfigView; @@ -293,6 +295,11 @@ public class QwpQueryClient implements QuietCloseable { private boolean tlsEnabled; // Only meaningful when tlsEnabled. Default is full validation against the JVM's trust store. private int tlsValidationMode = ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL; + // Supplies a fresh Bearer token at each WebSocket upgrade (the initial + // connect and every failover reconnect), so a long-lived client follows + // token rotation. Mutually exclusive with the fixed authorizationHeader + // synthesized by withBearerToken/withBasicAuth; null when unset. + private HttpTokenProvider tokenProvider; private char[] trustStorePassword; private String trustStorePath; private volatile WebSocketClient webSocketClient; @@ -645,6 +652,26 @@ public void close() { } connected = false; lastCloseTimedOut = false; + // Teardown must not be cancellable by a flag the CALLER merely arrived with. Thread.join(long) + // throws InterruptedException the instant the calling thread's flag is set, WITHOUT ever looking + // at whether the I/O thread has exited -- so a carried flag turns the join below into an + // immediate throw and takes the "could not join" return, skipping closePool() and + // webSocketClient.close(). Those are the only frees for sendScratch, the decoder and the + // batch-buffer pool, and there is no second attempt to preserve them for: closedFlag was CAS'd + // on entry, so every later close() returns at the guard above, and a pooled worker has already + // been removed from QueryClientPool.all by reapIdle() before shutdown() gets here, so the pool's + // own close() never sees it either. The leak is permanent and silent. + // + // This is not hypothetical: PoolHousekeeper.stop() interrupts the housekeeper thread to break a + // recovery build's credential pull, and that same thread runs queryPool.reapIdle() straight + // afterwards with the flag still set. + // + // Clear it for the duration and restore it in the finally -- the interrupt-neutral shape + // FileTokenStore.load()/save() already use, and bounded by shutdownJoinMs. The timeout branch + // below is unaffected: with the flag cleared the join really waits, so a genuinely stuck I/O + // thread still takes the leak-rather-than-SIGSEGV path, and an interrupt delivered DURING the + // wait still means "we could not join" and still returns. + final boolean wasInterrupted = Thread.interrupted(); try { if (ioThread != null) { ioThread.shutdown(); @@ -693,6 +720,11 @@ public void close() { // (submitQuery copies its bytes into sendScratch), so it is safe to free // even when we otherwise leak the I/O thread and buffer pool. bindValues.close(); + if (wasInterrupted) { + // Hand the caller's cancellation back exactly as it arrived. Restoring it here rather + // than earlier keeps it out of the joins above, which is the whole point. + Thread.currentThread().interrupt(); + } } } @@ -713,6 +745,11 @@ public void close() { * observed so callers can distinguish "no primary available" from "all * endpoints unreachable" (the latter surfaces as a plain * {@link HttpClientException}). + *

+ * A configured token provider is queried once here, before the walk. A + * provider failure (not signed in, a failed silent refresh, a rejected + * token) is cluster-wide, so it fails fast with the provider's own error + * rather than being retried across endpoints as a transport failure. */ public synchronized void connect() { if (closedFlag.get()) { @@ -731,6 +768,12 @@ public synchronized void connect() { QwpServerInfo lastObservedMismatch = null; QwpIngressRoleRejectedException lastUpgradeRoleReject = null; Throwable lastTransportError = null; + // Resolve the bearer credential once, before the endpoint walk: a token is cluster-wide, so a + // token-provider failure (not signed in, a failed silent refresh, a rejected token) is not a + // per-endpoint transport fault. Resolving here lets it propagate as the provider's own error + // instead of being folded into "all endpoints unreachable", and avoids re-querying the provider + // once per endpoint. + String authHeader = resolveAuthorizationHeader(); while (true) { int i = hostTracker.pickNext(); if (i < 0) { @@ -738,7 +781,7 @@ public synchronized void connect() { } Endpoint ep = endpoints.get(i); try { - connectToEndpoint(ep); + connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); throw ae; @@ -895,11 +938,12 @@ public int getCompressionLevelForTest() { /** * Test-only hook: the synthesized {@code Authorization} header value * ({@code Basic ...} or {@code Bearer ...}), or null when no credentials - * were configured. + * were configured. When a token provider is configured, queries it and + * validates the returned token, exactly as a real upgrade would. */ @TestOnly public String getAuthorizationHeaderForTest() { - return authorizationHeader; + return resolveAuthorizationHeader(); } /** @@ -1082,6 +1126,9 @@ public QwpQueryClient withConnectTimeout(int connectTimeoutMs) { */ public QwpQueryClient withBasicAuth(String username, String password) { checkPreConnect("withBasicAuth"); + if (tokenProvider != null) { + throw new IllegalStateException("withBasicAuth cannot be combined with withBearerTokenProvider"); + } if (username == null || password == null) { throw new IllegalArgumentException("username and password must not be null"); } @@ -1099,6 +1146,9 @@ public QwpQueryClient withBasicAuth(String username, String password) { */ public QwpQueryClient withBearerToken(String token) { checkPreConnect("withBearerToken"); + if (tokenProvider != null) { + throw new IllegalStateException("withBearerToken cannot be combined with withBearerTokenProvider"); + } if (token == null) { throw new IllegalArgumentException("token must not be null"); } @@ -1106,6 +1156,36 @@ public QwpQueryClient withBearerToken(String token) { return this; } + /** + * Configures HTTP Bearer authentication with a token supplied on demand by + * {@code provider}, instead of the fixed token captured once by + * {@link #withBearerToken(String)}. The provider is queried for a fresh + * token at every WebSocket upgrade -- the initial {@link #connect()} and + * each failover reconnect -- so a long-lived client keeps working as the + * token rotates (for example an OIDC device-flow token: + * {@code .withBearerTokenProvider(auth::getToken)}). + *

+ * {@link HttpTokenProvider#getToken()} runs on the connect and reconnect + * paths, so it must return promptly and must not block on interactive + * input; a quick silent refresh is fine. Each returned token is validated + * ({@link HttpTokenProvider#validateToken(CharSequence)}) before it is sent, + * and a provider that throws fails that connection attempt. Mutually + * exclusive with {@link #withBearerToken(String)} and + * {@link #withBasicAuth(String, String)}. Must be called before + * {@link #connect}. + */ + public QwpQueryClient withBearerTokenProvider(HttpTokenProvider provider) { + checkPreConnect("withBearerTokenProvider"); + if (provider == null) { + throw new IllegalArgumentException("provider must not be null"); + } + if (authorizationHeader != null) { + throw new IllegalStateException("withBearerTokenProvider cannot be combined with withBearerToken or withBasicAuth"); + } + this.tokenProvider = provider; + return this; + } + /** * Overrides the default I/O buffer pool depth (4). Larger pools let the * I/O thread decode further ahead of the consumer at the cost of memory; @@ -1458,7 +1538,7 @@ private void cleanupFailedConnect() { currentEndpointIndex = -1; } - private void connectToEndpoint(Endpoint ep) { + private void connectToEndpoint(Endpoint ep, String authHeader) { if (tlsEnabled) { webSocketClient = WebSocketClientFactory.newTlsInstance( new ClientTlsConfiguration(trustStorePath, trustStorePassword, tlsValidationMode)); @@ -1470,7 +1550,7 @@ private void connectToEndpoint(Endpoint ep) { webSocketClient.setQwpAcceptEncoding(buildAcceptEncodingHeader()); webSocketClient.setQwpMaxBatchRows(maxBatchRows); webSocketClient.setConnectTimeout(connectTimeoutMs); - runUpgradeWithTimeout(ep); + runUpgradeWithTimeout(ep, authHeader); negotiatedQwpVersion = webSocketClient.getServerQwpVersion(); negotiatedZstdLevel = webSocketClient.getServerNegotiatedZstdLevel(); @@ -1789,6 +1869,10 @@ private void reconnectViaTracker() { QwpServerInfo lastMismatch = null; Throwable lastError = null; boolean retriedAfterReset = false; + // Resolve the bearer credential once per reconnect, before the endpoint walk, for the same + // reason as connect(): a provider failure is cluster-wide, so surface it directly rather than + // as a per-endpoint transport error retried across every host. + String authHeader = resolveAuthorizationHeader(); while (true) { int i = hostTracker.pickNext(); if (i < 0) { @@ -1801,7 +1885,7 @@ private void reconnectViaTracker() { } Endpoint ep = endpoints.get(i); try { - connectToEndpoint(ep); + connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); throw ae; @@ -1846,6 +1930,34 @@ private void reconnectViaTracker() { + ", lastError=" + (lastError == null ? "" : lastError.getMessage()) + ']'); } + private String resolveAuthorizationHeader() { + // With a token provider, query it once per connect()/reconnect (the caller resolves before the + // endpoint walk) so a reconnect presents a freshly refreshed token; validateToken rejects a + // null/empty/blank return, or one carrying a control or non-ASCII character, before it reaches + // the "Bearer " header. A provider that throws (a failed silent refresh, or not signed in yet) + // fails connect()/reconnect as a LineSenderException, preserving the provider failure as its cause. + if (tokenProvider != null) { + CharSequence pulled; + try { + pulled = tokenProvider.getToken(); + } catch (LineSenderException e) { + throw e; + } catch (RuntimeException e) { + throw new LineSenderException( + e.getMessage() == null + ? "token provider failed to supply a credential" + : e.getMessage(), + e); + } + // snapshot before validating, for the reason HttpTokenProvider.validateToken gives: the + // concatenation below re-reads the sequence, and the provider may be reusing its buffer + CharSequence token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); + return "Bearer " + token; + } + return authorizationHeader; + } + private long resolveQueryFlags(boolean resetSymbolDict) { if (!resetSymbolDict) { return 0L; @@ -1856,7 +1968,7 @@ private long resolveQueryFlags(boolean resetSymbolDict) { : 0L; } - private void runUpgradeWithTimeout(Endpoint ep) { + private void runUpgradeWithTimeout(Endpoint ep, String authHeader) { // Connect first, OUTSIDE the upgrade try. A connect-phase failure -- // including a connect_timeout overage flagged via flagAsTimeout() -- must // keep its own message ("connect timed out ...") and must NOT be relabeled @@ -1867,7 +1979,7 @@ private void runUpgradeWithTimeout(Endpoint ep) { int timeoutMs = (int) Math.min(authTimeoutMs, Integer.MAX_VALUE); try { - webSocketClient.upgrade(DEFAULT_ENDPOINT_PATH, timeoutMs, authorizationHeader); + webSocketClient.upgrade(DEFAULT_ENDPOINT_PATH, timeoutMs, authHeader); } catch (HttpClientException ex) { if (ex.isTimeout()) { // Reachable only for an upgrade/auth-phase timeout now, so the diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUdpSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUdpSender.java index f02e87c39..f13b1f516 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUdpSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUdpSender.java @@ -1413,7 +1413,10 @@ private void validateTableName(CharSequence name) { if (name.length() > MAX_TABLE_NAME_LENGTH) { throw new LineSenderException("table name too long [maxLength=" + MAX_TABLE_NAME_LENGTH + "]"); } - throw new LineSenderException("table name contains illegal characters: " + name); + // sanitize the rejected name before it reaches the message (and any log/terminal): a name that + // failed validation can carry BOM/bidi/zero-width/control chars that would otherwise reorder, hide + // or forge what a human reads, matching how the ILP name/error render escapes untrusted text + throw new LineSenderException("table name contains illegal characters: ").putAsPrintable(name); } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 7aed14192..41cc0a8c9 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -79,6 +79,7 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicReference; import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Supplier; /** * QWP v1 WebSocket client sender for streaming data to QuestDB. @@ -145,7 +146,15 @@ public class QwpWebSocketSender implements Sender { // enough window to preserve the trailing category distribution. private static final int MIN_ERROR_INBOX_CAPACITY = 16; private static final String WRITE_PATH = "/write/v4"; - private final String authorizationHeader; + // Yields the Authorization header value presented on each WebSocket upgrade. A constant for a + // fixed token or Basic credential; for an httpTokenProvider it pulls a freshly refreshed token, + // so the initial connect and every reconnect re-handshake carry the current token. May be null + // when no auth is configured. Evaluated once per (re)connect round in buildAndConnect, before the + // endpoint walk (not once per endpoint); a throwing provider is wrapped as + // QwpCredentialUnavailableException: the foreground/SYNC initial connect fails fast with the provider's + // own exception, while the running background drainer treats it as a transient outage and retries it + // indefinitely (never bounded by the reconnect budget, never terminal) per store-and-forward Invariant B. + private final Supplier authorizationHeaderSupplier; private final int autoFlushBytes; private final long autoFlushIntervalNanos; // Auto-flush configuration @@ -410,14 +419,14 @@ private QwpWebSocketSender( int autoFlushRows, int autoFlushBytes, long autoFlushIntervalNanos, - String authorizationHeader + Supplier authorizationHeaderSupplier ) { if (endpoints == null || endpoints.isEmpty()) { throw new IllegalArgumentException("endpoints must be non-empty"); } this.endpoints = Collections.unmodifiableList(new ArrayList<>(endpoints)); this.hostTracker = new QwpHostHealthTracker(this.endpoints.size()); - this.authorizationHeader = authorizationHeader; + this.authorizationHeaderSupplier = authorizationHeaderSupplier; this.tlsConfig = tlsConfig; this.encoder = new QwpWebSocketEncoder(DEFAULT_BUFFER_SIZE); this.tableBuffers = new CharSequenceObjHashMap<>(); @@ -700,8 +709,8 @@ public static QwpWebSocketSender connect( long durableAckKeepaliveIntervalMillis, long authTimeoutMs ) { - return connect(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, - autoFlushIntervalNanos, authorizationHeader, + return connectWithCredentialSupplier(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, + autoFlushIntervalNanos, fixedAuthHeader(authorizationHeader), requestDurableAck, cursorEngine, closeFlushTimeoutMillis, reconnectMaxDurationMillis, reconnectInitialBackoffMillis, reconnectMaxBackoffMillis, @@ -710,18 +719,67 @@ public static QwpWebSocketSender connect( 0, null, SenderConnectionDispatcher.DEFAULT_CAPACITY); } + /** + * Constant-credential form of the connection-listener variant below, kept so callers compiled against + * the {@code String authorizationHeader} signature keep linking after the parameter became a + * {@link Supplier}. Wraps the header with {@link #fixedAuthHeader(String)}, which also tags it as a + * CONSTANT credential for the store-and-forward drainer's terminal policy -- the same thing the older + * signature implied. + *

+ * A rotating credential goes to {@code connectWithCredentialSupplier} instead, which carries a distinct + * name precisely so this form keeps its exact descriptor and a bare {@code null} credential stays + * unambiguous. + */ + public static QwpWebSocketSender connect( + List endpoints, + ClientTlsConfiguration tlsConfig, + int autoFlushRows, + int autoFlushBytes, + long autoFlushIntervalNanos, + String authorizationHeader, + boolean requestDurableAck, + CursorSendEngine cursorEngine, + long closeFlushTimeoutMillis, + long reconnectMaxDurationMillis, + long reconnectInitialBackoffMillis, + long reconnectMaxBackoffMillis, + Sender.InitialConnectMode initialConnectMode, + SenderErrorHandler errorHandler, + int errorInboxCapacity, + long durableAckKeepaliveIntervalMillis, + long authTimeoutMs, + int connectTimeoutMs, + SenderConnectionListener connectionListener, + int connectionListenerInboxCapacity + ) { + return connectWithCredentialSupplier(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, + autoFlushIntervalNanos, fixedAuthHeader(authorizationHeader), + requestDurableAck, cursorEngine, + closeFlushTimeoutMillis, reconnectMaxDurationMillis, + reconnectInitialBackoffMillis, reconnectMaxBackoffMillis, + initialConnectMode, errorHandler, errorInboxCapacity, + durableAckKeepaliveIntervalMillis, authTimeoutMs, connectTimeoutMs, + connectionListener, connectionListenerInboxCapacity); + } + /** * Multi-endpoint variant that also accepts the async connection-event * listener and its dispatcher inbox capacity. Uses the default * poison-frame detector threshold. + *

+ * Named apart from {@code connect} rather than overloading it: the constant-credential + * {@code connect(..., String, ...)} form must keep its exact descriptor for callers compiled against + * it, and a {@code String} / {@code Supplier} overload pair of equal arity makes a bare + * {@code null} credential argument ambiguous -- neither parameter type is more specific than the + * other. A distinct name keeps both forms callable with no cast. */ - public static QwpWebSocketSender connect( + public static QwpWebSocketSender connectWithCredentialSupplier( List endpoints, ClientTlsConfiguration tlsConfig, int autoFlushRows, int autoFlushBytes, long autoFlushIntervalNanos, - String authorizationHeader, + Supplier authorizationHeaderSupplier, boolean requestDurableAck, CursorSendEngine cursorEngine, long closeFlushTimeoutMillis, @@ -737,8 +795,8 @@ public static QwpWebSocketSender connect( SenderConnectionListener connectionListener, int connectionListenerInboxCapacity ) { - return connect(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, - autoFlushIntervalNanos, authorizationHeader, requestDurableAck, + return connectWithCredentialSupplier(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, + autoFlushIntervalNanos, authorizationHeaderSupplier, requestDurableAck, cursorEngine, closeFlushTimeoutMillis, reconnectMaxDurationMillis, reconnectInitialBackoffMillis, reconnectMaxBackoffMillis, initialConnectMode, errorHandler, errorInboxCapacity, @@ -750,18 +808,71 @@ public static QwpWebSocketSender connect( } /** - * Master connect overload — also accepts the poison-frame detector + * Constant-credential form of the master overload below, kept so callers compiled against the + * {@code String authorizationHeader} signature keep linking after the parameter became a + * {@link Supplier}. Wraps the header with {@link #fixedAuthHeader(String)}, which also tags it as a + * CONSTANT credential for the store-and-forward drainer's terminal policy -- the same thing the older + * signature implied. + *

+ * A rotating credential goes to {@code connectWithCredentialSupplier} instead, which carries a distinct + * name precisely so this form keeps its exact descriptor and a bare {@code null} credential stays + * unambiguous. + */ + public static QwpWebSocketSender connect( + List endpoints, + ClientTlsConfiguration tlsConfig, + int autoFlushRows, + int autoFlushBytes, + long autoFlushIntervalNanos, + String authorizationHeader, + boolean requestDurableAck, + CursorSendEngine cursorEngine, + long closeFlushTimeoutMillis, + long reconnectMaxDurationMillis, + long reconnectInitialBackoffMillis, + long reconnectMaxBackoffMillis, + Sender.InitialConnectMode initialConnectMode, + SenderErrorHandler errorHandler, + int errorInboxCapacity, + long durableAckKeepaliveIntervalMillis, + long authTimeoutMs, + int connectTimeoutMs, + SenderConnectionListener connectionListener, + int connectionListenerInboxCapacity, + int maxFrameRejections, + long poisonMinEscalationWindowMillis, + long catchUpCapGapMinEscalationWindowMillis + ) { + return connectWithCredentialSupplier(endpoints, tlsConfig, autoFlushRows, autoFlushBytes, + autoFlushIntervalNanos, fixedAuthHeader(authorizationHeader), + requestDurableAck, cursorEngine, + closeFlushTimeoutMillis, reconnectMaxDurationMillis, + reconnectInitialBackoffMillis, reconnectMaxBackoffMillis, + initialConnectMode, errorHandler, errorInboxCapacity, + durableAckKeepaliveIntervalMillis, authTimeoutMs, connectTimeoutMs, + connectionListener, connectionListenerInboxCapacity, + maxFrameRejections, poisonMinEscalationWindowMillis, + catchUpCapGapMinEscalationWindowMillis); + } + + /** + * Master connect entry point — also accepts the poison-frame detector * threshold ({@code max_frame_rejections}): consecutive server-active * rejections of the same head-of-line frame, with no ack progress in * between, before the loop escalates to a typed terminal. + *

+ * Named apart from {@code connect} for the reason given on + * {@link #connectWithCredentialSupplier(List, ClientTlsConfiguration, int, int, long, Supplier, + * boolean, CursorSendEngine, long, long, long, long, Sender.InitialConnectMode, SenderErrorHandler, + * int, long, long, int, SenderConnectionListener, int)}. */ - public static QwpWebSocketSender connect( + public static QwpWebSocketSender connectWithCredentialSupplier( List endpoints, ClientTlsConfiguration tlsConfig, int autoFlushRows, int autoFlushBytes, long autoFlushIntervalNanos, - String authorizationHeader, + Supplier authorizationHeaderSupplier, boolean requestDurableAck, CursorSendEngine cursorEngine, long closeFlushTimeoutMillis, @@ -783,7 +894,7 @@ public static QwpWebSocketSender connect( QwpWebSocketSender sender = new QwpWebSocketSender( endpoints, tlsConfig, autoFlushRows, autoFlushBytes, autoFlushIntervalNanos, - authorizationHeader + authorizationHeaderSupplier ); try { sender.requestDurableAck = requestDurableAck; @@ -850,7 +961,7 @@ public static QwpWebSocketSender createForTesting(String host, int port, String return new QwpWebSocketSender( singleEndpoint(host, port), null, DEFAULT_AUTO_FLUSH_ROWS, DEFAULT_AUTO_FLUSH_BYTES, DEFAULT_AUTO_FLUSH_INTERVAL_NANOS, - authorizationHeader + fixedAuthHeader(authorizationHeader) ); } @@ -878,6 +989,26 @@ public static QwpWebSocketSender createForTesting( ); } + /** + * Wraps a CONSTANT {@code Authorization} header value as a supplier, tagged so the store-and-forward + * drainer can tell it apart from an {@code httpTokenProvider}-backed rotating credential. Callers that + * synthesize a fixed header (a static bearer token, a Basic credential) must route it through here + * rather than through a bare lambda, or the drainer misreads the credential as rotating. + *

+ * The distinction is load-bearing for the orphan drainer's terminal policy. A {@code 401} against a + * fixed credential is a permanent misconfiguration, so quarantining the slot immediately is right. The + * same {@code 401} against a rotating credential can be a recoverable window - clock skew past the + * token's skew margin, a mid-flight revocation, an identity provider rotating its signing keys - where + * a later attempt carrying a freshly pulled token succeeds. See + * {@code BackgroundDrainer.connectWithDurableAckRetry}. + * + * @param header the constant header value, or null when no credential is configured + * @return a tagged supplier yielding {@code header}, or null when {@code header} is null + */ + public static Supplier fixedAuthHeader(String header) { + return header == null ? null : new FixedAuthHeader(header); + } + @Override public void at(long timestamp, ChronoUnit unit) { checkNotClosed(); @@ -1146,205 +1277,233 @@ public QwpWebSocketSender charColumn(CharSequence columnName, char value) { public void close() { if (!closed) { closed = true; - Runnable hook = closeStartedHook; - closeStartedHook = null; - if (hook != null) { - try { - hook.run(); - } catch (Throwable t) { - // A test witness must never prevent production resource cleanup. - LOG.error("Error in close-started test hook: {}", String.valueOf(t)); + // Interrupt-neutral for the duration, the same shape QwpQueryClient.close() and + // FileTokenStore.load()/save() use. PoolHousekeeper.stop() interrupts a housekeeper blocked in + // a pooled credential pull, while SenderPool's provider-free private driver may interrupt an + // unexpected overrun in a direct recovery operation. The interrupted thread can then run + // senderPool.reapIdle() or a startup-recovery step's finally, both of which close a delegate. + // A CARRIED flag is fatal to that close: CountDownLatch.await(t, u) tests Thread.interrupted() before it + // ever consults the latch, so CursorWebSocketSendLoop.close()'s shutdown await would throw + // having waited 0 ms, take the failed-stop path, and report the SF slot flock still held -- + // the exact outcome the interrupt was added to prevent. Worse, that path re-asserts the + // flag, so every remaining delegate in the same reap sweep failed the same way. + // + // Clearing it here restores the intended meaning: the interrupt breaks the operation it was + // aimed at, and the teardown that follows runs normally. drainOnClose() also records and consumes + // an interrupt delivered while it is pacing that wait; restoring it before the I/O-loop shutdown + // would make the next CountDownLatch.await fail at 0ms after a successful ACK drain. Carry both + // observations to this outer boundary, after every teardown wait. An interrupt that instead lands + // during the I/O-loop shutdown still reaches that await and takes its genuine failed-stop branch. + final boolean[] restoreInterrupt = {Thread.interrupted()}; + try { + close0(restoreInterrupt); + } finally { + if (restoreInterrupt[0]) { + Thread.currentThread().interrupt(); } } - boolean ioThreadStopped = true; - // Captures the first error from the flush/drain path AND any - // secondary errors from cleanup steps (added via addSuppressed). - // Silently swallowing any of these would hide latched terminal - // SenderError HALTs (server-side rejections like MESSAGE_TOO_BIG, - // SCHEMA_MISMATCH HALT) from users who only call close() and - // never call flush() afterwards. - Throwable terminalError = null; - // Snapshot the exact terminal error instance that a user-thread - // API call ALREADY caught (via flush()/at()) before close() ran. - // If flushPendingRows/drainOnClose below also rethrow the same - // instance, dropping it at the final rethrow avoids - // try-with-resources self-suppression: Throwable.addSuppressed - // raises IllegalArgumentException when primary == suppressed. - // Must stay this single read: the snapshot needs the identity of - // the error the user already owns, and only - // getSynchronouslySurfacedError() holds it. Deriving it from two - // separate latch reads races the I/O thread -- a terminal latched - // between the reads would be adopted as user-owned and silently - // dropped (see CloseOwnershipRaceTest). - Throwable alreadyOwnedByUser = cursorSendLoop != null - ? cursorSendLoop.getSynchronouslySurfacedError() : null; + } + } + private void close0(boolean[] restoreInterrupt) { + Runnable hook = closeStartedHook; + closeStartedHook = null; + if (hook != null) { try { - // Only drain when both the engine and the I/O loop are wired - // up — close() is also called from createForTesting() teardown - // and from connect() rollback paths where one or both may be null. - if (connectionError.get() == null && cursorEngine != null && cursorSendLoop != null) { - // 1) Flush user-thread state into the engine (encoded - // rows -> mmap'd / malloc'd ring). After this, the - // cursor engine's publishedFsn reflects the final - // target the I/O loop must drive ackedFsn up to. - // A pre-flight rejection means this batch cannot fit - // the current cap however it is split. It is - // RETAINED by design so it can go out once a - // larger-cap node is reached -- but on close there is - // no later flush, and letting the throw escape here - // skips sendCommitMessage, sealAndSwapBuffer and - // drainOnClose, abandoning every row an earlier - // successful flush already published. The message - // that path emits tells the caller to close the - // sender to discard the batch, so honour that: - // discard it, remember the error, and let the rest of - // close() run. rethrowTerminal below still surfaces it. - try { - flushPendingRows(deferCommit); - } catch (BatchTooLargeForCapException e) { - resetTableBuffersAfterFlush(); - terminalError = captureCloseError(terminalError, e); - } catch (Throwable t) { - // Same reasoning as the pre-flight rejection above, for the - // failures a size check cannot see: sealAndSwapBuffer's - // buffer-recycle timeout and appendBlocking's backpressure - // deadline. Letting those escape to the outer catch skipped - // sendCommitMessage, sealAndSwapBuffer and drainOnClose -- so a - // flush that had already published deferred dictionary chunks - // left their group open forever, and every row an EARLIER - // successful flush published was abandoned unacked. The batch is - // NOT discarded here (unlike the over-cap case, this failure is - // not a verdict on the batch's contents), but the rest of close() - // must still run. rethrowTerminal below surfaces it. - terminalError = captureCloseError(terminalError, t); - } - if (!deferCommit && hasDeferredMessages) { - sendCommitMessage(); - } - if (activeBuffer != null && activeBuffer.hasData()) { - sealAndSwapBuffer(); - if (!deferCommit) { - lastCommitBoundaryFsn = cursorEngine.publishedFsn(); - } - } - // 2) Safety-net rethrow: surface the latched terminal - // error only when no other channel has already - // delivered THIS terminal to the user. "Already - // delivered" means either the producer thread saw it - // synchronously via flush()/append() (checkUnsurfacedError - // is silent in that case) or the async dispatcher - // actually delivered the latched terminal to a - // user-installed custom handler - // (hasDeliveredTerminalToCustomHandler, checked here). - // The test is terminal-specific on purpose: an earlier - // routine RETRIABLE rejection delivered to the - // handler must NOT suppress a later genuine TERMINAL - // error (the "any error ever" flag did, silently - // losing it). It also stays false when the terminal - // reached only the default handler after a - // setErrorHandler(null) revert, or is still - // queued/abandoned behind a slow handler -- so a - // config-string-only caller, and a reverting caller, - // both still get the loud rethrow on shutdown. - boolean terminalOwnedByCustomHandler = errorDispatcher != null - && errorDispatcher.hasDeliveredTerminalToCustomHandler(); - if (!terminalOwnedByCustomHandler) { - cursorSendLoop.checkUnsurfacedError(); - } - // 3) Bounded drain: block until the server has ACK'd - // everything we just published, or until the - // configured timeout elapses. closeFlushTimeoutMillis - // <= 0 opts out (fast close, may lose memory-mode - // data on JVM exit). Pass the same ownership flag the - // step-2 safety net used: when the custom handler - // already owns THIS terminal, the drain must stop on it - // without re-throwing (re-throwing would double-signal - // an error the user already handled). Otherwise the - // drain keeps the loud safety net and surfaces it. - if (closeFlushTimeoutMillis > 0L) { - drainOnClose(terminalOwnedByCustomHandler); - } - } + hook.run(); } catch (Throwable t) { - terminalError = t; + // A test witness must never prevent production resource cleanup. + LOG.error("Error in close-started test hook: {}", String.valueOf(t)); } + } + boolean ioThreadStopped = true; + // Captures the first error from the flush/drain path AND any + // secondary errors from cleanup steps (added via addSuppressed). + // Silently swallowing any of these would hide latched terminal + // SenderError HALTs (server-side rejections like MESSAGE_TOO_BIG, + // SCHEMA_MISMATCH HALT) from users who only call close() and + // never call flush() afterwards. + Throwable terminalError = null; + // Snapshot the exact terminal error instance that a user-thread + // API call ALREADY caught (via flush()/at()) before close() ran. + // If flushPendingRows/drainOnClose below also rethrow the same + // instance, dropping it at the final rethrow avoids + // try-with-resources self-suppression: Throwable.addSuppressed + // raises IllegalArgumentException when primary == suppressed. + // Must stay this single read: the snapshot needs the identity of + // the error the user already owns, and only + // getSynchronouslySurfacedError() holds it. Deriving it from two + // separate latch reads races the I/O thread -- a terminal latched + // between the reads would be adopted as user-owned and silently + // dropped (see CloseOwnershipRaceTest). + Throwable alreadyOwnedByUser = cursorSendLoop != null + ? cursorSendLoop.getSynchronouslySurfacedError() : null; - // Shut down the I/O thread before closing the socket or buffers - // it may be using. Must run even if the flush above failed. - if (cursorSendLoop != null) { + try { + // Only drain when both the engine and the I/O loop are wired + // up — close() is also called from createForTesting() teardown + // and from connect() rollback paths where one or both may be null. + if (connectionError.get() == null && cursorEngine != null && cursorSendLoop != null) { + // 1) Flush user-thread state into the engine (encoded + // rows -> mmap'd / malloc'd ring). After this, the + // cursor engine's publishedFsn reflects the final + // target the I/O loop must drive ackedFsn up to. + // A pre-flight rejection means this batch cannot fit + // the current cap however it is split. It is + // RETAINED by design so it can go out once a + // larger-cap node is reached -- but on close there is + // no later flush, and letting the throw escape here + // skips sendCommitMessage, sealAndSwapBuffer and + // drainOnClose, abandoning every row an earlier + // successful flush already published. The message + // that path emits tells the caller to close the + // sender to discard the batch, so honour that: + // discard it, remember the error, and let the rest of + // close() run. rethrowTerminal below still surfaces it. try { - cursorSendLoop.close(); - } catch (Throwable e) { - ioThreadStopped = false; - LOG.error("Error closing cursor send loop: {}", String.valueOf(e)); + flushPendingRows(deferCommit); + } catch (BatchTooLargeForCapException e) { + resetTableBuffersAfterFlush(); terminalError = captureCloseError(terminalError, e); + } catch (Throwable t) { + // Same reasoning as the pre-flight rejection above, for the + // failures a size check cannot see: sealAndSwapBuffer's + // buffer-recycle timeout and appendBlocking's backpressure + // deadline. Letting those escape to the outer catch skipped + // sendCommitMessage, sealAndSwapBuffer and drainOnClose -- so a + // flush that had already published deferred dictionary chunks + // left their group open forever, and every row an EARLIER + // successful flush published was abandoned unacked. The batch is + // NOT discarded here (unlike the over-cap case, this failure is + // not a verdict on the batch's contents), but the rest of close() + // must still run. rethrowTerminal below surfaces it. + terminalError = captureCloseError(terminalError, t); } - } - // Drainer pool closes after the foreground I/O loop is wound - // down. Drainers share buildAndConnect's endpoint walk and - // hostTracker state with the foreground (never its observable - // connection state or event stream), but their - // connect gate is their own stop flag — NOT the foreground - // loop's liveness — so the pool's graceful-drain window below - // still lets in-flight drainers finish (including reconnects) - // even though cursorSendLoop is already stopped. - if (drainerPool != null) { - try { - drainerPool.close(); - } catch (Throwable e) { - LOG.error("Error closing drainer pool: {}", String.valueOf(e)); - terminalError = captureCloseError(terminalError, e); + if (!deferCommit && hasDeferredMessages) { + sendCommitMessage(); + } + if (activeBuffer != null && activeBuffer.hasData()) { + sealAndSwapBuffer(); + if (!deferCommit) { + lastCommitBoundaryFsn = cursorEngine.publishedFsn(); + } + } + // 2) Safety-net rethrow: surface the latched terminal + // error only when no other channel has already + // delivered THIS terminal to the user. "Already + // delivered" means either the producer thread saw it + // synchronously via flush()/append() (checkUnsurfacedError + // is silent in that case) or the async dispatcher + // actually delivered the latched terminal to a + // user-installed custom handler + // (hasDeliveredTerminalToCustomHandler, checked here). + // The test is terminal-specific on purpose: an earlier + // routine RETRIABLE rejection delivered to the + // handler must NOT suppress a later genuine TERMINAL + // error (the "any error ever" flag did, silently + // losing it). It also stays false when the terminal + // reached only the default handler after a + // setErrorHandler(null) revert, or is still + // queued/abandoned behind a slow handler -- so a + // config-string-only caller, and a reverting caller, + // both still get the loud rethrow on shutdown. + boolean terminalOwnedByCustomHandler = errorDispatcher != null + && errorDispatcher.hasDeliveredTerminalToCustomHandler(); + if (!terminalOwnedByCustomHandler) { + cursorSendLoop.checkUnsurfacedError(); + } + // 3) Bounded drain: block until the server has ACK'd + // everything we just published, or until the + // configured timeout elapses. closeFlushTimeoutMillis + // <= 0 opts out (fast close, may lose memory-mode + // data on JVM exit). Pass the same ownership flag the + // step-2 safety net used: when the custom handler + // already owns THIS terminal, the drain must stop on it + // without re-throwing (re-throwing would double-signal + // an error the user already handled). Otherwise the + // drain keeps the loud safety net and surfaces it. + if (closeFlushTimeoutMillis > 0L) { + drainOnClose(terminalOwnedByCustomHandler, restoreInterrupt); } } + } catch (Throwable t) { + terminalError = t; + } - // Always free resources the I/O thread never touches: - // encoder and table buffers are user-thread-only. + // Shut down the I/O thread before closing the socket or buffers + // it may be using. Must run even if the flush above failed. + if (cursorSendLoop != null) { try { - encoder.close(); - ObjList keys = tableBuffers.keys(); - for (int i = 0, n = keys.size(); i < n; i++) { - CharSequence key = keys.getQuick(i); - if (key != null) { - Misc.free(tableBuffers.get(key)); - } - } - tableBuffers.clear(); - } catch (Throwable t) { - LOG.error("Error closing encoder or table buffers: {}", String.valueOf(t)); - terminalError = captureCloseError(terminalError, t); + cursorSendLoop.close(); + } catch (Throwable e) { + ioThreadStopped = false; + LOG.error("Error closing cursor send loop: {}", String.valueOf(e)); + terminalError = captureCloseError(terminalError, e); + } + } + // Drainer pool closes after the foreground I/O loop is wound + // down. Drainers share buildAndConnect's endpoint walk and + // hostTracker state with the foreground (never its observable + // connection state or event stream), but their + // connect gate is their own stop flag — NOT the foreground + // loop's liveness — so the pool's graceful-drain window below + // still lets in-flight drainers finish (including reconnects) + // even though cursorSendLoop is already stopped. + if (drainerPool != null) { + try { + drainerPool.close(); + } catch (Throwable e) { + LOG.error("Error closing drainer pool: {}", String.valueOf(e)); + terminalError = captureCloseError(terminalError, e); } + } - if (!ioThreadStopped) { - // The worker may still touch every resource below. Hand the - // complete sender-owned tail to its exit path rather than - // permanently leaking everything except the engine. The - // callback is idempotence-gated by closeRemainingResources(). - if (ownsCursorEngine && cursorEngine != null) { - retainedEngine = cursorEngine; - } - Runnable closeCallback = () -> closeRemainingResources(null); - if (cursorSendLoop != null && cursorSendLoop.delegateClose(closeCallback)) { - rethrowTerminal(terminalError); - return; + // Always free resources the I/O thread never touches: + // encoder and table buffers are user-thread-only. + try { + encoder.close(); + ObjList keys = tableBuffers.keys(); + for (int i = 0, n = keys.size(); i < n; i++) { + CharSequence key = keys.getQuick(i); + if (key != null) { + Misc.free(tableBuffers.get(key)); } - // The worker exited between close() failing and delegation. - // Cleanup is safe here and its failures remain suppressed on - // the original close error. - terminalError = closeRemainingResources(terminalError); - } else { - terminalError = closeRemainingResources(terminalError); } + tableBuffers.clear(); + } catch (Throwable t) { + LOG.error("Error closing encoder or table buffers: {}", String.valueOf(t)); + terminalError = captureCloseError(terminalError, t); + } - // If close() ended up holding the same instance the user already - // caught earlier, suppress the rethrow. The user's catch block - // wraps close() (try-with-resources), and Throwable refuses - // self-suppression. - if (terminalError != null && terminalError == alreadyOwnedByUser) { - terminalError = null; + if (!ioThreadStopped) { + // The worker may still touch every resource below. Hand the + // complete sender-owned tail to its exit path rather than + // permanently leaking everything except the engine. The + // callback is idempotence-gated by closeRemainingResources(). + if (ownsCursorEngine && cursorEngine != null) { + retainedEngine = cursorEngine; + } + Runnable closeCallback = () -> closeRemainingResources(null); + if (cursorSendLoop != null && cursorSendLoop.delegateClose(closeCallback)) { + rethrowTerminal(terminalError); + return; } - rethrowTerminal(terminalError); + // The worker exited between close() failing and delegation. + // Cleanup is safe here and its failures remain suppressed on + // the original close error. + terminalError = closeRemainingResources(terminalError); + } else { + terminalError = closeRemainingResources(terminalError); + } + + // If close() ended up holding the same instance the user already + // caught earlier, suppress the rethrow. The user's catch block + // wraps close() (try-with-resources), and Throwable refuses + // self-suppression. + if (terminalError != null && terminalError == alreadyOwnedByUser) { + terminalError = null; } + rethrowTerminal(terminalError); } @TestOnly @@ -1979,6 +2138,23 @@ public QwpTableBuffer getTableBuffer(String tableName) { return buffer; } + /** + * Test seam over {@link #hasDynamicCredential()}: whether this sender's configured + * credential is re-derived per handshake (an {@code httpTokenProvider}) rather than + * captured once (an {@code httpToken} or {@code httpUsernamePassword}). + *

+ * The tag is set by the builder, several classes away from the orphan drainer whose + * terminal policy consumes it, and a mis-tag is silent at build time. Tagging a + * rotating credential as fixed makes the first {@code 401} of an orphan drain drop a + * {@code .failed} sentinel that nothing in production clears -- replayable rows + * abandoned for good over a token the next pull would have refreshed. Hence a seam a + * test can assert on a real, built sender. + */ + @TestOnly + public boolean isCredentialDynamic() { + return hasDynamicCredential(); + } + /** * Whether this sender is still in delta-encoded mode. Flips to {@code false} * permanently once {@link #disableDeltaDict} fires (a persisted-dictionary @@ -2653,12 +2829,15 @@ public synchronized void startOrphanDrainers( // Install the user listener as the pool's submit-time default so // the drainers submitted below observe it from their first event. drainerPool.setListener(this.drainerListener); - // Route drainer data-loss reports through the sender's own error + // Route the drainers' reports through the sender's own error // dispatcher: async, bounded, and contained exactly like every - // other SenderError. The dispatcher field is read lazily because - // it is created on connect, which can complete after this pool is - // built; a null dispatcher (never connected) leaves the site's own - // LOG line as the only announcement, same as before this sink. + // other SenderError. Two kinds arrive -- the data-loss report when a + // drainer abandons a slot, and the non-terminal faults its drain loop + // rides out (above all a credential the token provider cannot supply, + // which nothing else would ever surface). The dispatcher field is read + // lazily because it is created on connect, which can complete after + // this pool is built; a null dispatcher (never connected) leaves the + // site's own LOG line as the only announcement, same as before this sink. drainerPool.setErrorSink(err -> { SenderErrorDispatcher d = errorDispatcher; if (d != null) { @@ -2965,11 +3144,6 @@ public static int effectiveConnectTimeoutMs(boolean background, int configuredMs return background && configuredMs <= 0 ? DEFAULT_BACKGROUND_CONNECT_TIMEOUT_MS : configuredMs; } - /** - * Builds the per-attempt WebSocket client for {@link #buildAndConnect}. - * Production path delegates to {@link WebSocketClientFactory}; tests may - * install {@link #clientFactoryOverride} to substitute a stub. - */ /** * Best-effort close for a client being abandoned because a JVM Error is * about to be rethrown: under OOM {@code close()} itself can throw, and a @@ -2984,6 +3158,11 @@ private static void closeQuietlyOnError(WebSocketClient client) { } } + /** + * Builds the per-attempt WebSocket client for {@link #buildAndConnect}. + * Production path delegates to {@link WebSocketClientFactory}; tests may + * install {@link #clientFactoryOverride} to substitute a stub. + */ private WebSocketClient newWebSocketClient() { java.util.function.Supplier override = clientFactoryOverride; if (override != null) { @@ -3125,6 +3304,55 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo HttpClientException terminalUpgradeError = null; QwpIngressRoleRejectedException lastRoleReject = null; Endpoint lastEndpoint = null; + // Honor a close/stop that raced this (re)connect before doing any work - a token pull can make a + // blocking network call - mirroring the per-endpoint check at the top of the walk below. Use the + // context's abort gate, not the foreground loop's state: a background drainer must still be able to + // (re)connect during the sender's close sequence, when the foreground loop is already stopped. + if (ctx.isAborted()) { + throw new LineSenderException(ctx.abortMessage()); + } + // Resolve the Authorization header ONCE per (re)connect round, before the endpoint walk. For an + // httpTokenProvider this queries the provider a single time - a fresh token per handshake round, as the + // javadoc documents - NOT once per endpoint: a token-provider failure is cluster-wide (a failed silent + // refresh, or not signed in), not a per-endpoint transport fault, so re-querying it per endpoint would + // hammer the token endpoint with the same dead credential and mislabel the failure as "all endpoints + // unreachable". A throw here is wrapped as QwpCredentialUnavailableException (below): the + // foreground/SYNC initial connect unwraps it and fails fast with the provider's own message, while the + // running background drainer treats it as a transient outage and retries it indefinitely with capped + // backoff (never bounded by the reconnect budget, never terminal), so even a persistent credential + // outage keeps the buffered rows in store-and-forward rather than terminating the sender (Invariant B). + // Mirrors QwpQueryClient, which likewise resolves the credential once before its endpoint walk. + // Publish this thread as being inside the pull BEFORE making it, then re-check cancellation, exactly + // as the per-endpoint connect below does with the WebSocketClient. The pull is the one blocking call + // in the walk that cancel()'s closeTraffic() cannot reach - it runs caller-supplied HttpTokenProvider + // code - and it can outlast close()'s shutdown budget, so cancel() breaks it with an interrupt + // instead. Skipped on the foreground path, where cancellation is null and close() is not racing us. + if (cancellation != null) { + cancellation.publishCredentialPull(Thread.currentThread()); + if (cancellation.isCancelled()) { + cancellation.clearCredentialPull(); + throw new LineSenderException(ctx.abortMessage()); + } + } + final String authHeader; + try { + authHeader = authorizationHeaderSupplier == null ? null : authorizationHeaderSupplier.get(); + } catch (RuntimeException e) { + // Tag the failure CLASS so each context applies the right policy: a credential we cannot acquire is + // not a transport outage. A foreground/SYNC connect unwraps this and rethrows the provider's own + // exception, so build() surfaces the provider's error directly rather than an internal wrapper. The + // running background drainer, by contrast, treats it as a transient outage and retries it + // indefinitely under Invariant B -- never bounding it by the reconnect budget, never latching a + // terminal -- so a recoverable credential outage never drops a producer store-and-forward promised + // to keep alive. + throw new QwpCredentialUnavailableException(e); + } finally { + // Drop the marker as soon as the pull returns, so a later cancel() cannot interrupt this thread + // at an arbitrary point in the walk. Mirrors ConnectCancellation.clear() for the in-flight client. + if (cancellation != null) { + cancellation.clearCredentialPull(); + } + } while (true) { if (ctx.isAborted()) { throw new LineSenderException(ctx.abortMessage()); @@ -3164,7 +3392,10 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo } newClient.connect(ep.host, ep.port); int upgradeTimeoutMs = (int) Math.min(authTimeoutMs, Integer.MAX_VALUE); - newClient.upgrade(WRITE_PATH, upgradeTimeoutMs, authorizationHeader); + // Present the header resolved once above for this handshake. On a failover to a later endpoint + // the same round's token is reused (a token is cluster-wide), so the provider is queried once + // per reconnect round, not once per endpoint. + newClient.upgrade(WRITE_PATH, upgradeTimeoutMs, authHeader); if (cancellation != null) { // connect()+upgrade() completed: this client is no longer // blocking, so drop it from the in-flight handle before it @@ -3648,11 +3879,12 @@ private void dispatchConnectionEvent( * double-signalled either. * * - * @param errorOwnedByCustomHandler whether the async dispatcher has - * already delivered a terminal to a + * @param errorOwnedByCustomHandler whether the async dispatcher has already delivered a terminal to a * user-installed handler + * @param restoreInterrupt close()-scoped carrier that restores any consumed interrupt only after + * the remaining teardown waits have completed */ - private void drainOnClose(boolean errorOwnedByCustomHandler) { + private void drainOnClose(boolean errorOwnedByCustomHandler, boolean[] restoreInterrupt) { if (closeFlushTimeoutMillis <= 0L) { return; } @@ -3686,37 +3918,52 @@ private void drainOnClose(boolean errorOwnedByCustomHandler) { } } long deadlineNanos = System.nanoTime() + closeFlushTimeoutMillis * 1_000_000L; - while (cursorEngine.ackedFsn() < target) { - // Stop on a latched terminal (acks will never reach target); - // surface it only when no other channel already delivered it. - if (errorOwnedByCustomHandler) { - if (cursorSendLoop.getTerminalError() != null) { - return; + restoreInterrupt[0] |= Thread.interrupted(); + try { + while (cursorEngine.ackedFsn() < target) { + // PoolHousekeeper.stop() escalates to interrupt when this close runs past its join budget. A carried + // interrupt makes parkNanos return immediately without clearing the flag, turning the remainder of a + // close drain (up to 60s by default) into a full-core spin. Consume any later interrupt before the + // next paced wait, remember it, and restore the flag once the whole close exits. + if (Thread.interrupted()) { + restoreInterrupt[0] = true; } - } else { - cursorSendLoop.checkError(); - } - if (System.nanoTime() >= deadlineNanos) { - long acked = cursorEngine.ackedFsn(); - // Name the outage the I/O thread is riding out, when there is one. A - // foreground sender now retries endpoint-policy rejections indefinitely, - // so a revoked token reaches the operator HERE, and blaming timeout - // tuning for what is actually an auth failure would misdirect them. - CursorWebSocketSendLoop loop = cursorSendLoop; - Throwable outage = loop == null ? null : loop.lastReconnectError(); - LOG.warn("close() drain timed out after {}ms [target={} acked={}], pending data may be lost{}", - closeFlushTimeoutMillis, target, acked, - outage == null ? "" : "; wire is not draining: " + outage.getMessage()); - throw new LineSenderException("close() drain timed out after ") - .put(closeFlushTimeoutMillis).put(" ms [targetFsn=") - .put(target).put(", ackedFsn=").put(acked) - .put("] - server did not acknowledge ") - .put(target - acked) - .put(outage == null - ? " pending batches; data may be lost (use larger closeFlushTimeoutMillis or smaller batches)" - : " pending batches; the wire is not draining: " + outage.getMessage()); + // Stop on a latched terminal (acks will never reach target); + // surface it only when no other channel already delivered it. + if (errorOwnedByCustomHandler) { + if (cursorSendLoop.getTerminalError() != null) { + return; + } + } else { + cursorSendLoop.checkError(); + } + if (System.nanoTime() >= deadlineNanos) { + long acked = cursorEngine.ackedFsn(); + // Name the outage the I/O thread is riding out, when there is one. A + // foreground sender now retries endpoint-policy rejections indefinitely, + // so a revoked token reaches the operator HERE, and blaming timeout + // tuning for what is actually an auth failure would misdirect them. + CursorWebSocketSendLoop loop = cursorSendLoop; + Throwable outage = loop == null ? null : loop.lastReconnectError(); + LOG.warn("close() drain timed out after {}ms [target={} acked={}], pending data may be lost{}", + closeFlushTimeoutMillis, target, acked, + outage == null ? "" : "; wire is not draining: " + outage.getMessage()); + throw new LineSenderException("close() drain timed out after ") + .put(closeFlushTimeoutMillis).put(" ms [targetFsn=") + .put(target).put(", ackedFsn=").put(acked) + .put("] - server did not acknowledge ") + .put(target - acked) + .put(outage == null + ? " pending batches; data may be lost (use larger closeFlushTimeoutMillis or smaller batches)" + : " pending batches; the wire is not draining: " + outage.getMessage()); + } + java.util.concurrent.locks.LockSupport.parkNanos(50_000L); } - java.util.concurrent.locks.LockSupport.parkNanos(50_000L); + } finally { + // Close the last-iteration race: an interrupt can land after the loop observes its final ACK but + // before it exits. Keep teardown interrupt-neutral and hand the signal back only at close()'s outer + // boundary, after cursorSendLoop.close() and drainerPool.close() have spent their join budgets. + restoreInterrupt[0] |= Thread.interrupted(); } } @@ -3798,6 +4045,11 @@ private void ensureConnected() { default: try { client = reconnectFactory.reconnect(); + } catch (QwpCredentialUnavailableException e) { + // The caller configured the token provider, so surface the provider's own exception (and + // its message) rather than the internal marker: a credential failure on a foreground + // connect is the caller's to see, not a transport-shaped wrapper. + throw e.providerFailure(); } catch (RuntimeException e) { throw e; } catch (Exception e) { @@ -4377,6 +4629,29 @@ private void disableDeltaDict(Throwable cause) { cause); } + /** + * On-wire byte cost of one symbol-dictionary entry, exactly as + * {@code NativeBufferWriter.putString} writes it: {@code [varint utf8Len][utf8]}. + * Both of that method's branches (the ASCII fast path, which reserves + * {@code varintSize(charLen) == varintSize(utf8Len)}, and the two-pass fallback) + * produce this size, so the chunker below sizes frames against the same + * arithmetic the encoder will use rather than an independent estimate. + */ + private int dictionaryEntryWireBytes(int id) { + int utf8Len = NativeBufferWriter.utf8Length(globalSymbolDictionary.getSymbol(id)); + return NativeBufferWriter.varintSize(utf8Len) + utf8Len; + } + + /** + * Whether the {@code Authorization} header is re-derived on every handshake from a caller-supplied + * token provider, rather than being a constant captured once. A rotating credential makes a + * {@code 401} potentially recoverable, which the orphan drainer's terminal policy depends on; see + * {@link #fixedAuthHeader(String)}. + */ + private boolean hasDynamicCredential() { + return authorizationHeaderSupplier != null && !(authorizationHeaderSupplier instanceof FixedAuthHeader); + } + /** * Writes the ids the surviving frames contributed above the persisted prefix back * into {@code .symbol-dict}, immediately, before any new frame can be published. @@ -4401,19 +4676,6 @@ private void disableDeltaDict(Throwable cause) { *

* Healing here, eagerly and in full, restores the invariant before the window opens. */ - /** - * On-wire byte cost of one symbol-dictionary entry, exactly as - * {@code NativeBufferWriter.putString} writes it: {@code [varint utf8Len][utf8]}. - * Both of that method's branches (the ASCII fast path, which reserves - * {@code varintSize(charLen) == varintSize(utf8Len)}, and the two-pass fallback) - * produce this size, so the chunker below sizes frames against the same - * arithmetic the encoder will use rather than an independent estimate. - */ - private int dictionaryEntryWireBytes(int id) { - int utf8Len = NativeBufferWriter.utf8Length(globalSymbolDictionary.getSymbol(id)); - return NativeBufferWriter.varintSize(utf8Len) + utf8Len; - } - private void healPersistedDictionary(PersistedSymbolDict pd) { if (pd == null || !deltaDictEnabled) { return; @@ -5045,7 +5307,10 @@ private void validateTableName(CharSequence name) { if (name.length() > MAX_TABLE_NAME_LENGTH) { throw new LineSenderException("table name too long [maxLength=" + MAX_TABLE_NAME_LENGTH + "]"); } - throw new LineSenderException("table name contains illegal characters: " + name); + // sanitize the rejected name before it reaches the message (and any log/terminal): a name that + // failed validation can carry BOM/bidi/zero-width/control chars that would otherwise reorder, hide + // or forge what a human reads, matching how the ILP name/error render escapes untrusted text + throw new LineSenderException("table name contains illegal characters: ").putAsPrintable(name); } } @@ -5059,6 +5324,24 @@ public Endpoint(String host, int port) { } } + /** + * A constant {@code Authorization} header value. Its identity as a type - not the value it yields - is + * what {@link #hasDynamicCredential()} reads, so the drainer can apply the right terminal policy to a + * {@code 401}. See {@link #fixedAuthHeader(String)}. + */ + private static final class FixedAuthHeader implements Supplier { + private final String header; + + private FixedAuthHeader(String header) { + this.header = header; + } + + @Override + public String get() { + return header; + } + } + private final class ReconnectSupplier implements CursorWebSocketSendLoop.ReconnectFactory { /** * Optional caller-owned liveness gate. {@code null} means this factory @@ -5086,6 +5369,11 @@ String abortMessage() { return abortCheck != null ? abortMessage : "sender closed during connect"; } + @Override + public boolean hasDynamicCredential() { + return QwpWebSocketSender.this.hasDynamicCredential(); + } + /** * True when this factory serves a background drainer. Background * connects share buildAndConnect's endpoint walk and hostTracker diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java index a9b5545ed..facd872bb 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java @@ -29,6 +29,7 @@ import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.QwpCredentialUnavailableException; import io.questdb.client.cutlass.qwp.client.QwpDurableAckMismatchException; import io.questdb.client.cutlass.qwp.client.QwpIngressRoleRejectedException; import io.questdb.client.cutlass.qwp.client.QwpRoleMismatchException; @@ -40,6 +41,7 @@ import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.LockSupport; +import java.util.function.Consumer; /** * Empties one orphan slot, then exits. Owned by @@ -92,6 +94,70 @@ public final class BackgroundDrainer implements Runnable { * cluster-wide misconfig hang the drainer forever. */ public static final int DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS = 16; + /** + * Attempt threshold for {@code 401}/{@code 403} rejections an orphan drainer rides out before it + * may quarantine the slot, when - and only when - the credential is a ROTATING one + * ({@link CursorWebSocketSendLoop.ReconnectFactory#hasDynamicCredential()}). Against a constant + * credential a rejection stays terminal on the first sweep, as it always was. + *

+ * The attempt threshold is necessary but not sufficient: the rejection must also PERSIST for at least + * the dwell returned by {@link #dynamicCredentialAuthDwellNanos(long)} - {@code + * reconnectMaxDurationMillis}, clamped so an unbounded configuration cannot disable the escalation - + * measured from the first rejection of the current uninterrupted run. A transient in between (role + * reject, transport, credential-unavailable) restarts that measurement, because time the drainer spent + * unable to reach anyone is not time the credential spent rejected. This wall-clock floor gives IdP + * signing-key and resource-server JWKS caches time to converge even when capped backoff can accumulate + * six attempts in only a few seconds. A credential that stays rejected still reaches a human after both + * thresholds are met rather than pinning the slot and a drainer-pool worker forever. + * Note it cannot repair a PERSISTENT clock skew: the provider keeps serving the same cached token, so + * those sweeps eventually exhaust both thresholds and quarantine, which is the right end state for a + * condition that is not healing. + */ + public static final int DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS = 6; + /** + * Hard ceiling on rotating-credential {@code 401}/{@code 403} rejections within one no-ack-progress + * episode, whatever the dwell says. + *

+ * The dwell is a wall clock anchored at the first rejection of the current run, and the + * capability-gap, role-reject and transport arms all rewind that anchor - deliberately, so an + * unrelated outage cannot satisfy the floor for free. The attempt threshold beside it is only ever + * reset by real ack progress. That asymmetry is what the gate needs in the ordinary case and what + * breaks it in the pathological one: a cluster that ALTERNATES - reject, blip, reject, blip - rewinds + * the anchor before every rejection, so the elapsed dwell is always ~0, the AND can never be + * satisfied, and the ride-out never ends. The drainer then sweeps forever with no ack progress: no + * {@code .failed} sentinel, no {@code DATA_LOSS} report, the slot lock held, and one worker of a + * fixed-size {@link BackgroundDrainerPool} pinned for the life of the process - the exact outcome + * {@link #MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS} exists to prevent, reached by a different route. + *

+ * An attempt cap is the right backstop precisely because nothing rewinds it: it shares the attempt + * counter's episode scope, so it bounds the ride-out however the rejections are spaced. It is set far + * above {@link #DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS} on purpose - the dwell decides every + * case that is not pathological, and this only ever fires when the dwell has been rendered + * unsatisfiable. + */ + public static final int MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS_PER_EPISODE = 256; + /** + * Ceiling on the rotating-credential {@code 401}/{@code 403} wall-clock dwell, independent of the user + * knob it is otherwise derived from. + *

+ * The dwell is an AND with the attempt threshold - both must be exhausted before an orphan slot is + * quarantined - and it is taken from {@code reconnect_max_duration_millis}, which is validated only as + * {@code > 0} and whose documented way to ask a reconnect never to give up is {@code Long.MAX_VALUE}. + * {@code TimeUnit} saturates that, so the dwell conjunct became unsatisfiable and the ride-out never + * ended: the drainer swept forever, never wrote the {@code .failed} sentinel, never reported + * {@code DATA_LOSS}, and pinned the slot lock plus one worker of a FIXED-size + * {@link BackgroundDrainerPool} for the life of the process, starving every other orphan slot. The + * capability-gap gate below survives the same saturation because it is an OR; this gate cannot be an OR + * without losing the dwell floor that stops a healing credential being abandoned in the seconds capped + * backoff needs to spend six attempts, so it is clamped instead. + *

+ * Set to the DEFAULT reconnect budget rather than a new figure: five minutes is already what this design + * calls a settle budget, and a larger configured value is a statement about reconnect persistence, not + * about how long a credential failure may stay hidden from an operator. A smaller configured value is + * honoured as-is, so tuning down still works. + */ + public static final long MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS = + CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS; private static final Logger LOG = LoggerFactory.getLogger(BackgroundDrainer.class); /** How often to wake and re-check ackedFsn vs target. */ private static final long POLL_NANOS = 50_000_000L; // 50 ms @@ -113,6 +179,36 @@ public final class BackgroundDrainer implements Runnable { private final long sfMaxTotalBytes; private final String slotPath; private final long syncIntervalNanos; + /** + * Escalation counters for the two bounded ride-outs, held per DRAIN rather than per call, plus the + * ack watermark that scopes them. + *

+ * The counters were locals of {@link #connectWithDurableAckRetry()}, which {@link #run()} re-enters + * after every mid-drain terminal - so each recycle refilled the budget it is supposed to spend. A + * cluster that flaps (connect accepted, drop, {@code 401}, recycle, repeat) looped forever with no ack + * progress: the escalation never arrived, the slot lock was never released, and one of {@code + * max_background_drainers} workers - four by default - stayed pinned, so four such slots starve every + * other orphan slot of a drainer. No data is lost, but none is delivered either, and no operator ever + * sees the quarantine. + *

+ * "No ack progress" is the actual condition, and {@code ackProgressWatermark} is what measures it. + * Spanning the whole drain instead over-counts in the opposite direction: a drain that connects, + * DELIVERS, then meets a later gap accumulates both runs toward one threshold, so 16 sweeps that were + * never consecutive quarantine a slot the cluster is still draining - abandoning replayable rows behind + * a {@code .failed} sentinel nothing in production clears. Both terminals are documented as CONSECUTIVE + * sweeps, so {@link #noteAckProgress(long)} ends the episode the moment the wire delivers anything: a + * durable ack past this watermark is positive proof the cluster accepted us, which is strictly stronger + * evidence than the role-reject and transport arms that already reset these. A flap that delivers + * nothing never advances the watermark and still escalates on schedule. + *

+ * The wall-clock helpers beside them ({@code capabilityGapElapsedNanos}, {@code lastCapabilityGapNanos}) + * deliberately stay per-call: they measure an UNINTERRUPTED run, and a successful connect plus a drain + * is an interruption, so a fresh call should start their accounting over. + */ + private long ackProgressWatermark = Long.MIN_VALUE; + private int capabilityGapAttempts; + private int dynamicCredentialAuthAttempts; + private long firstDynamicCredentialAuthFailureNanos; /** Latest known {@code engine.ackedFsn()}; published for visibility. */ private volatile long ackedFsn = -1L; /** @@ -121,10 +217,15 @@ public final class BackgroundDrainer implements Runnable { * reference an already-closed engine once the drain ends. */ private volatile CursorSendEngine engineForTesting; - // Sink for SenderError.dataLoss reports fired when this drainer - // permanently abandons a slot behind a .failed sentinel. Volatile for the - // same reason as `listener`: applied by the pool at submit time, read on - // the drainer thread. Null means the abandonment is announced only via + @TestOnly + private Consumer afterAckPollHookForTesting; + // Sink for this drainer's SenderError reports. Two feeds: the dataLoss fired + // when it permanently abandons a slot behind a .failed sentinel, and the + // non-TERMINAL reports of the drain loop itself -- an unobtainable credential + // above all, plus the server rejections it replays through -- which run() + // forwards by handing the loop a SenderErrorDispatcher over this sink. + // Volatile for the same reason as `listener`: applied by the pool at submit + // time, read on the drainer thread. Null means both are announced only via // LOG -- a NOP for apps without an slf4j binding -- which is exactly the // silence this sink exists to break. private volatile SenderErrorHandler errorSink; @@ -253,6 +354,23 @@ public BackgroundDrainer() { CursorWebSocketSendLoop.DEFAULT_MAX_HEAD_FRAME_REJECTIONS, 0L, 0L); } + /** + * The effective wall-clock dwell the rotating-credential {@code 401} ride-out uses: the configured + * reconnect budget, clamped to {@link #MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS} so that an + * "effectively unbounded" configuration cannot disable the escalation entirely. + *

+ * Public and pure so the clamp can be asserted directly. The alternative - proving it end to end - means + * waiting out the ceiling, which is five minutes of wall clock in a test. + * + * @param reconnectMaxDurationMillis the configured {@code reconnect_max_duration_millis} + * @return the dwell in nanoseconds, always finite + */ + public static long dynamicCredentialAuthDwellNanos(long reconnectMaxDurationMillis) { + return Math.min( + TimeUnit.MILLISECONDS.toNanos(reconnectMaxDurationMillis), + TimeUnit.MILLISECONDS.toNanos(MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS)); + } + /** * Budgeted connect with retry on whole-cluster durable-ack unavailability: * the initial connect, and re-entered from {@link #run()} whenever a @@ -281,8 +399,9 @@ public BackgroundDrainer() { * transport error -- are retried indefinitely (Invariant B) and never * consume the budget. Either transient restarts the attempt count and wall * clock so only uninterrupted capability-gap sweeps can escalate. - * Genuine terminals (auth failure, non-421 upgrade reject) preserve - * the original behavior: mark failed, exit. + * Genuine terminals (a constant-credential auth failure, a rotating-credential + * auth failure that exhausts both its attempt threshold and wall-clock floor, + * or a non-421 upgrade reject) mark the slot failed and exit. * * @return a fresh durable-ack-capable client, or {@code null} if * {@link #outcome} has been set to FAILED or STOPPED @@ -301,10 +420,27 @@ public WebSocketClient connectWithDurableAckRetry() { // intervening role or transport state resets the episode: after the // cluster leaves the capability-gap state, later gaps must establish a // fresh consecutive run before quarantine is permitted. - int capabilityGapAttempts = 0; + // (capabilityGapAttempts is a field - see its declaration) + // 401/403 sweeps ridden out so far, counted only for a ROTATING credential (see + // DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS). Not reset by the transient arms below, so a + // credential alternating rejected/unreachable inside ONE connect attempt cannot refill it and + // stall the quarantine an operator needs to see - unlike the capability-gap episode, and unlike + // the dwell anchor beside it, which does restart because it measures persistence rather than + // count. + // + // It is a FIELD, so a mid-drain recycle cannot refill it - see its declaration. + // (dynamicCredentialAuthAttempts is a field) + // The rotating-auth wall-clock floor is anchored at the first 401/403 of the CURRENT run of + // rejections: a transient class in between (role reject, transport, credential-unavailable) restarts + // it, because the dwell measures how long the rejection persisted, not how long the drainer has been + // running. A recycle is NOT such an interruption for this anchor: it is a field, so the dwell spans + // recycles, which is what lets a flapping credential reach the escalation at all - the attempt + // threshold alone cannot, because the gate is an AND. The attempt threshold, unlike this, never + // resets during the drain. A zero value means no rejection has been observed. + // (firstDynamicCredentialAuthFailureNanos is a field) // Wall-clock time accumulated across uninterrupted gap-to-gap // intervals of the current episode; escalates once it reaches - // capabilityGapBudgetNanos (or the attempt cap fires first). + // reconnectBudgetNanos (or the attempt cap fires first). long capabilityGapElapsedNanos = 0L; // Timestamp of the previous capability-gap sweep; 0 = the next gap // charges nothing because a fresh episode is starting. @@ -318,8 +454,13 @@ public WebSocketClient connectWithDurableAckRetry() { // more tolerance would buy exactly none. TimeUnit clamps at Long.MAX_VALUE, which // is the intended "effectively unbounded". CursorWebSocketSendLoop's dwell // conversion guards the same way for the same reason. - final long capabilityGapBudgetNanos = + final long reconnectBudgetNanos = TimeUnit.MILLISECONDS.toNanos(reconnectMaxDurationMillis); + // The rotating-401 gate needs its own, clamped copy: it is an AND, so an unbounded value there is + // not "effectively unbounded" but "never escalates". The capability-gap gate below keeps the raw + // budget - it is an OR, so its attempt cap fires regardless. + final long dynamicCredentialAuthDwellNanos = + dynamicCredentialAuthDwellNanos(reconnectMaxDurationMillis); // Observability-only counter for the transient all-replica window; // never consulted for escalation (Invariant B). int roleRejectAttempts = 0; @@ -338,17 +479,74 @@ public WebSocketClient connectWithDurableAckRetry() { try { return clientFactory.reconnect(); } catch (QwpAuthFailedException | WebSocketUpgradeException e) { - // Genuinely non-retriable across the cluster (auth 401/403, or a - // non-421 upgrade reject): waiting will not fix it, so quarantine - // immediately under the orphan reconnect policy. - String msg = e.getMessage(); - LOG.error("drainer terminal upgrade/auth error for slot {}: {}", slotPath, msg); - lastErrorMessage = msg; - String reason = "auth/upgrade: " + msg; - OrphanScanner.markFailed(slotPath, reason); - dispatchDataLoss(reason); - outcome = DrainOutcome.FAILED; - return null; + // A non-421 upgrade reject, and a 401/403 against a CONSTANT credential, are genuinely + // non-retriable across the cluster: waiting will not fix them, so quarantine immediately + // under the orphan reconnect policy. + // + // A 401/403 against a ROTATING credential is a different condition. The header is + // re-derived from the caller's token provider on every sweep, so the rejection can be a + // window that heals itself, and the next sweep carries a freshly pulled token. Quarantining + // on the first one would permanently abandon replayable data - nothing in production clears + // the .failed sentinel - on a fault that repairs itself. Require BOTH enough rejection + // attempts and a minimum wall-clock dwell before quarantine: capped backoff can otherwise spend + // the attempt threshold in seconds, far sooner than IdP signing-key/JWKS caches commonly + // converge. + boolean retryDynamicCredentialAuth = false; + long dynamicCredentialAuthElapsedNanos = 0L; + if (e instanceof QwpAuthFailedException && clientFactory.hasDynamicCredential()) { + dynamicCredentialAuthAttempts++; + long now = System.nanoTime(); + if (firstDynamicCredentialAuthFailureNanos == 0L) { + firstDynamicCredentialAuthFailureNanos = now; + } + dynamicCredentialAuthElapsedNanos = now - firstDynamicCredentialAuthFailureNanos; + // Both thresholds still gate the quarantine - a healing credential is never abandoned + // early - but the dwell is the CLAMPED one, so a saturated reconnect_max_duration_millis + // cannot make the second conjunct unsatisfiable and turn "ride it out" into "never + // escalate". See MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS. + // The AND of the two thresholds, under a cap that nothing can rewind. The transient + // arms below deliberately restart the dwell anchor, so an ALTERNATING cluster - + // reject, blip, reject, blip - re-anchors before every rejection and leaves the + // elapsed dwell permanently at ~0, making the AND unsatisfiable and the ride-out + // endless. The attempt counter shares this episode's scope (only ack progress clears + // it), so capping on it bounds the ride-out however the rejections are spaced while + // leaving the dwell to decide every non-pathological case. See + // MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS_PER_EPISODE. + retryDynamicCredentialAuth = + (dynamicCredentialAuthAttempts < DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS + || dynamicCredentialAuthElapsedNanos < dynamicCredentialAuthDwellNanos) + && dynamicCredentialAuthAttempts < MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS_PER_EPISODE; + } + if (retryDynamicCredentialAuth) { + lastErrorMessage = e.getMessage(); + // An auth rejection is unrelated to any open durable-ack episode: we reached a node and + // it refused the credential, which says nothing about its batch cap. Restart the episode + // so a later gap gets its full settle budget, exactly as the transport arm below does. + capabilityGapAttempts = 0; + capabilityGapElapsedNanos = 0L; + lastCapabilityGapNanos = 0L; + // the CLAMPED dwell, which is the one the gate above just applied. Reporting the raw + // reconnect_max_duration_millis here was misleading exactly where the clamp matters: a + // saturated budget rendered as "dwell 12ms/9223372036854775807ms", telling an operator the + // ride-out would never end when it was in fact bounded at the ceiling. + LOG.warn("drainer slot {} attempt {} (threshold {}, dwell {}ms/{}ms): " + + "the rotating credential was rejected ({}); retrying with a freshly pulled " + + "token after backoff", + slotPath, dynamicCredentialAuthAttempts, + DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS, + dynamicCredentialAuthElapsedNanos / 1_000_000L, + dynamicCredentialAuthDwellNanos / 1_000_000L, e.getMessage()); + // fall through to the shared capped-backoff block + } else { + String msg = e.getMessage(); + LOG.error("drainer terminal upgrade/auth error for slot {}: {}", slotPath, msg); + lastErrorMessage = msg; + String reason = "auth/upgrade: " + msg; + OrphanScanner.markFailed(slotPath, reason); + dispatchDataLoss(reason); + outcome = DrainOutcome.FAILED; + return null; + } } catch (QwpRoleMismatchException | QwpIngressRoleRejectedException e) { // INVARIANT B: every reachable endpoint is a REPLICA right now. // A replica is promotable and a primary will reappear, so this is @@ -365,6 +563,16 @@ public WebSocketClient connectWithDurableAckRetry() { capabilityGapAttempts = 0; capabilityGapElapsedNanos = 0L; lastCapabilityGapNanos = 0L; + // The rotating-401 dwell measures how long the REJECTION has persisted, so time spent in + // an unrelated state is not part of it. Restart its anchor for the same reason the + // capability-gap episode restarts above: without this, a 401, then an outage outlasting the + // dwell, then a sixth rejection satisfies both thresholds at once and quarantines a slot on + // a credential that was only rejected for seconds - abandoning replayable rows behind a + // .failed sentinel nothing in production clears. The attempt counter deliberately does NOT + // reset (a credential alternating rejected/unreachable must not refill it indefinitely), and + // the clamped dwell (MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS) keeps the escalation reachable + // regardless of what reconnect_max_duration_millis is set to. + firstDynamicCredentialAuthFailureNanos = 0L; BackgroundDrainerListener l = listener; if (l != null) { try { @@ -388,6 +596,14 @@ public WebSocketClient connectWithDurableAckRetry() { // stays terminal for the drainer -- give the cluster a bounded // settle budget (rolling upgrade), then quarantine the slot. capabilityGapAttempts++; + // Symmetry with the arm above, which restarts the capability-gap episode because an auth + // rejection says nothing about a node's batch cap: a capability gap says nothing about the + // credential either. We reached a node and it answered - it simply cannot do durable ack - + // so this is not time the credential spent REJECTED, and charging it to the rotating-401 + // dwell would let a rolling upgrade satisfy that floor for free. The settle budget below can + // legitimately run for the whole reconnect budget, which is exactly the span the dwell is + // meant to require of an uninterrupted rejection. + firstDynamicCredentialAuthFailureNanos = 0L; long now = System.nanoTime(); if (lastCapabilityGapNanos != 0L) { // Charge only the interval since the PREVIOUS gap sweep, @@ -399,7 +615,7 @@ public WebSocketClient connectWithDurableAckRetry() { lastCapabilityGapNanos = now; long elapsedMs = capabilityGapElapsedNanos / 1_000_000L; boolean exhausted = capabilityGapAttempts >= DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS - || capabilityGapElapsedNanos >= capabilityGapBudgetNanos; + || capabilityGapElapsedNanos >= reconnectBudgetNanos; BackgroundDrainerListener l = listener; if (exhausted) { LOG.error("drainer giving up on slot {} after {} durable-ack-mismatch attempts ({}ms): {}", @@ -455,6 +671,11 @@ public WebSocketClient connectWithDurableAckRetry() { // WebSocketUpgradeException) and is intentionally retried under // Invariant B -- but it is NOT a transport outage, so log it // truthfully below rather than mislabelling it "cluster unreachable". + // The same holds for a credential the client cannot ACQUIRE + // (QwpCredentialUnavailableException extends LineSenderException, so it + // matches none of the typed arms above): retried indefinitely here, for + // the reason CursorWebSocketSendLoop's matching arm spells out, but + // named for what it is. lastErrorMessage = t.getMessage(); // This unrelated state breaks the consecutive capability-gap // run. Restart both halves of the settle budget so a later gap @@ -462,6 +683,16 @@ public WebSocketClient connectWithDurableAckRetry() { capabilityGapAttempts = 0; capabilityGapElapsedNanos = 0L; lastCapabilityGapNanos = 0L; + // The rotating-401 dwell measures how long the REJECTION has persisted, so time spent in + // an unrelated state is not part of it. Restart its anchor for the same reason the + // capability-gap episode restarts above: without this, a 401, then an outage outlasting the + // dwell, then a sixth rejection satisfies both thresholds at once and quarantines a slot on + // a credential that was only rejected for seconds - abandoning replayable rows behind a + // .failed sentinel nothing in production clears. The attempt counter deliberately does NOT + // reset (a credential alternating rejected/unreachable must not refill it indefinitely), and + // the clamped dwell (MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS) keeps the escalation reachable + // regardless of what reconnect_max_duration_millis is set to. + firstDynamicCredentialAuthFailureNanos = 0L; long nowWarn = System.nanoTime(); if (nowWarn - lastTransportWarnNanos >= 5_000_000_000L) { if (t instanceof QwpVersionMismatchException) { @@ -478,6 +709,18 @@ public WebSocketClient connectWithDurableAckRetry() { + "QWP protocol version ({}); retrying (rolling-upgrade window) -- " + "if this persists the client is version-incompatible with the cluster", slotPath, t.getMessage()); + } else if (t instanceof QwpCredentialUnavailableException) { + // Nothing was attempted on the wire: the configured token provider + // threw instead of handing over a credential (a failed silent + // refresh, a revoked or expired refresh token, an unreachable IdP, + // or an interactive sign-in not finished yet). The cluster may be + // perfectly healthy, so "cluster unreachable" sends the operator + // after a network fault that does not exist while the slot's rows + // sit undrained. Point at the credential instead. + LOG.warn("drainer slot {}: the token provider failed to supply a credential ({}); " + + "retrying after backoff -- the slot stays un-drained until a token " + + "is available", + slotPath, t.getMessage()); } else { LOG.warn("drainer slot {}: cluster unreachable ({}), retrying after backoff", slotPath, t.getMessage()); @@ -497,7 +740,7 @@ public WebSocketClient connectWithDurableAckRetry() { long sleepMillis = backoffMillis + jitter; if (boundedByBudget) { sleepMillis = Math.min(sleepMillis, - Math.max(0L, (capabilityGapBudgetNanos - capabilityGapElapsedNanos) / 1_000_000L)); + Math.max(0L, (reconnectBudgetNanos - capabilityGapElapsedNanos) / 1_000_000L)); } if (sleepMillis > 0L && !stopRequested) { long parkDeadlineNanos = System.nanoTime() + sleepMillis * 1_000_000L; @@ -547,6 +790,27 @@ public boolean isStopRequested() { return stopRequested; } + /** + * Pre-ages the rotating-credential rejection anchor so a test can reach the + * {@link #MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS} ceiling without waiting it out in real time. + *

+ * The clamp on the connect loop's dwell is otherwise unobservable end to end. Every other drainer test + * configures a dwell far below the ceiling, where {@code Math.min} returns its first argument either + * way, so the loop reverting to the raw budget leaves them all green - while a saturated + * {@code reconnect_max_duration_millis} makes the gate's second conjunct unsatisfiable and an orphan + * drainer sweeps forever, pinning the slot lock and one pool worker. Proving it honestly instead means + * waiting out the ceiling, which is five minutes of wall clock per run. + * + * @param ageNanos how long ago the current run of rejections should appear to have begun + */ + @TestOnly + public void ageDynamicCredentialAuthAnchorForTesting(long ageNanos) { + long anchor = System.nanoTime() - ageNanos; + // 0 is the "no rejection observed yet" sentinel, and the connect loop overwrites it on the next + // rejection - which would silently undo the ageing and make the test pass for the wrong reason. + firstDynamicCredentialAuthFailureNanos = anchor != 0L ? anchor : 1L; + } + /** * Engine this drainer constructed, or {@code null} until {@link #run()} * gets past engine construction. The reference outlives the drain, so @@ -557,6 +821,16 @@ public CursorSendEngine getEngineForTesting() { return engineForTesting; } + /** + * Installs a test-only hook after the drain loop reads {@code ackedFsn} but before it checks the loop's + * terminal latch. This makes the late-ack ordering deterministic without changing either production + * thread's publication order. + */ + @TestOnly + public void setAfterAckPollHookForTesting(Consumer hook) { + afterAckPollHookForTesting = hook; + } + /** * Periodic SF checkpoint interval this drainer inherited from the * adopting sender at construction time. @@ -618,6 +892,31 @@ private void dispatchDataLoss(String reason) { } } + /** + * Ends any open escalation episode once the wire has durably acked something new. Both bounded + * ride-outs are documented as CONSECUTIVE sweeps, and a durable ack past the watermark is the + * cluster accepting this drainer - stronger evidence than a role reject or a transport error, both + * of which already reset these counters. Without it the two per-drain counters span every session of + * the drain, so a rolling upgrade that delivers between two gap windows accumulates both toward one + * threshold and quarantines a slot that was still draining. + *

+ * Runs on the drainer thread only, from {@link #run()}'s poll loop, so the plain field reads and + * writes need no synchronisation. Deliberately NOT called from {@link #connectWithDurableAckRetry()}: + * a successful connect on its own delivers nothing, and treating it as progress would restore the + * per-call refill that let a flapping cluster recycle forever. + * + * @param acked the engine's current durable-ack watermark + */ + private void noteAckProgress(long acked) { + if (acked <= ackProgressWatermark) { + return; + } + ackProgressWatermark = acked; + capabilityGapAttempts = 0; + dynamicCredentialAuthAttempts = 0; + firstDynamicCredentialAuthFailureNanos = 0L; + } + @Override public void run() { runnerThread = Thread.currentThread(); @@ -625,6 +924,12 @@ public void run() { CursorSendEngine engine = null; WebSocketClient client = null; CursorWebSocketSendLoop loop = null; + // Async delivery arm for the drain loop's own SenderError reports. Built + // only when a sink is installed, and only once per run() -- it outlives + // the mid-drain loop recycles below, which would otherwise churn a thread + // per wire session. Closed by the finally, after loop.close(), so errors + // dispatched during the loop's shutdown still reach the sink. + SenderErrorDispatcher loopErrorDispatcher = null; try { // Scanner results are only snapshots. Serialize adoption against // a producer's close -> quarantine rename -> fresh-slot recreate @@ -762,20 +1067,66 @@ public void run() { outcome = DrainOutcome.SUCCESS; return; } + // Seed the progress watermark from what a previous run already durably acked, so only acks + // THIS drain earns count as progress. Seeding from the -1 field default would make the first + // poll of a partially-drained slot read as progress and hand back a budget the initial connect + // had legitimately spent. + ackProgressWatermark = engine.ackedFsn(); client = connectWithDurableAckRetry(); if (client == null) { // outcome already set (FAILED or STOPPED); markFailed sentinel // already dropped on the FAILED path. return; } - // One iteration per wire session. Re-entered ONLY when a mid-drain - // reconnect sweep hit a durable-ack CAPABILITY gap: that is the - // exact rolling-upgrade condition the settle budget in - // connectWithDurableAckRetry() exists for, so it must not - // quarantine on the first sweep the way the initial-connect path - // never does. The engine stays alive across sessions (it holds the - // slot lock; only loop + client are recycled), and target remains - // valid -- the slot is orphaned, nothing appends to it. + // Read the sink once: like `listener` it is volatile because the pool + // applies it at submit time and it is consumed on the drainer thread. + SenderErrorHandler sink = errorSink; + if (sink != null) { + // The I/O thread must never run the sink inline -- it is caller-supplied + // code and may block -- so it reaches the sink through the same bounded, + // drop-oldest, off-thread arm the foreground sender uses. + // + // TERMINAL is dropped on the way through: on an ORPHAN loop it does not + // mean what it means to a foreground producer. It is the loop handing the + // slot back to this drainer, which then decides -- ride the fault out and + // finish the drain, or quarantine and report the abandonment itself with + // dispatchDataLoss. Forwarding it would announce a dead producer for a + // rotating credential the very next sweep accepts, and would double-report + // the quarantine the drainer already names. Everything the loop rides out + // (RETRIABLE / RETRIABLE_OTHER) has no such owner and is forwarded verbatim. + loopErrorDispatcher = new SenderErrorDispatcher( + err -> { + if (err.getAppliedPolicy() != SenderError.Policy.TERMINAL) { + // This sink belongs to the live sender, while err's FSNs belong to the orphan + // engine being drained. Strip that foreign correlation span before forwarding; + // otherwise an operator can join it to unrelated live rows with the same FSNs. + sink.onError(new SenderError( + err.getCategory(), + err.getAppliedPolicy(), + err.getServerStatusByte(), + err.getServerMessage(), + err.getMessageSequence(), + SenderError.NO_MESSAGE_SEQUENCE, + SenderError.NO_MESSAGE_SEQUENCE, + err.getTableName(), + err.getDetectedAtNanos())); + } + }, + SenderErrorDispatcher.DEFAULT_CAPACITY, "qdb-sf-drainer-error-dispatcher"); + } + + // One iteration per wire session. Re-entered on either of the two + // RECOVERABLE mid-drain terminals the recycle branch below tests + // for -- a durable-ack CAPABILITY gap, or a 401/403 against a + // ROTATING credential. Both are conditions a later sweep can clear + // (a rolling upgrade settling; the next pulled token being + // accepted), so neither may quarantine on its first sweep the way + // the initial-connect path never does; connectWithDurableAckRetry() + // owns the bounded budget for each. Every other wire error still + // quarantines the slot without re-entering. The engine stays alive + // across sessions (it holds the slot lock; only loop + client are + // recycled), and target remains valid -- the slot is orphaned, + // nothing appends to it. drain: while (!stopRequested) { loop = new CursorWebSocketSendLoop( @@ -790,10 +1141,22 @@ public void run() { poisonMinEscalationWindowMillis, catchUpCapGapMinEscalationWindowMillis, CursorWebSocketSendLoop.ReconnectPolicy.ORPHAN); + // Without this the loop's ridden-out reports -- above all + // "credential-unavailable", the one endpoint-policy failure an ORPHAN + // loop retries rather than latching -- are dispatched into a null, and + // the outage is announced only by a throttled slf4j WARN, which is a + // NOP in an app with no binding configured. The foreground sender wires + // the same arm (QwpWebSocketSender.ensureConnected, where it builds the + // loop); an orphan drainer rides out the same faults and + // must be just as observable, or a revoked token reads as a disk-sizing + // problem once SF fills. Null when no sink is installed, which + // setErrorDispatcher accepts and dispatchError treats as before. + loop.setErrorDispatcher(loopErrorDispatcher); loop.start(); while (!stopRequestedOrInterrupted()) { long acked = engine.ackedFsn(); + noteAckProgress(acked); this.ackedFsn = acked; if (acked >= target) { outcome = DrainOutcome.SUCCESS; @@ -801,6 +1164,10 @@ public void run() { slotPath, target, acked); return; } + Consumer afterAckPollHook = afterAckPollHookForTesting; + if (afterAckPollHook != null) { + afterAckPollHook.accept(loop); + } try { loop.checkError(); } catch (Throwable t) { @@ -814,17 +1181,45 @@ public void run() { if (t.getCause() instanceof Error) { throw (Error) t.getCause(); } - if (loop.capabilityGapTerminal() != null) { - // Capability gap mid-drain: recycle the wire, NOT - // the slot. connectWithDurableAckRetry() owns the - // episode budget (16 consecutive gap sweeps / - // wall clock) and drops the sentinel itself if the - // gap persists. The loop's own failed sweep is not - // counted toward the fresh episode -- an off-by-one - // that is immaterial at budget 16. - LOG.warn("drainer slot {}: durable-ack capability gap " - + "mid-drain ({}), re-entering settle budget", - slotPath, t.getMessage()); + if (loop.capabilityGapTerminal() != null || loop.authTerminal() != null) { + // The I/O thread publishes a durable ack before it latches a later terminal. + // That publication can land after the poll at the top of this iteration but + // before checkError() observes the terminal. Re-read here so recycling the wire + // cannot carry a spent escalation budget across progress we actually made. + long ackedAfterTerminal = engine.ackedFsn(); + noteAckProgress(ackedAfterTerminal); + this.ackedFsn = ackedAfterTerminal; + if (ackedAfterTerminal >= target) { + outcome = DrainOutcome.SUCCESS; + LOG.info("drainer fully drained slot {} before recoverable terminal " + + "(target={}, acked={})", + slotPath, target, ackedAfterTerminal); + return; + } + // Mid-drain RECOVERABLE terminal: recycle the wire, NOT + // the slot. connectWithDurableAckRetry() owns the matching + // bounded budget and drops the sentinel itself if the + // condition persists, so a fault that heals -- a rolling + // upgrade settling, or a rotating credential's next token + // being accepted -- never abandons replayable data on its + // first sweep. The loop's own failed sweep is not counted + // toward the fresh budget -- an off-by-one immaterial at + // either budget. Two classes route here: + // - capability gap: the 16 consecutive-sweep / wall-clock + // settle budget. + // - rotating-credential 401/403 (authTerminal): the + // DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS ride-out. + // Only an ORPHAN loop with a dynamic credential sets it; + // a constant credential stays fatal and quarantines below. + if (loop.authTerminal() != null) { + LOG.warn("drainer slot {}: rotating credential rejected mid-drain ({}), " + + "re-entering the rotating-401 ride-out", + slotPath, t.getMessage()); + } else { + LOG.warn("drainer slot {}: durable-ack capability gap " + + "mid-drain ({}), re-entering settle budget", + slotPath, t.getMessage()); + } try { loop.close(); } catch (Throwable closeFailure) { @@ -940,6 +1335,41 @@ public void run() { slotPath, e.getMessage()); } } + if (loopErrorDispatcher != null) { + // After loop.close() so anything the I/O loop reported on its way + // out is still admitted, and before the sink can outlive this run. + // Safe on the failed-stop path above too: a still-live I/O thread's + // later offer() is rejected by the closed dispatcher rather than + // resurrecting its delivery thread. + // + // Interrupt-neutral for this ONE call, then restored. Everything above + // deliberately runs with the flag set -- stopRequestedOrInterrupted() leaves + // it so loop.close()'s latch await throws rather than blocking on a wedged + // I/O thread -- but SenderErrorDispatcher.close() drains by joining its + // delivery thread against a refreshed deadline, and its catch re-asserts the + // flag before looping. Arriving with the flag set makes Thread.join(millis) + // throw on arrival on every pass, so the loop BUSY-SPINS for as long as the + // delivery thread stays alive, capped at the 100ms drain deadline. + // + // It still drains correctly -- join() returns normally the moment the thread + // is no longer alive, so neither the wait's duration nor which errors get + // delivered changes. What changes is the cost: measured at 15k-53k join + // attempts, one core pinned for that window, per closing drainer, and + // max_background_drainers is 4 by default. Same clear-and-restore the sibling + // teardowns use (QueryWorker.shutdown, QwpQueryClient.close, + // QwpWebSocketSender.close). + final boolean wasInterrupted = Thread.interrupted(); + try { + loopErrorDispatcher.close(); + } catch (Throwable e) { + LOG.warn("drainer slot {}: error dispatcher close failed ({})", + slotPath, e.getMessage()); + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } if (client != null && ioThreadStopped) { // Skipped on a failed stop: the thread may be mid-send on // this very client; ioLoop's finally closes the loop's diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerPool.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerPool.java index a35bf9340..67110a59c 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerPool.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainerPool.java @@ -55,11 +55,25 @@ * that don't exit in time (typically parked in a blocking native connect * that neither unpark nor interrupt cancels) are left to finish on their * own — the pool's underlying executor uses daemon threads so they don't - * block JVM exit. An interrupted {@code close()} skips the graceful - * window: every active drainer is stop-signaled immediately, then the - * executor is shut down hard. A drainer cut down mid-drain exits STOPPED - * with its unacked rows still in SF — re-adopted by the next orphan scan, - * never dropped. + * block JVM exit. A drainer cut down mid-drain exits STOPPED with its + * unacked rows still in SF — re-adopted by the next orphan scan, never + * dropped. + *

+ * An interrupt that lands during the graceful window ends it: every + * active drainer is stop-signaled immediately, then the executor is shut + * down hard. An interrupt the caller merely arrives with does not, + * and the usual caller is the one that carries one — a task cancelled by + * {@code ExecutorService.shutdownNow()} closing its {@code QuestDB} handle. + * {@link io.questdb.client.cutlass.qwp.client.QwpWebSocketSender#close()} + * clears the flag for the duration of its close and restores it on the way + * out, because a carried flag used to make every wait beneath it throw on + * arrival — which is how a reap sweep came to report slots with their SF + * flock still held. The graceful window therefore runs in full on that + * path, and a cancelled close costs up to + * {@code GRACEFUL_DRAIN_MILLIS + STOP_GRACE_MILLIS} per sender that has an + * orphan drainer actively delivering. That is the intended trade: the + * split stop above already exempts drainers that are merely retrying a + * connect, so what is waited on is a drainer with rows on the wire. */ public final class BackgroundDrainerPool implements QuietCloseable { diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 5a66c3029..6643bf036 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -33,6 +33,7 @@ import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.NativeBufferWriter; import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.QwpCredentialUnavailableException; import io.questdb.client.cutlass.qwp.client.QwpDurableAckMismatchException; import io.questdb.client.cutlass.qwp.client.QwpIngressRoleRejectedException; import io.questdb.client.cutlass.qwp.client.QwpRoleMismatchException; @@ -86,16 +87,6 @@ */ public final class CursorWebSocketSendLoop implements QuietCloseable { - /** - * Default cadence for the keepalive PING the I/O loop emits while - * waiting on STATUS_DURABLE_ACK frames. See - * {@link #sendDurableAckKeepaliveIfDue()} for the rationale: the OSS - * server only flushes pending durable-ack frames on inbound recv - * events, so an opted-in idle client has to prod it. {@code 200} ms - * trades one PING per 200 ms per idle opted-in connection for - * sub-second confirmation latency once the upload completes - * server-side. {@code 0} or negative disables the keepalive entirely. - */ /** * Bounded-await backstop for {@link #close()}: the maximum time close() * waits for the I/O thread to stop (count down {@code shutdownLatch}) @@ -119,6 +110,16 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { * rather than waiting it out. */ public static final long DEFAULT_CLOSE_SHUTDOWN_AWAIT_MILLIS = 30_000L; + /** + * Default cadence for the keepalive PING the I/O loop emits while + * waiting on STATUS_DURABLE_ACK frames. See + * {@link #sendDurableAckKeepaliveIfDue()} for the rationale: the OSS + * server only flushes pending durable-ack frames on inbound recv + * events, so an opted-in idle client has to prod it. {@code 200} ms + * trades one PING per 200 ms per idle opted-in connection for + * sub-second confirmation latency once the upload completes + * server-side. {@code 0} or negative disables the keepalive entirely. + */ public static final long DEFAULT_DURABLE_ACK_KEEPALIVE_INTERVAL_MILLIS = 200L; public static final long DEFAULT_PARK_NANOS = 50_000L; // 50us idle backoff /** @@ -437,6 +438,17 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // is why a revoked token surfaced to operators as "sf_max_total_bytes too small". private volatile Throwable lastReconnectError; private volatile Thread ioThread; + // Typed marker for a ROTATING-credential auth terminal (401/403): set (before the + // terminalError latch, so a checkError() caller that observes the latch is guaranteed to + // observe this marker too) when a mid-drain reconnect sweep on an ORPHAN drainer whose + // credential is dynamic (reconnectFactory.hasDynamicCredential()) threw QwpAuthFailedException. + // The orphan drainer consults it to route such a rejection into its bounded rotating-401 + // ride-out (BackgroundDrainer.connectWithDurableAckRetry) instead of quarantining the slot on + // the first rejection: the header is re-derived from the token provider every attempt, so a 401 + // can be a self-healing window that a freshly pulled token clears -- the same reasoning the + // capability-gap recycle rests on. A CONSTANT credential never sets it (it stays fatal), and + // foreground reconnects never set it either. Write-once alongside terminalError. + private volatile QwpAuthFailedException authTerminal; // Typed marker for a durable-ack CAPABILITY-GAP terminal: set (before the // terminalError latch, so a checkError() caller that observes the latch is // guaranteed to observe this marker too) when a reconnect sweep threw @@ -1038,6 +1050,19 @@ public static WebSocketClient connectWithRetry( LOG.error("{} hit terminal upgrade error, won't retry: {}", contextLabel, e.getMessage()); throw e; + } catch (QwpCredentialUnavailableException e) { + // A credential the client cannot ACQUIRE (the configured token provider threw) is NOT a + // transport outage: retrying the connect cannot conjure a token the provider will not hand + // over, so fail fast with the provider's own exception rather than burn the whole connect + // budget treating it as a reachable-server problem (which would block build() for up to + // maxDurationMillis, default 5 min, and surface a transport-shaped wrapper). Mirrors the + // foreground OFF-mode connect (QwpWebSocketSender) and the background reconnect loop above, + // which both give credential acquisition its own terminal handling; only this SYNC + // initial-connect path lacked it. QwpCredentialUnavailableException is a LineSenderException, + // disjoint from the HttpClientException-based terminal set above, so it reaches here. + LOG.error("{} could not acquire a credential, won't retry: {}", + contextLabel, e.getMessage()); + throw e.providerFailure(); } catch (Throwable e) { if (e instanceof Error) { // JVM/programming failure (OOM, LinkageError): not a @@ -1176,6 +1201,22 @@ public void checkError() { } } + /** + * The typed rotating-credential auth terminal (401/403), or {@code null} if the loop's terminal + * (if any) is a different failure class. Non-null only after {@link #checkError()} started + * throwing: the marker is written before the {@code terminalError} latch, both on the I/O thread. + *

+ * Consumer contract: the orphan drainer ({@code BackgroundDrainer}) checks this after a + * {@code checkError()} throw to route a mid-drain 401/403 against a ROTATING credential into its + * bounded rotating-401 ride-out (the header is re-derived per attempt, so the rejection can be a + * self-healing window a freshly pulled token clears) rather than quarantining the slot. A constant + * credential never sets it. Package-private on purpose -- the foreground sender must not branch + * on it. + */ + QwpAuthFailedException authTerminal() { + return authTerminal; + } + /** * The typed durable-ack capability-gap terminal, or {@code null} if the * loop's terminal (if any) is a different failure class. Non-null only @@ -1709,11 +1750,13 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM // INVARIANT B: a store-and-forward loop must NEVER terminate on a // wall-clock reconnect budget. A replica-only / all-endpoints-replica // window is TRANSIENT -- a replica gets promoted, a primary reappears -- - // so this background loop retries for as long as it is running, backing - // off between attempts. Endpoint-policy failures (auth / non-421 - // upgrade / durable-ack capability gap) are terminal only for orphan - // drainers. Foreground senders retry them from asynchronous startup onward - // so a credential or cluster capability rotation cannot stop the producer. SF + // and so is a token-provider failure -- the IdP becomes reachable again, + // or the user completes an interactive sign-in -- so this background loop + // retries all of them for as long as it is running, backing off between + // attempts. Endpoint-policy failures (auth / non-421 upgrade / + // durable-ack capability gap) are terminal only for orphan drainers. + // Foreground senders retry them from asynchronous startup onward so a + // credential or cluster capability rotation cannot stop the producer. SF // exhaustion is surfaced to the PRODUCER as append backpressure, never // here. reconnect_max_duration_millis is intentionally NOT consulted by // THIS loop. Its holders pass it explicitly where it does apply: the @@ -1781,11 +1824,37 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM resetCatchUpCapGapEpisode(); } catch (QwpAuthFailedException | WebSocketUpgradeException e) { if (endpointPolicyFailureIsTerminal()) { - // Orphans return control to their quarantine owner. - // WebSocketUpgradeException reaching here is always non-421: - // role rejects are classified into the transient branch below. - LOG.error("terminal upgrade error during {} -- won't retry: {}", - phase, e.getMessage()); + // A 401/403 against a ROTATING credential on an orphan drainer is NOT uniformly + // fatal: the Authorization header is re-derived from the caller's token provider on + // every attempt, so the rejection can be a self-healing window (a revocation landing + // mid-flight, the IdP rotating signing keys, clock skew past the token's own margin) + // that a freshly pulled token clears. Hand it back as a RECOVERABLE auth-terminal -- + // exactly as a capability gap does -- so BackgroundDrainer recycles it through + // connectWithDurableAckRetry()'s bounded rotating-401 ride-out instead of + // quarantining on the first rejection and permanently abandoning replayable data. + // A CONSTANT credential (or a non-421 upgrade reject) is uniformly rejected across + // the cluster and stays fatal. This gates on reconnectPolicy == ORPHAN, so an + // INITIALIZING foreground 401 (endpointPolicyFailureIsTerminal via !hasEverConnected) + // is never masked and still reaches the caller. + final boolean rotatingCredentialReject = reconnectPolicy == ReconnectPolicy.ORPHAN + && e instanceof QwpAuthFailedException + && reconnectFactory.hasDynamicCredential(); + if (rotatingCredentialReject) { + if (terminalError == null) { + // Publish the marker before terminalError, the volatile first-writer-wins + // latch the owner observes -- same ordering as capabilityGapTerminal. + authTerminal = (QwpAuthFailedException) e; + } + LOG.warn("rotating credential rejected during {} -- handing the slot back to the " + + "drainer to retry with a freshly pulled token: {}", + phase, e.getMessage()); + } else { + // Orphans return control to their quarantine owner. + // WebSocketUpgradeException reaching here is always non-421: + // role rejects are classified into the transient branch below. + LOG.error("terminal upgrade error during {} -- won't retry: {}", + phase, e.getMessage()); + } long fromFsn = engine.ackedFsn() + 1L; long toFsn = Math.max(fromFsn, engine.publishedFsn()); SenderError err = new SenderError( @@ -1856,6 +1925,41 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM phase, attempts, e.getMessage()); lastLogNanos = now; } + } catch (QwpCredentialUnavailableException e) { + // The token provider threw instead of returning a credential (a failed silent refresh, an + // interactive sign-in in progress on another thread, or not signed in yet). In the RUNNING + // background drainer this is a TRANSIENT outage like any other under Invariant B: the provider + // hands over a token again once the IdP is reachable or the user finishes signing in, and the + // un-acked rows stay safe in on-disk SF meanwhile. So retry indefinitely with capped backoff -- + // NEVER bound by a wall-clock budget and NEVER latch a terminal, which would drop a producer + // that store-and-forward promised to keep alive on a recoverable fault. The foreground/SYNC + // initial connect still fails fast with the provider's own exception (connectWithRetry, and the + // OFF-mode connect in QwpWebSocketSender), because a connectivity error is only the caller's to + // see DURING initialization, not after the drainer is running. + // + // Ends any open cap-gap episode like every other unrelated reconnect state: we never reached a + // node to observe its batch cap, so this outage's wall clock must not accrue toward the orphan + // cap-gap dwell (see MAX_CATCHUP_CAP_GAP_ATTEMPTS). + resetCatchUpCapGapEpisode(); + lastReconnectError = e; + // Retrying must not be programmatically INVISIBLE, exactly as for the auth/upgrade and + // durable-ack policy failures above: a revoked refresh token or a permanently unreachable IdP + // is not self-healing, yet flush() keeps returning success while SF absorbs the rows. Without + // this dispatch the only signal is a throttled slf4j WARN - and this library ships embedded, + // frequently with no binding configured - until SF fills and the failure resurfaces as ring + // backpressure, pointing the operator at disk sizing instead of at their credentials. It stays + // RETRIABLE, not TERMINAL: the handler learns the wire is down while the producer stays alive + // and no data is at risk (Invariant B). + dispatchRetriedEndpointPolicyFailure( + SenderError.Category.SECURITY_ERROR, "credential-unavailable: " + e.getMessage()); + long now = System.nanoTime(); + if (now - lastLogNanos >= RECONNECT_LOG_THROTTLE_NANOS) { + LOG.warn("{} attempt {}: the token provider failed ({}); retrying with capped backoff -- " + + "the sender keeps buffering to SF and recovers once a token is available", + phase, attempts, e.getMessage()); + lastLogNanos = now; + } + // fall through to the shared capped-backoff block } catch (QwpRoleMismatchException | QwpIngressRoleRejectedException e) { // Role mismatch: every reachable endpoint role-rejected the // upgrade -- right now they are all replicas / primary-catchup. @@ -3527,6 +3631,24 @@ public enum ReconnectPolicy { public interface ReconnectFactory { WebSocketClient reconnect() throws Exception; + /** + * Whether this factory re-derives its {@code Authorization} header from a caller-supplied token + * provider on every attempt, rather than presenting a constant captured once. + *

+ * The orphan drainer's terminal policy reads this. A {@code 401} against a CONSTANT credential is a + * permanent misconfiguration, so quarantining the slot on the first one is correct. Against a + * ROTATING credential the same {@code 401} can be a recoverable window - clock skew past the + * token's own skew margin, a revocation landing mid-flight, an identity provider rotating signing + * keys - and a later attempt carries a freshly pulled token, so quarantining immediately would + * abandon replayable data permanently on a fault that heals itself. + *

+ * Default: {@code false}, the conservative answer. A factory that cannot tell (a test double, a + * transport with no credential at all) keeps the pre-existing fail-fast behaviour. + */ + default boolean hasDynamicCredential() { + return false; + } + /** * Cancellable variant of {@link #reconnect()}. The loop passes a * per-attempt {@link ConnectCancellation} so a transport that blocks @@ -3581,11 +3703,33 @@ public static final class ConnectCancellation { // Latched once close() requested cancellation. Written by the owner // thread (cancel); read by the I/O thread's pre-connect guard. private volatile boolean cancelled; + // The I/O thread while it is inside a credential pull -- the one + // blocking call in the connect walk that closeTraffic() cannot reach, + // because it runs caller-supplied HttpTokenProvider code. Written by + // the I/O thread only (publishCredentialPull/clearCredentialPull); + // read by the owner thread (cancel). null when no pull is in flight. + private volatile Thread credentialPullThread; + + public void clearCredentialPull() { + credentialPullThread = null; + } public boolean isCancelled() { return cancelled; } + /** + * I/O-thread hook: record this thread as being about to enter a + * credential pull, BEFORE the blocking call. Pairs with + * {@link #cancel()} the same way {@link #publish(WebSocketClient)} + * does, except the break lever is an interrupt rather than + * {@code closeTraffic()} -- a token provider is caller code and owns + * no socket the sender can shut down. + */ + public void publishCredentialPull(Thread thread) { + credentialPullThread = thread; + } + /** * I/O-thread hook: record the client the walk is about to block on, * BEFORE the blocking {@code connect()}. Pairs with {@link #cancel()} @@ -3623,6 +3767,24 @@ void cancel() { if (c != null) { c.closeTraffic(); } + // A credential pull is caller code, so closeTraffic() cannot reach it, yet it can block far + // longer than close()'s shutdown budget: OidcDeviceAuth.getToken() waits up to + // 6 x httpTimeoutMillis (180s by default) behind a peer's silent refresh, against a 30s + // DEFAULT_CLOSE_SHUTDOWN_AWAIT_MILLIS. During an IdP outage the drainer sits inside a pull for + // most of every retry cycle, so close() lands there routinely, not just in a narrow race. An + // interrupt is the only lever that reaches a Java-level wait; OidcDeviceAuth converts it into a + // provider failure, which the reconnect loop treats as a transient outage and then observes the + // abort. Fires while the connect walk is inside its credential-pull window, which the walk enters + // whenever it carries a cancellation -- including with no token provider configured, since it + // publishes the marker before it looks at the supplier. That costs nothing: close() sets + // running=false before cancel(), so every path a late interrupt can reach is already winding down, + // and the I/O thread's exit is native frees plus a latch countdown, none of it interruptible. + // It does NOT cover a provider stalled in an OS-level TCP connect, which ignores interrupts -- + // close() still loud-fails on its budget there, as it did before. + Thread t = credentialPullThread; + if (t != null) { + t.interrupt(); + } } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java index 34168b44d..183876a71 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java @@ -978,14 +978,6 @@ private static PersistedSymbolDict openExisting(FilesFacade ff, String filePath, } } - /** - * Validates every chunk and copies the entries of each proven-good one into - * {@code dstAddr}, in a single pass. {@code dstAddr} must have room for {@code len} - * bytes -- the entry region is a subset of the file, so that always suffices. - * Stops at the first chunk that is torn, fails its CRC, or is internally - * inconsistent, exactly as the two-pass version did, so the trusted prefix is - * unchanged. - */ /** * Stats {@code filePath} and, when the stat fails, captures {@code errno} with NO * intervening call. @@ -1019,6 +1011,14 @@ private static long statLength(FilesFacade ff, String filePath, int[] errnoOut) } } + /** + * Validates every chunk and copies the entries of each proven-good one into + * {@code dstAddr}, in a single pass. {@code dstAddr} must have room for {@code len} + * bytes -- the entry region is a subset of the file, so that always suffices. + * Stops at the first chunk that is torn, fails its CRC, or is internally + * inconsistent, exactly as the two-pass version did, so the trusted prefix is + * unchanged. + */ private static RecoveryScan scanAndCopyRecoveredChunks(long inputAddr, int len, long dstAddr) { Varint v = new Varint(); int count = 0; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java b/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java index 1bfd2617e..9f01f0ab8 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java @@ -222,10 +222,13 @@ public ColumnBuffer getOrCreateColumn(CharSequence name, byte type, boolean useN inProgressColumnCount++; return col; } - throw new LineSenderException( - name.length() > MAX_COLUMN_NAME_LENGTH ? "column name too long [maxLength=" + MAX_COLUMN_NAME_LENGTH + "]" - : "column name contains illegal characters: " + name - ); + if (name.length() > MAX_COLUMN_NAME_LENGTH) { + throw new LineSenderException("column name too long [maxLength=" + MAX_COLUMN_NAME_LENGTH + "]"); + } + // sanitize the rejected name before it reaches the message (and any log/terminal): a name that failed + // validation can carry BOM/bidi/zero-width/control chars that would otherwise reorder, hide or forge what + // a human reads, matching how the ILP name/error render escapes untrusted text + throw new LineSenderException("column name contains illegal characters: ").putAsPrintable(name); } public ColumnBuffer getOrCreateDesignatedTimestampColumn(byte type) { diff --git a/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java b/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java index d5ff3db45..0778104b9 100644 --- a/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java +++ b/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java @@ -24,6 +24,8 @@ package io.questdb.client.impl; +import java.util.concurrent.TimeUnit; + /** * Daemon thread that periodically asks both pools to reap idle / over-age * slots. Owned by {@link QuestDBImpl}; one instance per {@code QuestDB} @@ -36,10 +38,13 @@ final class PoolHousekeeper { // in flight when close() arrives finishes well within this join (C1 fix). // The recovery build that precedes the drain is bounded separately -- // recoverers force initial_connect_mode=OFF, so the build makes at most one - // connect attempt rather than a SYNC reconnect-budget retry (M1). The lone - // case that can still overrun this join is an in-flight connect to a - // black-holed host (no application-level connect timeout in the transport); - // see the residual-window note on SenderPool.recoverOneSlotStep. + // connect attempt rather than a SYNC reconnect-budget retry (M1). A recovery + // build also pulls a credential when a token provider is configured, and + // that wait dwarfs this join; stop() escalates to an interrupt for it. The + // lone case that survives even that is an in-flight connect to a black-holed + // host, which blocks in a syscall no interrupt breaks (the transport exposes + // no application-level connect timeout); see the residual-window note on + // SenderPool.recoverOneSlotStep. static final long STOP_TIMEOUT_MILLIS = 2_000; private final long intervalMillis; @@ -66,11 +71,84 @@ void stop() { synchronized (signalLock) { signalLock.notifyAll(); } + // Clear the caller's cancellation for the duration and hand it back at the end -- the shape + // QueryWorker.shutdown() and QwpQueryClient.close() already use. Thread.join(millis) consults the + // CALLING thread's interrupt flag before it ever looks at whether the target is alive, so a caller + // that arrives interrupted -- a close() from a task cancelled by shutdownNow(), or from a finally on + // a thread the application cancelled -- made the first join throw at 0 ms and skip the escalation + // below entirely. That is the one case the escalation is most needed in: it exists because the + // target may be parked in a credential pull only an interrupt can break, and skipping it returns + // from close() with the recoverer still holding its store-and-forward slot flock. The flag is + // restored before returning, so the caller's own cancellation bookkeeping still sees it. A fresh + // interrupt during either join is handled the same way: remember it, finish the shutdown protocol, + // then restore it. + boolean callerWasInterrupted = Thread.interrupted(); try { - thread.join(STOP_TIMEOUT_MILLIS); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); + callerWasInterrupted |= joinIgnoringCallerInterrupts(thread, STOP_TIMEOUT_MILLIS); + if (thread.isAlive()) { + // The stop flag only reaches the loop BETWEEN steps. A step blocked inside a recovery + // build is unreachable by it, and since recovery builds acquired a token provider the + // longest such block is a credential pull: OidcDeviceAuth.getToken() documents a wait of up + // to six times httpTimeoutMillis behind a peer's refresh, plus a token-store lock wait, + // which together dwarf this join. Returning anyway leaves the recoverer holding its + // store-and-forward slot flock after close() has returned, so an immediate reopen fails + // with "sf slot already in use" and the detached build's engine, mmaps and I/O thread leak + // -- the very window this pool's per-slot ids and the drain_orphans(false) forced on + // recovery builds exist to eliminate. + // + // Interrupt and re-join. The waits this is aimed at are interruptible: acquireForGetToken + // polls a timed tryLock, and FileTokenStore's two lock waits abandon and re-assert the flag. + // The pull then throws, the step's caller swallows it (recovery is best-effort), and the + // loop reaches its stop check and releases the flock on its own. + // + // Not ALL of the pull is interruptible, and the join above is the only bound on the rest: + // the token POST's connect, send, await and parse phases run on the native HTTP client + // (raw fd + epoll/kqueue), which no interrupt breaks -- each is bounded by + // httpTimeoutMillis, and DNS resolution is not bounded at all. So a pull already inside its + // round trip outlives both joins, exactly as an in-flight connect to a black-holed host + // does. This escalation shortens the common case; it does not make the window impossible. + // + // The flag must not outlive the interrupt's target. Sender.close() and QwpQueryClient + // close() are interrupt-neutral precisely because this thread goes on to close delegates: + // a CARRIED flag makes CountDownLatch.await return instantly and would report a flock still + // held that was released fine. + thread.interrupt(); + callerWasInterrupted |= joinIgnoringCallerInterrupts(thread, STOP_TIMEOUT_MILLIS); + } + } finally { + if (callerWasInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + + /** + * Waits up to the supplied budget without letting cancellation skip the caller's remaining shutdown + * work. Every {@link InterruptedException} clears the caller's flag, so remember it and spend only the + * remainder of the original budget before handing the information back to {@link #stop()}. + *

+ * Package-private so SenderPool's direct recovery driver can use the same deadline-preserving shutdown + * primitive. Keeping the two stop paths identical matters when an interrupt arrives during the first + * join: it must be remembered without skipping the target interrupt and second join that follow. + */ + static boolean joinIgnoringCallerInterrupts(Thread target, long timeoutMillis) { + final long timeoutNanos = TimeUnit.MILLISECONDS.toNanos(timeoutMillis); + final long deadlineNanos = System.nanoTime() + timeoutNanos; + boolean callerWasInterrupted = false; + while (target.isAlive()) { + final long remainingNanos = deadlineNanos - System.nanoTime(); + if (remainingNanos <= 0L) { + break; + } + final long waitMillis = TimeUnit.NANOSECONDS.toMillis(remainingNanos); + final int waitNanos = (int) (remainingNanos - TimeUnit.MILLISECONDS.toNanos(waitMillis)); + try { + target.join(waitMillis, waitNanos); + } catch (InterruptedException e) { + callerWasInterrupted = true; + } } + return callerWasInterrupted; } private void runLoop() { @@ -84,9 +162,10 @@ private void runLoop() { // forced OFF (at most one connect attempt, never a SYNC // reconnect-budget retry -- M1), and we re-check stop every step, so // a close() landing mid-recovery normally only waits out a single - // bounded drain and the join in stop() does not time out. The sole - // residual overrun is an in-flight connect to a black-holed host; - // see SenderPool.recoverOneSlotStep. + // bounded drain and the join in stop() does not time out. A step + // blocked in a credential pull is broken by stop()'s interrupt; the + // sole residual overrun is an in-flight connect to a black-holed + // host, which no interrupt breaks. See SenderPool.recoverOneSlotStep. // While recovery still has work we skip the idle wait so the backlog // drains promptly; once done we fall back to the normal interval. // No-op once recovery completes or the pool is closing. Best-effort: diff --git a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java index ca3aec15e..8e858c019 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java +++ b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.QueryException; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import org.jetbrains.annotations.TestOnly; @@ -91,6 +92,7 @@ public final class QueryClientPool implements AutoCloseable { private final int maxSize; private final int minSize; private final AtomicInteger nextSlotIndex = new AtomicInteger(); + private final HttpTokenProvider tokenProvider; private final Condition workerReleased; private volatile boolean closed; // Upper bound on the Query.close() drain wait; see @@ -113,7 +115,7 @@ public QueryClientPool( long maxLifetimeMillis ) { this(configurationString, minSize, maxSize, acquireTimeoutMillis, - idleTimeoutMillis, maxLifetimeMillis, null); + idleTimeoutMillis, maxLifetimeMillis, null, null, null); } // Constructor exposing the connectHook seam. Production (QuestDBImpl) passes @@ -131,7 +133,7 @@ public QueryClientPool( Consumer connectHook ) { this(configurationString, minSize, maxSize, acquireTimeoutMillis, - idleTimeoutMillis, maxLifetimeMillis, connectHook, null); + idleTimeoutMillis, maxLifetimeMillis, connectHook, null, null); } // Constructor exposing both the connectHook and startHook seams. Production @@ -148,12 +150,28 @@ public QueryClientPool( long maxLifetimeMillis, Consumer connectHook, Consumer startHook + ) { + this(configurationString, minSize, maxSize, acquireTimeoutMillis, + idleTimeoutMillis, maxLifetimeMillis, connectHook, startHook, null); + } + + QueryClientPool( + String configurationString, + int minSize, + int maxSize, + long acquireTimeoutMillis, + long idleTimeoutMillis, + long maxLifetimeMillis, + Consumer connectHook, + Consumer startHook, + HttpTokenProvider tokenProvider ) { if (minSize < 0 || maxSize < 1 || minSize > maxSize) { throw new IllegalArgumentException("invalid pool sizing: min=" + minSize + ", max=" + maxSize); } this.connectHook = connectHook != null ? connectHook : QwpQueryClient::connect; this.startHook = startHook != null ? startHook : QueryWorker::start; + this.tokenProvider = tokenProvider; this.configurationString = configurationString; this.minSize = minSize; this.maxSize = maxSize; @@ -578,6 +596,9 @@ public void setCreationWaitRetryHookForTesting(Runnable hook) { private QueryWorker createUnlocked() { QwpQueryClient client = QwpQueryClient.fromConfig(configurationString); try { + if (tokenProvider != null) { + client.withBearerTokenProvider(tokenProvider); + } connectHook.accept(client); } catch (Throwable e) { // Catch Throwable, not just RuntimeException: connect() runs a heavy diff --git a/core/src/main/java/io/questdb/client/impl/QueryWorker.java b/core/src/main/java/io/questdb/client/impl/QueryWorker.java index 040ade630..568ac0dfe 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryWorker.java +++ b/core/src/main/java/io/questdb/client/impl/QueryWorker.java @@ -206,6 +206,10 @@ void releaseToPool(long gen) { } void shutdown() { + // Take the caller's cancellation out of the way for the whole teardown and hand it back at the + // end. Every join below would otherwise throw on arrival rather than on a real timeout; see the + // join site and QwpQueryClient.close() for what that costs. + boolean callerWasInterrupted = Thread.interrupted(); shuttingDown = true; signalLock.lock(); try { @@ -236,9 +240,15 @@ void shutdown() { // the worker thread and the client's native socket/buffers. } try { + // Interrupt-neutral for the same reason QwpQueryClient.close() is: reapIdle() reaches + // here on the housekeeper thread, which PoolHousekeeper.stop() may have interrupted to + // break a credential pull. A carried flag makes this join throw instantly without ever + // checking whether the dispatch thread exited, so client.close() below would run + // alongside a still-live dispatch thread. Restored by the caller-flag handling in + // shutdown()'s outer finally. thread.join(SHUTDOWN_JOIN_MILLIS); } catch (InterruptedException e) { - Thread.currentThread().interrupt(); + callerWasInterrupted = true; } } finally { // close() must run even if cancel()/join() threw, otherwise the @@ -249,6 +259,9 @@ void shutdown() { client.close(); } catch (Throwable ignored) { } + if (callerWasInterrupted) { + Thread.currentThread().interrupt(); + } } } diff --git a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java index 75ee0a267..574d6b59e 100644 --- a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java +++ b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.QuestDB; import io.questdb.client.Query; import io.questdb.client.Sender; @@ -70,10 +71,33 @@ public QuestDBImpl( ) { this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, - housekeeperIntervalMillis, queryCloseTimeoutMillis, null, null, + housekeeperIntervalMillis, queryCloseTimeoutMillis, null, errorHandler, connectionListener, drainerListener); } + public QuestDBImpl( + String ingestConfig, + String queryConfig, + int senderMin, + int senderMax, + int queryMin, + int queryMax, + long acquireTimeoutMillis, + long idleTimeoutMillis, + long maxLifetimeMillis, + long housekeeperIntervalMillis, + long queryCloseTimeoutMillis, + HttpTokenProvider tokenProvider, + SenderErrorHandler errorHandler, + SenderConnectionListener connectionListener, + BackgroundDrainerListener drainerListener + ) { + this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, + acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, + housekeeperIntervalMillis, queryCloseTimeoutMillis, null, null, + tokenProvider, errorHandler, connectionListener, drainerListener); + } + // Test-only constructor exposing the senderFactory and connectHook seams: // production uses the public overload above, which passes null for both -> // the real native build/connect paths. White-box error-safety tests in @@ -98,7 +122,7 @@ public QuestDBImpl( this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, housekeeperIntervalMillis, QueryClientPool.DEFAULT_CLOSE_QUERY_TIMEOUT_MILLIS, - senderFactory, connectHook, null, null, null); + senderFactory, connectHook, null, null, null, null); } // Full constructor adding the ingest-side errorHandler/connectionListener/ @@ -120,6 +144,7 @@ public QuestDBImpl( long queryCloseTimeoutMillis, IntFunction senderFactory, Consumer connectHook, + HttpTokenProvider tokenProvider, SenderErrorHandler errorHandler, SenderConnectionListener connectionListener, BackgroundDrainerListener drainerListener @@ -135,10 +160,10 @@ public QuestDBImpl( // build() never blocks on a slow / reachable-but-not-acking // server; the housekeeper drives it via runStartupRecoveryStep(). true, - errorHandler, connectionListener, drainerListener); + errorHandler, connectionListener, drainerListener, tokenProvider); builtQueryPool = new QueryClientPool( queryConfig, queryMin, queryMax, acquireTimeoutMillis, - idleTimeoutMillis, maxLifetimeMillis, connectHook); + idleTimeoutMillis, maxLifetimeMillis, connectHook, null, tokenProvider); builtQueryPool.closeQueryTimeoutMillis(queryCloseTimeoutMillis); builtHousekeeper = new PoolHousekeeper(builtSenderPool, builtQueryPool, housekeeperIntervalMillis); builtHousekeeper.start(); diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java index a8a10d621..912d3b444 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderPool.java +++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionListener; import io.questdb.client.SenderError; @@ -154,6 +155,7 @@ public final class SenderPool implements AutoCloseable { private final BackgroundDrainerListener drainerListener; private final SenderErrorHandler errorHandler; private final long idleTimeoutMillis; + private final HttpTokenProvider tokenProvider; // Delivery channel for recovery-delegate errors that pass the // isRecoveryEventUserRelevant filter. Pool-owned so a slow user handler // can never stall the recovery driver / housekeeper thread or overrun @@ -354,7 +356,7 @@ public SenderPool( long maxLifetimeMillis ) { this(configurationString, minSize, maxSize, acquireTimeoutMillis, - idleTimeoutMillis, maxLifetimeMillis, null, false, null, null, null, null, null, null, null); + idleTimeoutMillis, maxLifetimeMillis, null, false, null, null, null, null, null, null, null, null); } // Test-only constructor exposing the senderFactory seam: production builds @@ -397,7 +399,7 @@ public SenderPool( ) { this(configurationString, minSize, maxSize, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, senderFactory, - deferStartupRecovery, null, null, null, null, null, null, null); + deferStartupRecovery, null, null, null, null, null, null, null, null); } // Test-only constructor adding a deterministic fault hook for the ownership @@ -416,7 +418,7 @@ public SenderPool( ) { this(configurationString, minSize, maxSize, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, senderFactory, - deferStartupRecovery, null, null, null, postFactoryHook, null, null, null); + deferStartupRecovery, null, null, null, postFactoryHook, null, null, null, null); } @TestOnly @@ -433,6 +435,7 @@ public static SenderPool createWithRecoveryControlsForTesting( return new SenderPool(configurationString, minSize, maxSize, acquireTimeoutMillis, Long.MAX_VALUE, Long.MAX_VALUE, senderFactory, false, null, null, null, null, recoveryThreadFactory, recoveryWaiter, + null, beforeFailedRecoveryJoinHook); } @@ -457,7 +460,27 @@ public static SenderPool createWithRecoveryControlsForTesting( this(configurationString, minSize, maxSize, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, senderFactory, deferStartupRecovery, errorHandler, connectionListener, - drainerListener, null, null, null, null); + drainerListener, null); + } + + SenderPool( + String configurationString, + int minSize, + int maxSize, + long acquireTimeoutMillis, + long idleTimeoutMillis, + long maxLifetimeMillis, + IntFunction senderFactory, + boolean deferStartupRecovery, + SenderErrorHandler errorHandler, + SenderConnectionListener connectionListener, + BackgroundDrainerListener drainerListener, + HttpTokenProvider tokenProvider + ) { + this(configurationString, minSize, maxSize, acquireTimeoutMillis, + idleTimeoutMillis, maxLifetimeMillis, senderFactory, + deferStartupRecovery, errorHandler, connectionListener, + drainerListener, null, null, null, tokenProvider, null); } private SenderPool( @@ -475,6 +498,7 @@ private SenderPool( Runnable postFactoryHook, ThreadFactory recoveryThreadFactory, Runnable recoveryWaiter, + HttpTokenProvider tokenProvider, Runnable beforeFailedRecoveryJoinHook ) { if (minSize < 0 || maxSize < 1 || minSize > maxSize) { @@ -483,6 +507,7 @@ private SenderPool( this.errorHandler = errorHandler; this.connectionListener = connectionListener; this.drainerListener = drainerListener; + this.tokenProvider = tokenProvider; this.senderFactory = senderFactory != null ? senderFactory : this::defaultSender; // An injected factory (tests) drives recovery too, preserving the // white-box recovery seam; production recovery forces OFF-mode connects @@ -511,6 +536,11 @@ private SenderPool( // us whether SF is on and, if so, the base slot id to derive // per-sender ids from. Sender.LineSenderBuilder probe = Sender.builder(configurationString); + if (tokenProvider != null) { + // Validate fixed-config credentials vs. the provider even when the + // pool is fully lazy and no sender is built yet. + probe.httpTokenProvider(tokenProvider); + } this.storeAndForward = probe.isStoreAndForwardEnabled(); this.slotBaseId = this.storeAndForward ? probe.getConfiguredSenderId() : null; this.sfDir = this.storeAndForward ? probe.getConfiguredSfDir() : null; @@ -730,7 +760,12 @@ boolean runStartupRecoveryStep() { * minutes-long block a {@code reconnect_*}-tuned config used to cause (M1). * One residual window remains and is NOT closed here: a single in-flight * connect to a black-holed/firewalled host blocks on the OS connect timeout - * (the transport exposes no application-level connect timeout to clamp it). + * (the transport exposes no application-level connect timeout to clamp it) + * and the stop path's interrupt cannot break that syscall. Only a deferred, + * PoolHousekeeper-driven pool can carry a token provider: QuestDBImpl is the + * sole caller of that constructor and passes {@code deferStartupRecovery=true}. + * Its potentially much longer credential pull is therefore interrupted by + * {@link PoolHousekeeper#stop()}, never by this pool's private-driver stop. * If {@code close()} lands during that one connect, its driver join can * still time out and the detached build releases the slot flock shortly * after {@code close()} returns. No data is lost (the slot stays durable on @@ -1675,13 +1710,33 @@ private void stopStartupRecoveryDriver() { if (beforeStartupRecoveryJoinHook != null) { beforeStartupRecoveryJoinHook.run(); } + // A private startup-recovery driver and a token provider are mutually exclusive by construction: + // QuestDBImpl passes deferStartupRecovery=true on the only path that supplies a provider. This + // escalation therefore does NOT break credential pulls (PoolHousekeeper.stop() owns that live + // protection). It is still a last resort for an unexpected interruptible overrun in a direct + // driver's build, drain, or teardown after the closed signal and normal unpark failed to stop it. + // + // Keep the whole two-join protocol interrupt-neutral. A carried caller flag, or an interrupt + // delivered DURING the first join, must be remembered without jumping past the target interrupt + // and second join; restore it only once the shutdown protocol has completed. + boolean callerWasInterrupted = Thread.interrupted(); try { - startupRecoveryThread.join(PoolHousekeeper.STOP_TIMEOUT_MILLIS); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - if (afterStartupRecoveryJoinHook != null) { - afterStartupRecoveryJoinHook.run(); + callerWasInterrupted |= PoolHousekeeper.joinIgnoringCallerInterrupts( + startupRecoveryThread, PoolHousekeeper.STOP_TIMEOUT_MILLIS); + if (startupRecoveryThread.isAlive()) { + // The closed flag reaches the driver only between operations. Interrupt an overrun and + // spend a second bounded join giving its finally/close path time to release the slot flock. + startupRecoveryThread.interrupt(); + callerWasInterrupted |= PoolHousekeeper.joinIgnoringCallerInterrupts( + startupRecoveryThread, PoolHousekeeper.STOP_TIMEOUT_MILLIS); + } + if (afterStartupRecoveryJoinHook != null) { + afterStartupRecoveryJoinHook.run(); + } + } finally { + if (callerWasInterrupted) { + Thread.currentThread().interrupt(); + } } } } @@ -1993,6 +2048,13 @@ public void onError(SenderError error) { return builder; } + private Sender.LineSenderBuilder applyTokenProvider(Sender.LineSenderBuilder builder) { + if (tokenProvider != null) { + builder.httpTokenProvider(tokenProvider); + } + return builder; + } + // Applies the user-supplied ingest callbacks to a sender builder. Null // callbacks are skipped so the sender keeps its loud-not-silent default. private Sender.LineSenderBuilder applyUserCallbacks(Sender.LineSenderBuilder builder) { @@ -2021,7 +2083,7 @@ private static boolean isRecoveryEventUserRelevant(SenderError e) { private Sender buildManagedSlotSender(int slotIndex, boolean forRecovery) { if (!storeAndForward) { - return applyUserCallbacks(Sender.builder(configurationString)).build(); + return applyUserCallbacks(applyTokenProvider(Sender.builder(configurationString))).build(); } // Give this pooled sender its own slot dir /- // so concurrent SF senders sharing one sf_dir never collide on @@ -2089,6 +2151,7 @@ private Sender buildManagedSlotSender(int slotIndex, boolean forRecovery) { // SenderErrorDispatcher so a slow handler cannot stall the recovery // driver or housekeeper thread. connectionListener and drainerListener // remain unset on recovery builds. + builder = applyTokenProvider(builder); return (forRecovery ? applyRecoveryCallbacks(builder) : applyUserCallbacks(builder)).build(); } diff --git a/core/src/main/java/io/questdb/client/std/Numbers.java b/core/src/main/java/io/questdb/client/std/Numbers.java index c7b8acc97..78a3bf4c7 100644 --- a/core/src/main/java/io/questdb/client/std/Numbers.java +++ b/core/src/main/java/io/questdb/client/std/Numbers.java @@ -343,6 +343,28 @@ public static long parseHexLong(CharSequence sequence) throws NumericException { return parseHexLong(sequence, 0, sequence.length()); } + /** + * Parses a hexadecimal sequence into a long, reading a full-width 16-digit word as its + * TWO'S-COMPLEMENT value: {@code ffffffffffffffff} is {@code -1}, not an error. This matches + * {@link #parseHexInt(CharSequence, int, int)} beside it and the server-side {@code io.questdb.std.Numbers} + * of the same name, whose {@code Long256} decoding depends on the wrap. + *

+ * Anything longer than 16 significant digits silently discards its high bits, and a caller + * parsing a COUNT it did not choose must bound the digits itself rather than lean on this method to + * do it. Each overflow residue breaks a length-prefixed format its own way, and + * {@code AbstractChunkedResponse} is the worked example: an HTTP chunk size of + * {@code 8000000000000000} wraps negative and hangs a framing state machine, one of + * {@code 10000000000000000} wraps to zero and reads as the terminal chunk (a truncated body reported + * as complete), and longer values wrap to short positive counts that mis-frame everything after them. + * It guards by counting significant digits BEFORE calling here, which is the only form that works - + * the zero residue is indistinguishable from a genuine {@code 0} once parsed. + * + * @param sequence the characters to parse + * @param lo inclusive start + * @param hi exclusive end + * @return the parsed value, wrapping on overflow + * @throws NumericException if the sequence is empty or holds a non-hex character + */ public static long parseHexLong(CharSequence sequence, int lo, int hi) throws NumericException { if (hi == 0) { throw NumericException.instance().put("empty hex string"); diff --git a/core/src/main/java/io/questdb/client/std/str/DirectUtf8Sink.java b/core/src/main/java/io/questdb/client/std/str/DirectUtf8Sink.java index 5b1921a67..8aae811dc 100644 --- a/core/src/main/java/io/questdb/client/std/str/DirectUtf8Sink.java +++ b/core/src/main/java/io/questdb/client/std/str/DirectUtf8Sink.java @@ -25,6 +25,7 @@ package io.questdb.client.std.str; import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; import io.questdb.client.std.bytes.DirectByteSink; import io.questdb.client.std.bytes.NativeByteSink; import org.jetbrains.annotations.NotNull; @@ -103,6 +104,28 @@ public DirectUtf8Sink put(byte b) { return this; } + /** + * Appends the bytes of {@code src} in {@code [lo, hi)} verbatim in a single bulk copy, rather than byte by + * byte. The ascii hint is set to {@code false} conservatively (the bytes are treated as opaque), so callers + * that rely on {@link #isAscii()} should not use this overload for ascii-only content. + */ + public DirectUtf8Sink put(byte[] src, int lo, int hi) { + // a real check, not an assert: this is public API doing an unchecked Unsafe.copyMemory, and client apps + // typically run without -ea, so a bad range must fail with a clear exception rather than a native + // out-of-bounds read that corrupts memory or crashes the JVM + if (lo < 0 || hi > src.length || lo > hi) { + throw new IndexOutOfBoundsException("put(byte[]) range out of bounds [lo=" + lo + ", hi=" + hi + ", len=" + src.length + ']'); + } + final int len = hi - lo; + if (len > 0) { + setAscii(false); + final long dest = sink.ensureCapacity(len); + Unsafe.getUnsafe().copyMemory(src, Unsafe.BYTE_OFFSET + lo, null, dest, len); + sink.advance(len); + } + return this; + } + @Override public DirectUtf8Sink putAny(byte b) { setAscii(isAscii() & b >= 0); diff --git a/core/src/main/java/io/questdb/client/std/str/DisplaySafe.java b/core/src/main/java/io/questdb/client/std/str/DisplaySafe.java new file mode 100644 index 000000000..c22c6b11d --- /dev/null +++ b/core/src/main/java/io/questdb/client/std/str/DisplaySafe.java @@ -0,0 +1,100 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.std.str; + +import static io.questdb.client.std.Numbers.hexDigits; + +/** + * Shared classifier for whether a code point is safe to show in a terminal or a log line. It is the one + * source of truth for the client's display-escaping: {@link Utf16Sink#putAsPrintable(CharSequence)} escapes + * everything it rejects, and the OIDC auth layer strips it from untrusted identity-provider text. Left raw, + * attacker-influenced text - an ILP server's error body, a column name, a verification URL - could reorder, + * hide, or forge what a human reads (a right-to-left override, a zero-width joiner, an ANSI escape). + */ +public final class DisplaySafe { + + private DisplaySafe() { + } + + /** + * Returns {@code true} when {@code cp} can be shown verbatim, {@code false} when it must be escaped or + * stripped. A code point is unsafe if it is a control char (C0/C1, DEL), a Unicode format char (bidi + * embeddings/overrides/isolates, LRM/RLM marks, zero-width joiners, the BOM, supplementary-plane tag + * chars), a Unicode line/paragraph separator (U+2028/U+2029, which break a rendered log line), or a + * surrogate (a lone half, with no displayable meaning). + */ + public static boolean isDisplaySafe(int cp) { + // Printable ASCII is the overwhelmingly common case and is never a control, format or surrogate char, + // so a single range check returns it without the Character.getType table lookup. + if (cp >= 0x20 && cp < 0x7f) { + return true; + } + if (Character.isISOControl(cp)) { + return false; + } + final int type = Character.getType(cp); + // FORMAT covers bidi/zero-width/joiners/BOM/tag chars; SURROGATE a lone half. LINE_SEPARATOR (U+2028) + // and PARAGRAPH_SEPARATOR (U+2029) are Unicode line breaks that split a rendered log line in + // ECMAScript/GUI/JSON log consumers, yet they are neither C0/C1 (isISOControl) nor FORMAT, so catch + // them here rather than let a tampered field forge an apparent extra log line. + if (type == Character.FORMAT || type == Character.SURROGATE + || type == Character.LINE_SEPARATOR || type == Character.PARAGRAPH_SEPARATOR) { + return false; + } + // The explicit bidi/BOM set is redundant with the FORMAT category on a conformant JDK, but kept as + // belt-and-suspenders on one that categorizes these differently. Hex literals (not char escapes) keep + // this source ASCII, so it carries none of the chars it guards. + return !(cp >= 0x202A && cp <= 0x202E) // LRE, RLE, PDF, LRO, RLO + && !(cp >= 0x2066 && cp <= 0x2069) // LRI, RLI, FSI, PDI + && cp != 0x200E && cp != 0x200F // LRM, RLM + && cp != 0xFEFF; // BOM / zero-width no-break space + } + + /** + * The inverse of {@link #isDisplaySafe(int)}: {@code true} when {@code cp} must not reach a terminal or + * log line raw. + */ + public static boolean isUnsafeForDisplay(int cp) { + return !isDisplaySafe(cp); + } + + // Escapes a code point to one (BMP) or two (supplementary, as its surrogate pair) visible \\uXXXX + // sequences, so the escaped value still names the original char. Emitting all four hex digits keeps a + // char above U+00FF (e.g. U+202E) correct rather than truncated to its low byte. A static helper here + // (not a private method on Utf16Sink) keeps the source Java 8 - private interface methods are Java 9. + static void putUnicodeEscape(Utf16Sink sink, int cp) { + if (cp > 0xFFFF) { + putUnicodeEscape(sink, Character.highSurrogate(cp)); + putUnicodeEscape(sink, Character.lowSurrogate(cp)); + return; + } + sink.put('\\'); + sink.put('u'); + sink.put(hexDigits[(cp >> 12) & 0xF]); + sink.put(hexDigits[(cp >> 8) & 0xF]); + sink.put(hexDigits[(cp >> 4) & 0xF]); + sink.put(hexDigits[cp & 0xF]); + } +} diff --git a/core/src/main/java/io/questdb/client/std/str/StringSink.java b/core/src/main/java/io/questdb/client/std/str/StringSink.java index 3c644aab5..aae8a83f9 100644 --- a/core/src/main/java/io/questdb/client/std/str/StringSink.java +++ b/core/src/main/java/io/questdb/client/std/str/StringSink.java @@ -28,6 +28,8 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.util.Arrays; + public class StringSink implements MutableUtf16Sink, CharSequence, Utf16Sink { private char[] buffer; @@ -125,6 +127,25 @@ public String toString() { return new String(buffer, 0, pos); } + /** + * Empties the sink AND overwrites its whole backing buffer, so nothing it has held remains readable + * through it. Best-effort hygiene for a sink that carried a secret - a bearer token, a refresh token, a + * device code - where {@link #clear()} is not enough: clear only rewinds the write position, leaving + * every character past that position in the array, so a long secret followed by a short write stays + * legible in the tail. It cannot reach a copy already handed out (a {@link #toString()} result, anything + * downstream wrote elsewhere), only this sink's own storage. + *

+ * Storage the sink has OUTGROWN is covered, but not by this method: {@link #checkCapacity(int)} zeroes + * each array as it hands off to a larger one, because by the time wipe() runs those generations are + * unreachable from here. Without that a sink small enough to grow while holding a secret - which is + * every default-sized one that carries a token - would leave a full copy per growth on the heap, and + * wiping the survivor would say nothing about them. + */ + public void wipe() { + Arrays.fill(buffer, (char) 0); + pos = 0; + } + private void checkCapacity(int extra) { int len = pos + extra; if (buffer.length >= len) { @@ -133,6 +154,15 @@ private void checkCapacity(int extra) { len = Math.max(pos * 2, len); final char[] n = new char[len]; System.arraycopy(buffer, 0, n, 0, pos); + // Zero the array being abandoned. wipe() can only reach the CURRENT buffer, so without this every + // generation growth leaves behind keeps its contents legible on the heap until the collector happens + // to overwrite that memory - which it is under no obligation to do, and a heap dump taken meanwhile + // shows the lot. That is not hypothetical for the sinks wipe() exists for: OidcDeviceAuth's formSink + // starts at 16 chars and builds "...&refresh_token=&client_id=...&scope=...", so it grows + // several times while already holding the whole refresh token, and each array it hands off carries a + // copy of it. The WHOLE array is zeroed, not just the live prefix: a sink cleared after holding a + // long secret keeps that secret past pos, which is the same retention wipe() itself closes. + Arrays.fill(buffer, (char) 0); buffer = n; } } diff --git a/core/src/main/java/io/questdb/client/std/str/Utf16Sink.java b/core/src/main/java/io/questdb/client/std/str/Utf16Sink.java index f53e1ae5f..b18c29853 100644 --- a/core/src/main/java/io/questdb/client/std/str/Utf16Sink.java +++ b/core/src/main/java/io/questdb/client/std/str/Utf16Sink.java @@ -26,8 +26,6 @@ import org.jetbrains.annotations.Nullable; -import static io.questdb.client.std.Numbers.hexDigits; - /** * Family of sinks that write out character value as UTF16 encoded bytes. This interface * is separate from {@link CharSink} to achieve two goals: @@ -45,24 +43,61 @@ default Utf16Sink put(@Nullable Utf8Sequence us) { } default void putAsPrintable(CharSequence nonPrintable) { - for (int i = 0, n = nonPrintable.length(); i < n; i++) { - char c = nonPrintable.charAt(i); - putAsPrintable(c); + // Scan by code point, not UTF-16 unit. A supplementary-plane format char (e.g. a U+E00xx language + // tag char) arrives as a surrogate pair whose halves report SURROGATE rather than FORMAT, and a + // lone surrogate likewise - per-unit scanning would pass both through raw. Judging the whole code + // point (via DisplaySafe, the shared classifier) escapes them, while a normal supplementary char + // such as an emoji is neither control nor format and is emitted verbatim. + // + // Classify before copying, and when nothing needs escaping hand the whole sequence to + // put(CharSequence) instead of walking it a character at a time. The escaping loop below appends + // with put(char), so an implementation like StringSink pays a capacity check per CHARACTER, where + // its put(CharSequence) override pays one for the whole sequence and then copies in a tight loop. + // That matters because the biggest input here is a server-supplied error body on a failed ILP + // flush, which the client does not cap - and which is almost always entirely printable, so the + // scan finds nothing and the copy is the bulk one. Mixed input costs this extra scan and then + // takes the loop as before; that is the rare case, and it is the one where correctness, not + // speed, is the point. Mirrors OidcDeviceAuth.sanitizeForDisplay, which returns its input + // untouched on the same test. + final int n = nonPrintable.length(); + int firstUnsafe = -1; + for (int i = 0; i < n; ) { + final int cp = Character.codePointAt(nonPrintable, i); + if (!DisplaySafe.isDisplaySafe(cp)) { + firstUnsafe = i; + break; + } + i += Character.charCount(cp); + } + if (firstUnsafe < 0) { + put(nonPrintable); + return; + } + for (int i = 0; i < n; ) { + final int cp = Character.codePointAt(nonPrintable, i); + final int count = Character.charCount(cp); + if (DisplaySafe.isDisplaySafe(cp)) { + if (count == 1) { + put((char) cp); // BMP: cp already is the char, so skip the redundant charAt re-read + } else { + put(nonPrintable.charAt(i)); + put(nonPrintable.charAt(i + 1)); + } + } else { + DisplaySafe.putUnicodeEscape(this, cp); + } + i += count; } } default void putAsPrintable(char c) { - if (c > 0x1F && c != 0x7F) { + // A single UTF-16 unit: escape control chars, Unicode format chars, and a lone surrogate (which has + // no displayable meaning). Supplementary-plane format chars are caught by the code-point-aware + // putAsPrintable(CharSequence). + if (DisplaySafe.isDisplaySafe(c)) { put(c); } else { - put('\\'); - put('u'); - - final int s = (int) c & 0xFF; - put('0'); - put('0'); - put(hexDigits[s / 0x10]); - put(hexDigits[s % 0x10]); + DisplaySafe.putUnicodeEscape(this, c); } } @@ -93,5 +128,4 @@ default Utf16Sink putNonAscii(long lo, long hi) { Utf8s.utf8ToUtf16(lo, hi, this); return this; } - -} \ No newline at end of file +} diff --git a/core/src/main/java/module-info.java b/core/src/main/java/module-info.java index ada19961c..8383221e4 100644 --- a/core/src/main/java/module-info.java +++ b/core/src/main/java/module-info.java @@ -27,7 +27,18 @@ requires static org.jetbrains.annotations; requires static java.management; requires jdk.management; - requires java.desktop; + // STATIC, not mandatory: the only java.desktop reference is BrowserLauncher's java.awt.Desktop, used + // best-effort by the default DeviceCodePrompt.openBrowser() to pop the verification URL. A mandatory + // requires is resolved BEFORE any code runs, so on a runtime without java.desktop - a jlink image, a + // --limit-modules run - the whole module failed to resolve at startup and the LinkageError catch in + // openBrowser() never got the chance to degrade to "print the URL and carry on". As a static requires + // the dependency is compile-time only: java.desktop's absence surfaces as the NoClassDefFoundError that + // catch already handles. Note the consequence for a MODULAR application - a static requires is not + // followed during runtime resolution, so such an application only gets the browser launch when + // java.desktop is in its graph anyway (it requires it, or --add-modules java.desktop); everything else, + // including every class-path application, is unaffected because java.desktop is resolved there by + // default. See DeviceCodePrompt#openBrowser(). + requires static java.desktop; requires java.sql; requires org.slf4j; diff --git a/core/src/test/java/io/questdb/client/test/HttpTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/HttpTokenProviderTest.java new file mode 100644 index 000000000..bda404e1c --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/HttpTokenProviderTest.java @@ -0,0 +1,82 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test; + +import io.questdb.client.HttpTokenProvider; +import io.questdb.client.cutlass.line.LineSenderException; +import org.junit.Assert; +import org.junit.Test; + +public class HttpTokenProviderTest { + + @Test + public void testValidateTokenAcceptsPrintableAscii() { + // a real bearer token is printable ASCII (base64url JWT segments joined by dots); validateToken + // must pass it through unchanged + HttpTokenProvider.validateToken("eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJfc3NvIn0.abc-DEF_123"); + HttpTokenProvider.validateToken("a~b"); // 0x7e (~) is the top of the allowed range + HttpTokenProvider.validateToken("a b"); // an interior space (0x20) is allowed; only an all-blank token is rejected + } + + @Test + public void testValidateTokenNeverEchoesTheToken() { + // the token is the secret this guards; it must never appear in the exception message + try { + HttpTokenProvider.validateToken("SUPERSECRET" + (char) 0x0d + (char) 0x0a + "TOKEN"); + Assert.fail("expected the token to be rejected"); + } catch (LineSenderException e) { + Assert.assertFalse(e.getMessage(), e.getMessage().contains("SUPERSECRET")); + } + } + + @Test + public void testValidateTokenRejectsBlank() { + assertRejected(null, "null or empty token"); + assertRejected("", "null or empty token"); + assertRejected(" ", "null or empty token"); + } + + @Test + public void testValidateTokenRejectsControlOrNonAscii() { + // a control char would break out of the "Authorization: Bearer " header (CR/LF injects into + // the request line); a non-ASCII char is silently truncated to one byte by the ASCII header writer. + // The strings are built with explicit char values to keep this source pure ASCII. + assertRejected("abc" + (char) 0x0d + (char) 0x0a + "def", "control or non-ASCII character"); // CR/LF + assertRejected("tok" + (char) 0x00 + "en", "control or non-ASCII character"); // NUL + assertRejected((char) 0x1b + "[31mred", "control or non-ASCII character"); // ANSI escape (ESC) + assertRejected("a" + (char) 0x1f + "b", "control or non-ASCII character"); // 0x1f, just below the 0x20 lower bound + assertRejected("a" + (char) 0x7f + "b", "control or non-ASCII character"); // DEL (0x7f), just above the 0x7e upper bound + assertRejected("tok" + (char) 0xe9 + "n", "control or non-ASCII character"); // non-ASCII (e-acute, 0xe9) + } + + private static void assertRejected(CharSequence token, String expectedMessage) { + try { + HttpTokenProvider.validateToken(token); + Assert.fail("expected token to be rejected: " + token); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java b/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java index 6cf5eb09a..e338aa72c 100644 --- a/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java +++ b/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java @@ -24,13 +24,20 @@ package io.questdb.client.test; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.QuestDB; import io.questdb.client.QuestDBBuilder; +import io.questdb.client.Query; +import io.questdb.client.Sender; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; import org.junit.Test; +import java.util.Arrays; +import java.util.HashSet; +import java.util.Set; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.BooleanSupplier; public class QuestDBBuilderTest { @@ -79,6 +86,118 @@ public void testConnectSingleStringValidatesAndBuilds() { } } + @Test + public void testConnectTokenProviderSuppliesBothPoolsAndPoolGrowth() throws Exception { + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + })) { + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + AtomicInteger tokenCalls = new AtomicInteger(); + HttpTokenProvider provider = () -> "ROTATING-" + tokenCalls.incrementAndGet(); + String cfg = "ws::addr=localhost:" + server.getPort() + ";" + + "sender_pool_min=1;sender_pool_max=2;" + + "query_pool_min=1;query_pool_max=2;" + + "auth_timeout_ms=2000;"; + + try (QuestDB db = QuestDB.connect(cfg, provider)) { + // Each prewarmed connection must obtain its own current token. + // Pool startup order is deliberately not part of the contract. + assertAuthorizationHeaders( + server, + "Bearer ROTATING-1", + "Bearer ROTATING-2"); + + // Exhaust each prewarmed slot so the elastic pools grow. The + // newly created sender and query client must pull again rather + // than reuse either token captured during prewarm. + try (Sender sender1 = db.borrowSender(); Sender sender2 = db.borrowSender()) { + Assert.assertNotNull(sender1); + Assert.assertNotNull(sender2); + assertAuthorizationHeaders(server, "Bearer ROTATING-3"); + } + try (Query query1 = db.borrowQuery(); Query query2 = db.borrowQuery()) { + Assert.assertNotNull(query1); + Assert.assertNotNull(query2); + assertAuthorizationHeaders(server, "Bearer ROTATING-4"); + } + } + Assert.assertEquals(4, tokenCalls.get()); + } + } + + @Test(timeout = 30_000) + public void testEagerBuildAndBorrowQuerySurfaceAProviderFailure() throws Exception { + // The other half of the lazy_connect contract, and the half that must FAIL loudly: without + // lazy_connect the pools initialize eagerly, so a credential the provider cannot supply is a + // startup error the caller has to see rather than a sender that silently never authenticates. + // Driven against a LIVE server so the failure is unambiguously the credential and not connectivity. + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + })) { + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + HttpTokenProvider failing = () -> { + throw new IllegalStateException("not signed in yet"); + }; + String eagerCfg = "ws::addr=localhost:" + server.getPort() + ";" + + "sender_pool_min=1;sender_pool_max=1;" + + "query_pool_min=1;query_pool_max=1;auth_timeout_ms=2000;"; + try { + QuestDB.connect(eagerCfg, failing).close(); + Assert.fail("an eager build must surface a provider that cannot supply a credential"); + } catch (RuntimeException e) { + assertCarriesProviderCause(e); + } + + // And the deferred read path: query_pool_min=0 prewarms nothing, so the first borrowQuery() is + // where the pull happens. It must report the provider's failure rather than hand back a client + // that never authenticated. + String lazyQueryCfg = "ws::addr=localhost:" + server.getPort() + ";" + + "sender_pool_min=0;sender_pool_max=1;" + + "query_pool_min=0;query_pool_max=1;auth_timeout_ms=2000;"; + try (QuestDB db = QuestDB.connect(lazyQueryCfg, failing)) { + try (Query ignored = db.borrowQuery()) { + Assert.fail("borrowQuery() must surface a provider that cannot supply a credential"); + } catch (RuntimeException e) { + assertCarriesProviderCause(e); + } + } + } + } + + @Test + public void testTokenProviderRejectsFixedConfigCredentialsBeforePoolCreation() { + HttpTokenProvider provider = () -> "TOKEN"; + assertTokenProviderAuthRejected( + "ws::addr=127.0.0.1:1;token=fixed;sender_pool_min=0;query_pool_min=0;", + provider); + assertTokenProviderAuthRejected( + "ws::addr=127.0.0.1:1;username=user;password=pass;sender_pool_min=0;query_pool_min=0;", + provider); + } + + @Test + public void testTokenProviderRejectsNull() { + try { + QuestDB.builder().httpTokenProvider(null); + Assert.fail("expected a null provider to be rejected"); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("must not be null")); + } + + try { + QuestDB.connect( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;", + null).close(); + Assert.fail("expected a null provider to be rejected"); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("must not be null")); + } + } + @Test public void testMalformedEgressConfigRejectedAtBuildWithMinZero() { // query_pool_min=0 pre-warms nothing, so build() never constructs a @@ -280,6 +399,43 @@ private static void assertBuildRejected(String config, String expectedFragment) } } + private static void assertCarriesProviderCause(Throwable thrown) { + // The provider's own message must survive to the caller: "not signed in yet" is actionable, + // a transport-shaped wrapper naming the endpoint is not. + for (Throwable t = thrown; t != null; t = t.getCause()) { + if (t.getMessage() != null && t.getMessage().contains("not signed in yet")) { + return; + } + } + Assert.fail("the provider's own failure must reach the caller, got: " + thrown); + } + + private static void assertAuthorizationHeaders( + TestWebSocketServer server, + String... expected + ) throws InterruptedException { + Set actual = new HashSet<>(); + for (int i = 0; i < expected.length; i++) { + String header = server.pollAuthorizationHeader(5, TimeUnit.SECONDS); + Assert.assertNotNull("timed out waiting for an Authorization header", header); + Assert.assertTrue("duplicate Authorization header: " + header, actual.add(header)); + } + Assert.assertEquals(new HashSet<>(Arrays.asList(expected)), actual); + } + + private static void assertTokenProviderAuthRejected(String config, HttpTokenProvider provider) { + try { + QuestDB.builder() + .fromConfig(config) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("expected fixed credentials and the token provider to be mutually exclusive"); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("cannot be combined")); + } + } + private static void assertSchemaRejected(Runnable action) { try { action.run(); diff --git a/core/src/test/java/io/questdb/client/test/QuestDBLazyConnectTest.java b/core/src/test/java/io/questdb/client/test/QuestDBLazyConnectTest.java index ef70d08a8..d140812e2 100644 --- a/core/src/test/java/io/questdb/client/test/QuestDBLazyConnectTest.java +++ b/core/src/test/java/io/questdb/client/test/QuestDBLazyConnectTest.java @@ -24,6 +24,7 @@ package io.questdb.client.test; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.QuestDB; import io.questdb.client.QuestDBBuilder; import io.questdb.client.Sender; @@ -31,6 +32,8 @@ import org.junit.Assert; import org.junit.Test; +import java.util.concurrent.atomic.AtomicInteger; + /** * {@code lazy_connect=true} makes a {@link QuestDB} facade tolerate the server * being down at startup without disabling reads: the ingest side @@ -75,6 +78,42 @@ public void testLazyConnectStartsAndWritesWhileServerDown() { } } + @Test(timeout = 30_000) + public void testLazyConnectBuildsAndWritesDespiteAFailingTokenProvider() { + int port = TestPorts.findUnusedPort(); + // lazy_connect and httpTokenProvider are both documented, and their COMBINATION decides who sees a + // credential failure at startup - but nothing drove them together. The contract: connectivity and + // credential errors are the caller's problem only DURING initialization, and under lazy_connect + // there is no eager initialization to fail. The ingest side resolves to ASYNC (client is null, no + // pull at build) and the read pool defaults to min=0, so build() must return and a write must + // buffer even though this provider can supply nothing at all. + // + // Getting this wrong is a data-loss shape, not an inconvenience: a producer that hard-fails at + // build() instead of buffering drops the rows store-and-forward promised to keep. + AtomicInteger pulls = new AtomicInteger(); + HttpTokenProvider failing = () -> { + pulls.incrementAndGet(); + throw new IllegalStateException("not signed in yet"); + }; + try (QuestDB db = QuestDB.connect("ws::addr=localhost:" + port + + ";lazy_connect=true;reconnect_max_duration_millis=200" + + ";reconnect_initial_backoff_millis=10;reconnect_max_backoff_millis=50" + + ";close_flush_timeout_millis=0;", failing)) { + Sender sender = db.borrowSender(); + Assert.assertNotNull("build() must not fail-fast on a provider that cannot supply a token yet", + sender); + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.close(); + } catch (RuntimeException ignored) { + // acceptable: the close-flush runs against a server that never came up + } + } + // Deliberately not asserting a pull count. The async connect thread may or may not have attempted + // one by now, and that timing is not the contract - what is, is that neither build() nor the write + // above surfaced the provider's failure to the caller. + } + @Test(timeout = 30_000) public void testLazyConnectKeepsReadsEnabledWhileServerDown() { int port = TestPorts.findUnusedPort(); diff --git a/core/src/test/java/io/questdb/client/test/SenderBuilderErrorApiTest.java b/core/src/test/java/io/questdb/client/test/SenderBuilderErrorApiTest.java index cccbcdab9..704705e14 100644 --- a/core/src/test/java/io/questdb/client/test/SenderBuilderErrorApiTest.java +++ b/core/src/test/java/io/questdb/client/test/SenderBuilderErrorApiTest.java @@ -230,11 +230,89 @@ public void testConnectStringRejectsConnectionListenerInboxCapacityOnNonWebSocke @Test public void testCategoryAndPolicyAreStillEnumerable() { - // Cross-check that the enum surface is fully reachable from - // user-side code via the builder import path. - SenderError.Category c = SenderError.Category.SCHEMA_MISMATCH; - SenderError.Policy p = SenderError.Policy.RETRIABLE; - Assert.assertNotNull(c); - Assert.assertNotNull(p); + // Cross-check that the user-facing SenderError enum surface is intact, driven by NAME strings the + // compiler does not resolve, so a rename or removal fails this test at RUNTIME (valueOf throws + // IllegalArgumentException). Using a compiled constant reference (SenderError.Category.SCHEMA_MISMATCH) as + // the expected value instead would only fail to COMPILE on a rename - the source, not the assertion, + // would break - so it would test nothing at runtime. + Assert.assertEquals("SCHEMA_MISMATCH", SenderError.Category.valueOf("SCHEMA_MISMATCH").name()); + Assert.assertEquals("RETRIABLE", SenderError.Policy.valueOf("RETRIABLE").name()); + } + + @Test + public void testHttpTokenProviderIsMutuallyExclusiveWithOtherAuth() { + // a provider cannot be combined with a static token or username/password, in either order + try { + Sender.builder(Sender.Transport.HTTP).address("localhost:9000") + .httpToken("static").httpTokenProvider(() -> "dynamic"); + Assert.fail("expected token-already-configured"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token was already configured")); + } + try { + Sender.builder(Sender.Transport.HTTP).address("localhost:9000") + .httpTokenProvider(() -> "dynamic").httpToken("static"); + Assert.fail("expected token-provider-already-configured"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token provider was already configured")); + } + try { + Sender.builder(Sender.Transport.HTTP).address("localhost:9000") + .httpUsernamePassword("u", "p").httpTokenProvider(() -> "dynamic"); + Assert.fail("expected username-already-configured"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("username was already configured")); + } + } + + @Test + public void testHttpTokenProviderNullRejectedAndExclusiveWithLaterUsernamePassword() { + // a null provider is rejected up front + try { + Sender.builder(Sender.Transport.HTTP).address("localhost:9000").httpTokenProvider(null); + Assert.fail("expected a null provider to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token provider cannot be null")); + } + // the reverse of the mutual-exclusion case above: provider first, then username/password. This hits a + // distinct guard in httpUsernamePassword(), which the provider-then-token / token-then-provider / + // username-then-provider orderings above do not reach + try { + Sender.builder(Sender.Transport.HTTP).address("localhost:9000") + .httpTokenProvider(() -> "dynamic").httpUsernamePassword("u", "p"); + Assert.fail("expected token-provider-already-configured"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token provider authentication is already configured")); + } + } + + @Test + public void testHttpTokenProviderAcceptedForWebSocket() { + // the provider is supported over WebSocket (queried at each upgrade handshake): it must pass + // build-time validation and fail only on the connection itself, never with a "not supported" + // rejection. 127.0.0.1:1 is refused promptly, and InitialConnectMode defaults to OFF (fail fast). + try (Sender ignored = Sender.builder(Sender.Transport.WEBSOCKET).address("127.0.0.1:1") + .httpTokenProvider(() -> "dynamic").build()) { + Assert.fail("expected a connection failure against a dead address"); + } catch (LineSenderException e) { + Assert.assertFalse(e.getMessage(), e.getMessage().contains("not supported for WebSocket")); + } + } + + @Test + public void testHttpTokenProviderRejectedForTcpAndUdp() { + // TCP uses challenge-response key auth and UDP has no auth; neither carries a bearer token, + // so both must reject the provider at build time + assertProviderRejected(Sender.Transport.TCP, "token provider authentication is not supported for TCP protocol"); + assertProviderRejected(Sender.Transport.UDP, "token provider authentication is not supported for UDP transport"); + } + + private static void assertProviderRejected(Sender.Transport transport, String expectedMessage) { + try (Sender ignored = Sender.builder(transport).address("localhost:9009") + .httpTokenProvider(() -> "dynamic").build()) { + Assert.fail("expected the token provider to be rejected for " + transport); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + } } } diff --git a/core/src/test/java/io/questdb/client/test/compat/ExportedApiCompatibilityTest.java b/core/src/test/java/io/questdb/client/test/compat/ExportedApiCompatibilityTest.java new file mode 100644 index 000000000..8ace8e77c --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/compat/ExportedApiCompatibilityTest.java @@ -0,0 +1,194 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.compat; + +import io.questdb.client.cutlass.http.client.Fragment; +import io.questdb.client.cutlass.http.client.Response; +import io.questdb.client.cutlass.line.http.AbstractLineHttpSender; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; +import org.junit.Assert; +import org.junit.Test; + +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.util.LinkedHashSet; +import java.util.Set; +import java.util.TreeSet; + +/** + * Pins the public signatures this branch had to restore after replacing them in place. + *

+ * Three exported methods were changed rather than added to - {@code Response.recv(int)} arrived as an + * abstract interface method, two {@code QwpWebSocketSender.connect(..., String, ...)} overloads were retyped + * to {@code Supplier}, and the multi-host {@code AbstractLineHttpSender.createLineSender} gained a + * parameter in place. All three sit in packages {@code module-info.java} exports and that ship a javadoc jar, + * so a caller compiled against an earlier release would have failed with {@code NoSuchMethodError}, and an + * external {@code Response} implementation with {@code AbstractMethodError}. Nothing in this repository, in + * questdb, or in questdb-enterprise calls them, which is why the break was latent rather than observed - and + * why nothing would have caught it coming back. + *

+ * There is no japicmp or revapi gate on this build, so this test is the gate. The expected signatures below + * are the ones present at this branch's merge base ({@code 2489b243}); they are written out literally rather + * than derived from the current classes, because a pin computed from the thing it pins proves nothing. + * Adding an overload is fine and this test stays green; retyping or removing one turns it red. + */ +public class ExportedApiCompatibilityTest { + + /** + * Every {@code QwpWebSocketSender.connect} and {@code AbstractLineHttpSender.createLineSender} signature + * that existed at the merge base, as {@code name(paramType,...)returnType} over erased type names. + */ + private static final String[] PRE_BRANCH_SIGNATURES = { + // ---- QwpWebSocketSender.connect ---- + "connect(java.lang.String,int)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode,io.questdb.client.SenderErrorHandler,int)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.lang.String,int,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode,io.questdb.client.SenderErrorHandler,int,long)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.util.List,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode,io.questdb.client.SenderErrorHandler,int,long,long)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.util.List,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode,io.questdb.client.SenderErrorHandler,int,long,long,int,io.questdb.client.SenderConnectionListener,int)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + "connect(java.util.List,io.questdb.client.ClientTlsConfiguration,int,int,long,java.lang.String,boolean,io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine,long,long,long,long,io.questdb.client.Sender$InitialConnectMode,io.questdb.client.SenderErrorHandler,int,long,long,int,io.questdb.client.SenderConnectionListener,int,int,long,long)io.questdb.client.cutlass.qwp.client.QwpWebSocketSender", + // ---- AbstractLineHttpSender.createLineSender ---- + "createLineSender(java.lang.String,int,java.lang.String,io.questdb.client.HttpClientConfiguration,io.questdb.client.ClientTlsConfiguration,int,java.lang.String,java.lang.String,java.lang.String,int,long,int,long,long,int)io.questdb.client.cutlass.line.http.AbstractLineHttpSender", + "createLineSender(io.questdb.client.std.ObjList,io.questdb.client.std.IntList,java.lang.String,io.questdb.client.HttpClientConfiguration,io.questdb.client.ClientTlsConfiguration,int,java.lang.String,java.lang.String,java.lang.String,int,long,int,long,long,int)io.questdb.client.cutlass.line.http.AbstractLineHttpSender", + }; + + @Test + public void testPreBranchCreateLineSenderOverloadsStillLink() { + assertSignaturesPresent(AbstractLineHttpSender.class, "createLineSender"); + } + + @Test + public void testPreBranchQwpWebSocketSenderConnectOverloadsStillLink() { + assertSignaturesPresent(QwpWebSocketSender.class, "connect"); + } + + @Test + public void testResponseRecvIntIsDefaultNotAbstract() throws Exception { + // The defect was recv(int) arriving as an abstract interface method: an external implementation + // written against the earlier Response compiles fine and then fails at run time with + // AbstractMethodError, which is exactly the failure a unit test of this library never sees. + Method recvWithTimeout = Response.class.getMethod("recv", int.class); + Assert.assertFalse("Response.recv(int) must stay a default method - an implementation written " + + "before the overload existed has no override for it", + Modifier.isAbstract(recvWithTimeout.getModifiers())); + Assert.assertEquals(Fragment.class, recvWithTimeout.getReturnType()); + + Method recv = Response.class.getMethod("recv"); + Assert.assertTrue("recv() is the one method an implementation must supply", + Modifier.isAbstract(recv.getModifiers())); + Assert.assertEquals(Fragment.class, recv.getReturnType()); + } + + @Test + public void testResponseImplementorOverridingOnlyRecvStillWorks() { + // LegacyResponse below is the compile-time half: it implements Response and overrides recv() ONLY, + // exactly as an implementation predating the overload does. If recv(int) went back to being + // abstract, this class stops compiling and the whole test module goes with it - which is the point. + LegacyResponse legacy = new LegacyResponse(); + Fragment first = legacy.recv(); + Assert.assertNotNull(first); + Assert.assertEquals(1, legacy.calls); + + // and the default must keep the PREVIOUS behaviour, not merely link: it ignores the bound and + // defers to recv(), which is what such an implementation did before the overload existed + Fragment bounded = legacy.recv(5_000); + Assert.assertSame("the default must delegate to recv()", legacy.fragment, bounded); + Assert.assertEquals("and must not read anything of its own", 2, legacy.calls); + + Fragment unbounded = legacy.recv(0); + Assert.assertSame("a non-positive timeout is the legacy unbounded path, same delegation", + legacy.fragment, unbounded); + Assert.assertEquals(3, legacy.calls); + } + + private static void assertSignaturesPresent(Class type, String methodName) { + final Set actual = new TreeSet<>(); + for (Method m : type.getMethods()) { + if (m.getName().equals(methodName)) { + actual.add(signatureOf(m)); + } + } + final Set missing = new LinkedHashSet<>(); + int expected = 0; + for (String signature : PRE_BRANCH_SIGNATURES) { + if (!signature.startsWith(methodName + "(")) { + continue; + } + expected++; + if (!actual.contains(signature)) { + missing.add(signature); + } + } + Assert.assertTrue("expected at least one pinned signature for " + methodName, expected > 0); + Assert.assertTrue( + "these " + type.getSimpleName() + '.' + methodName + " signatures existed at the merge base " + + "and no longer do, so a caller compiled against an earlier release breaks with " + + "NoSuchMethodError. Add an overload instead of retyping one.\n missing:\n " + + String.join("\n ", missing) + "\n present:\n " + + String.join("\n ", actual), + missing.isEmpty()); + } + + private static String signatureOf(Method m) { + final StringBuilder sb = new StringBuilder(m.getName()).append('('); + final Class[] params = m.getParameterTypes(); + for (int i = 0; i < params.length; i++) { + if (i > 0) { + sb.append(','); + } + sb.append(params[i].getName()); + } + return sb.append(')').append(m.getReturnType().getName()).toString(); + } + + /** + * A {@link Response} written before {@code recv(int)} existed: it overrides {@code recv()} and nothing + * else. Its value is mostly at compile time - it does not compile against an abstract {@code recv(int)}. + */ + private static final class LegacyResponse implements Response { + private final Fragment fragment = new Fragment() { + @Override + public long hi() { + return 128L; + } + + @Override + public long lo() { + return 64L; + } + }; + private int calls; + + @Override + public Fragment recv() { + calls++; + return fragment; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/AwtErrorPromptMain.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/AwtErrorPromptMain.java new file mode 100644 index 000000000..b97018f21 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/AwtErrorPromptMain.java @@ -0,0 +1,118 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.DeviceAuthorizationChallenge; +import io.questdb.client.cutlass.auth.DeviceCodePrompt; + +import java.lang.reflect.InvocationTargetException; + +/** + * The child half of {@link BrowserLauncherAwtErrorTest}: run by that test in a second JVM whose AWT + * toolkit cannot initialise, so the browser launch throws a {@code java.awt.AWTError}. + *

+ * Two modes, run as two separate processes because the failure is ONE-SHOT: {@code Toolkit}'s assistive + * technology loading throws on the first {@code getDefaultToolkit()} and then completes, so a second call + * in the same JVM succeeds. A probe and the real call therefore cannot share a process - the probe would + * consume the only throw and leave the real call asserting nothing. + *

+ * The probe reaches {@code Desktop} reflectively for the same reason {@code DesktopFreePromptMain} does: + * this test tree compiles as module {@code io.questdb.test}, which does not read {@code java.desktop}, so a + * direct reference would not compile. At run time the child is on the class path, where it resolves. + *

+ * Deliberately Java 8 source level, like the rest of this test tree. + */ +public final class AwtErrorPromptMain { + + /** + * Printed by {@link #MODE_PROMPT} once the default prompt returned despite the broken toolkit. + */ + static final String DEGRADED_MARKER = "AWT-ERROR-DEGRADED-OK"; + /** + * Exit code for "the precondition did not hold": the toolkit initialised fine, so this JVM is not the + * hostile one the test needs and nothing it observes would prove anything. + */ + static final int EXIT_NO_AWT_ERROR = 2; + static final String MODE_PROBE = "probe"; + static final String MODE_PROMPT = "prompt"; + /** + * Printed by {@link #MODE_PROBE} once it has seen the {@code AWTError} the other mode relies on. + */ + static final String PROBE_MARKER = "AWT-ERROR-OBSERVED"; + + private AwtErrorPromptMain() { + } + + public static void main(String[] args) throws Exception { + if (args.length != 1) { + System.out.println("usage: " + AwtErrorPromptMain.class.getName() + + " <" + MODE_PROBE + '|' + MODE_PROMPT + '>'); + System.exit(EXIT_NO_AWT_ERROR); + return; + } + if (MODE_PROBE.equals(args[0])) { + probe(); + } else { + prompt(); + } + } + + /** + * Proves this JVM's desktop stack is genuinely broken, so the sibling process's quiet return is a + * DEGRADE rather than a no-op. Reports the throwable's type as well, since the whole point is that it + * is an {@code AWTError} - neither an {@code Exception} nor a {@code LinkageError}, so neither of the + * two guards on the launch path catches it by category. + */ + private static void probe() throws Exception { + Throwable raised = null; + try { + Class.forName("java.awt.Desktop").getMethod("isDesktopSupported").invoke(null); + } catch (InvocationTargetException e) { + // reflection wraps whatever the toolkit raised; the cause is the throwable the launch path + // would have met directly + raised = e.getCause(); + } + if (raised == null) { + System.out.println("the AWT toolkit initialised, so this JVM is not the hostile one this test " + + "needs: check that java.awt.headless is off and the assistive-technology property is set"); + System.exit(EXIT_NO_AWT_ERROR); + return; + } + System.out.println(PROBE_MARKER + ' ' + raised.getClass().getName() + + " isLinkageError=" + (raised instanceof LinkageError) + + " isException=" + (raised instanceof Exception)); + } + + /** + * The real path, run FIRST in this process so it meets the one-shot {@code AWTError} rather than the + * clean toolkit a probe would have left behind. The promise is that the browser launch is best-effort + * and never fatal, so the prompt must render the challenge and return. + */ + private static void prompt() { + DeviceCodePrompt.openBrowser().promptUser(new DeviceAuthorizationChallenge( + "WDJB-MJHT", "https://verify.example/device", null, 300, 5)); + System.out.println(DEGRADED_MARKER); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherAwtErrorTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherAwtErrorTest.java new file mode 100644 index 000000000..aaec5daed --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherAwtErrorTest.java @@ -0,0 +1,144 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.DeviceCodePrompt; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.Logger; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.InputStream; +import java.security.CodeSource; + +/** + * Pins that a desktop stack which cannot initialise degrades to "no browser" instead of failing sign-in. + *

+ * {@code BrowserLauncher.open()} caught {@code Exception} and {@code DeviceCodePrompt.openBrowser()} + * catches {@code LinkageError}. {@link java.awt.AWTError} is neither: it extends {@code Error} directly, so + * it passed through both and escaped {@code signIn()} - aborting a sign-in the human could have completed + * from the URL already printed, as a type the caller's documented {@code catch (OidcAuthException)} does not + * handle. {@code Toolkit} raises it whenever {@code assistive_technologies} names a class the runtime cannot + * load, which is the stock configuration on several Linux distributions, and again when a set {@code DISPLAY} + * points at no X server. + *

+ * Two child processes rather than one, because the failure is one-shot - see {@link AwtErrorPromptMain}. The + * probe runs first and its assertion is what keeps the other half honest: without it a prompt that returned + * because the toolkit was FINE would read exactly like one that returned because the guard caught the error. + * It is also what keeps this test from opening a real browser on a developer's machine - the launch is only + * ever reached in a JVM the probe has already shown cannot get past {@code Desktop.isDesktopSupported()}. + *

+ * Java 8 source level, like the rest of this test tree. + */ +public class BrowserLauncherAwtErrorTest { + + /** + * A class no runtime can load. The real-world value here is {@code org.GNOME.Accessibility.AtkWrapper}, + * shipped in {@code accessibility.properties} by several Linux distributions without the package that + * provides it; any absent name reaches the same {@code AWTError}. + */ + private static final String MISSING_AT_CLASS = "io.questdb.client.test.NoSuchAssistiveTechnology"; + + @Test(timeout = 60_000) + public void testAnUninitialisableToolkitDegradesInsteadOfFailingSignIn() throws Exception { + // The precondition, asserted BEFORE the launch half is ever forked: this JVM configuration must + // genuinely raise an AWTError, or the assertion below proves nothing and the launch could reach a + // real browser. + String probe = runChild(AwtErrorPromptMain.MODE_PROBE); + Assert.assertTrue("the assistive-technology property did not break the toolkit, so nothing below " + + "would be exercised:\n" + probe, probe.contains(AwtErrorPromptMain.PROBE_MARKER)); + // and it must be the shape that defeats both guards by category - an Error that is not a + // LinkageError. A JDK that started raising, say, a HeadlessException here would make this test pass + // for the wrong reason, since the pre-existing catch (Exception) already handled that. + Assert.assertTrue("the toolkit failure must be an Error that is neither an Exception nor a " + + "LinkageError, or it was already caught before this fix:\n" + probe, + probe.contains("isLinkageError=false isException=false")); + + String prompt = runChild(AwtErrorPromptMain.MODE_PROMPT); + Assert.assertTrue("the default prompt did not survive an AWTError from the browser launch:\n" + + prompt, prompt.contains(AwtErrorPromptMain.DEGRADED_MARKER)); + // degrading to "no browser" must not degrade to "no instructions": a user whose browser could not + // be opened still needs the URL and the code + Assert.assertTrue("the verification URL must still be printed:\n" + prompt, + prompt.contains("https://verify.example/device")); + Assert.assertTrue("the user code must still be printed:\n" + prompt, prompt.contains("WDJB-MJHT")); + } + + private static String locationOf(Class type) { + CodeSource source = type.getProtectionDomain().getCodeSource(); + Assert.assertNotNull("no code source for " + type.getName(), source); + return new File(source.getLocation().getPath()).getPath(); + } + + private static String readFully(InputStream in) throws Exception { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[8192]; + int read; + while ((read = in.read(buffer)) != -1) { + out.write(buffer, 0, read); + } + return out.toString("UTF-8"); + } + + /** + * Runs {@link AwtErrorPromptMain} in a second JVM whose AWT toolkit cannot initialise, and returns its + * merged stdout/stderr. Asserts a clean exit, so a child that failed its own precondition reports + * through its printed reason rather than through a silent skip. + */ + private static String runChild(String mode) throws Exception { + // The CLASS path, deliberately, not the module path this suite itself runs on: io.questdb.test does + // not read java.desktop, so a child in that module could not touch Desktop at all and would prove + // nothing. On the class path everything lands in the unnamed module, which reads every resolved + // module, and java.desktop resolves by default there. + String classPath = locationOf(AwtErrorPromptMain.class) // the child and this test + + File.pathSeparator + locationOf(DeviceCodePrompt.class) // the client under test + + File.pathSeparator + locationOf(Logger.class); // org.slf4j, a mandatory client requires + String javaBin = System.getProperty("java.home") + File.separator + "bin" + File.separator + "java"; + + ProcessBuilder pb = new ProcessBuilder( + javaBin, + // headless=false explicitly, not merely by default: headless mode short-circuits the + // assistive-technology loading altogether, so a CI runner that forces headless through + // JAVA_TOOL_OPTIONS would otherwise turn this into a silent no-op. It is also what makes the + // failure deterministic on a machine with no display, where the toolkit instead raises its + // "cannot connect to the X11 window server" AWTError - the same category, the same guard. + "-Djava.awt.headless=false", + "-Djavax.accessibility.assistive_technologies=" + MISSING_AT_CLASS, + // deliberately NOT setting questdb.client.oidc.open.browser: the kill-switch returns before + // BrowserLauncher touches java.awt.Desktop, so a run with it set would never reach the error + // this test exercises + "-classpath", classPath, + AwtErrorPromptMain.class.getName(), + mode); + pb.redirectErrorStream(true); + Process process = pb.start(); + String output = readFully(process.getInputStream()); + int exitCode = process.waitFor(); + Assert.assertEquals("the " + mode + " run of " + AwtErrorPromptMain.class.getSimpleName() + + " failed:\n" + output, 0, exitCode); + return output; + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherTest.java new file mode 100644 index 000000000..e926a2bd2 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/BrowserLauncherTest.java @@ -0,0 +1,132 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import org.junit.Assert; +import org.junit.Test; + +import java.lang.reflect.Method; +import java.net.URI; + +/** + * Covers {@code BrowserLauncher}, the best-effort browser launch behind the default device-code prompt. + *

+ * REFLECTION, deliberately: the class and all three methods are package-private, and the only public route + * to them is {@code DeviceCodePrompt.openBrowser().promptUser(...)}, whose whole contract is that it does + * nothing observable - it swallows every failure and, on a headless machine, never launches anything either + * way. There is no public path to assert on, so the choice is reflection or no coverage of the scheme + * allowlist at all. The one behaviour that IS observable from outside is what {@code open()} does with the + * kill-switch, and that is pinned without reflection by + * {@link DesktopFreeModulePathTest#testTheBrowserKillSwitchIsHonouredByOpenItself}, which runs it where + * {@code java.desktop} is absent. + */ +public class BrowserLauncherTest { + + @Test + public void testAcceptsHttpAndHttps() throws Exception { + Assert.assertNotNull(invokeSafeHttpUri("https://idp.example.com/device?user_code=ABCD")); + Assert.assertNotNull(invokeSafeHttpUri("http://localhost:8080/device")); + // the scheme allowlist is case-insensitive + Assert.assertNotNull(invokeSafeHttpUri("HTTPS://idp.example.com")); + } + + @Test + public void testOpenIsBestEffortForRejectedUrls() throws Exception { + // these URLs are rejected by the scheme/parse allowlist, so open() returns at the safeHttpUri null + // check before touching java.awt.Desktop. Assert the rejection holds so the no-op below is provably + // the URL-rejection path (not an incidental headless no-op), then confirm open() tolerates each + // without throwing (and never launches a real browser, so the test is safe on a desktop machine too) + Assert.assertNull(invokeSafeHttpUri("javascript:alert(1)")); + Assert.assertNull(invokeSafeHttpUri("not a url")); + invokeOpen(null); + invokeOpen("javascript:alert(1)"); + invokeOpen("not a url"); + } + + @Test + public void testOpenRespectsDisableProperty() throws Exception { + // SCOPE: this test proves the property READ flips, and nothing more. A browser launch is + // unobservable from here, and on a headless JVM open() is a no-op whether or not it ever consulted + // the flag - so the invokeOpen call below would stay green against an open() that ignored the + // kill-switch outright (verified: removing the gate from open() leaves every assertion in this class + // passing). What open() DOES with the flag is pinned by + // DesktopFreeModulePathTest.testTheBrowserKillSwitchIsHonouredByOpenItself, which runs it where + // java.desktop is absent: reaching Desktop throws there, so the two directions become distinguishable + // - quiet with the kill-switch off, LinkageError with it on. A VALID http(s) URL is used below so the + // no-op under "false" is at least not URL rejection, and so this class never pops a browser. + String validUrl = "https://idp.example.com/device?user_code=ABCD"; + Assert.assertNotNull("the URL must be one open() would otherwise launch", invokeSafeHttpUri(validUrl)); + String prop = "questdb.client.oidc.open.browser"; + String prev = System.getProperty(prop); + try { + System.clearProperty(prop); + Assert.assertTrue("the browser launch must default to enabled", invokeIsBrowserOpenEnabled()); + System.setProperty(prop, "true"); + Assert.assertTrue("\"true\" must enable the browser launch", invokeIsBrowserOpenEnabled()); + System.setProperty(prop, "false"); + Assert.assertFalse("\"false\" must disable the browser launch (the kill-switch)", invokeIsBrowserOpenEnabled()); + invokeOpen(validUrl); // kill-switch off: must return without launching and without throwing + } finally { + if (prev == null) { + System.clearProperty(prop); + } else { + System.setProperty(prop, prev); + } + } + } + + @Test + public void testRejectsDangerousOrMalformedUrls() throws Exception { + // an attacker-influenced verification URI must not smuggle a non-http(s) scheme to the OS handler + Assert.assertNull(invokeSafeHttpUri("javascript:alert(1)")); + Assert.assertNull(invokeSafeHttpUri("data:text/html,")); + Assert.assertNull(invokeSafeHttpUri("file:///etc/passwd")); + Assert.assertNull(invokeSafeHttpUri("ftp://example.com/x")); + Assert.assertNull(invokeSafeHttpUri("not a url")); + Assert.assertNull(invokeSafeHttpUri("//idp.example.com/device")); + Assert.assertNull(invokeSafeHttpUri("")); + Assert.assertNull(invokeSafeHttpUri(null)); + } + + // BrowserLauncher is a package-private helper; the client is an open module, so reflection reaches its + // static methods without widening production visibility for the test. + private static boolean invokeIsBrowserOpenEnabled() throws Exception { + Method m = Class.forName("io.questdb.client.cutlass.auth.BrowserLauncher").getDeclaredMethod("isBrowserOpenEnabled"); + m.setAccessible(true); + return (boolean) m.invoke(null); + } + + private static void invokeOpen(String url) throws Exception { + Method m = Class.forName("io.questdb.client.cutlass.auth.BrowserLauncher").getDeclaredMethod("open", String.class); + m.setAccessible(true); + m.invoke(null, url); + } + + private static URI invokeSafeHttpUri(String url) throws Exception { + Method m = Class.forName("io.questdb.client.cutlass.auth.BrowserLauncher").getDeclaredMethod("safeHttpUri", String.class); + m.setAccessible(true); + return (URI) m.invoke(null, url); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeKillSwitchMain.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeKillSwitchMain.java new file mode 100644 index 000000000..0a061f527 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeKillSwitchMain.java @@ -0,0 +1,99 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; + +/** + * Proves that {@code BrowserLauncher.open} itself honours the + * {@code questdb.client.oidc.open.browser} kill-switch, rather than merely that + * {@code isBrowserOpenEnabled()} reads the property. + *

+ * In an ordinary JVM the two are indistinguishable: a browser launch is unobservable from a test, and on a + * headless machine {@code Desktop.isDesktopSupported()} answers false, so {@code open()} is a no-op whether + * or not it ever consulted the property. Run it where {@code java.desktop} does NOT exist and the difference + * becomes loud - reaching {@code Desktop} throws a {@link LinkageError}: + *

+ * The second half is what makes the first half mean something: without it, an {@code open()} that returned + * immediately for any reason at all would look like a working kill-switch. + *

+ * Java 8 source level and no {@code java.lang.module} API, like the test that launches it - the JDK 8 + * release profile compiles this test tree. + */ +public final class DesktopFreeKillSwitchMain { + + static final int EXIT_DESKTOP_REACHABLE = 2; + static final int EXIT_NO_LINKAGE_ERROR = 3; + static final int EXIT_THREW_WHILE_DISABLED = 4; + static final String SUCCESS_MARKER = "DESKTOP-FREE-KILL-SWITCH-OK"; + private static final String OPEN_BROWSER_PROPERTY = "questdb.client.oidc.open.browser"; + // a URL open() would otherwise hand to the OS: http(s), so it survives the scheme allowlist and the run + // reaches the Desktop call. Nothing can open it here - this JVM has no java.desktop at all. + private static final String VALID_URL = "https://verify.example/device?user_code=WDJB-MJHT"; + + private DesktopFreeKillSwitchMain() { + } + + public static void main(String[] args) throws Exception { + try { + Class.forName("java.awt.Desktop"); + System.out.println("java.awt.Desktop is reachable, so neither half below proves anything"); + System.exit(EXIT_DESKTOP_REACHABLE); + } catch (ClassNotFoundException expected) { + // desktop-free, as this run requires + } + + final Method open = Class.forName("io.questdb.client.cutlass.auth.BrowserLauncher") + .getDeclaredMethod("open", String.class); + open.setAccessible(true); + + System.setProperty(OPEN_BROWSER_PROPERTY, "false"); + try { + open.invoke(null, VALID_URL); + } catch (InvocationTargetException e) { + System.out.println("open() must return at the kill-switch, before java.awt.Desktop: " + e.getCause()); + System.exit(EXIT_THREW_WHILE_DISABLED); + } + + System.setProperty(OPEN_BROWSER_PROPERTY, "true"); + try { + open.invoke(null, VALID_URL); + System.out.println("open() did not reach java.awt.Desktop with the kill-switch ON, so the quiet " + + "run above was not the kill-switch doing its job"); + System.exit(EXIT_NO_LINKAGE_ERROR); + } catch (InvocationTargetException e) { + if (!(e.getCause() instanceof LinkageError)) { + System.out.println("expected a LinkageError from the missing java.desktop, got: " + e.getCause()); + System.exit(EXIT_NO_LINKAGE_ERROR); + } + } + + System.out.println(SUCCESS_MARKER); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java new file mode 100644 index 000000000..5ee97eee8 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java @@ -0,0 +1,149 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.DeviceCodePrompt; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.Test; +import org.slf4j.Logger; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.InputStream; +import java.security.CodeSource; + +/** + * Guards the promise {@link DeviceCodePrompt#openBrowser()} makes - that the browser launch is skipped + * "on a runtime without the {@code java.desktop} module" and never prevents sign-in - for the one + * configuration where the promise used to be unkeepable: a build of this source used as an EXPLICIT + * module. + *

+ * {@code module-info.java} declared a mandatory {@code requires java.desktop}. Mandatory requires are + * satisfied during module RESOLUTION, before a single line of client code runs, so on a runtime image + * without {@code java.desktop} the JVM failed at startup with {@code FindException} and the + * {@code LinkageError} catch in {@code openBrowser()} - the thing that implements the promise - never got + * to run. The published artifact is built on JDK 8 and carries no descriptor, so it is an automatic module + * and was never affected; a build from this source is. + *

+ * The test runs a second JVM with the client on the MODULE path and {@code --limit-modules} naming only + * the client, which limits the universe to the client plus the closure of its mandatory requires. A + * {@code requires static} is not part of that closure, so {@code java.desktop} is absent and the child + * proves three things at once: the module resolved without it, {@code java.awt.Desktop} really is + * unreachable (see {@link DesktopFreePromptMain}, which fails the run if it is not - this is what catches + * a revert to a mandatory requires, since the closure would then drag {@code java.desktop} back in), and + * the default prompt still renders the challenge and returns. + *

+ * Java 8 source level, and no {@code java.lang.module} API: the JDK 8 release profile compiles this test + * tree, only excluding {@code module-info.java}. + */ +public class DesktopFreeModulePathTest { + + @Test(timeout = 60_000) + public void testTheBrowserKillSwitchIsHonouredByOpenItself() throws Exception { + // BrowserLauncherTest can only assert that isBrowserOpenEnabled() reads the property: a browser + // launch is unobservable, and on a headless JVM open() is a no-op whether or not it ever consulted + // the flag, so that test passes even against an open() that ignores it. Without java.desktop the two + // become distinguishable - see DesktopFreeKillSwitchMain, which drives both directions. + String output = runInDesktopFreeJvm(DesktopFreeKillSwitchMain.class); + Assert.assertTrue("the kill-switch is not what stopped the browser launch:\n" + output, + output.contains(DesktopFreeKillSwitchMain.SUCCESS_MARKER)); + } + + @Test(timeout = 60_000) + public void testTheModuleResolvesAndPromptsWithoutJavaDesktop() throws Exception { + String output = runInDesktopFreeJvm(DesktopFreePromptMain.class); + Assert.assertTrue("the child never reached the end of the prompt:\n" + output, + output.contains(DesktopFreePromptMain.SUCCESS_MARKER)); + // the challenge itself must still have been shown - degrading to "no browser" must not degrade to + // "no instructions", which would leave a user with no way to sign in at all + Assert.assertTrue("the verification URL must still be printed:\n" + output, + output.contains("https://verify.example/device")); + Assert.assertTrue("the user code must still be printed:\n" + output, output.contains("WDJB-MJHT")); + } + + private static File locationOf(Class type) { + CodeSource source = type.getProtectionDomain().getCodeSource(); + Assert.assertNotNull("no code source for " + type.getName(), source); + return new File(source.getLocation().getPath()); + } + + /** + * Runs {@code main} in a second JVM that has the client on the MODULE path and no {@code java.desktop}, + * and returns its merged stdout/stderr. Asserts a clean exit, so a child that failed its own + * preconditions reports through its printed reason rather than through a silent skip. + */ + private static String runInDesktopFreeJvm(Class main) throws Exception { + File clientLocation = locationOf(DeviceCodePrompt.class); + Assume.assumeFalse("the module system arrived in Java 9; nothing to resolve on a Java 8 runtime", + "1.8".equals(System.getProperty("java.specification.version"))); + // A JDK 8 build produces no module-info.class: the artifact is then an automatic module, which + // reads every observable module and is exactly the configuration this defect never reached. + Assume.assumeTrue("no module descriptor next to " + clientLocation + " (a JDK 8 build)", + new File(clientLocation, "module-info.class").isFile()); + + File slf4jLocation = locationOf(Logger.class); // org.slf4j is a mandatory requires of the client + File testClasses = locationOf(DesktopFreeModulePathTest.class); + String javaBin = System.getProperty("java.home") + File.separator + "bin" + File.separator + "java"; + + ProcessBuilder pb = new ProcessBuilder( + javaBin, + // Second net under each child's own desktop-free check, and independent of it: should + // java.desktop ever be present, Desktop.isDesktopSupported() answers false in headless mode, + // so nothing can reach a real browser on a developer's machine. It does not weaken either + // test - headless changes what Desktop ANSWERS, not whether the class reference links. + "-Djava.awt.headless=true", + "--module-path", clientLocation.getPath() + File.pathSeparator + slf4jLocation.getPath(), + // the universe: io.questdb.client and the closure of its MANDATORY requires, and nothing + // else. This is what makes the child desktop-free - and what makes a mandatory + // `requires java.desktop` visible, because the closure would then include it. + "--limit-modules", "io.questdb.client", + // the main class runs from the class path, so the client is not a root by default + "--add-modules", "io.questdb.client", + "-classpath", testClasses.getPath(), + main.getName()); + // deliberately NOT setting questdb.client.oidc.open.browser here: the kill-switch returns before + // BrowserLauncher touches java.awt.Desktop, so a run with it set would never reach the LinkageError + // these tests exercise. What keeps a browser from opening is each child's own precondition - the + // prompt and the launch run only in the arm that proved Desktop unreachable - plus the headless flag. + pb.redirectErrorStream(true); + Process process = pb.start(); + String output = readFully(process.getInputStream()); + int exitCode = process.waitFor(); + Assert.assertEquals("the desktop-free module-path run of " + main.getSimpleName() + " failed:\n" + + output, 0, exitCode); + return output; + } + + private static String readFully(InputStream in) throws Exception { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[8192]; + int read; + while ((read = in.read(buffer)) != -1) { + out.write(buffer, 0, read); + } + return out.toString("UTF-8"); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreePromptMain.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreePromptMain.java new file mode 100644 index 000000000..1248cd314 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreePromptMain.java @@ -0,0 +1,84 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.DeviceAuthorizationChallenge; +import io.questdb.client.cutlass.auth.DeviceCodePrompt; + +/** + * The child half of {@link DesktopFreeModulePathTest}: run by that test in a second JVM which has + * {@code io.questdb.client} on the MODULE path and a universe that does not contain + * {@code java.desktop}. Started from the class path, so this class itself is in the unnamed module and + * reads whatever the module graph resolved. + *

+ * Deliberately written to Java 8 source level, like the test that launches it: the JDK 8 release profile + * compiles this same test tree (it just excludes {@code module-info.java}), so a {@code ModuleLayer} or + * {@code java.lang.module} reference here would break that build. Reachability of {@code java.awt.Desktop} + * is therefore probed with {@code Class.forName}, which answers the same question. + */ +public final class DesktopFreePromptMain { + + /** + * Exit code for "the precondition did not hold": {@code java.awt.Desktop} was reachable, so nothing + * below would have proven anything. + */ + static final int EXIT_DESKTOP_REACHABLE = 2; + /** + * Printed on success. The launching test asserts on it rather than on the exit code alone, so a JVM + * that exited 0 without running this far cannot read as a pass. + */ + static final String SUCCESS_MARKER = "DESKTOP-FREE-PROMPT-OK"; + + private DesktopFreePromptMain() { + } + + public static void main(String[] args) { + // The precondition, checked FIRST and by itself fatal: this JVM must genuinely lack java.desktop. + // It is also the regression guard. --limit-modules closes over the MANDATORY requires of the + // module it is given, so a module-info that says "requires java.desktop" drags java.desktop back + // into the universe and lands here - reachable - even though the run asked for a desktop-free one. + try { + Class.forName("java.awt.Desktop"); + System.out.println("java.awt.Desktop is reachable, so this JVM is not desktop-free: " + + "io.questdb.client must declare `requires static java.desktop`, not a mandatory requires"); + System.exit(EXIT_DESKTOP_REACHABLE); + } catch (ClassNotFoundException expected) { + // Desktop-free, as intended - the module resolved without java.desktop. The prompt runs HERE, in + // the arm that proved it, and not after the try: this is the one place in the suite that drives + // the real openBrowser() with the questdb.client.oidc.open.browser kill-switch left enabled, so + // "we checked first" must be structural rather than a matter of statement order that a later + // edit could undo. A reachable Desktop can then never reach the launch below - and the test + // that starts this JVM passes -Djava.awt.headless=true as a second, independent net. + // + // The promise DeviceCodePrompt.openBrowser() documents: the browser open is best-effort and + // "skipped on a runtime without the java.desktop module", never fatal. Reaching BrowserLauncher + // throws a LinkageError here, which openBrowser() swallows, leaving the printed URL and code. + DeviceCodePrompt.openBrowser().promptUser(new DeviceAuthorizationChallenge( + "WDJB-MJHT", "https://verify.example/device", null, 300, 5)); + + System.out.println(SUCCESS_MARKER); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/FileTokenStoreTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/FileTokenStoreTest.java new file mode 100644 index 000000000..264c11b5b --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/FileTokenStoreTest.java @@ -0,0 +1,3337 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.FileTokenStore; +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.auth.PersistedToken; +import io.questdb.client.cutlass.auth.TokenStore; +import io.questdb.client.cutlass.auth.TokenStoreKey; +import io.questdb.client.std.Os; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.nio.charset.StandardCharsets; +import java.nio.file.AccessDeniedException; +import java.nio.file.DirectoryStream; +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.nio.file.attribute.FileTime; +import java.nio.file.attribute.PosixFilePermission; +import java.nio.file.attribute.PosixFilePermissions; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; +import static io.questdb.client.test.tools.TestUtils.repeat; + +/** + * Coverage for {@link FileTokenStore}. + *

+ * PLATFORM SCOPE. CI runs Linux only, so the store's Windows-motivated arms are covered here to the extent a + * POSIX host can reach them, and no further: + *

+ * Closing the last two needs a Windows CI agent, or a filesystem-provider fixture that reports neither POSIX + * attributes nor atomic moves. + */ +public class FileTokenStoreTest { + + private static final Set OWNER_ONLY_DIR_PERMS = + PosixFilePermissions.fromString("rwx------"); + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test + public void testAdvancedConstructorRejectsNonPositiveTimings() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + // a non-positive acquire budget or staleness window is rejected: a tiny/zero staleness would make + // every freshly created lock look abandoned, so acquirers would steal each other's live locks + try { + new FileTokenStore(dir, 0, 1000); + Assert.fail("a zero lock acquire budget must be rejected"); + } catch (OidcAuthException expected) { + // expected + } + try { + new FileTokenStore(dir, -1, 1000); + Assert.fail("a negative lock acquire budget must be rejected"); + } catch (OidcAuthException expected) { + // expected + } + try { + new FileTokenStore(dir, 1000, 0); + Assert.fail("a zero lock staleness window must be rejected"); + } catch (OidcAuthException expected) { + // expected + } + try { + new FileTokenStore(dir, 1000, -1); + Assert.fail("a negative lock staleness window must be rejected"); + } catch (OidcAuthException expected) { + // expected + } + }); + } + + @Test + public void testAdvancedConstructorRejectsOverCapAcquireBudget() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + // the acquire budget is capped: getToken() can wait it out on the latency-sensitive flush path, so + // an unbounded budget would let a misconfiguration stall a flush; the cap also keeps a waiter + // degrading well before it could begin stealing live locks + try { + new FileTokenStore(dir, 30_001, 600_000); + Assert.fail("an over-cap lock acquire budget must be rejected"); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("lockAcquireBudgetMillis")); + } + // the cap boundary itself is accepted + new FileTokenStore(dir, 30_000, 600_000); + }); + } + + @Test + public void testArrayWrappedJsonReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // a valid entry, then the same object wrapped in a top-level array. The wrapper leaves the + // fingerprint fields untouched, so only the non-object-root rejection - not a fingerprint mismatch - + // can reject it: the parser must refuse a shape that is not a single flat JSON object + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("the plain object must load", store.load(key)); + byte[] obj = Files.readAllBytes(tokenFile(dir, key)); + byte[] wrapped = new byte[obj.length + 2]; + wrapped[0] = '['; + System.arraycopy(obj, 0, wrapped, 1, obj.length); + wrapped[wrapped.length - 1] = ']'; + Files.write(tokenFile(dir, key), wrapped); + Assert.assertNull("an array-wrapped object must be rejected as a malformed shape", store.load(key)); + }); + } + + @Test + public void testAudienceNullVersusEmptyFingerprint() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey nullAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + TokenStoreKey withAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", "api://billing", false); + + // a null audience round-trips: the writer omits the member, and nullableEquals matches an absent + // file value against a null key audience + store.save(nullAud, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull(store.load(nullAud)); + byte[] nullAudBytes = Files.readAllBytes(tokenFile(dir, nullAud)); + + store.save(withAud, sampleToken("ACCESS-2", "REFRESH-2")); + byte[] withAudBytes = Files.readAllBytes(tokenFile(dir, withAud)); + + // place each file under the *other* key's name to isolate the in-file audience fingerprint check + // from the hash-based file naming: a recorded audience must not match a null-audience key, and an + // absent audience must not match an audience-bearing key + Files.write(tokenFile(dir, nullAud), withAudBytes); + Assert.assertNull("a recorded audience must not match a null-audience key", store.load(nullAud)); + Files.write(tokenFile(dir, withAud), nullAudBytes); + Assert.assertNull("an absent audience must not match an audience-bearing key", store.load(withAud)); + }); + } + + @Test + public void testClearDeletesFile() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertTrue(Files.exists(tokenFile(dir, key))); + + store.clear(key); + Assert.assertFalse(Files.exists(tokenFile(dir, key))); + Assert.assertNull(store.load(key)); + // clearing a missing entry is a no-op, not an error + store.clear(key); + }); + } + + @Test + public void testClearErasesTheEntryOnAnInterruptCarryingThread() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-SECRET")); + Assert.assertTrue(Files.exists(tokenFile(dir, key))); + + // A sign-out runs on shutdown and cleanup paths, which is exactly where a thread carries an + // interrupt flag - the standard cancellation idiom re-asserts it. Routed through inLock, the + // carried flag made the delete never run, and clear() discarded the false return: the plaintext + // refresh token stayed on disk with no exception and no warning, and the next process start + // silently resumed the old identity. A local delete has nothing to abandon on a cancellation. + Thread.currentThread().interrupt(); + final boolean flagSurvived; + try { + store.clear(key); + flagSurvived = Thread.currentThread().isInterrupted(); + } finally { + Thread.interrupted(); // do not leak the flag into the next test + } + + // erasure first: it is the claim this test exists for, so it is the one a regression must break + Assert.assertFalse("clear() must erase the credential even on an interrupt-carrying thread", + Files.exists(tokenFile(dir, key))); + Assert.assertNull(store.load(key)); + Assert.assertTrue("the caller's cancellation signal must survive clear()", flagSurvived); + }); + } + + @Test + public void testClearOnEmptyStoreIsNoOp() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); // a non-existent subdirectory + FileTokenStore store = new FileTokenStore(dir); + // clearing an identity that was never saved must be a no-op and must not create the store directory + // just to run the now-locked delete + store.clear(sampleKey()); + Assert.assertFalse("clear must not create the store directory", Files.exists(dir)); + }); + } + + @Test + public void testConcurrentStealContentionDegradesCleanly() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + // a lock abandoned by a crashed holder, backdated well past the staleness window + Path lock = lockFile(dir, key); + Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000)); + + // Several "processes" run the FULL inLock path against the one abandoned lock, and it must + // degrade CLEANLY: every contender runs its critical section (none is starved or wedged), each + // under a lock it actually holds, and no atomic-capture temp file leaks. + // + // SCOPE NOTE: these four contenders do NOT race at the file-lock layer. inLock() takes the + // in-process PROCESS_LOCKS entry for key.hash() before any lock-file logic, and that map is + // static, so four distinct FileTokenStore instances on one identity are serialized whatever the + // file lock does - the first reclaims the abandoned lock, and each of the rest finds the path free + // and creates its own. The genuinely concurrent capture race is + // testConcurrentStealersLeaveExactlyOneWinner, which drives stealIfStale directly because that is + // the only way to reach it inside one JVM. + // + // It deliberately does NOT assert strict mutual exclusion. stealIfStale is best-effort by design + // and documents a three-actor residual - a peer recreating the lock in the isStale->capture gap + // while a second captures that fresh live lock and a third claims the momentarily-free path, all + // at once - under which two holders can briefly run concurrently. That residual needs three or + // more contenders and degrades only to one extra token refresh (a re-prompt on a + // rotating-refresh-token IdP), never a torn or forged credential, since the Layer-1 atomic-rename + // write is independent of the lock. + // testSameProcessContendersSerializeAndBothStealStaleLock covers the two-contender same-JVM case + // (where PROCESS_LOCKS, not the file-lock capture, provides the exclusion it asserts). + final int threads = 4; + AtomicInteger ran = new AtomicInteger(); + // the stamp on the lock file while each contender runs, so the assertions below can tell an + // acquisition apart from a no-op + List stampWhileRunning = Collections.synchronizedList(new ArrayList<>()); + TokenStore.CriticalSection section = () -> { + stampWhileRunning.add(readLockStamp(lock)); + Os.sleep(100); + ran.incrementAndGet(); + return true; + }; + + // A contender that THREW instead of running would die on its own thread and leave the counts + // below looking like a clean degrade, so carry the first failure back to the test thread. + AtomicReference workerError = new AtomicReference<>(); + Thread[] ts = new Thread[threads]; + for (int i = 0; i < threads; i++) { + // a generous acquire budget so a contender waits for the lock rather than giving up early + FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000); + ts[i] = new Thread(() -> { + try { + store.inLock(key, section); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }, "steal-contender-" + i); + } + for (Thread t : ts) { + t.start(); + } + for (Thread t : ts) { + joinOrFail(t, "a steal contender"); + } + + Assert.assertNull("a contender failed instead of running its critical section: " + workerError.get(), + workerError.get()); + Assert.assertEquals("every contender must run its critical section", threads, ran.get()); + // Teeth the run count alone does not have: threads==ran holds even with the whole acquire deleted, + // since inLock() runs the section lock-free when it cannot get a lock. A contender that never + // acquired would have run with the crashed holder's stamp still in place (or with no lock file at + // all), so require a live stamp - one that is neither absent nor the crashed holder's - under + // every critical section. + Assert.assertEquals(threads, stampWhileRunning.size()); + for (String stamp : stampWhileRunning) { + Assert.assertNotNull("a contender ran with no lock file at all, so it never acquired one", stamp); + Assert.assertNotEquals("a contender ran while the crashed holder's lock was still in place", + "crashed-holder-stamp", stamp); + } + Assert.assertFalse("the last holder must have released its lock", Files.exists(lock)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testConcurrentStealersLeaveExactlyOneWinner() throws Exception { + assertMemoryLeak(() -> { + // The capture race stealIfStale is written for: several stealers judge the same abandoned lock + // stale at once, exactly one wins the ATOMIC_MOVE capture and drops it, and the losers take the + // NoSuchFileException arm and fall back to the wait rather than deleting anything. inLock() cannot + // reach this race inside one JVM - PROCESS_LOCKS serializes same-identity threads ahead of every + // lock-file syscall - so drive the steal itself. Reflection is the seam: the test tree is a + // separate io.questdb.client.test.* package with its own module-info, so package-private access + // is structurally unavailable. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000)); + + Method stealIfStale = FileTokenStore.class.getDeclaredMethod("stealIfStale", Path.class); + stealIfStale.setAccessible(true); + + final int stealers = 8; + CyclicBarrier start = new CyclicBarrier(stealers); + AtomicReference failure = new AtomicReference<>(); + Thread[] ts = new Thread[stealers]; + for (int i = 0; i < stealers; i++) { + // one store per "process", as in the sibling test + FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000); + ts[i] = new Thread(() -> { + try { + start.await(10, TimeUnit.SECONDS); + stealIfStale.invoke(store, lock); + } catch (Throwable t) { + failure.compareAndSet(null, t); + } + }, "steal-contender"); + } + for (Thread t : ts) { + t.start(); + } + for (Thread t : ts) { + joinOrFail(t, "a stealer"); + } + + Assert.assertNull("a stealer failed outright: " + failure.get(), failure.get()); + Assert.assertFalse("the abandoned lock must be gone - one stealer captures it and drops it, and a " + + "loser must not restore what it never captured", Files.exists(lock)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testRestoreCapturedLockPutsAPeersLockBackByteForByte() throws Exception { + assertMemoryLeak(() -> { + // The arm testConcurrentStealersLeaveExactlyOneWinner cannot reach. Its three observables - no + // stealer threw, the lock is gone, no capture temp survives - all hold under the bare + // deleteIfExists(lock) that stealIfStale's own comment says must never be used, because a bare + // delete also removes the lock and leaves no temp. What separates the two is what happens when the + // capture-verify says "this is NOT the lock we judged stale": the capture must go BACK, with the + // peer's exact bytes, because releaseLock verifies the stamp before deleting and a peer whose + // stamp we corrupted can no longer release its own lock. + // + // Driving stealIfStale itself cannot get here: confirmedStale is false only when a peer replaces + // the file between the staleness read and the ATOMIC_MOVE, an interleaving no test can force + // without a production seam. So drive the restore directly. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Path captured = dir.resolve(lock.getFileName().toString() + ".capture.tmp"); + + byte[] peerStamp = "peer-owner-nonce-9f3c".getBytes(StandardCharsets.UTF_8); + Files.write(captured, peerStamp); + + Method restore = FileTokenStore.class.getDeclaredMethod( + "restoreCapturedLock", Path.class, Path.class); + restore.setAccessible(true); + restore.invoke(new FileTokenStore(dir, 30_000, 60_000), lock, captured); + + Assert.assertTrue("the peer's lock must be back at the lock path", Files.exists(lock)); + Assert.assertArrayEquals("the peer's owner stamp must survive byte for byte, or releaseLock's " + + "own-stamp check will refuse to let that peer release its own lock", + peerStamp, Files.readAllBytes(lock)); + Assert.assertFalse("the capture copy must not be left behind", Files.exists(captured)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testRestoreCapturedLockLeavesAThirdPartysLockUntouched() throws Exception { + assertMemoryLeak(() -> { + // The reason the restore links rather than renames. Files.move without REPLACE_EXISTING stats the + // target and then renames, and rename(2) replaces silently - so a third party that claimed the + // freed path between those two steps would have its live lock destroyed by the very call whose + // comment promises to leave it intact. createLink fails outright instead. Here the third party has + // already claimed the path when the restore runs, which is the deterministic end of that race and + // needs no interleaving to reproduce. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Path captured = dir.resolve(lock.getFileName().toString() + ".capture.tmp"); + + byte[] thirdPartyStamp = "third-party-live-nonce".getBytes(StandardCharsets.UTF_8); + Files.write(lock, thirdPartyStamp); + Files.write(captured, "our-captured-copy".getBytes(StandardCharsets.UTF_8)); + + Method restore = FileTokenStore.class.getDeclaredMethod( + "restoreCapturedLock", Path.class, Path.class); + restore.setAccessible(true); + restore.invoke(new FileTokenStore(dir, 30_000, 60_000), lock, captured); + + Assert.assertArrayEquals("a third party's LIVE lock must survive the restore untouched - " + + "overwriting it admits two holders at once", + thirdPartyStamp, Files.readAllBytes(lock)); + Assert.assertFalse("our capture copy must be dropped, not left to accumulate", + Files.exists(captured)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testProcessLocksDoNotGrowWithTheIdentityCount() throws Exception { + assertMemoryLeak(() -> { + // TokenStoreKey is public and inLock() is public API, so a process mints as many identities as its + // caller needs - one per end user in a multi-tenant service. An unpruned map roots a 64-char hash + // plus a lock for every identity EVER SEEN, for the life of the JVM. Bound it by the identities + // actually in flight instead: once the last caller on an identity leaves, its entry goes. + Field field = FileTokenStore.class.getDeclaredField("PROCESS_LOCKS"); + field.setAccessible(true); + java.util.Map locks = (java.util.Map) field.get(null); + + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir, 30_000, 600_000); + AtomicInteger ran = new AtomicInteger(); + final int identities = 500; + for (int i = 0; i < identities; i++) { + TokenStoreKey key = new TokenStoreKey("client-" + i, "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + Assert.assertTrue(store.inLock(key, () -> { + ran.incrementAndGet(); + // the entry has to EXIST while its critical section runs, or the lock is serializing + // nothing; the retirement below is only interesting because of this + Assert.assertFalse("the identity's lock must be held for the critical section", + locks.isEmpty()); + return true; + })); + } + Assert.assertEquals("every identity must still have run its critical section", identities, ran.get()); + + // Nothing is in flight now, so nothing may be left behind. This is the assertion the old + // stripe-table version could not make: it asserted the table was the same ARRAY of the same + // LENGTH, which is true of any immutable array whether or not the code under test works. + Assert.assertEquals("no entry may outlive its last caller [left=" + locks + "]", 0, locks.size()); + // and it must be usable, not merely empty: a fresh identity still serializes + Assert.assertTrue(store.inLock(sampleKey(), () -> true)); + Assert.assertEquals("the fresh identity must be retired too", 0, locks.size()); + }); + } + + @Test(timeout = 60_000) + public void testSameIdentityInDifferentDirectoriesDoesNotSerialize() throws Exception { + assertMemoryLeak(() -> { + // The mirror of testUnrelatedIdentitiesDoNotSerializeOnEachOther, along the other axis. That one + // varies the identity within one directory; this one keeps the identity and varies the + // directory - which is the shape this class's javadoc and the README actually prescribe for + // signing several application users in at once: "a store each, on a per-user directory". + // + // Those two stores hold DIFFERENT files, so serializing them buys nothing and costs everything + // the sibling test describes: the lock spans a whole token-endpoint round trip, its acquire has + // no budget, and getToken() sits on the ILP flush path. + // + // Same CyclicBarrier trick: it trips only when both callers are inside their critical section at + // once, which two callers sharing one lock can never be. + // distinct directories, not storeDir() twice - that helper returns one fixed path, and two + // stores over ONE directory are the case that must keep serializing + Path dirA = temp.getRoot().toPath().resolve("oidc-tokens-user-a"); + Path dirB = temp.getRoot().toPath().resolve("oidc-tokens-user-b"); + createStoreDir(dirA); + createStoreDir(dirB); + FileTokenStore storeA = new FileTokenStore(dirA, 30_000, 600_000); + FileTokenStore storeB = new FileTokenStore(dirB, 30_000, 600_000); + TokenStoreKey key = sampleKey(); + + CyclicBarrier bothInside = new CyclicBarrier(2); + AtomicReference workerError = new AtomicReference<>(); + AtomicInteger ran = new AtomicInteger(); + TokenStore.CriticalSection section = () -> { + try { + ran.incrementAndGet(); + bothInside.await(20, TimeUnit.SECONDS); + return true; + } catch (Exception e) { + throw new RuntimeException(e); + } + }; + + Thread tA = new Thread(() -> { + try { + Assert.assertTrue(storeA.inLock(key, section)); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }, "store-a"); + Thread tB = new Thread(() -> { + try { + Assert.assertTrue(storeB.inLock(key, section)); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }, "store-b"); + tA.start(); + tB.start(); + joinOrFail(tA, "store A"); + joinOrFail(tB, "store B"); + + Assert.assertNull("one identity in two directories must not queue on a single in-process lock; " + + "the barrier times out when they share one", workerError.get()); + Assert.assertEquals("both critical sections must have run", 2, ran.get()); + }); + } + + @Test(timeout = 60_000) + public void testUnrelatedIdentitiesDoNotSerializeOnEachOther() throws Exception { + assertMemoryLeak(() -> { + // The in-process lock owes exactly one guarantee: two callers on the SAME identity must not run + // the read-refresh-write concurrently and double-POST a rotating refresh token. It owes unrelated + // identities nothing, and serializing them is not the free trade it looks - the lock is held + // across a whole token-endpoint round trip while the caller also holds its OidcDeviceAuth + // instance lock, and the acquire has no budget. One tenant's ILP flush blocking on another + // tenant's stalled refresh is a stall the flush path cannot see coming. + // + // MORE identities than a 64-entry stripe table has stripes, so the pigeonhole principle - not a + // probability - guarantees a collision under any fixed table of that size. Every identity must + // still be able to sit inside its critical section at once. + final int identities = 65; + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir, 30_000, 600_000); + + CyclicBarrier allInside = new CyclicBarrier(identities); + AtomicReference workerError = new AtomicReference<>(); + AtomicInteger inside = new AtomicInteger(); + List workers = new ArrayList<>(); + for (int i = 0; i < identities; i++) { + TokenStoreKey key = new TokenStoreKey("tenant-" + i, "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + Thread t = new Thread(() -> { + try { + Assert.assertTrue(store.inLock(key, () -> { + try { + inside.incrementAndGet(); + // Trips only once every identity is holding its own lock. Two identities + // sharing one lock can never both get here, so a stripe table deadlocks the + // barrier and the await below times out. + allInside.await(30, TimeUnit.SECONDS); + return true; + } catch (Exception e) { + throw new RuntimeException(e); + } + })); + } catch (Throwable e) { + workerError.compareAndSet(null, e); + allInside.reset(); // unblock the peers so the test fails loudly, not by timing out + } + }, "tenant-lock-" + i); + t.setDaemon(true); + workers.add(t); + t.start(); + } + for (Thread t : workers) { + joinOrFail(t, "tenant lock holder"); + } + if (workerError.get() != null) { + throw new AssertionError("unrelated identities did not hold their locks concurrently; " + + identities + " identities, " + inside.get() + " got inside", workerError.get()); + } + Assert.assertEquals("every identity must have entered its critical section", + identities, inside.get()); + }); + } + + @Test + public void testOneConfigurationHoldsOneActiveLogin() throws Exception { + assertMemoryLeak(() -> { + // The store is keyed on a CONFIGURATION - client id, endpoints, scope, audience, + // groups-in-token mode - and no field of TokenStoreKey names a subject. Two people signing in + // through the same configuration therefore address the same file, and the later sign-in + // overwrites the earlier one: a store holds a single active login, which is the boundary the + // README and the FileTokenStore javadoc now state. Anyone reading "one file per identity" as + // "one file per person" would size a multi-user deployment on a guarantee that does not exist. + FileTokenStore store = new FileTokenStore(storeDir()); + TokenStoreKey first = sampleKey(); + TokenStoreKey second = sampleKey(); // a separate instance, identical configuration + Assert.assertEquals("identical configurations must address the same entry", + first.hash(), second.hash()); + + store.save(first, sampleToken("ACCESS-ALICE", "REFRESH-ALICE")); + store.save(second, sampleToken("ACCESS-BOB", "REFRESH-BOB")); + + PersistedToken loaded = store.load(first); + Assert.assertNotNull(loaded); + Assert.assertEquals("the later sign-in must own the entry", "ACCESS-BOB", loaded.getAccessToken()); + Assert.assertEquals("REFRESH-BOB", loaded.getRefreshToken()); + // and the first login is gone rather than merged or kept alongside + Assert.assertEquals("one configuration keeps one entry, not one per person", + "ACCESS-BOB", store.load(second).getAccessToken()); + + // separating them is the caller's job, and a separate store directory is what does it + Path aliceDir = temp.getRoot().toPath().resolve("alice"); + FileTokenStore aliceStore = FileTokenStore.at(aliceDir); + aliceStore.save(first, sampleToken("ACCESS-ALICE", "REFRESH-ALICE")); + Assert.assertEquals("a per-user store keeps a per-user login", + "ACCESS-ALICE", aliceStore.load(first).getAccessToken()); + Assert.assertEquals("and does not disturb the shared one", + "ACCESS-BOB", store.load(first).getAccessToken()); + }); + } + + @Test + public void testReplaceTargetGivesUpAfterTheRetryBudget() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to deny the rename", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + Assume.assumeFalse("a root process bypasses the directory permissions this denial relies on", + "root".equals(System.getProperty("user.name"))); + assertMemoryLeak(() -> { + // The other half of the Windows sharing-violation arm: a denial that never clears must surface, + // not be retried forever or swallowed. save() turns the throw into its best-effort degrade. + Path dir = storeDir(); + createStoreDir(dir); + Path tmp = Files.write(dir.resolve("payload.tmp"), "NEW".getBytes(StandardCharsets.UTF_8)); + Path target = Files.write(dir.resolve("payload.json"), "OLD".getBytes(StandardCharsets.UTF_8)); + + Method replaceTarget = FileTokenStore.class.getDeclaredMethod("replaceTarget", Path.class, Path.class); + replaceTarget.setAccessible(true); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("r-x------")); + try { + long start = System.currentTimeMillis(); + try { + replaceTarget.invoke(null, tmp, target); + Assert.fail("a rename denied on every attempt must be reported, not swallowed"); + } catch (InvocationTargetException e) { + Assert.assertTrue("the LAST denial must be the one rethrown, was: " + e.getCause(), + e.getCause() instanceof AccessDeniedException); + } + // 5 attempts means 4 backoff sleeps of 20ms; a shape that gave up on the first denial + // (the pre-retry behaviour, and what Windows would routinely trip over) returns at once + long elapsed = System.currentTimeMillis() - start; + Assert.assertTrue("it must have spent the whole retry budget, took " + elapsed + "ms", + elapsed >= 80); + } finally { + // restore, or the temp-folder rule cannot delete the tree + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwx------")); + } + Assert.assertEquals("a failed replace must leave the previous entry intact", + "OLD", new String(Files.readAllBytes(target), StandardCharsets.UTF_8)); + }); + } + + @Test + public void testReplaceTargetPreservesAnInterruptDeliveredDuringItsBackoff() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to deny the rename", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + Assume.assumeFalse("a root process bypasses the directory permissions this denial relies on", + "root".equals(System.getProperty("user.name"))); + assertMemoryLeak(() -> { + // replaceTarget's retry backoff is the one interruptible wait on the save path. save() parks and + // restores only the flag it saw on ENTRY, so an interrupt arriving mid-save has to survive this + // sleep on its own - and that interrupt is exactly what PoolHousekeeper.stop() delivers to break a + // recovery step. Os.sleep() swallowed it: it catches InterruptedException, sleeps on to its + // deadline and never re-asserts the flag, so the stop signal was destroyed and the caller's later + // isInterrupted() checks read false. + Path dir = storeDir(); + createStoreDir(dir); + Path tmp = Files.write(dir.resolve("payload.tmp"), "NEW".getBytes(StandardCharsets.UTF_8)); + Path target = Files.write(dir.resolve("payload.json"), "OLD".getBytes(StandardCharsets.UTF_8)); + + Method replaceTarget = FileTokenStore.class.getDeclaredMethod("replaceTarget", Path.class, Path.class); + replaceTarget.setAccessible(true); + // Deny the rename the same way the sibling test does, and prove the denial bites on THIS host + // before resting on it: without a denial the first attempt succeeds, no backoff runs, and the + // assertion below would pass without exercising anything. + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("r-x------")); + try { + try { + Files.move(tmp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + Assert.fail("the rename must be denied while the store directory is not writable"); + } catch (AccessDeniedException expected) { + // as intended - every attempt inside replaceTarget will now be denied, so it reaches its + // backoff and stays there for the whole budget + } + + // Set the flag BEFORE the call rather than racing a second thread into the 20ms window: the + // first backoff observes it either way, and this leaves nothing to time. + Thread.currentThread().interrupt(); + try { + replaceTarget.invoke(null, tmp, target); + Assert.fail("a permanently denied rename must surface its AccessDeniedException"); + } catch (InvocationTargetException e) { + Assert.assertTrue("expected the denial to propagate, got " + e.getCause(), + e.getCause() instanceof AccessDeniedException); + } + Assert.assertTrue("replaceTarget must hand back an interrupt delivered during its backoff, " + + "or a stop signal aimed at the save path is lost", + Thread.currentThread().isInterrupted()); + } finally { + Thread.interrupted(); // do not leak the flag into the next test + // whatever happened above, leave the tree deletable + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwx------")); + } + }); + } + + @Test + public void testReplaceTargetRetriesADeniedRenameThenSucceeds() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to deny the rename", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + Assume.assumeFalse("a root process bypasses the directory permissions this denial relies on", + "root".equals(System.getProperty("user.name"))); + assertMemoryLeak(() -> { + // replaceTarget retries a denied rename because on WINDOWS a concurrent reader holding the target + // open makes the atomic replace fail transiently with AccessDeniedException. CI is Linux-only, so + // the denial is produced the one way a POSIX host can: rename(2) needs write permission on the + // containing directory, so taking it away denies the move exactly as the sharing violation does, + // and restoring it mid-retry stands in for the Windows reader closing its handle. + Path dir = storeDir(); + createStoreDir(dir); + Path tmp = Files.write(dir.resolve("payload.tmp"), "NEW".getBytes(StandardCharsets.UTF_8)); + Path target = Files.write(dir.resolve("payload.json"), "OLD".getBytes(StandardCharsets.UTF_8)); + + Method replaceTarget = FileTokenStore.class.getDeclaredMethod("replaceTarget", Path.class, Path.class); + replaceTarget.setAccessible(true); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("r-x------")); + // Prove the denial is real on THIS host before the test rests on it. Without this the whole test + // passes vacuously wherever the mode bits do not bite - the first attempt inside replaceTarget + // simply succeeds and no retry is ever exercised. + try { + Files.move(tmp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + Assert.fail("the rename must be denied while the store directory is not writable"); + } catch (AccessDeniedException expected) { + // exactly what a Windows sharing violation produces, and what the retry loop is written for + } + Thread reopener = new Thread(() -> { + // after the first backoff (20ms) but well inside the 5-attempt budget + Os.sleep(30); + try { + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwx------")); + } catch (IOException e) { + throw new AssertionError("could not restore the directory permissions", e); + } + }, "denial-clearer"); + reopener.setDaemon(true); + reopener.start(); + try { + replaceTarget.invoke(null, tmp, target); + } finally { + reopener.join(10_000); + // whatever happened above, leave the tree deletable + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwx------")); + } + + Assert.assertEquals("the retry must complete the replace once the denial clears", + "NEW", new String(Files.readAllBytes(target), StandardCharsets.UTF_8)); + Assert.assertFalse("an atomic move consumes the temp file", Files.exists(tmp)); + }); + } + + @Test + public void testSameProcessContendersSerializeAndBothStealStaleLock() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + // a lock abandoned by a crashed holder, backdated well past the staleness window + Path lock = lockFile(dir, key); + Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000)); + + // Two threads of the SAME JVM contend for the one abandoned lock. SCOPE NOTE: the mutual exclusion + // asserted below (overlaps==0, maxInside==1) is provided by the in-process PROCESS_LOCKS entry, + // which inLock() takes for key.hash() BEFORE any file-lock logic - so it would hold + // even if the file-lock steal were broken. What this test genuinely proves is that two same-process + // contenders each steal the stale lock and run their critical section (ran==2), serialized, without + // leaving an orphaned capture temp (assertNoCaptureTempFiles). The CROSS-process capture-verify in + // stealIfStale - that among separate OS PROCESSES exactly one steal wins - is masked by PROCESS_LOCKS + // here and cannot be exercised in a single JVM; it is verified by inspection, and a two-holder + // outcome is a documented best-effort residual anyway. The N-way degrade path is + // testConcurrentStealContentionDegradesCleanly. + final int threads = 2; + AtomicInteger inside = new AtomicInteger(); + AtomicInteger maxInside = new AtomicInteger(); + AtomicInteger overlaps = new AtomicInteger(); + AtomicInteger ran = new AtomicInteger(); + TokenStore.CriticalSection section = () -> { + int now = inside.incrementAndGet(); + maxInside.accumulateAndGet(now, Math::max); + if (now > 1) { + overlaps.incrementAndGet(); + } + Os.sleep(100); + inside.decrementAndGet(); + ran.incrementAndGet(); + return true; + }; + + // A contender that THREW would never enter the section, so `inside` never rises and the + // exclusion assertions below pass on a test that proved nothing. Carry the first failure back. + AtomicReference workerError = new AtomicReference<>(); + Thread[] ts = new Thread[threads]; + for (int i = 0; i < threads; i++) { + // a generous acquire budget so a contender waits for the lock rather than degrading to a + // lock-free run (a degraded action runs without the lock and could legitimately overlap) + FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000); + ts[i] = new Thread(() -> { + try { + store.inLock(key, section); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }, "same-process-contender-" + i); + } + for (Thread t : ts) { + t.start(); + } + for (Thread t : ts) { + joinOrFail(t, "a same-process contender"); + } + + Assert.assertNull("a contender failed instead of running its critical section: " + workerError.get(), + workerError.get()); + Assert.assertEquals("every contender must run its critical section", threads, ran.get()); + Assert.assertEquals("same-process contenders must never overlap (PROCESS_LOCKS serializes them)", 0, overlaps.get()); + Assert.assertEquals("at most one holder at a time", 1, maxInside.get()); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testControlCharactersRoundTrip() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + TokenStoreKey key = sampleKey(); + // a refresh token carrying every control-escape branch of the JSON writer - the short escapes + // (\b \f \n \r \t) and the \\u00XX arm - plus a quote and a backslash must round-trip byte for byte; + // the served-token char check lives in OidcDeviceAuth, so the store itself must preserve these + String refresh = "R\b\f\n\r\t\"\\Z"; + // also exercise the hex-escape branch: control chars below 0x20 that are not one of the short escapes + refresh = refresh + (char) 0x01 + (char) 0x1f; + store.save(key, new PersistedToken("ACCESS-1", null, refresh, 1L, 1000L)); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals("ACCESS-1", loaded.getAccessToken()); + Assert.assertEquals(refresh, loaded.getRefreshToken()); + }); + } + + @Test + public void testClearRemovesOrphanedWriteTemps() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + // A crash between createTempFile and the atomic rename orphans this: it holds the FULL + // serialized entry - access, id and refresh tokens in plaintext. save()'s sweep only reclaims + // temps past the staleness window and only ever runs from save(), so a caller that cleared and + // never signed in again left a live refresh token on disk indefinitely, contradicting clear()'s + // "removes any persisted entry for this identity". + Path orphan = dir.resolve(key.hash() + "9999.tmp"); + Files.write(orphan, "{\"refresh_token\":\"REFRESH-1\"}".getBytes(StandardCharsets.UTF_8)); + + store.clear(key); + + Assert.assertFalse("clear must remove the token file", Files.exists(tokenFile(dir, key))); + Assert.assertFalse("clear must also reclaim an orphaned write temp holding the refresh token", + Files.exists(orphan)); + }); + } + + @Test + public void testClearRemovesAnOrphanedWriteTempStampedInTheFuture() throws Exception { + assertMemoryLeak(() -> { + // The sibling test's orphan is stamped in the past, which every clock agrees on. clear() passes + // minAgeMillis 0 to mean "at ANY age", but that went through the same "now - mtime >= minAge" + // comparison save()'s staleness-bounded sweep uses - and an mtime AHEAD of now makes the left + // side negative, which is not >= 0. So the one sweep that is supposed to ignore the clock was + // the one the clock could veto, and save()'s sweep skips the same file against a larger + // threshold, leaving nothing in the class that would ever reclaim it. + // + // A future mtime needs no attacker: a network home whose server clock leads the client's (which + // this class documents as in scope), or a wall-clock step back from an NTP correction, a VM + // snapshot restore, or a container started before its time sync. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + Path orphan = dir.resolve(key.hash() + "9999.tmp"); + Files.write(orphan, "{\"refresh_token\":\"REFRESH-1\"}".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(orphan, + FileTime.fromMillis(System.currentTimeMillis() + 3_600_000L)); + Assert.assertTrue("the fixture must be stamped in the future - that is the whole point", + Files.getLastModifiedTime(orphan).toMillis() > System.currentTimeMillis()); + + store.clear(key); + + Assert.assertFalse("clear must remove the token file", Files.exists(tokenFile(dir, key))); + Assert.assertFalse("clear() is an explicit 'forget this credential', so its sweep must not be " + + "conditional on a clock: a temp stamped in the future holds the same plaintext " + + "refresh token, and no later sweep in this class reclaims it either", + Files.exists(orphan)); + }); + } + + @Test + public void testCorruptFileReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + createStoreDir(dir); + Files.write(tokenFile(dir, key), "this is not json {{{".getBytes(StandardCharsets.UTF_8)); + Assert.assertNull(store.load(key)); + }); + } + + @Test + public void testDirectoryLockHeartbeatKeepsALiveRecoveryLockFresh() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to drive directory recovery", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + FileTokenStore store = new FileTokenStore(dir); + Field hookField = FileTokenStore.class.getDeclaredField("beforeUntrustedDiscardHook"); + hookField.setAccessible(true); + hookField.set(store, (Runnable) () -> { + Path lock = dir.resolve(".store.lock"); + try { + long firstModified = Files.getLastModifiedTime(lock).toMillis(); + long deadline = System.nanoTime() + 2_000_000_000L; + while (System.nanoTime() - deadline < 0 + && Files.getLastModifiedTime(lock).toMillis() <= firstModified) { + Thread.sleep(25L); + } + Assert.assertTrue("a live directory lock must renew its lease while recovery is paused", + Files.getLastModifiedTime(lock).toMillis() > firstModified); + } catch (IOException e) { + throw new RuntimeException(e); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + }); + + store.save(sampleKey(), sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertFalse("the directory lock must still be released after the heartbeat", + Files.exists(dir.resolve(".store.lock"))); + }); + } + + @Test + public void testEmptyAudienceNormalizesToNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + // an empty-string audience is normalised to null: getAudience() reports null, it shares the + // null-audience identity hash/file, and its save->load round-trips (the pre-fix "" broke its own + // round-trip because the writer recorded "audience":"" but the fingerprint treated it as absent) + TokenStoreKey emptyAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", "", false); + TokenStoreKey nullAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + Assert.assertNull("an empty audience must normalise to null", emptyAud.getAudience()); + Assert.assertEquals("null and empty audiences must share one identity hash", nullAud.hash(), emptyAud.hash()); + + store.save(emptyAud, sampleToken("ACCESS-1", "REFRESH-1")); + PersistedToken loaded = store.load(emptyAud); + Assert.assertNotNull("an empty-audience key must load the entry it just saved", loaded); + Assert.assertEquals("ACCESS-1", loaded.getAccessToken()); + }); + } + + @Test + public void testEmptyDirectoryLockAbandonedByACrashedWriterIsReclaimed() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + // The narrower crash window: the process died after CREATE_NEW but before its nonce write. The + // required directory lock cannot use the refresh lock's 5s empty-file grace because its default + // acquisition budget is only 3s; unlike inLock, it cannot safely degrade after that budget. + Path directoryLock = dir.resolve(".store.lock"); + Files.createFile(directoryLock); + + PersistedToken loaded = new FileTokenStore(dir).load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertFalse("the empty abandoned directory lock must be stolen and released", + Files.exists(directoryLock)); + }); + } + + @Test + public void testEmptyFileReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + createStoreDir(dir); + Files.write(tokenFile(dir, key), new byte[0]); + Assert.assertNull(store.load(key)); + }); + } + + @Test + public void testEmptyLockStolenAfterGraceWithinStaleWindow() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // a holder that crashed between creating its lock and stamping it leaves an empty, unstamped lock. + // It must be reclaimable on the short empty-lock grace, not held un-stealable until the full + // staleness window elapses: here the staleness window is large (60s) but the empty lock is backdated + // only past the grace, so a steal here can only come from the empty-lock-grace path. Without that + // path the empty lock would not be stale (10s < 60s) and would wedge this acquirer into a lock-free + // degrade, leaving the lock in place. + FileTokenStore store = new FileTokenStore(dir, 2000, 60_000); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.createFile(lock); // empty, unstamped + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 10_000)); + + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(key, () -> { + ran.set(true); + return true; + }); + + Assert.assertTrue("the action must run", ran.get()); + Assert.assertTrue(result); + Assert.assertFalse("an empty (unstamped) lock past the grace must be stolen and acquired (then released)," + + " not wedge the acquirer for the full staleness window", Files.exists(lock)); + }); + } + + @Test + public void testEmptyLockGraceIsNotShortenedByASmallStaleWindow() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // A staleness window far below the 5s empty-lock grace. The grace is the ONLY thing standing + // between a peer caught between its exclusive create and its stamp and having its live lock + // stolen, which is why the frozen cross-language contract says a client MUST NOT shorten it. + // Clamping the grace down to lockStaleMillis did exactly that, silently. + FileTokenStore store = new FileTokenStore(dir, 300, 100); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.createFile(lock); // empty: a peer momentarily between its create and its stamp + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 1_000)); + + AtomicBoolean ran = new AtomicBoolean(); + store.inLock(key, () -> { + ran.set(true); + return true; + }); + + Assert.assertTrue("inLock must still run, degrading to lock-free", ran.get()); + Assert.assertTrue("a 1s-old empty lock is inside the 5s grace and must not be stolen", + Files.exists(lock)); + Assert.assertEquals("the peer's lock must be left exactly as it was", 0, Files.size(lock)); + }); + } + + @Test + public void testEnsureDirectoryTightensPreExistingDirPerms() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + // a pre-existing, world-accessible store directory (a permissive umask, a prior tool, or a hostile + // local pre-create) must be tightened to owner-only before a token is written into it + createStoreDir(dir); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + + FileTokenStore store = new FileTokenStore(dir); + store.save(sampleKey(), sampleToken("ACCESS-1", "REFRESH-1")); + + Assert.assertEquals("a pre-existing directory must be re-restricted to owner-only", + PosixFilePermissions.fromString("rwx------"), Files.getPosixFilePermissions(dir)); + }); + } + + @Test + public void testFingerprintMismatchReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + // save under one identity, then read with a key that hashes the same name but differs in the + // stored fingerprint - simulated by writing the saved bytes under a *different* key's file name + TokenStoreKey saved = sampleKey(); + store.save(saved, sampleToken("ACCESS-1", "REFRESH-1")); + byte[] bytes = Files.readAllBytes(tokenFile(dir, saved)); + TokenStoreKey other = new TokenStoreKey("other-client", saved.getTokenEndpoint(), + saved.getDeviceAuthorizationEndpoint(), saved.getScope(), null, false); + Files.write(tokenFile(dir, other), bytes); + // the file exists under other.hash(), but its in-file fingerprint says client_id=questdb, so the + // load for `other` must reject it rather than serve questdb's token + Assert.assertNull(store.load(other)); + }); + } + + @Test + public void testFrozenSchemaEndpointsCarryAnExplicitPort() throws Exception { + assertMemoryLeak(() -> { + // The file NAME hash is pinned by testHashMatchesFrozenCrossLanguageContract; the file BODY was + // not. The two endpoint fields are part of the fingerprint and are compared with an exact string + // compare, not a URL compare, so they must carry the canonical rendering - port always explicit - + // that design/oidc-token-persistence.md specifies. A peer client (the Python one) that writes the + // default port implicitly produces a file this client silently ignores: load() returns null, the + // process re-prompts and re-persists in its own encoding, and the two never converge. That is + // invisible in every other test, because they all round-trip through this client's own writer. + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + String withPort = "{\"v\":1,\"client_id\":\"questdb\"," + + "\"token_endpoint\":\"https://idp.example.com:443/token\"," + + "\"device_authorization_endpoint\":\"https://idp.example.com:443/device\"," + + "\"scope\":\"openid\",\"groups_in_token\":false," + + "\"access_token\":\"ACCESS-1\",\"refresh_token\":\"REFRESH-1\"," + + "\"expires_at_millis\":1730000000000,\"token_ttl_millis\":300000}"; + Files.write(tokenFile(dir, key), withPort.getBytes(StandardCharsets.UTF_8)); + Assert.assertNotNull("the documented encoding must load", store.load(key)); + + // the same document with the default ports omitted - the shape a naive reading of the schema + // example invites - must NOT load, which is exactly why the spec pins the explicit port + String withoutPort = withPort + .replace("https://idp.example.com:443/token", "https://idp.example.com/token") + .replace("https://idp.example.com:443/device", "https://idp.example.com/device"); + Files.write(tokenFile(dir, key), withoutPort.getBytes(StandardCharsets.UTF_8)); + Assert.assertNull("an implicit default port must not match the canonical fingerprint", + store.load(key)); + }); + } + + @Test + public void testHashMatchesFrozenCrossLanguageContract() throws Exception { + assertMemoryLeak(() -> { + // the file name is a frozen cross-language contract (the Python client mirrors it byte for byte): + // lowercase-hex SHA-256 of "questdb-oidc-token-v1" and the six identity fields, NUL-separated, a + // null audience rendered as "" and groups_in_token as '1'/'0'. Pin it to golden values so a change + // to the prefix, separator, field order, or null/boolean encoding that would silently stop two + // clients sharing one file is caught here. + TokenStoreKey withAudience = new TokenStoreKey("questdb", + "https://idp.example.com:443/as/token", "https://idp.example.com:443/as/device", + "openid", "api://billing", false); + Assert.assertEquals("eee1a742a27499d176bcdaed8635c14a3edbdef1d68b61c05c3c2158a5bfbcca", withAudience.hash()); + + // a null audience hashes as an empty field, not the literal "null" + TokenStoreKey nullAudience = new TokenStoreKey("questdb", + "https://idp.example.com:443/as/token", "https://idp.example.com:443/as/device", + "openid", null, false); + Assert.assertEquals("1dca0e8192ae529b94c1ac5493f09f8a45e641e4e0ec316333c0cbfeeccfef0e", nullAudience.hash()); + + // groups_in_token participates in the identity, so it flips the hash to a different file + TokenStoreKey groups = new TokenStoreKey("questdb", + "https://idp.example.com:443/as/token", "https://idp.example.com:443/as/device", + "openid", "api://billing", true); + Assert.assertEquals("5193f668130b28cd9430f5271011f1044b3b1c1e78bfc4f45d7688a3d9b1ceb0", groups.hash()); + Assert.assertNotEquals(withAudience.hash(), groups.hash()); + }); + } + + @Test + public void testKeyIsUsableAsAMapKey() throws Exception { + assertMemoryLeak(() -> { + // TokenStore's contract says entries are keyed by TokenStoreKey, and its javadoc invites a + // custom store backed by a keychain or a vault. Without value equality that reads as an + // invitation to a Map that never hits: OidcDeviceAuth builds its key once per instance, so a + // Map-backed store looks correct until a second instance - or a restart - rebuilds an equal key, + // misses, and sends the user back through the device flow on every refresh. The bundled + // FileTokenStore is unaffected only because it keys by hash() for the file name. + TokenStoreKey a = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid groups", "api://billing", true); + TokenStoreKey sameIdentity = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid groups", "api://billing", true); + TokenStoreKey otherClient = new TokenStoreKey("other", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid groups", "api://billing", true); + + Assert.assertEquals("two keys naming one identity must be equal", a, sameIdentity); + Assert.assertEquals("equal keys must share a hashCode", a.hashCode(), sameIdentity.hashCode()); + Assert.assertNotEquals("a different client id is a different identity", a, otherClient); + Assert.assertNotEquals(a, null); + Assert.assertNotEquals(a, "not a key"); + + Map byKey = new HashMap<>(); + byKey.put(a, "entry"); + Assert.assertEquals("a rebuilt key must find the entry the original stored", "entry", + byKey.get(sameIdentity)); + Assert.assertNull("a different identity must not read another's entry", byKey.get(otherClient)); + byKey.put(sameIdentity, "replaced"); + Assert.assertEquals("an equal key must replace, not duplicate", 1, byKey.size()); + + // equality means "the same store entry", so it follows the constructor's null/empty audience + // normalisation rather than the raw arguments - the two below share one file, and now one + // Map slot too + TokenStoreKey emptyAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", "", false); + TokenStoreKey nullAud = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + Assert.assertEquals("keys addressing one entry must be equal", emptyAud, nullAud); + Assert.assertEquals(emptyAud.hashCode(), nullAud.hashCode()); + }); + } + + @Test + public void testInLockAbandonsFileLockWaitOnInterrupt() throws Exception { + assertMemoryLeak(() -> { + // The lock-file poll used Os.sleep, which catches InterruptedException, keeps sleeping to its own + // deadline and never re-asserts the flag - so a cancellation aimed at this wait was swallowed and + // the whole budget elapsed regardless. The budget maxes out at 30s, the same as QWP's close() + // shutdown budget, so a caller stuck here made close() time out and delegate the teardown of the + // native client, the cursor engine and the store-and-forward slot lock. + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir, 30_000, 600_000); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + // a live peer's stamped lock: not empty, so the empty-lock grace does not apply, and far inside + // the staleness window, so it is never stolen - the waiter can only poll + Files.write(lock, "live-peer-nonce".getBytes(StandardCharsets.UTF_8)); + + AtomicBoolean ran = new AtomicBoolean(); + AtomicReference result = new AtomicReference<>(); + AtomicReference waiterError = new AtomicReference<>(); + AtomicBoolean flagLeftSet = new AtomicBoolean(); + Thread waiter = new Thread(() -> { + try { + result.set(store.inLock(key, () -> { + ran.set(true); + return true; + })); + flagLeftSet.set(Thread.currentThread().isInterrupted()); + } catch (Throwable t) { + // without this the throw dies on this thread and the assertions below read it as + // "result was never set" - a null-vs-FALSE mismatch that names nothing + waiterError.compareAndSet(null, t); + } + }, "file-lock-waiter"); + waiter.setUncaughtExceptionHandler((t, e) -> waiterError.compareAndSet(null, e)); + waiter.setDaemon(true); + waiter.start(); + // Read off the waiter's own stack that it is INSIDE the poll before interrupting it. The latch + // this replaced counted down at the top of the thread body, so it proved only that the thread had + // been scheduled: an interrupt landing before the call becomes a CARRIED flag, which inLock + // answers by returning false without ever entering the wait, and every assertion below would then + // pass on a path this test does not mean to exercise. + awaitInside(waiter, "acquireLock"); + + long start = System.currentTimeMillis(); + waiter.interrupt(); + waiter.join(10_000); + long elapsed = System.currentTimeMillis() - start; + + Assert.assertNull("the waiter failed instead of abandoning its wait: " + waiterError.get(), + waiterError.get()); + Assert.assertFalse("the waiter must not still be polling out the 30s budget", waiter.isAlive()); + Assert.assertTrue("the interrupt must cut the poll short, took " + elapsed + "ms", elapsed < 5_000); + Assert.assertFalse("the refresh must not start once the wait was cancelled", ran.get()); + Assert.assertEquals("an abandoned wait reports no refresh", Boolean.FALSE, result.get()); + // The false above is not self-describing: a refresh that RAN and failed returns the same value. + // Only the restored flag separates them, and OidcDeviceAuth acts on the difference - a bare + // false sends signIn() into the interactive device flow (a browser, then a poll loop on Os.sleep + // that ignores interrupts) on a thread its owner just cancelled, and makes getToken() arm the + // instance-wide refresh back-off over a credential that is fine. This assertion used to require + // the opposite, on the reasoning that consuming the signal was "acting on it"; consuming it is + // what made the two cases indistinguishable. + Assert.assertTrue("inLock must leave the interrupt flag set when a cancellation abandoned its " + + "wait, or the caller cannot tell that apart from a failed refresh", flagLeftSet.get()); + Assert.assertTrue("the peer's live lock must be left alone", Files.exists(lock)); + }); + } + + @Test + public void testInLockAbandonsProcessLockWaitOnInterrupt() throws Exception { + assertMemoryLeak(() -> { + // The in-process lock that serializes same-identity threads was taken with lock(), which no + // interrupt can break. A peer thread holds it for a whole refresh round trip, so a caller behind + // it was unreachable by the one lever QWP's ConnectCancellation has. + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore holderStore = new FileTokenStore(dir, 30_000, 600_000); + FileTokenStore waiterStore = new FileTokenStore(dir, 30_000, 600_000); + TokenStoreKey key = sampleKey(); + + CountDownLatch holding = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + Thread holder = new Thread(() -> holderStore.inLock(key, () -> { + holding.countDown(); + try { + release.await(30, TimeUnit.SECONDS); + } catch (InterruptedException ignored) { + Thread.currentThread().interrupt(); + } + return true; + }), "process-lock-holder"); + holder.setDaemon(true); + holder.start(); + Assert.assertTrue("the holder must enter its critical section", holding.await(5, TimeUnit.SECONDS)); + + AtomicBoolean ran = new AtomicBoolean(); + AtomicReference result = new AtomicReference<>(); + AtomicBoolean flagAfterReturn = new AtomicBoolean(); + AtomicReference waiterError = new AtomicReference<>(); + Thread waiter = new Thread(() -> { + try { + result.set(waiterStore.inLock(key, () -> { + ran.set(true); + return true; + })); + // sampled INSIDE the thread and immediately after the return, because that is the + // instant OidcDeviceAuth inspects it to tell "the wait was cancelled" from "the + // refresh ran and failed" + flagAfterReturn.set(Thread.currentThread().isInterrupted()); + } catch (Throwable t) { + // see the sibling test: a throw here must arrive as itself, not as a missing result + waiterError.compareAndSet(null, t); + } + }, "process-lock-waiter"); + waiter.setUncaughtExceptionHandler((t, e) -> waiterError.compareAndSet(null, e)); + waiter.setDaemon(true); + waiter.start(); + // inside inLock is the right point here: it is where the process lock is taken, and inLock's + // carried-interrupt check has already run by then, so the interrupt below is unambiguously the + // LIVE cancellation this test is about. See the sibling test for what the latch could not prove. + awaitInside(waiter, "inLock"); + + long start = System.currentTimeMillis(); + waiter.interrupt(); + waiter.join(10_000); + long elapsed = System.currentTimeMillis() - start; + + Assert.assertNull("the waiter failed instead of abandoning its wait: " + waiterError.get(), + waiterError.get()); + Assert.assertFalse("the waiter must not still be blocked on the process lock", waiter.isAlive()); + Assert.assertTrue("the interrupt must break the process-lock wait, took " + elapsed + "ms", + elapsed < 5_000); + Assert.assertFalse("the refresh must not start once the wait was cancelled", ran.get()); + Assert.assertEquals("an abandoned wait reports no refresh", Boolean.FALSE, result.get()); + // Same contract as the file-lock sibling: false alone cannot be told from a failed refresh, and + // OidcDeviceAuth answers a failed refresh with the interactive device flow. + Assert.assertTrue("inLock must leave the interrupt flag set when a cancellation abandoned its " + + "wait, or the caller cannot tell that apart from a failed refresh", + flagAfterReturn.get()); + + release.countDown(); + holder.join(10_000); + Assert.assertFalse("the holder must finish its critical section", holder.isAlive()); + }); + } + + @Test + public void testInLockHonoursItsAcquireBudgetBehindALivePeerLock() throws Exception { + assertMemoryLeak(() -> { + // The budget is a PROMISE to the caller: inLock waits at most lockAcquireBudgetMillis for a peer's + // lock and then runs the critical section lock-free, because getToken() reaches this on an ILP + // producer's flush path. Anything blocking added inside the acquire breaks that promise silently - + // as ManagementFactory.getRuntimeMXBean().getName() did in the owner stamp, resolving the local + // hostname (InetAddress.getLocalHost()) for 3.2s inside a 200ms budget, once per JVM, on the first + // credential refresh. That one was caught end-to-end by + // OidcDeviceAuthPersistenceTest.testGetTokenDegradesWhenStoreLockHeld; this pins the same bound + // directly on the store, where such a call would live. + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir, 200, 600_000); + TokenStoreKey key = sampleKey(); + // a live peer's stamped lock: neither the empty-lock grace nor the staleness steal applies, so the + // acquire can only poll it out and degrade + Files.write(lockFile(dir, key), "live-peer-nonce".getBytes(StandardCharsets.UTF_8)); + + AtomicBoolean ran = new AtomicBoolean(); + long start = System.nanoTime(); + boolean result = store.inLock(key, () -> { + ran.set(true); + return true; + }); + long elapsedMillis = (System.nanoTime() - start) / 1_000_000L; + + Assert.assertTrue("a peer's lock must not stop the critical section, only unserialize it", ran.get()); + Assert.assertTrue(result); + Assert.assertTrue("the whole budget must be spent polling, was " + elapsedMillis + "ms", + elapsedMillis >= 200); + // Generous, because this is a wall-clock bound on a shared machine: it must ride out a GC pause or + // a scheduling hiccup, while still failing on the kind of multi-second blocking call it exists to + // keep out of the acquire. + Assert.assertTrue("the acquire must degrade on its budget, not stall, was " + elapsedMillis + "ms", + elapsedMillis < 2_000); + Assert.assertTrue("the peer's live lock must be left alone", Files.exists(lockFile(dir, key))); + }); + } + + @Test + public void testInLockPreservesACarriedInterruptFlag() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + AtomicBoolean ran = new AtomicBoolean(); + + // The root cause behind clear() losing a credential, pinned on its own. The lock is FREE and + // uncontended here, so nothing is being waited on: a carried flag is the caller's own state and + // must survive. ReentrantLock.lockInterruptibly() begins with Thread.interrupted(), so testing + // the flag only after the acquire read it as a live cancellation and consumed it - which also + // made getToken() report "could not be refreshed" on a reachable endpoint while destroying the + // caller's signal. + Thread.currentThread().interrupt(); + boolean result; + boolean flagSurvived; + try { + result = store.inLock(sampleKey(), () -> { + ran.set(true); + return true; + }); + flagSurvived = Thread.currentThread().isInterrupted(); + } finally { + Thread.interrupted(); // do not leak the flag into the next test + } + + Assert.assertTrue("a carried interrupt must survive inLock", flagSurvived); + Assert.assertFalse("inLock must not start the critical section for a cancelled caller", ran.get()); + Assert.assertFalse("and must report that the action did not run", result); + }); + } + + @Test + public void testInLockDegradesWhenDirectoryUnusable() throws Exception { + assertMemoryLeak(() -> { + // a regular file standing where the store directory's parent must be makes directory preparation + // throw IOException; inLock must still run the action lock-free rather than fail a sign-in + Path blocker = temp.getRoot().toPath().resolve("blocker"); + Files.write(blocker, new byte[]{1}); + FileTokenStore store = new FileTokenStore(blocker.resolve("oidc-tokens")); + + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(sampleKey(), () -> { + ran.set(true); + return true; + }); + + Assert.assertTrue("must run the action even when the directory cannot be created (degrade)", ran.get()); + Assert.assertTrue(result); + }); + } + + @Test + public void testInLockDegradesWhenHeldByFreshLock() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // small acquire budget, large staleness: a fresh foreign lock cannot be acquired or stolen + FileTokenStore store = new FileTokenStore(dir, 200, 60_000); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.createFile(lock); // a live holder's fresh lock + + AtomicBoolean ran = new AtomicBoolean(); + long start = System.currentTimeMillis(); + boolean result = store.inLock(key, () -> { + ran.set(true); + return true; + }); + long elapsed = System.currentTimeMillis() - start; + + Assert.assertTrue("must run the action even when it cannot lock (degrade)", ran.get()); + Assert.assertTrue(result); + Assert.assertTrue("must not steal a fresh foreign lock", Files.exists(lock)); + Assert.assertTrue("must wait out the acquire budget before degrading, was " + elapsed, elapsed >= 150); + }); + } + + @Test + public void testInLockIsMutuallyExclusiveAcrossInstances() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + // two instances over one directory model two concurrent users of one identity; a generous acquire + // budget makes a contender wait rather than degrade, and a large staleness window stops either from + // stealing the other's live lock - so the two critical sections must run strictly one at a time. In a + // single JVM the in-process lock (keyed on the identity) is what serializes them; it stands in for the + // cross-process file lock that only genuinely separate processes would exercise. + FileTokenStore storeA = new FileTokenStore(dir, 10_000, 600_000); + FileTokenStore storeB = new FileTokenStore(dir, 10_000, 600_000); + + AtomicInteger inside = new AtomicInteger(); + AtomicInteger maxInside = new AtomicInteger(); + AtomicInteger overlaps = new AtomicInteger(); + AtomicInteger ran = new AtomicInteger(); + AtomicReference workerError = new AtomicReference<>(); + TokenStore.CriticalSection section = () -> { + int now = inside.incrementAndGet(); + maxInside.accumulateAndGet(now, Math::max); + if (now > 1) { + overlaps.incrementAndGet(); + } + Os.sleep(200); + inside.decrementAndGet(); + ran.incrementAndGet(); + return true; + }; + + // a barrier forces the two threads to genuinely contend, rather than one running and finishing before + // the other starts (which would satisfy the overlap check without ever exercising mutual exclusion) + CyclicBarrier barrier = new CyclicBarrier(2); + Thread tA = new Thread(() -> { + try { + barrier.await(); + storeA.inLock(key, section); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }); + Thread tB = new Thread(() -> { + try { + barrier.await(); + storeB.inLock(key, section); + } catch (Throwable t) { + workerError.compareAndSet(null, t); + } + }); + tA.start(); + tB.start(); + joinOrFail(tA, "contender A"); + joinOrFail(tB, "contender B"); + + // capture a worker throwable on the main thread: without this, a contender that THREW instead of + // waiting would die silently and leave the other holder looking (falsely) like correct exclusion + Assert.assertNull("a contender thread failed instead of running its critical section", workerError.get()); + Assert.assertEquals("both critical sections must have run", 2, ran.get()); + Assert.assertEquals("the two critical sections must never overlap", 0, overlaps.get()); + Assert.assertEquals("at most one holder at a time", 1, maxInside.get()); + }); + } + + @Test + public void testInLockReleaseDoesNotDeleteAStolenLock() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // a tiny staleness window so our own in-progress hold is judged stale and a peer can steal it + FileTokenStore store = new FileTokenStore(dir, 1000, 50); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + + // our critical section outlives the 50ms staleness window; while we are still inside it, a peer + // process judges our lock stale, steals it (deletes and recreates) and writes its own owner stamp. + // releaseLock must verify ownership and leave the peer's live lock intact, not delete it by bare + // path - otherwise a third acquirer could enter alongside the peer, defeating mutual exclusion. + store.inLock(key, () -> { + Os.sleep(120); + try { + Files.deleteIfExists(lock); + Files.write(lock, "peer-owner-stamp".getBytes(StandardCharsets.UTF_8)); + } catch (Exception e) { + throw new RuntimeException(e); + } + return true; + }); + + Assert.assertTrue("releaseLock must not delete a lock a peer has stolen", Files.exists(lock)); + Assert.assertEquals("the peer's lock content must survive our release", + "peer-owner-stamp", new String(Files.readAllBytes(lock), StandardCharsets.UTF_8)); + }); + } + + @Test + public void testInLockReleasesLockWhenActionThrows() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + + RuntimeException boom = new RuntimeException("action failed"); + try { + store.inLock(key, () -> { + Assert.assertTrue("the lock must be held while the action runs", Files.exists(lock)); + throw boom; + }); + Assert.fail("the action's exception must propagate out of inLock"); + } catch (RuntimeException e) { + Assert.assertSame(boom, e); + } + Assert.assertFalse("inLock must release the lock even when the action throws", Files.exists(lock)); + }); + } + + @Test + public void testInLockReleasesLockWhenSectionLeavesThreadInterrupted() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + + // The critical section is a token refresh, and close() breaks a drainer stuck in one by + // interrupting its thread - so inLock's release routinely runs with the flag already set. + // releaseLock reads the lock's owner stamp through a FileChannel, an InterruptibleChannel: + // with the flag set that read throws ClosedByInterruptException, which releaseLock swallows, + // so the lock file survives its whole staleness window (10 minutes by default) while every + // peer degrades to an unserialized refresh - the rotating-refresh-token race the lock exists + // to prevent. Release must therefore be interrupt-neutral. + boolean released = store.inLock(key, () -> { + Assert.assertTrue("the lock must be held while the action runs", Files.exists(lock)); + Thread.currentThread().interrupt(); + return true; + }); + + Assert.assertTrue(released); + try { + Assert.assertFalse("inLock must release the lock even when the section leaves the thread " + + "interrupted", Files.exists(lock)); + Assert.assertTrue("the caller's interrupt must be preserved, not consumed", + Thread.currentThread().isInterrupted()); + } finally { + // never leak the flag into the rest of the suite + Thread.interrupted(); + } + }); + } + + @Test + public void testInLockRunsActionAndManagesLockFile() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(key, () -> { + ran.set(true); + Assert.assertTrue("lock file must exist while the action runs", Files.exists(lock)); + return true; + }); + + Assert.assertTrue(ran.get()); + Assert.assertTrue(result); + Assert.assertFalse("lock file must be released after the action", Files.exists(lock)); + }); + } + + @Test + public void testInLockStealsStaleLock() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // staleness threshold 100ms; the pre-created lock is backdated well past it + FileTokenStore store = new FileTokenStore(dir, 2000, 100); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.createFile(lock); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 10_000)); + + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(key, () -> { + ran.set(true); + return true; + }); + + Assert.assertTrue("must steal the stale lock and run", ran.get()); + Assert.assertTrue(result); + Assert.assertFalse("having acquired the stolen lock, it must be released", Files.exists(lock)); + }); + } + + @Test + public void testLiteralNullStringTokenRoundTrip() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + TokenStoreKey key = sampleKey(); + // a token whose value is exactly the 4 characters "null" must survive the round trip: the writer + // omits absent fields rather than emitting a JSON null, so a present "null" is unambiguous on read + PersistedToken saved = new PersistedToken("null", null, "null", 1_730_000_000_000L, 300_000L); + store.save(key, saved); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals("null", loaded.getAccessToken()); + Assert.assertNull("an absent id token must stay null, not become the string \"null\"", loaded.getIdToken()); + Assert.assertEquals("null", loaded.getRefreshToken()); + }); + } + + @Test + public void testLoadAndSaveSurviveACarriedInterruptFlag() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + + // Every file operation here goes through FileChannel, an InterruptibleChannel: a thread that + // merely CARRIES a set interrupt flag makes the first read or write throw + // ClosedByInterruptException, and the flag survives. Callers arrive that way routinely - an ILP + // producer on a pooled thread where interrupt is the cancellation signal, and the sender's own + // I/O thread, which close() interrupts to break a stuck credential pull. Neither means "abandon + // the token store", so the store must clear the flag around its own I/O and restore it after. + Thread.currentThread().interrupt(); + try { + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertTrue("save must complete with the flag set", Thread.currentThread().isInterrupted()); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull("load must complete with the flag set, not throw on the channel", loaded); + Assert.assertEquals("ACCESS-1", loaded.getAccessToken()); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertTrue("the caller's interrupt must be preserved, not consumed", + Thread.currentThread().isInterrupted()); + } finally { + // never leak the flag into the rest of the suite + Thread.interrupted(); + } + }); + } + + @Test + public void testLoadMissingReturnsNull() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + Assert.assertNull(store.load(sampleKey())); + }); + } + + @Test + public void testLoadStealsAStampedDirectoryLockAbandonedByACrashedWriter() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + // The lock is fresh and stamped, exactly as a process killed after createLockFile() leaves it. + // The default load must outwait the directory lease and reclaim it inside its 3s acquire budget, + // rather than applying the refresh lock's 10-minute stale window and throwing. + Path directoryLock = dir.resolve(".store.lock"); + Files.write(directoryLock, "dead-directory-owner".getBytes(StandardCharsets.UTF_8)); + + PersistedToken loaded = new FileTokenStore(dir).load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertFalse("the abandoned directory lock must be stolen and released", + Files.exists(directoryLock)); + }); + } + + @Test + public void testLoadDiscardsOnlyTheStoresOwnFilesFromAWorldWritableDirectory() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // Discarding every ENTRY in an untrusted directory is right - the sibling test pins it. What the + // discard must not do is decide "entry" means "any .json", because the directory it is emptying + // is one the operator chose and may share. questdb.client.oidc.token.store.dir pointed at an + // existing config directory that happens to be group-writable is enough: one getToken() then + // deletes files the client never wrote. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + // Entries chosen to pin both sides of the shape test: plain and short-hex JSON names must + // survive, while a different full-length fingerprint is indistinguishable from an entry this + // store wrote for another key and must be discarded with the real entry. The foreign temp pins + // the same hash-prefix requirement on the .tmp arm. (No uppercase-hex case: the store renders + // its digests lowercase, but on a case-insensitive filesystem such a name is the same file as + // the real entry, so the assertion would be about the filesystem rather than about the filter.) + Path plainJson = dir.resolve("my-important-settings.json"); + Path shortHexJson = dir.resolve("abc123.json"); + Path otherFingerprintJson = dir.resolve(repeat("a", 64) + ".json"); + Path foreignTemp = dir.resolve("scratch-notes.tmp"); + Assert.assertNotEquals("the full-length fixture must not be the sample key's entry", + tokenFile(dir, key), otherFingerprintJson); + Files.write(plainJson, "{\"keep\":true}".getBytes(StandardCharsets.UTF_8)); + Files.write(shortHexJson, "{\"keep\":true}".getBytes(StandardCharsets.UTF_8)); + Files.write(otherFingerprintJson, "{\"discard\":true}".getBytes(StandardCharsets.UTF_8)); + Files.write(foreignTemp, "keep".getBytes(StandardCharsets.UTF_8)); + + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + + Assert.assertNull("an entry from a directory other local users could write must not be adopted", + store.load(key)); + Assert.assertFalse("the store's own entry is still discarded - that half is unchanged", + Files.exists(tokenFile(dir, key))); + + Assert.assertTrue("a file the store never wrote must survive: " + plainJson.getFileName(), + Files.exists(plainJson)); + Assert.assertTrue("a short hex name is not a 64-char fingerprint: " + shortHexJson.getFileName(), + Files.exists(shortHexJson)); + Assert.assertFalse("a different store-shaped JSON entry must be discarded: " + + otherFingerprintJson.getFileName(), + Files.exists(otherFingerprintJson)); + Assert.assertTrue("a foreign .tmp is not a store write temp: " + foreignTemp.getFileName(), + Files.exists(foreignTemp)); + }); + } + + @Test + public void testLoadRejectsAndDiscardsAnEntryFromAWorldWritableDirectory() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("baseline: an entry written into an owner-only directory is trusted", + store.load(key)); + + // adopt() already rejects an entry carrying ONLY a refresh token, but a COMPLETE plant - a dummy + // access token, the attacker's refresh token, an expiry already in the past - takes the normal + // path and the next silent refresh presents their credential. Closing that needs the container + // checked too: an entry sitting in a directory other local users can WRITE was never ours to + // trust, whatever it contains. + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + + Assert.assertNull("an entry from a directory other local users could write must not be adopted", + store.load(key)); + Assert.assertFalse("the untrusted entry must be discarded, not left for the next load to adopt", + Files.exists(tokenFile(dir, key))); + Assert.assertEquals("load() must tighten the store directory, as the write paths already do", + PosixFilePermissions.fromString("rwx------"), Files.getPosixFilePermissions(dir)); + Assert.assertNull(store.load(key)); + + // the store stays usable: a fresh sign-in persists and loads normally over the tightened directory + store.save(key, sampleToken("ACCESS-2", "REFRESH-2")); + PersistedToken reloaded = store.load(key); + Assert.assertNotNull(reloaded); + Assert.assertEquals("REFRESH-2", reloaded.getRefreshToken()); + }); + } + + @Test + public void testWorldWritableVerdictIsNotConsumedByWhicheverIdentityLoadsFirst() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // restrictToOwner reads the permissions, chmods to 0700, and returns the verdict it computed + // BEFORE the chmod - so the verdict is destroyed by the act of reporting it. One store directory + // holds one file per configuration, and identity A's load tightens the directory for everybody: + // by the time identity B loads, the directory is 0700, B's verdict is "trusted", and B adopts + // whatever .json happens to be sitting there. A discarded its own entry and left B's. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey a = sampleKey(); + TokenStoreKey b = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid profile", null, false); + Assert.assertNotEquals("the two identities must address different files", a.hash(), b.hash()); + + store.save(a, sampleToken("ACCESS-A", "REFRESH-A")); + store.save(b, sampleToken("ACCESS-B", "REFRESH-B")); + + // the window: while this stands, any local user can replace either entry + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + byte[] planted = Files.readAllBytes(tokenFile(dir, b)); + Files.write(tokenFile(dir, b), + new String(planted, StandardCharsets.UTF_8) + .replace("REFRESH-B", "REFRESH-PLANTED") + .getBytes(StandardCharsets.UTF_8)); + + // A loads first and correctly refuses - and tightens the directory on the way through + Assert.assertNull("A must refuse an entry from a world-writable directory", store.load(a)); + Assert.assertEquals("A's load tightens the directory for every later caller", + PosixFilePermissions.fromString("rwx------"), Files.getPosixFilePermissions(dir)); + + // B now loads over a directory that LOOKS owner-only, because A made it so + Assert.assertNull("B must not adopt an entry that was exposed in the same window, merely " + + "because A's load already spent the directory's untrusted verdict", store.load(b)); + Assert.assertFalse("and the exposed entry must be discarded, not left for the next load", + Files.exists(tokenFile(dir, b))); + + // the store stays usable for both identities over the tightened directory + store.save(a, sampleToken("ACCESS-A2", "REFRESH-A2")); + store.save(b, sampleToken("ACCESS-B2", "REFRESH-B2")); + Assert.assertEquals("REFRESH-A2", store.load(a).getRefreshToken()); + Assert.assertEquals("REFRESH-B2", store.load(b).getRefreshToken()); + }); + } + + @Test + public void testWorldWritableVerdictSurvivesASaveTouchingTheDirectoryFirst() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // Exercise the same distrust verdict through a WRITE path. A save arriving before any load must + // tighten and sweep the directory before publishing its own token; otherwise planted entries can + // look as though they had always been protected. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey a = sampleKey(); + TokenStoreKey b = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid profile", null, false); + store.save(b, sampleToken("ACCESS-B", "REFRESH-B")); + + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + byte[] planted = Files.readAllBytes(tokenFile(dir, b)); + Files.write(tokenFile(dir, b), + new String(planted, StandardCharsets.UTF_8) + .replace("REFRESH-B", "REFRESH-PLANTED") + .getBytes(StandardCharsets.UTF_8)); + + // a save for an unrelated identity is the first thing to touch the directory + store.save(a, sampleToken("ACCESS-A", "REFRESH-A")); + Assert.assertEquals("the save tightens the directory, as it always did", + PosixFilePermissions.fromString("rwx------"), Files.getPosixFilePermissions(dir)); + + Assert.assertNull("an entry exposed before that save must not be adopted afterwards", + store.load(b)); + }); + } + + @Test + public void testAnUntrustedSentinelKeepsAnOwnerOnlyDirectoryDistrusted() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The invariant the concurrent fix rests on, pinned directly: while the .untrusted sentinel is + // present, the directory is distrusted whatever its permission bits say. It reconstructs, without + // threads, the state a second caller observes mid-race - a peer detected the world-writable + // directory, dropped the sentinel and chmodded to 0700, but has not yet swept - an owner-only + // directory that still holds a valid-looking entry AND the sentinel. A loader that trusts on the + // bits alone adopts the entry; it must not. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("baseline: an entry in an owner-only directory is trusted", store.load(key)); + Assert.assertEquals("baseline: the directory is owner-only", OWNER_ONLY_DIR_PERMS, + Files.getPosixFilePermissions(dir)); + + // mark untrusted while leaving the permissions owner-only: the "tightened but not yet swept" state + Path sentinel = dir.resolve(".untrusted"); + Files.write(sentinel, new byte[0]); + + Assert.assertNull("a marked directory must be distrusted even while it looks owner-only", + store.load(key)); + Assert.assertFalse("the entry under a marked directory must be discarded", + Files.exists(tokenFile(dir, key))); + Assert.assertFalse("a complete sweep must clear the sentinel", Files.exists(sentinel)); + + // recovery: with the sentinel gone the store trusts the owner-only directory again + store.save(key, sampleToken("ACCESS-2", "REFRESH-2")); + PersistedToken reloaded = store.load(key); + Assert.assertNotNull(reloaded); + Assert.assertEquals("REFRESH-2", reloaded.getRefreshToken()); + }); + } + + @Test + public void testConcurrentLoadWaitsForTheUntrustedDirectorySweep() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The concurrent case the sibling sequential verdict tests cannot reach. restrictToOwner tightens + // the directory to 0700 and returns the verdict in one breath, but the planted entries are not + // swept until discardUntrustedDirectoryContents runs afterwards. A SECOND caller - another thread + // or process - must wait on the directory recovery lock until that sweep is complete. Otherwise a + // loader can adopt a plant in the chmod-to-sweep gap, and a saver can write a fresh entry which the + // first caller's later directory-wide sweep silently deletes. + // + // beforeUntrustedDiscardHook drops us into exactly that gap: it fires after the tighten-and-mark, + // before the sweep and while the directory lock is held. From inside it a fresh store (no hook) + // starts loading the OTHER identity and must block until the first caller has swept. No real + // concurrency can force a peer into this sub-syscall gap deterministically, which is why the seam + // exists (as beforeCaptureHook does for stealIfStale). + Path dir = storeDir(); + FileTokenStore first = new FileTokenStore(dir); + FileTokenStore second = new FileTokenStore(dir); + TokenStoreKey a = sampleKey(); + TokenStoreKey b = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid profile", null, false); + Assert.assertNotEquals("the two identities must address different files", a.hash(), b.hash()); + + first.save(a, sampleToken("ACCESS-A", "REFRESH-A")); + first.save(b, sampleToken("ACCESS-B", "REFRESH-B")); + + // the window: while this stands, any local user can replace either entry + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + byte[] planted = Files.readAllBytes(tokenFile(dir, b)); + Files.write(tokenFile(dir, b), + new String(planted, StandardCharsets.UTF_8) + .replace("REFRESH-B", "REFRESH-PLANTED") + .getBytes(StandardCharsets.UTF_8)); + + AtomicReference secondSaw = new AtomicReference<>(); + AtomicReference secondFailure = new AtomicReference<>(); + AtomicReference> permsInGap = new AtomicReference<>(); + AtomicBoolean plantPresentInGap = new AtomicBoolean(); + AtomicBoolean secondReturned = new AtomicBoolean(); + AtomicReference secondThread = new AtomicReference<>(); + Field hookField = FileTokenStore.class.getDeclaredField("beforeUntrustedDiscardHook"); + hookField.setAccessible(true); + hookField.set(first, (Runnable) () -> { + try { + permsInGap.set(Files.getPosixFilePermissions(dir)); + } catch (IOException e) { + throw new RuntimeException(e); + } + plantPresentInGap.set(Files.exists(tokenFile(dir, b))); + Assert.assertTrue("the cross-process directory lock must cover the whole chmod-to-sweep gap", + Files.isRegularFile(dir.resolve(".store.lock"), LinkOption.NOFOLLOW_LINKS)); + Thread loader = new Thread(() -> { + try { + secondSaw.set(second.load(b)); + secondReturned.set(true); + } catch (Throwable t) { + secondFailure.compareAndSet(null, t); + } + }, "concurrent-untrusted-loader"); + loader.setDaemon(true); + secondThread.set(loader); + loader.start(); + try { + awaitWaitingInside(loader, "withDirectoryLock"); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + Assert.assertFalse("the second load must not return while the first caller still owns the " + + "directory recovery lock", secondReturned.get()); + }); + + // A refuses its own identity AND, through the hook, parks the concurrent B at the gap + Assert.assertNull("A must refuse an entry from a world-writable directory", first.load(a)); + joinOrFail(secondThread.get(), "the concurrent loader"); + + Assert.assertEquals("the directory was already tightened to 0700 when B observed it", + OWNER_ONLY_DIR_PERMS, permsInGap.get()); + Assert.assertTrue("the plant was still present when B observed it - the sweep had not run yet", + plantPresentInGap.get()); + Assert.assertNull("the concurrent load failed instead of waiting for the sweep: " + secondFailure.get(), + secondFailure.get()); + Assert.assertTrue("the concurrent load must finish after the sweep releases the directory lock", + secondReturned.get()); + Assert.assertNull("B must not adopt the plant after it is allowed through", secondSaw.get()); + Assert.assertFalse("the plant must be discarded, not left for the next load", + Files.exists(tokenFile(dir, b))); + + // the store recovers over the tightened directory: the sentinel is gone and both identities work + first.save(a, sampleToken("ACCESS-A2", "REFRESH-A2")); + first.save(b, sampleToken("ACCESS-B2", "REFRESH-B2")); + Assert.assertEquals("REFRESH-A2", first.load(a).getRefreshToken()); + Assert.assertEquals("REFRESH-B2", first.load(b).getRefreshToken()); + }); + } + + @Test + public void testConcurrentSaveWaitsForTheUntrustedDirectorySweepAndSurvives() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // Regression for the lost-persistence interleaving: A detects a world-writable directory, marks + // and tightens it, then pauses before its directory-wide sweep. B saves another identity and + // returns. Without one lock covering BOTH the sweep and the write, A resumes and deletes B's + // completed file; a headless process then cannot resume after restart and needs a human sign-in. + Path dir = storeDir(); + FileTokenStore first = new FileTokenStore(dir); + FileTokenStore second = new FileTokenStore(dir); + TokenStoreKey a = sampleKey(); + TokenStoreKey b = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid profile", null, false); + Assert.assertNotEquals("the two identities must address different files", a.hash(), b.hash()); + + first.save(a, sampleToken("ACCESS-A", "REFRESH-A")); + first.save(b, sampleToken("ACCESS-B", "REFRESH-B")); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + + AtomicBoolean saveReturned = new AtomicBoolean(); + AtomicReference saveFailure = new AtomicReference<>(); + AtomicReference saveThread = new AtomicReference<>(); + Field hookField = FileTokenStore.class.getDeclaredField("beforeUntrustedDiscardHook"); + hookField.setAccessible(true); + hookField.set(first, (Runnable) () -> { + Assert.assertTrue("the cross-process directory lock must remain held through the sweep", + Files.isRegularFile(dir.resolve(".store.lock"), LinkOption.NOFOLLOW_LINKS)); + Thread saver = new Thread(() -> { + try { + second.save(b, sampleToken("ACCESS-B2", "REFRESH-B2")); + saveReturned.set(true); + } catch (Throwable t) { + saveFailure.compareAndSet(null, t); + } + }, "concurrent-untrusted-saver"); + saver.setDaemon(true); + saveThread.set(saver); + saver.start(); + try { + awaitWaitingInside(saver, "withDirectoryLock"); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + Assert.assertFalse("save must not report success while an earlier distrust sweep can still " + + "delete its file", saveReturned.get()); + }); + + Assert.assertNull("A must refuse an entry from a world-writable directory", first.load(a)); + joinOrFail(saveThread.get(), "the concurrent saver"); + + Assert.assertNull("the concurrent save failed instead of waiting for recovery: " + saveFailure.get(), + saveFailure.get()); + Assert.assertTrue("the save must complete once the directory sweep releases the lock", + saveReturned.get()); + PersistedToken reloaded = new FileTokenStore(dir).load(b); + Assert.assertNotNull("a save that returned successfully must survive the earlier sweep", reloaded); + Assert.assertEquals("REFRESH-B2", reloaded.getRefreshToken()); + }); + } + + @Test + public void testADanglingSymlinkAtTheSentinelNameKeepsTheDirectoryDistrusted() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The sibling test plants the sentinel as a REGULAR FILE, which is the only shape this store + // writes - and so the only shape it ever proved the distrust through. The party the sentinel + // defends against is the one who can write this directory, and they choose the shape. A DANGLING + // symlink at the name reports absent to any link-following test, so a verdict built on + // Files.exists reads "no sentinel" and trusts the directory on its permission bits alone - the + // mechanism disabled outright, with no race to win and nothing in any log. The exclusive-create + // mark cannot displace it either: O_CREAT|O_EXCL answers EEXIST for a symlink exactly as it does + // for a peer's mark, so markUntrusted reads the squatter as "already marked" and returns happy. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-PLANT")); + Assert.assertNotNull("baseline: an entry in an owner-only directory is trusted", store.load(key)); + + Path sentinel = dir.resolve(".untrusted"); + Files.createSymbolicLink(sentinel, dir.resolve("no-such-target")); + Assert.assertTrue("the fixture must be a symlink", Files.isSymbolicLink(sentinel)); + Assert.assertFalse("the fixture must DANGLE - that is what a link-following test misreads", + Files.exists(sentinel)); + + Assert.assertNull("a symlink standing at the sentinel's name must distrust the directory just as " + + "a regular file does; a link-following presence test hands an attacker who can " + + "write this directory a way to switch the sentinel off", + store.load(key)); + Assert.assertFalse("the entry under a distrusted directory must be discarded", + Files.exists(tokenFile(dir, key))); + }); + } + + @Test + public void testMarkUntrustedDisplacesASymlinkSquattingTheSentinelName() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The other half. Distrusting THROUGH a squatter (the sibling test) keeps THIS caller safe, but + // the sentinel's job is to carry the verdict to a CONCURRENT one across the chmod. That peer + // reads the name itself, so the name has to end up holding a mark rather than the attacker's + // symlink. markUntrusted must therefore tell a peer's mark from a squatter - which the + // FileAlreadyExistsException alone cannot do - and displace the squatter. + // + // beforeUntrustedDiscardHook drops us into the chmod-to-sweep gap, the same seam and the same + // window as testConcurrentLoadWaitsForTheUntrustedDirectorySweep, and asserts what a + // peer arriving there would find at the name. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + Path sentinel = dir.resolve(".untrusted"); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + Files.createSymbolicLink(sentinel, dir.resolve("no-such-target")); + + AtomicBoolean regularFileInGap = new AtomicBoolean(); + AtomicBoolean stillASymlinkInGap = new AtomicBoolean(); + Field hookField = FileTokenStore.class.getDeclaredField("beforeUntrustedDiscardHook"); + hookField.setAccessible(true); + hookField.set(store, (Runnable) () -> { + regularFileInGap.set(Files.isRegularFile(sentinel, LinkOption.NOFOLLOW_LINKS)); + stillASymlinkInGap.set(Files.isSymbolicLink(sentinel)); + }); + + Assert.assertNull("the world-writable directory must be distrusted", store.load(key)); + + Assert.assertFalse("markUntrusted must not leave the attacker's symlink standing at the name: a " + + "peer reading it in this gap follows it, finds nothing, and trusts the plant", + stillASymlinkInGap.get()); + Assert.assertTrue("the sentinel name must hold a real mark - a regular file - once markUntrusted " + + "has run, so a concurrent caller in the chmod-to-sweep gap distrusts", + regularFileInGap.get()); + Assert.assertFalse("a complete sweep must still clear the mark it wrote", + Files.exists(sentinel, LinkOption.NOFOLLOW_LINKS)); + }); + } + + @Test + public void testAnUnliftableUntrustedMarkNamesItselfRatherThanKillingPersistenceInSilence() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The sentinel is meant to be transient: mark, sweep, clear. Its retention on a failed sweep is + // deliberate and fail-closed ("the next caller re-sweeps"), but that reasoning assumes the + // failure goes away. When it does not, nothing else can lift the mark - the sweep skips the + // sentinel by design and markUntrusted only runs while the directory is still other-writable - + // so restrictToOwner distrusts on every later call. load() then returns null over an entry + // save() has just written, for good, and the only thing that ever said so was a warning about + // the directory not being owner-only, which is a different condition with a different fix. + // + // A non-empty directory squatting the sentinel's name reaches that state with one mkdir and + // survives the chmod that ends the attacker's write access: deleteIfExists cannot remove it, so + // the clear fails on every pass. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("baseline: the store works before the name is squatted", store.load(key)); + + Path sentinel = dir.resolve(".untrusted"); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxrwxrwx")); + Files.createDirectory(sentinel); + Files.write(sentinel.resolve("occupant"), new byte[]{1}); + + Field latch = FileTokenStore.class.getDeclaredField("warnedStuckUntrustedSentinel"); + latch.setAccessible(true); + ((AtomicBoolean) latch.get(null)).set(false); + + ch.qos.logback.classic.Logger storeLogger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(FileTokenStore.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = storeLogger.getLevel(); + storeLogger.setLevel(ch.qos.logback.classic.Level.ALL); + storeLogger.addAppender(appender); + try { + // three full rounds: the mark cannot be lifted, so every one of them refuses + for (int i = 0; i < 3; i++) { + store.save(key, sampleToken("ACCESS-" + i, "REFRESH-" + i)); + Assert.assertNull("a mark that cannot be lifted keeps the directory distrusted", + store.load(key)); + } + } finally { + storeLogger.detachAppender(appender); + storeLogger.setLevel(savedLevel); + appender.stop(); + } + + Assert.assertTrue("the squatter must still be standing - that is what makes the state permanent", + Files.isDirectory(sentinel, LinkOption.NOFOLLOW_LINKS)); + + int stuck = 0; + for (ch.qos.logback.classic.spi.ILoggingEvent e : appender.list) { + String msg = e.getFormattedMessage(); + if (msg.contains("marked untrusted and the mark cannot be lifted") + && msg.contains("no token will be persisted or read") + && msg.contains(".untrusted")) { + stuck++; + } + } + Assert.assertEquals("a permanently latched store must say so, naming the entry to remove: " + + "persistence that merely stops working leaves an operator with a headless " + + "producer that re-prompts every restart and nothing to act on. Warned " + + "exactly once per JVM, like its siblings, because load() is on the flush path", + 1, stuck); + }); + } + + @Test + public void testLoadThrowsRatherThanReportsEmptyWhenTheDirectoryIsUnusable() throws Exception { + assertMemoryLeak(() -> { + // Same fixture as testInLockDegradesWhenDirectoryUnusable: a regular file standing where the + // store directory's parent must be makes directory preparation throw IOException. That fault is + // TRANSIENT in the field - a home directory not mounted yet, EIO/ESTALE on an NFS home, a + // momentarily read-only or full filesystem. + Path blocker = temp.getRoot().toPath().resolve("blocker"); + Files.write(blocker, new byte[]{1}); + FileTokenStore store = new FileTokenStore(blocker.resolve("oidc-tokens")); + + try { + PersistedToken token = store.load(sampleKey()); + Assert.fail("load must not report a definitive empty store for a transient directory fault; " + + "returned " + token); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), + expected.getMessage().contains("could not prepare the OIDC token store directory")); + } + // Why the distinction is not cosmetic: null is load()'s DEFINITIVE answer. OidcDeviceAuth + // latches storeLoadAttempted on it and never reads the store again for the life of the + // instance, so a momentary mount fault at the first getToken() would send a process that owns a + // good refresh token back through the interactive device flow - a hard failure for the headless + // consumer this persistence exists to serve. A throw is retried under the store-load back-off. + // save() already lets this same exception propagate; only load() disagreed. + }); + } + + @Test + public void testTransientLoadFailureDoesNotSuppressUntrustedDirectoryWarning() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to expose the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Field ioLatchField = FileTokenStore.class.getDeclaredField("warnedStoreDirIoFailure"); + ioLatchField.setAccessible(true); + AtomicBoolean ioLatch = (AtomicBoolean) ioLatchField.get(null); + Field securityLatchField = FileTokenStore.class.getDeclaredField("warnedUnprotectedStoreDir"); + securityLatchField.setAccessible(true); + AtomicBoolean securityLatch = (AtomicBoolean) securityLatchField.get(null); + boolean ioLatchWasSet = ioLatch.getAndSet(false); + boolean securityLatchWasSet = securityLatch.getAndSet(false); + + ch.qos.logback.classic.Logger storeLogger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(FileTokenStore.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = storeLogger.getLevel(); + storeLogger.setLevel(ch.qos.logback.classic.Level.ALL); + storeLogger.addAppender(appender); + try { + Path blocker = temp.getRoot().toPath().resolve("warning-blocker"); + Files.write(blocker, new byte[]{1}); + FileTokenStore unavailableStore = new FileTokenStore(blocker.resolve("oidc-tokens")); + try { + unavailableStore.load(sampleKey()); + Assert.fail("the transient directory failure must still propagate"); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), + expected.getMessage().contains("could not prepare the OIDC token store directory")); + } + + Path exposedDir = temp.newFolder("exposed-store").toPath(); + FileTokenStore exposedStore = new FileTokenStore(exposedDir); + TokenStoreKey key = sampleKey(); + exposedStore.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Files.setPosixFilePermissions(exposedDir, PosixFilePermissions.fromString("rwxrwxrwx")); + Assert.assertNull("contents found after other-user write access must be discarded", + exposedStore.load(key)); + } finally { + storeLogger.detachAppender(appender); + storeLogger.setLevel(savedLevel); + appender.stop(); + ioLatch.set(ioLatchWasSet); + securityLatch.set(securityLatchWasSet); + } + + int ioWarnings = 0; + int securityWarnings = 0; + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + String message = event.getFormattedMessage(); + if (message.contains("could not prepare the OIDC token store directory")) { + ioWarnings++; + Assert.assertTrue("the transient warning must carry the filesystem cause: " + message, + message.contains("warning-blocker")); + Assert.assertFalse("a filesystem fault must not be misreported as a permissions finding: " + + message, + message.contains("not owner-only")); + } + if (message.contains("the OIDC token store directory is not owner-only")) { + securityWarnings++; + Assert.assertTrue("the security warning must explain the discarded credentials: " + message, + message.contains("every entry found in it was discarded")); + } + } + Assert.assertEquals("the transient I/O condition has its own one-shot warning", 1, ioWarnings); + Assert.assertEquals("the transient warning must not consume the security warning's latch", + 1, securityWarnings); + }); + } + + @Test + public void testOwnerOnlyPermissionSubsetsAreNotWidenedTo0700() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to inspect and preserve strict directory modes", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + Files.createDirectories(dir); + FileTokenStore store = new FileTokenStore(dir); + Method restrictToOwner = FileTokenStore.class.getDeclaredMethod("restrictToOwner"); + restrictToOwner.setAccessible(true); + try { + for (String mode : new String[]{"r-x------", "rw-------"}) { + Set expected = PosixFilePermissions.fromString(mode); + Files.setPosixFilePermissions(dir, expected); + restrictToOwner.invoke(store); + Assert.assertEquals("an already-owner-only " + mode + + " directory must not gain missing owner permissions", + expected, Files.getPosixFilePermissions(dir)); + } + } finally { + // Leave the temporary-folder rule enough access to delete the directory tree. + Files.setPosixFilePermissions(dir, OWNER_ONLY_DIR_PERMS); + } + }); + } + + @Test + public void testTightenWarningIsLatchedOnlyAfterChmodSucceeds() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to force and observe the chmod", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Field latchField = FileTokenStore.class.getDeclaredField("warnedTightenedStoreDir"); + latchField.setAccessible(true); + AtomicBoolean latch = (AtomicBoolean) latchField.get(null); + boolean latchWasSet = latch.getAndSet(false); + + Path dir = storeDir(); + Files.createDirectories(dir); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxr-xr-x")); + FileTokenStore store = new FileTokenStore(dir); + Method restrictToOwner = FileTokenStore.class.getDeclaredMethod("restrictToOwner"); + restrictToOwner.setAccessible(true); + Field hookField = FileTokenStore.class.getDeclaredField("beforeDirectoryTightenHook"); + hookField.setAccessible(true); + + ch.qos.logback.classic.Logger storeLogger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(FileTokenStore.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = storeLogger.getLevel(); + storeLogger.setLevel(ch.qos.logback.classic.Level.ALL); + storeLogger.addAppender(appender); + try { + // Remove the empty directory after restrictToOwner has statted it but before the real chmod. + // setPosixFilePermissions then throws NoSuchFileException, deterministically exercising the + // syscall-failure ordering without a synthetic FileSystemProvider. + hookField.set(store, (Runnable) () -> { + try { + Files.delete(dir); + } catch (IOException e) { + throw new AssertionError("could not remove the directory before chmod", e); + } + }); + try { + restrictToOwner.invoke(store); + Assert.fail("chmod on the removed directory must fail"); + } catch (InvocationTargetException e) { + Assert.assertTrue("the real chmod failure must propagate, got: " + e.getCause(), + e.getCause() instanceof NoSuchFileException); + } + Assert.assertFalse("a failed chmod must not consume the one-time success-warning latch", + latch.get()); + Assert.assertEquals("a failed chmod must not announce a completed tighten", + 0, countTightenWarnings(appender)); + + hookField.set(store, null); + Files.createDirectories(dir); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxr-xr-x")); + restrictToOwner.invoke(store); + + Assert.assertEquals("the later directory must genuinely be tightened", + OWNER_ONLY_DIR_PERMS, Files.getPosixFilePermissions(dir)); + Assert.assertTrue("a successful chmod must consume the warning latch", latch.get()); + Assert.assertEquals("the successful tighten after a failure must still be announced once", + 1, countTightenWarnings(appender)); + } finally { + hookField.set(store, null); + storeLogger.detachAppender(appender); + storeLogger.setLevel(savedLevel); + appender.stop(); + latch.set(latchWasSet); + if (!Files.exists(dir, LinkOption.NOFOLLOW_LINKS)) { + Files.createDirectories(dir); + } + Files.setPosixFilePermissions(dir, OWNER_ONLY_DIR_PERMS); + } + }); + } + + @Test + public void testLoadTrustsAWorldREADABLEDirectory() throws Exception { + Assume.assumeTrue("POSIX permissions are needed to loosen the store directory", + FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + // The 0755 a default umask produces is NOT the attack surface: no other user can create or + // replace a file in it, and the entry itself is 0600. Distrusting it would discard honest tokens + // - and make every negative assertion in this suite pass for the wrong reason. + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Files.setPosixFilePermissions(dir, PosixFilePermissions.fromString("rwxr-xr-x")); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull("a merely world-READABLE directory must not invalidate its entry", loaded); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertEquals("and it is still tightened on the way through", + PosixFilePermissions.fromString("rwx------"), Files.getPosixFilePermissions(dir)); + }); + } + + @Test + public void testLockFilePermissionsOwnerOnly() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + // the lock file is created owner-only too: it briefly records an owner stamp and sits beside the + // 0600 token file, so it must not widen the directory's exposure. Assert while the lock is held; inLock + // deletes it on return and propagates a thrown AssertionError after releasing it. + store.inLock(key, () -> { + try { + Assert.assertEquals("the lock file must be owner-only (0600)", + PosixFilePermissions.fromString("rw-------"), Files.getPosixFilePermissions(lock)); + } catch (java.io.IOException e) { + throw new AssertionError(e); + } + return true; + }); + }); + } + + @Test + public void testLongFieldsSerializeAsDigitsNotBareNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // the two long fields are present, non-nullable integers, so Long.MIN_VALUE must serialize as its + // digits. serialize() reserves an omitted member (not a bare null) for an absent value, so a null + // here would be indistinguishable from absent and breaks the frozen cross-language contract; the + // reader would also round-trip that null back to 0 (parseLongOrZero), a silent corruption. + store.save(key, new PersistedToken("ACCESS-1", null, "REFRESH-1", Long.MIN_VALUE, Long.MIN_VALUE)); + + String json = new String(Files.readAllBytes(tokenFile(dir, key)), StandardCharsets.UTF_8); + Assert.assertTrue("expires_at_millis must be written as digits, not a bare null [json=" + json + ']', + json.contains("\"expires_at_millis\":-9223372036854775808")); + Assert.assertTrue("token_ttl_millis must be written as digits, not a bare null [json=" + json + ']', + json.contains("\"token_ttl_millis\":-9223372036854775808")); + + // and the extreme value round-trips verbatim rather than collapsing to 0 on read + PersistedToken loaded = store.load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals(Long.MIN_VALUE, loaded.getExpiresAtMillis()); + Assert.assertEquals(Long.MIN_VALUE, loaded.getTokenTtlMillis()); + }); + } + + @Test + public void testNoLeftoverTempFileAfterSave() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + store.save(key, sampleToken("ACCESS-2", "REFRESH-2")); // overwrite + + File[] files = dir.toFile().listFiles(); + Assert.assertNotNull(files); + int jsonCount = 0; + for (File f : files) { + Assert.assertFalse("leftover temp file: " + f.getName(), f.getName().endsWith(".tmp")); + if (f.getName().endsWith(".json")) { + jsonCount++; + } + } + Assert.assertEquals(1, jsonCount); + }); + } + + @Test + public void testOutOfContractNumberIsRejectedNotQuietlyParsed() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Path file = tokenFile(dir, key); + String json = new String(Files.readAllBytes(file), StandardCharsets.UTF_8); + + // QuestDB's Numbers.parseLong accepts an 'L' suffix and '_' thousands separators. JSON allows + // neither, and neither does the frozen cross-language format - so "1L" is a schema version only + // THIS client can read, and accepting it would let a file diverge silently from every other + // language client sharing the directory. It must read as unusable instead. + String tampered = json.replace("\"v\":1,", "\"v\":1L,"); + Assert.assertNotEquals("the fixture must actually have been tampered with", json, tampered); + Files.write(file, tampered.getBytes(StandardCharsets.UTF_8)); + + Assert.assertNull("a number only this client's parser accepts must not satisfy the schema gate", + store.load(key)); + }); + } + + @Test + public void testOversizedFileReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // Baseline: a normal, fingerprint-matching token loads. This proves the oversized file below is + // rejected by the size cap ALONE, not by a fingerprint/version/parse mismatch (the flaw in the old + // all-spaces file, which parsed to version 0 and would return null even with the cap removed). + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("a normal valid token must load", store.load(key)); + // A valid, fingerprint-matching token whose two large fields push the FILE past MAX_FILE_BYTES + // (1 MiB) while each field stays under the per-value lexer limit (also 1 MiB), so without the size + // cap this parses and loads. readBounded caps on channel.size() before reading, so the oversized + // file is rejected up front - the real point of the cap (avoid an unbounded read / OOM on an + // attacker-grown file), which the guard now demonstrably enforces. + char[] big = new char[600_000]; + Arrays.fill(big, 'a'); + String bigField = new String(big); + store.save(key, sampleToken(bigField, bigField)); + Assert.assertTrue("the test file must exceed the 1 MiB size cap to isolate it", + Files.size(tokenFile(dir, key)) > (1 << 20)); + Assert.assertNull("an oversized but otherwise valid, fingerprint-matching file must be rejected by the size cap", + store.load(key)); + }); + } + + @Test + public void testOversizedStaleLockIsStolen() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // Stale window 60s, lock backdated only 10s: a STAMPED (readable) lock this fresh would NOT be + // stolen (10s < 60s). So the steal below can only happen because the oversized lock reads as + // unreadable/null via MAX_LOCK_FILE_BYTES and is stolen on the shorter empty-lock grace (5s < 10s). + // This isolates the read cap: remove it and readLockHolder reads the 64 KiB as a live stamp -> the + // lock is judged fresh, not stolen, acquisition degrades lock-free, and the lock file is NOT + // released, failing the Files.exists assertion below. The old 100ms window stole on staleness + // regardless of the cap, so the cap was untested. + FileTokenStore store = new FileTokenStore(dir, 2000, 60_000); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + // a corrupt/hostile lock far larger than the read cap. The steal reads the owner stamp with a hard + // cap (not Files.readAllBytes, which on an attacker-grown lock could OutOfMemoryError on the refresh + // path): a bounded read reports an oversized lock as unreadable, which the steal treats as abandoned + // junk - it must still acquire, not wedge. + byte[] huge = new byte[64 * 1024]; + Arrays.fill(huge, (byte) 'x'); + Files.write(lock, huge); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 10_000)); + + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(key, () -> { + ran.set(true); + return true; + }); + + Assert.assertTrue("an oversized stale lock must be stolen, not wedge acquisition", ran.get()); + Assert.assertTrue(result); + Assert.assertFalse("the acquired (stolen) lock must be released", Files.exists(lock)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testPerFieldFingerprintMismatchReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey saved = sampleKey(); + store.save(saved, sampleToken("ACCESS-1", "REFRESH-1")); + byte[] bytes = Files.readAllBytes(tokenFile(dir, saved)); + + // each key differs from the saved fingerprint in exactly one field; writing the saved bytes under + // the differing key's file name isolates the in-file fingerprint re-check (the file is found, but + // its recorded identity does not match), so a hash collision or a copied file never serves another + // identity's token. groups_in_token (id-token vs access-token credential) and audience (the + // distinct nullableEquals path) are the riskiest fields. + TokenStoreKey[] mismatches = { + new TokenStoreKey("questdb", "https://idp.example.com:443/OTHER-token", + "https://idp.example.com:443/device", "openid", null, false), + new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/OTHER-device", "openid", null, false), + new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid groups", null, false), + new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", "api://other", false), + new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, true), + }; + for (TokenStoreKey other : mismatches) { + Files.write(tokenFile(dir, other), bytes); + Assert.assertNull("a fingerprint mismatch must be rejected: " + other.hash(), store.load(other)); + } + }); + } + + @Test + public void testPermissionsOwnerOnly() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertEquals(PosixFilePermissions.fromString("rw-------"), + Files.getPosixFilePermissions(tokenFile(dir, key))); + Assert.assertEquals(PosixFilePermissions.fromString("rwx------"), + Files.getPosixFilePermissions(dir)); + }); + } + + @Test + public void testSaveDoesNotDegradeWhenTheDirectoryLockIsHeld() throws Exception { + assertMemoryLeak(() -> { + // The per-identity refresh lock deliberately degrades after its acquire budget, but applying that + // policy to .store.lock reopens the lost-save race: a writer can return while a peer still holds an + // old distrust verdict and later sweeps its file. Directory coordination must fail the best-effort + // persistence call instead of making a false success claim. + Path dir = storeDir(); + createStoreDir(dir); + Path directoryLock = dir.resolve(".store.lock"); + byte[] peerStamp = "live-directory-owner".getBytes(StandardCharsets.UTF_8); + Files.write(directoryLock, peerStamp); + FileTokenStore store = new FileTokenStore(dir, 200, 60_000); + TokenStoreKey key = sampleKey(); + + try { + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.fail("save must not run without the required directory lock"); + } catch (OidcAuthException expected) { + // wrapped as the same best-effort persistence failure OidcDeviceAuth already handles + } + + Assert.assertFalse("a timed-out save must not write a token outside the directory lock", + Files.exists(tokenFile(dir, key))); + Assert.assertArrayEquals("the live peer's directory lock must be left untouched", + peerStamp, Files.readAllBytes(directoryLock)); + }); + } + + @Test + public void testSaveDoesNotUseTheShortDirectoryLeaseWhileARecoveryIsPending() throws Exception { + assertMemoryLeak(() -> { + // A short lease is safe only after every distrust sweep is complete. If an old sweeper were + // displaced and later resumed, it could delete a new holder's completed save. Keep the configured + // 60s stale window while the sentinel stands even though this fake lock is older than the trusted + // directory lease (2s). + Path dir = storeDir(); + createStoreDir(dir); + Files.createFile(dir.resolve(".untrusted")); + Path directoryLock = dir.resolve(".store.lock"); + byte[] peerStamp = "recovering-directory-owner".getBytes(StandardCharsets.UTF_8); + Files.write(directoryLock, peerStamp); + Files.setLastModifiedTime(directoryLock, FileTime.fromMillis(System.currentTimeMillis() - 3_000)); + FileTokenStore store = new FileTokenStore(dir, 200, 60_000); + TokenStoreKey key = sampleKey(); + + try { + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.fail("save must not displace a pending directory recovery"); + } catch (OidcAuthException expected) { + // The required lock fails closed until the recovery holder releases it or reaches 60s stale. + } + + Assert.assertFalse("a timed-out save must not write while recovery is pending", + Files.exists(tokenFile(dir, key))); + Assert.assertArrayEquals("the recovering peer's directory lock must be left untouched", + peerStamp, Files.readAllBytes(directoryLock)); + }); + } + + @Test + public void testSaveFailureLeavesNoTempFileAndThrows() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // make the atomic rename fail: the target path already exists as a NON-EMPTY directory, which a + // file-over-directory replace cannot overwrite, driving save() down its IOException path + Path target = tokenFile(dir, key); + Files.createDirectories(target); + Files.createFile(target.resolve("blocker")); + + try { + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.fail("save must throw when it cannot replace the target"); + } catch (OidcAuthException expected) { + // the write-temp / flush / atomic-rename protocol must surface a wrapped failure, never a raw + // IOException, and never a half-written credential + } + + // the temp file is the durability point of the protocol; a failed save must clean it up rather than + // leave a *.tmp credential fragment behind + boolean hasTmp; + try (java.nio.file.DirectoryStream entries = Files.newDirectoryStream(dir, "*.tmp")) { + hasTmp = entries.iterator().hasNext(); + } + Assert.assertFalse("a failed save must not leave a *.tmp file behind", hasTmp); + }); + } + + @Test + public void testSaveThenLoadRoundTrip() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + TokenStoreKey key = sampleKey(); + PersistedToken saved = new PersistedToken("ACCESS-1", "ID-1", "REFRESH-1", 1_730_000_000_000L, 300_000L); + store.save(key, saved); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull(loaded); + Assert.assertEquals("ACCESS-1", loaded.getAccessToken()); + Assert.assertEquals("ID-1", loaded.getIdToken()); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertEquals(1_730_000_000_000L, loaded.getExpiresAtMillis()); + Assert.assertEquals(300_000L, loaded.getTokenTtlMillis()); + }); + } + + @Test + public void testSchemaVersionMismatchReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + createStoreDir(dir); + // a future schema version with an otherwise-matching fingerprint must be ignored, not served: the + // version is the forward-compat guard the frozen cross-language contract rests on + String v2 = "{\"v\":2,\"client_id\":\"questdb\"," + + "\"token_endpoint\":\"https://idp.example.com:443/token\"," + + "\"device_authorization_endpoint\":\"https://idp.example.com:443/device\"," + + "\"scope\":\"openid\",\"groups_in_token\":false," + + "\"access_token\":\"ACCESS-1\",\"refresh_token\":\"REFRESH-1\"," + + "\"expires_at_millis\":1730000000000,\"token_ttl_millis\":300000}"; + Files.write(tokenFile(dir, key), v2.getBytes(StandardCharsets.UTF_8)); + Assert.assertNull("a future schema version must be rejected", store.load(key)); + + // sanity: the identical body at the live version IS accepted, proving the rejection is the version + // and not a malformed document + Files.write(tokenFile(dir, key), v2.replace("\"v\":2", "\"v\":1").getBytes(StandardCharsets.UTF_8)); + Assert.assertNotNull("the same body at the live schema version must load", store.load(key)); + }); + } + + @Test + public void testSpecialCharactersAndNullsRoundTrip() throws Exception { + assertMemoryLeak(() -> { + FileTokenStore store = new FileTokenStore(storeDir()); + // a non-null audience that needs JSON escaping, and null access/id tokens + TokenStoreKey key = new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid groups", "api://q\"uote\\slash", true); + PersistedToken saved = new PersistedToken(null, null, "REFRESH-\t-1", 42L, 60_000L); + store.save(key, saved); + + PersistedToken loaded = store.load(key); + Assert.assertNotNull(loaded); + Assert.assertNull(loaded.getAccessToken()); + Assert.assertNull(loaded.getIdToken()); + Assert.assertEquals("REFRESH-\t-1", loaded.getRefreshToken()); + Assert.assertEquals(42L, loaded.getExpiresAtMillis()); + }); + } + + @Test + public void testStaleTempFilesAreSweptOnSave() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + // 1s staleness window so the test does not have to wait + FileTokenStore store = new FileTokenStore(dir, 3_000, 1_000); + TokenStoreKey key = sampleKey(); + // an orphan temp left by a crashed save (backdated past the staleness window) must be reaped on the + // next save; a fresh temp (recent mtime - a concurrent writer's) must be left untouched + Path staleTmp = dir.resolve(key.hash() + "stale.tmp"); + Files.createFile(staleTmp); + Files.setLastModifiedTime(staleTmp, FileTime.fromMillis(System.currentTimeMillis() - 10_000)); + Path freshTmp = dir.resolve(key.hash() + "fresh.tmp"); + Files.createFile(freshTmp); + + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + Assert.assertFalse("a stale orphan temp must be swept on save", Files.exists(staleTmp)); + Assert.assertTrue("a fresh temp (a concurrent writer's) must not be swept", Files.exists(freshTmp)); + Assert.assertNotNull("the save must still succeed", store.load(key)); + }); + } + + @Test + public void testStealIfStaleRestoresALockAPeerRecreatedInTheCaptureGap() throws Exception { + assertMemoryLeak(() -> { + // The arm that ran only in production. stealIfStale judges a lock stale, captures it with an + // ATOMIC_MOVE, then re-reads the capture to confirm it took the stamp it judged rather than a + // LIVE lock a peer recreated in the gap between those two steps. Its own comment says a bare + // deleteIfExists(lock) "would admit two holders at once", yet replacing the whole + // capture/verify/restore with exactly that left the suite green: the two + // testRestoreCapturedLock* cases drive restoreCapturedLock DIRECTLY by reflection, so they + // pass unchanged when nothing calls it. + // + // Reaching the arm needs the peer to land inside that gap, which no amount of concurrency can + // force deterministically -- hence beforeCaptureHook, which runs there and nowhere else. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + byte[] peerLive = "peer-owner-stamp".getBytes(StandardCharsets.UTF_8); + Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000)); + + FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000); + Field hookField = FileTokenStore.class.getDeclaredField("beforeCaptureHook"); + hookField.setAccessible(true); + // In the gap: the abandoned lock is replaced by a peer's freshly-created live one. + hookField.set(store, (Runnable) () -> { + try { + Files.write(lock, peerLive); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis())); + } catch (IOException e) { + throw new RuntimeException(e); + } + }); + + Method stealIfStale = FileTokenStore.class.getDeclaredMethod("stealIfStale", Path.class); + stealIfStale.setAccessible(true); + stealIfStale.invoke(store, lock); + + Assert.assertTrue("the peer's LIVE lock must survive the capture gap; removing it admits two " + + "holders at once, which is the double-POST of one rotating refresh token that a " + + "reuse-detecting provider answers by revoking the whole family", Files.exists(lock)); + Assert.assertArrayEquals("the peer's lock must go back byte for byte, or releaseLock's " + + "owner-stamp check refuses to delete it and the peer wedges every later acquire", + peerLive, Files.readAllBytes(lock)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testStealIfStaleRestoresALockRenewedInTheCaptureGap() throws Exception { + assertMemoryLeak(() -> { + // A directory-lock heartbeat preserves the owner stamp and changes only the mtime. Simulate a + // renewal after the age check but before capture: comparing only owner bytes would misclassify the + // captured live lock as unchanged and delete it, admitting a second holder. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + byte[] owner = "live-owner-stamp".getBytes(StandardCharsets.UTF_8); + Files.write(lock, owner); + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000)); + + FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000); + Field hookField = FileTokenStore.class.getDeclaredField("beforeCaptureHook"); + hookField.setAccessible(true); + hookField.set(store, (Runnable) () -> { + try { + Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis())); + } catch (IOException e) { + throw new RuntimeException(e); + } + }); + + Method stealIfStale = FileTokenStore.class.getDeclaredMethod("stealIfStale", Path.class); + stealIfStale.setAccessible(true); + stealIfStale.invoke(store, lock); + + Assert.assertTrue("a renewed live lock must be restored after capture", Files.exists(lock)); + Assert.assertArrayEquals("renewal must not alter the owner stamp", owner, Files.readAllBytes(lock)); + Assert.assertTrue("the restored lock must retain the renewal timestamp", + System.currentTimeMillis() - Files.getLastModifiedTime(lock).toMillis() < 5_000); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testADirectorySquattingTheDirectoryLockNameIsDisplaced() throws Exception { + assertSquattedDirectoryLockIsReclaimed("a directory", Files::createDirectory); + } + + @Test + public void testADanglingSymlinkSquattingTheDirectoryLockNameIsDisplaced() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + assertSquattedDirectoryLockIsReclaimed("a dangling symlink", + lock -> Files.createSymbolicLink(lock, lock.resolveSibling("no-such-target"))); + } + + @Test + public void testAnUnreadableStaleLockIsStolenRatherThanWedgingPersistence() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + Assume.assumeFalse("root reads every file, so an unreadable lock cannot be modelled here", + "root".equals(System.getProperty("user.name"))); + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + // The shape a run under a DIFFERENT uid leaves when it is killed holding the lock: correctly + // shaped and correctly named, but with a mode that denies this uid the stamp read. No attacker + // needed - a sudo -E start or a re-mapped container uid produces it. Unlike a squatter it is + // genuinely ambiguous (it may be a live holder's), so it is aged rather than displaced on sight; + // backdate it well past both staleness windows. + Path directoryLock = dir.resolve(".store.lock"); + // Backdate BEFORE the chmod: setLastModifiedTime opens the file, so it cannot touch a mode-000 one + // even when we own it. + Files.write(directoryLock, "another-uids-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(directoryLock, FileTime.fromMillis(System.currentTimeMillis() - 3_600_000L)); + Files.setPosixFilePermissions(directoryLock, PosixFilePermissions.fromString("---------")); + + PersistedToken loaded = new FileTokenStore(dir).load(key); + Assert.assertNotNull("an abandoned unreadable directory lock must not wedge persistence", loaded); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertFalse("the stale unreadable lock must be stolen and released", + Files.exists(directoryLock, LinkOption.NOFOLLOW_LINKS)); + }); + } + + @Test + public void testAnUnreadableFreshLockIsNotStolen() throws Exception { + Assume.assumeTrue(FileSystems.getDefault().supportedFileAttributeViews().contains("posix")); + Assume.assumeFalse("root reads every file, so an unreadable lock cannot be modelled here", + "root".equals(System.getProperty("user.name"))); + assertMemoryLeak(() -> { + // The safety property the ageing buys, and the reason an unreadable lock is not displaced the way + // a directory or a symlink is: another uid's LIVE lock is equally unreadable to us, and only its + // age tells it apart from the abandoned one above. Stealing it would admit two holders and + // double-POST one rotating refresh token. + Path dir = storeDir(); + createStoreDir(dir); + TokenStoreKey key = sampleKey(); + Path lock = lockFile(dir, key); + Files.write(lock, "live-owner-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setPosixFilePermissions(lock, PosixFilePermissions.fromString("---------")); + + FileTokenStore store = new FileTokenStore(dir, 3_000, 60_000); + Method stealIfStale = FileTokenStore.class.getDeclaredMethod("stealIfStale", Path.class); + stealIfStale.setAccessible(true); + stealIfStale.invoke(store, lock); + + Assert.assertTrue("a fresh unreadable lock may be a live holder's and must not be stolen", + Files.exists(lock, LinkOption.NOFOLLOW_LINKS)); + assertNoCaptureTempFiles(dir, key); + }); + } + + @Test + public void testSweepDoesNotDeleteStealCapturedLock() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + // a 1s staleness window so an old file is well past it + FileTokenStore store = new FileTokenStore(dir, 3_000, 1_000); + TokenStoreKey key = sampleKey(); + store.save(key, sampleToken("ACCESS-0", "REFRESH-0")); // create the directory + // a steal in another process captures a stale lock by atomically renaming it to + // .lock..tmp; ATOMIC_MOVE preserves the stale lock's old mtime onto the capture. Even + // though that name matches the *.tmp write-temp glob and is past the staleness window, the + // save's temp-sweep must NOT delete it - it is a live cross-process steal in progress, and deleting + // it would destroy a lock the stealer may be about to restore to its live owner. + Path capture = dir.resolve(key.hash() + ".lock." + java.util.UUID.randomUUID() + ".tmp"); + Files.write(capture, "stale-owner-stamp".getBytes(StandardCharsets.UTF_8)); + Files.setLastModifiedTime(capture, FileTime.fromMillis(System.currentTimeMillis() - 10_000)); + + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); // runs sweepStaleTempFiles + + Assert.assertTrue("the temp-sweep must not delete an in-flight steal-captured lock", Files.exists(capture)); + }); + } + + @Test + public void testTokenStoreKeyRejectsNullRequiredFields() throws Exception { + assertMemoryLeak(() -> { + // the identity fields are required; a null must fail fast with a clear OidcAuthException rather than + // surface later as a raw NullPointerException inside save()/load() (audience stays optional) + try { + new TokenStoreKey(null, "https://idp/token", "https://idp/device", "openid", null, false); + Assert.fail("a null clientId must be rejected"); + } catch (OidcAuthException expected) { + // required identity field + } + try { + new TokenStoreKey("questdb", null, "https://idp/device", "openid", null, false); + Assert.fail("a null tokenEndpoint must be rejected"); + } catch (OidcAuthException expected) { + // required identity field + } + try { + new TokenStoreKey("questdb", "https://idp/token", null, "openid", null, false); + Assert.fail("a null deviceAuthorizationEndpoint must be rejected"); + } catch (OidcAuthException expected) { + // required identity field + } + try { + new TokenStoreKey("questdb", "https://idp/token", "https://idp/device", null, null, false); + Assert.fail("a null scope must be rejected"); + } catch (OidcAuthException expected) { + // required identity field + } + }); + } + + @Test + public void testTruncatedJsonReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + createStoreDir(dir); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // a crash mid-write on a filesystem without atomic rename, or a torn read, can leave a valid JSON + // prefix cut off before the closing brace. parseLast() must reject the truncated document rather + // than serve a half-parsed credential + Files.write(tokenFile(dir, key), + "{\"v\":1,\"client_id\":\"questdb\"".getBytes(StandardCharsets.UTF_8)); + Assert.assertNull("a truncated-but-prefix-valid file must be ignored", store.load(key)); + }); + } + + @Test + public void testVersionOverflowReturnsNull() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore store = new FileTokenStore(dir); + TokenStoreKey key = sampleKey(); + // a tampered version that narrows to SCHEMA_VERSION when cast to int (1 + 2^32) must not pass the + // schema gate: the parser keeps the version as a long and compares it as a long + store.save(key, sampleToken("ACCESS-1", "REFRESH-1")); + Assert.assertNotNull("the valid entry must load", store.load(key)); + byte[] valid = Files.readAllBytes(tokenFile(dir, key)); + String tampered = new String(valid, StandardCharsets.UTF_8).replace("\"v\":1", "\"v\":4294967297"); + Files.write(tokenFile(dir, key), tampered.getBytes(StandardCharsets.UTF_8)); + Assert.assertNull("a version that truncates to 1 as an int must be rejected", store.load(key)); + }); + } + + private static int countTightenWarnings( + ch.qos.logback.core.read.ListAppender appender + ) { + int warnings = 0; + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + if (event.getFormattedMessage().contains( + "the OIDC token store directory was not owner-only and has been tightened to 0700")) { + warnings++; + } + } + return warnings; + } + + private static TokenStoreKey sampleKey() { + return new TokenStoreKey("questdb", "https://idp.example.com:443/token", + "https://idp.example.com:443/device", "openid", null, false); + } + + private static PersistedToken sampleToken(String access, String refresh) { + return new PersistedToken(access, null, refresh, System.currentTimeMillis() + 300_000L, 300_000L); + } + + private void assertNoCaptureTempFiles(Path dir, TokenStoreKey key) throws Exception { + // a successful steal deletes its atomic-capture file and a restore moves it back; neither must leak a + // .lock..tmp behind (an orphan is otherwise only reclaimed by a later save's sweep) + try (DirectoryStream stream = Files.newDirectoryStream(dir, key.hash() + "*.tmp")) { + for (Path p : stream) { + Assert.fail("a steal must not leak a capture temp file: " + p.getFileName()); + } + } + } + + /** + * A shape this store never writes, sitting on the required {@code .store.lock} name, must be displaced on + * sight rather than waited out. {@code CREATE_NEW} reports EEXIST for a directory and for a symlink + * exactly as it does for a peer's live lock, so acquireLock spends its whole budget and throws - and + * throws again on every later call, since nothing reclaims that name (it carries no hash prefix, so the + * untrusted sweep skips it, and it is not a {@code *.tmp}). Persistence then dies silently: load() and + * save() both fail, OidcDeviceAuth degrades to "continuing without persistence", and every process start + * re-runs the interactive device flow - a hard failure for the headless consumer this feature exists for. + */ + private void assertSquattedDirectoryLockIsReclaimed(String shape, LockPlanter planter) throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + TokenStoreKey key = sampleKey(); + new FileTokenStore(dir).save(key, sampleToken("ACCESS-1", "REFRESH-1")); + + Path directoryLock = dir.resolve(".store.lock"); + planter.plant(directoryLock); + Assert.assertFalse("the fixture must leave a NON-regular shape at the lock name [" + shape + ']', + Files.isRegularFile(directoryLock, LinkOption.NOFOLLOW_LINKS)); + + // On the FIRST call, not after a staleness window: a squatter is not a lock and no wait turns it + // into one. A dangling symlink would never age out at all, since NOFOLLOW stats the link and the + // link is exactly as young as the planting above. + PersistedToken loaded = new FileTokenStore(dir).load(key); + Assert.assertNotNull("persistence must survive " + shape + " squatting the directory lock", loaded); + Assert.assertEquals("REFRESH-1", loaded.getRefreshToken()); + Assert.assertFalse("the squatter must be gone from the lock name [" + shape + ']', + Files.exists(directoryLock, LinkOption.NOFOLLOW_LINKS)); + + // and the store is fully usable afterwards, not merely readable once + new FileTokenStore(dir).save(key, sampleToken("ACCESS-2", "REFRESH-2")); + Assert.assertEquals("REFRESH-2", new FileTokenStore(dir).load(key).getRefreshToken()); + }); + } + + @FunctionalInterface + private interface LockPlanter { + void plant(Path lock) throws IOException; + } + + private static void awaitInside(Thread t, String method) throws InterruptedException { + // poll the thread's own stack for the named FileTokenStore frame: the only evidence that a helper + // thread has actually ENTERED the call, as opposed to having been scheduled at all + final long deadline = System.currentTimeMillis() + 10_000; + while (System.currentTimeMillis() < deadline) { + for (StackTraceElement frame : t.getStackTrace()) { + if (FileTokenStore.class.getName().equals(frame.getClassName()) + && method.equals(frame.getMethodName())) { + return; + } + } + Thread.sleep(5); + } + Assert.fail("the waiter never entered FileTokenStore." + method + " [state=" + t.getState() + ']'); + } + + private static void awaitWaitingInside(Thread t, String method) throws InterruptedException { + // Stronger than awaitInside: prove the helper has reached the named store operation AND is parked + // there, which distinguishes real directory-lock exclusion from a thread that was merely scheduled. + final long deadline = System.currentTimeMillis() + 10_000; + while (System.currentTimeMillis() < deadline) { + boolean isInside = false; + for (StackTraceElement frame : t.getStackTrace()) { + if (FileTokenStore.class.getName().equals(frame.getClassName()) + && method.equals(frame.getMethodName())) { + isInside = true; + break; + } + } + final Thread.State state = t.getState(); + if (isInside && (state == Thread.State.WAITING || state == Thread.State.TIMED_WAITING)) { + return; + } + Thread.sleep(5); + } + Assert.fail("the waiter never blocked inside FileTokenStore." + method + " [state=" + t.getState() + ']'); + } + + /** + * Creates the store directory owner-only, the way {@code FileTokenStore} itself creates it - NOT the + * way the JVM's umask happens to. + *

+ * A fixture that calls {@code Files.createDirectories(dir)} bare inherits the umask, so on a host with + * a group-writable one (002, the default on the Linux CI agents) the directory arrives {@code + * rwxrwxr-x}. {@code load()} then reads it as a directory another local user could have planted an + * entry in, discards the entry and returns null BEFORE it opens the file - which fails every test whose + * first assertion is that a valid entry loads, and, far worse, silently satisfies every test asserting + * that some malformed entry does NOT load. Those pass for the wrong reason: proved by feeding + * {@code testCorruptFileReturnsNull} a perfectly valid document, which fails the test at umask 022 and + * passes it at 002. + *

+ * A test that wants a loose directory sets the permissions itself right after this call; that is an + * explicit statement rather than a property of whoever ran the build. + */ + private static void createStoreDir(Path dir) throws Exception { + if (!FileSystems.getDefault().supportedFileAttributeViews().contains("posix")) { + Files.createDirectories(dir); // Windows: no POSIX bits to set, and load() trusts it either way + return; + } + Files.createDirectories(dir, PosixFilePermissions.asFileAttribute(OWNER_ONLY_DIR_PERMS)); + // createDirectories applies the attribute only to directories it actually creates, so assert + // rather than assume: a fixture that silently reverted to the umask must not go unnoticed again. + Assert.assertEquals("the fixture must not leave the store directory at the mercy of the umask", + OWNER_ONLY_DIR_PERMS, Files.getPosixFilePermissions(dir)); + } + + private static void joinOrFail(Thread t, String what) throws InterruptedException { + // never a bare join(): a contender that wedges on a lock it should have degraded out of would hang + // the suite until the 20-minute surefire timeout, reported as an opaque stall with no failing + // assertion. Bounded, then asserted, so the wedge fails as itself. + t.join(30_000); + Assert.assertFalse(what + " did not finish within 30s [state=" + t.getState() + ']', t.isAlive()); + } + + private Path lockFile(Path dir, TokenStoreKey key) { + return dir.resolve(key.hash() + ".lock"); + } + + private String readLockStamp(Path lock) { + // the owner nonce a live holder stamped into the lock, or null when there is no lock file at all. An + // IO error here is a harness fault, so it fails loudly rather than reading as "no lock" + try { + return Files.exists(lock) ? new String(Files.readAllBytes(lock), StandardCharsets.UTF_8) : null; + } catch (IOException e) { + throw new AssertionError("could not read the lock stamp: " + lock, e); + } + } + + private Path storeDir() { + // a non-existent subdirectory so the store creates it (and we can assert its permissions) + return temp.getRoot().toPath().resolve("oidc-tokens"); + } + + private Path tokenFile(Path dir, TokenStoreKey key) { + return dir.resolve(key.hash() + ".json"); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServer.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServer.java new file mode 100644 index 000000000..33977fd8f --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServer.java @@ -0,0 +1,501 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.std.str.StringSink; + +import java.io.Closeable; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.net.SocketException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.atomic.AtomicReference; + +/** + * A minimal HTTP/1.1 server for tests that impersonates an OIDC identity provider (and, + * when needed, the QuestDB {@code /settings} endpoint). It speaks just enough HTTP to drive + * {@link io.questdb.client.cutlass.auth.OidcDeviceAuth}: it reads a request, hands the path + * and body to a {@link Handler}, and writes back a {@code Content-Length}-framed response on + * a keep-alive connection. + */ +public class MockOidcServer implements Closeable { + private final Thread acceptThread; + private final List connSockets = Collections.synchronizedList(new ArrayList<>()); + private final List connThreads = Collections.synchronizedList(new ArrayList<>()); + private final Handler handler; + // The FIRST throwable a Handler raised on a daemon connection thread (typically an assertion inside a + // handler). handleConnection captures it here rather than letting it die on that thread as a mere + // transport drop the client may swallow; close() resurfaces it on the test thread. + private final AtomicReference handlerError = new AtomicReference<>(); + private final List requestAuthHeaders = Collections.synchronizedList(new ArrayList<>()); + private final ServerSocket serverSocket; + + public MockOidcServer(Handler handler) throws IOException { + this.handler = handler; + this.serverSocket = new ServerSocket(0, 50, InetAddress.getLoopbackAddress()); + this.acceptThread = new Thread(this::acceptLoop, "mock-oidc-accept"); + this.acceptThread.setDaemon(true); + this.acceptThread.start(); + } + + public static MockResponse chunkedJson(int status, String body) { + return new MockResponse(status, body, true); + } + + public static MockResponse dropConnection() { + // close the connection without responding, so the client sees a transport failure (connection + // reset / EOF) on this request - used to simulate an unreachable endpoint that is co-located with + // a working one on the same mock origin + MockResponse response = new MockResponse(0, "", false); + response.dropConnection = true; + return response; + } + + public static MockResponse json(int status, String body) { + return new MockResponse(status, body, false); + } + + public static MockResponse oversizedJson(long bodyBytes) { + // stream a chunked body larger than the client's response-size cap (MAX_RESPONSE_BODY_BYTES), so the + // bounded read aborts on the cap instead of letting a hostile or MITM'd server stream an endless body + // and wedge the thread. The payload is all whitespace, which the JSON lexer skips, so the byte cap is + // what trips - not a parse error, and not the lexer's per-value length limit + MockResponse response = new MockResponse(200, "", true); + response.oversizedBodyBytes = bodyBytes; + return response; + } + + public static MockResponse raw(String rawResponse) { + // write the supplied bytes verbatim as the whole HTTP response, so a test can craft a malformed + // status line (for example a status code carrying control bytes) that the int-typed status factories + // cannot express + MockResponse response = new MockResponse(0, "", false); + response.rawResponse = rawResponse; + return response; + } + + public static MockResponse dribble() { + return dribble(200); + } + + /** + * A dribbled chunked body under an arbitrary status, so a test can drive the response-body read bound + * on the ERROR path (where the status is the verdict and the body is only detail) as well as on the + * success path. + */ + public static MockResponse dribble(int status) { + MockResponse response = new MockResponse(status, "", true); + response.dribble = true; + return response; + } + + /** + * Dribbles the response HEAD - the status line and headers - one byte at a time and never terminates it, + * so the client never reaches a complete header block. The body-dribbling {@link #dribble()} above cannot + * reach this: a client only starts reading a body once the head has parsed, so the head read is a + * separate bound with a separate loop ({@code ResponseHeaders.await}, not {@code Response.recv}). + */ + public static MockResponse dribbleHead() { + MockResponse response = new MockResponse(200, "", true); + response.dribbleHead = true; + return response; + } + + public static MockResponse stall() { + MockResponse response = new MockResponse(200, "", true); + response.stall = true; + return response; + } + + @Override + public void close() throws IOException { + // tear the server down deterministically so a test's threads are gone before its assertions (and + // assertMemoryLeak's native-memory check) run, instead of lingering as daemon threads that can + // perturb a later test: stop accepting, drop every connection (which unblocks a handler reading a + // socket), then interrupt and join the accept and connection threads (interrupt wakes a stalled + // handler that is sleeping on the response body) + serverSocket.close(); + synchronized (connSockets) { + for (Socket s : connSockets) { + try { + s.close(); + } catch (IOException ignore) { + // already closed + } + } + } + interruptAndJoin(acceptThread); + synchronized (connThreads) { + for (Thread t : connThreads) { + interruptAndJoin(t); + } + } + // Resurface a failure a Handler raised on its daemon connection thread (see handleConnection): it is + // otherwise visible only as a transport drop the client may swallow, turning a broken assertion into a + // green test. Teardown ran first, so nothing leaks; rethrow after it so a test whose body otherwise + // passed still fails with the real cause (and one whose body already failed gets it as a suppressed + // exception on the primary failure). + Throwable handlerFailure = handlerError.get(); + if (handlerFailure != null) { + if (handlerFailure instanceof Error) { + throw (Error) handlerFailure; + } + if (handlerFailure instanceof RuntimeException) { + throw (RuntimeException) handlerFailure; + } + throw new RuntimeException(handlerFailure); + } + } + + /** + * How many TCP connections this server has accepted since it started. + *

+ * The discriminator for "did the client drop the connection?", which no other observable exposes: a + * keep-alive reuse and a reconnect send the same bytes and produce the same request count, and this + * mock never sends {@code Connection: close}, so the client's own choice is the only thing that moves + * this number. {@code acceptLoop} records a socket before it starts the thread that answers on it, so + * a client that has read a response head is guaranteed to see that connection counted here. + * + * @return the number of accepted connections + */ + public int connectionsAccepted() { + return connSockets.size(); + } + + public String httpUrl(String path) { + return "http://127.0.0.1:" + port() + path; + } + + public int port() { + return serverSocket.getLocalPort(); + } + + public List requestAuthHeaders() { + return requestAuthHeaders; + } + + private static void interruptAndJoin(Thread t) { + t.interrupt(); + try { + t.join(5_000); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + private static String readLine(InputStream in) throws IOException { + StringSink sb = new StringSink(); + boolean any = false; + int c; + while ((c = in.read()) != -1) { + any = true; + if (c == '\r') { + continue; + } + if (c == '\n') { + return sb.toString(); + } + sb.put((char) c); + } + return any ? sb.toString() : null; + } + + private static Request readRequest(InputStream in) throws IOException { + String requestLine = readLine(in); + if (requestLine == null || requestLine.isEmpty()) { + return null; + } + String[] parts = requestLine.split(" "); + String method = parts[0]; + String path = parts.length > 1 ? parts[1] : ""; + int contentLength = 0; + String authorization = null; + String line; + while ((line = readLine(in)) != null && !line.isEmpty()) { + int idx = line.indexOf(':'); + if (idx > 0) { + String name = line.substring(0, idx).trim(); + if ("content-length".equalsIgnoreCase(name)) { + contentLength = Integer.parseInt(line.substring(idx + 1).trim()); + } else if ("authorization".equalsIgnoreCase(name)) { + authorization = line.substring(idx + 1).trim(); + } + } + } + String body = ""; + if (contentLength > 0) { + byte[] buf = new byte[contentLength]; + int read = 0; + while (read < contentLength) { + int n = in.read(buf, read, contentLength - read); + if (n < 0) { + break; + } + read += n; + } + body = new String(buf, 0, read, StandardCharsets.UTF_8); + } + return new Request(method, path, body, authorization); + } + + private static String reason(int status) { + switch (status) { + case 200: + return "OK"; + case 400: + return "Bad Request"; + case 401: + return "Unauthorized"; + case 403: + return "Forbidden"; + case 404: + return "Not Found"; + default: + return "Status"; + } + } + + private static void writeChunked(OutputStream out, byte[] body) throws IOException { + // split into small chunks so a multi-KB value spans several, exercising the chunked decoder + final int chunkSize = 64; + for (int off = 0; off < body.length; off += chunkSize) { + int len = Math.min(chunkSize, body.length - off); + out.write((Integer.toHexString(len) + "\r\n").getBytes(StandardCharsets.US_ASCII)); + out.write(body, off, len); + out.write("\r\n".getBytes(StandardCharsets.US_ASCII)); + } + out.write("0\r\n\r\n".getBytes(StandardCharsets.US_ASCII)); // terminal chunk + } + + private static void writeOversized(OutputStream out, long bodyBytes) throws IOException { + // chunked body of the requested size, all whitespace after the opening brace so the JSON lexer keeps + // consuming (no per-value limit) until the client trips its response-size cap. The client aborts and + // closes the connection mid-stream once the cap is crossed, so tolerate the write failing under us + out.write("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n".getBytes(StandardCharsets.US_ASCII)); + final int chunkLen = 64 * 1024; + final byte[] chunk = new byte[chunkLen]; + Arrays.fill(chunk, (byte) ' '); + chunk[0] = '{'; // open an object once; the rest is whitespace, an unterminated body the cap cuts short + final byte[] crlf = "\r\n".getBytes(StandardCharsets.US_ASCII); + try { + long remaining = bodyBytes; + while (remaining > 0) { + final int len = (int) Math.min(chunkLen, remaining); + out.write((Integer.toHexString(len) + "\r\n").getBytes(StandardCharsets.US_ASCII)); + out.write(chunk, 0, len); + out.write(crlf); + chunk[0] = ' '; // only the first chunk opens the object; the rest is pure whitespace + remaining -= len; + } + out.write("0\r\n\r\n".getBytes(StandardCharsets.US_ASCII)); + out.flush(); + } catch (IOException ignore) { + // expected: the client aborts on its response-size cap mid-stream and closes the connection + } + } + + private static void writeResponse(OutputStream out, MockResponse response) throws IOException { + if (response.rawResponse != null) { + out.write(response.rawResponse.getBytes(StandardCharsets.US_ASCII)); + out.flush(); + return; + } + if (response.oversizedBodyBytes > 0) { + writeOversized(out, response.oversizedBodyBytes); + return; + } + if (response.stall) { + // send chunked headers then block without sending the body, so the client must abort on its + // own configured deadline rather than wedging on the HttpClient default timeout + out.write("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n".getBytes(StandardCharsets.US_ASCII)); + out.flush(); + try { + Thread.sleep(30_000); + } catch (InterruptedException ignore) { + } + return; + } + if (response.dribbleHead) { + // Dribble the response HEAD one byte at a time and never terminate it: no blank line ever + // arrives, so the header parser stays incomplete and ResponseHeaders.await keeps looping. Each + // read makes progress within its budget, so a client that re-armed a per-read timeout would run + // for (bytes x timeout) - long enough that the @Test timeout fires instead - while one bounding + // the WHOLE call aborts on its own deadline. Header bytes only, so nothing here can be mistaken + // for a body. Stop once the client aborts and closes the socket (the write throws). + // Well-formed throughout - a status line followed by endlessly repeated padding headers - so the + // client aborts on its deadline rather than on a parse error, which would prove nothing about + // the bound. The blank line that would end the head is never sent. + final StringBuilder head = new StringBuilder("HTTP/1.1 200 OK\r\n"); + for (int i = 0; i < 400; i++) { + head.append("X-Pad-").append(i).append(": pad\r\n"); + } + final byte[] headBytes = head.toString().getBytes(StandardCharsets.US_ASCII); + try { + for (byte headByte : headBytes) { + out.write(headByte); + out.flush(); + Thread.sleep(50); + } + } catch (IOException | InterruptedException ignore) { + // the client aborted on its whole-read deadline and closed the socket + } + return; + } + if (response.dribble) { + // send chunked headers, then dribble the chunk-size LINE one hex digit at a time (never the + // terminating CRLF), so the client's single recv() keeps looping on the incomplete line while + // wall-clock accumulates. This exercises the WHOLE-read timeout bound, not the per-read one: each + // recvOrDie makes progress (gets a byte) within its shrinking budget, so a client that only re-armed + // a per-read timeout would loop forever, while one bounding the whole read aborts on its deadline. + // A modest digit count keeps the accumulated chunk size within a long. Stop once the client aborts + // and closes the socket (the write throws). The status is whatever the test asked dribble() + // for, so the same dribble drives the success path and the error path. + out.write(("HTTP/1.1 " + response.status + " STATUS\r\nContent-Type: application/json\r\n" + + "Transfer-Encoding: chunked\r\n\r\n").getBytes(StandardCharsets.US_ASCII)); + out.flush(); + try { + for (int i = 0; i < 100; i++) { + // leading-zero hex digits of a never-terminated chunk-size line: the parsed size stays 0 + // (so nothing overflows) while the line never completes, keeping recv() looping + out.write('0'); + out.flush(); + Thread.sleep(100); + } + } catch (IOException | InterruptedException ignore) { + // the client aborted on its whole-read deadline and closed the socket + } + return; + } + byte[] bodyBytes = response.body.getBytes(StandardCharsets.UTF_8); + StringSink head = new StringSink(); + head.put("HTTP/1.1 ").put(response.status).put(' ').put(reason(response.status)).put("\r\n"); + head.put("Content-Type: application/json\r\n"); + if (response.chunked) { + head.put("Transfer-Encoding: chunked\r\n"); + head.put("\r\n"); + out.write(head.toString().getBytes(StandardCharsets.US_ASCII)); + writeChunked(out, bodyBytes); + } else { + head.put("Content-Length: ").put(bodyBytes.length).put("\r\n"); + head.put("\r\n"); + out.write(head.toString().getBytes(StandardCharsets.US_ASCII)); + out.write(bodyBytes); + } + out.flush(); + } + + private void acceptLoop() { + while (!serverSocket.isClosed()) { + try { + Socket socket = serverSocket.accept(); + connSockets.add(socket); + Thread connThread = new Thread(() -> handleConnection(socket), "mock-oidc-conn"); + connThread.setDaemon(true); + connThreads.add(connThread); + connThread.start(); + } catch (IOException e) { + // server socket closed, stop accepting + return; + } + } + } + + private void handleConnection(Socket socket) { + try (InputStream in = socket.getInputStream(); OutputStream out = socket.getOutputStream()) { + Request request; + while ((request = readRequest(in)) != null) { + requestAuthHeaders.add(request.authorization); + MockResponse response; + try { + response = handler.handle(request.method, request.path, request.body); + } catch (Throwable t) { + // The Handler runs on this daemon connection thread, where an uncaught throwable - an + // assertion inside a handler, most often - is otherwise swallowed: the client sees only a + // transport drop it may tolerate (a silent false pass) or retry into an opaque @Test + // timeout. Capture the FIRST such failure so close() can resurface it on the test thread, + // then drop the connection exactly as a return would, leaving client-visible behaviour + // unchanged. + handlerError.compareAndSet(null, t); + return; + } + if (response.dropConnection) { + // returning closes the socket (try-with-resources on its streams), so the client's + // in-flight read fails with a transport error + return; + } + writeResponse(out, response); + } + } catch (SocketException e) { + // client closed the connection, expected + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + @FunctionalInterface + public interface Handler { + MockResponse handle(String method, String path, String body); + } + + public static class MockResponse { + final String body; + final boolean chunked; + final int status; + boolean dribble; + boolean dribbleHead; + boolean dropConnection; + long oversizedBodyBytes; + String rawResponse; + boolean stall; + + MockResponse(int status, String body, boolean chunked) { + this.status = status; + this.body = body; + this.chunked = chunked; + } + } + + public static class Request { + final String authorization; + final String body; + final String method; + final String path; + + Request(String method, String path, String body, String authorization) { + this.method = method; + this.path = path; + this.body = body; + this.authorization = authorization; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServerTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServerTest.java new file mode 100644 index 000000000..f2b29c777 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/MockOidcServerTest.java @@ -0,0 +1,117 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import org.junit.Assert; +import org.junit.Test; + +import java.util.List; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Self-tests for {@link MockOidcServer}, the harness the OIDC suites assert through. + *

+ * Its load-bearing property is not that it serves JSON - every OIDC test would fail loudly if it did not - + * but that a {@link MockOidcServer.Handler} failure REACHES THE TEST THREAD. A handler runs on a daemon + * connection thread, where an uncaught throwable is otherwise invisible: the client sees a dropped + * connection, which most of these tests tolerate as one more transport failure, so a broken assertion inside + * a handler reads as a passing test. {@code handleConnection} captures the first such throwable and + * {@code close()} rethrows it, and that path had no test of its own - every suite depended on it while + * nothing proved it worked. + */ +public class MockOidcServerTest { + + private static final String DEVICE_PATH = "/device"; + + @Test(timeout = 30_000) + public void testAHandlerAssertionFailureReachesTheTestThread() throws Exception { + assertMemoryLeak(() -> { + // The shape that matters: an assertion inside a handler. Without the capture-and-rethrow, the + // client below sees a dropped connection, turns it into an OidcAuthException the test could + // easily be written to expect, and the broken assertion is never heard from again. + AssertionError thrownByHandler = new AssertionError("the handler asserted something and it failed"); + boolean rethrown = false; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + throw thrownByHandler; + })) { + try (OidcDeviceAuth auth = newAuth(server)) { + auth.signIn(); + Assert.fail("the handler threw, so the client cannot have completed a sign-in"); + } catch (OidcAuthException expected) { + // the client's view of a handler failure: the connection simply dropped + } + } catch (AssertionError e) { + rethrown = true; + Assert.assertSame("close() must resurface the handler's OWN throwable, not a copy", + thrownByHandler, e); + } + Assert.assertTrue("close() must resurface a handler failure on the test thread", rethrown); + }); + } + + @Test(timeout = 30_000) + public void testAHealthyRunClosesQuietlyAndRecordsItsRequests() throws Exception { + assertMemoryLeak(() -> { + // The control for the test above: without it, a close() that rethrew unconditionally - or a + // server that recorded a phantom failure - would look exactly like a working propagation path. + AtomicInteger deviceCalls = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, "{\"device_code\":\"DEV-CODE\",\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\",\"expires_in\":300," + + "\"interval\":1}"); + } + return MockOidcServer.json(200, "{\"token_type\":\"Bearer\",\"expires_in\":3600," + + "\"access_token\":\"ACCESS-1\"}"); + })) { + try (OidcDeviceAuth auth = newAuth(server)) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + } + Assert.assertEquals(1, deviceCalls.get()); + // requestAuthHeaders() records one entry per request READ, header or not - the OIDC endpoints + // are unauthenticated, so these are nulls, and the count is what QwpQueryClientTokenProviderTest + // asserts on + List headers = server.requestAuthHeaders(); + Assert.assertTrue("every request read must be recorded: " + headers, headers.size() >= 2); + } // a clean close: no throwable to resurface, so this must not throw + }); + } + + private static OidcDeviceAuth newAuth(MockOidcServer server) { + return OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl("/token")) + .allowInsecureTransport(true) + .prompt(challenge -> { + }) + .build(); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcAuthExceptionTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcAuthExceptionTest.java new file mode 100644 index 000000000..e8656af99 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcAuthExceptionTest.java @@ -0,0 +1,64 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.OidcAuthException; +import org.junit.Assert; +import org.junit.Test; + +public class OidcAuthExceptionTest { + + @Test + public void testOauthErrorCleanCodePassesThrough() { + OidcAuthException e = OidcAuthException.oauthError("access_denied", "the user declined"); + Assert.assertEquals("access_denied", e.getOauthError()); + Assert.assertEquals( + "the identity provider returned an error [error=access_denied, description=the user declined]", + e.getMessage() + ); + } + + @Test + public void testOauthErrorNullCodeYieldsNull() { + OidcAuthException e = OidcAuthException.oauthError(null, null); + Assert.assertNull(e.getOauthError()); + Assert.assertEquals("the identity provider returned an error [error=]", e.getMessage()); + } + + @Test + public void testOauthErrorStripsControlCharsFromAccessorAndMessage() { + // JsonLexer decodes JSON escapes, so a hostile identity provider can put a real ESC (and CR/LF) into + // the error code. getOauthError() is public and a caller may log it verbatim, so the accessor must not + // return raw control bytes that would inject ANSI sequences or forge log lines - the same guarantee the + // rendered message already made. Build the control chars from their code points to keep the source ASCII. + String raw = "access_denied" + (char) 0x1b + "[2Jx" + (char) 0x0d + (char) 0x0a + "y"; + OidcAuthException e = OidcAuthException.oauthError(raw, null); + Assert.assertEquals("access_denied[2Jxy", e.getOauthError()); + Assert.assertEquals( + "the identity provider returned an error [error=access_denied[2Jxy]", + e.getMessage() + ); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthPersistenceTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthPersistenceTest.java new file mode 100644 index 000000000..1b57cc3c8 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthPersistenceTest.java @@ -0,0 +1,2450 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.FileTokenStore; +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import io.questdb.client.cutlass.auth.PersistedToken; +import io.questdb.client.cutlass.auth.TokenStore; +import io.questdb.client.cutlass.auth.TokenStoreKey; +import io.questdb.client.test.tools.NoBrowserLaunch; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.ClassRule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.LockSupport; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +public class OidcDeviceAuthPersistenceTest { + private static final String DEVICE_PATH = "/device"; + private static final String TOKEN_PATH = "/token"; + + // a restored sign-in here reaches the device-code prompt; see NoBrowserLaunch for why this is a rule + @ClassRule + public static final NoBrowserLaunch NO_BROWSER = new NoBrowserLaunch(); + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testAdoptTrustsStoredIssuedTtlNotRemainingSpan() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-FRESH", "REFRESH-FRESH", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a token issued for 5m (stored ttl) loaded with only ~100s of life left. adopt() must set + // tokenTtlMillis from the stored ISSUED lifetime (5m), NOT the remaining span (~100s): the + // remaining-span form shrinks the effectiveSkewMillis basis as a token ages and collapses the + // clock-skew margin near expiry (guarded by testAdoptedTokenNearExpiryStillRefreshesOnFlushPath). + // A tampered ttl can only shrink the skew (never inflate it past CLOCK_SKEW_MILLIS) and the server + // still enforces the real expiry, so trusting the stored value is no less safe. + long now = System.currentTimeMillis(); + fake.loadReturns = new PersistedToken("ACCESS-1", null, "REFRESH-1", now + 100_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("the still-valid persisted token is served", "ACCESS-1", auth.signIn()); + long ttl = readPrivateLong(auth, "tokenTtlMillis"); + Assert.assertEquals("tokenTtlMillis must be the stored 5m issued lifetime, not the ~100s remaining span", + 300_000L, ttl); + } + Assert.assertEquals("no device flow for a valid persisted token", 0, device.get()); + Assert.assertEquals("no refresh for a valid persisted token", 0, token.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testAdoptedTokenNearExpiryStillRefreshesOnFlushPath() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-REFRESHED", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a token issued for 5m (stored ttl) but loaded with only ~20s of life left. The 30s clock-skew + // margin exceeds the remaining life, so getToken() (the flush path) must silently refresh rather + // than serve a token that would expire mid-request. Deriving the skew basis from the remaining + // span (the pre-fix bug) collapses the margin to ~10s and serves the near-expired token instead. + long now = System.currentTimeMillis(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", now + 20_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("a token inside the clock-skew margin must be refreshed, not served", + "ACCESS-REFRESHED", auth.getToken()); + } + Assert.assertEquals("device flow must not run; a silent refresh suffices", 0, device.get()); + Assert.assertTrue("the token endpoint must be hit for the refresh", token.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testCancelledStoreLockWaitDoesNotArmTheSharedRefreshBackOff() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // an expired access token with a live refresh token: getToken() must want a silent refresh + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", + System.currentTimeMillis() - 1_000, 300_000); + fake.cancelWaitInsteadOfRunning = true; + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + try { + auth.getToken(); + Assert.fail("getToken must report the cancellation rather than a refresh failure"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("interrupted")); + } + // refreshFailedAtMillis is INSTANCE state shared by every producer holding this + // OidcDeviceAuth. Arming it here would fail all of them for + // MIN_REFRESH_RETRY_INTERVAL_MILLIS over a credential that is fine and an identity + // provider that was never contacted - the cancelled wait ran no refresh at all. + Assert.assertEquals("a cancelled lock wait must not arm the shared refresh back-off", + 0L, readPrivateLong(auth, "refreshFailedAtMillis")); + Assert.assertEquals("the token endpoint must not have been called", 0, token.get()); + } finally { + // the store set the flag on this thread by design; do not leak it into later tests + Thread.interrupted(); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFailedRefreshWithLateInterruptArmsBackOffWhenStoreActionRan() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(500, "{\"error\":\"unexpected_device_flow\"}"); + } + tokenCalls.incrementAndGet(); + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-REVOKED", + System.currentTimeMillis() - 1_000, 300_000); + // Model cancellation arriving after action.run(): the refresh really reached the IdP and + // failed, but the caller carries an interrupt by the time getToken() classifies the result. + fake.interruptAfterAction = true; + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + final String message; + try { + auth.getToken(); + Assert.fail("the revoked refresh token must fail"); + return; + } catch (OidcAuthException e) { + message = e.getMessage(); + } + + Assert.assertTrue("the store must have run the refresh action", fake.actionRuns.get() > 0); + Assert.assertEquals("the failed refresh must reach the token endpoint once", 1, tokenCalls.get()); + Assert.assertFalse("a refresh that ran must not be reported as an abandoned store wait: " + + message, + message.contains("no silent token refresh was attempted") + || message.contains("token store lock")); + Assert.assertTrue("the actual failed attempt must arm the shared stampede guard", + readPrivateLong(auth, "refreshFailedAtMillis") > 0); + Assert.assertTrue("the caller's cancellation signal must survive", + Thread.currentThread().isInterrupted()); + } finally { + Thread.interrupted(); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFailedRefreshWithLateInterruptArmsBackOffWithoutTokenStore() throws Exception { + assertMemoryLeak(() -> { + CountDownLatch refreshEntered = new CountDownLatch(1); + CountDownLatch releaseRefresh = new CountDownLatch(1); + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + if (tokenCalls.incrementAndGet() == 1) { + return MockOidcServer.json(200, + tokenJson("ACCESS-INITIAL", null, "REFRESH-REVOKED", 3600)); + } + refreshEntered.countDown(); + try { + if (!releaseRefresh.await(10, TimeUnit.SECONDS)) { + throw new AssertionError("the test did not release the failed refresh response"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("the mock token endpoint was interrupted", e); + } + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + }; + + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = baseBuilder(server).build()) { + Assert.assertEquals("ACCESS-INITIAL", auth.signIn()); + OidcDeviceAuthTest.expireCachedToken(auth); + + AtomicReference failure = new AtomicReference<>(); + AtomicBoolean callerInterruptSurvived = new AtomicBoolean(); + Thread caller = new Thread(() -> { + try { + auth.getToken(); + } catch (Throwable e) { + failure.set(e); + } finally { + callerInterruptSurvived.set(Thread.currentThread().isInterrupted()); + Thread.interrupted(); + } + }, "oidc-interrupted-failed-refresh"); + caller.setDaemon(true); + caller.start(); + try { + Assert.assertTrue("the refresh did not reach the token endpoint", + refreshEntered.await(10, TimeUnit.SECONDS)); + caller.interrupt(); + releaseRefresh.countDown(); + caller.join(10_000); + } finally { + releaseRefresh.countDown(); + caller.interrupt(); + caller.join(10_000); + } + + Assert.assertFalse("the interrupted getToken() caller did not finish", caller.isAlive()); + Assert.assertTrue("the revoked refresh must surface as an OIDC failure", + failure.get() instanceof OidcAuthException); + String message = failure.get().getMessage(); + Assert.assertFalse("a no-store refresh cannot have been abandoned behind a store lock: " + + message, + message.contains("no silent token refresh was attempted") + || message.contains("token store lock")); + Assert.assertEquals("one device grant and one failed refresh must reach the token endpoint", + 2, tokenCalls.get()); + Assert.assertTrue("the actual failed attempt must arm the shared stampede guard", + readPrivateLong(auth, "refreshFailedAtMillis") > 0); + Assert.assertTrue("the caller's cancellation signal must survive", callerInterruptSurvived.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testCancelledStoreLockWaitDoesNotStartTheDeviceFlow() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", + System.currentTimeMillis() - 1_000, 300_000); + fake.cancelWaitInsteadOfRunning = true; + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + try { + auth.signIn(); + Assert.fail("signIn must decline once a cancellation abandoned the store lock wait"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("interrupted")); + } + // The device flow is the expensive wrong answer: it launches a browser and then polls + // to the device-code lifetime on Os.sleep, which ignores interrupts - so a caller that + // cancelled this thread cannot get it back, and shutdown does not complete. A plain + // false from inLock reads as "the refresh failed", which is exactly what sends signIn() + // here. + Assert.assertEquals("a cancelled lock wait must not start the interactive device flow", + 0, device.get()); + Assert.assertEquals("and must not reach the token endpoint either", 0, token.get()); + } finally { + Thread.interrupted(); + } + } + }); + } + + @Test(timeout = 30_000) + public void testCloseWaitsForCustomStoreClearToDeleteCredential() throws Exception { + assertMemoryLeak(() -> { + FakeTokenStore store = new FakeTokenStore(); + store.stored = new PersistedToken( + "ACCESS-STALE", null, "REFRESH-SECRET", + System.currentTimeMillis() + 300_000, 300_000); + store.clearEntered = new CountDownLatch(1); + store.releaseClear = new CountDownLatch(1); + AtomicReference clearFailure = new AtomicReference<>(); + AtomicReference closeFailure = new AtomicReference<>(); + AtomicBoolean clearInterruptedAtReturn = new AtomicBoolean(); + + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint("http://127.0.0.1:1/device") + .tokenEndpoint("http://127.0.0.1:1/token") + .tokenStore(store) + .build(); + Thread clearer = new Thread(() -> { + try { + auth.clearCache(); + } catch (Throwable e) { + clearFailure.set(e); + } finally { + clearInterruptedAtReturn.set(Thread.currentThread().isInterrupted()); + } + }, "oidc-custom-store-clearer"); + Thread closer = new Thread(() -> { + try { + auth.close(); + } catch (Throwable e) { + closeFailure.set(e); + } + }, "oidc-custom-store-closer"); + clearer.setDaemon(true); + closer.setDaemon(true); + + try { + clearer.start(); + Assert.assertTrue("custom clear did not start", store.clearEntered.await(10, TimeUnit.SECONDS)); + closer.start(); + awaitOidcCloseLockWait(closer); + + Assert.assertNotNull("the credential must remain while its backend deletion is in flight", + store.stored); + Assert.assertFalse("close() must not interrupt an arbitrary custom TokenStore.clear()", + store.clearInterrupted.get()); + + store.releaseClear.countDown(); + clearer.join(10_000L); + closer.join(10_000L); + } finally { + store.releaseClear.countDown(); + clearer.interrupt(); + closer.interrupt(); + clearer.join(10_000L); + closer.join(10_000L); + auth.close(); + } + + Assert.assertFalse("custom clear did not finish", clearer.isAlive()); + Assert.assertFalse("close did not finish after custom clear", closer.isAlive()); + Assert.assertNull("custom clear failed", clearFailure.get()); + Assert.assertNull("close failed", closeFailure.get()); + Assert.assertFalse("custom clear must not inherit close()'s cancellation signal", + clearInterruptedAtReturn.get()); + Assert.assertNull("both calls returned while the persisted credential remained reloadable", + store.stored); + }); + } + + @Test(timeout = 30_000) + public void testCloseInterruptsClearCacheWaitingBehindPeerInstance() throws Exception { + assertMemoryLeak(() -> { + Path dir = storeDir(); + FileTokenStore holderStore = new FileTokenStore(dir); + FileTokenStore clearingStore = new FileTokenStore(dir); + TokenStoreKey key = new TokenStoreKey( + "questdb", + "http://127.0.0.1:1/token", + "http://127.0.0.1:1/device", + "openid", + null, + false); + holderStore.save(key, new PersistedToken( + "ACCESS-STALE", null, "REFRESH-SECRET", + System.currentTimeMillis() - 1_000, 300_000)); + + CountDownLatch holderEntered = new CountDownLatch(1); + CountDownLatch releaseHolder = new CountDownLatch(1); + CountDownLatch closeStarted = new CountDownLatch(1); + AtomicBoolean clearInterruptedAtReturn = new AtomicBoolean(); + AtomicReference holderFailure = new AtomicReference<>(); + AtomicReference clearFailure = new AtomicReference<>(); + + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint("http://127.0.0.1:1/device") + .tokenEndpoint("http://127.0.0.1:1/token") + .tokenStore(clearingStore) + .build()) { + Thread holder = new Thread(() -> { + try { + holderStore.inLock(key, () -> { + holderEntered.countDown(); + try { + if (!releaseHolder.await(10, TimeUnit.SECONDS)) { + throw new AssertionError("the test did not release the token-store holder"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("the token-store holder was interrupted", e); + } + return true; + }); + } catch (Throwable e) { + holderFailure.set(e); + } + }, "oidc-clear-peer-holder"); + Thread clearer = new Thread(() -> { + try { + auth.clearCache(); + } catch (Throwable e) { + clearFailure.set(e); + } finally { + clearInterruptedAtReturn.set(Thread.currentThread().isInterrupted()); + } + }, "oidc-clear-waiter"); + // Prevent an unfixed close() from pinning the test thread until the holder's 10-second fallback. + Thread fallbackRelease = new Thread(() -> { + try { + closeStarted.await(); + Thread.sleep(2_000L); + releaseHolder.countDown(); + } catch (InterruptedException ignore) { + // The fixed path releases the holder itself and stops this fallback promptly. + } + }, "oidc-clear-close-fallback"); + holder.setDaemon(true); + clearer.setDaemon(true); + fallbackRelease.setDaemon(true); + holder.start(); + fallbackRelease.start(); + + long closeElapsedMillis; + try { + Assert.assertTrue("the peer did not enter the token-store critical section", + holderEntered.await(10, TimeUnit.SECONDS)); + clearer.start(); + awaitProcessLockWait(clearer); + + closeStarted.countDown(); + long closeStartNanos = System.nanoTime(); + auth.close(); + closeElapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - closeStartNanos); + clearer.join(5_000L); + } finally { + releaseHolder.countDown(); + fallbackRelease.interrupt(); + clearer.interrupt(); + holder.join(10_000L); + clearer.join(10_000L); + fallbackRelease.join(10_000L); + } + + Assert.assertTrue("close() waited " + closeElapsedMillis + + "ms for a peer token-store operation instead of interrupting clearCache()", + closeElapsedMillis < 1_000L); + Assert.assertFalse("the interrupted clearCache() thread did not unwind", clearer.isAlive()); + Assert.assertNull("the peer token-store holder failed", holderFailure.get()); + Assert.assertNull("clearCache() failed", clearFailure.get()); + Assert.assertTrue("clearCache() must preserve close()'s cancellation signal", + clearInterruptedAtReturn.get()); + Assert.assertFalse("clearCache() must still erase the persisted credential when interrupted", + Files.exists(dir.resolve(key.hash() + ".json"))); + } + }); + } + + @Test(timeout = 30_000) + public void testCloseInterruptsRefreshWaitingBehindPeerInstance() throws Exception { + assertMemoryLeak(() -> { + CountDownLatch peerRefreshInFlight = new CountDownLatch(1); + CountDownLatch releasePeerRefresh = new CountDownLatch(1); + CountDownLatch closeStarted = new CountDownLatch(1); + AtomicInteger refreshPosts = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(500, "{\"error\":\"unexpected_device_flow\"}"); + } + int refreshNumber = refreshPosts.incrementAndGet(); + if (refreshNumber == 1) { + peerRefreshInFlight.countDown(); + try { + if (!releasePeerRefresh.await(10, TimeUnit.SECONDS)) { + throw new AssertionError("the test did not release the peer refresh"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("the mock peer refresh was interrupted", e); + } + // Leave the persisted entry stale. Without close() interrupting the waiting instance, that + // instance acquires the shared process lock next and starts a fresh token POST after closed. + return MockOidcServer.json(500, "{\"error\":\"temporarily_unavailable\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-UNEXPECTED", null, "REFRESH-2", 3600)); + }; + + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + new FileTokenStore(dir).save(keyFor(server), new PersistedToken( + "ACCESS-STALE", null, "REFRESH-1", System.currentTimeMillis() - 1_000, 300_000)); + AtomicReference peerFailure = new AtomicReference<>(); + AtomicReference waitingFailure = new AtomicReference<>(); + try (OidcDeviceAuth peer = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build(); + OidcDeviceAuth closing = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + Thread peerRefresh = new Thread(() -> { + try { + peer.getToken(); + } catch (Throwable e) { + peerFailure.set(e); + } + }, "oidc-peer-refresh"); + Thread waitingRefresh = new Thread(() -> { + try { + closing.getToken(); + } catch (Throwable e) { + waitingFailure.set(e); + } + }, "oidc-waiting-refresh"); + // Prevent an unfixed close() from pinning the test thread for the full mock/HTTP timeout. + // The fixed close returns before this fallback fires, while the peer still owns the lock. + Thread fallbackRelease = new Thread(() -> { + try { + closeStarted.await(); + Thread.sleep(2_000); + releasePeerRefresh.countDown(); + } catch (InterruptedException ignore) { + // The fixed path releases the peer itself and stops this fallback promptly. + } + }, "oidc-close-test-fallback"); + peerRefresh.setDaemon(true); + waitingRefresh.setDaemon(true); + fallbackRelease.setDaemon(true); + peerRefresh.start(); + fallbackRelease.start(); + + long closeElapsedMillis; + try { + Assert.assertTrue("the peer refresh did not reach the token endpoint", + peerRefreshInFlight.await(10, TimeUnit.SECONDS)); + waitingRefresh.start(); + // The peer is now inside the token POST while holding FileTokenStore's JVM-wide + // per-directory/per-identity process lock. Wait until the second instance queues on it. + long waitDeadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (waitingRefresh.isAlive() + && waitingRefresh.getState() != Thread.State.WAITING + && System.nanoTime() - waitDeadline < 0) { + Thread.sleep(10); + } + Assert.assertEquals("the second instance did not queue on the shared process lock", + Thread.State.WAITING, waitingRefresh.getState()); + + closeStarted.countDown(); + long closeStartNanos = System.nanoTime(); + closing.close(); + closeElapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - closeStartNanos); + waitingRefresh.join(5_000); + } finally { + releasePeerRefresh.countDown(); + fallbackRelease.interrupt(); + waitingRefresh.interrupt(); + peerRefresh.join(10_000); + waitingRefresh.join(10_000); + fallbackRelease.join(10_000); + } + + Assert.assertTrue("close() waited " + closeElapsedMillis + + "ms for another instance's refresh instead of interrupting its own waiter", + closeElapsedMillis < 1_000); + Assert.assertFalse("the interrupted refresh thread did not unwind", waitingRefresh.isAlive()); + Assert.assertTrue("the waiting refresh must report cancellation", + waitingFailure.get() instanceof OidcAuthException); + Assert.assertTrue(waitingFailure.get().getMessage(), + waitingFailure.get().getMessage().contains("interrupted")); + Assert.assertTrue("the peer refresh must fail on the mock 500 response", + peerFailure.get() instanceof OidcAuthException); + Assert.assertEquals("a closing instance must not start another token POST", 1, refreshPosts.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testCloseDoesNotInterruptSuccessfulTokenStoreLoad() throws Exception { + assertMemoryLeak(() -> { + CountDownLatch loadEntered = new CountDownLatch(1); + AtomicBoolean releaseLoad = new AtomicBoolean(); + AtomicReference returnedToken = new AtomicReference<>(); + AtomicReference failure = new AtomicReference<>(); + AtomicBoolean interruptedAfterReturn = new AtomicBoolean(); + PersistedToken valid = new PersistedToken( + "ACCESS-VALID", null, null, System.currentTimeMillis() + 300_000, 300_000); + TokenStore store = new TokenStore() { + @Override + public void clear(TokenStoreKey key) { + } + + @Override + public PersistedToken load(TokenStoreKey key) { + loadEntered.countDown(); + while (!releaseLoad.get()) { + LockSupport.park(); + } + return valid; + } + + @Override + public void save(TokenStoreKey key, PersistedToken token) { + } + }; + + try (MockOidcServer server = new MockOidcServer( + (method, path, body) -> MockOidcServer.json(500, "{\"error\":\"unexpected_request\"}"))) { + OidcDeviceAuth auth = baseBuilder(server).tokenStore(store).build(); + Thread tokenThread = new Thread(() -> { + try { + returnedToken.set(auth.getToken()); + } catch (Throwable e) { + failure.set(e); + } finally { + interruptedAfterReturn.set(Thread.currentThread().isInterrupted()); + } + }, "oidc-successful-store-load"); + Thread closeThread = new Thread(auth::close, "oidc-close-during-store-load"); + tokenThread.setDaemon(true); + closeThread.setDaemon(true); + tokenThread.start(); + try { + Assert.assertTrue("getToken() did not enter the token store load", + loadEntered.await(10, TimeUnit.SECONDS)); + closeThread.start(); + // Give close() time to publish closed and wait for the instance lock. Before the fix it + // interrupts the instance-lock holder here, causing park() to return with the flag set. + Thread.sleep(200); + releaseLoad.set(true); + LockSupport.unpark(tokenThread); + tokenThread.join(10_000); + closeThread.join(10_000); + } finally { + releaseLoad.set(true); + LockSupport.unpark(tokenThread); + tokenThread.interrupt(); + tokenThread.join(10_000); + closeThread.join(10_000); + auth.close(); + } + + Assert.assertFalse("getToken() thread did not finish", tokenThread.isAlive()); + Assert.assertFalse("close() thread did not finish", closeThread.isAlive()); + Assert.assertNull("getToken() failed", failure.get()); + Assert.assertEquals("ACCESS-VALID", returnedToken.get()); + Assert.assertFalse("close() left an interrupt on a successful getToken() caller", + interruptedAfterReturn.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testClosePreventsRefreshPostAfterStoreReturnsLate() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenPosts = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(500, "{\"error\":\"unexpected_device_flow\"}"); + } + tokenPosts.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-UNEXPECTED", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", + System.currentTimeMillis() - 1_000, 300_000); + fake.inLockEntered = new CountDownLatch(1); + fake.releaseInLock = new CountDownLatch(1); + AtomicReference refreshFailure = new AtomicReference<>(); + CountDownLatch closeStarted = new CountDownLatch(1); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Thread refresh = new Thread(() -> { + try { + auth.getToken(); + } catch (Throwable e) { + refreshFailure.set(e); + } + }, "oidc-late-store-refresh"); + // This deliberately models a TokenStore implementation that finishes its wait despite an + // interrupt. close() must still acquire the instance lock before freeing resources, so let + // the store return shortly after close begins and verify the refresh checks closed before I/O. + Thread delayedRelease = new Thread(() -> { + try { + closeStarted.await(); + Thread.sleep(200); + fake.releaseInLock.countDown(); + } catch (InterruptedException ignore) { + // Test cleanup releases the latch directly. + } + }, "oidc-late-store-release"); + refresh.setDaemon(true); + delayedRelease.setDaemon(true); + refresh.start(); + delayedRelease.start(); + try { + Assert.assertTrue("the refresh did not enter the delayed store section", + fake.inLockEntered.await(10, TimeUnit.SECONDS)); + closeStarted.countDown(); + auth.close(); + } finally { + fake.releaseInLock.countDown(); + delayedRelease.interrupt(); + refresh.join(10_000); + delayedRelease.join(10_000); + } + Assert.assertFalse("the delayed refresh thread did not unwind", refresh.isAlive()); + Assert.assertTrue("the delayed refresh must observe close", + refreshFailure.get() instanceof OidcAuthException); + Assert.assertTrue(refreshFailure.get().getMessage(), + refreshFailure.get().getMessage().contains("closed")); + Assert.assertEquals("a refresh resuming after close must not issue a token POST", + 0, tokenPosts.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testBuildRejectsFileTokenStoreWithTooSmallStaleWindow() throws Exception { + assertMemoryLeak(() -> { + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(200, "{}"))) { + Path dir = storeDir(); + // a lock-staleness window below LOCK_HOLD_HTTP_TIMEOUT_MULTIPLE (6) x httpTimeoutMillis would let a + // peer judge a live holder's lock stale and steal it mid-refresh, reopening the rotating-refresh- + // token race the lock prevents; build() must reject the combination rather than ship the race. + // The multiple counts the TCP connect and the TLS handshake as the two separate budgets + // HttpClient spends on them, on top of send, await, parse and the drain. + try { + baseBuilder(server) + .httpTimeoutMillis(30_000) + .tokenStore(new FileTokenStore(dir, 3_000, 179_999)) + .build(); + Assert.fail("a lockStaleMillis below 6x httpTimeoutMillis must be rejected"); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("lockStaleMillis")); + } + // exactly 6x httpTimeoutMillis is the boundary and builds + try (OidcDeviceAuth ignored = baseBuilder(server) + .httpTimeoutMillis(30_000) + .tokenStore(new FileTokenStore(dir, 3_000, 180_000)) + .build()) { + // building at the boundary succeeds + } + } + }); + } + + @Test(timeout = 30_000) + public void testBuilderRejectsHttpTimeoutAboveCap() throws Exception { + assertMemoryLeak(() -> { + // the HTTP timeout is capped (120s): a token-endpoint round-trip never needs longer, and bounding + // it keeps a refresh held under the FileTokenStore cross-process lock safely shorter than that + // store's staleness window, so a slow refresh's live lock is not stolen by a peer. Above the cap is + // rejected; the boundary value builds. + try { + OidcDeviceAuth.builder().httpTimeoutMillis(120_001); + Assert.fail("a httpTimeoutMillis above the cap must be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("httpTimeoutMillis")); + } + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint("https://idp.example/device") + .tokenEndpoint("https://idp.example/token") + .httpTimeoutMillis(120_000) + .build()) { + // building at the cap boundary succeeds + } + }); + } + + @Test(timeout = 30_000) + public void testClearCacheDeletesPersistedEntry() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + Path file = dir.resolve(keyFor(server).hash() + ".json"); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + auth.signIn(); + Assert.assertTrue(Files.exists(file)); + + auth.clearCache(); + Assert.assertFalse("clearCache must remove the persisted entry", Files.exists(file)); + + int before = device.get(); + auth.signIn(); + Assert.assertTrue("a cleared cache must re-run the device flow", device.get() > before); + } + } + }); + } + + @Test(timeout = 30_000) + public void testClearCacheDoesNotReloadStaleEntry() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-NEW", null, "REFRESH-NEW", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // load() keeps returning a valid entry even after clear() (loadReturns is not cleared); this + // proves clearCache() does not re-read the store - it relies on storeLoadAttempted, not on the + // store having actually forgotten the entry - so a fresh device flow runs rather than re-adopting + fake.loadReturns = new PersistedToken("ACCESS-OLD", null, "REFRESH-OLD", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-OLD", auth.signIn()); + int loadsAfterFirst = fake.loads.get(); + + auth.clearCache(); + Assert.assertEquals("clearCache must clear the persisted entry exactly once", 1, fake.clears.get()); + + // even though load() would still hand back ACCESS-OLD, clearCache must not let it be reloaded + Assert.assertEquals("ACCESS-NEW", auth.signIn()); + Assert.assertEquals("clearCache must not trigger a re-read of the store", + loadsAfterFirst, fake.loads.get()); + Assert.assertEquals("a cleared cache must re-run the device flow", 1, device.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDeviceGrantWithoutRefreshTokenClearsPreviousUsersRefreshToken() throws Exception { + assertMemoryLeak(() -> { + // Cross-account confusion. storeTokens() keeps the current refresh token whenever a response + // omits one -- correct for a REFRESH response, which RFC 6749 6 lets omit it, but wrong for a + // fresh device grant. A device grant is a NEW authorization and may be a DIFFERENT human: if it + // returns no refresh token, the previous user's must not survive it, or the next silent refresh + // signs back in as them with no interaction and no signal. + // + // A: persisted, expired access token plus a live refresh token. + // B: signs in interactively after A's refresh hits a transient IdP failure, and B's grant carries + // no refresh token of its own. + AtomicInteger device = new AtomicInteger(); + AtomicInteger refreshCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + if (body.contains("grant_type=refresh_token")) { + // A's refresh token is live, not revoked -- the first attempt just lands on a 503. That is + // what makes this a confusion bug rather than a dead credential: the token still works, so + // any later use of it silently resumes A's session. + if (refreshCalls.incrementAndGet() == 1) { + return MockOidcServer.json(503, "{}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-A2", null, "REFRESH-A", 3600)); + } + // B's device grant: a served token and deliberately NO refresh token. expires_in=1 so B's + // token goes stale inside the test without stubbing the clock. + return MockOidcServer.json(200, tokenJson("ACCESS-B", null, null, 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.stored = new PersistedToken("ACCESS-A", null, "REFRESH-A", System.currentTimeMillis() - 1, 3600_000); + + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("A's refresh fails, so B signs in interactively", + "ACCESS-B", auth.signIn()); + Assert.assertEquals("the device flow must have run for B", 1, device.get()); + + // B's token is issued for 1s and effectiveSkewMillis caps the skew at half that, so it + // reads as stale ~500ms in. Wait past that, then ask for a token the way the sender does. + Thread.sleep(1_000L); + try { + String served = auth.getToken(); + Assert.fail("B's expired token must not be refreshed with A's retained refresh token; " + + "getToken() served [" + served + "] after " + refreshCalls.get() + " refresh calls"); + } catch (OidcAuthException e) { + Assert.assertTrue("expected a prompt to sign in again, got: " + e.getMessage(), + e.getMessage().contains("could not be refreshed without an interactive sign-in")); + } + Assert.assertEquals("no refresh may be attempted once B's grant carried no refresh token", + 1, refreshCalls.get()); + } + + // Persistence half: A's refresh token must not outlive B's sign-in on disk either, or the next + // process start adopts it and resumes as A. + Assert.assertNotNull("B's grant must have been persisted over A's entry", fake.stored); + Assert.assertNull("A's refresh token must not survive in the store: " + fake.stored.getRefreshToken(), + fake.stored.getRefreshToken()); + + // Restart over the same store. + int refreshesBeforeRestart = refreshCalls.get(); + try (OidcDeviceAuth restarted = baseBuilder(server).tokenStore(fake).build()) { + try { + String served = restarted.getToken(); + Assert.fail("a restart must not resume A's session; getToken() served [" + served + "]"); + } catch (OidcAuthException e) { + Assert.assertTrue("expected a sign-in prompt after restart, got: " + e.getMessage(), + e.getMessage().contains("could not be refreshed without an interactive sign-in")); + } + } + Assert.assertEquals("a restart must not refresh with A's token either", + refreshesBeforeRestart, refreshCalls.get()); + } + }); + } + + @Test + public void testDefaultInLockRunsTheAction() { + // TokenStore.inLock has a default that simply runs the action (no cross-process coordination). It is a + // public extension point users implement, so a store that does NOT override inLock must still run its + // critical section and return the action's result. Both in-tree stores override inLock, so this pins the + // default directly; a regression to, say, "return false" without running the action would fail here. + TokenStore store = new TokenStore() { + @Override + public void clear(TokenStoreKey key) { + } + + @Override + public PersistedToken load(TokenStoreKey key) { + return null; + } + + @Override + public void save(TokenStoreKey key, PersistedToken token) { + } + }; + AtomicBoolean ran = new AtomicBoolean(); + boolean result = store.inLock(null, () -> { + ran.set(true); + return true; + }); + Assert.assertTrue("the default inLock must run the action", ran.get()); + Assert.assertTrue("the default inLock must return the action's result", result); + } + + @Test + public void testEntryWithNoTokenOfEitherKindIsNotAdopted() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicBoolean sawPlantedRefreshToken = new AtomicBoolean(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + if (body != null && body.contains("REFRESH-PLANTED")) { + sawPlantedRefreshToken.set(true); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // The cheapest credential swap there is: an entry carrying ONLY a refresh token. Every entry + // this client writes carries at least one token kind, so this shape came from somewhere else - + // an attacker who can WRITE the store directory, without ever reading our 0600 file. Adopted, + // the next silent refresh would present THEIR refresh token and the client would resume as + // them, with no prompt and nothing in any log recording the change of identity. + fake.loadReturns = new PersistedToken(null, null, "REFRESH-PLANTED", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-FRESH", auth.signIn()); + } + Assert.assertFalse("the planted refresh token must never reach the token endpoint", + sawPlantedRefreshToken.get()); + Assert.assertTrue("a rejected entry must fall back to the device flow, not to a silent refresh", + device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenBacksOffAfterAFailedRefreshInsteadOfFloodingTheIdp() throws Exception { + assertMemoryLeak(() -> { + // getToken() runs once per ILP flush and once per (re)connect, and a producer retrying its rows + // calls it in a tight loop. Without a back-off a revoked refresh token cost a full + // token-endpoint round trip on EVERY call: a sustained request flood at the provider - enough to + // trip its rate limits and lengthen the outage being retried - and a producer blocked for each + // round trip, up to the OS TCP-connect timeout against a black-holed endpoint. + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + tokenCalls.incrementAndGet(); + // the shape of a revoked refresh token + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-REVOKED", + System.currentTimeMillis() - 1, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + for (int i = 0; i < 25; i++) { + try { + auth.getToken(); + Assert.fail("a revoked refresh token must not yield a usable token"); + } catch (OidcAuthException expected) { + // every call still reports the failure - only the network attempt is rate-limited + } + } + } + Assert.assertEquals("25 getToken() calls must not mean 25 token-endpoint round trips", + 1, tokenCalls.get()); + } + }); + } + + @Test(timeout = 60_000) + public void testGetTokenBackOffExpiresWhileAProducerKeepsCalling() throws Exception { + assertMemoryLeak(() -> { + // The companion of the throttle test above, and the half that cannot be checked by a tight + // loop: the back-off must EXPIRE on its own while the caller keeps calling. Arming the latch + // on a call the back-off itself skipped slides its window forward by one call every time, so + // it never elapses for any caller returning faster than the retry interval - and getToken() + // runs once per ILP flush, at a default auto-flush interval of one second. One transient + // identity-provider failure then wedges the sender for the life of the process. + // + // Drives real wall clock because that is the only thing that distinguishes the two shapes: + // both serve the same token, throttle to one round trip inside the window, and differ only in + // whether a later call is ever allowed through. + final long retryIntervalMillis = 5_000L; // OidcDeviceAuth.MIN_REFRESH_RETRY_INTERVAL_MILLIS + AtomicBoolean healthy = new AtomicBoolean(); + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + tokenCalls.incrementAndGet(); + if (!healthy.get()) { + // a transient outage, not a revoked grant: tryRefresh reports failure and leaves the + // cached refresh token intact, so the next attempt would succeed + return MockOidcServer.json(503, "{\"error\":\"temporarily_unavailable\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-RECOVERED", null, "REFRESH-NEW", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-OK", + System.currentTimeMillis() - 1, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + try { + auth.getToken(); + Assert.fail("the first call must attempt the refresh and surface its failure"); + } catch (OidcAuthException expected) { + } + Assert.assertEquals("the first call must have reached the token endpoint", + 1, tokenCalls.get()); + + // the provider recovers, and the caller keeps polling well inside the window, exactly + // as a flushing producer does + healthy.set(true); + String token = null; + final long deadlineNanos = System.nanoTime() + 4 * retryIntervalMillis * 1_000_000L; + while (token == null && System.nanoTime() - deadlineNanos < 0) { + Thread.sleep(50); + try { + token = auth.getToken(); + } catch (OidcAuthException stillBackedOff) { + } + } + Assert.assertEquals("the back-off must expire on its own; the identity provider " + + "recovered and getToken() never retried it", + "ACCESS-RECOVERED", token); + Assert.assertEquals("exactly one retry, once the window had elapsed", + 2, tokenCalls.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testSignInDeclinesOnAnInterruptCarryingThread() throws Exception { + assertMemoryLeak(() -> { + // The interrupt guard used to live only inside FileTokenStore.inLock, so signIn()'s behaviour + // depended on whether a store was configured - and was wrong in OPPOSITE directions either way. + // + // with a FileTokenStore: inLock declined the carried interrupt by returning false, signIn() + // read that as "the refresh failed" and started the DEVICE FLOW - a browser prompt and a poll + // loop Os.sleep cannot be interrupted out of, so the cancelled caller was parked for up to the + // device-code lifetime, having skipped a refresh it could have completed; + // + // with no store: tryRefreshCoordinated() went straight to tryRefresh() and POSTed to the token + // endpoint on that same cancelled thread. + // + // Driven twice in one test on purpose: the defect was the ASYMMETRY, so the two configurations + // agreeing is the property worth pinning. Both halves reach signIn() holding a usable refresh + // token and an expired access token, so each has real network work to decline rather than a + // cache hit to serve. + for (boolean withStore : new boolean[]{true, false}) { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + OidcDeviceAuth.Builder builder = baseBuilder(server); + if (withStore) { + // a REAL FileTokenStore, not a double: its inLock is what declines a carried + // interrupt, and that decline is the half of the asymmetry that ended in a prompt + Path dir = storeDir(); + new FileTokenStore(dir).save(keyFor(server), + new PersistedToken("ACCESS-STALE", null, "REFRESH-OK", + System.currentTimeMillis() - 1, 300_000)); + builder.tokenStore(new FileTokenStore(dir)); + } + try (OidcDeviceAuth auth = builder.build()) { + if (!withStore) { + // nothing on disk to restore from, so earn a refresh token the ordinary way + Assert.assertEquals("ACCESS-1", auth.signIn()); + OidcDeviceAuthTest.expireCachedToken(auth); + } + final int deviceBefore = device.get(); + final int tokenBefore = token.get(); + + Thread.currentThread().interrupt(); + try { + String served = auth.signIn(); + Assert.fail("a cancelled caller must not be signed in [withStore=" + withStore + + ", served=" + served + ", deviceFlows=" + (device.get() - deviceBefore) + + ", tokenCalls=" + (token.get() - tokenBefore) + "]"); + } catch (OidcAuthException e) { + Assert.assertTrue("withStore=" + withStore + ": " + e.getMessage(), + e.getMessage().contains("interrupted")); + } finally { + Assert.assertTrue("the caller's cancellation signal must survive signIn() " + + "[withStore=" + withStore + "]", Thread.interrupted()); + } + Assert.assertEquals("no device flow may be started on a cancelled thread " + + "[withStore=" + withStore + "]", deviceBefore, device.get()); + Assert.assertEquals("no token-endpoint round trip may be made on a cancelled thread " + + "[withStore=" + withStore + "]", tokenBefore, token.get()); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testSignInClearsTheRefreshBackOff() throws Exception { + assertMemoryLeak(() -> { + // The back-off must never strand a caller: signIn() is the explicit action a user takes to + // recover, so it re-attempts immediately and falls through to the device flow. + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + if (body != null && body.contains("REFRESH-REVOKED")) { + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-NEW", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-REVOKED", + System.currentTimeMillis() - 1, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + try { + auth.getToken(); + Assert.fail("expected the revoked refresh to fail"); + } catch (OidcAuthException expected) { + // now latched + } + Assert.assertEquals("ACCESS-FRESH", auth.signIn()); + Assert.assertTrue("signIn() must not be held off by the back-off", device.get() >= 1); + } + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenDeclinesTheRefreshOnAnInterruptCarryingThreadWithoutLatchingTheBackOff() throws Exception { + assertMemoryLeak(() -> { + // A cancelled caller must not drive a network round trip, must keep its cancellation signal, must + // be told what actually happened, and must not suppress the next five seconds of legitimate + // refreshes for every other thread sharing this instance. + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + tokenCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-REFRESHED", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", + System.currentTimeMillis() - 1, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Thread.currentThread().interrupt(); + final boolean flagSurvived; + final String message; + try { + auth.getToken(); + Assert.fail("a cancelled caller must not get a token out of a network refresh"); + return; + } catch (OidcAuthException e) { + message = e.getMessage(); + flagSurvived = Thread.currentThread().isInterrupted(); + } finally { + Thread.interrupted(); // do not leak the flag into the next test + } + + Assert.assertEquals("no refresh may be attempted on a cancelled thread", 0, tokenCalls.get()); + Assert.assertTrue("the message must name the interrupt, not blame the credential: " + message, + message.contains("interrupted")); + Assert.assertFalse("it must not send the user to re-authenticate: " + message, + message.contains("call signIn()")); + Assert.assertTrue("the caller's cancellation signal must survive getToken()", flagSurvived); + + // and the decline must not have latched the back-off: a clean caller refreshes at once + Assert.assertEquals("ACCESS-REFRESHED", auth.getToken()); + Assert.assertEquals(1, tokenCalls.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenDeclinesTheRefreshOnAnInterruptCarryingThreadWithNoTokenStore() throws Exception { + assertMemoryLeak(() -> { + // The only interrupt guard used to live inside FileTokenStore.inLock, so with NO store configured + // a cancelled thread POSTed to the token endpoint regardless. The guard is in getToken() now, so + // both shapes behave the same. + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + tokenCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = baseBuilder(server).build()) { + Assert.assertEquals("ACCESS-1", auth.signIn()); // one device grant, one token call + Assert.assertEquals(1, tokenCalls.get()); + OidcDeviceAuthTest.expireCachedToken(auth); + + Thread.currentThread().interrupt(); + try { + auth.getToken(); + Assert.fail("a cancelled caller must not drive a refresh even with no token store"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("interrupted")); + } finally { + Thread.interrupted(); + } + Assert.assertEquals("no refresh may be attempted on a cancelled thread", 1, tokenCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenAsFirstCallAfterRestore() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + new FileTokenStore(dir).save(keyFor(server), + new PersistedToken("ACCESS-1", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000)); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + // getToken() without a prior signIn(): a restored process can flush immediately + Assert.assertEquals("ACCESS-1", auth.getToken()); + } + Assert.assertEquals(0, device.get()); + Assert.assertEquals(0, token.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenDegradesWhenStoreLockHeld() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + // a small acquire budget so the test does not wait the 3s default; a large staleness window so + // the pre-created lock is treated as a live peer's and not stolen + new FileTokenStore(dir, 200, 600_000).save(keyFor(server), + new PersistedToken("OLD-ACCESS", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000)); + // a peer holds the per-identity lock: getToken() must wait out only its short acquire budget, + // then degrade to a lock-free refresh rather than stall the flush path or fail + final int acquireBudgetMillis = 200; + Path lock = dir.resolve(keyFor(server).hash() + ".lock"); + Files.createFile(lock); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore( + new FileTokenStore(dir, acquireBudgetMillis, 600_000)).build()) { + long start = System.currentTimeMillis(); + Assert.assertEquals("ACCESS-2", auth.getToken()); + long elapsed = System.currentTimeMillis() - start; + // Bounded on BOTH sides, because one side alone cannot tell a degrade from an acquire. + // Below: the whole acquire budget must have been spent polling for a lock this call never + // gets - a run that skipped the wait (or took the lock) returns in single-digit millis. + // The peer's lock is empty, so only the 5s EMPTY_LOCK_STEAL_GRACE_MILLIS governs a steal, + // and a 200ms budget cannot reach it: the wait is deterministic, not racy. + Assert.assertTrue("getToken must wait out the acquire budget before degrading, was " + elapsed, + elapsed >= acquireBudgetMillis); + // Above: budget + one loopback refresh, with a wide margin. The old bound was 10s, which + // a regression to the 3s DEFAULT acquire budget would have sailed through. + Assert.assertTrue("getToken must degrade promptly, not stall, was " + elapsed, elapsed < 2_000); + } + // the definitive degrade-vs-acquire evidence: had getToken() acquired (or stolen) the lock, it + // would have deleted the file on release + Assert.assertTrue("the peer's lock must be left exactly where it was", Files.exists(lock)); + Assert.assertEquals("device flow must not run; getToken degrades to a lock-free refresh", 0, device.get()); + Assert.assertTrue("the refresh must hit the token endpoint", token.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testNoStorePersistsNothing() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = baseBuilder(server).build()) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals("ACCESS-1", auth.getToken()); + } + }); + } + + @Test(timeout = 30_000) + public void testNonRotatingRefreshDoesNotRewrite() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + if (body.contains("grant_type=refresh_token")) { + // a non-rotating provider returns no new refresh token + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, null, 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.stored = new PersistedToken("OLD", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-2", auth.signIn()); + } + Assert.assertEquals("an unchanged refresh token must not rewrite the file", 0, fake.saves.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshUnderLockAdoptsPeerTokenAndSkipsNetwork() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // our own (expired) entry: adopted on load, leaving us in sync with what we last persisted + fake.stored = new PersistedToken("OLD-ACCESS", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000); + // a peer refreshes and writes a fresh, still-valid entry while we hold the cross-process lock + fake.peerInstallsOnLock = new PersistedToken("PEER-ACCESS", null, "REFRESH-2", System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("must adopt the peer's fresh token from inside the lock", "PEER-ACCESS", auth.signIn()); + } + Assert.assertEquals("a peer's still-valid token must be served without a token-endpoint call", 0, token.get()); + Assert.assertEquals("device flow must not run", 0, device.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshUnderLockKeepsLiveRefreshTokenWhenPeerEntryOmitsIt() throws Exception { + // regression: a peer (or cross-language client, or a tampered file) persists a valid-but-expired served + // token with NO refresh_token while we hold a live refresh token in memory. The coordinated re-read must + // keep our refresh token, not null it - nulling it made tryRefresh() urlEncode(null) and throw an + // uncaught NPE that aborted getToken()/signIn() instead of degrading. With the fix REFRESH-1 is kept and + // the refresh succeeds. + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (path.startsWith(DEVICE_PATH)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + // the token endpoint honours the kept refresh token and returns a fresh access token + return MockOidcServer.json(200, tokenJson("REFRESHED-ACCESS", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + long now = System.currentTimeMillis(); + // our own entry, adopted on load: an expired served token carrying REFRESH-1 + fake.stored = new PersistedToken("OLD-ACCESS", null, "REFRESH-1", now - 60_000, 300_000); + // a peer overwrites the file with a valid-but-expired served token and NO refresh_token (the + // frozen on-disk format permits omitting it) while we hold the cross-process lock + fake.peerInstallsOnLock = new PersistedToken("PEER-ACCESS", null, null, now - 60_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("the live refresh token must be kept and used, not nulled into an NPE", + "REFRESHED-ACCESS", auth.getToken()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshUnderLockResavesKeptRefreshTokenWhenPeerEntryOmitsIt() throws Exception { + // the other half of the NPE fix: when the coordinated re-read adopts a peer entry that omits the refresh + // token, adopt() keeps the live refresh token AND records that the file carried none + // (lastPersistedRefreshToken=null). The refresh that follows must therefore RE-SAVE the kept token, so a + // restart still finds it on disk. If adopt() instead marked the kept token as already-persisted, the save + // would be skipped and the refresh token would silently vanish from disk, forcing a needless re-prompt. + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (path.startsWith(DEVICE_PATH)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + // non-rotating refresh: the same REFRESH-1 comes back, so ONLY the null-vs-REFRESH-1 lastPersisted + // bookkeeping (not a token change) decides whether persistIfRotated re-saves + return MockOidcServer.json(200, tokenJson("REFRESHED-ACCESS", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + long now = System.currentTimeMillis(); + // our own entry, adopted on load: an expired served token carrying REFRESH-1 + fake.stored = new PersistedToken("OLD-ACCESS", null, "REFRESH-1", now - 60_000, 300_000); + // a peer overwrites the file with an expired served token and NO refresh_token while we hold the + // lock; the re-read adopts it, keeps REFRESH-1, and records that the file carried no refresh token + fake.peerInstallsOnLock = new PersistedToken("PEER-ACCESS", null, null, now - 60_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("REFRESHED-ACCESS", auth.getToken()); + } + Assert.assertTrue("the kept refresh token must be re-saved (the file carried none), not skipped as already-persisted", + fake.saves.get() >= 1); + Assert.assertNotNull("the re-saved entry must exist", fake.stored); + Assert.assertEquals("the re-saved entry must carry the kept refresh token", "REFRESH-1", fake.stored.getRefreshToken()); + } + }); + } + + @Test(timeout = 30_000) + public void testRestartRefreshesExpiredTokenSkippingDeviceFlow() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + // seed an already-expired access token plus a valid refresh token, as if persisted before a restart + new FileTokenStore(dir).save(keyFor(server), + new PersistedToken("OLD-ACCESS", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000)); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + Assert.assertEquals("ACCESS-2", auth.signIn()); + } + Assert.assertEquals("device flow must not run; a silent refresh suffices", 0, device.get()); + Assert.assertTrue("the token endpoint must be hit for the refresh", token.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testRestartServesPersistedIdTokenWithGroupsInToken() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + token.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + try (OidcDeviceAuth first = baseBuilder(server).groupsInToken(true).tokenStore(new FileTokenStore(dir)).build()) { + // with groups encoded in the token, signIn() serves the id token, and that is what persists + Assert.assertEquals("ID-1", first.signIn()); + } + Assert.assertEquals(1, device.get()); + // the persisted entry must record the id-token identity, so an access-token-mode client rejects it + String json = new String(Files.readAllBytes(dir.resolve(keyForGroups(server).hash() + ".json")), StandardCharsets.UTF_8); + Assert.assertTrue("file must record groups_in_token=true: " + json, json.contains("\"groups_in_token\":true")); + device.set(0); + token.set(0); + + // a restart over the same store serves the persisted id token with no network + try (OidcDeviceAuth restarted = baseBuilder(server).groupsInToken(true).tokenStore(new FileTokenStore(dir)).build()) { + Assert.assertEquals("ID-1", restarted.signIn()); + } + Assert.assertEquals("device flow must not run on restart", 0, device.get()); + Assert.assertEquals("a valid persisted id token needs no token-endpoint call", 0, token.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRestartServesPersistedTokenWithoutNetwork() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + try (OidcDeviceAuth first = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + Assert.assertEquals("ACCESS-1", first.signIn()); + } + Assert.assertEquals(1, device.get()); + device.set(0); + token.set(0); + + // a new instance over the same store mimics a restart: it serves the persisted (still valid) + // token with no calls to either endpoint + try (OidcDeviceAuth restarted = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + Assert.assertEquals("ACCESS-1", restarted.signIn()); + } + Assert.assertEquals("device flow must not run on restart", 0, device.get()); + Assert.assertEquals("a valid persisted token needs no token-endpoint call", 0, token.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRotatingRefreshRewritesStore() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.stored = new PersistedToken("OLD", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-2", auth.signIn()); + } + Assert.assertEquals(0, device.get()); + Assert.assertEquals("the coordinated refresh must run through the store's cross-process lock", 1, fake.locks.get()); + Assert.assertEquals("a rotated refresh token must be persisted", 1, fake.saves.get()); + Assert.assertEquals("REFRESH-2", fake.stored.getRefreshToken()); + } + }); + } + + @Test(timeout = 30_000) + public void testSaveFailureIsNonFatal() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + FakeTokenStore fake = new FakeTokenStore(); + fake.failSave = true; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + // the store's save throws, but the failure is swallowed (warned best-effort, then ignored) and + // the sign-in still yields the valid in-memory token + Assert.assertEquals("ACCESS-1", auth.signIn()); + } + // the save was actually attempted, so the throwing path was exercised rather than skipped + Assert.assertTrue("the token store save must have been attempted", fake.saves.get() >= 1); + }); + } + + @Test(timeout = 30_000) + public void testSaveFailureThenRefreshDoesNotReplayRevokedToken() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicBoolean refresh1Consumed = new AtomicBoolean(); + // a rotating identity provider: REFRESH-1 mints REFRESH-2 once (and is then revoked, so replaying + // it is rejected); REFRESH-2 mints REFRESH-3. The first refreshed token is short-lived, forcing a + // second refresh while the rotated REFRESH-2 is still only in memory (every save fails). + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + if (body.contains("refresh_token=REFRESH-2")) { + return MockOidcServer.json(200, tokenJson("ACCESS-3", null, "REFRESH-3", 3600)); + } + if (body.contains("refresh_token=REFRESH-1")) { + if (refresh1Consumed.compareAndSet(false, true)) { + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 1)); + } + // a rotated-away refresh token is revoked: replaying it must be rejected + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.failSave = true; // every persist fails, so the rotated REFRESH-2 never reaches disk + fake.stored = new PersistedToken("OLD-ACCESS", null, "REFRESH-1", System.currentTimeMillis() - 60_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + // first refresh rotates REFRESH-1 -> REFRESH-2 (save fails, so disk still says REFRESH-1) + Assert.assertEquals("ACCESS-2", auth.signIn()); + // let the short-lived access token expire so the next call refreshes again + Thread.sleep(1_200); + // the second refresh must use the in-memory REFRESH-2, not re-read the stale (now revoked) + // REFRESH-1 from disk - otherwise the replay is rejected and we are forced to re-prompt + Assert.assertEquals("ACCESS-3", auth.signIn()); + } + Assert.assertEquals("a swallowed save must not force the device flow on the next refresh", 0, device.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testStoreThrowingBeforeTheActionDegradesToOneUncoordinatedRefresh() throws Exception { + assertMemoryLeak(() -> { + // TokenStore is a user-implemented SPI and persistence is documented best-effort, but inLock was + // called bare: a store that threw took the whole sign-in down with it and refreshed nothing, even + // though the client held a perfectly good refresh token. The degrade is a single uncoordinated + // refresh - exactly one, because the lock exists to stop a rotating refresh token being POSTed + // twice, and a reuse-detecting provider answers a replay by revoking the whole family. + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-REFRESHED", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.stored = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", System.currentTimeMillis() - 1, 300_000); + fake.throwBeforeAction = new RuntimeException("LOCK-DOWN"); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("a throwing store must not fail a sign-in it cannot help with", + "ACCESS-REFRESHED", auth.getToken()); + } + Assert.assertEquals("the degrade must still refresh", 1, token.get()); + Assert.assertEquals("the interactive flow must not be needed", 0, device.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testStoreThrowingAfterTheActionKeepsTheCompletedRefresh() throws Exception { + assertMemoryLeak(() -> { + // The mirror case, and the one where a blind retry does real damage. The store threw on the way + // OUT - releasing its lock, closing a handle - so the refresh already happened and the token is + // live. Re-running it would be the duplicate POST of a rotating refresh token the lock exists to + // prevent, and propagating would tell the caller a completed sign-in failed. + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-1", "REFRESH-1", "ACCESS-REFRESHED", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + fake.stored = new PersistedToken("ACCESS-STALE", null, "REFRESH-1", System.currentTimeMillis() - 1, 300_000); + fake.throwAfterAction = new RuntimeException("RELEASE-FAILED"); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("a completed refresh must be reported, not undone by a bookkeeping throw", + "ACCESS-REFRESHED", auth.getToken()); + } + Assert.assertEquals("the refresh must not be replayed after it already completed", + 1, token.get()); + Assert.assertEquals("the interactive flow must not be needed", 0, device.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testStoreLoadedAtMostOncePerInstance() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a valid persisted token, so every getToken()/signIn() is a cache hit + fake.loadReturns = new PersistedToken("ACCESS-1", null, "REFRESH-1", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + auth.getToken(); + auth.signIn(); + auth.getToken(); + Assert.assertEquals("the store must be read at most once per instance, not on every call", + 1, fake.loads.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedBareJsonNullServedTokenIsRefusedNotServedAsBearerNull() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-FRESH", "REFRESH-FRESH", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + // Start from a file a conforming writer produced, so the fingerprint and the file name are + // exactly right, then replace the served token with a BARE JSON null - the one encoding + // design/oidc-token-persistence.md forbids, and the natural output of + // json.dumps({"access_token": None}) in a peer client sharing this store. + new FileTokenStore(dir).save(keyFor(server), + new PersistedToken("ACCESS-PLANTED", null, "REFRESH-1", + System.currentTimeMillis() + 300_000, 300_000)); + Path file = dir.resolve(keyFor(server).hash() + ".json"); + String conforming = new String(Files.readAllBytes(file), StandardCharsets.UTF_8); + Assert.assertTrue("the writer must emit a present token as a QUOTED string, or this test is " + + "not planting what it thinks: " + conforming, + conforming.contains("\"access_token\":\"ACCESS-PLANTED\"")); + Files.write(file, conforming + .replace("\"access_token\":\"ACCESS-PLANTED\"", "\"access_token\":null") + .getBytes(StandardCharsets.UTF_8)); + + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + // JsonLexer reports a bare null and a quoted "null" identically, so the entry reaches + // adopt() as the four characters "null" - non-blank, printable ASCII, and with a + // fingerprint that matches, so nothing before adopt() turns it away. Served, it becomes + // "Bearer null", which the server answers with 401; and because the persisted expiry is + // still valid, getToken() would go on serving it rather than refreshing, so the producer + // 401s with nothing naming the cause until the expiry lapses. + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals("a bare JSON null must never be served as the credential", + "null", result); + Assert.assertNotEquals("null", auth.getToken()); + Assert.assertEquals("Bearer ACCESS-FRESH", auth.getAuthorizationHeaderValue()); + } + Assert.assertTrue("a bare JSON null must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedBlankServedTokenRejectedOnLoad() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a tampered persisted entry with a BLANK (whitespace-only) served token is NOT isEmpty() and + // passes hasOnlyTokenChars vacuously (space is 0x20), yet is served as a blank "Bearer " header + // the server only answers with 401 - so adopt() must reject it (via Chars.isBlank, matching the + // sender's own HttpTokenProvider.validateToken) and fall back to the device flow, not wedge on it + fake.loadReturns = new PersistedToken(" ", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals(" ", result); + } + Assert.assertTrue("a rejected blank persisted token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedEmptyServedTokenRejectedOnLoad() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a tampered persisted entry with an EMPTY served token passes hasOnlyTokenChars vacuously but + // would be served as a blank "Bearer " header; adopt() must reject it (not serve "") and fall + // back to the device flow, exactly like a control-char token + fake.loadReturns = new PersistedToken("", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals("", result); + } + Assert.assertTrue("a rejected empty persisted token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedFarFutureExpiryIsBoundedNotTrustedForever() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-FRESH", "REFRESH-FRESH", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a tampered entry claims the access token never expires. adopt() must clamp the trust window + // to MAX_EXPIRES_IN_SECONDS rather than copy the far-future expiry verbatim, and the clamp + // arithmetic (now + maxLife, Math.min over the persisted value) must not overflow on + // Long.MAX_VALUE. Within the clamped hour the token is still valid, so it is served with no + // network. + fake.loadReturns = new PersistedToken("ACCESS-1", null, "REFRESH-1", Long.MAX_VALUE, Long.MAX_VALUE); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("a still-valid persisted token is served within the clamped window", "ACCESS-1", auth.signIn()); + // assert the clamp actually bounds the trust window, not merely that it avoids overflow: + // a verbatim copy of the Long.MAX_VALUE expiry would still pass the assertion above but + // fail these. MAX_EXPIRES_IN_SECONDS is 3600, so the window is at most one hour from now. + long maxLifeMillis = 3_600_000L; + Assert.assertTrue("a tampered far-future expiry must be clamped to <= now + 1h", + readPrivateLong(auth, "expiresAtMillis") <= System.currentTimeMillis() + maxLifeMillis); + Assert.assertTrue("a tampered ttl must be clamped to <= 1h", + readPrivateLong(auth, "tokenTtlMillis") <= maxLifeMillis); + } + Assert.assertEquals("no device flow for a valid persisted token", 0, device.get()); + Assert.assertEquals("no token-endpoint call for a valid persisted token", 0, token.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedFarPastExpiryIsNotServedAsValid() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-FRESH", "REFRESH-FRESH", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a tampered entry claims an absurd, far-PAST expiry near Long.MIN_VALUE. adopt() clamps the + // expiry to [0, now + maxLife]; flooring at 0 is what keeps the validity check + // (now < expiresAtMillis - skew) underflow-safe - without the floor a near-Long.MIN_VALUE + // expiry wraps that subtraction to a huge positive and would serve the garbage-expiry token as + // valid forever. It must instead read as expired and fall back to a silent refresh. + fake.loadReturns = new PersistedToken("ACCESS-1", null, "REFRESH-1", Long.MIN_VALUE, Long.MIN_VALUE); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + String result = auth.signIn(); + Assert.assertEquals("a far-past expiry must not be served; the refresh token supplies a fresh one", "ACCESS-2", result); + Assert.assertNotEquals("a garbage-expiry token must never be served as valid", "ACCESS-1", result); + } + Assert.assertEquals("a valid refresh token needs no device flow", 0, device.get()); + Assert.assertTrue("the expired persisted token must trigger a token-endpoint refresh", token.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedFileWithCrlfTokenFallsBackToDeviceFlow() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = countingHandler(device, token, "ACCESS-FRESH", "REFRESH-FRESH", "ACCESS-2", "REFRESH-2"); + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + // a genuine on-disk file (valid fingerprint) whose served token carries CR/LF: the JSON writer + // escapes it and the lexer decodes it back to real control bytes on load, so adopt() must + // reject it and fall back rather than route a header-injecting credential onto the wire + new FileTokenStore(dir).save(keyFor(server), + new PersistedToken("AC\r\nCESS", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000)); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(new FileTokenStore(dir)).build()) { + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals("AC\r\nCESS", result); + } + Assert.assertTrue("a rejected on-disk token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedIdTokenRejectedOnLoadWithGroupsInToken() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + token.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", "ID-FRESH", "REFRESH-FRESH", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + Path dir = storeDir(); + // groups-in-token mode serves the ID token, so adopt() must validate the ID token, not the access + // token. A genuine on-disk file (valid groups fingerprint) with a CLEAN access token but a CR/LF + // id token must be rejected and fall back to the device flow, never routing the tampered id token + // onto the wire. A bug that validated the access token would accept this entry and serve "I\r\nD". + new FileTokenStore(dir).save(keyForGroups(server), + new PersistedToken("ACCESS-CLEAN", "I\r\nD", "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000)); + try (OidcDeviceAuth auth = baseBuilder(server).groupsInToken(true).tokenStore(new FileTokenStore(dir)).build()) { + String result = auth.signIn(); + Assert.assertEquals("ID-FRESH", result); + Assert.assertNotEquals("I\r\nD", result); + } + Assert.assertTrue("a rejected on-disk id token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedServedTokenRejectedOnLoad() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // a tampered persisted access token carrying CR/LF must never be served + fake.loadReturns = new PersistedToken("AC\r\nCESS", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals("AC\r\nCESS", result); + } + Assert.assertTrue("a rejected persisted token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testTamperedServedTokenWithNonAsciiRejectedOnLoad() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // hasOnlyTokenChars rejects a non-ASCII char (> 0x7e), not just a control char: a persisted served + // token carrying one (here U+00E9) is the byte the ASCII Authorization-header writer would + // truncate, so adopt() must reject the entry and fall back rather than serve a corrupt credential + fake.loadReturns = new PersistedToken("ACC\u00e9SS", null, "REFRESH-1", System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + String result = auth.signIn(); + Assert.assertEquals("ACCESS-FRESH", result); + Assert.assertNotEquals("ACC\u00e9SS", result); + } + Assert.assertTrue("a rejected non-ASCII persisted token must fall back to the device flow", device.get() >= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testPersistedEntryWithoutServedTokenStillRefreshes() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + AtomicInteger token = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + token.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-REFRESHED", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // An entry with a good refresh token but no SERVED kind is reachable: under + // groupsInToken=false a grant that returns only an id_token has storeTokens null the access + // token, and persistIfRotated writes the entry anyway - and a cross-language peer can produce + // the same shape. adopt() used to discard such an entry whole, throwing away the refresh + // token, which is the one thing persistence exists to preserve. The restart must therefore + // spend one silent refresh, not send a human back through the device flow. + fake.loadReturns = new PersistedToken(null, "ID-1", "REFRESH-1", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-REFRESHED", auth.signIn()); + } + Assert.assertEquals("the persisted refresh token must be spent on a silent refresh", + 1, token.get()); + Assert.assertEquals("a usable persisted refresh token must not force the device flow", + 0, device.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testTransientStoreLoadFailureIsRetriedNotLatched() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // The first read fails transiently - the shape a carried interrupt flag produces, since + // FileChannel is an InterruptibleChannel and throws ClosedByInterruptException on a thread + // that merely carries the flag. Latching "already attempted" on that failure disables + // persistence for the whole life of the instance, so a process holding a perfectly good + // refresh token on disk re-runs the interactive device flow instead - a hard failure for the + // headless getToken() consumer persistence exists for, not a degraded one. Only a read that + // COMPLETES (even yielding nothing) is a definitive answer worth latching. + fake.failLoadTimes = 1; + fake.loadReturns = new PersistedToken("ACCESS-PERSISTED", null, "REFRESH-1", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + try { + auth.getToken(); + Assert.fail("the first call must report no usable token after the read failed"); + } catch (OidcAuthException expected) { + // the read threw, so nothing was adopted and there is no token to serve yet + } + Assert.assertEquals("the failed read must not be retried within one call", 1, fake.loads.get()); + + // the store recovers: the very next call must re-read it and serve the persisted token + Assert.assertEquals("ACCESS-PERSISTED", auth.getToken()); + Assert.assertEquals("a failed read must leave the store re-readable", 2, fake.loads.get()); + Assert.assertEquals("a recovered store must not force the interactive device flow", + 0, device.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testRepeatedStoreLoadFailureIsThrottledNotRetriedOnEveryCall() throws Exception { + assertMemoryLeak(() -> { + // A store that never becomes readable - a chmod or uid mismatch in a container, EIO/ESTALE on an + // NFS home - must not cost a blocking read on every call. maybeLoadFromStore() runs on the + // getToken() path AHEAD of the cache check, and getToken() runs once per ILP flush, so without a + // back-off the producer thread paid a file open, two stack trace fills and a WARN line per flush, + // forever, while holding this instance's lock. + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // never recovers, unlike the single transient fault above + fake.failLoadTimes = Integer.MAX_VALUE; + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + for (int i = 0; i < 25; i++) { + try { + auth.getToken(); + Assert.fail("an unreadable store leaves no token to serve"); + } catch (OidcAuthException expected) { + // every call still reports the failure - only the store read is rate-limited + } + } + // two reads, not 25: the first failure, plus the free retry it arms so a one-shot fault + // still recovers at once. The second failure arms the real back-off, which the remaining + // 23 calls run inside of. + Assert.assertEquals("25 getToken() calls must not mean 25 store reads", 2, fake.loads.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testARotatedRefreshWithNoServedKindIsNotPersistedAsAnUnadoptableEntry() throws Exception { + assertMemoryLeak(() -> { + // adopt() rejects a refresh token carried with NEITHER token kind, treating it as positive + // evidence of a foreign writer - an attacker who can write the store dropping in their own + // refresh token. That reasoning only holds while this client cannot produce the shape. + // + // It can. Under groupsInToken the served kind is the id token, so a stored entry carrying only + // an access token takes adopt()'s served-kind-absent branch, which nulls BOTH kinds and keeps + // the refresh token. A refresh that then rotates the refresh token but still returns no id + // token reaches adoptRotatedRefreshToken() -> persistIfRotated() with both null, and writing + // that snapshot leaves a file this client refuses for the life of the entry: every restart + // re-runs the device flow over a refresh token sitting on disk, which for a headless + // getToken() consumer is a hard failure. + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + // rotates the refresh token, still no id_token: the grant the branch above exists for + return MockOidcServer.json(200, tokenJson("ACCESS-NEW", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + long now = System.currentTimeMillis(); + PersistedToken seeded = new PersistedToken("ACCESS-OLD", null, "REFRESH-1", + now + 300_000, 300_000); + fake.stored = seeded; + fake.loadReturns = seeded; + try (OidcDeviceAuth auth = baseBuilder(server).groupsInToken(true).tokenStore(fake).build()) { + try { + // no id token anywhere: the seeded entry has none and the refresh does not produce + // one, so this necessarily fails - the point is what it leaves on disk + auth.getToken(); + Assert.fail("groupsInToken with no id token must not yield a served token"); + } catch (OidcAuthException expected) { + // expected: selectToken() reports the missing served kind + } + Assert.assertEquals("the refresh must have run, or this test proves nothing about what " + + "adoptRotatedRefreshToken() persists", 0, device.get()); + + PersistedToken after = fake.stored; + Assert.assertNotNull("the pre-existing entry must not be replaced by nothing", after); + Assert.assertFalse("the client must never persist the one shape adopt() rejects as " + + "foreign: a refresh token with neither token kind", + after.getAccessToken() == null && after.getIdToken() == null); + } + } + }); + } + + @Test(timeout = 30_000) + public void testARecoveredStoreReadDoesNotRevertACompletedSignIn() throws Exception { + assertMemoryLeak(() -> { + // maybeLoadFromStore() deliberately leaves its latch UNSET when a read THROWS, so a transient + // fault is retried rather than disabling persistence for the life of the instance. But it runs + // at the top of getToken(), AHEAD of the cache check, and adopt() assigns the served kind, the + // expiry and the ttl unconditionally - with no comparison against what is already in memory. + // + // So a store that is unavailable across signIn() and readable afterwards used to undo it: an + // unmounted home or a container started before its volume attaches fails the read AND the save + // (one root cause, both through ensureDirectory), the human authenticates, and then the next + // getToken() - one per ILP flush - re-reads and installs the PREVIOUS entry over the grant just + // obtained. The failed save is what makes it stick: nothing rewrote the entry to match memory. + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-FRESH", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + long now = System.currentTimeMillis(); + // a PREVIOUS login, still unexpired, so adopt() would take it and getToken() would serve it + fake.loadReturns = new PersistedToken("ACCESS-STALE", null, "REFRESH-STALE", + now + 300_000, 300_000); + fake.failLoadTimes = 1; // the read inside signIn() + fake.failSave = true; // ...and the save that follows it, same root cause + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + Assert.assertEquals("ACCESS-FRESH", auth.signIn()); + Assert.assertEquals("the device flow must have run", 1, device.get()); + Assert.assertTrue("the save must have been attempted and failed, or this test is not " + + "reproducing the stuck-stale-entry case", fake.saves.get() > 0); + + // the store is readable again from here on (failLoadTimes is spent) + Assert.assertEquals("a recovered store read must not install a previous login over the " + + "grant signIn() just obtained", "ACCESS-FRESH", auth.getToken()); + Assert.assertEquals("ACCESS-FRESH", auth.getToken()); + Assert.assertEquals("once this instance holds its own tokens the store is no longer " + + "authoritative for it and must not be re-read", 1, fake.loads.get()); + } + } + }); + } + + + @Test(timeout = 30_000) + public void testSignInClearsTheStoreLoadBackOff() throws Exception { + assertMemoryLeak(() -> { + // The back-off must never strand a caller: signIn() is the explicit action a user takes to + // recover, and sending a human through the device flow over a refresh token that is sitting on + // disk - readable again by then - is exactly what persistence exists to avoid. + AtomicInteger device = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-FRESH", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + FakeTokenStore fake = new FakeTokenStore(); + // the two getToken() reads fail, arming the back-off; the store is readable again by the time + // signIn() is called + fake.failLoadTimes = 2; + fake.loadReturns = new PersistedToken("ACCESS-PERSISTED", null, "REFRESH-1", + System.currentTimeMillis() + 300_000, 300_000); + try (OidcDeviceAuth auth = baseBuilder(server).tokenStore(fake).build()) { + for (int i = 0; i < 2; i++) { + try { + auth.getToken(); + Assert.fail("the store read failed, so there is no token to serve yet"); + } catch (OidcAuthException expected) { + // the second failure arms the back-off + } + } + Assert.assertEquals(2, fake.loads.get()); + + Assert.assertEquals("ACCESS-PERSISTED", auth.signIn()); + Assert.assertEquals("signIn() must re-read a store the back-off is holding off", + 3, fake.loads.get()); + Assert.assertEquals("a readable store must not force the interactive device flow", + 0, device.get()); + } + } + }); + } + + private static void awaitProcessLockWait(Thread thread) throws InterruptedException { + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (System.nanoTime() - deadlineNanos < 0) { + boolean inFileStoreLock = false; + boolean inInterruptibleAcquire = false; + for (StackTraceElement frame : thread.getStackTrace()) { + if (FileTokenStore.class.getName().equals(frame.getClassName()) + && "inLock".equals(frame.getMethodName())) { + inFileStoreLock = true; + } + if ("lockInterruptibly".equals(frame.getMethodName())) { + inInterruptibleAcquire = true; + } + } + if (thread.getState() == Thread.State.WAITING && inFileStoreLock && inInterruptibleAcquire) { + return; + } + Thread.sleep(5L); + } + Assert.fail("clearCache() never reached FileTokenStore's interruptible process-lock wait"); + } + + private static void awaitOidcCloseLockWait(Thread thread) throws InterruptedException { + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (System.nanoTime() - deadlineNanos < 0) { + boolean inClose = false; + for (StackTraceElement frame : thread.getStackTrace()) { + if (OidcDeviceAuth.class.getName().equals(frame.getClassName()) + && "close".equals(frame.getMethodName())) { + inClose = true; + break; + } + } + if (thread.getState() == Thread.State.WAITING && inClose) { + return; + } + Thread.sleep(5L); + } + Assert.fail("close() never waited for the custom TokenStore.clear() call"); + } + + private static OidcDeviceAuth.Builder baseBuilder(MockOidcServer server) { + return OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .scope("openid") + .allowInsecureTransport(true) + .prompt(challenge -> { + }); + } + + private static MockOidcServer.Handler countingHandler( + AtomicInteger device, AtomicInteger token, + String access, String refresh, String refreshedAccess, String refreshedRefresh + ) { + return (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + device.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthJson()); + } + token.incrementAndGet(); + if (body.contains("grant_type=refresh_token")) { + return MockOidcServer.json(200, tokenJson(refreshedAccess, null, refreshedRefresh, 3600)); + } + return MockOidcServer.json(200, tokenJson(access, null, refresh, 3600)); + }; + } + + private static String deviceAuthJson() { + return "{\"device_code\":\"DEV-CODE\",\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\",\"expires_in\":300,\"interval\":1}"; + } + + private static TokenStoreKey keyFor(MockOidcServer server) { + return new TokenStoreKey( + "questdb", + "http://127.0.0.1:" + server.port() + TOKEN_PATH, + "http://127.0.0.1:" + server.port() + DEVICE_PATH, + "openid", + null, + false); + } + + private static TokenStoreKey keyForGroups(MockOidcServer server) { + return new TokenStoreKey( + "questdb", + "http://127.0.0.1:" + server.port() + TOKEN_PATH, + "http://127.0.0.1:" + server.port() + DEVICE_PATH, + "openid", + null, + true); + } + + private static long readPrivateLong(Object target, String fieldName) throws Exception { + java.lang.reflect.Field field = target.getClass().getDeclaredField(fieldName); + field.setAccessible(true); + return field.getLong(target); + } + + private static String tokenJson(String access, String id, String refresh, int expiresIn) { + StringBuilder sb = new StringBuilder(); + sb.append("{\"token_type\":\"Bearer\",\"expires_in\":").append(expiresIn); + if (access != null) { + sb.append(",\"access_token\":\"").append(access).append('"'); + } + if (id != null) { + sb.append(",\"id_token\":\"").append(id).append('"'); + } + if (refresh != null) { + sb.append(",\"refresh_token\":\"").append(refresh).append('"'); + } + sb.append('}'); + return sb.toString(); + } + + private Path storeDir() { + return temp.getRoot().toPath().resolve("oidc-tokens"); + } + + private static final class FakeTokenStore implements TokenStore { + final AtomicInteger clears = new AtomicInteger(); + final AtomicInteger loads = new AtomicInteger(); + final AtomicInteger locks = new AtomicInteger(); + final AtomicInteger saves = new AtomicInteger(); + final AtomicInteger actionRuns = new AtomicInteger(); + // models a CONFORMANT coordinating store whose lock wait a cancellation abandoned: per TokenStore's + // contract it returns false without running the action AND leaves the interrupt flag set, which is + // preserved as the caller's cancellation signal; OidcDeviceAuth uses action entry to distinguish this + // from a refresh that ran and failed + boolean cancelWaitInsteadOfRunning; + CountDownLatch clearEntered; + final AtomicBoolean clearInterrupted = new AtomicBoolean(); + // number of leading load() calls to fail before the first one is allowed to succeed; models a + // transient store fault (an interrupted channel, a momentary IO error), as opposed to a store that + // simply has nothing to return + int failLoadTimes; + boolean failSave; + boolean interruptAfterAction; + CountDownLatch inLockEntered; + PersistedToken loadReturns; + PersistedToken peerInstallsOnLock; + CountDownLatch releaseInLock; + CountDownLatch releaseClear; + PersistedToken stored; + RuntimeException throwAfterAction; + RuntimeException throwBeforeAction; + + @Override + public void clear(TokenStoreKey key) { + clears.incrementAndGet(); + if (clearEntered != null) { + clearEntered.countDown(); + try { + releaseClear.await(); + } catch (InterruptedException e) { + clearInterrupted.set(true); + Thread.currentThread().interrupt(); + throw new RuntimeException("custom token-store deletion was interrupted", e); + } + } + stored = null; + } + + @Override + public boolean inLock(TokenStoreKey key, CriticalSection action) { + locks.incrementAndGet(); + if (throwBeforeAction != null) { + throw throwBeforeAction; + } + if (cancelWaitInsteadOfRunning) { + Thread.currentThread().interrupt(); + return false; + } + if (inLockEntered != null) { + inLockEntered.countDown(); + boolean interrupted = false; + while (true) { + try { + releaseInLock.await(); + break; + } catch (InterruptedException e) { + // Model a store whose own wait is not cancelled, while preserving the caller's signal. + interrupted = true; + } + } + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + if (peerInstallsOnLock != null) { + // simulate a peer process refreshing and writing a fresh entry while we hold the lock + stored = peerInstallsOnLock; + peerInstallsOnLock = null; + } + actionRuns.incrementAndGet(); + boolean result = action.run(); + if (interruptAfterAction) { + Thread.currentThread().interrupt(); + } + if (throwAfterAction != null) { + // a bookkeeping failure on the way out - releasing the lock, closing a handle - AFTER the + // critical section already completed + throw throwAfterAction; + } + return result; + } + + @Override + public PersistedToken load(TokenStoreKey key) { + loads.incrementAndGet(); + if (failLoadTimes > 0) { + failLoadTimes--; + throw new RuntimeException("token store read failed"); + } + return loadReturns != null ? loadReturns : stored; + } + + @Override + public void save(TokenStoreKey key, PersistedToken token) { + saves.incrementAndGet(); + if (failSave) { + throw new RuntimeException("disk full"); + } + stored = token; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTest.java new file mode 100644 index 000000000..dae0b5753 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTest.java @@ -0,0 +1,4676 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.DeviceAuthorizationChallenge; +import io.questdb.client.cutlass.auth.DeviceCodePrompt; +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import io.questdb.client.cutlass.http.client.Fragment; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.http.client.Response; +import io.questdb.client.cutlass.json.JsonLexer; +import io.questdb.client.cutlass.json.JsonParser; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Os; +import io.questdb.client.std.Unsafe; +import io.questdb.client.std.str.StringSink; +import io.questdb.client.test.tools.NoBrowserLaunch; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.ClassRule; +import org.junit.Test; + +import java.lang.reflect.Field; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.net.InetAddress; +import java.net.ServerSocket; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Supplier; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +public class OidcDeviceAuthTest { + + /** + * Every flow here that reaches the device-code prompt would otherwise pop a real browser tab on a + * developer machine. A class rule rather than a static initializer, so the override is undone + * afterwards instead of leaking into every later class in the surefire JVM. + */ + @ClassRule + public static final NoBrowserLaunch NO_BROWSER = new NoBrowserLaunch(); + + private static final String DEVICE_PATH = "/device"; + private static final JsonParser NOOP_JSON_PARSER = (code, tag, position) -> { + }; + private static final String SETTINGS_PATH = "/settings"; + private static final String TOKEN_PATH = "/token"; + private static final String WELL_KNOWN_PATH = "/.well-known/openid-configuration"; + + @Test(timeout = 30_000) + public void testAccessDeniedSurfacesOauthError() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"error\":\"access_denied\",\"error_description\":\"the user declined\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "the user declined"); + Assert.assertEquals("access_denied", e.getOauthError()); + } + }); + } + + @Test(timeout = 30_000) + public void testAllControlVerificationUriCompleteTreatedAsAbsent() throws Exception { + assertMemoryLeak(() -> { + // a verification_uri_complete that is all control chars is non-empty on the wire but sanitizes to + // empty; it must be treated as absent (null), so the prompt shows no blank "(or open this URL ...)" + // line and the browser launcher is never handed an empty string + String allControl = jsonUnicodeEscape(0x0001) + jsonUnicodeEscape(0x0002) + jsonUnicodeEscape(0x0003); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"verification_uri_complete\":\"" + allControl + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + Assert.assertNull(challenge.getVerificationUriComplete()); + } + }); + } + + @Test(timeout = 30_000) + public void testAllControlVerificationUriRejectedAsIncomplete() throws Exception { + assertMemoryLeak(() -> { + // a verification_uri made entirely of control chars is non-empty on the wire but sanitizes to empty + // - it would display as a blank URL the user cannot open, so the response is rejected as incomplete + // (the valid token below would let an unfixed client proceed to a successful but unusable sign-in) + String allControl = jsonUnicodeEscape(0x0001) + jsonUnicodeEscape(0x0002); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"" + allControl + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "incomplete", + "expected an all-control verification_uri to be rejected as incomplete"); + } + }); + } + + @Test(timeout = 30_000) + public void testAudienceParameterSentToDeviceEndpoint() throws Exception { + assertMemoryLeak(() -> { + // the optional audience builder parameter must be url-encoded into the device authorization request + AtomicReference deviceBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceBody.set(body); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-AUD", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .audience("api://questdb") + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + Assert.assertEquals("ACCESS-AUD", auth.signIn()); + Assert.assertTrue(deviceBody.get(), deviceBody.get().contains("audience=api%3A%2F%2Fquestdb")); + } + }); + } + + @Test(timeout = 30_000) + public void testAudienceSentOnRefresh() throws Exception { + assertMemoryLeak(() -> { + // the audience must also be url-encoded into the refresh request, matching the Python client + AtomicReference refreshBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + refreshBody.set(body); + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 60)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .audience("api://questdb") + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); // force the silent-refresh path on the next call + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertTrue(refreshBody.get(), refreshBody.get().contains("audience=api%3A%2F%2Fquestdb")); + } + }); + } + + @Test(timeout = 30_000) + public void testBuilderIssuerPinAcceptsHostCasingAndImplicitPort() throws Exception { + assertMemoryLeak(() -> { + // the origin pin (isSameOrigin) folds host case (ASCII) and treats an implicit https port as 443, so + // an endpoint differing from the issuer only in host case or an explicit :443 is still same-origin + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://IDP.Example:443/as/device") + .tokenEndpoint("https://idp.example/as/token") + .issuer("https://Idp.Example") + .build() + ) { + // accepted: host-case and implicit-vs-explicit 443 differences do not defeat the origin pin + } + }); + } + + @Test(timeout = 30_000) + public void testBuilderIssuerPinAcceptsMatchingOrigin() throws Exception { + assertMemoryLeak(() -> { + // endpoints that belong to the pinned issuer origin are accepted; only the origin is pinned, so + // the differing paths of the device and token endpoints are fine + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/as/device") + .tokenEndpoint("https://idp.example/as/token") + .issuer("https://idp.example") + .build() + ) { + // accepted: build() did not reject the matching-origin endpoints + } + }); + } + + @Test(timeout = 30_000) + public void testBuilderIssuerPinRejectsOffOriginEndpoints() { + // the token/device endpoints do not belong to the pinned issuer origin; build() must reject them + // rather than send the device code and refresh token outside the trusted issuer + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/device") + .tokenEndpoint("https://idp.example/token") + .issuer("https://other-idp.example") + .build() + ) { + Assert.fail("expected the issuer pin to reject off-origin endpoints"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("does not match the issuer origin")); + } + } + + @Test(timeout = 30_000) + public void testBuilderRejectsMissingRequiredOptions() { + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder().deviceAuthorizationEndpoint("https://h/d").tokenEndpoint("https://h/t").build()) { + Assert.fail("expected clientId validation to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("clientId")); + } + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder().clientId("c").tokenEndpoint("https://h/t").build()) { + Assert.fail("expected deviceAuthorizationEndpoint validation to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("deviceAuthorizationEndpoint")); + } + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder().clientId("c").deviceAuthorizationEndpoint("https://h/d").build()) { + Assert.fail("expected tokenEndpoint validation to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("tokenEndpoint")); + } + } + + @Test(timeout = 30_000) + public void testBuilderRejectsNonPositiveHttpTimeout() { + // every other timing input is clamped; a non-positive HTTP timeout yields an already-expired read + // deadline and an unbounded recv(int), so the setter rejects it (matching Sender.Builder) + for (int bad : new int[]{0, -1}) { + try { + OidcDeviceAuth.builder().httpTimeoutMillis(bad); + Assert.fail("expected httpTimeoutMillis(" + bad + ") to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("httpTimeoutMillis")); + } + } + } + + @Test(timeout = 30_000) + public void testBuilderRejectsSplitOriginEndpoints() { + // the token and device authorization endpoints are on different origins; RFC 8628 co-locates them + // on one authorization server, so build() must refuse to spread the credential POSTs across hosts + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://device.example/device") + .tokenEndpoint("https://token.example/token") + .build() + ) { + Assert.fail("expected split-origin endpoints to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("different origins")); + } + } + + @Test(timeout = 30_000) + public void testChallengeStripsBidiAndZeroWidthFromDisplayFields() throws Exception { + assertMemoryLeak(() -> { + // a hostile or MITM'd IdP smuggles bidi/zero-width formatting into the display fields. Here a + // right-to-left override (U+202E) arrives as a JSON unicode escape, which this client's lexer + // decodes into the real character before it reaches the prompt; a BOM, a zero-width space and a + // bidi isolate arrive the same way. The challenge shown to the user must strip them all, so the + // verification URL a human reads matches the one their browser opens + String evilUri = "https://verify.example/" + jsonUnicodeEscape(0x202E) + "evil"; // RTL override + String evilComplete = "https://verify.example/" + jsonUnicodeEscape(0xFEFF) + "device?x=1"; // BOM + String evilUserCode = "W" + jsonUnicodeEscape(0x200B) + "D" + jsonUnicodeEscape(0x2066) + "JB"; // ZWSP + LRI + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"" + evilUserCode + "\"," + + "\"verification_uri\":\"" + evilUri + "\"," + + "\"verification_uri_complete\":\"" + evilComplete + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + // the bidi/zero-width/BOM characters are removed, the readable text is preserved + Assert.assertEquals("https://verify.example/evil", challenge.getVerificationUri()); + Assert.assertEquals("https://verify.example/device?x=1", challenge.getVerificationUriComplete()); + Assert.assertEquals("WDJB", challenge.getUserCode()); + assertNoUnsafeDisplayChars(challenge.getUserCode()); + assertNoUnsafeDisplayChars(challenge.getVerificationUri()); + assertNoUnsafeDisplayChars(challenge.getVerificationUriComplete()); + } + }); + } + + @Test(timeout = 30_000) + public void testChallengeStripsControlCharactersFromDisplayFields() throws Exception { + assertMemoryLeak(() -> { + // an attacker-influenced device-auth response embeds ANSI/control characters; the challenge + // shown to the user must have them stripped so it cannot rewrite or spoof the terminal + String evilUserCode = "WD\u001b[2JJB"; // ESC clear-screen sequence + String evilUri = "https://verify.example/\r\nFAKE: enter 000"; // CRLF line injection + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"" + evilUserCode + "\"," + + "\"verification_uri\":\"" + evilUri + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + // the control characters are removed, the rest of the value is preserved + Assert.assertEquals("WD[2JJB", challenge.getUserCode()); + Assert.assertEquals("https://verify.example/FAKE: enter 000", challenge.getVerificationUri()); + assertNoControlChars(challenge.getUserCode()); + assertNoControlChars(challenge.getVerificationUri()); + } + }); + } + + @Test(timeout = 30_000) + public void testChallengeStripsLoneSurrogates() throws Exception { + assertMemoryLeak(() -> { + // a hostile IdP smuggles unpaired UTF-16 surrogates into display fields via single backslash-u-XXXX escapes + // the lexer emits verbatim (it does not pair them). codePointAt surfaces a lone surrogate as a + // SURROGATE code point, which the sanitizer must strip - while a legitimate adjacent high+low pair + // (an emoji) that codePointAt reassembles survives. + String loneHigh = jsonUnicodeEscape(0xD83D); // high surrogate, no low half + String loneLow = jsonUnicodeEscape(0xDE00); // low surrogate, no high half + String emoji = jsonUnicodeEscape(0xD83D) + jsonUnicodeEscape(0xDE00); // U+1F600, a valid pair + String evilUserCode = "WD" + loneHigh + "JB"; + String evilUri = "https://verify.example/" + loneLow + "evil" + emoji; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"" + evilUserCode + "\"," + + "\"verification_uri\":\"" + evilUri + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + // the unpaired surrogates are removed; the readable text and the legitimate emoji survive + Assert.assertEquals("WDJB", challenge.getUserCode()); + Assert.assertEquals("https://verify.example/evil" + new String(Character.toChars(0x1F600)), + challenge.getVerificationUri()); + assertNoUnsafeDisplayChars(challenge.getUserCode()); + assertNoUnsafeDisplayChars(challenge.getVerificationUri()); + } + }); + } + + @Test(timeout = 30_000) + public void testChallengeStripsSupplementaryPlaneFormatChars() throws Exception { + assertMemoryLeak(() -> { + // a hostile IdP smuggles a supplementary-plane (>= U+10000) format char - U+E0001 LANGUAGE TAG, + // an invisible Unicode "tag" character (category Cf) used to hide or spoof text - via a + // surrogate-pair JSON unicode escape the lexer reassembles. A per-UTF-16-unit filter misses it + // (each surrogate half is neither a control nor Cf); the sanitizer must judge it per code point + // and strip it, while leaving a legitimate astral character (an emoji) intact. + String evilTag = jsonUnicodeEscape(0xDB40) + jsonUnicodeEscape(0xDC01); // U+E0001 as a surrogate pair + String emoji = jsonUnicodeEscape(0xD83D) + jsonUnicodeEscape(0xDE00); // U+1F600 grinning face + String evilUserCode = "WD" + evilTag + "JB"; + String evilUri = "https://verify.example/" + evilTag + "evil" + emoji; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"" + evilUserCode + "\"," + + "\"verification_uri\":\"" + evilUri + "\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + // the invisible tag char is removed; the readable text and the legitimate emoji survive + Assert.assertEquals("WDJB", challenge.getUserCode()); + Assert.assertEquals("https://verify.example/evil" + new String(Character.toChars(0x1F600)), + challenge.getVerificationUri()); + assertNoUnsafeDisplayChars(challenge.getUserCode()); + assertNoUnsafeDisplayChars(challenge.getVerificationUri()); + } + }); + } + + @Test(timeout = 30_000) + public void testChunkedTokenResponseParses() throws Exception { + assertMemoryLeak(() -> { + // real IdPs use Transfer-Encoding: chunked; a multi-KB id token split across chunks must parse + String idToken = TestUtils.repeat("a", 3000); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.chunkedJson(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.chunkedJson(200, tokenJson("ACCESS-CHUNKED", idToken, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + // groups-in-token mode serves the id token; it arrived chunked and is 3 KB long + Assert.assertEquals(idToken, auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testClearCacheForcesFreshSignIn() throws Exception { + assertMemoryLeak(() -> { + // clearCache() must drop the cached token AND the refresh token, so the next signIn() runs a + // fresh interactive sign-in (a device-code grant) rather than a silent refresh + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger refreshCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + refreshCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-R", null, "REFRESH-R", 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + auth.clearCache(); + // the next call must run a second device-code sign-in, not a refresh (the refresh token was dropped) + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals("clearCache must force a second interactive sign-in", 2, deviceCalls.get()); + Assert.assertEquals("clearCache must drop the refresh token so no refresh is attempted", 0, refreshCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testClockSkewCappedAtHalfTokenLifetime() throws Exception { + assertMemoryLeak(() -> { + // the fixed 30s clock skew is capped at half the token lifetime (matching the Python client), so a + // short-lived token is served from cache for the first half of its life rather than being treated + // as expired the instant it is issued - which a flat 30s skew would do to any sub-60s token + AtomicInteger refreshCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + refreshCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 10)); // 10s lifetime + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + // a flat 30s skew would mark this 10s token expired immediately (now < expiresAt - 30s is + // false); the lifetime/2 cap (5s) keeps it valid, so the second call is a cache hit, not a refresh + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals("the capped skew kept the short token cached - no refresh", 0, refreshCalls.get()); + + // once the token is genuinely past expiry, signIn() takes the silent-refresh path + expireCachedToken(auth); + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertEquals(1, refreshCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testCloseCancelsInFlightSignIn() throws Exception { + // a sign-in is waiting for the user: the token endpoint keeps returning authorization_pending. + // close() from another caller must abort the in-flight signIn() promptly, instead of letting + // it hold the instance lock and poll until the device code expires + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 10)); + } + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + }; + CountDownLatch polling = new CountDownLatch(1); + AtomicReference outcome = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, challenge -> polling.countDown())) { + Thread signIn = new Thread(() -> { + try { + auth.signIn(); + outcome.set(new AssertionError("signIn() should have been cancelled by close()")); + } catch (Throwable t) { + outcome.set(t); + } + }, "oidc-sign-in"); + signIn.setDaemon(true); + signIn.start(); + // wait until the flow has prompted and is polling, then close from this thread + Assert.assertTrue("the sign-in did not reach the polling stage", polling.await(10, TimeUnit.SECONDS)); + auth.close(); + signIn.join(10_000); + Assert.assertFalse("signIn() did not return promptly after close()", signIn.isAlive()); + Throwable t = outcome.get(); + Assert.assertTrue("expected an OidcAuthException, got " + t, t instanceof OidcAuthException); + Assert.assertTrue(t.getMessage(), t.getMessage().contains("closed")); + } + }); + } + + @Test(timeout = 30_000) + public void testConcurrentSignInStartsSingleSignIn() throws Exception { + assertMemoryLeak(() -> { + // several callers race signIn() on a fresh instance; the synchronized method must serialize + // them so exactly one interactive sign-in runs and the rest get the cached token + AtomicInteger deviceCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-CONCURRENT", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + int workerCount = 4; + CountDownLatch ready = new CountDownLatch(workerCount); + CountDownLatch go = new CountDownLatch(1); + AtomicReference error = new AtomicReference<>(); + String[] tokens = new String[workerCount]; + Thread[] workers = new Thread[workerCount]; + for (int i = 0; i < workerCount; i++) { + final int idx = i; + workers[i] = new Thread(() -> { + ready.countDown(); + try { + go.await(); + tokens[idx] = auth.signIn(); + } catch (Throwable t) { + error.set(t); + } + }, "oidc-signIn-" + i); + workers[i].setDaemon(true); + workers[i].start(); + } + Assert.assertTrue(ready.await(10, TimeUnit.SECONDS)); + go.countDown(); + for (Thread w : workers) { + w.join(10_000); + } + Assert.assertNull("a worker failed: " + error.get(), error.get()); + Assert.assertEquals("only one interactive sign-in must run", 1, deviceCalls.get()); + for (int i = 0; i < workerCount; i++) { + Assert.assertEquals("ACCESS-CONCURRENT", tokens[i]); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDeviceCodeLifetimeClamped() throws Exception { + assertMemoryLeak(() -> { + // a missing or zero expires_in defaults to 600s, and an absurd value is capped at 1800s (matching + // the Python client), so a hostile or buggy provider cannot make the client poll for an absurd + // duration; the clamped value is the one shown to the user (challenge.getExpiresInSeconds()) + AtomicReference shown = new AtomicReference<>(); + MockOidcServer.Handler missingExpiry = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"device_code\":\"DEV\",\"user_code\":\"UC\"," + + "\"verification_uri\":\"https://verify.example/device\",\"interval\":1}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-DEFAULT-TTL", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(missingExpiry); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-DEFAULT-TTL", auth.signIn()); + Assert.assertEquals(600, shown.get().getExpiresInSeconds()); + } + MockOidcServer.Handler absurdExpiry = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 999_999)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-CAPPED-TTL", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(absurdExpiry); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-CAPPED-TTL", auth.signIn()); + Assert.assertEquals(1800, shown.get().getExpiresInSeconds()); + } + }); + } + + @Test(timeout = 30_000) + public void testDeviceEndpointReturnsOauthError() throws Exception { + assertMemoryLeak(() -> { + // the device authorization request itself is rejected (e.g. the client is not allowed) + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(400, "{\"error\":\"invalid_client\",\"error_description\":\"unknown client\"}"); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "unknown client"); + Assert.assertEquals("invalid_client", e.getOauthError()); + } + }); + } + + @Test(timeout = 30_000) + public void testDeviceFlowHappyPath() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + Assert.assertTrue(body, body.contains("client_id=questdb")); + Assert.assertTrue(body, body.contains("scope=openid")); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + // first poll: still pending, second poll: success + Assert.assertTrue(body, body.contains("grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code")); + Assert.assertTrue(body, body.contains("device_code=DEV-CODE")); + if (tokenCalls.getAndIncrement() == 0) { + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals("Bearer ACCESS-1", auth.getAuthorizationHeaderValue()); + Assert.assertEquals(2, tokenCalls.get()); + + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + Assert.assertEquals("WDJB-MJHT", challenge.getUserCode()); + Assert.assertEquals("https://verify.example/device", challenge.getVerificationUri()); + Assert.assertEquals("https://verify.example/device?user_code=WDJB-MJHT", challenge.getVerificationUriComplete()); + } + }); + } + + @Test(timeout = 30_000) + public void testMalformedResponseHeadDuringDiscoveryIsAnOidcAuthException() throws Exception { + assertMemoryLeak(() -> { + // HttpHeaderParser rejects a response head it cannot parse - here a header block past its fixed + // 4096-byte buffer, the shape a WAF or proxy stacking Set-Cookie/CSP produces - by throwing + // HttpException. That is a SIBLING of HttpClientException, not a subclass, so it escaped both of + // fetchJson's catches and left fromQuestDB throwing a type its own javadoc does not name, past + // every caller's catch (OidcAuthException) degrade handler. + MockOidcServer.Handler handler = (method, path, body) -> { + if (SETTINGS_PATH.equals(path)) { + StringBuilder padding = new StringBuilder(); + for (int i = 0; i < 5000; i++) { + padding.append('A'); + } + return MockOidcServer.raw("HTTP/1.1 200 OK\r\n" + + "X-Pad: " + padding + "\r\n" + + "Content-Length: 0\r\n\r\n"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + try { + OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure()).close(); + Assert.fail("an unparseable response head must not gate discovery open"); + } catch (OidcAuthException expected) { + // the documented type; an HttpException escaping here is the regression + } + } + }); + } + + @Test(timeout = 30_000) + public void testMalformedResponseHeadDuringPollingIsTransient() throws Exception { + assertMemoryLeak(() -> { + // The same unparseable head on the TOKEN endpoint, mid-poll. Escaping as HttpException it missed + // postForm's catch and the client.disconnect() with it, so the cached keep-alive connection kept a + // half-read response for the next poll to parse as its own; it also missed pollForToken's + // classification, aborting the whole interactive sign-in on a condition the same loop rides out + // when it arrives as a transport error. One malformed answer must not end a sign-in. + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (tokenCalls.incrementAndGet() == 1) { + StringBuilder padding = new StringBuilder(); + for (int i = 0; i < 5000; i++) { + padding.append('A'); + } + return MockOidcServer.raw("HTTP/1.1 200 OK\r\n" + + "X-Pad: " + padding + "\r\n" + + "Content-Length: 0\r\n\r\n"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-AFTER-RECOVERY", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-AFTER-RECOVERY", auth.signIn()); + Assert.assertTrue("the poll must have retried after the malformed head, on a clean connection", + tokenCalls.get() >= 2); + } + }); + } + + @Test(timeout = 30_000) + public void testNonNumericStatusCodeRejected() throws Exception { + assertMemoryLeak(() -> { + // a hostile or MITM'd identity provider returns a status line whose status-code token carries an + // ANSI escape (the HTTP header parser copies the token verbatim apart from SP/CR/LF). A status code + // is bare digits, so a non-digit byte is a malformed or hostile status line: the client must reject + // it - never echoing its bytes (which could rewrite a terminal or forge a log line) and never + // trusting its leading digit as a 2xx success gate + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}}"); + } + // status code "2[m00": an ANSI reset spliced into the token. The leading '2' would pass a + // first-char success check, but the non-digit bytes must make the client reject the response + return MockOidcServer.raw("HTTP/1.1 2\u001b[m00 OK\r\n" + + "Content-Type: application/json\r\n" + + "Content-Length: 2\r\n" + + "\r\n" + + "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + auth.signIn(); + Assert.fail("expected a malformed status code to be rejected"); + } catch (OidcAuthException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("malformed HTTP status code")); + Assert.assertFalse("raw ESC must not leak into the message: " + msg, msg.indexOf('\u001b') >= 0); + } + } + }); + } + + @Test(timeout = 30_000) + public void testNonNumericStatusCodeRejectedDuringPolling() throws Exception { + assertMemoryLeak(() -> { + // the malformed-status guard must also fire on the token-poll path, where readResponse handles the + // POSTs that carry the device code on every poll (testNonNumericStatusCodeRejected covers the + // device-authorization POST). The device step succeeds, then the token endpoint returns a status + // line whose status-code token splices in an ANSI escape; the client must reject it - never echoing + // its bytes, never trusting its leading '2' as a 2xx success gate + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.raw("HTTP/1.1 2\u001b[m00 OK\r\n" + + "Content-Type: application/json\r\n" + + "Content-Length: 2\r\n" + + "\r\n" + + "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "malformed HTTP status code", + "expected a malformed status code on the poll path to be rejected"); + String msg = e.getMessage(); + Assert.assertFalse("raw ESC must not leak into the message: " + msg, msg.indexOf('\u001b') >= 0); + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryDefaultsScopeToOpenid() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + AtomicReference deviceBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + // settings advertise no scope, so the client must default to "openid" + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}}"); + } + if (DEVICE_PATH.equals(path)) { + deviceBody.set(body); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-SCOPE", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.assertEquals("ACCESS-SCOPE", auth.signIn()); + Assert.assertTrue(deviceBody.get(), deviceBody.get().contains("scope=openid")); + Assert.assertFalse(deviceBody.get(), deviceBody.get().contains("groups")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryRejectsMalformedStatusWithoutEchoingIt() throws Exception { + assertMemoryLeak(() -> { + // The header parser copies the status-line token verbatim apart from SP/CR/LF, so a non-digit + // byte means a malformed or hostile status line. It must not be read as a 2xx by its leading + // digit, and none of it may reach the exception message, which lands in logs and terminals. + // A short all-digit status is malformed too, for the same "leading digit is not the class" + // reason. + // A COMPLETE, otherwise-valid settings body, so the only thing standing between this response + // and a working instance is the status gate. A partial body would fail later on a missing key + // and prove nothing about the status. + for (String statusToken : new String[]{"2\u001b[31m0", "2"}) { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, requestBody) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + String settings = settingsJson(true, true, + server.httpUrl(TOKEN_PATH), server.httpUrl(DEVICE_PATH)); + return MockOidcServer.raw( + "HTTP/1.1 " + statusToken + " OK\r\n" + + "Content-Type: application/json\r\n" + + "Content-Length: " + settings.length() + "\r\n\r\n" + + settings); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + OidcAuthException e = assertOidcFails( + () -> OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure()), + "malformed HTTP status code", + "a malformed status [" + statusToken + "] must not gate discovery open"); + Assert.assertFalse("the raw status must not be echoed: " + e.getMessage(), + e.getMessage().indexOf('\u001b') >= 0); + } + } + }); + } + + @Test(timeout = 30_000) + public void testSettingsUnderErrorStatusNotTrustedAsConfig() throws Exception { + assertMemoryLeak(() -> { + // /settings was parsed without looking at the status, so a body carrying the right keys was + // read as configuration whatever the response claimed to be. A 500 is not a settings document: + // an error envelope, a proxy's branded page or a captive portal could supply the endpoints the + // user then signs in against, and the refresh token is POSTed to. The status gate must refuse + // it before the body is parsed at all. + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(500, + settingsJson(true, true, server.httpUrl(TOKEN_PATH), server.httpUrl(DEVICE_PATH))); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + OidcAuthException e = assertOidcFails( + () -> OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure()), + "did not return its settings", + "a 500 /settings body must not be trusted as OIDC configuration"); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("httpStatus=500")); + } + }); + } + + @Test(timeout = 30_000) + public void testWellKnownUnderErrorStatusNotTrustedAsDiscoveryDoc() throws Exception { + assertMemoryLeak(() -> { + // The same hole on the .well-known fallback, which is what a pinned issuer falls back to when + // /settings advertises no device endpoint. A 404 body is not a discovery document - a + // tenant-not-found stub is exactly the shape that reaches this path - so it must not be able to + // name the token and device endpoints. + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson(true, false, server.httpUrl(TOKEN_PATH), null)); + } + if (WELL_KNOWN_PATH.equals(path)) { + return MockOidcServer.json(404, + wellKnownJson(server.httpUrl(DEVICE_PATH), server.httpUrl(TOKEN_PATH), server.httpUrl(""))); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB( + server.httpUrl(""), insecure().issuer(server.httpUrl("")))) { + Assert.fail("a 404 .well-known body must not be trusted as a discovery document"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), + e.getMessage().contains("did not return an OIDC discovery document")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("httpStatus=404")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryIgnoresArrayWrappedConfig() throws Exception { + assertMemoryLeak(() -> { + // a tampered /settings wraps the config object in an ARRAY - {"config":[{...}]} - so the config + // keys sit inside an array element rather than the trusted top-level "config" object. The parser + // must not surface an array element's object as config (mirroring FileTokenStore's array + // rejection), so OIDC reads as disabled and fromQuestDB fails rather than trusting wrapped config. + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":[{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.scope\":\"openid\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}]}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + assertOidcFails(() -> OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure()), + "OIDC is not enabled", "array-wrapped config must not be trusted as OIDC config"); + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryIgnoresPreferencesKeys() throws Exception { + assertMemoryLeak(() -> { + // the unprivileged-writable "preferences" object tries to poison discovery (flip enabled + // off, flip groups-in-token, inject scope); only the trusted top-level "config" object + // must feed discovery + AtomicReference serverRef = new AtomicReference<>(); + AtomicReference deviceBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.scope\":\"openid\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "},\"preferences.version\":0,\"preferences\":{" + + "\"acl.oidc.enabled\":false," + + "\"acl.oidc.groups.encoded.in.token\":true," + + "\"acl.oidc.scope\":\"INJECTED\"" + + "}}"); + } + if (DEVICE_PATH.equals(path)) { + deviceBody.set(body); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-TRUSTED", "ID-TRUSTED", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + // enabled stayed true (no DoS), groups-in-token stayed false (access token served), + // scope stayed "openid" (no injection) + Assert.assertEquals("ACCESS-TRUSTED", auth.signIn()); + Assert.assertTrue(deviceBody.get(), deviceBody.get().contains("scope=openid")); + Assert.assertFalse(deviceBody.get(), deviceBody.get().contains("INJECTED")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryReadsAudience() throws Exception { + assertMemoryLeak(() -> { + // the audience advertised by /settings (acl.oidc.audience) must be url-encoded into the device + // authorization request, matching the Python client + AtomicReference serverRef = new AtomicReference<>(); + AtomicReference deviceBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.audience\":\"api://questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}}"); + } + if (DEVICE_PATH.equals(path)) { + deviceBody.set(body); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-AUD-D", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.assertEquals("ACCESS-AUD-D", auth.signIn()); + Assert.assertTrue(deviceBody.get(), deviceBody.get().contains("audience=api%3A%2F%2Fquestdb")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryRejectsMissingClientId() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + // OIDC enabled, endpoints advertised, but no client id + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("client id")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryRejectsMissingTokenEndpoint() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + // OIDC enabled with a client id, but no token endpoint + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token endpoint")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testBodyReadAbortsOnItsElapsedDeadline() throws Exception { + assertMemoryLeak(() -> { + // parseBody bounds the WHOLE body read against an untrusted identity provider: a server that + // keeps delivering, slowly, for longer than httpTimeoutMillis must not hold the thread. The bound + // sits on a hot path - getToken() runs once per ILP flush - so losing it stalls ingestion rather + // than failing it. + // + // Two things can end that read and they are NOT interchangeable: the per-call recv bound + // ("timed out reading the chunked response body") and parseBody's own elapsed deadline. Only the + // second one catches a peer whose every individual read SUCCEEDS while the read as a whole runs + // past the budget, and nothing exercised it - a socket-level test races the two bounds and pins + // whichever wins on the day. + // + // Driving parseBody directly removes the race. The Response below hands back an EMPTY fragment + // immediately and forever: every recv succeeds, so the recv bound can never fire; totalBytes + // never grows, so the 4 MiB cap can never fire either; and the lexer is fed nothing, so it + // cannot throw. The elapsed deadline is the only exit, which is exactly the line under test. + Method parseBody = OidcDeviceAuth.class.getDeclaredMethod( + "parseBody", Response.class, JsonLexer.class, JsonParser.class, int.class); + parseBody.setAccessible(true); + + Fragment empty = new Fragment() { + @Override + public long hi() { + return 0; + } + + @Override + public long lo() { + return 0; + } + }; + AtomicInteger reads = new AtomicInteger(); + Response alwaysReady = new Response() { + @Override + public Fragment recv() { + return recv(0); + } + + @Override + public Fragment recv(int timeout) { + reads.incrementAndGet(); + return empty; + } + }; + + try (JsonLexer lexer = new JsonLexer(1024, 1024)) { + long startMillis = System.currentTimeMillis(); + try { + parseBody.invoke(null, alwaysReady, lexer, NOOP_JSON_PARSER, 200); + Assert.fail("a body that never ends must abort on the elapsed deadline"); + } catch (InvocationTargetException e) { + Assert.assertTrue("expected the elapsed-deadline abort, got: " + e.getCause(), + e.getCause() instanceof HttpClientException); + Assert.assertEquals("timed out reading the identity provider response body", + e.getCause().getMessage()); + } + long elapsedMillis = System.currentTimeMillis() - startMillis; + Assert.assertTrue("the deadline must bound the read, not merely end it eventually: " + + elapsedMillis + "ms", elapsedMillis < 10_000); + Assert.assertTrue("every read must have succeeded, or this pinned the recv bound instead", + reads.get() > 0); + } + }); + } + + @Test(timeout = 30_000) + public void testClearCacheWipesTheLexerDecodeBuffers() throws Exception { + assertMemoryLeak(() -> { + // The sibling below covers close(), which cannot see this: close() runs the same sweep and THEN + // does jsonLexer = Misc.free(jsonLexer), so the field is null by the time the reflective walk + // looks and the lexer is garbage either way. clearCache() deliberately keeps the lexer alive - + // the instance stays usable for a later signIn() - so whatever it still holds stays reachable + // from this object. + // + // What it holds is the token itself. The lexer ASSEMBLES every name and value in its own decode + // sinks before a listener ever sees one, and stashes split values in a native cache, so + // TokenResponseParser's copy is not the only copy. Wiping the parsers left the originals + // untouched. JsonLexer.clear() resets parse state only, and StringSink.clear() would just rewind + // the write position anyway. + // + // So a caller who called clearCache() to sign this process out still had the access, id and + // refresh tokens legible on the heap or in native memory - the exact retention wipe() exists to + // close. + String refreshToken = "REFRESH-LEXER-WIPE-ME-" + TestUtils.repeat("r", 3_000); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEVCODE-LEXER\"," + + "\"user_code\":\"USERCODE-LEXER\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300,\"interval\":1}"); + } + // The access token carries a JSON \\u002D escape (which decodes to the '-' already in the + // string), so the lexer resolves it through unescapeSink rather than returning the sink + // verbatim - the SECOND decode buffer JsonLexer.wipe() clears. A plain-concatenated token + // carries no escape, skips unescape(), and leaves unescapeSink empty, so its wipe is never + // exercised and dropping it goes unnoticed. The decoded access-token value keeps the same + // WIPE-ME marker, while the long refresh token below independently exercises the native stash. + // Force the long refresh token across 64-byte HTTP chunks. JsonLexer copies the fragment + // prefix into its native split-value cache, then resets cacheSize to zero after emitting the + // completed value without erasing the allocation. + return MockOidcServer.chunkedJson(200, + tokenJson("ACCESS\\u002DLEXER-WIPE-ME", "ID-LEXER-WIPE-ME", refreshToken, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + OidcDeviceAuth auth = newAuth(server, false, noopPrompt()); + try { + Assert.assertEquals("ACCESS-LEXER-WIPE-ME", auth.signIn()); + // The decode buffers really carry a secret, or clearing them below proves nothing. The + // sink is reused per value, so which one survives is whichever the parse ended on plus + // whatever is still legible in the tail past it - the retention itself, so assert on the + // set rather than on one field's position in the response. + String before = lexerBuffers(auth); + boolean holdsOne = false; + for (String secret : new String[]{ + "ACCESS-LEXER-WIPE-ME", "ID-LEXER-WIPE-ME", "REFRESH-LEXER-WIPE-ME"}) { + holdsOne |= before.contains(secret); + } + Assert.assertTrue("the lexer must hold a parsed token before the wipe, otherwise this " + + "test cannot fail: " + before, holdsOne); + String nativeBefore = lexerNativeCache(auth); + Assert.assertTrue("the chunk-split refresh token must be legible in the lexer's native " + + "stash before the wipe", + nativeBefore.contains("REFRESH-LEXER-WIPE-ME")); + + auth.clearCache(); + + String buffers = lexerBuffers(auth); + for (String secret : new String[]{ + "ACCESS-LEXER-WIPE-ME", "ID-LEXER-WIPE-ME", "REFRESH-LEXER-WIPE-ME"}) { + Assert.assertFalse("clearCache() left \"" + secret + "\" legible in the lexer's " + + "decode buffers", buffers.contains(secret)); + } + String nativeAfter = lexerNativeCache(auth); + Assert.assertFalse("clearCache() left the split refresh token legible in the lexer's " + + "native stash", + nativeAfter.contains("REFRESH-LEXER-WIPE-ME")); + for (int i = 0, n = nativeAfter.length(); i < n; i++) { + Assert.assertEquals("JsonLexer.wipe() must zero the whole native stash allocation", + 0, nativeAfter.charAt(i)); + } + // and nothing else on the instance kept a copy either + assertHoldsNowhere(auth, "ACCESS-LEXER-WIPE-ME"); + assertHoldsNowhere(auth, "REFRESH-LEXER-WIPE-ME"); + } finally { + auth.close(); + } + } + }); + } + + @Test(timeout = 30_000) + public void testCloseWipesCredentialState() throws Exception { + assertMemoryLeak(() -> { + // close() disables every token operation, so nothing it holds can be needed again - yet the + // instance went on holding all of it: the served token and refresh token in their String fields, + // and the raw grant in the sinks that carried it. formSink keeps the last request body, which on + // the refresh path is literally "refresh_token="; the two response parsers keep every + // field of the last response, device code included. All are reused, and clear() only rewinds the + // write position, so a long secret followed by a short write stays legible in the tail. + // + // The walk below is deliberately reflective and generic rather than a list of field names: a sink + // added to this class or to either parser later is covered without anyone remembering to extend + // the test. + AtomicInteger deviceCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEVCODE-WIPE-ME\"," + + "\"user_code\":\"USERCODE-WIPE-ME\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300,\"interval\":1}"); + } + if (body.contains("grant_type=refresh_token")) { + return MockOidcServer.json(200, + tokenJson("ACCESS-REFRESHED-WIPE-ME", "ID-REFRESHED-WIPE-ME", "REFRESH-2-WIPE-ME", 3600)); + } + return MockOidcServer.json(200, + tokenJson("ACCESS-WIPE-ME", "ID-WIPE-ME", "REFRESH-1-WIPE-ME", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + OidcDeviceAuth auth = newAuth(server, false, noopPrompt()); + try { + Assert.assertEquals("ACCESS-WIPE-ME", auth.signIn()); + expireCachedToken(auth); + // spend the refresh token too, so it passes through formSink as a request parameter + Assert.assertEquals("ACCESS-REFRESHED-WIPE-ME", auth.getToken()); + Assert.assertEquals(1, deviceCalls.get()); + // the state is genuinely there before the close - otherwise the sweep below proves nothing + assertHoldsSomewhere(auth, "REFRESH-2-WIPE-ME"); + } finally { + auth.close(); + } + + Assert.assertNull("the served access token must not survive close()", + readField(auth, "accessToken")); + Assert.assertNull("the id token must not survive close()", readField(auth, "idToken")); + Assert.assertNull("the refresh token must not survive close()", readField(auth, "refreshToken")); + Assert.assertNull("the last-persisted refresh token must not survive close()", + readField(auth, "lastPersistedRefreshToken")); + for (String secret : new String[]{ + "ACCESS-WIPE-ME", "ID-WIPE-ME", "REFRESH-1-WIPE-ME", + "ACCESS-REFRESHED-WIPE-ME", "ID-REFRESHED-WIPE-ME", "REFRESH-2-WIPE-ME", + "DEVCODE-WIPE-ME", "USERCODE-WIPE-ME"}) { + assertHoldsNowhere(auth, secret); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDiscoveryTransportFailureDoesNotLeakNativeMemory() throws Exception { + // discoverSettings allocates a JSON lexer (NATIVE_TEXT_PARSER_RSS) and an HTTP client (NATIVE_DEFAULT + // buffers) and frees both in a finally; a transport failure during discovery must not leak either. + // assertMemoryLeak covers EVERY tag - its LeakCheck asserts per-tag equality across the whole + // MemoryTag range, then total equality - so it is the outer guard here rather than something to work + // around. The two explicit tag assertions stay because they name the buffer that leaked, which a + // blanket "total native memory" mismatch does not. + assertMemoryLeak(() -> { + int deadPort; + try (ServerSocket probe = new ServerSocket(0, 1, InetAddress.getLoopbackAddress())) { + deadPort = probe.getLocalPort(); + } // closed now - nothing listens on deadPort + long parserMemBefore = Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS); + long clientMemBefore = Unsafe.getMemUsedByTag(MemoryTag.NATIVE_DEFAULT); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB("http://127.0.0.1:" + deadPort, insecure())) { + Assert.fail("expected discovery to fail against a dead port"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("could not reach the QuestDB server")); + } + Assert.assertEquals("the discovery JSON lexer native buffer leaked", + parserMemBefore, Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS)); + Assert.assertEquals("the discovery HTTP client native buffers leaked", + clientMemBefore, Unsafe.getMemUsedByTag(MemoryTag.NATIVE_DEFAULT)); + }); + } + + @Test(timeout = 30_000) + public void testDuplicateJsonKeysDoNotConcatenate() throws Exception { + assertMemoryLeak(() -> { + // a buggy/hostile IdP repeats a key; the parser must keep the last value, not concatenate it + // onto the first (e.g. AAABBB), which would corrupt the served token and the device code + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV-CODE\"," + + "\"user_code\":\"WRONG\",\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, "{\"token_type\":\"Bearer\",\"expires_in\":3600," + + "\"access_token\":\"AAA\",\"access_token\":\"ACCESS-LAST\"}"); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + // the duplicate access_token resolves to the last value, not "AAAACCESS-LAST" + Assert.assertEquals("ACCESS-LAST", auth.signIn()); + // the duplicate user_code resolves to the last value, not "WRONGWDJB-MJHT" + Assert.assertEquals("WDJB-MJHT", shown.get().getUserCode()); + } + }); + } + + @Test(timeout = 30_000) + public void testEndpointParseRejectsDisplayUnsafeUrl() { + // a url carrying a display-unsafe character is rejected, and the rejection message itself must carry + // none: otherwise a tampered /settings or discovery endpoint url could reorder, hide or forge the + // log line / exception text it lands in. The control-char scan alone does not catch these higher + // code points (bidi, zero-width, BOM, supplementary-plane tag chars), the last scanned per code point + String[] unsafe = { + String.valueOf((char) 0x202E), // right-to-left override + String.valueOf((char) 0x200B), // zero-width space + String.valueOf((char) 0xFEFF), // BOM / zero-width no-break space + new String(Character.toChars(0xE0001)) // U+E0001 LANGUAGE TAG (supplementary-plane format char) + }; + for (int i = 0; i < unsafe.length; i++) { + String marker = unsafe[i]; + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/dev" + marker + "ice") + .tokenEndpoint("https://idp.example/t") + .build() + ) { + Assert.fail("expected the display-unsafe url to be rejected [index=" + i + "]"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("illegal character")); + // the raw unsafe character must not survive into the message + assertNoUnsafeDisplayChars(e.getMessage()); + } + } + } + + @Test(timeout = 30_000) + public void testEndpointParseAcceptsUppercaseScheme() throws Exception { + assertMemoryLeak(() -> { + // RFC 3986 schemes are case-insensitive, so HTTPS/Http must build - matching BrowserLauncher's + // case-insensitive scheme allowlist. (Endpoint.parse lower-cases only ASCII, so a homoglyph scheme + // is still rejected as "expected http or https".) + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("HTTPS://idp.example/device") + .tokenEndpoint("Https://idp.example/token") + .build() + ) { + // accepted: build() did not reject the mixed-case https scheme + } + }); + } + + @Test(timeout = 30_000) + public void testEndpointParseRejectsMalformedUrls() { + // Endpoint.parse rejects malformed endpoint URLs at build time + assertBuildFails("ftp://idp/d", "https://idp/t", "expected http or https"); + assertBuildFails("idp/d", "https://idp/t", "expected a scheme"); + assertBuildFails("https://idp/d", "https://idp:notaport/t", "could not parse the port"); + assertBuildFails("https:///d", "https://idp/t", "the host is empty"); + assertBuildFails("https://[::1]:9000/d", "https://idp/t", "IPv6 literal hosts are not supported"); + // userinfo (user@host or user:pass@host) is unsupported: the HTTP layer would connect to the literal + // "user@host", so reject it rather than mis-resolve it or report a misleading port-parse error + assertBuildFails("https://user@idp/d", "https://idp/t", "userinfo"); + assertBuildFails("https://idp/d", "https://user:pass@idp/t", "userinfo"); + // an out-of-range port (0, negative, or above 65535) is rejected rather than passed to the transport + assertBuildFails("https://idp:99999/d", "https://idp/t", "between 1 and 65535"); + assertBuildFails("https://idp:0/d", "https://idp/t", "between 1 and 65535"); + assertBuildFails("https://idp:-1/d", "https://idp/t", "between 1 and 65535"); + assertBuildFails("https://idp/d", "https://idp:70000/t", "between 1 and 65535"); + // a leading '+' on the port is rejected: Integer.parseInt would read ":+443" as 443, but a real + // authority port is bare digits (a leading '-' is already caught by the range check above) + assertBuildFails("https://idp:+443/d", "https://idp/t", "could not parse the port"); + // a host carrying control characters or whitespace (e.g. a smuggled CR/LF that would inject into the + // outbound Host header) is rejected rather than passed verbatim to the transport + assertBuildFails("https://ho\r\nst/d", "https://idp/t", "illegal character"); + assertBuildFails("https://h\tst/d", "https://idp/t", "illegal character"); + assertBuildFails("https://h st/d", "https://idp/t", "illegal character"); + assertBuildFails("https://idp/d", "https://e\nvil/t", "illegal character"); + // a control character or whitespace in the path or query is rejected too: postForm sends the path + // verbatim on the request line, so a smuggled CR/LF there would inject a header / smuggle a request + assertBuildFails("https://idp/devic\r\ne", "https://idp/t", "illegal character"); + assertBuildFails("https://idp/d", "https://idp/toke\r\nX-Injected:1", "illegal character"); + assertBuildFails("https://idp/d", "https://idp/t?a=b\nc", "illegal character"); + // a fragment (#...) is rejected: pathOnly() strips it before the issuer-path pin while postForm sends + // endpoint.path verbatim on the wire, so folding a "#/../other/token" past the '#' would let a lenient + // server that normalizes '..' resolve the request-target to a path the pin never validated. Fail closed + assertBuildFails("https://idp/realms/acme#/../other/device", "https://idp/realms/acme/token", "fragment"); + assertBuildFails("https://idp/d", "https://idp/realms/acme#/../other/token", "fragment"); + assertBuildFails("https://idp/d#", "https://idp/t", "fragment"); + // a query (?...) is rejected for the same pin-bypass reason: pathOnly() strips it before the issuer-path + // pin while postForm sends endpoint.path - query included - verbatim, so a "?..." the pin never validated + // would still reach the wire. An OIDC device/token endpoint carries its parameters in the request body, + // never the url query, so fail closed (the user-facing verification url may carry one, but it is parsed + // by BrowserLauncher, not Endpoint.parse) + assertBuildFails("https://idp/realms/acme/device?x=/../other", "https://idp/realms/acme/token", "query"); + assertBuildFails("https://idp/d", "https://idp/realms/acme/token?client_id=evil", "query"); + assertBuildFails("https://idp/d?a=b", "https://idp/t", "query"); + // a non-ASCII host is rejected: it would not resolve (the HTTP layer sends the host to the OS resolver + // as raw UTF-8, no IDNA), and equalsIgnoreCase folds several non-ASCII letters onto ASCII (U+0130 -> i, + // U+212A -> k, ...), so a homoglyph host could otherwise pass the origin pin against a pinned issuer + assertBuildFails("https://\u0130dp/d", "https://idp/t", "non-ASCII"); // U+0130, folds to i + assertBuildFails("https://idp/d", "https://\u212Aelvin/t", "non-ASCII"); // U+212A Kelvin, folds to k + // a backslash in the host is rejected: the WHATWG URL spec folds '\' to '/', so a lenient consumer + // could re-split good.com\.evil.com into a different authority + assertBuildFails("https://good.com\\.evil.com/d", "https://idp/t", "backslash"); + } + + @Test(timeout = 30_000) + public void testEscapedDeviceCodeRoundTripsDecoded() throws Exception { + assertMemoryLeak(() -> { + // an IdP that escapes a character in device_code (here a slash) must have it decoded before the + // client posts it back, otherwise the polled device_code never matches what the IdP issued + AtomicReference pollBody = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\\/CODE\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + pollBody.set(body); + return MockOidcServer.json(200, tokenJson("ACCESS-DC", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-DC", auth.signIn()); + // device_code was "DEV\/CODE" in JSON; decoded to "DEV/CODE" and url-encoded as DEV%2FCODE + Assert.assertTrue(pollBody.get(), pollBody.get().contains("device_code=DEV%2FCODE")); + } + }); + } + + @Test(timeout = 30_000) + public void testEscapedErrorDescriptionDecoded() throws Exception { + assertMemoryLeak(() -> { + // an error_description with JSON-escaped characters must be decoded in the exception message, + // not shown with literal backslashes + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"error\":\"access_denied\",\"error_description\":\"it\\\"s a \\/ test\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "it\"s a / test"); + Assert.assertEquals("access_denied", e.getOauthError()); + // the escapes are decoded, not shown literally + Assert.assertFalse(e.getMessage(), e.getMessage().contains("\\/")); + } + }); + } + + @Test(timeout = 30_000) + public void testEscapedVerificationUrlIsUnescapedForDisplay() throws Exception { + assertMemoryLeak(() -> { + // some identity providers JSON-escape forward slashes (PHP json_encode does by default), e.g. + // "https:\/\/...". The challenge shown to the user must decode the escapes, not display literal + // backslashes that break the link + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV-CODE\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https:\\/\\/verify.example\\/device\"," + + "\"verification_uri_complete\":\"https:\\/\\/verify.example\\/device?user_code=WDJB-MJHT\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-ESC", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-ESC", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + Assert.assertEquals("https://verify.example/device", challenge.getVerificationUri()); + Assert.assertEquals("https://verify.example/device?user_code=WDJB-MJHT", challenge.getVerificationUriComplete()); + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoversDeviceEndpointFromIssuer() throws Exception { + assertMemoryLeak(() -> { + // the server advertises a token endpoint but not the device authorization endpoint (today's + // servers); pinning the issuer lets the client discover the device endpoint from the issuer's + // .well-known/openid-configuration document and complete the flow + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson(true, false, server.httpUrl(TOKEN_PATH), null)); + } + if (WELL_KNOWN_PATH.equals(path)) { + return MockOidcServer.json(200, wellKnownJson(server.httpUrl(DEVICE_PATH), server.httpUrl(TOKEN_PATH), server.httpUrl(""))); + } + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-WK", "ID-WK", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + // the issuer is the mock itself, which also serves the .well-known document and the IdP endpoints + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("")))) { + // settings advertise groups.encoded.in.token=true, so signIn() returns the id token + Assert.assertEquals("ID-WK", auth.signIn()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryAcceptsPathIssuerWithTrailingSlash() throws Exception { + assertMemoryLeak(() -> assertFromQuestDbDiscoveryAcceptsTrailingSlashIssuer("/realms/acme/")); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryAcceptsRootIssuerWithTrailingSlash() throws Exception { + assertMemoryLeak(() -> assertFromQuestDbDiscoveryAcceptsTrailingSlashIssuer("/")); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryDocMissingDeviceEndpointRejected() throws Exception { + assertMemoryLeak(() -> { + // discovery runs against the pinned issuer, but the discovery document does not advertise a + // device authorization endpoint (the identity provider lacks the device grant); fail clearly + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson(true, false, server.httpUrl(TOKEN_PATH), null)); + } + // a discovery document with a token endpoint and issuer but no device_authorization_endpoint + return MockOidcServer.json(200, "{" + + "\"issuer\":\"" + server.httpUrl("") + "\"," + + "\"token_endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"" + + "}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("")))) { + Assert.fail("expected discovery to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("device_authorization_endpoint")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryRejectsMismatchedIssuer() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson( + true, + false, + server.httpUrl("/realms/acme/token"), + null + )); + } + if (("/realms/acme" + WELL_KNOWN_PATH).equals(path)) { + // Every endpoint passes the existing origin/path checks. Only the returned issuer reveals + // that this is metadata for a different tenant on the same authorization server. + return MockOidcServer.json(200, wellKnownJson( + server.httpUrl("/realms/acme/device"), + server.httpUrl("/realms/acme/token"), + server.httpUrl("/realms/other") + )); + } + return MockOidcServer.json(500, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + assertOidcFails( + () -> OidcDeviceAuth.fromQuestDB( + server.httpUrl(""), + insecure().issuer(server.httpUrl("/realms/acme")) + ), + "issuer does not exactly match the pinned issuer" + ); + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryRejectsMissingIssuer() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson( + true, + false, + server.httpUrl(TOKEN_PATH), + null + )); + } + if (WELL_KNOWN_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"token_endpoint\":\"" + server.httpUrl(TOKEN_PATH) + "\"," + + "\"device_authorization_endpoint\":\"" + server.httpUrl(DEVICE_PATH) + "\"" + + "}"); + } + return MockOidcServer.json(500, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + assertOidcFails( + () -> OidcDeviceAuth.fromQuestDB( + server.httpUrl(""), + insecure().issuer(server.httpUrl("")) + ), + "does not contain the required issuer" + ); + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbDiscoveryRunsFlow() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson(true, true, server.httpUrl(TOKEN_PATH), server.httpUrl(DEVICE_PATH))); + } + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-D", "ID-D", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + // discovery advertises groups.encoded.in.token=true, so signIn() must return the id token + Assert.assertEquals("ID-D", auth.signIn()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbIssuerPinAcceptsOffOriginDiscoveredEndpoints() throws Exception { + assertMemoryLeak(() -> { + // The Google case: the pinned issuer hosts its discovery document on one origin but serves its + // token and device endpoints on another. /settings advertises neither endpoint, so both are + // discovered from the issuer's own .well-known (a trusted, out-of-band source) and must be accepted + // wherever the issuer hosts them, NOT origin-pinned to the issuer. An endpoint the untrusted + // /settings advertised IS still origin-pinned - see testFromQuestDbIssuerPinRejectsOffOriginAdvertisedEndpoint. + AtomicReference idpRef = new AtomicReference<>(); + AtomicReference issuerRef = new AtomicReference<>(); + // the IdP endpoint host: a different origin (port) than the issuer below + MockOidcServer.Handler idpHandler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OFF", null, null, 3600)); + }; + try (MockOidcServer idp = new MockOidcServer(idpHandler)) { + idpRef.set(idp); + // the QuestDB server doubles as the pinned issuer: it serves /settings (advertising neither + // endpoint) and the .well-known document, which points the device/token endpoints at the + // off-origin idp + MockOidcServer.Handler issuerHandler = (method, path, body) -> { + MockOidcServer endpointHost = idpRef.get(); + MockOidcServer iss = issuerRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.scope\":\"openid\"" + + "}}"); + } + if (WELL_KNOWN_PATH.equals(path)) { + return MockOidcServer.json(200, wellKnownJson( + endpointHost.httpUrl(DEVICE_PATH), endpointHost.httpUrl(TOKEN_PATH), iss.httpUrl(""))); + } + return MockOidcServer.json(404, "{}"); + }; + try (MockOidcServer issuer = new MockOidcServer(issuerHandler)) { + issuerRef.set(issuer); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(issuer.httpUrl(""), insecure().issuer(issuer.httpUrl("")))) { + // the off-origin discovered endpoints are accepted; the device flow completes against them + Assert.assertEquals("ACCESS-OFF", auth.signIn()); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbIssuerPinRejectsOffOriginAdvertisedEndpoint() throws Exception { + assertMemoryLeak(() -> { + // the server advertises both endpoints directly, but they do not belong to the pinned issuer + // origin; the issuer pin must reject them rather than route credentials off the trusted issuer + // (this is the protection against a compromised-but-reachable server redirecting the sign-in) + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, settingsJson(true, true, server.httpUrl(TOKEN_PATH), server.httpUrl(DEVICE_PATH))); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer("https://idp.attacker.example"))) { + Assert.fail("expected the issuer pin to reject the off-origin endpoints"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("is not on the pinned identity-provider origin")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbRejectsCrlfInjectedAdvertisedEndpoint() throws Exception { + assertMemoryLeak(() -> { + // a tampered /settings advertises a token endpoint whose path carries a JSON-escaped CR/LF; the + // lexer decodes it to real control characters, and Endpoint.parse must reject it rather than let + // it inject into the outbound request line (header smuggling against the identity provider) + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + String crlf = jsonUnicodeEscape(0x0d) + jsonUnicodeEscape(0x0a); + String injectedToken = server.httpUrl(TOKEN_PATH) + crlf + "X-Injected:1"; + return MockOidcServer.json(200, settingsJson(true, true, injectedToken, server.httpUrl(DEVICE_PATH))); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected the CR/LF-injected token endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("illegal character")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbRejectsInsecureServerUrl() { + // the default-secure fromQuestDB overload must reject an http:// QuestDB server url (the discovery + // response and the sign-in it bootstraps would travel in cleartext) unless insecure transport is + // explicitly opted in + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB("http://questdb.example:9000")) { + Assert.fail("expected an http server url to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("QuestDB server url")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("insecure http")); + } + } + + @Test(timeout = 30_000) + public void testFromQuestDbRejectsMissingDeviceEndpoint() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + // OIDC enabled, but no device authorization endpoint advertised (an older server) + return MockOidcServer.json(200, settingsJson(true, false, server.httpUrl(TOKEN_PATH), null)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("device authorization endpoint")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testFromQuestDbRejectsOidcDisabled() throws Exception { + assertMemoryLeak(() -> { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(200, settingsJson(false, false, serverRef.get().httpUrl(TOKEN_PATH), null)); + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to fail"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("OIDC is not enabled")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testGarbledRefreshResponseFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // the cached token expires and the refresh hits a transient non-JSON body (e.g. a gateway + // 502 HTML page). The client must fall back to the interactive flow, not propagate the parse + // failure out of signIn() + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // a transient gateway error page instead of a token JSON + return MockOidcServer.json(502, "502 Bad Gateway"); + } + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the cached token is expired and the refresh body is garbled, so the client must re-run + // the interactive flow instead of throwing the parse error + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (initial + fallback)", 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenDoesNotBlockBehindInteractiveSignIn() throws Exception { + assertMemoryLeak(() -> { + // an interactive signIn() is parked polling (authorization_pending), holding the instance + // lock for the whole device-code lifetime. A flush-path getToken() on another thread + // must NOT block behind it - it must fail fast, so a Sender flush is never stalled by a + // concurrent sign-in. (With the old synchronized model it blocked until the code expired.) + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 10)); + } + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + }; + CountDownLatch polling = new CountDownLatch(1); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, challenge -> polling.countDown())) { + Thread signIn = new Thread(() -> { + try { + auth.signIn(); + } catch (Throwable ignore) { + // expected: cancelled by close() at the end of the test + } + }, "oidc-sign-in"); + signIn.setDaemon(true); + signIn.start(); + try { + // wait until the interactive flow has prompted and is polling (it holds the lock now) + Assert.assertTrue("the sign-in did not reach the polling stage", polling.await(10, TimeUnit.SECONDS)); + // getToken() must return control promptly (here: throw), NOT block ~10s until + // the device code expires and signIn() releases the lock + long startNanos = System.nanoTime(); + OidcAuthException e = assertOidcFails(auth::getToken, "in progress", + "expected getToken() to fail fast while a sign-in is in progress"); + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + Assert.assertTrue("getToken() blocked " + elapsedMillis + "ms behind the in-flight sign-in", + elapsedMillis < 2_000); + } finally { + auth.close(); // cancel the in-flight sign-in + signIn.join(10_000); // let the daemon thread unwind before the leak check + } + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenSucceedsWhenCallingThreadIsInterrupted() throws Exception { + assertMemoryLeak(() -> { + // getToken()'s uncontended lock acquire must NOT fail merely because the calling thread carries a + // set interrupt flag. An ILP producer on a pooled/managed thread commonly does (interrupt is the + // standard cancellation signal), and the old timed tryLock threw InterruptedException even on a FREE + // lock and then re-armed the flag, so every getToken() on that thread failed with a valid token + // sitting in the cache. The untimed fast-path acquire fixes it. + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); // seed a valid cached token + + Thread.currentThread().interrupt(); // the calling (producer) thread carries a pending interrupt + try { + // uncontended lock, valid cached token: getToken() must return it, not throw on the interrupt + Assert.assertEquals("ACCESS-1", auth.getToken()); + // and it must not silently swallow the caller's interrupt (the untimed acquire preserves it) + Assert.assertTrue("getToken() must not clear the caller's interrupt flag", + Thread.currentThread().isInterrupted()); + } finally { + Thread.interrupted(); // clear so the flag does not leak into later tests sharing this fork + } + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenSucceedsWhenInterruptedAfterContendedFastPath() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + + // Force the first untimed CAS to miss, modelling contention that ends before the timed arm. + // AQS checks the carried interrupt before that arm attempts its own CAS, so pre-fix getToken() + // threw at 0ms despite the lock now being free and ACCESS-1 still being valid. + MissOnceReentrantLock testLock = new MissOnceReentrantLock(); + Field lockField = OidcDeviceAuth.class.getDeclaredField("lock"); + lockField.setAccessible(true); + lockField.set(auth, testLock); + + Thread.currentThread().interrupt(); + try { + Assert.assertEquals("a valid cached token must survive ended contention", "ACCESS-1", + auth.getToken()); + Assert.assertTrue("getToken() must preserve the caller's interrupt", + Thread.currentThread().isInterrupted()); + Assert.assertEquals("the contended timed arm must have been reached", 1, + testLock.timedTryLockCalls); + Assert.assertEquals("the catch must make one final untimed acquire", 2, + testLock.untimedTryLockCalls); + } finally { + Thread.interrupted(); + } + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenWaitsBehindSilentRefreshInsteadOfFailing() throws Exception { + assertMemoryLeak(() -> { + // When another thread's SILENT REFRESH (not an interactive sign-in) holds the lock, a second + // getToken() must WAIT for that bounded refresh and then serve the freshly refreshed token - NOT + // fail fast. Failing fast would make every concurrent caller sharing one OidcDeviceAuth (the + // documented shared-provider pattern) spuriously throw on each token refresh. The token endpoint + // blocks the refresh response until the test releases it, pinning the lock on the refresher thread + // while the second caller waits for it. (This is the fix for the old fail-fast-on-any-contention + // behaviour: the HttpTokenProvider contract permits a brief wait behind a silent refresh.) + CountDownLatch refreshInFlight = new CountDownLatch(1); + CountDownLatch releaseRefresh = new CountDownLatch(1); + AtomicReference handlerError = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + refreshInFlight.countDown(); + try { + if (!releaseRefresh.await(30, TimeUnit.SECONDS)) { + // on a MockOidcServer thread: JUnit would swallow an Assert.fail here, so record it + // and let the main thread assert on it at the end + handlerError.set("the test never released the refresh within 30s"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); // initial device_code grant + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + auth.signIn(); // sign in once: caches ACCESS-1 and a refresh token + expireCachedToken(auth); // so the refresher thread's getToken() takes the refresh path + // The refresher is not scenery: it is the thread that holds the lock, performs the refresh + // and produces ACCESS-2. Swallowing its failure let the test pass on a run where the refresh + // never happened - the waiter would simply refresh for itself and still see ACCESS-2, so + // every assertion below still held while the contention this test exists for never occurred. + AtomicReference refresherError = new AtomicReference<>(); + AtomicReference refresherResult = new AtomicReference<>(); + Thread refresher = new Thread(() -> { + try { + refresherResult.set(auth.getToken()); + } catch (Throwable t) { + refresherError.set(t); + } + }, "oidc-silent-refresh"); + refresher.setDaemon(true); + refresher.start(); + Assert.assertTrue("the silent refresh did not start", refreshInFlight.await(10, TimeUnit.SECONDS)); + + // a refresh holds the lock now; a second getToken() must WAIT for it, not fail fast + AtomicReference waiterResult = new AtomicReference<>(); + AtomicReference waiterError = new AtomicReference<>(); + Thread waiter = new Thread(() -> { + try { + waiterResult.set(auth.getToken()); + } catch (Throwable t) { + waiterError.set(t); + } + }, "oidc-getToken-waiter"); + waiter.setDaemon(true); + waiter.start(); + // Wait until the waiter is genuinely INSIDE getToken(), read off its own stack. The latch this + // replaced counted down as the first statement of the thread body - BEFORE the call it claimed + // to gate - so it proved only that the thread had been scheduled, and every "still blocked" + // assertion below rested on the sleep that follows instead. + Assert.assertTrue("the waiter never entered getToken()", awaitInside(waiter, "getToken", 10_000)); + try { + // give the waiter time to (wrongly) fail fast if it were going to; while the refresh is held + // it must instead still be blocked INSIDE getToken() - a fail-fast throw would have left that + // frame (and finished the thread) + Thread.sleep(500); + Assert.assertTrue("getToken() must still be blocked behind the peer's refresh, not finished", + waiter.isAlive()); + Assert.assertTrue("getToken() must still be inside the call, waiting out the peer's refresh", + isInside(waiter, "getToken")); + Assert.assertNull("getToken() must not fail fast behind a silent refresh, but threw: " + waiterError.get(), + waiterError.get()); + Assert.assertNull("getToken() must wait, not return, while the peer's refresh is still in flight", + waiterResult.get()); + } finally { + releaseRefresh.countDown(); + waiter.join(10_000); + refresher.join(10_000); + } + // once the peer's refresh completed and released the lock, the waiter served the fresh token + Assert.assertNull("the refresher itself failed, so the wait was never behind a real refresh: " + + refresherError.get(), refresherError.get()); + Assert.assertEquals("the refresher must have completed the refresh it was holding the lock for", + "ACCESS-2", refresherResult.get()); + Assert.assertNull("getToken() must not throw when it waits out a peer's refresh: " + waiterError.get(), + waiterError.get()); + Assert.assertEquals("getToken() must serve the freshly refreshed token after waiting", "ACCESS-2", waiterResult.get()); + Assert.assertNull("the mock server handler must not have reported an error", handlerError.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenRefreshesWhenServedKindIsNullButRefreshTokenExists() throws Exception { + assertMemoryLeak(() -> { + // groupsInToken=true, but the device-code grant returns an access_token + refresh_token and NO + // id_token: signIn() rejects that grant (the served id_token is missing) yet leaves the refresh token + // in memory. A later getToken() must then attempt a silent refresh - which here yields the id_token - + // rather than give up with "no token has been obtained yet". M5: the refresh is no longer foreclosed + // just because the served-kind token is currently null. + AtomicInteger refreshCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + refreshCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-2", "ID-2", "REFRESH-2", 3600)); // now with id_token + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 3600)); // initial grant: no id_token + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { // groupsInToken=true + assertOidcFails(auth::signIn, "no id_token", + "signIn() must reject a grant with no id_token when groups are encoded in the token"); + // the partial grant left a refresh token in memory; getToken() must refresh to obtain the id_token + Assert.assertEquals("ID-2", auth.getToken()); + Assert.assertEquals("getToken() must have performed a silent refresh", 1, refreshCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGetTokenRefreshesWithoutPrompting() throws Exception { + assertMemoryLeak(() -> { + // getToken() returns the cached token, silently refreshes it when it expires, and never + // prompts; if it cannot produce a token without an interactive sign-in, it throws + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger promptCalls = new AtomicInteger(); + AtomicBoolean refreshOk = new AtomicBoolean(true); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + return refreshOk.get() + ? MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 1)) + : MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, ch -> promptCalls.incrementAndGet())) { + // before any sign-in, getToken() must not prompt - it throws + assertOidcFails(auth::getToken, "no token", "expected getToken() to fail before sign-in"); + // sign in once interactively + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the cached token is expired, so getToken() refreshes silently + Assert.assertEquals("ACCESS-2", auth.getToken()); + // now make the refresh fail; getToken() must throw, not start the device flow + refreshOk.set(false); + expireCachedToken(auth); + assertOidcFails(auth::getToken, "interactive sign-in", + "expected getToken() to fail when the refresh is rejected"); + // the device flow ran exactly once (the initial signIn), and the user was prompted once + Assert.assertEquals(1, deviceCalls.get()); + Assert.assertEquals(1, promptCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testBlankServedTokenFromWireIsNotServed() throws Exception { + assertMemoryLeak(() -> { + // a hostile or broken IdP returns a whitespace-only access token on the grant: it is non-empty and + // passes the control/non-ASCII char check vacuously (space is 0x20), but must NOT be cached and + // served as a blank "Bearer " header (which only draws a 401). storeTokens folds a blank served + // token to absent, so signIn() fails with the actionable "no access_token" rather than serving " " + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson(" ", null, "REFRESH-1", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "no access_token", + "expected signIn() to reject a blank served token from the wire"); + } + }); + } + + @Test(timeout = 30_000) + public void testBlankTokenFromRefreshFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // a non-conformant IdP answers a SILENT REFRESH with a 2xx carrying a whitespace-only access token. + // The refresh gate (hasRequiredToken) must treat it as absent with the same Chars.isBlank contract + // storeTokens uses, so tryRefresh() reports failure and signIn() falls back to the interactive device + // flow - rather than caching a token storeTokens then nulls, which would make signIn() throw "no + // access_token" while a fresh interactive sign-in was still possible. + AtomicInteger deviceCodePolls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // blank served token on refresh: the gate must fall back, not cache-and-serve it + return MockOidcServer.json(200, tokenJson(" ", null, null, 3600)); + } + // the device-code grant: the first poll mints the initial short-lived token; the second is the + // interactive fallback after the blank refresh and mints a fresh, usable one + return deviceCodePolls.incrementAndGet() == 1 + ? MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 60)) + : MockOidcServer.json(200, tokenJson("ACCESS-FALLBACK", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); // force the silent-refresh path on the next sign-in + // with the blank-refresh gate fixed, signIn() falls back to the device flow instead of throwing + Assert.assertEquals("ACCESS-FALLBACK", auth.signIn()); + Assert.assertEquals("the interactive device flow must run again as the fallback", + 2, deviceCodePolls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testGroupsInTokenButNoIdTokenFails() throws Exception { + assertMemoryLeak(() -> { + // groups encoded in token, but the IdP returns only an access token on the initial grant + // (e.g. the requested scope omitted openid); signIn() must fail with an actionable message + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-ONLY", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + assertOidcFails(auth::signIn, "no id_token"); + } + }); + } + + @Test(timeout = 30_000) + public void testGroupsInTokenReturnsIdToken() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-X", "ID-X", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + Assert.assertEquals("ID-X", auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testHttpSenderProviderFailureAfterFlushDoesNotCorruptSender() throws Exception { + assertMemoryLeak(() -> { + // regression: the per-request token must be pulled lazily when a row starts, never eagerly when + // the post-flush request is rebuilt. A provider that throws on a later pull (e.g. + // OidcDeviceAuth::getToken when a refresh fails) must NOT turn an already-successful + // flush into a thrown exception, and must NOT leave a half-built request that corrupts the + // sender so later rows go out malformed + MockOidcServer.Handler handler = (method, path, body) -> MockOidcServer.json(204, ""); + AtomicInteger pulls = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer(handler); + Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V2) + .httpTokenProvider(() -> { + int n = pulls.incrementAndGet(); + if (n == 2) { + // the second pull - for the request after the first, successful flush - fails + throw new OidcAuthException("the cached token expired and could not be refreshed"); + } + return "TOKEN-" + n; + }) + .build()) { + // first batch: the token is pulled when the row starts (TOKEN-1); the flush sends it and must + // succeed. The failing *next* pull must not strike here - the eager post-flush pull was the bug + sender.table("t").doubleColumn("x", 1.0).atNow(); + sender.flush(); + + // next batch: the deferred pull runs when the row starts and the provider throws there; the + // failure must surface cleanly, leaving the previous successful flush and its data untouched + try { + sender.table("t").doubleColumn("x", 2.0).atNow(); + Assert.fail("expected the failing provider pull to surface on the next row"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("could not be refreshed")); + Assert.assertTrue("the provider failure must be retained as the cause", + e.getCause() instanceof OidcAuthException); + } + + // the provider recovers (pull #3 -> TOKEN-3); the failed pull must not have corrupted the + // sender, so this row produces a well-formed request the server accepts + sender.table("t").doubleColumn("x", 3.0).atNow(); + sender.flush(); + + java.util.List seen = server.requestAuthHeaders(); + Assert.assertTrue(seen.toString(), seen.contains("Bearer TOKEN-1")); + Assert.assertTrue(seen.toString(), seen.contains("Bearer TOKEN-3")); + // the failed pull never reached the wire as a partial request + Assert.assertFalse(seen.toString(), seen.contains("Bearer TOKEN-2")); + } + }); + } + + @Test(timeout = 30_000) + public void testHttpSenderPullsTokenProviderPerRequest() throws Exception { + assertMemoryLeak(() -> { + // a long-lived HTTP Sender must pull the token from the provider on each request, so a rotating + // token (as OidcDeviceAuth produces on refresh) reaches the wire without rebuilding the sender + MockOidcServer.Handler handler = (method, path, body) -> MockOidcServer.json(204, ""); + AtomicInteger tokenSeq = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer(handler); + Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V2) + .httpTokenProvider(() -> "TOKEN-" + tokenSeq.incrementAndGet()) + .build()) { + sender.table("t").doubleColumn("x", 1.0).atNow(); + sender.flush(); + sender.table("t").doubleColumn("x", 2.0).atNow(); + sender.flush(); + // each flush built a fresh request and pulled a fresh token; the server saw successive bearers + java.util.List seen = server.requestAuthHeaders(); + Assert.assertTrue("expected at least 2 write requests, got " + seen, seen.size() >= 2); + Assert.assertTrue(seen.toString(), seen.contains("Bearer TOKEN-1")); + Assert.assertTrue(seen.toString(), seen.contains("Bearer TOKEN-2")); + Assert.assertNotEquals("the token must rotate per request", seen.get(0), seen.get(1)); + } + }); + } + + @Test(timeout = 30_000) + public void testIncompleteDeviceResponseRejected() throws Exception { + assertMemoryLeak(() -> { + // the device endpoint returns 200 but omits user_code and verification_uri + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(200, "{\"device_code\":\"DEV\",\"expires_in\":300,\"interval\":1}"); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "incomplete device authorization"); + } + }); + } + + @Test(timeout = 30_000) + public void testIdpEndpointsRequireHttpsExceptLoopback() throws Exception { + assertMemoryLeak(() -> { + // a non-loopback http identity-provider endpoint carries the device code and refresh token in + // cleartext, so it must be refused + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("http://idp.example/device") + .tokenEndpoint("https://idp.example/token") + .build() + ) { + Assert.fail("expected the http device authorization endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("device authorization endpoint")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("insecure http")); + } + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/device") + .tokenEndpoint("http://idp.example/token") + .build() + ) { + Assert.fail("expected the http token endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("token endpoint")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("insecure http")); + } + // allowInsecureTransport must NOT relax the identity provider endpoints (unlike the QuestDB + // link), matching the Python client; a non-loopback http endpoint stays rejected, and the + // error says so + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("http://idp.example/device") + .tokenEndpoint("http://idp.example/token") + .allowInsecureTransport(true) + .build() + ) { + Assert.fail("allowInsecureTransport must not relax a non-loopback http identity provider endpoint"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("insecure http")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("allowInsecureTransport relaxes only the QuestDB")); + } + // loopback http is allowed without any flag: the request never leaves the host + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("http://127.0.0.1:9999/device") + .tokenEndpoint("http://127.0.0.1:9999/token") + .build() + ) { + // accepted: loopback endpoints never put the device code or refresh token on the network + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingAcceptsEndpointsUnderIssuerPath() throws Exception { + assertMemoryLeak(() -> { + // a path-based identity provider (Keycloak-style /realms/{realm}): the issuer carries a path and + // /settings advertises the endpoints under it, so the flow completes + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/acme/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl("/realms/acme/device") + "\"" + + "}}"); + } + if ("/realms/acme/device".equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-REALM", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("/realms/acme")))) { + Assert.assertEquals("ACCESS-REALM", auth.signIn()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsEncodedSlash() throws Exception { + assertMemoryLeak(() -> { + // the device endpoint hides an extra path segment behind a %2f-encoded slash; decoding it would + // split acme%2fevil into acme/evil and slip the "/realms/acme" scope, so an encoded path separator + // must be rejected outright + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/acme/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl("/realms/acme%2fevil/device") + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("/realms/acme")))) { + Assert.fail("expected the %2f-encoded device endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("not under the pinned issuer")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsEncodedTraversal() throws Exception { + assertMemoryLeak(() -> { + // the device endpoint hides a parent traversal as %2e%2e; decoding must unmask it and reject it, + // since the server would normalize /realms/acme/../evil to a different realm + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/acme/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl("/realms/acme/%2e%2e/evil/device") + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("/realms/acme")))) { + Assert.fail("expected the encoded ..-traversal device endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("not under the pinned issuer")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsMatrixParamTraversal() throws Exception { + assertMemoryLeak(() -> { + // the device endpoint hides a parent traversal as an RFC 3986 ";matrix" segment (..;): a server or + // proxy that strips matrix params resolves /realms/acme/..;/evil to /realms/evil, a DIFFERENT realm. + // The plain "." / ".." dot-segment check does not match "..;", so the check must strip the ";suffix" + // first and reject it - the origin pin alone cannot stop a sibling-tenant redirect on one host. + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/acme/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl("/realms/acme/..;/evil/device") + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("/realms/acme")))) { + Assert.fail("expected the ..;-matrix-param traversal device endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("not under the pinned issuer")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsSiblingRealm() throws Exception { + assertMemoryLeak(() -> { + // a tampered /settings advertises a token endpoint under a DIFFERENT realm on the same origin; the + // origin check alone would accept it, but path scoping must reject it + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/evil/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl("/realms/acme/device") + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(server.httpUrl("/realms/acme")))) { + Assert.fail("expected the off-path (sibling realm) token endpoint to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("not under the pinned issuer")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsRawDotSegments() throws Exception { + assertMemoryLeak(() -> { + // A RAW (unencoded) ".." or "." segment carries no '%' or '\\' and no '?'/'#'/control, so it slips + // every earlier gate and reaches the dot-segment scan - the only cases that do. A lenient server + // normalizes .../realms/acme/../evil/device into a different realm, so the pin must reject it. + assertIssuerScopeAccepts("/realms/acme/device"); + assertIssuerScopeRejects("/realms/acme/../evil/device"); + assertIssuerScopeRejects("/realms/acme/./../evil/device"); + assertIssuerScopeRejects("/realms/./acme/device"); + }); + } + + @Test(timeout = 30_000) + public void testIssuerPathScopingRejectsSplitEncodedAndBackslashSeparators() throws Exception { + assertMemoryLeak(() -> { + // An encoded path separator can hide behind a SPLIT encoding (%2%66 -> %2f -> '/'), a double + // encoding (%252f), or a backslash that decodePathSegments folds to '/'. Each lets an extra + // segment masquerade as being under the issuer path while a different raw path travels on the + // wire. Overlong UTF-8 (%c0%ae, %e0%80%ae) and an IIS-style %u002e encode a '.' that a permissive + // server resolves but a byte-oriented decode leaves as high bytes, so any '%' in an endpoint path + // is refused outright. + assertIssuerScopeAccepts("/realms/acme/protocol/device"); + assertIssuerScopeRejects("/realms/acme%2%66evil/device"); + assertIssuerScopeRejects("/realms/acme%252fevil/device"); + assertIssuerScopeRejects("/realms/acme\\evil/device"); + assertIssuerScopeRejects("/realms/acme%5cevil/device"); + assertIssuerScopeRejects("/realms/acme%5Cevil/device"); + assertIssuerScopeRejects("/realms/acme/%c0%ae%c0%ae/evil/device"); + assertIssuerScopeRejects("/realms/acme/%e0%80%ae%e0%80%ae/evil/device"); + assertIssuerScopeRejects("/realms/acme/%u002e%u002e/evil/device"); + }); + } + + @Test(timeout = 30_000) + public void testLargeSplitTokenValueParsesWithConfiguredLexerSizing() throws Exception { + assertMemoryLeak(() -> { + // A real id_token (a JWT with group claims) runs to several KB, and a single JSON string value + // can arrive split across HTTP response fragments. OidcDeviceAuth must size its JSON lexer so + // such a split value still parses. Drive the production OidcDeviceAuth instance rather than a + // test-local JsonLexer configured with the same literal: reverting JSON_LEXER_MAX_VALUE_BYTES + // to its old 1024 value must make this flow fail with "String is too long". MockOidcServer emits + // 64-byte HTTP chunks, so the 4 KiB value necessarily exercises the lexer's split-value cache. + String idToken = TestUtils.repeat("a", 4000); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.chunkedJson(200, tokenJson("ACCESS-LARGE", idToken, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + Assert.assertEquals("the production lexer must return the complete split id_token", + idToken, auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testPlaintextIdpEndpointIsAllowedOnlyForLoopbackHosts() { + // The rule the loopback classifier exists to serve: the identity provider endpoints must use https, + // because the device code and the refresh token travel over them - EXCEPT to a loopback host, where + // the request never leaves the machine. Driven through build(), which does no network I/O, rather + // than by reflecting on the private classifier: this asserts the outcome a user actually gets, and + // it survives that predicate being renamed, inlined or replaced. + // + // Both endpoints use the same host because build() also requires them to share an origin; that check + // is not what is under test here, it just has to be satisfied for the loopback rows to reach a verdict. + String[] loopback = { + "localhost", "LOCALHOST", "LocalHost", + "127.0.0.1", "127.0.0.0", "127.1.2.3", "127.255.255.255", "127.0.0.255" + }; + for (String host : loopback) { + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("http://" + host + "/device") + .tokenEndpoint("http://" + host + "/token") + .build()) { + Assert.assertNotNull("plaintext to a loopback host must be allowed: [" + host + ']', auth); + } + } + + // Everything else must be refused over plaintext, so the MITM pin fires. A classifier that accepted + // any of these would silently send a device code and a refresh token across the network in the clear. + String[] notLoopback = { + "example.com", "questdb.example", + "127.evil.com", // starts with "127." but is not a dotted-IPv4 literal + "localhost.evil.com", // not an exact localhost match + "evil.localhost", + "0x7f.0.0.1", // hex form is not the dotted 127.0.0.0/8 literal + "127.1", "127.0.1", "127", // short forms the OS would expand are deliberately not accepted + "127.0.0.256", // octet out of range + "127.0.0.1.evil.com", // extra label after a valid prefix + "127.0.0.1.", // trailing dot + "127..0.1", // empty octet + "1270.0.0.1", // does not start with "127." + "227.0.0.1", // not the 127 block + "0.0.0.0", "10.0.0.1", "192.168.0.1", + // A name is accepted on the strength of what it RESOLVES to, not how it is spelt - RFC 6761 + // says localhost must be loopback, but a host with no /etc/hosts entry leaves that to DNS. + // One that does not resolve must fail CLOSED, i.e. be treated as non-loopback. (The hostile + // half - localhost resolving off loopback - needs the host's resolver rewritten, so it is + // unasserted by design rather than by omission.) + "no-such-host.invalid" + }; + for (String host : notLoopback) { + assertBuildFails("http://" + host + "/device", "http://" + host + "/token", "use an https url"); + } + + // Two forms the classifier never sees, because the endpoint parser rejects them first. Asserted here + // so the list above is not silently assumed to cover them. + assertBuildFails("http:///device", "http:///token", "the host is empty"); + assertBuildFails("http://[::1]:9000/device", "http://[::1]:9000/token", + "IPv6 literal hosts are not supported"); + } + + @Test(timeout = 30_000) + public void testPlaintextSettingsWithAdvertisedEndpointsRequiresPin() throws Exception { + // The "127.1" reachability trick below depends on the OS resolver expanding the abbreviated IPv4 + // form to 127.0.0.1 (inet_aton, on Linux/macOS). Windows getaddrinfo - which the native HTTP client + // resolves through - does not accept the short form, so the loopback mock is unreachable there. + Assume.assumeTrue("requires inet_aton-style short-form IPv4 resolution, unavailable on Windows", Os.type != Os.WINDOWS); + assertMemoryLeak(() -> { + // the end-to-end firing path of the plaintext-channel MITM pin, which a 127.0.0.1-bound mock + // cannot otherwise reach: a non-loopback http /settings that advertises BOTH endpoints (so the + // missing-endpoint discovery pin does not apply) must be refused unless the identity provider is + // pinned out of band - otherwise a tampered response could route the device code and refresh token + // to an attacker. Reaching the mock through "127.1" is the trick: the OS resolver expands the short + // form to 127.0.0.1 so the loopback mock answers, but the loopback classifier deliberately rejects + // the short form, so the server host is non-loopback and the pin fires. + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, settingsJson(true, true, server.httpUrl(TOKEN_PATH), server.httpUrl(DEVICE_PATH))); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + String questdbUrl = "http://127.1:" + server.port(); + // without an out-of-band pin the plaintext channel is untrusted: the pin fires + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(questdbUrl, insecure())) { + Assert.fail("expected the plaintext-channel pin to reject /settings-supplied endpoints without a pin"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("reached over insecure http")); + } + // pinning the issuer to the advertised endpoints' origin satisfies the pin over the very same + // plaintext channel, so construction succeeds - proving the pin, not some unrelated rejection, + // is what gated the unpinned call above + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(questdbUrl, insecure().issuer(server.httpUrl("")))) { + Assert.assertNotNull(auth); + } + } + }); + } + + @Test(timeout = 30_000) + public void testRejectedBuildDoesNotLeakNativeMemory() throws Exception { + // A build rejected during validation must not leak. build() parses and validates every endpoint + // BEFORE the constructor runs, and the constructor allocates the native JSON lexer LAST (after + // urlEncode and the TokenStoreKey build, either of which can throw), so a rejected build never + // allocates the lexer and the never-returned instance cannot be closed to free it. Use a + // parseable-but-rejected config - endpoints that parse cleanly but fail the https requirement - so the + // rejection lands AFTER endpoint parsing, exercising more of build() than a syntactically bad url + // would. testSuccessfulBuildAndCloseDoNotLeakNativeMemory covers the complementary lexer-allocated + // path. assertMemoryLeak guards every tag; the parser-tag assertion stays because it names the buffer. + assertMemoryLeak(() -> { + long parserMemBefore = Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS); + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("http://idp.example/device") // parses fine, but plaintext http to a non-loopback host + .tokenEndpoint("https://idp.example/token") + .allowInsecureTransport(false) + .build() + ) { + Assert.fail("expected the https requirement to reject the plaintext device endpoint"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("use an https url")); + } + Assert.assertEquals("a rejected build must not leak the JSON lexer native buffer", + parserMemBefore, Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS)); + }); + } + + @Test(timeout = 30_000) + public void testSuccessfulBuildAndCloseDoNotLeakNativeMemory() throws Exception { + // The complement to the rejected-build case: a SUCCESSFUL build is the only path that allocates the + // native JSON lexer, so this is the block that actually exercises a lexer-allocated instance, and + // close() must free it. Loop a few build->close cycles so any per-cycle leak accrues, then assert the + // parser tag returns to its baseline. build() does no network I/O (discovery is separate), so valid + // co-located https endpoints construct offline. assertMemoryLeak guards every tag; the parser-tag + // assertion stays because it names the buffer. + assertMemoryLeak(() -> { + long parserMemBefore = Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS); + for (int i = 0; i < 4; i++) { + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/device") + .tokenEndpoint("https://idp.example/token") + .build()) { + Assert.assertNotNull(auth); + } + } + Assert.assertEquals("close() must free the JSON lexer native buffer allocated by a successful build", + parserMemBefore, Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS)); + }); + } + + @Test(timeout = 30_000) + public void testNoAccessTokenWhenGroupsDisabledFails() throws Exception { + assertMemoryLeak(() -> { + // groups not in token, but the IdP returns only an id token; signIn() must fail + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson(null, "ID-ONLY", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "no access_token"); + } + }); + } + + @Test(timeout = 30_000) + public void testNonSuccessDeviceAuthorizationResponseRejected() throws Exception { + assertMemoryLeak(() -> { + // RFC 8628 3.2: a device authorization grant is a 2xx response. A non-2xx body that nonetheless + // carries device_code/user_code/verification_uri and no OAuth error must be rejected - the client + // must not prompt the user and poll on a response the server never signalled success for + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(403, deviceAuthorizationJson(1, 300)); + AtomicBoolean prompted = new AtomicBoolean(false); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, challenge -> prompted.set(true))) { + assertOidcFails(auth::signIn, "unexpected response from the device authorization endpoint", + "expected the non-2xx device authorization response to be rejected"); + Assert.assertFalse("the user must not be prompted on a rejected device authorization response", prompted.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testNullAccessTokenNotServedAsLiteralNull() throws Exception { + assertMemoryLeak(() -> { + // a JSON null arrives from the lexer as the literal "null"; "access_token": null must be treated + // as absent, not stored and served as the 4-char token "null" + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, "{\"token_type\":\"Bearer\",\"expires_in\":3600,\"access_token\":null}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + // null is absent, so a 2xx with no token is a definitive but malformed answer. The token the + // call would have served, had it wrongly served the literal "null", is in the failure message + // assertOidcFails builds. + assertOidcFails(auth::signIn, "unexpected response", + "a JSON null access_token must not be served as the literal token \"null\""); + } + }); + } + + @Test(timeout = 30_000) + public void testNullJsonErrorIsTreatedAsAbsent() throws Exception { + assertMemoryLeak(() -> { + // "error": null in a device-auth response must be treated as absent, not as an OAuth error whose + // code is the literal string "null"; the flow must proceed to prompt and poll + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"error\":null," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-OK", auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testNullPromptDefaultsToSystemOut() throws Exception { + assertMemoryLeak(() -> { + // builder.prompt(null) must fall back to the default prompt rather than NPE during the flow + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-NP", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .prompt(null) + .allowInsecureTransport(true) + .build()) { + // no NPE: the flow runs to completion with the default SYSTEM_OUT prompt + Assert.assertEquals("ACCESS-NP", auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testOauthErrorMessageStripsBidiControls() throws Exception { + assertMemoryLeak(() -> { + // an IdP error_description carrying a right-to-left override and a zero-width space (as JSON + // unicode escapes the lexer decodes) must not reach the exception message verbatim; they would + // let a malicious IdP reorder or hide text when the message is rendered to a terminal or a log + String desc = "denied" + jsonUnicodeEscape(0x202E) + "reversed" + jsonUnicodeEscape(0x200B) + "end"; + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(400, "{\"error\":\"access_denied\",\"error_description\":\"" + desc + "\"}"); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "access_denied"); + Assert.assertEquals("access_denied", e.getOauthError()); + String msg = e.getMessage(); + assertNoUnsafeDisplayChars(msg); + Assert.assertTrue(msg, msg.contains("deniedreversedend")); // readable text survives, controls gone + } + }); + } + + @Test(timeout = 30_000) + public void testOauthErrorMessageStripsControlChars() throws Exception { + assertMemoryLeak(() -> { + // an IdP error_description carrying ANSI/CRLF control chars must not reach the exception + // message verbatim (it would let a malicious IdP rewrite the terminal or forge log lines) + String desc = "denied" + ((char) 0x1b) + "[2J\r\nFAKE: paste your token"; + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(400, "{\"error\":\"access_denied\",\"error_description\":\"" + desc + "\"}"); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "access_denied"); + Assert.assertEquals("access_denied", e.getOauthError()); + String msg = e.getMessage(); + assertNoControlChars(msg); + Assert.assertTrue(msg, msg.contains("FAKE: paste your token")); // readable text survives + } + }); + } + + @Test(timeout = 30_000) + public void testOutOfRangePollIntervalAndExpiryAreClamped() throws Exception { + assertMemoryLeak(() -> { + // a hostile or misconfigured identity provider reports an absurd interval/expires_in; the + // client must clamp both, so interval*1000 cannot overflow into a zero-delay busy loop and + // the wait cannot run absurdly long + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(2_000_000_000, 2_000_000_000)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-CLAMP", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-CLAMP", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + // the absurd interval/expires_in are clamped to the documented maxima: the poll interval to + // MAX_POLL_INTERVAL_SECONDS (60) and the device-code lifetime to MAX_DEVICE_CODE_TTL_SECONDS (1800) + Assert.assertEquals(60, challenge.getIntervalSeconds()); + Assert.assertEquals(1800, challenge.getExpiresInSeconds()); + } + }); + } + + @Test(timeout = 30_000) + public void testOversizedSettingsBodyAbortsAtSizeCap() throws Exception { + assertMemoryLeak(() -> { + // a hostile or MITM'd server streams a /settings body larger than the client's response-size cap + // (MAX_RESPONSE_BODY_BYTES, 4 MiB); the bounded read must abort on the cap rather than consume the + // body without limit. Stream well past the cap - the client stops reading and closes the + // connection once it crosses 4 MiB + MockOidcServer.Handler handler = (method, path, body) -> MockOidcServer.oversizedJson(8L * 1024 * 1024); + try (MockOidcServer server = new MockOidcServer(handler)) { + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to abort on the response-size cap"); + } catch (OidcAuthException e) { + // the size-cap failure surfaces as the cause; the body (which carries access/id/refresh + // tokens on a real response) is never embedded in the message + Throwable cause = e.getCause(); + Assert.assertNotNull("expected the size-cap failure as the cause", cause); + Assert.assertTrue(cause.getMessage(), cause.getMessage().contains("exceeded the size limit")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testPollAbortDropsDirtyConnectionAndReconnects() throws Exception { + assertMemoryLeak(() -> { + // the token endpoint stalls the body on the first poll, so the bounded read aborts with the + // response half-read and unconsumed bytes left in the cached keep-alive connection. The poll loop + // must drop that connection and reconnect for the next poll, not reuse it: the stalled mock thread + // never reads a reused connection, so reusing it would leave every later poll unanswered until the + // device code expires (and, for a non-stalled dirty connection, would mis-frame the next response + // against this one's leftovers). With the reconnect, the second poll reaches a fresh connection and + // succeeds. Without the fix this test hangs until the 10s device-code lifetime and signIn throws. + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + // short lifetime, well under the 30s mock stall and the 30s test timeout, so the no-fix + // failure (poll the dirty connection until expiry) surfaces deterministically and fast + return MockOidcServer.json(200, deviceAuthorizationJson(1, 10)); + } + if (tokenCalls.getAndIncrement() == 0) { + return MockOidcServer.stall(); + } + return MockOidcServer.json(200, tokenJson("ACCESS-RECONNECTED", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .httpTimeoutMillis(1_000) // abort the stalled body read quickly + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + Assert.assertEquals("ACCESS-RECONNECTED", auth.signIn()); + Assert.assertEquals(2, tokenCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testPollIntervalClampedTo60() throws Exception { + assertMemoryLeak(() -> { + // the identity-provider-reported poll interval is capped at 60s (matching the Python client); the + // clamped value is the one shown to the user and used between polls. A short-lived device code + // ends the flow quickly via timeout, once the interval has been captured by the prompt. + AtomicReference shown = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(999, 2)); + } + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + assertOidcFails(auth::signIn, "device code expired", "expected the device code to expire"); + Assert.assertEquals(60, shown.get().getIntervalSeconds()); + } + }); + } + + @Test(timeout = 30_000) + public void testRateLimited429WithTerminalErrorAbortsImmediately() throws Exception { + assertMemoryLeak(() -> { + // a 429 that ALSO carries a terminal OAuth error must fail fast on the error, not back off and poll + // to the device-code deadline: pollOnce handles the OAuth error before the 429 rate-limit backoff + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 5)); + } + return MockOidcServer.json(429, "{\"error\":\"access_denied\",\"error_description\":\"the user declined\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + try { + auth.signIn(); + Assert.fail("expected the terminal OAuth error to abort despite the 429 status"); + } catch (OidcAuthException e) { + Assert.assertEquals("access_denied", e.getOauthError()); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("device code expired")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testRateLimitedTokenEndpointBacksOffInsteadOfFailingFast() throws Exception { + assertMemoryLeak(() -> { + // HTTP 429 is a transient backoff (poll slower, keep polling), matching the Python client, not a + // terminal rejection. The token endpoint always returns 429, so the flow ends only when the + // short-lived device code expires - proving polling continued rather than failing fast. + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 4)); + } + return MockOidcServer.json(429, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "device code expired", + "expected the device code to expire while the token endpoint kept returning 429"); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("rejected the request")); + } + }); + } + + @Test(timeout = 30_000) + public void testPersistentTransportFailureKeepsPollingToDeadline() throws Exception { + assertMemoryLeak(() -> { + // the device endpoint works, but the (co-located) token endpoint drops the connection on every + // poll. Matching the Python client, a transport failure is transient - the user may already have + // authorized - so polling continues until the device code expires rather than failing fast. The + // endpoints share one origin so the build-time co-location check passes; the mock simulates the + // unreachable token endpoint by dropping the connection. + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 3)); + } + return MockOidcServer.dropConnection(); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + auth.signIn(); + Assert.fail("expected the device code to expire while the token endpoint kept dropping"); + } catch (OidcAuthException e) { + // polled to the deadline (device code expired), not a fast transport abort + Assert.assertTrue(e.getMessage(), e.getMessage().contains("device code expired")); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("unreachable")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testPersistent5xxDuringPollingKeepsPollingToDeadline() throws Exception { + assertMemoryLeak(() -> { + // a 5xx from the token endpoint is a transient server/gateway condition: keep polling to the + // device-code deadline rather than failing fast, matching the Python client + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 3)); + } + return MockOidcServer.json(503, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "device code expired", + "expected the device code to expire while the token endpoint returned 503"); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("rejected the request")); + } + }); + } + + @Test(timeout = 30_000) + public void testTerminal4xxDuringPollingFailsFast() throws Exception { + assertMemoryLeak(() -> { + // a 4xx from the token endpoint with no OAuth error (e.g. a WAF or proxy rejection) is terminal: + // fail fast rather than poll on to a misleading "device code expired", matching the Python client + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(403, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "rejected the request", + "expected a terminal 4xx to fail fast"); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("device code expired")); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshErrorFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // the cached token expires and the refresh is rejected (revoked/expired refresh token); + // the client must fall back to a fresh interactive sign-in + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + } + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the refresh is rejected, so the flow re-runs the interactive sign-in + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (initial + fallback)", 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshedTokenWithControlCharFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // A silent refresh whose 200 response carries a served token with a control character - here an + // escaped \r that JsonLexer now decodes into a real CR byte - must be rejected by storeTokens -> + // validateTokenChars, and tryRefresh must SWALLOW that rejection and fall back to the interactive + // device flow rather than let it propagate out of signIn()/getToken(). Guards the tryRefresh + // storeTokens try/catch: without it, this signIn() throws instead of returning the fallback token. + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // valid-JSON 200, but the access_token carries an escaped CR (\r on the wire); the served + // kind is validated, so validateTokenChars must reject it before it is cached + return MockOidcServer.json(200, tokenJson("ACCESS\\r2", null, "REFRESH-2", 3600)); + } + // the initial device-code grant uses a short TTL so the next signIn() triggers a refresh + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + } + // the fallback interactive grant, after the poisoned refresh is rejected + return MockOidcServer.json(200, tokenJson("ACCESS-3", null, "REFRESH-3", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the refresh returns a control-char token -> rejected -> fall back to a fresh interactive sign-in + Assert.assertEquals("ACCESS-3", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (initial + fallback after the rejected refresh)", + 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshedTokenWithControlCharStillAdoptsTheRotatedRefreshToken() throws Exception { + assertMemoryLeak(() -> { + // The sibling test above pins the FALLBACK when a refresh returns an unusable served token. This + // one pins what happens to the refresh_token that arrived beside it. + // + // The rejection does not undo the exchange: the provider accepted the token we presented and + // answered a clean 2xx, so a rotating provider has already burned it and the refresh_token in + // that body is the live one. That is the rule adoptRotatedRefreshToken() states, and it applied + // only to the branch where the served kind was ABSENT - the branch where it arrives and is + // rejected returned first and dropped the rotation with the rest of the response. + // + // signIn() hides the loss, because its device-flow fallback overwrites the refresh token before + // anything can replay it. getToken() is where it bites: it never prompts, so the spent token + // stays cached and goes back on the wire, and a reuse-detecting provider answers a replay by + // revoking the whole family - the credential lost outright rather than one refresh failed. + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + List presented = Collections.synchronizedList(new ArrayList<>()); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + presented.add(body.contains("refresh_token=REFRESH-1") ? "REFRESH-1" + : body.contains("refresh_token=REFRESH-2") ? "REFRESH-2" : "OTHER:" + body); + // clean 2xx: the exchange happened. The served token carries an escaped CR, so + // validateTokenChars rejects it - but REFRESH-2 is live and REFRESH-1 is now spent. + return MockOidcServer.json(200, tokenJson("ACCESS\\r2", null, "REFRESH-2", 3600)); + } + deviceCodeGrants.getAndIncrement(); + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + + for (int attempt = 0; attempt < 2; attempt++) { + expireCachedToken(auth); + clearRefreshBackOff(auth); + try { + auth.getToken(); + Assert.fail("an unusable served token must not be served"); + } catch (OidcAuthException expected) { + // getToken() never prompts, so it reports the failure and leaves the caller to + // signIn(); what matters is the credential it holds when it does + } + } + + Assert.assertEquals("both attempts must reach the token endpoint", 2, presented.size()); + Assert.assertEquals("the first refresh presents the token we started with", + "REFRESH-1", presented.get(0)); + Assert.assertEquals("the second refresh must present the ROTATED token: the provider burned " + + "REFRESH-1 answering the first, so replaying it is what a reuse-detecting " + + "provider revokes the whole family over", + "REFRESH-2", presented.get(1)); + Assert.assertEquals("no interactive flow may run on the getToken() path", 1, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshKeepsExistingRefreshTokenWhenOmitted() throws Exception { + assertMemoryLeak(() -> { + // a refresh response that omits refresh_token (RFC 6749 permits this) must not drop the existing + // refresh token; a later refresh must reuse it rather than fall back to a fresh interactive sign-in + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger refreshCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // every refresh must present the ORIGINAL refresh token, and returns a short-lived + // access token WITHOUT a new refresh_token + Assert.assertTrue(body, body.contains("refresh_token=REFRESH-1")); + int n = refreshCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-R" + n, null, null, 1)); + } + // the initial device-code grant: a short-lived access token plus the refresh token + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // first refresh omits refresh_token, so REFRESH-1 must be kept + Assert.assertEquals("ACCESS-R1", auth.signIn()); + expireCachedToken(auth); + // second refresh must still present the retained REFRESH-1 (asserted in the handler) + Assert.assertEquals("ACCESS-R2", auth.signIn()); + Assert.assertEquals("no extra interactive sign-in", 1, deviceCalls.get()); + Assert.assertEquals(2, refreshCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshTokenAlongsideErrorFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // a refresh response that carries an OAuth error (under a non-2xx status) must not be trusted + // even if it also returns a token; the client ignores the smuggled token and falls back to a + // fresh interactive sign-in rather than caching it + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // malformed: a 400 error together with a token + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\",\"access_token\":\"SHOULD-NOT-BE-USED\"}"); + } + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 1)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", null, "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the refresh carries an error+token, so the client must ignore the smuggled token and + // re-run the interactive flow + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (initial + fallback)", 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testInterruptDuringTheRefreshDoesNotStartTheDeviceFlow() throws Exception { + assertMemoryLeak(() -> { + // signIn() guards the interrupt flag on entry, but everything after that guard is network work: + // the silent refresh is a round trip bounded by six times httpTimeoutMillis plus an OS connect + // stall. A cancellation landing inside it is the ORDINARY case rather than a narrow race, because + // a caller gives up precisely when a refresh is dragging - and proceeding then launches a browser + // and parks for the device-code lifetime on a thread whose owner already asked it to stop. + AtomicInteger deviceCalls = new AtomicInteger(); + CountDownLatch refreshReceived = new CountDownLatch(1); + CountDownLatch interruptSent = new CountDownLatch(1); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // hold the refresh open until the test has interrupted the worker, then fail it so the + // flow reaches the point where it would otherwise prompt + refreshReceived.countDown(); + try { + interruptSent.await(20, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return MockOidcServer.json(400, "{\"error\":\"invalid_grant\"}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + Assert.assertEquals(1, deviceCalls.get()); + expireCachedToken(auth); + + AtomicReference failure = new AtomicReference<>(); + AtomicBoolean flagSurvived = new AtomicBoolean(); + Thread worker = new Thread(() -> { + try { + auth.signIn(); + failure.compareAndSet(null, new AssertionError("signIn() must abandon a cancelled sign-in")); + } catch (OidcAuthException expected) { + if (!expected.getMessage().contains("interrupted")) { + failure.compareAndSet(null, expected); + } + } catch (Throwable t) { + failure.compareAndSet(null, t); + } finally { + // read it, do not clear it: the caller's shutdown path is what waits on this flag + flagSurvived.set(Thread.currentThread().isInterrupted()); + } + }, "cancelled-signin"); + worker.start(); + + Assert.assertTrue("the refresh must reach the server", + refreshReceived.await(20, TimeUnit.SECONDS)); + worker.interrupt(); + interruptSent.countDown(); + worker.join(20_000); + Assert.assertFalse("signIn() did not return after the cancellation", worker.isAlive()); + + Assert.assertNull(String.valueOf(failure.get()), failure.get()); + Assert.assertEquals("a cancelled sign-in must not open a browser or start a device grant", + 1, deviceCalls.get()); + Assert.assertTrue("the interrupt is the caller's cancellation signal and must survive " + + "signIn(), or their own shutdown bookkeeping reads as never-cancelled", + flagSurvived.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testInterruptDuringThePollLoopAbandonsItAndKeepsTheFlag() throws Exception { + assertMemoryLeak(() -> { + // The other half: an interrupt that arrives once the poll loop is already running. Os.sleep + // catches InterruptedException, recomputes its deadline and keeps sleeping - and Thread.sleep + // clears the flag when it throws - so the loop both ignored the cancellation AND destroyed the + // evidence of it, then polled on to the device-code lifetime. + // + // The prompt runs on the sign-in thread, immediately before the poll loop, so interrupting from + // there lands the cancellation exactly where the loop must notice it. + AtomicInteger tokenPolls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 1800)); + } + tokenPolls.incrementAndGet(); + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + }; + DeviceCodePrompt cancellingPrompt = challenge -> Thread.currentThread().interrupt(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, cancellingPrompt)) { + long startNanos = System.nanoTime(); + try { + auth.signIn(); + Assert.fail("the poll loop must abandon a cancelled sign-in"); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), + expected.getMessage().contains("interrupted")); + } + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + + Assert.assertTrue("the interrupt must survive the poll loop", Thread.interrupted()); + // the device code is good for 1800s; a loop that ignores the interrupt runs to the @Test + // timeout instead, so this ceiling is what separates the two + Assert.assertTrue("the poll loop ran on past the cancellation: " + elapsedMillis + "ms", + elapsedMillis < 10_000); + Assert.assertTrue("the loop must not keep polling after the cancellation, saw " + + tokenPolls.get() + " polls", tokenPolls.get() <= 1); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshWithoutIdTokenAdoptsRotatedRefreshToken() throws Exception { + assertMemoryLeak(() -> { + // A refresh may legally answer 2xx without an id_token (RFC 6749 6; OIDC Core 12.2), which a + // groupsInToken client cannot serve - but that response is still a clean grant, and a ROTATING + // provider has already invalidated the refresh token we presented. Dropping the whole response + // therefore keeps a spent credential: every later refresh replays it, and a reuse-detecting + // provider answers a replay by revoking the entire token family. + // + // The observable is what reaches the wire: each refresh records the refresh_token it presented, + // so a replay shows up as the same value twice. The device endpoint is available for the FIRST + // sign-in only - afterwards it errors, so the interactive fallback cannot mint a fresh refresh + // token and mask which one the refresh path kept. + StringSink presented = new StringSink(); + AtomicInteger deviceAuthCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + if (deviceAuthCalls.getAndIncrement() == 0) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"error\":\"access_denied\"}"); + } + if (body.contains("grant_type=refresh_token")) { + if (presented.length() > 0) { + presented.put(','); + } + presented.put(refreshTokenParam(body)); + // a clean 2xx, no id_token, and the refresh token ROTATES + return MockOidcServer.json(200, tokenJson("ACCESS-R", null, "REFRESH-2", 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + Assert.assertEquals("ID-1", auth.signIn()); + + // signIn() attempts the silent refresh BEFORE prompting, so each of these two calls puts + // exactly one refresh on the wire and then fails over to a device endpoint that refuses. + for (int i = 0; i < 2; i++) { + expireCachedToken(auth); + try { + auth.signIn(); + Assert.fail("the device endpoint refuses after the first sign-in"); + } catch (OidcAuthException expected) { + Assert.assertTrue(expected.getMessage(), + expected.getMessage().contains("access_denied")); + } + } + + Assert.assertEquals("the rotated refresh token must replace the one the provider burned; " + + "presenting the same value twice is the replay a reuse-detecting provider " + + "answers by revoking the whole token family", + "REFRESH-1,REFRESH-2", presented.toString()); + } + }); + } + + @Test(timeout = 30_000) + public void testRefreshWithoutIdTokenFallsBackToInteractiveFlow() throws Exception { + assertMemoryLeak(() -> { + // groups are encoded in the token (the default enterprise config), so signIn() serves the + // id token. The cached token expires and the refresh response omits id_token (RFC 6749 makes + // it optional on refresh), so the client must re-run the interactive flow rather than fail. + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + // a refresh that returns a fresh access token but no id_token + return MockOidcServer.json(200, tokenJson("ACCESS-R", null, null, 3600)); + } + // the device-code grant: first a soon-expired token, then (after fallback) a fresh one + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 1)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", "ID-2", "REFRESH-2", 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + Assert.assertEquals("ID-1", auth.signIn()); + expireCachedToken(auth); + // the refresh returns no id_token, so the flow falls back to interactive sign-in and + // returns the fresh id token instead of throwing "returned no id_token" + Assert.assertEquals("ID-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (initial + fallback)", 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testServerErrorDuringPollingRetries() throws Exception { + assertMemoryLeak(() -> { + // the token endpoint returns a gateway 5xx with an empty body once (no JSON error), then a + // token. An empty-bodied upstream blip must be retried, not aborted as an "unexpected response" + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (tokenCalls.getAndIncrement() == 0) { + return MockOidcServer.json(502, ""); + } + return MockOidcServer.json(200, tokenJson("ACCESS-RECOVERED-5XX", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-RECOVERED-5XX", auth.signIn()); + Assert.assertEquals(2, tokenCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testSilentRefreshWhenTokenExpired() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger promptCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (body.contains("grant_type=refresh_token")) { + Assert.assertTrue(body, body.contains("refresh_token=REFRESH-1")); + return MockOidcServer.json(200, tokenJson("ACCESS-2", "ID-2", null, 3600)); + } + // initial device-code grant, hand out a token that is already expired vs the clock skew + return MockOidcServer.json(200, tokenJson("ACCESS-1", "ID-1", "REFRESH-1", 1)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, ch -> promptCalls.incrementAndGet())) { + Assert.assertEquals("ACCESS-1", auth.signIn()); + expireCachedToken(auth); + // the cached token is expired, so the second call refreshes silently + Assert.assertEquals("ACCESS-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run only once", 1, deviceCalls.get()); + Assert.assertEquals("the user must be prompted only once", 1, promptCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testSlowDownIncreasesIntervalAndSucceeds() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenCalls = new AtomicInteger(); + AtomicLong firstPollNanos = new AtomicLong(); + AtomicLong secondPollNanos = new AtomicLong(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + int call = tokenCalls.getAndIncrement(); + if (call == 0) { + firstPollNanos.set(System.nanoTime()); + return MockOidcServer.json(400, "{\"error\":\"slow_down\"}"); + } + if (call == 1) { + secondPollNanos.set(System.nanoTime()); + } + return MockOidcServer.json(200, tokenJson("ACCESS-S", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-S", auth.signIn()); + Assert.assertEquals(2, tokenCalls.get()); + // base interval is 1s; the slow_down must add ~5s, so the SECOND poll lands ~6s after + // the first. Assert the inter-poll gap directly, not just total elapsed - without the + // increment the gap would be ~1s. + long gapMillis = (secondPollNanos.get() - firstPollNanos.get()) / 1_000_000L; + Assert.assertTrue("inter-poll gap=" + gapMillis + "ms", gapMillis >= 4_000); + } + }); + } + + @Test(timeout = 30_000) + public void testStalledResponseBodyAbortsWithinTimeout() throws Exception { + assertMemoryLeak(() -> { + // a server that sends headers then stalls the body must not wedge the thread on the 10-minute + // HttpClient default timeout; the body read aborts on the configured OIDC timeout instead + MockOidcServer.Handler handler = (method, path, body) -> MockOidcServer.stall(); + try (MockOidcServer server = new MockOidcServer(handler)) { + long startNanos = System.nanoTime(); + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .httpTimeoutMillis(1_000) + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + auth.signIn(); + Assert.fail("expected the stalled body read to abort"); + } catch (OidcAuthException e) { + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // aborted on the configured ~1s OIDC timeout: not instantly (which would be a different + // failure path) and not on the 600s HttpClient default (or an indefinite wedge). The window + // proves the 1s timeout fired, with generous headroom for a slow CI host + Assert.assertTrue("aborted too fast to be the 1s timeout: " + elapsedMillis + "ms", elapsedMillis >= 500); + Assert.assertTrue("aborted too slowly for the 1s timeout: " + elapsedMillis + "ms", elapsedMillis < 5_000); + } + } + }); + } + + @Test(timeout = 30_000) + public void testTimesOutWhenCodeExpires() throws Exception { + assertMemoryLeak(() -> { + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + // very short lifetime so the poll loop gives up quickly + return MockOidcServer.json(200, deviceAuthorizationJson(1, 1)); + } + return MockOidcServer.json(400, "{\"error\":\"authorization_pending\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "timed out", "expected a timeout"); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenAlongsideOauthErrorIsRejected() throws Exception { + assertMemoryLeak(() -> { + // RFC 6749 5.2: an error response must not be treated as a grant even if the body also carries + // a token. A hostile or buggy IdP returns access_denied together with an access_token; the + // client must surface the error, not cache the smuggled token + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"error\":\"access_denied\",\"access_token\":\"SHOULD-NOT-BE-USED\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + try { + auth.signIn(); + Assert.fail("expected the error response to be rejected, not the smuggled token accepted"); + } catch (OidcAuthException e) { + Assert.assertEquals("access_denied", e.getOauthError()); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("SHOULD-NOT-BE-USED")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testTokenCachedAcrossCalls() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + tokenCalls.incrementAndGet(); + return MockOidcServer.json(200, tokenJson("ACCESS-C", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-C", auth.signIn()); + Assert.assertEquals("ACCESS-C", auth.signIn()); + Assert.assertEquals("ACCESS-C", auth.signIn()); + Assert.assertEquals("the interactive flow must run only once", 1, deviceCalls.get()); + Assert.assertEquals("the token endpoint must be hit only once", 1, tokenCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenEndpointErrorDoesNotLeakSecretsInMessage() throws Exception { + assertMemoryLeak(() -> { + final String secret = "SUPER-SECRET-TOKEN-VALUE-0123456789"; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + // a 4xx (terminal) carrying a token but malformed JSON: the parser fails, and the raw body + // (with the token) must NOT be echoed into the exception message + return MockOidcServer.json(400, "{\"access_token\":\"" + secret + "\" not-valid-json}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "httpStatus="); + Assert.assertFalse("the token must not leak into the message: " + e.getMessage(), + e.getMessage().contains(secret)); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenResponseExpiresInIsClamped() throws Exception { + assertMemoryLeak(() -> { + // an absurd token-response expires_in (here Integer.MAX_VALUE, ~68 years) must be clamped to + // MAX_EXPIRES_IN_SECONDS (1h) like the device-side value, so the client does not trust a stale + // cached token for decades (the server still enforces the real expiry). + AtomicInteger deviceCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + // no refresh_token, so an expired cache forces a fresh device flow rather than a silent refresh + return MockOidcServer.json(200, tokenJson("ACCESS-OK", null, null, Integer.MAX_VALUE)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .scope("openid") + .prompt(noopPrompt()) + .allowInsecureTransport(true) + .build()) { + long before = System.currentTimeMillis(); + Assert.assertEquals("ACCESS-OK", auth.signIn()); + long after = System.currentTimeMillis(); + Assert.assertEquals("first sign-in runs the device flow once", 1, deviceCalls.get()); + + // the cached expiry must be ~1h out (the clamp), not ~68 years + long maxLifetimeMillis = 3600L * 1000L; + long expiresAt = readExpiresAtMillis(auth); + Assert.assertTrue("expiry must be clamped to <= 1h ahead, was " + (expiresAt - before) + "ms ahead", + expiresAt <= after + maxLifetimeMillis); + Assert.assertTrue("expiry must be ~1h ahead (the clamp), was " + (expiresAt - after) + "ms ahead", + expiresAt >= before + maxLifetimeMillis - 5_000L); + + // once the clamped token is past expiry, with no refresh token signIn() re-runs the device flow + expireCachedToken(auth); + Assert.assertEquals("ACCESS-OK", auth.signIn()); + Assert.assertEquals("expired clamped token forces a fresh sign-in", 2, deviceCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenResponseExpiresInZeroUsesDefaultTtl() throws Exception { + assertMemoryLeak(() -> { + // a token response with a non-positive expires_in (here 0) must fall back to + // DEFAULT_TOKEN_TTL_SECONDS (5 min), not be treated as already-expired or cached forever. + // testTokenResponseExpiresInIsClamped covers the absurd-large end; this covers the <= 0 default. + AtomicInteger deviceCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + // no refresh_token, so an expired cache forces a fresh device flow rather than a silent refresh + return MockOidcServer.json(200, tokenJson("ACCESS-DEF", null, null, 0)); // expires_in = 0 + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + long before = System.currentTimeMillis(); + Assert.assertEquals("ACCESS-DEF", auth.signIn()); + long after = System.currentTimeMillis(); + Assert.assertEquals("first sign-in runs the device flow once", 1, deviceCalls.get()); + + // the cached expiry must be ~5min out (the default), neither ~now (treated as expired) nor far + long defaultTtlMillis = 300L * 1000L; + long expiresAt = readExpiresAtMillis(auth); + Assert.assertTrue("expiry must be ~5min ahead (the default), was " + (expiresAt - after) + "ms ahead", + expiresAt >= before + defaultTtlMillis - 5_000L); + Assert.assertTrue("expiry must be ~5min ahead (the default), not longer, was " + (expiresAt - before) + "ms ahead", + expiresAt <= after + defaultTtlMillis); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenUnderNonSuccessStatusIsNotAccepted() throws Exception { + assertMemoryLeak(() -> { + // RFC 6749 5.1: a token must come from a 2xx response. A token under a non-2xx status with no + // OAuth error is a malformed or hostile answer; the client must not cache it - a 4xx is a + // terminal rejection that fails fast rather than trusting the token + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"access_token\":\"SHOULD-NOT-BE-USED\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "rejected the request", + "expected a token under a 400 to be rejected, not accepted"); + Assert.assertFalse(e.getMessage(), e.getMessage().contains("SHOULD-NOT-BE-USED")); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenWithControlCharsRejected() throws Exception { + assertMemoryLeak(() -> { + // a hostile or man-in-the-middled identity provider returns an access token whose JSON value carries + // an escaped CR/LF; the lexer decodes it to real control bytes, which - sent verbatim in the + // Authorization header to the trusted QuestDB server - would inject into the request line. storeTokens + // must reject the token rather than cache and serve it, and must not leak the token into the message + String injected = "header.payload" + jsonUnicodeEscape(0x0d) + jsonUnicodeEscape(0x0a) + "X-Injected:1"; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson(injected, null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "disallowed control or non-ASCII", + "expected a token with control characters to be rejected"); + // the token bytes must never leak into the message + Assert.assertFalse(e.getMessage(), e.getMessage().contains("X-Injected")); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenWithNonAsciiCharRejected() throws Exception { + assertMemoryLeak(() -> { + // the > 0x7e arm of the token guard (testTokenWithControlCharsRejected covers the < 0x20 arm): + // a non-ASCII char (here U+00E9, not a control char) in the access token would be silently + // truncated to one byte by the ASCII Authorization-header writer, yielding a corrupt credential. + // storeTokens must reject it, and must not leak the token into the message + String injected = "header.payload" + jsonUnicodeEscape(0x00e9) + "SHOULD-NOT-LEAK"; // e-acute, > 0x7e + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, tokenJson(injected, null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + OidcAuthException e = assertOidcFails(auth::signIn, "disallowed control or non-ASCII", + "expected a token with a non-ASCII character to be rejected"); + // the token bytes must never leak into the message + Assert.assertFalse(e.getMessage(), e.getMessage().contains("SHOULD-NOT-LEAK")); + } + }); + } + + @Test(timeout = 30_000) + public void testTransientParseFailureDuringPollingRecovers() throws Exception { + assertMemoryLeak(() -> { + // the token endpoint returns a garbled (non-JSON) body once, then a valid token; a transient + // parse failure is retried like a transport blip rather than aborting the sign-in + AtomicInteger tokenCalls = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (tokenCalls.getAndIncrement() == 0) { + return MockOidcServer.json(200, "502 Bad Gateway"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-RECOVERED", null, null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("ACCESS-RECOVERED", auth.signIn()); + Assert.assertEquals(2, tokenCalls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testTruncatedSettingsResponseRejected() throws Exception { + assertMemoryLeak(() -> { + // the /settings body is cut off mid-object (HTTP framing satisfied, JSON unterminated). discovery + // must reject it as a parse failure, not silently discover from the partial document and report a + // misleading "does not advertise ..." error + MockOidcServer.Handler handler = (method, path, body) -> + MockOidcServer.json(200, "{\"config\":{\"acl.oidc.enabled\":true,\"acl.oidc.client.id\":\"questdb\""); + try (MockOidcServer server = new MockOidcServer(handler)) { + try (OidcDeviceAuth ignored = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure())) { + Assert.fail("expected discovery to reject the truncated settings body"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("could not parse")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testTruncatedTokenResponseRejected() throws Exception { + assertMemoryLeak(() -> { + // a token response whose Content-Length is satisfied but whose JSON is unterminated must be + // rejected (parseLast catches the dangling value), not silently treated as no token. A 4xx makes + // the parse failure terminal so it surfaces immediately (a malformed 2xx is retried as transient). + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(400, "{\"access_token\":\"abc"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "could not parse"); + } + }); + } + + @Test(timeout = 30_000) + public void testUnexpectedTokenResponseRejected() throws Exception { + assertMemoryLeak(() -> { + // the token endpoint returns 200 with neither tokens nor an error + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + return MockOidcServer.json(200, "{\"token_type\":\"Bearer\"}"); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + assertOidcFails(auth::signIn, "unexpected response"); + } + }); + } + + @Test(timeout = 30_000) + public void testUnreachableDeviceEndpointThrowsOidcAuthException() throws Exception { + assertMemoryLeak(() -> { + // a connection failure to the device endpoint must surface as OidcAuthException (signIn's + // documented failure type), not a raw HttpClientException + int deadPort; + try (ServerSocket probe = new ServerSocket(0, 1, InetAddress.getLoopbackAddress())) { + deadPort = probe.getLocalPort(); + } // closed now - nothing listens on deadPort + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint("http://127.0.0.1:" + deadPort + "/device") + .tokenEndpoint("http://127.0.0.1:" + deadPort + "/token") + .allowInsecureTransport(true) + .prompt(noopPrompt()) + .build()) { + auth.signIn(); + Assert.fail("expected an OidcAuthException"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("device authorization endpoint")); + } + }); + } + + @Test(timeout = 30_000) + public void testUseAfterCloseThrowsClearly() { + // calling signIn()/clearCache() after close() must fail with a clear "closed" error rather than + // NPE on the freed JSON lexer or resurrect (and leak) a fresh native HTTP client + long parserMemBefore = Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS); + // close() is the subject under test, so it is called explicitly mid-body; the try-with-resources + // close at scope exit is a harmless idempotent second close that also covers an early assertion throw + try (OidcDeviceAuth auth = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint("https://idp.example/device") + .tokenEndpoint("https://idp.example/token") + .build() + ) { + auth.close(); + assertOidcFails(auth::signIn, "closed", "expected signIn() after close() to be rejected"); + try { + auth.clearCache(); + Assert.fail("expected clearCache() after close() to be rejected"); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("closed")); + } + // signIn() must reject before resurrecting a native HTTP client, and close() must have freed + // the JSON lexer, so the parser-tag memory returns to its pre-construction level + Assert.assertEquals("a closed instance must not leak or resurrect native memory", + parserMemBefore, Unsafe.getMemUsedByTag(MemoryTag.NATIVE_TEXT_PARSER_RSS)); + } + } + + @Test(timeout = 30_000) + public void testVerificationUrlAliasesParsed() throws Exception { + assertMemoryLeak(() -> { + // some identity providers (historically Google) return verification_url / verification_url_complete + // instead of the RFC 8628 verification_uri / verification_uri_complete; both spellings must populate + // the challenge shown to the user + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV-CODE\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_url\":\"https://verify.example/device\"," + + "\"verification_url_complete\":\"https://verify.example/device?user_code=WDJB-MJHT\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.json(200, tokenJson("ACCESS-ALIAS", null, null, 3600)); + }; + AtomicReference shown = new AtomicReference<>(); + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, shown::set)) { + Assert.assertEquals("ACCESS-ALIAS", auth.signIn()); + DeviceAuthorizationChallenge challenge = shown.get(); + Assert.assertNotNull(challenge); + Assert.assertEquals("https://verify.example/device", challenge.getVerificationUri()); + Assert.assertEquals("https://verify.example/device?user_code=WDJB-MJHT", challenge.getVerificationUriComplete()); + } + }); + } + + @Test(timeout = 30_000) + public void testWrongTokenKindDoesNotWedgeCache() throws Exception { + assertMemoryLeak(() -> { + // groups-in-token mode, but the IdP returns only an access token on the first grant (e.g. the + // requested scope omitted openid). signIn() must fail the first call, then re-run the + // interactive flow on the next call - not cache the unusable access token as valid and keep + // throwing "no id_token" on every later call + AtomicInteger deviceCalls = new AtomicInteger(); + AtomicInteger deviceCodeGrants = new AtomicInteger(); + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + deviceCalls.incrementAndGet(); + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + // first grant: access token only (no id_token); second grant: a proper id token + if (deviceCodeGrants.getAndIncrement() == 0) { + return MockOidcServer.json(200, tokenJson("ACCESS-ONLY", null, null, 3600)); + } + return MockOidcServer.json(200, tokenJson("ACCESS-2", "ID-2", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, true, noopPrompt())) { + assertOidcFails(auth::signIn, "no id_token", "expected an OidcAuthException on the first call"); + // the unusable grant must NOT be cached as valid: the next call re-runs the flow and succeeds + Assert.assertEquals("ID-2", auth.signIn()); + Assert.assertEquals("the interactive flow must run twice (failed first, recovered second)", 2, deviceCalls.get()); + } + }); + } + + private static void assertBuildFails(String deviceEndpoint, String tokenEndpoint, String expectedMessage) { + try (OidcDeviceAuth ignored = OidcDeviceAuth.builder() + .clientId("c") + .deviceAuthorizationEndpoint(deviceEndpoint) + .tokenEndpoint(tokenEndpoint) + .build() + ) { + Assert.fail("expected build to fail for device=" + deviceEndpoint + " token=" + tokenEndpoint); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + } + } + + private static void assertFromQuestDbDiscoveryAcceptsTrailingSlashIssuer(String issuerPath) throws Exception { + final String endpointPrefix = issuerPath.substring(0, issuerPath.length() - 1); + final String discoveryPath = endpointPrefix + WELL_KNOWN_PATH; + final String devicePath = endpointPrefix + DEVICE_PATH; + final String tokenPath = endpointPrefix + TOKEN_PATH; + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + if (SETTINGS_PATH.equals(path)) { + return MockOidcServer.json(200, settingsJson(true, false, server.httpUrl(tokenPath), null)); + } + if (discoveryPath.equals(path)) { + return MockOidcServer.json(200, wellKnownJson( + server.httpUrl(devicePath), + server.httpUrl(tokenPath), + server.httpUrl(issuerPath) + )); + } + if (devicePath.equals(path)) { + return MockOidcServer.json(200, deviceAuthorizationJson(1, 300)); + } + if (tokenPath.equals(path)) { + return MockOidcServer.json(200, tokenJson("ACCESS-TRAILING-SLASH", "ID-TRAILING-SLASH", null, 3600)); + } + return MockOidcServer.json(404, "{}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB( + server.httpUrl(""), + insecure().issuer(server.httpUrl(issuerPath)) + )) { + Assert.assertEquals("ID-TRAILING-SLASH", auth.signIn()); + } + } + } + + /** + * Fails unless NO sink or String reachable from {@code instance} - its own fields, and the fields of any + * client object they point at - still carries {@code secret}. A StringSink is read through its whole + * backing array, not just up to its write position, because that tail is precisely what a plain clear() + * leaves behind. + */ + private static void assertHoldsNowhere(Object instance, String secret) throws Exception { + String holder = findHolderOf(instance, secret); + Assert.assertNull("close() left \"" + secret + "\" readable in " + holder, holder); + } + + private static void assertHoldsSomewhere(Object instance, String secret) throws Exception { + Assert.assertNotNull("the state this test is about was never there: no field holds \"" + secret + '"', + findHolderOf(instance, secret)); + } + + /** + * Drives one issuer-path case through the PUBLIC path a user takes: a QuestDB {@code /settings} that + * advertises {@code devicePath} while the caller pins the issuer to {@code /realms/acme}. Asserts the + * outcome a caller sees - fromQuestDB throwing - rather than the return value of the private scan, so a + * rename or an inline of that scan leaves the coverage intact. The sibling scenario tests + * (testIssuerPathScopingRejectsEncodedSlash and friends) use the same shape; this exists so the encoding + * table can stay a table. + */ + private static void assertIssuerScope(String devicePath, boolean accepted) throws Exception { + AtomicReference serverRef = new AtomicReference<>(); + MockOidcServer.Handler handler = (method, path, body) -> { + MockOidcServer server = serverRef.get(); + return MockOidcServer.json(200, "{\"config\":{" + + "\"acl.oidc.enabled\":true," + + "\"acl.oidc.client.id\":\"questdb\"," + + "\"acl.oidc.token.endpoint\":\"" + server.httpUrl("/realms/acme/token") + "\"," + + "\"acl.oidc.device.authorization.endpoint\":\"" + server.httpUrl(devicePath) + "\"" + + "}}"); + }; + try (MockOidcServer server = new MockOidcServer(handler)) { + serverRef.set(server); + final String issuer = server.httpUrl("/realms/acme"); + if (accepted) { + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(issuer))) { + Assert.assertNotNull("an endpoint genuinely under the issuer path must be accepted: " + + devicePath, auth); + } + } else { + assertOidcFails(() -> OidcDeviceAuth.fromQuestDB(server.httpUrl(""), insecure().issuer(issuer)), + "not under the pinned issuer", + "an endpoint that escapes the issuer path must be rejected: " + devicePath); + } + } + } + + private static void assertIssuerScopeAccepts(String devicePath) throws Exception { + assertIssuerScope(devicePath, true); + } + + private static void assertIssuerScopeRejects(String devicePath) throws Exception { + assertIssuerScope(devicePath, false); + } + + private static void assertNoControlChars(String value) { + for (int i = 0; i < value.length(); i++) { + Assert.assertFalse("control char at index " + i + " in '" + value + "'", Character.isISOControl(value.charAt(i))); + } + } + + private static void assertNoUnsafeDisplayChars(String value) { + // mirrors OidcAuthException.isUnsafeForDisplay: no controls, no Cf format chars, no bidi/BOM - + // checked per code point so a supplementary-plane (>= U+10000) format/control char is not missed + for (int i = 0; i < value.length(); ) { + int cp = value.codePointAt(i); + boolean unsafe = Character.isISOControl(cp) + || Character.getType(cp) == Character.FORMAT + || Character.getType(cp) == Character.SURROGATE + || (cp >= 0x202A && cp <= 0x202E) + || (cp >= 0x2066 && cp <= 0x2069) + || cp == 0x200E || cp == 0x200F + || cp == 0xFEFF; + Assert.assertFalse("display-unsafe char U+" + Integer.toHexString(cp) + " at index " + i + " in '" + value + "'", unsafe); + i += Character.charCount(cp); + } + } + + /** + * Asserts that {@code call} - a {@code signIn()}, a {@code getToken()} or a discovery that must not + * succeed - throws an {@link OidcAuthException} whose message carries {@code expectedMessage}, and hands + * that exception back so a caller with more to check keeps asserting on it. Same idiom as + * {@link #assertBuildFails}, applied to the seven-line try/fail/catch this file used to stamp out at + * roughly thirty sites. + */ + private static OidcAuthException assertOidcFails(Supplier call, String expectedMessage) { + return assertOidcFails(call, expectedMessage, "the call must not succeed"); + } + + private static OidcAuthException assertOidcFails(Supplier call, String expectedMessage, String whatMustFail) { + final Object returned; + try { + returned = call.get(); + } catch (OidcAuthException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + return e; + } + // The call SUCCEEDED. Close what it handed back before failing - a construction that should have been + // rejected must not leak past the assertion - then report the value itself, which on a token path IS + // the credential an over-permissive check let through. + if (returned instanceof AutoCloseable) { + try { + ((AutoCloseable) returned).close(); + } catch (Exception ignore) { + // the failure below is the one that matters + } + } + throw new AssertionError(whatMustFail + " [expected an OidcAuthException containing \"" + expectedMessage + + "\", got " + returned + ']'); + } + + private static boolean awaitInside(Thread t, String method, long timeoutMillis) throws InterruptedException { + // poll the thread's own stack until the named OidcDeviceAuth frame shows up: the only evidence that a + // helper thread has actually ENTERED a call, as opposed to having been scheduled at all + final long deadline = System.currentTimeMillis() + timeoutMillis; + while (System.currentTimeMillis() < deadline) { + if (isInside(t, method)) { + return true; + } + Thread.sleep(10); + } + return false; + } + + private static String deviceAuthorizationJson(int interval, int expiresIn) { + return "{" + + "\"device_code\":\"DEV-CODE\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"verification_uri_complete\":\"https://verify.example/device?user_code=WDJB-MJHT\"," + + "\"expires_in\":" + expiresIn + "," + + "\"interval\":" + interval + + "}"; + } + + // DiscoveryOptions permitting insecure http with a no-op prompt: tests must never print to the console + // or try to open a real browser, which the default prompt now does. The common shape for tests reaching + // a plaintext mock server. + private static OidcDeviceAuth.DiscoveryOptions insecure() { + return new OidcDeviceAuth.DiscoveryOptions().allowInsecureTransport(true).prompt(noopPrompt()); + } + + @Test(timeout = 30_000) + public void testControlCharInUnusedTokenKindDoesNotAbortGrant() throws Exception { + assertMemoryLeak(() -> { + // groupsInToken=false, so signIn() serves and sends only the access_token; the id_token is + // cached but never placed in a header or a PG-wire password. A control char in that unused id_token + // must not reject an otherwise-usable grant - only the served kind is validated for wire safety + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + // a clean access_token (the served kind) alongside an id_token carrying a decoded control char + return MockOidcServer.json(200, tokenJson("CLEAN-ACCESS", "bad" + jsonUnicodeEscape(0x0001) + "id", null, 3600)); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + Assert.assertEquals("CLEAN-ACCESS", auth.signIn()); + } + }); + } + + @Test(timeout = 30_000) + public void testShortAllDigitStatusIsNotTreatedAsSuccess() throws Exception { + assertMemoryLeak(() -> { + // a real HTTP status is exactly 3 digits; a malformed 1-digit "2" (all digits, so readResponse + // accepts it) must not be classified as a 2xx success by its leading digit and accepted as a grant + String tokenBody = tokenJson("SHOULD-NOT-ACCEPT", null, null, 3600); + String rawToken = "HTTP/1.1 2 OK\r\n" + + "Content-Type: application/json\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(tokenBody.length()) + "\r\n" + tokenBody + "\r\n" + + "0\r\n\r\n"; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.raw(rawToken); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + try { + auth.signIn(); + Assert.fail("expected a malformed 1-digit status to be rejected, not accepted as success"); + } catch (OidcAuthException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("rejected the request") || msg.contains("refusing to keep polling")); + Assert.assertFalse("the unaccepted token must not leak: " + msg, msg.contains("SHOULD-NOT-ACCEPT")); + } + } + }); + } + + @Test(timeout = 30_000) + public void testShortAllDigitStatusNotTreatedAsTransientOrTerminal() throws Exception { + // a real HTTP status is exactly 3 digits. A malformed 1-digit "5" must not be read as a transient 5xx + // (which would poll on to the device-code deadline), nor a 1-digit "4" as a terminal 4xx, by the leading + // digit alone; both fall through to the fast terminal reject rather than an infinite poll. + for (String shortStatus : new String[]{"5", "4"}) { + assertMemoryLeak(() -> { + String tokenBody = tokenJson("SHOULD-NOT-ACCEPT", null, null, 3600); + String rawToken = "HTTP/1.1 " + shortStatus + " X\r\n" + + "Content-Type: application/json\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(tokenBody.length()) + "\r\n" + tokenBody + "\r\n" + + "0\r\n\r\n"; + MockOidcServer.Handler handler = (method, path, body) -> { + if (DEVICE_PATH.equals(path)) { + return MockOidcServer.json(200, "{" + + "\"device_code\":\"DEV\"," + + "\"user_code\":\"WDJB-MJHT\"," + + "\"verification_uri\":\"https://verify.example/device\"," + + "\"expires_in\":300," + + "\"interval\":1" + + "}"); + } + return MockOidcServer.raw(rawToken); + }; + try (MockOidcServer server = new MockOidcServer(handler); + OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) { + try { + auth.signIn(); + Assert.fail("expected malformed 1-digit status '" + shortStatus + "' to be rejected fast"); + } catch (OidcAuthException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("rejected the request") || msg.contains("refusing to keep polling")); + // must NOT have polled to the device-code deadline (that would be a mis-classified transient) + Assert.assertFalse(msg, msg.contains("device code expired")); + Assert.assertFalse("the unaccepted token must not leak: " + msg, msg.contains("SHOULD-NOT-ACCEPT")); + } + } + }); + } + } + + // Forces the cached access/id token to look expired WITHOUT dropping the refresh token, so the next + // signIn()/getToken() takes the silent-refresh (or interactive re-sign-in) path. Reflection + // because the field is private and there is no configurable clock skew to lean on anymore; the client is + // an open module, so this reaches it without widening production visibility for the test. + // package-private, not private: OidcDeviceAuthPersistenceTest needs the same thing and a second copy of + // this reflection would be the third in the package. There is no non-reflective route - expires_in is + // clamped to a default when non-positive, and the smallest usable value still leaves a live window that + // would have to be slept out. + // Disarms the 5s stampede latch a failed silent refresh arms, so a test can drive two refresh attempts + // back to back without sleeping through MIN_REFRESH_RETRY_INTERVAL_MILLIS. + static void clearRefreshBackOff(OidcDeviceAuth auth) throws Exception { + Field f = OidcDeviceAuth.class.getDeclaredField("refreshFailedAtMillis"); + f.setAccessible(true); + f.setLong(auth, 0L); + } + + static void expireCachedToken(OidcDeviceAuth auth) throws Exception { + Field f = OidcDeviceAuth.class.getDeclaredField("expiresAtMillis"); + f.setAccessible(true); + f.setLong(auth, 0L); // any "now" is past 0 minus the (capped, non-negative) skew, so the token reads as expired + } + + // Reads the cached token's absolute expiry (epoch millis) so a test can assert the lifetime clamp directly. + private static Object readField(Object instance, String name) throws Exception { + Field f = instance.getClass().getDeclaredField(name); + f.setAccessible(true); + return f.get(instance); + } + + private static long readExpiresAtMillis(OidcDeviceAuth auth) throws Exception { + Field f = OidcDeviceAuth.class.getDeclaredField("expiresAtMillis"); + f.setAccessible(true); + return f.getLong(auth); + } + + // isEndpointUnderIssuerPath is a private static security check (it scopes a /settings-advertised endpoint + // to the pinned issuer's path); the client is an open module, so reflection reaches it without widening + // production visibility for the test + /** + * Returns a description of the first field holding {@code secret}, or null when none does. Walks the + * instance's declared fields and, one level deeper, the fields of any {@code io.questdb.client} object + * among them - which is what reaches the sinks inside the two response parsers. + */ + private static String findHolderOf(Object instance, String secret) throws Exception { + for (Field f : instance.getClass().getDeclaredFields()) { + if (Modifier.isStatic(f.getModifiers())) { + continue; + } + f.setAccessible(true); + Object value = f.get(instance); + if (value == null) { + continue; + } + if (value instanceof StringSink && sinkContents((StringSink) value).contains(secret)) { + return instance.getClass().getSimpleName() + '.' + f.getName(); + } + if (value instanceof String && ((String) value).contains(secret)) { + return instance.getClass().getSimpleName() + '.' + f.getName(); + } + if (value != instance && value.getClass().getName().startsWith("io.questdb.client.") + && !(value instanceof StringSink)) { + for (Field nested : value.getClass().getDeclaredFields()) { + if (Modifier.isStatic(nested.getModifiers())) { + continue; + } + nested.setAccessible(true); + Object nestedValue = nested.get(value); + if (nestedValue instanceof StringSink + && sinkContents((StringSink) nestedValue).contains(secret)) { + return f.getName() + '.' + nested.getName(); + } + if (nestedValue instanceof String && ((String) nestedValue).contains(secret)) { + return f.getName() + '.' + nested.getName(); + } + } + } + } + return null; + } + + // Whether a thread is currently executing the named OidcDeviceAuth method, read off its stack. Used to + // wait for a peer to be parked inside a specific phase rather than sleeping and hoping. + private static boolean isInside(Thread t, String method) { + for (StackTraceElement frame : t.getStackTrace()) { + if (OidcDeviceAuth.class.getName().equals(frame.getClassName()) + && method.equals(frame.getMethodName())) { + return true; + } + } + return false; + } + + // builds a JSON unicode escape (backslash-u-XXXX) for a BMP code point without writing one literally + // in this source (char 92 is REVERSE SOLIDUS), so the file stays ASCII; the client's JSON lexer decodes + // the escape back into the real character, exercising the same decode-then-display path a hostile IdP hits + private static String jsonUnicodeEscape(int codePoint) { + String hex = Integer.toHexString(codePoint); + return ((char) 92) + "u" + "0000".substring(hex.length()) + hex; + } + + /** + * The whole backing array of every {@link StringSink} the instance's {@code jsonLexer} owns, past the + * write position too - which is where a cleared but unwiped secret survives. Targets the lexer + * directly rather than going through findHolderOf, which reports only the FIRST holder it meets and + * would name the String field instead while the token was still cached. + */ + private static String lexerBuffers(OidcDeviceAuth auth) throws Exception { + Object lexer = readField(auth, "jsonLexer"); + Assert.assertNotNull("clearCache() must keep the lexer alive; close() is the one that frees it", + lexer); + StringSink out = new StringSink(); + for (Field f : lexer.getClass().getDeclaredFields()) { + if (Modifier.isStatic(f.getModifiers())) { + continue; + } + f.setAccessible(true); + Object value = f.get(lexer); + if (value instanceof StringSink) { + out.put(f.getName()).put('=').put(sinkContents((StringSink) value)).put(' '); + } + } + return out.toString(); + } + + /** + * The whole native split-value cache owned by the instance's {@link JsonLexer}, including the tail beyond + * its logical cacheSize. A completed split value resets cacheSize to zero, so inspecting only the logical + * range would miss precisely the retained credential this helper is meant to expose. + */ + private static String lexerNativeCache(OidcDeviceAuth auth) throws Exception { + Object lexer = readField(auth, "jsonLexer"); + Assert.assertNotNull("clearCache() must keep the lexer alive; close() is the one that frees it", lexer); + Field cacheField = JsonLexer.class.getDeclaredField("cache"); + Field capacityField = JsonLexer.class.getDeclaredField("cacheCapacity"); + cacheField.setAccessible(true); + capacityField.setAccessible(true); + long cache = cacheField.getLong(lexer); + int capacity = capacityField.getInt(lexer); + StringSink out = new StringSink(capacity); + for (int i = 0; i < capacity; i++) { + out.put((char) (Unsafe.getUnsafe().getByte(cache + i) & 0xff)); + } + return out.toString(); + } + + private static final class MissOnceReentrantLock extends ReentrantLock { + private int timedTryLockCalls; + private int untimedTryLockCalls; + + @Override + public boolean tryLock() { + untimedTryLockCalls++; + return untimedTryLockCalls > 1 && super.tryLock(); + } + + @Override + public boolean tryLock(long timeout, TimeUnit unit) throws InterruptedException { + timedTryLockCalls++; + return super.tryLock(timeout, unit); + } + } + + private static OidcDeviceAuth newAuth(MockOidcServer server, boolean groupsInToken, DeviceCodePrompt prompt) { + return OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(server.httpUrl(DEVICE_PATH)) + .tokenEndpoint(server.httpUrl(TOKEN_PATH)) + .scope("openid groups") + .groupsInToken(groupsInToken) + .prompt(prompt) + .allowInsecureTransport(true) + .build(); + } + + private static DeviceCodePrompt noopPrompt() { + return challenge -> { + }; + } + + /** + * The sink's WHOLE backing array as a String - past the write position too, which is where a cleared but + * unwiped secret survives. + */ + // Reads the refresh_token the client actually presented, so a test can tell a rotation from a replay. + // The form body is "grant_type=refresh_token&refresh_token=&client_id=...", so the leading '&' + // is what separates the parameter from the grant_type value that shares its name. + private static String refreshTokenParam(String body) { + final String marker = "&refresh_token="; + final int at = body.indexOf(marker); + if (at < 0) { + return ""; + } + final int from = at + marker.length(); + final int to = body.indexOf('&', from); + return to < 0 ? body.substring(from) : body.substring(from, to); + } + + private static String sinkContents(StringSink sink) throws Exception { + Field buffer = StringSink.class.getDeclaredField("buffer"); + buffer.setAccessible(true); + return new String((char[]) buffer.get(sink)); + } + + private static String settingsJson(boolean enabled, boolean withDeviceEndpoint, String tokenEndpoint, String deviceEndpoint) { + StringSink config = new StringSink(); + config.put("{\"config\":{"); + config.put("\"acl.oidc.enabled\":").put(Boolean.toString(enabled)).put(','); + config.put("\"acl.oidc.client.id\":\"questdb\","); + config.put("\"acl.oidc.scope\":\"openid groups\","); + config.put("\"acl.oidc.groups.encoded.in.token\":true,"); + config.put("\"acl.oidc.token.endpoint\":\"").put(tokenEndpoint).put('"'); + if (withDeviceEndpoint) { + config.put(",\"acl.oidc.device.authorization.endpoint\":\"").put(deviceEndpoint).put('"'); + } + config.put("},\"preferences.version\":0,\"preferences\":{}}"); + return config.toString(); + } + + private static String tokenJson(String accessToken, String idToken, String refreshToken, int expiresIn) { + StringSink sb = new StringSink(); + sb.put("{\"token_type\":\"Bearer\",\"expires_in\":").put(expiresIn); + if (accessToken != null) { + sb.put(",\"access_token\":\"").put(accessToken).put('"'); + } + if (idToken != null) { + sb.put(",\"id_token\":\"").put(idToken).put('"'); + } + if (refreshToken != null) { + sb.put(",\"refresh_token\":\"").put(refreshToken).put('"'); + } + sb.put('}'); + return sb.toString(); + } + + private static String wellKnownJson(String deviceEndpoint, String tokenEndpoint, String issuer) { + return "{" + + "\"issuer\":\"" + issuer + "\"," + + "\"authorization_endpoint\":\"" + issuer + "/authorize\"," + + "\"token_endpoint\":\"" + tokenEndpoint + "\"," + + "\"device_authorization_endpoint\":\"" + deviceEndpoint + "\"" + + "}"; + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTransportBudgetTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTransportBudgetTest.java new file mode 100644 index 000000000..7e238dec3 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTransportBudgetTest.java @@ -0,0 +1,98 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.HttpClientConfiguration; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import org.junit.Assert; +import org.junit.Test; + +import java.lang.reflect.Method; + +/** + * Pins that the HTTP clients {@link OidcDeviceAuth} builds take their CONNECTION budgets from + * {@code httpTimeoutMillis}, not from the transport defaults. + *

+ * {@code HttpClient.connect} reads both: it leaves the TCP connect to the OS when the connect timeout is 0, + * and it sizes the TLS handshake as {@code connectTimeout > 0 ? connectTimeout : defaultTimeout}. Taking + * {@code DefaultHttpClientConfiguration.INSTANCE} - 0 and 600s - therefore gave the handshake alone a 600s + * budget derived from nothing the caller set, so neither {@code MAX_HTTP_TIMEOUT_MILLIS} (120s) nor + * {@code Builder.build()}'s {@code lockStaleMillis} floor constrained it. + *

+ * The consequence is not a slow request. A silent refresh runs inside {@code FileTokenStore}'s cross-process + * lock, whose file is stamped once at creation and never re-stamped, so a hold outrunning + * {@code DEFAULT_LOCK_STALE_MILLIS} (600s) is judged abandoned and stolen by a peer. Both holders then POST + * the same rotating refresh token, and an identity provider with reuse detection revokes the whole family - + * on a headless producer, ingestion stops until a human re-runs the device flow. + *

+ * Asserted on the configuration rather than end to end because a real TLS handshake stall needs a + * certificate, and the client's test tree has none - the TLS fixture ({@code TlsProxyRule}) lives in the + * Enterprise tree, where {@code OidcDeviceAuthTlsTest} drives the flow over a real TLS socket. Reflection + * because every test class here is in {@code io.questdb.client.test.*}, so a package-private hook would not + * be reachable either; {@code FileTokenStoreTest} reaches this class's private statics the same way. + */ +public class OidcDeviceAuthTransportBudgetTest { + + @Test + public void testConnectionBudgetsDeriveFromTheHttpTimeout() throws Exception { + final Method httpConfig = OidcDeviceAuth.class.getDeclaredMethod("httpConfig", int.class); + httpConfig.setAccessible(true); + + // A value distinct from every default in play (0, 30_000, 600_000), so a config that quietly fell + // back to any of them fails rather than coincidentally matching. + final int timeoutMillis = 7_777; + final HttpClientConfiguration config = (HttpClientConfiguration) httpConfig.invoke(null, timeoutMillis); + + Assert.assertEquals("the TLS handshake budget is connectTimeout when it is positive, so leaving this " + + "at 0 hands the handshake the 600s request-timeout default instead", + timeoutMillis, config.getConnectTimeout()); + Assert.assertEquals("the request timeout must be the caller's figure, not the 600s default", + timeoutMillis, config.getTimeout()); + + // Guard the premise: a zero connect timeout is precisely what routes HttpClient.connect to the OS + // for the TCP connect and to defaultTimeout for the handshake, so a regression to the shared + // DefaultHttpClientConfiguration.INSTANCE reads as 0 here. + Assert.assertTrue("a positive connect timeout is what bounds both the TCP connect and the TLS " + + "handshake; 0 restores the unbounded shape", config.getConnectTimeout() > 0); + } + + @Test + public void testDiscoveryClientsCarryTheSameDerivedBudgets() throws Exception { + // Discovery runs before an instance exists, so it cannot take a builder value - but it reads + // /settings and .well-known from the same untrusted network position and must be bounded too. + final java.lang.reflect.Field discoveryConfig = OidcDeviceAuth.class.getDeclaredField("DISCOVERY_HTTP_CONFIG"); + discoveryConfig.setAccessible(true); + final HttpClientConfiguration config = (HttpClientConfiguration) discoveryConfig.get(null); + + final java.lang.reflect.Field defaultTimeout = OidcDeviceAuth.class.getDeclaredField("DEFAULT_HTTP_TIMEOUT_MILLIS"); + defaultTimeout.setAccessible(true); + final int expected = (Integer) defaultTimeout.get(null); + + Assert.assertEquals("discovery's connect/TLS budget must be the default HTTP timeout, not 0", + expected, config.getConnectTimeout()); + Assert.assertEquals("discovery's request timeout must be the default HTTP timeout, not 600s", + expected, config.getTimeout()); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/client/ChunkedResponseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/client/ChunkedResponseTest.java index 6bd20a94d..beda163d5 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/http/client/ChunkedResponseTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/client/ChunkedResponseTest.java @@ -26,6 +26,7 @@ import io.questdb.client.cutlass.http.client.AbstractChunkedResponse; import io.questdb.client.cutlass.http.client.Fragment; +import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Numbers; import io.questdb.client.std.ObjList; @@ -186,6 +187,185 @@ public void testFuzz() { createChunks(rnd, encoded.toString(), fragCount)); } + @Test(timeout = 30_000) + public void testNoArgRecvHonoursPositiveDefaultTimeout() { + // The ILP flush path reads a chunked response via the no-arg recv(), which delegates to + // recv(defaultTimeout). With a positive defaultTimeout (the production HttpClient timeout) the + // whole-call bound applies on that path too, so a server dribbling a never-terminated chunk size + // cannot wedge a single recv() past the timeout. The explicit recv(int) path is covered by + // testRecvHonoursTotalTimeoutWhileChunkSizeDribbles. + final long memSize = 64; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + final AbstractChunkedResponse rsp = new AbstractChunkedResponse(mem, mem + memSize, 50) { // positive default + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + if (bufLo >= bufHi) { + return 0; // buffer full of a CRLF-less chunk size: no forward progress + } + Unsafe.getUnsafe().putByte(bufLo, (byte) '0'); // a hex digit, never the terminating CR + return 1; + } + }; + rsp.begin(mem, mem); + try { + rsp.recv(); // no-arg: delegates to recv(defaultTimeout=50) + Assert.fail("expected the no-arg recv to time out on a dribbled, never-terminated chunk size"); + } catch (HttpClientException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("timed out")); + } + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + + @Test(timeout = 30_000) + public void testOverflowingChunkSizeIsRejectedRatherThanSpun() { + // A chunk-size line of 16 or more hex digits overflows an unchecked val << 4 accumulation, and the + // residue decides how the damage shows up. All three must be rejected; only the first one was. + // + // 8000000000000000 -> NEGATIVE. Matches neither the "size > 0" data branch nor the "size == 0" + // terminator, so the state machine breaks straight back to the top of the + // loop. The preceding chunk left receive == false with bytes still buffered, + // so the read gate is skipped too and the loop spins with nothing to stop it. + // 10000000000000000 -> ZERO, and 10000000000000001 -> 1; both report success with the wrong + // bytes, and get their own tests below. Rejecting only the negative case + // left those two, which are the dangerous ones: a spin is at least visible, + // whereas truncated JSON parses. + // + // defaultTimeout is -1 on purpose, so no deadline can rescue the spinning case and the test passes + // only because the size itself is rejected. The server chooses that size line, and for a discovery + // or token response that server is untrusted. + // a trailing byte keeps dataLo < dataHi so the read gate stays shut and the spin is reachable + assertChunkSizeRejected("8000000000000000", "X", "negative overflow residue"); + } + + @Test(timeout = 30_000) + public void testZeroWrappingChunkSizeIsRejectedRatherThanTruncating() { + // 10000000000000000 wraps to ZERO, which the state machine reads as the terminal chunk: recv() + // returns null and the caller sees a complete-looking body that is actually truncated. Worse than + // the spin the negative residue causes, because nothing looks wrong -- truncated JSON parses, and + // the connection's framing is lost for the next keep-alive response on it. The size line is chosen + // by the server, untrusted for an OIDC discovery or token response. + // A proper CRLF terminator here, so the pre-fix parser really does complete the body rather than + // stall waiting for one. + assertChunkSizeRejected("10000000000000000", "\r\n", "zero overflow residue"); + } + + @Test(timeout = 30_000) + public void testPositiveWrappingChunkSizeIsRejectedRatherThanMisframing() { + // 10000000000000001 wraps to 1: a one-byte data chunk that frames the following bytes as chunk + // furniture. Like the zero residue this reports success, just with the wrong bytes. + assertChunkSizeRejected("10000000000000001", "X", "positive overflow residue"); + } + + @Test + public void testZeroPaddedChunkSizeIsStillAccepted() { + // The guard counts SIGNIFICANT hex digits, so a server that pads its size line is not mistaken for + // one overflowing it. A raw length check would reject this - it is 20 characters against a 15-digit + // bound - and would break framing against a perfectly conformant peer, which is a worse failure + // than the one the bound exists to prevent. + final long memSize = 128; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + final String wire = "00000000000000000001\r\nZ\r\n0\r\n\r\n"; + final AbstractChunkedResponse rsp = new AbstractChunkedResponse(mem, mem + memSize, -1) { + boolean delivered; + + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + if (delivered) { + return 0; + } + delivered = true; + for (int i = 0; i < wire.length(); i++) { + Unsafe.getUnsafe().putByte(bufLo + i, (byte) wire.charAt(i)); + } + return wire.length(); + } + }; + rsp.begin(mem, mem); + Fragment first = rsp.recv(); + Assert.assertNotNull("a zero-padded size line must frame its chunk normally", first); + Assert.assertEquals('Z', (char) Unsafe.getUnsafe().getByte(first.lo())); + Assert.assertEquals(1, first.hi() - first.lo()); + Assert.assertNull("and the terminator must still terminate", rsp.recv()); + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + + private static void assertChunkSizeRejected(String sizeLine, String tail, String what) { + final long memSize = 128; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + // one well-formed chunk (leaves receive == false), then the overflowing size line, then a + // trailing byte so dataLo < dataHi holds the read gate shut + final String wire = "1\r\nA\r\n" + sizeLine + "\r\n" + tail; + final AbstractChunkedResponse rsp = new AbstractChunkedResponse(mem, mem + memSize, -1) { + boolean delivered; + + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + if (delivered) { + return 0; + } + delivered = true; + for (int i = 0; i < wire.length(); i++) { + Unsafe.getUnsafe().putByte(bufLo + i, (byte) wire.charAt(i)); + } + return wire.length(); + } + }; + rsp.begin(mem, mem); + Fragment first = rsp.recv(); + Assert.assertNotNull(what + ": the first chunk must still be delivered", first); + Assert.assertEquals('A', (char) Unsafe.getUnsafe().getByte(first.lo())); + try { + Fragment second = rsp.recv(); + Assert.fail(what + ": expected the overflowing chunk size to be rejected as malformed, got " + + (second == null ? "a terminal chunk (a truncated body reported as complete)" + : "a data chunk")); + } catch (HttpClientException e) { + Assert.assertTrue(what + ": " + e.getMessage(), + e.getMessage().contains("malformed chunk size")); + } + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + + @Test(timeout = 30_000) + public void testRecvHonoursTotalTimeoutWhileChunkSizeDribbles() { + // a server that dribbles the chunk-size line and never sends its terminating CRLF must not keep a + // single recv() running past its timeout. recv(timeout) bounds the whole call (not each socket read), + // so the loop scanning the never-terminated chunk size aborts once the timeout elapses. Without the + // bound this recv() never returns and the @Test timeout fires instead. + final long memSize = 64; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + final AbstractChunkedResponse rsp = new AbstractChunkedResponse(mem, mem + memSize, -1) { + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + if (bufLo >= bufHi) { + return 0; // buffer full of a CRLF-less chunk size: no forward progress + } + Unsafe.getUnsafe().putByte(bufLo, (byte) '0'); // a hex digit, never the terminating CR + return 1; + } + }; + rsp.begin(mem, mem); + try { + rsp.recv(50); + Assert.fail("expected recv to time out on a dribbled, never-terminated chunk size"); + } catch (HttpClientException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("timed out")); + } + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + @Test public void testSingleFragment() { String[] fragments = { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientConstructorLeakTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientConstructorLeakTest.java new file mode 100644 index 000000000..4670d5077 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientConstructorLeakTest.java @@ -0,0 +1,212 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.http.client; + +import io.questdb.client.DefaultHttpClientConfiguration; +import io.questdb.client.HttpClientConfiguration; +import io.questdb.client.cutlass.http.client.HttpClient; +import io.questdb.client.cutlass.http.client.HttpClientFactory; +import io.questdb.client.network.EpollFacade; +import io.questdb.client.network.EpollFacadeImpl; +import io.questdb.client.network.KqueueFacade; +import io.questdb.client.network.KqueueFacadeImpl; +import io.questdb.client.network.NetworkFacade; +import io.questdb.client.network.SelectFacade; +import io.questdb.client.std.Os; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.Test; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * A constructor that fails partway leaves an object nobody can close. It never reaches the caller, so no + * {@code finally}, no try-with-resources and no {@code close()} ever runs on it, and whatever it had already + * taken is lost for the life of the process. + *

+ * {@link HttpClient}'s base constructor takes a socket and two native buffers, then each platform subclass + * builds its poller. A poller that fails to initialise therefore stranded all of that. What makes it worth + * guarding is the trigger: {@code epoll_create}/{@code kqueue} fail on fd exhaustion, and the two mallocs + * fail under memory pressure, so the failure arrives exactly when resources are already scarce - and a + * caller that retries compounds the loss each time. OIDC discovery newly exposes it by building a client per + * fetch. + *

+ * One test covers the base constructor's own staging and runs everywhere; the other three cover the poller, + * and only the one matching the running platform executes. All assert through {@code assertMemoryLeak} that + * nothing survives the throw. The injection differs because the clean failure point does: epoll and kqueue + * are reached through a facade, so a facade returning a negative descriptor mimics fd exhaustion without + * touching real descriptors, while FDSet and the base buffers have no facade and take a failing size + * instead. Removing the rollback leaks 131072 bytes on the poller path and 65536 on the base path. + *

+ * Only the base test and the platform test for the developing machine can be run locally; the other two + * platforms' tests are exercised by CI. + */ +public class HttpClientConstructorLeakTest { + + @Test + public void testEpollCreateFailureLeaksNothing() throws Exception { + Assume.assumeTrue("epoll is the Linux poller", Os.type == Os.LINUX); + assertMemoryLeak(() -> assertConstructionFailureLeaksNothing(new DefaultHttpClientConfiguration() { + @Override + public EpollFacade getEpollFacade() { + return new EpollFacade() { + @Override + public int epollCreate() { + return -1; // as on fd exhaustion + } + + @Override + public int epollCtl(int epfd, int op, int fd, long eventPtr) { + return EpollFacadeImpl.INSTANCE.epollCtl(epfd, op, fd, eventPtr); + } + + @Override + public int epollWait(int epfd, long eventPtr, int eventCount, int timeout) { + return EpollFacadeImpl.INSTANCE.epollWait(epfd, eventPtr, eventCount, timeout); + } + + @Override + public int errno() { + return 24; // EMFILE + } + + @Override + public NetworkFacade getNetworkFacade() { + return EpollFacadeImpl.INSTANCE.getNetworkFacade(); + } + }; + } + })); + } + + @Test + public void testKqueueCreateFailureLeaksNothing() throws Exception { + Assume.assumeTrue("kqueue is the BSD/macOS poller", Os.type == Os.DARWIN || Os.type == Os.FREEBSD); + assertMemoryLeak(() -> assertConstructionFailureLeaksNothing(new DefaultHttpClientConfiguration() { + @Override + public KqueueFacade getKQueueFacade() { + return new KqueueFacade() { + @Override + public NetworkFacade getNetworkFacade() { + return KqueueFacadeImpl.INSTANCE.getNetworkFacade(); + } + + @Override + public int kevent(int kq, long changeList, int nChanges, long eventList, int nEvents, int timeout) { + return KqueueFacadeImpl.INSTANCE.kevent(kq, changeList, nChanges, eventList, nEvents, timeout); + } + + @Override + public int kqueue() { + return -1; // as on fd exhaustion + } + }; + } + })); + } + + @Test + public void testBaseConstructorFailureLeaksNothing() throws Exception { + // The base constructor's OWN staging, independent of any platform poller: it takes a socket, then the + // request buffer, then the response-parser buffer, then hands the last one to ResponseHeaders. A + // failure at any step must not strand the earlier ones. A negative response-buffer size makes the + // second malloc fail while the first has already succeeded - the shape a real allocation failure + // takes under memory pressure - and it needs no platform-specific injection point. + assertMemoryLeak(() -> assertConstructionFailureLeaksNothing(new DefaultHttpClientConfiguration() { + @Override + public int getResponseBufferSize() { + return -1; + } + })); + } + + @Test + public void testSelectFacadeFailureLeaksNothingIncludingTheFdSet() throws Exception { + Assume.assumeTrue("select/FDSet is the Windows poller", Os.type == Os.WINDOWS); + // The OTHER arm of the Windows guard, and the one the sibling above cannot reach: here FDSet is + // constructed successfully and the throw lands on the next statement, so the guard has to free the + // FDSet as well as everything the base constructor took. getSelectFacade() is a caller-supplied + // extension point evaluated inside the try for exactly this reason, and until now nothing drove it. + // Deterministic, with no arithmetic to rot. + assertMemoryLeak(() -> assertConstructionFailureLeaksNothing(new DefaultHttpClientConfiguration() { + @Override + public SelectFacade getSelectFacade() { + throw new IllegalStateException("injected select facade failure"); + } + })); + } + + @Test + public void testSelectFdSetFailureLeaksNothing() throws Exception { + Assume.assumeTrue("select/FDSet is the Windows poller", Os.type == Os.WINDOWS); + // FDSet reaches no facade, so the injection is its size instead: the constructor computes + // ARRAY_OFFSET + 8 * capacity in INT arithmetic, and a capacity that overflows it negative makes + // allocateMemory reject the size. An allocation that simply fails is the shape a real one takes + // under memory pressure, and FDSet throwing rather than the statement after it is what exercises + // the guard's null-tolerant Misc.free(fdSet). + // + // The capacity has to overflow to a LARGE negative, not merely a negative. Integer.MAX_VALUE - the + // obvious choice, and what this used - makes 8 * capacity exactly -8, so the size works out to + // ARRAY_OFFSET - 8: negative only where ARRAY_OFFSET is 0 or 4. On Windows fd_set is + // { u_int fd_count; SOCKET fd_array[]; } with an 8-byte SOCKET, so arrayOffset() reports 8, the + // size lands on exactly 0, and allocateMemory(0) succeeds and hands back a null pointer instead of + // failing - construction completed and the test asserted nothing. 1 << 28 makes 8 * capacity + // overflow to exactly Integer.MIN_VALUE, so the size is negative whatever arrayOffset() reports. + assertMemoryLeak(() -> assertConstructionFailureLeaksNothing(new DefaultHttpClientConfiguration() { + @Override + public int getWaitQueueCapacity() { + return 1 << 28; + } + })); + } + + private static void assertConstructionFailureLeaksNothing(HttpClientConfiguration configuration) { + HttpClient client = null; + // The "it threw" assertion CANNOT be an Assert.fail() inside the try: fail() throws AssertionError, + // which the catch below swallows, so an injection that stopped failing would report a green test + // having injected nothing - and all four tests share this helper, so all four would go green at once. + // The catch has to stay this broad, which is why the flag is needed rather than a narrower catch: the + // four injections share no supertype below Throwable. Epoll and Kqueue throw NetworkError, which + // extends Error, while the two failing allocations throw IllegalArgumentException out of + // Unsafe.malloc. + boolean threw = false; + try { + client = HttpClientFactory.newPlainTextInstance(configuration); + } catch (Throwable expected) { + // the point of the test is what assertMemoryLeak checks around it: the socket and the two native + // buffers the base constructor took must not survive a throw from the subclass + threw = true; + } finally { + // defensive: if construction unexpectedly succeeded, do not leak it out of the test + if (client != null) { + client.close(); + } + } + Assert.assertTrue( + "construction succeeded, so this test's injected failure no longer fires and it proved nothing", + threw + ); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientRequestTrimTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientRequestTrimTest.java new file mode 100644 index 000000000..748e72b9a --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientRequestTrimTest.java @@ -0,0 +1,103 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.http.client; + +import io.questdb.client.cutlass.http.client.HttpClient; +import io.questdb.client.cutlass.http.client.HttpClientFactory; +import org.junit.Assert; +import org.junit.Test; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Pins {@code Request.trimContentToLen}'s sentinel guard, whose absence is a SIGSEGV rather than a failed + * assertion. + *

+ * {@code newRequest()} sets {@code contentStart = -1} and only {@code withContent()} replaces it with a real + * address, so a request still at the header stage has no content section. Trimming one anyway computes + * {@code -1 + contentLen} as the write pointer, and the next write to the buffer takes the process down - + * surefire reports "The forked VM terminated without properly saying goodbye", with no test named. + *

+ * That state is ordinary rather than exotic: an ILP request built with an {@code httpTokenProvider} defers + * {@code withContent()} until the first row stamps the Authorization header, so it sits at the header stage + * between every flush and the next row - which is when {@code cancelRow()} can arrive. {@code cancelRow()} + * used to carry a second {@code isTokenPending} check of its own that returned before the trim. The two were + * mutually masking: removing either alone left the whole suite green, and only removing both crashed, so + * neither was pinned and either could have been dropped by a refactor with CI green. The caller-side one is + * gone; this pins the one that remains, which is also the only protection an external caller of this + * exported method has. + *

+ * Asserted on the pointer rather than by writing through it: an assertion names what broke, where a write + * would just kill the fork. + */ +public class HttpClientRequestTrimTest { + + @Test + public void testTrimContentToLenOnAHeaderStageRequestLeavesThePointerValid() throws Exception { + assertMemoryLeak(() -> { + try (HttpClient client = HttpClientFactory.newPlainTextInstance()) { + // header stage only - no withContent(), so contentStart is still the -1 sentinel. No socket + // is involved: newRequest() just resets the buffer and the request state. + HttpClient.Request request = client.newRequest("127.0.0.1", 9000); + request.GET().url("/write").header("Authorization", "Bearer GOODTOKEN"); + + Assert.assertEquals("precondition: no content section, so no content length", + 0, request.getContentLength()); + Assert.assertEquals("precondition: and getContentStart() reports 0, not the sentinel", + 0, request.getContentStart()); + final long ptrAfterHeaders = request.getPtr(); + Assert.assertTrue("precondition: the headers advanced the write pointer", + ptrAfterHeaders > 0); + + // what cancelRow() does on a row that never started + request.trimContentToLen(0); + Assert.assertEquals("trimming a request with no content section must not move the write " + + "pointer - contentStart is -1, so the arithmetic yields an invalid " + + "pointer the next write segfaults on", + ptrAfterHeaders, request.getPtr()); + + // and with a stale non-zero bookmark, which is what rowBookmark holds from the previous + // request until stampTokenIfPending resets it + request.trimContentToLen(37); + Assert.assertEquals("a stale non-zero bookmark must not move it either", + ptrAfterHeaders, request.getPtr()); + + // the request is still usable afterwards: the content section opens where it should, and + // writing through it does not touch a rewound pointer + request.withContent(); + final long contentStart = request.getContentStart(); + Assert.assertTrue("withContent() must open a real content section", contentStart > 0); + request.putAscii("t v=1i\n"); + Assert.assertEquals(7, request.getContentLength()); + + // now that a content section exists, the trim is a real rewind rather than a no-op + request.trimContentToLen(0); + Assert.assertEquals("with a content section, trimming must actually rewind", + 0, request.getContentLength()); + Assert.assertEquals(contentStart, request.getPtr()); + } + }); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientResponseHeadTimeoutTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientResponseHeadTimeoutTest.java new file mode 100644 index 000000000..f24d2b191 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/client/HttpClientResponseHeadTimeoutTest.java @@ -0,0 +1,95 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.http.client; + +import io.questdb.client.DefaultHttpClientConfiguration; +import io.questdb.client.HttpClientConfiguration; +import io.questdb.client.cutlass.http.client.HttpClient; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.http.client.HttpClientFactory; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import org.junit.Assert; +import org.junit.Test; + +/** + * Pins the whole-call bound on the response HEAD read, {@code ResponseHeaders.await(int)}. + *

+ * The body reads got this bound first ({@code AbstractResponse.recv}, {@code AbstractChunkedResponse.recv}); + * the head read kept re-arming the full timeout on every pass. That is not a slower version of the same + * thing, it is unbounded: {@code recvOrDie} returns 0 whenever a read yields no application bytes, a 0 + * leaves {@code totalBytesReceived} unmoved, and an unmoved counter neither advances the header parser nor + * fills its buffer - so the "header is too large" escape never fires either. + *

+ * It matters because {@code OidcDeviceAuth} reads this head from an identity provider on the + * {@code getToken()} path, which an ILP sender built with {@code httpTokenProvider} calls once per flush. + * The IdP endpoints are required to be {@code https}, and a partial TLS record decrypting to no application + * bytes is exactly the 0-length read above. + *

+ * Driven over plaintext with a head dribbled a byte at a time rather than with a stubbed {@code recvOrDie}: + * the point is the elapsed-time bound a caller asked for, and a dribbling peer defeats it the same way. + */ +public class HttpClientResponseHeadTimeoutTest { + + @Test(timeout = 30_000) + public void testAwaitHonoursTotalTimeoutWhileTheHeadDribbles() throws Exception { + // 500ms against a head dribbled at 50ms/byte: the bound must fire in ~500ms. Without it every read + // makes progress inside its own re-armed 500ms, so await() runs for (bytes x 50ms) and the @Test + // timeout fires instead of this assertion. + final int timeoutMillis = 500; + final HttpClientConfiguration config = new DefaultHttpClientConfiguration() { + @Override + public int getTimeout() { + return timeoutMillis; + } + }; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.dribbleHead())) { + try (HttpClient client = HttpClientFactory.newPlainTextInstance(config)) { + HttpClient.Request request = client.newRequest("127.0.0.1", server.port()) + .GET() + .url("/head"); + final HttpClient.ResponseHeaders headers = request.send(timeoutMillis); + final long startNanos = System.nanoTime(); + try { + headers.await(timeoutMillis); + Assert.fail("expected await to time out while the response head dribbled"); + } catch (HttpClientException e) { + // Either terminator is the bound working. Against a peer that dribbles, the shrinking + // per-pass budget starves ioWait's poll first, so the throw comes from there + // ("timed out [errno=..]"); against one whose reads yield no application bytes at all - + // the partial-TLS-record case, which consumes no budget - the loop's own deadline check + // fires instead ("timed out reading the response head"). What neither can do is keep + // running, which is what the elapsed assertion below pins. + Assert.assertTrue(e.getMessage(), e.getMessage().contains("timed out")); + } + final long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // The ceiling is the assertion that matters: it is what a per-read re-arm cannot satisfy. + // Generous against the 500ms budget so a loaded CI box does not turn a real bound into a + // red test, and still an order of magnitude below the unbounded behaviour. + Assert.assertTrue("await must abort on its own deadline, not run on with the dribble; took " + + elapsedMillis + "ms against a " + timeoutMillis + "ms budget", elapsedMillis < 10_000); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/client/ResponseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/client/ResponseTest.java index 6c9901db8..2867c27d0 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/http/client/ResponseTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/client/ResponseTest.java @@ -27,6 +27,7 @@ import io.questdb.client.cutlass.http.client.AbstractResponse; import io.questdb.client.cutlass.http.client.Fragment; +import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Os; import io.questdb.client.std.Unsafe; @@ -37,6 +38,34 @@ public class ResponseTest { + @Test(timeout = 30_000) + public void testNoArgRecvHonoursPositiveDefaultTimeout() { + // The ILP flush path reads via the no-arg recv(), which delegates to recv(defaultTimeout). With a + // positive defaultTimeout (the production HttpClient timeout) the whole-call bound applies on that + // path too, so a server yielding no application bytes cannot wedge a single recv() past the timeout. + // The explicit recv(int) path is covered by testRecvHonoursTotalTimeoutWhenNoApplicationBytesArrive. + final long memSize = 64; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + final AbstractResponse rsp = new AbstractResponse(mem, mem + memSize, 50) { // positive defaultTimeout + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + Os.sleep(1); // a readability wakeup that decrypts to no application bytes + return 0; + } + }; + rsp.begin(mem, mem, 16); // content length 16, nothing received yet + try { + rsp.recv(); // no-arg: delegates to recv(defaultTimeout=50) + Assert.fail("expected the no-arg recv to time out under a positive default timeout"); + } catch (HttpClientException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("timed out")); + } + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + @Test public void testNoSplit() { String[] expectedFragments = { @@ -48,6 +77,36 @@ public void testNoSplit() { assertResponse(expectedFragments, actualFragments); } + @Test(timeout = 30_000) + public void testRecvHonoursTotalTimeoutWhenNoApplicationBytesArrive() { + // A Content-Length response whose socket reads yield no application bytes - e.g. an incomplete or + // empty TLS record over a hostile or MITM'd link, where JavaTlsClientSocket.recv returns 0 on + // BUFFER_UNDERFLOW without a disconnect - must not keep a single recv() running past its timeout. + // recv(timeout) bounds the whole call, not each socket read, so the while (len == 0) loop aborts + // once the timeout elapses. Without the bound this recv() never returns and the @Test timeout + // fires instead. + final long memSize = 64; + final long mem = Unsafe.malloc(memSize, MemoryTag.NATIVE_DEFAULT); + try { + final AbstractResponse rsp = new AbstractResponse(mem, mem + memSize, -1) { + @Override + protected int recvOrDie(long bufLo, long bufHi, int timeout) { + Os.sleep(1); // a readability wakeup that decrypts to no application bytes + return 0; + } + }; + rsp.begin(mem, mem, 16); // content length 16, nothing received yet + try { + rsp.recv(50); + Assert.fail("expected recv to time out when no application bytes arrive"); + } catch (HttpClientException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("timed out")); + } + } finally { + Unsafe.free(mem, memSize, MemoryTag.NATIVE_DEFAULT); + } + } + @Test public void testSplit1() { String[] expectedFragments = { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/json/JsonLexerTest.java b/core/src/test/java/io/questdb/client/test/cutlass/json/JsonLexerTest.java index 2e8cd96ee..7ec005dce 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/json/JsonLexerTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/json/JsonLexerTest.java @@ -24,14 +24,17 @@ package io.questdb.client.test.cutlass.json; +import io.questdb.client.Sender; import io.questdb.client.cutlass.json.JsonException; import io.questdb.client.cutlass.json.JsonLexer; import io.questdb.client.cutlass.json.JsonParser; +import io.questdb.client.cutlass.line.http.AbstractLineHttpSender; import io.questdb.client.std.Files; import io.questdb.client.std.IntStack; import io.questdb.client.std.MemoryTag; import io.questdb.client.std.Mutable; import io.questdb.client.std.Unsafe; +import io.questdb.client.std.str.StringSink; import io.questdb.client.test.tools.TestUtils; import org.junit.AfterClass; import org.junit.Assert; @@ -243,7 +246,9 @@ public void testNestedObjects() throws Exception { @Test public void testQuoteEscape() throws Exception { - assertThat("{\"x\":\"a\\\"bc\"}", "{\"x\": \"a\\\"bc\"}"); + // the lexer decodes the escaped quote: the value a\"bc becomes a"bc (the assembling parser does + // not re-escape, so the decoded quote shows bare in the re-serialized form) + assertThat("{\"x\":\"a\"bc\"}", "{\"x\": \"a\\\"bc\"}"); } @Test @@ -663,6 +668,185 @@ public void testWrongQuote() { assertError("Unexpected symbol", 10, "{\"x\": \"a\"bc\",}"); } + @Test + public void testStringEscapesAreDecoded() throws Exception { + assertMemoryLeak(() -> { + // JSON string escapes must be resolved, not handed back to the listener literally + assertDecodedValue("{\"v\":\"https:\\/\\/h\\/p\"}", "https://h/p"); // escaped slash -> slash + assertDecodedValue("{\"v\":\"a\\\"b\"}", "a\"b"); // escaped quote -> quote + assertDecodedValue("{\"v\":\"a\\\\b\"}", "a\\b"); // escaped backslash -> backslash + assertDecodedValue("{\"v\":\"X\\u0041Y\"}", "XAY"); // 4-hex unicode escape decoded + assertDecodedValue("{\"v\":\"X\\u0041\"}", "XA"); // \\uXXXX at end of value (i+6==n boundary) + assertDecodedValue("{\"v\":\"tab\\tend\"}", "tab\tend"); // escaped tab -> tab + assertDecodedValue("{\"v\":\"plain\"}", "plain"); // no escapes (fast path) + }); + } + + @Test + public void testStringEscapesDecodedAcrossSplitParseCalls() throws Exception { + assertMemoryLeak(() -> { + // a value whose backslash escape straddles two parse() calls (a real HTTP-fragment boundary) + // must still be decoded: the "saw a backslash" flag that gates the unescape pass has to persist + // across the calls, not reset to false at the start of the second one + String json = "{\"v\":\"ab\\ncd\"}"; // value ab\ncd -> abcd + int len = json.length(); + long address = TestUtils.toMemory(json); + StringSink captured = new StringSink(); + JsonParser parser = (code, tag, position) -> { + if (code == JsonLexer.EVT_VALUE) { + captured.clear(); + captured.put(tag); + } + }; + try (JsonLexer lexer = new JsonLexer(4, 1024)) { + // split immediately after the backslash, so the escape's '\' is in the first chunk and the + // 'n' it escapes is in the second + int split = json.indexOf('\\') + 1; + lexer.parse(address, address + split, parser); + lexer.parse(address + split, address + len, parser); + lexer.parseLast(); + TestUtils.assertEquals("ab\ncd", captured); + } finally { + Unsafe.free(address, len, MemoryTag.NATIVE_DEFAULT); + } + }); + } + + @Test + public void testUnicodeEscapeDecodedAcrossSplitParseCalls() throws Exception { + assertMemoryLeak(() -> { + // a backslash-u-XXXX escape whose four hex digits straddle two parse() calls (a real HTTP-fragment + // boundary) must still decode to one character: the lexer stashes the partial value and resolves the + // escape only once the whole value is assembled, so parseHex4 never sees a truncated escape + String bs = String.valueOf((char) 92); // a single backslash, built without a literal escape + String json = "{\"v\":\"x" + bs + "u0041y\"}"; // value x then the escape for A then y -> xAy + int len = json.length(); + long address = TestUtils.toMemory(json); + StringSink captured = new StringSink(); + JsonParser parser = (code, tag, position) -> { + if (code == JsonLexer.EVT_VALUE) { + captured.clear(); + captured.put(tag); + } + }; + try (JsonLexer lexer = new JsonLexer(4, 1024)) { + // split inside the four hex digits: backslash-u-0-0 lands in the first chunk, 4-1 in the second + int split = json.indexOf(bs) + 4; + lexer.parse(address, address + split, parser); + lexer.parse(address + split, address + len, parser); + lexer.parseLast(); + TestUtils.assertEquals("xAy", captured); + } finally { + Unsafe.free(address, len, MemoryTag.NATIVE_DEFAULT); + } + }); + } + + @Test + public void testUnicodeEscapeWithNonAsciiCharInWindow() throws Exception { + assertMemoryLeak(() -> { + // a backslash-u escape whose four-hex window's first char is a non-ASCII code point, fed as the + // valid UTF-8 a hostile IdP response would carry (0xC3 0xA9 -> U+00E9, 233). parseHex4 indexes + // Numbers.hexNumbers (int[128]) only behind a c<128 guard, so 233 is a non-hex digit and the escape + // is kept verbatim (lenient), not decoded. WITHOUT the guard, hexNumbers[233] throws an + // ArrayIndexOutOfBoundsException that escapes as an unchecked exception - the OIDC callers catch + // only JsonException - so this pins that the guard is present. + byte[] bytes = {'{', '"', 'v', '"', ':', '"', 'x', '\\', 'u', + (byte) 0xC3, (byte) 0xA9, // valid UTF-8 for U+00E9 (e-acute); lands right after the backslash-u + 'A', 'B', 'C', 'y', '"', '}'}; + int len = bytes.length; + long address = Unsafe.malloc(len, MemoryTag.NATIVE_DEFAULT); + StringSink captured = new StringSink(); + JsonParser parser = (code, tag, position) -> { + if (code == JsonLexer.EVT_VALUE) { + captured.clear(); + captured.put(tag); + } + }; + try { + for (int i = 0; i < len; i++) { + Unsafe.getUnsafe().putByte(address + i, bytes[i]); + } + try (JsonLexer lexer = new JsonLexer(4, 1024)) { + lexer.parse(address, address + len, parser); + lexer.parseLast(); + // the escape stayed literal (lenient): x, then a verbatim backslash-u, then the decoded + // e-acute (U+00E9), then ABCy + TestUtils.assertEquals("x\\u\u00e9ABCy", captured); + } + } finally { + Unsafe.free(address, len, MemoryTag.NATIVE_DEFAULT); + } + }); + } + + @Test + public void testStringEscapesExoticAndLenient() throws Exception { + assertMemoryLeak(() -> { + String bs = String.valueOf((char) 92); // a single backslash, built without a literal escape + // a surrogate pair (two backslash-u escapes) reassembles into the supplementary point U+1F600 + assertDecodedValue("{\"v\":\"x" + bs + "uD83D" + bs + "uDE00y\"}", + "x" + new String(Character.toChars(0x1F600)) + "y"); + // the backspace and form-feed arms + assertDecodedValue("{\"v\":\"a" + bs + "bb" + bs + "fc\"}", + "a" + ((char) 8) + "b" + ((char) 12) + "c"); + // the lexer is deliberately lenient (not RFC 8259-strict) about malformed or unknown escapes: + // it keeps the backslash and the following text verbatim rather than failing the parse, so a + // literal backslash in non-conformant input is not silently lost. These pin that behavior and + // cover the lenient arms that otherwise carry most of the file's coverage: + assertDecodedValue("{\"v\":\"a" + bs + "xb\"}", "a" + bs + "xb"); // unknown escape -> kept verbatim + assertDecodedValue("{\"v\":\"a" + bs + "uZZZZb\"}", "a" + bs + "uZZZZb"); // non-hex unicode escape -> literal + assertDecodedValue("{\"v\":\"ab" + bs + "u12\"}", "ab" + bs + "u12"); // too few hex digits -> literal + // a lone (unpaired) high surrogate is emitted as-is, not dropped or replaced + assertDecodedValue("{\"v\":\"x" + bs + "uD83Dy\"}", "x" + ((char) 0xD83D) + "y"); + }); + } + + @Test + public void testSettingsParserKeysDecodedThroughUnescape() throws Exception { + assertMemoryLeak(() -> { + // the line-protocol version probe parses /settings with JsonSettingsParser, whose keys now flow + // through the lexer's unescape pass. An escaped key (here a JSON unicode escape standing in for + // the letter 'o') must decode to the real key, otherwise the probe would miss the advertised + // versions and silently fall back to V1. The backslash is built from char 92, so this source + // carries no literal backslash-u sequence. + String esc = ((char) 92) + "u006f"; // a JSON unicode escape for 'o' + String json = "{\"line.proto.support.versi" + esc + "ns\":[1,2,3],\"cairo.max.file.name.length\":127}"; + long address = TestUtils.toMemory(json); + int len = json.length(); + try (AbstractLineHttpSender.JsonSettingsParser parser = new AbstractLineHttpSender.JsonSettingsParser(); + JsonLexer lexer = new JsonLexer(1024, 1024)) { + lexer.parse(address, address + len, parser); + lexer.parseLast(); + // the escaped "versions" key decoded and matched, so the highest advertised version was + // picked; a non-decoded key would leave the versions empty and fall back to V1 + Assert.assertEquals(Sender.PROTOCOL_VERSION_V3, parser.getDefaultProtocolVersion()); + Assert.assertEquals(127, parser.getMaxNameLen()); + } finally { + Unsafe.free(address, len, MemoryTag.NATIVE_DEFAULT); + } + }); + } + + private static void assertDecodedValue(String json, String expected) throws JsonException { + int len = json.length(); + long address = TestUtils.toMemory(json); + StringSink captured = new StringSink(); + JsonParser parser = (code, tag, position) -> { + if (code == JsonLexer.EVT_VALUE) { + captured.clear(); + captured.put(tag); + } + }; + try (JsonLexer lexer = new JsonLexer(4, 1024)) { + lexer.parse(address, address + len, parser); + lexer.parseLast(); + TestUtils.assertEquals(expected, captured); + } finally { + Unsafe.free(address, len, MemoryTag.NATIVE_DEFAULT); + } + } + private void assertError(String expected, int expectedPosition, String input) { int len = input.length(); long address = TestUtils.toMemory(input); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderErrorResponseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderErrorResponseTest.java new file mode 100644 index 000000000..04504ec6c --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderErrorResponseTest.java @@ -0,0 +1,723 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.line; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Verifies that the error body a QuestDB HTTP endpoint returns on a failed flush is rendered safely + * into the {@link LineSenderException} message. A JSON error body has its string escapes resolved by the + * lexer, so a {@code message} or {@code errorId} field arrives fully decoded; an auth (401/403) body, a + * non-JSON body, and a body that fails to parse as JSON are echoed verbatim. In every case a hostile or + * proxied endpoint could otherwise smuggle real control characters, ANSI escapes or bidi overrides that + * forge a log line or rewrite a terminal when the exception text is printed. The sender must escape them, + * just as it does for column names in an error message. + *

+ * The dangerous bytes are built at runtime via {@code (char) 0x1b} (ESC) and {@code (char) 0x202e} (a + * right-to-left override), so this source file stays pure ASCII and carries none of the chars it guards. + */ +public class LineHttpSenderErrorResponseTest { + + // ESC: the lead byte of an ANSI escape sequence (terminal hijack) + private static final char ESC = 0x1b; + // U+202E RIGHT-TO-LEFT OVERRIDE: reorders displayed text (visual spoofing) + private static final char RLO = 0x202e; + + @Test(timeout = 30_000) + public void testMalformedResponseHeadOnFlushFailsOnceWithoutResending() throws Exception { + assertMemoryLeak(() -> { + // HttpHeaderParser rejects a response head it cannot parse - here a header block past its fixed + // 4096-byte buffer, the shape an intermediary stacking Set-Cookie/CSP produces - by throwing + // HttpException, a SIBLING of HttpClientException rather than a subclass. Uncaught it escaped + // flush0 entirely, taking with it the client.disconnect() that keeps the next flush off a + // connection holding a half-read response, and left flush() throwing a raw HttpException + // instead of the LineSenderException its contract promises. + // + // Caught, but NOT retried. The parser only ever runs on bytes that arrived, so the server + // answered: the batch is delivered, and the head is chosen by an intermediary, so the next + // attempt parses the same block and fails the same way. Routing it to the transport arm made a + // committed batch re-send until the retry budget ran out. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + StringBuilder padding = new StringBuilder(); + for (int i = 0; i < 5000; i++) { + padding.append('A'); + } + return MockOidcServer.raw("HTTP/1.1 204 No Content\r\n" + + "X-Pad: " + padding + "\r\n\r\n"); + })) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) // only the flush hits the mock + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(5_000) // a budget a re-send would visibly spend + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + long startNanos = System.nanoTime(); + try { + sender.flush(); + Assert.fail("an unparseable response head must fail the flush"); + } catch (LineSenderException e) { + // the documented type, and a message that names what went wrong rather than + // reporting a transport failure that did not happen + Assert.assertTrue(e.getMessage(), + e.getMessage().contains("Malformed HTTP response head")); + Assert.assertFalse("a head an intermediary will re-send identically is not retryable", + e.isRetryable()); + } + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + Assert.assertEquals("a batch the server already answered must be sent exactly once", + 1, requests.get()); + Assert.assertTrue("returned too slowly to have failed without retrying: " + + elapsedMillis + "ms", elapsedMillis < 5_000); + } + } + }); + } + + @Test(timeout = 30_000) + public void testMalformedResponseHeadSuppressesTheReflushOnClose() throws Exception { + assertMemoryLeak(() -> { + // The HttpException arm sets lastFlushFailed = true so close()'s auto-flush cannot re-send the + // batch: the parser only runs on bytes that arrived, so the server already received these rows, + // and a close-time re-send would duplicate every one of them on a table without DEDUP keys. The + // sibling above disables auto-flush and asserts inside the try, so close()'s suppression gate is + // never reached there; this leaves auto-flush ON (the default) so close() calls flush0(true), and + // asserts the batch is not sent a second time. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + StringBuilder padding = new StringBuilder(); + for (int i = 0; i < 5000; i++) { + padding.append('A'); + } + return MockOidcServer.raw("HTTP/1.1 204 No Content\r\n" + + "X-Pad: " + padding + "\r\n\r\n"); + })) { + Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) // only the flush hits the mock + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(5_000) + // auto-flush left ON (the default): close() runs flush0(true), so its gate is exercised + .build(); + try { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("an unparseable response head must fail the flush"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("Malformed HTTP response head")); + } + Assert.assertEquals("the failed flush sent the batch once", 1, requests.get()); + } finally { + // close() auto-flushes; lastFlushFailed must suppress it. Should a regression drop that + // flag, the re-send hits the same malformed head and close() itself throws - either way + // the counter has already reached 2, which the assertion below turns into a clear failure. + try { + sender.close(); + } catch (LineSenderException ignore) { + // a re-send that fails still reached the server and incremented the counter + } + } + Assert.assertEquals("close() must not re-send a batch the server already answered", + 1, requests.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testProtocolDetectionErrorBodyControlAndBidiAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // when the caller does not pin a protocol version, build() probes the server for one; a + // non-success, non-404 probe response body is captured into the "Failed to detect server line + // protocol version" exception. A hostile or proxied endpoint must not splice control, ANSI or + // bidi chars into that message any more than into a flush error + String errorBody = "probe denied " + ESC + "[2J forged\n" + RLO + "moc.live"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(400, errorBody))) { + try { + // no protocolVersion(...) -> build() runs the detection probe; retryTimeoutMillis(0) makes + // it give up after the first failed probe instead of retrying to a deadline + Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .retryTimeoutMillis(0) + .build() + .close(); + Assert.fail("expected protocol detection to fail and surface the server body"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("Failed to detect server line protocol version")); + Assert.assertTrue("visible text must be preserved: " + msg, msg.contains("probe denied")); + Assert.assertTrue("the ESC must be escaped: " + msg, msg.contains("\\u001b")); + Assert.assertTrue("the bidi override must be escaped: " + msg, msg.contains("\\u202e")); + Assert.assertFalse("a raw ESC must not leak: " + msg, msg.indexOf(0x1b) >= 0); + Assert.assertFalse("a raw newline must not leak: " + msg, msg.indexOf('\n') >= 0); + Assert.assertFalse("a raw bidi override must not leak: " + msg, msg.indexOf(0x202e) >= 0); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDribbledBodyUnderA2xxDoesNotResendTheBatch() throws Exception { + assertMemoryLeak(() -> { + // A flush whose response BODY dribbles (chunked headers sent, then the chunk-size line one byte at + // a time, never completing) aborts the read on the configured request timeout: the no-arg recv() + // the flush uses bounds the WHOLE body read, not each socket read. Drives that bound end to end + // over a real socket from a real flush (the Response classes are unit-tested in isolation; the ILP + // flush path - consumeChunkedResponse -> recv() - is covered here). Without the whole-read bound + // the dribble would re-arm the per-read timeout forever and this test would hit its @Test timeout. + // + // The status here is 200, so the server ALREADY COMMITTED these rows. The abort must therefore not + // reach flush0's catch, which treats HttpClientException as a transport error and re-sends the + // whole batch - duplicate rows on data the server accepted, with a retry budget that keeps trying. + // The bound is what made this reachable at all: base re-armed per socket read, so a + // dribbling-but-progressing body never aborted here. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + return MockOidcServer.dribble(); + })) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) // skip the build-time probe: only the flush hits the dribble + .httpTimeoutMillis(1_000) // the whole-body-read bound the no-arg recv() applies + .retryTimeoutMillis(3_000) // a budget a re-send would visibly spend + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + long startNanos = System.nanoTime(); + sender.flush(); + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // aborted on the ~1s whole-read bound. The mock dribbles for ~10s, so a per-read re-arm + // would not abort until ~11s (then the 30s @Test timeout); the < 5s ceiling fails on that + // path while giving the 1s bound generous CI headroom. + Assert.assertTrue("returned too fast to be the 1s read bound: " + elapsedMillis + "ms", elapsedMillis >= 500); + Assert.assertTrue("returned too slowly - re-armed per-read, or retried? " + elapsedMillis + "ms", elapsedMillis < 5_000); + Assert.assertEquals("a committed batch must be sent exactly once; a drain failure after a " + + "2xx must not re-send it", 1, requests.get()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDribbledResponseHeadFailsTheFlushWithinTheRetryBudget() throws Exception { + assertMemoryLeak(() -> { + // The response HEAD read is bounded on elapsed time the same way the body read is, and the two + // sibling tests above cover only the body. The head bound is the one existing non-OIDC senders + // are most exposed to, because it precedes every response, including the 204 QuestDB's own + // /write answers with. + // + // What separates it from the body cases: at the point await() aborts, NO STATUS HAS BEEN READ. + // So unlike the 2xx drain arm - which knows the server committed and reports success - and + // unlike the error arm - which has a verdict to surface - this abort carries no information + // about whether the batch landed. flush0 classifies it as a transport failure and retries, + // which is the only thing it can do, and the pre-existing ILP-over-HTTP at-least-once window + // is what that retry spends. Against a table without DEDUP keys, a peer that dribbles a head + // past the budget can therefore duplicate rows. + // + // This pins the two properties that keep that bounded and diagnosable: the flush TERMINATES on + // the retry budget instead of running on with the dribble, and it reports a transport timeout + // rather than something the operator cannot act on. Base could not reach it at all - await() + // re-armed its timeout on every socket read, so a head making progress never aborted. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + return MockOidcServer.dribbleHead(); + })) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) // skip the build-time probe: only the flush hits the dribble + .httpTimeoutMillis(500) // the whole-head-read bound + .retryTimeoutMillis(2_000) // and the budget the retries spend + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + long startNanos = System.nanoTime(); + try { + sender.flush(); + Assert.fail("a head that never completes must fail the flush"); + } catch (LineSenderException e) { + Assert.assertTrue("the operator must be told this was a transport timeout, not " + + "handed a bare parse error: " + e.getMessage(), + e.getMessage().contains("timed out")); + Assert.assertTrue("a head-read abort carries no status, so it stays retryable - " + + "unlike a malformed head, which is a verdict", + e.isRetryable()); + } + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // The mock dribbles for ~10s. A per-read re-arm would not abort until then, and the + // 30s @Test timeout would fire instead of this ceiling. + Assert.assertTrue("the head bound must fire, not run on with the dribble: " + + elapsedMillis + "ms", elapsedMillis < 15_000); + Assert.assertTrue("the flush must retry within its budget rather than give up on the " + + "first abort, and must stop when the budget is spent: " + + requests.get() + " sends", + requests.get() >= 1 && requests.get() < 20); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDrainAbortAfterA2xxDropsTheConnection() throws Exception { + assertMemoryLeak(() -> { + // The other half of the drain-abort contract, and the half nothing pinned. Its sibling above + // covers ONE flush and proves the abort does not re-send it. This covers the flush AFTER it, + // which is the only place the `!drained` term of the disconnect can be observed at all. + // + // The abort leaves the response body unconsumed, so the socket still carries the first + // response's chunk-size digits. Without the disconnect the next flush writes onto it and its + // await() reads those digits instead of a status line: the header block never completes, so + // await() spends its whole bound and throws HttpClientException, which flush0 classifies as a + // transport error and RETRIES. Measured against this fixture: three sends for two flushes - + // one extra copy of a batch the server had already committed, which is the duplicate-row harm + // the surrounding arm exists to prevent, arriving by the one route it does not guard against. + // + // So the request count is what goes red, and it is asserted first. The connection count is + // NOT a discriminator here and is not claimed as one - it is 2 either way, because the failed + // second flush disconnects and reconnects on its own before retrying. It is asserted anyway, + // as the direct statement of the guard: one flush, one connection. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + return MockOidcServer.dribble(); + })) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) // skip the build-time probe: only the flush hits the dribble + .httpTimeoutMillis(1_000) // the whole-body-read bound each drain aborts on + .retryTimeoutMillis(3_000) // a budget a re-send would visibly spend + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + + // The second flush is what needs a clean socket. Without the disconnect it lands on the + // poisoned one and throws instead of succeeding. + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); + + Assert.assertEquals("two flushes, two sends: a drain abort leaves the body unconsumed, so " + + "a reused socket fails the next flush's header read and it retries - " + + "re-sending a batch the server had already committed", + 2, requests.get()); + Assert.assertEquals("and each flush went out on its own connection", 2, + server.connectionsAccepted()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testDrainFailureAfterASuccessfulFlushReportsItsReason() throws Exception { + assertMemoryLeak(() -> { + // A 2xx IS the commit, so a body-drain abort after it changes no outcome - but it does drop the + // connection, because unconsumed bytes would mis-frame the next response. Against a server that + // dribbles every response that is one reconnect per flush, and the catch used to bind the + // exception and discard it, leaving the churn with nothing to explain it. + ch.qos.logback.classic.Logger senderLog = + (ch.qos.logback.classic.Logger) LoggerFactory.getLogger( + "io.questdb.client.cutlass.line.http.AbstractLineHttpSender"); + ListAppender appender = new ListAppender<>(); + appender.start(); + Level saved = senderLog.getLevel(); + senderLog.setLevel(Level.ALL); + senderLog.addAppender(appender); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.dribble(200))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(3_000) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); // the 2xx committed; only the drain aborts + } + String drainLine = null; + for (ILoggingEvent event : appender.list) { + if (event.getFormattedMessage().contains("could not drain the response body")) { + drainLine = event.getFormattedMessage(); + break; + } + } + Assert.assertNotNull("a drain abort that drops the connection must say so; logged: " + + appender.list, drainLine); + Assert.assertTrue("and it must carry WHY, not just that it happened: " + drainLine, + drainLine.contains("reason=") && !drainLine.contains("reason=null")); + } finally { + senderLog.detachAppender(appender); + senderLog.setLevel(saved); + } + }); + } + + @Test(timeout = 30_000) + public void testDribbledBodyUnderAnErrorStatusStillSurfacesTheStatus() throws Exception { + assertMemoryLeak(() -> { + // The mirror of the 2xx case on the error path. The STATUS is the verdict; the body is only detail + // for the message. Reading that body can now abort on the whole-read bound, and if the abort + // escapes it reaches flush0's catch, which reclassifies a definitive 401 as a transport failure: + // the sender then burns the whole retry budget re-sending against an endpoint that will keep + // refusing, and finally reports "Connection Failed", with the real status nowhere in the message. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> { + requests.incrementAndGet(); + return MockOidcServer.dribble(401); + })) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(3_000) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + long startNanos = System.nanoTime(); + try { + sender.flush(); + Assert.fail("expected the 401 to surface"); + } catch (LineSenderException e) { + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + String msg = e.getMessage(); + Assert.assertTrue("the real status must reach the caller: " + msg, + msg.contains("http-status=401")); + Assert.assertFalse("a definitive 401 must not be reported as a transport failure: " + msg, + msg.contains("Connection Failed")); + // and WHY the body could not be read, which is the half the status cannot supply: + // a read that timed out, a peer that vanished and a mangled chunk all arrive here + // as the same status, and only this tells an operator which one to act on + Assert.assertTrue("the reason the body read failed must reach the caller: " + msg, + msg.contains("reason=") && !msg.contains("reason=")); + Assert.assertTrue("and it must be the read abort, not something invented: " + msg, + msg.contains("timed out")); + Assert.assertTrue("a definitive status must not spend the retry budget: " + + elapsedMillis + "ms", elapsedMillis < 3_000); + Assert.assertEquals("a definitive status must not be retried", 1, requests.get()); + // The classification has to survive the wrapper too, not just the message. This is + // the one throw site that builds its exception from the status alone, having failed + // to read the body, so it re-passes `retryable` by hand - and a caller acting on + // isRetryable() would re-flush forever into a 401 that keeps refusing. The `true` + // direction is pinned by LineSenderExceptionRetryableTest, which is what makes this + // assertFalse mean "classified permanent" rather than "not classified at all". + Assert.assertFalse("a 401 must stay non-retryable through the unreadable-body wrapper", + e.isRetryable()); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerAuthErrorBodyControlAndBidiAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // a 401/403 body is echoed into the exception verbatim (read as raw bytes, not through the JSON + // parser), so a hostile or proxied endpoint could splice raw control, ANSI or bidi chars straight + // into the LineSenderException; the sender must escape them just like the JSON-field path + String errorBody = "denied " + ESC + "[2J forged\n" + RLO + "moc.live"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(401, errorBody))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's auth error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("authentication error")); + Assert.assertTrue("visible text must be preserved: " + msg, msg.contains("denied")); + Assert.assertTrue("the ESC must be escaped: " + msg, msg.contains("\\u001b")); + Assert.assertTrue("the bidi override must be escaped: " + msg, msg.contains("\\u202e")); + Assert.assertFalse("a raw ESC must not leak: " + msg, msg.indexOf(0x1b) >= 0); + Assert.assertFalse("a raw newline must not leak: " + msg, msg.indexOf('\n') >= 0); + Assert.assertFalse("a raw bidi override must not leak: " + msg, msg.indexOf(0x202e) >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerErrorStatusLineControlCharsAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // the HTTP status-line token is echoed into the exception as "[http-status=...]". The header parser + // copies it verbatim between the two spaces, so a hostile or proxied endpoint can smuggle control or + // ANSI bytes there; a non-3-char token bypasses the numeric status checks and reaches the generic + // error path, so the status render must escape them too, not just the body. A bidi override is a + // multi-byte char the raw-response writer's US-ASCII encoding would drop, so this case uses an ESC; + // the bidi cases above cover the body + String body = "upstream error"; + // a malformed status code "400[m" (6 chars, not 3) carries an ESC between the two spaces; + // text/plain keeps it off the JSON parser, so it reaches the generic path that renders the status + String rawResponse = "HTTP/1.1 400" + ESC + "[m FORGED\r\n" + + "Content-Type: text/plain\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(body.length()) + "\r\n" + body + "\r\n" + + "0\r\n\r\n"; + try (MockOidcServer server = new MockOidcServer((method, path, b) -> MockOidcServer.raw(rawResponse))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("Could not flush buffer")); + // the ESC smuggled into the status token arrives escaped, never as a raw byte that + // could drive an ANSI terminal sequence + Assert.assertTrue("the status-line ESC must be escaped: " + msg, msg.contains("\\u001b")); + Assert.assertFalse("a raw ESC must not leak from the status line: " + msg, msg.indexOf(0x1b) >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerJsonErrorBidiAndZeroWidthAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // beyond C0 controls, a hostile or proxied endpoint can smuggle bidi overrides and zero-width + // characters (as JSON \\uXXXX escapes the lexer decodes) that reorder or hide text in a terminal. + // The sender must escape these too, matching the OIDC display sanitizer, so the rendered message + // cannot be visually spoofed + String errorBody = "{" + + "\"code\":\"invalid\"," + + "\"message\":\"safe\\u202ehidden\\u200bend\"," + + "\"line\":1," + + "\"errorId\":\"E1\"" + + "}"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(400, errorBody))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's JSON error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + // the visible text survives, but the bidi override (U+202E) and the zero-width space + // (U+200B) arrive escaped, never as raw code points that could reorder or hide text + Assert.assertTrue("visible text must be preserved: " + msg, msg.contains("safe")); + Assert.assertTrue("visible text must be preserved: " + msg, msg.contains("hidden")); + Assert.assertTrue("the bidi override must be escaped: " + msg, msg.contains("\\u202e")); + Assert.assertTrue("the zero-width space must be escaped: " + msg, msg.contains("\\u200b")); + Assert.assertFalse("a raw bidi override must not leak: " + msg, msg.indexOf(0x202e) >= 0); + Assert.assertFalse("a raw zero-width space must not leak: " + msg, msg.indexOf(0x200b) >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testQuestDbRowErrorRendersTheDecodedNewlineAsAnEscape() throws Exception { + assertMemoryLeak(() -> { + // Pins the rendering of the single most common ILP failure. QuestDB's own + // LineHttpProcessorState builds its error as error.put("\nerror in line ")... - a REAL newline - + // and escapeJsonStr sends it as the JSON escape \n. Before the lexer decoded escapes the client + // copied those two characters through verbatim; now it decodes them to a newline and + // putAsPrintable re-escapes it, so the text a user (and their log scraper) sees changed from + // a two-character JSON escape to a six-character unicode escape. Neither form leaks a raw + // newline, which is the point of putAsPrintable, but the + // rendering is user-visible and nothing pinned it: the sibling tests here assert only that + // fragments either side of the newline survive, which holds under both. + String errorBody = "{" + + "\"code\":\"invalid\"," + + "\"message\":\"invalid field format\\nerror in line 1: table: t, column: v\"," + + "\"line\":1," + + "\"errorId\":\"ABC-1\"" + + "}"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(400, errorBody))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's row error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue("the newline must render as its unicode escape: " + msg, + msg.contains("invalid field format\\u000aerror in line 1: table: t, column: v")); + Assert.assertFalse("the raw JSON escape must not survive undecoded: " + msg, + msg.contains("format\\nerror")); + Assert.assertFalse("and no raw newline may reach the message: " + msg, + msg.indexOf('\n') >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerJsonErrorControlCharsAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // Feed REAL control bytes into the structured-error fields. JsonLexer currently also decodes + // valid JSON escapes, but spelling these as \\u001b/\\n on the wire would let the pre-decoding + // implementation pass with drainAndReset's old plain put(): the six printable escape characters + // were already safe. Raw bytes make this test discriminate on putAsPrintable itself. The parser + // deliberately accepts this malformed-JSON input so the sender can still render a hostile server + // response safely. + String esc = String.valueOf((char) 0x1b); + String errorBody = "{" + + "\"code\":\"invalid\"," + + "\"message\":\"bad" + esc + "[m\nthing\"," + + "\"line\":42," + + "\"errorId\":\"E" + esc + "ID\"" + + "}"; + // a chunked 400 with Content-Type application/json drives the flush failure through the sender's + // JSON error parser (a 4xx response is asserted to be chunked before parsing) + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(400, errorBody))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + // an explicit protocol version keeps build() from probing the server, so the only + // request is the flush below + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's JSON error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("Could not flush buffer")); + // the decoded message text survives... + Assert.assertTrue("decoded message text must be preserved: " + msg, msg.contains("bad")); + Assert.assertTrue("decoded message text must be preserved: " + msg, msg.contains("thing")); + Assert.assertTrue("errorId must be present with its ESC escaped: " + msg, msg.contains("id: E\\u001bID")); + // ...but no raw control byte reaches the message: no ESC (ANSI injection) and no + // newline (log-line forging); both arrive escaped instead + Assert.assertTrue("the decoded ESC must be escaped, not raw: " + msg, msg.contains("\\u001b")); + Assert.assertTrue("the decoded newline must be escaped, not raw: " + msg, + msg.contains("\\u000a")); + Assert.assertFalse("a raw ESC must not leak into the message: " + msg, msg.indexOf(0x1b) >= 0); + Assert.assertFalse("a raw newline must not leak into the message: " + msg, msg.indexOf('\n') >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerMalformedJsonErrorBodyControlAndBidiAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // a body sent as application/json but not parseable as a QuestDB error object (a proxy/WAF page, + // or an unexpected first key) makes the JSON parser throw; the fallback renders the raw body, which + // must still be escaped. The unexpected first key "forged" forces the parse failure; the ESC and + // bidi override ride in the value and must surface escaped, not raw + String errorBody = "{\"forged\":\"x " + ESC + "[2J y " + RLO + " z\"}"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.chunkedJson(400, errorBody))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the malformed server response to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("Could not flush buffer")); + // the raw body is shown (so the user can diagnose the unexpected response)... + Assert.assertTrue("the raw body must be preserved: " + msg, msg.contains("forged")); + // ...but the smuggled control and bidi chars arrive escaped, never raw + Assert.assertTrue("the ESC must be escaped: " + msg, msg.contains("\\u001b")); + Assert.assertTrue("the bidi override must be escaped: " + msg, msg.contains("\\u202e")); + Assert.assertFalse("a raw ESC must not leak: " + msg, msg.indexOf(0x1b) >= 0); + Assert.assertFalse("a raw bidi override must not leak: " + msg, msg.indexOf(0x202e) >= 0); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testServerNonJsonErrorBodyControlCharsAreEscaped() throws Exception { + assertMemoryLeak(() -> { + // a proxy or WAF can return a non-JSON error body (here text/plain) with raw ANSI/control bytes; + // it reaches the generic error path, which must escape them before they hit a log or terminal. + // The body is all ASCII (a real ESC and a newline) so it survives the raw response writer's + // US-ASCII encoding; bidi is covered by the auth/malformed cases above + String body = "upstream down " + ESC + "[31m forged\nsecond line"; + // hand-craft a chunked text/plain response: the generic path only reads the body when chunked, and + // a non-application/json content type keeps it off the JSON parser + String rawResponse = "HTTP/1.1 400 Bad Request\r\n" + + "Content-Type: text/plain\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(body.length()) + "\r\n" + body + "\r\n" + + "0\r\n\r\n"; + try (MockOidcServer server = new MockOidcServer((method, path, b) -> MockOidcServer.raw(rawResponse))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the server's non-JSON error to surface as a LineSenderException"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + Assert.assertTrue(msg, msg.contains("Could not flush buffer")); + Assert.assertTrue("visible text must be preserved: " + msg, msg.contains("upstream down")); + Assert.assertTrue("the ESC must be escaped: " + msg, msg.contains("\\u001b")); + Assert.assertFalse("a raw ESC must not leak: " + msg, msg.indexOf(0x1b) >= 0); + Assert.assertFalse("a raw newline must not leak: " + msg, msg.indexOf('\n') >= 0); + } + } + } + }); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderInterfaceTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderInterfaceTest.java index 0d8ae3d58..4ccf74e99 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderInterfaceTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderInterfaceTest.java @@ -25,10 +25,14 @@ package io.questdb.client.test.cutlass.line; import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.std.bytes.DirectByteSlice; import org.junit.Assert; import org.junit.Test; +import java.time.Instant; +import java.time.temporal.ChronoUnit; + /** * Exercises {@link Sender#reset()} and {@link Sender#bufferView()} on the * HTTP transport. The HTTP sender connects lazily on the first @@ -51,6 +55,97 @@ public void testBufferViewReflectsAccumulatedRows() { } } + @Test + public void testRejectedAtNowWritesNothing() { + // atNow() validates the row state before writing its terminator, the same guard at() carries in a + // separately deletable method. All four at() overloads are covered below; the one bare atNow() call + // in this suite runs in ADDING_COLUMNS, so the rejected states reach it only here. Both are exercised + // - no table name, and a table with no symbols or columns - over V1 and V2 (V3 inherits V2's). + for (int version = 1; version <= 2; version++) { + String config = "http::addr=127.0.0.1:1;auto_flush=off;protocol_version=" + version + ';'; + String where = "[version=" + version + ']'; + + try (Sender sender = Sender.fromConfig(config)) { + try { + sender.atNow(); + Assert.fail("atNow() with no table name must be rejected " + where); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no table name was provided")); + } + Assert.assertEquals("a rejected atNow() must not write a terminator " + where, + 0, sender.bufferView().size()); + } + + try (Sender sender = Sender.fromConfig(config)) { + sender.table("t"); + int afterTableName = sender.bufferView().size(); + Assert.assertTrue("preconditions: table() writes " + where, afterTableName > 0); + try { + sender.atNow(); + Assert.fail("atNow() with no symbols or columns must be rejected " + where); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), + e.getMessage().contains("no symbols or columns were provided")); + } + Assert.assertEquals("a rejected atNow() must not write a terminator " + where, + afterTableName, sender.bufferView().size()); + + // and the half-built row is still intact: finishing it properly must work + sender.longColumn("v", 1L); + sender.atNow(); + Assert.assertTrue("the row must still complete after the rejection " + where, + sender.bufferView().size() > afterTableName); + } + } + } + + @Test + public void testRejectedExplicitTimestampWritesNothing() { + // at(timestamp) validates BEFORE it writes, and that ordering is the whole reason it does not simply + // delegate to atNow(): a row rejected after the timestamp went into the buffer would leave those bytes + // for the NEXT row to inherit, splicing a stray " 1700000000000000000" into an otherwise valid line. + // Both rejected states are covered - no table name, and a table with no symbols or columns - over V1 + // and V2 (V3 inherits V2's at()), for both overloads. + for (int version = 1; version <= 2; version++) { + for (int overload = 0; overload < 2; overload++) { + String config = "http::addr=127.0.0.1:1;auto_flush=off;protocol_version=" + version + ';'; + String where = "[version=" + version + " overload=" + overload + ']'; + + try (Sender sender = Sender.fromConfig(config)) { + try { + at(sender, overload); + Assert.fail("at() with no table name must be rejected " + where); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no table name was provided")); + } + Assert.assertEquals("a rejected at() must not write a timestamp " + where, + 0, sender.bufferView().size()); + } + + try (Sender sender = Sender.fromConfig(config)) { + sender.table("t"); + int afterTableName = sender.bufferView().size(); + Assert.assertTrue("preconditions: table() writes " + where, afterTableName > 0); + try { + at(sender, overload); + Assert.fail("at() with no symbols or columns must be rejected " + where); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), + e.getMessage().contains("no symbols or columns were provided")); + } + Assert.assertEquals("a rejected at() must not write a timestamp " + where, + afterTableName, sender.bufferView().size()); + + // and the half-built row is still intact: finishing it properly must work + sender.longColumn("v", 1L); + sender.atNow(); + Assert.assertTrue("the row must still complete after the rejection " + where, + sender.bufferView().size() > afterTableName); + } + } + } + } + @Test public void testResetClearsBufferAndAllowsNewRows() { try (Sender sender = Sender.fromConfig("http::addr=127.0.0.1:1;auto_flush=off;protocol_version=1;")) { @@ -72,4 +167,12 @@ public void testResetClearsBufferAndAllowsNewRows() { sender.bufferView().size() > 0); } } + + private static void at(Sender sender, int overload) { + if (overload == 0) { + sender.at(1_700_000_000_000_000_000L, ChronoUnit.NANOS); + } else { + sender.at(Instant.ofEpochMilli(1_700_000_000_000L)); + } + } } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderTokenProviderTest.java new file mode 100644 index 000000000..fb94e08e0 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/LineHttpSenderTokenProviderTest.java @@ -0,0 +1,514 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.line; + +import io.questdb.client.HttpTokenProvider; +import io.questdb.client.Sender; +import io.questdb.client.std.bytes.DirectByteSlice; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.line.http.AbstractLineHttpSender; +import io.questdb.client.std.str.Utf8String; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import io.questdb.client.test.tools.HandOffCharSequence; +import org.junit.Assert; +import org.junit.Test; + +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Verifies that a {@link Sender} built with {@link Sender.LineSenderBuilder#httpTokenProvider} + * does not query the provider on the build path: the first token pull is deferred to the first + * row. That lets a provider which signs in lazily - the documented + * {@code .httpTokenProvider(auth::getToken)} - be wired before the interactive sign-in + * has completed. + *

+ * The deferral tests pin an explicit {@code protocol_version} to keep {@link Sender.LineSenderBuilder#build()} + * from probing the server, and disable auto-flush, so rows buffer against a port nobody listens on without + * opening a connection. The end-to-end tests instead flush against a {@link MockOidcServer} and assert the + * pulled token actually reaches the {@code Authorization: Bearer} header on the wire, is re-queried per + * request as a rotating provider refreshes, and is re-sent verbatim (not re-pulled) on a retry. Each test + * runs under {@code assertMemoryLeak} so the sender's native buffers are proven freed on close. + */ +public class LineHttpSenderTokenProviderTest { + + @Test + public void testBufferViewIsEmptyNotSentinelWhileTheTokenIsPending() { + // With a provider configured, newRequest() leaves the request at the header stage - withContent() is + // deferred until the first row stamps the Authorization header - so contentStart holds its -1 + // sentinel between every flush and the next row. getContentLength() already reported 0 for that + // state, so bufferView() handed out a view that is empty by length but whose base address is a + // non-zero, unusable pointer: a ptr() != 0 test reads as true, and arithmetic on it is nonsense. + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:1") + .httpTokenProvider(() -> "TOKEN") + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .build()) { + DirectByteSlice pending = sender.bufferView(); + Assert.assertEquals("an empty buffer must report a zero base address, not the -1 sentinel", + 0L, pending.ptr()); + Assert.assertEquals(0, pending.size()); + + // and once a row stamps the token and opens the content section, the view is real again + sender.table("t").longColumn("v", 1L).atNow(); + DirectByteSlice afterRow = sender.bufferView(); + Assert.assertTrue("a stamped request must expose a usable base address", afterRow.ptr() > 0); + Assert.assertTrue("and a non-empty buffer", afterRow.size() > 0); + } + } + + @Test(timeout = 30_000) + public void testAtNowWithoutTableDoesNotCorruptTheAuthorizationHeader() throws Exception { + assertMemoryLeak(() -> { + // The sibling below covers at(); atNow() shares the same guard (validateRowStarted) but its own, + // separately deletable, call site. With a provider, newRequest() leaves the request at the header + // stage (withContent() deferred until the first row stamps the Authorization header). If atNow() + // skipped the guard, its terminator '\n' would land in the HTTP HEADER block on a line of its own, + // folding the following "Authorization: Bearer ..." into it (RFC 7230 obs-fold) so the flush ships + // with NO credential - strictly worse than at()'s stray body byte. Covered over V1 and V2 (V3 + // inherits V2's). + int[] versions = {Sender.PROTOCOL_VERSION_V1, Sender.PROTOCOL_VERSION_V2}; + for (int i = 0; i < versions.length; i++) { + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(versions[i]) + .disableAutoFlush() + .httpTokenProvider(() -> "TOKEN") + .build()) { + try { + sender.atNow(); + Assert.fail("expected atNow() with no table name to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no table name was provided")); + } + // the documented recovery, and the sender must still be usable afterwards + sender.cancelRow(); + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals("exactly one flush must reach the server", 1, auth.size()); + // null here means the rejected atNow() spliced a terminator ahead of the header, so the + // mock's parser never saw a line whose field name is "Authorization" + Assert.assertEquals("the token must reach the wire as its own header", + "Bearer TOKEN", auth.get(0)); + } + } + }); + } + + @Test(timeout = 30_000) + public void testAtWithoutTableDoesNotCorruptTheAuthorizationHeader() throws Exception { + assertMemoryLeak(() -> { + // Regression: at() used to write the leading space and the timestamp BEFORE atNow() validated the + // row state. With a provider, newRequest() leaves the request at the header stage (withContent() + // deferred until the first row stamps the Authorization header), so those bytes landed in the HTTP + // HEADER block, on a line of their own. The next row's "Authorization: Bearer ..." was then appended + // to that line, making it an obs-fold continuation of User-Agent (RFC 7230) instead of a header of + // its own - so the flush went out with NO credential and the server answered 401, after which + // close() dropped the buffered rows. cancelRow() could not undo it: trimContentToLen only rewinds + // within the content section, and it early-returns while the token is pending anyway. + // Both at() overloads are covered, over V1 and V2 (V3 inherits V2's). + int[] versions = {Sender.PROTOCOL_VERSION_V1, Sender.PROTOCOL_VERSION_V2}; + for (int i = 0; i < versions.length; i++) { + for (int overload = 0; overload < 2; overload++) { + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(versions[i]) + .disableAutoFlush() + .httpTokenProvider(() -> "TOKEN") + .build()) { + try { + if (overload == 0) { + sender.at(1_700_000_000_000_000_000L, ChronoUnit.NANOS); + } else { + sender.at(Instant.ofEpochMilli(1_700_000_000_000L)); + } + Assert.fail("expected at() with no table name to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no table name was provided")); + } + // the documented recovery, and the sender must still be usable afterwards + sender.cancelRow(); + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals("exactly one flush must reach the server", 1, auth.size()); + // null here means the rejected at() spliced bytes ahead of the header, so the mock's + // parser never saw a line whose field name is "Authorization" + Assert.assertEquals("the token must reach the wire as its own header", + "Bearer TOKEN", auth.get(0)); + } + } + } + }); + } + + @Test + public void testBuildSucceedsWhenProviderHasNotSignedInYet() throws Exception { + assertMemoryLeak(() -> { + // a provider that throws until the caller has signed in, mirroring OidcDeviceAuth::getToken + AtomicBoolean signedIn = new AtomicBoolean(false); + HttpTokenProvider provider = () -> { + if (!signedIn.get()) { + throw new LineSenderException("no token has been obtained yet"); + } + return "TOKEN"; + }; + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:1") + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + // build() must succeed even though the provider cannot supply a token yet, so the natural + // "construct the sender, sign in, then send" ordering is possible + try { + sender.table("t").longColumn("v", 1L).atNow(); + Assert.fail("expected the not-yet-signed-in provider to fail the first row"); + } catch (LineSenderException e) { + // the deferred pull surfaces the provider's error at first use, not at build time + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no token has been obtained yet")); + } + // after signing in, the still-pending stamp is retried and the row is accepted + signedIn.set(true); + sender.table("t").longColumn("v", 1L).atNow(); + Assert.assertTrue("row must be buffered after signing in", sender.bufferView().size() > 0); + } + }); + } + + @Test(timeout = 30_000) + public void testCancelRowWithPendingTokenDoesNotCorruptRequest() throws Exception { + assertMemoryLeak(() -> { + // Regression: with an httpTokenProvider, newRequest() defers the token and leaves the request at the + // header stage (withContent() not yet run), so the native contentStart is still the -1 sentinel and + // no row bytes are buffered. cancelRow() must be a safe no-op in that window: trimContentToLen(0) + // would otherwise set the write pointer to contentStart + 0 == -1, and the next buffer write (the + // deferred Authorization header on the following row) would segfault the JVM. The window is entered + // after build() and again after every flush (reset() re-arms the pending token); a rejected table + // name - validateTableName() runs BEFORE the token is stamped - is a mainstream way to reach a + // cancelRow() with the token still pending. + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + AtomicInteger calls = new AtomicInteger(); + HttpTokenProvider provider = () -> "TOKEN-" + calls.incrementAndGet(); + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + // (1) cancelRow immediately after build(), token pending, nothing buffered: before the fix the + // write pointer went to -1 and the following row's write segfaulted the JVM + sender.cancelRow(); + Assert.assertEquals("cancelRow must not pull the deferred token", 0, calls.get()); + + // the sender is still usable: a real row buffers, flushes, and carries the token to the wire + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + + // (2) after a flush the token is pending again; cancelRow in that window must also be a safe + // no-op, and the next row must still send its (rotated) token + sender.cancelRow(); + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals("both flushes must reach the server", 2, auth.size()); + Assert.assertEquals("Bearer TOKEN-1", auth.get(0)); + Assert.assertEquals("Bearer TOKEN-2", auth.get(1)); + } + }); + } + + @Test(timeout = 30_000) + public void testChangedProviderTokenIsRevalidated() throws Exception { + assertMemoryLeak(() -> { + // every pulled token is validated per flush, so a token that CHANGES to a bad one must be rejected. + // First flush a valid token, then return a distinct CR/LF token and require the next flush to reject + // it rather than splice it onto the wire. (The same-instance-mutated case - a reused buffer whose + // content changes - is covered by testMutatedSameInstanceProviderTokenIsRevalidated.) + AtomicInteger calls = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + HttpTokenProvider provider = () -> calls.incrementAndGet() == 1 + ? "GOODTOKEN" + : "abc" + (char) 0x0d + (char) 0x0a + "def"; // second pull: CR/LF injected + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); // first flush: GOODTOKEN validated and sent + try { + // the second flush's first row re-pulls the provider -> the changed, bad token + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); + Assert.fail("a changed, bad token must be re-validated and rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("control or non-ASCII character")); + } + } + Assert.assertEquals("the provider is re-pulled per flush", 2, calls.get()); + } + }); + } + + @Test + public void testControlOrNonAsciiProviderTokenIsRejected() throws Exception { + assertMemoryLeak(() -> { + // a token carrying a control or non-ASCII char is forbidden by the HttpTokenProvider contract: a + // CR/LF would inject into the request line and a non-ASCII byte is silently truncated by the ASCII + // header writer, so the sender must reject it at first use rather than splice a corrupt or injected + // "Authorization: Bearer " header onto the wire. Strings are built with explicit char values to keep + // this source pure ASCII. + assertProviderTokenRejected(() -> "abc" + (char) 0x0d + (char) 0x0a + "def", "control or non-ASCII character"); // CR/LF + assertProviderTokenRejected(() -> "tok" + (char) 0x00 + "en", "control or non-ASCII character"); // NUL + assertProviderTokenRejected(() -> (char) 0x1b + "[31mred", "control or non-ASCII character"); // ANSI escape + assertProviderTokenRejected(() -> "tok" + (char) 0xe9 + "n", "control or non-ASCII character"); // non-ASCII + }); + } + + @Test(timeout = 30_000) + public void testFailedFlushReSendsSameTokenWithoutRePull() throws Exception { + assertMemoryLeak(() -> { + // a failed flush preserves the buffered request - token included - and re-sends it verbatim on retry + // rather than re-pulling the provider (the documented contract on httpTokenProvider()). Here the first + // send gets a retryable 500 and the retry must carry the SAME baked token, with the provider queried + // only once - so a rotating provider does not change the credential mid-retry of one buffered batch. + AtomicInteger requests = new AtomicInteger(); + try (MockOidcServer server = new MockOidcServer((method, path, body) -> + requests.incrementAndGet() == 1 + ? MockOidcServer.chunkedJson(500, "boom") // first send: retryable server error + : MockOidcServer.json(204, ""))) { // retry: success + AtomicInteger calls = new AtomicInteger(); + HttpTokenProvider provider = () -> "TOKEN-" + calls.incrementAndGet(); + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); // first send 500 -> retry -> 204 + } + Assert.assertEquals("the provider must be pulled once, not re-pulled on retry", 1, calls.get()); + List auth = server.requestAuthHeaders(); + Assert.assertEquals("the failed send plus its retry must be two requests", 2, auth.size()); + Assert.assertEquals("the first send carries the pulled token", "Bearer TOKEN-1", auth.get(0)); + Assert.assertEquals("the retry must re-send the same baked token", "Bearer TOKEN-1", auth.get(1)); + } + }); + } + + @Test(timeout = 30_000) + public void testMutatedSameInstanceProviderTokenIsRevalidated() throws Exception { + assertMemoryLeak(() -> { + // A provider may reuse one CharSequence buffer (the idiomatic zero-alloc style) and return the SAME + // instance every call. HttpTokenProvider.getToken() makes no immutability promise, so the sender + // must re-validate EVERY pulled token, not trust instance identity: a token mutated in place to + // carry a CR/LF between flushes must be rejected, not spliced verbatim into the "Authorization: + // Bearer" header (authToken writes it with no CR/LF filtering). This pins the fix that dropped the + // identity-cache skip; before it, the second flush injected a header past the auth line. + StringBuilder token = new StringBuilder("GOODTOKEN"); // one instance, mutated in place below + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(() -> token) // always the SAME instance + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); // first flush: GOODTOKEN validated and sent + // mutate the SAME instance to inject a CR/LF header break + token.setLength(0); + token.append("abc").append((char) 0x0d).append((char) 0x0a).append("X-Injected: pwned"); + try { + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); + Assert.fail("a mutated same-instance token carrying CR/LF must be re-validated and rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("control or non-ASCII character")); + } + } + // only the first (valid) flush reached the wire; the injected token was rejected before any send + List auth = server.requestAuthHeaders(); + Assert.assertEquals("only the valid first flush must reach the server", 1, auth.size()); + Assert.assertEquals("Bearer GOODTOKEN", auth.get(0)); + } + }); + } + + @Test + public void testTokenMutatedBetweenValidationAndTheHeaderCannotSplice() throws Exception { + assertMemoryLeak(() -> { + // The sibling test above covers a buffer mutated BETWEEN flushes, which re-validation catches. + // This is the window inside ONE flush: validateToken scanned the provider's sequence and + // authToken then re-read it, so a mutation landing between those two reads passed the check and + // was spliced verbatim into the Authorization header. HttpTokenProvider.getToken() explicitly + // invites a reused mutable buffer, and the SPI is exported, so the reader has to be the one that + // makes this safe: the pulled value is snapshotted before it is validated, and the bytes checked + // are the bytes sent. + // + // HandOffToken swaps its content the instant a full scan completes - i.e. exactly when + // validateToken finishes - so every later read sees the CR/LF splice. + final String clean = "GOODTOKEN"; + final String spliced = "abc" + (char) 0x0d + (char) 0x0a + "X-Injected: pwned"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(() -> new HandOffCharSequence(clean, spliced)) + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals(1, auth.size()); + Assert.assertEquals("the header must carry the bytes that were validated, not a value " + + "swapped in after the scan", "Bearer " + clean, auth.get(0)); + } + }); + } + + @Test + public void testNullOrEmptyProviderTokenIsRejected() throws Exception { + assertMemoryLeak(() -> { + // the HttpTokenProvider contract forbids a null or empty token; the sender must reject it with a + // clear LineSenderException at first use, rather than silently send a malformed "Authorization: + // Bearer " header that the server only answers with a 401 far from the cause + assertProviderTokenRejected(() -> null, "null or empty token"); + assertProviderTokenRejected(() -> "", "null or empty token"); + assertProviderTokenRejected(() -> " ", "null or empty token"); + }); + } + + @Test + public void testProviderTokenNotPulledAtBuildAndPulledOnFirstRow() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger calls = new AtomicInteger(); + HttpTokenProvider provider = () -> { + calls.incrementAndGet(); + return "TOKEN"; + }; + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:1") + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + // build() must not query the provider: a lazily-signing-in provider would not have a token yet + Assert.assertEquals("provider must not be queried at build time", 0, calls.get()); + // the first row pulls the deferred token so the first send will carry it + sender.table("t").longColumn("v", 1L).atNow(); + Assert.assertEquals("provider must be queried when the first row starts", 1, calls.get()); + // a second row in the same un-flushed batch reuses the same request, so it does not re-pull + sender.table("t").longColumn("v", 2L).atNow(); + Assert.assertEquals("provider must not be re-queried within the same batch", 1, calls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testPutRawMessageStampsPendingToken() throws Exception { + assertMemoryLeak(() -> { + // putRawMessage() sends a pre-formatted ILP line as the first row; it must stamp the deferred provider + // token first, or the raw message would ship with no Authorization header. F7: covers the + // stampTokenIfPending() call that putRawMessage() gained. + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + AtomicInteger calls = new AtomicInteger(); + HttpTokenProvider provider = () -> "TOKEN-" + calls.incrementAndGet(); + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + ((AbstractLineHttpSender) sender).putRawMessage(new Utf8String("t v=1i\n")); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals("the raw-message flush must reach the server", 1, auth.size()); + Assert.assertEquals("putRawMessage must carry the provider token", "Bearer TOKEN-1", auth.get(0)); + } + }); + } + + @Test(timeout = 30_000) + public void testTokenReachesAuthorizationHeaderAndRotatesPerFlush() throws Exception { + assertMemoryLeak(() -> { + // end-to-end against a real socket: the pulled token must reach the "Authorization: Bearer" header + // on the wire (not merely be pulled), and a rotating provider must be re-queried per request so a + // long-lived sender follows token refreshes rather than sending a token captured once. + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(204, ""))) { + AtomicInteger calls = new AtomicInteger(); + HttpTokenProvider provider = () -> "TOKEN-" + calls.incrementAndGet(); + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); + } + List auth = server.requestAuthHeaders(); + Assert.assertEquals("two flushes must send two requests", 2, auth.size()); + Assert.assertEquals("the first request must carry the first pulled token", "Bearer TOKEN-1", auth.get(0)); + Assert.assertEquals("the second flush must re-query the provider and carry the rotated token", "Bearer TOKEN-2", auth.get(1)); + } + }); + } + + private static void assertProviderTokenRejected(HttpTokenProvider provider, String expectedMessage) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:1") + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .disableAutoFlush() + .httpTokenProvider(provider) + .build()) { + try { + sender.table("t").longColumn("v", 1L).atNow(); + Assert.fail("expected an invalid provider token to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + } + } + } + +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionRetryableTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionRetryableTest.java new file mode 100644 index 000000000..ed178de03 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionRetryableTest.java @@ -0,0 +1,219 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.line; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import org.junit.Assert; +import org.junit.Test; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Covers {@link LineSenderException#isRetryable()}. + *

+ * The class documentation tells a caller to act on exactly this distinction - retry {@code flush()} on a + * transient failure, close or {@code reset()} on a permanent one - and the sender already computes the + * answer for every failure it raises. The two-argument constructor accepted that classification and then + * dropped it on the floor, so the advice was unactionable: six call sites passed a flag no caller could + * read. + */ +public class LineSenderExceptionRetryableTest { + + @Test + public void testConstructorsWithoutAClassificationReportNotRetryable() { + // false means "not classified as retryable", not "proven permanent". These constructors carry no + // classification at all, and false is the conservative direction for a caller that retries only + // on true - it stops rather than spins. + Assert.assertFalse(new LineSenderException("boom").isRetryable()); + Assert.assertFalse(new LineSenderException(new RuntimeException("boom")).isRetryable()); + Assert.assertFalse(new LineSenderException("boom", new RuntimeException("boom")).isRetryable()); + } + + @Test + public void testExplicitClassificationSurvivesConstruction() { + Assert.assertTrue(new LineSenderException("transient", true).isRetryable()); + Assert.assertFalse(new LineSenderException("permanent", false).isRetryable()); + // the flag must survive the fluent message building every call site does after construction + LineSenderException built = new LineSenderException("transient", true) + .put(" [http-status=").put(503).put(']'); + Assert.assertTrue("building the message must not lose the classification", built.isRetryable()); + } + + @Test(timeout = 30_000) + public void testADefinitiveStatusFromTheSenderIsNotRetryable() throws Exception { + assertMemoryLeak(() -> { + // End to end, through the sender's own classification rather than a hand-built exception: a 401 + // is definitive, so a caller must be able to tell it from a 503 and stop instead of re-flushing + // into an endpoint that will keep refusing. + // A CHUNKED 401: flush0 asserts response.isChunked() on the error branch, and MockOidcServer.json + // writes a Content-Length body, so the plain helper trips that assert (with -ea on) before the + // classification is ever reached. + final String errorBody = "{\"code\":\"unauthorized\"}"; + final String chunked401 = "HTTP/1.1 401 Unauthorized\r\n" + + "Content-Type: application/json\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(errorBody.length()) + "\r\n" + errorBody + "\r\n0\r\n\r\n"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> + MockOidcServer.raw(chunked401))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(1_000) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the 401 to surface"); + } catch (LineSenderException e) { + Assert.assertFalse("a 401 is definitive; a caller told to retry on it would spin " + + "against an endpoint that keeps refusing: " + e.getMessage(), + e.isRetryable()); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testARetryableStatusFromTheSenderIsRetryable() throws Exception { + assertMemoryLeak(() -> { + // The other direction, and the one that carries the risk. assertFalse cannot tell a CLASSIFIED + // "permanent" from an UNCLASSIFIED exception, because the three constructors that carry no + // classification also report false - so the 401 test above passes just as happily against a + // sender that stopped classifying altogether. Only a true here proves the flag is computed and + // survives the throw. + // + // A 503 exhausts the retry budget and then throws with retryable=true. Chunked, because flush0 + // asserts response.isChunked() on the error branch and the plain helper writes Content-Length. + final String errorBody = "{\"code\":\"unavailable\"}"; + final String chunked503 = "HTTP/1.1 503 Service Unavailable\r\n" + + "Content-Type: application/json\r\n" + + "Transfer-Encoding: chunked\r\n\r\n" + + Integer.toHexString(errorBody.length()) + "\r\n" + errorBody + "\r\n0\r\n\r\n"; + try (MockOidcServer server = new MockOidcServer((method, path, body) -> + MockOidcServer.raw(chunked503))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(100) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the 503 to surface once the retry budget is spent"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("503")); + Assert.assertTrue("a 503 is transient; a caller told to close or reset() on it would " + + "tear down a healthy sender and drop the buffered batch: " + + e.getMessage(), + e.isRetryable()); + } + } + } + }); + } + + @Test(timeout = 30_000) + public void testATransportFailureFromTheSenderIsRetryable() throws Exception { + assertMemoryLeak(() -> { + // The sender's other retryable=true site: the give-up throw after the retry budget is spent on a + // transport error rather than a status. It reaches the caller through a different constructor + // call than the status path, so it needs its own assertion. + final int deadPort; + try (java.net.ServerSocket probe = new java.net.ServerSocket(0, 1, + java.net.InetAddress.getLoopbackAddress())) { + deadPort = probe.getLocalPort(); + } + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + deadPort) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(100) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the unreachable endpoint to surface"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("Connection Failed")); + Assert.assertTrue("a transport failure is transient by definition: " + e.getMessage(), + e.isRetryable()); + } + } + }); + } + + @Test(timeout = 30_000) + public void testAnUnreadableRetryableBodyStaysRetryable() throws Exception { + assertMemoryLeak(() -> { + // The wrapper path, and the direction that carries the risk on it. Every other test here reaches + // throwOnHttpErrorResponse0, which builds its exception beside the body it just read. When that + // read ABORTS - a dribbled body, a peer that vanished, a mangled chunk - the outer + // throwOnHttpErrorResponse catches it and builds a different exception from the status alone, + // re-passing `retryable` by hand. Nothing pinned that hand-off, so hardcoding it either way was + // green. + // + // A 503 whose body dribbles: flush0 retries on the status until retryTimeoutMillis is spent (the + // head arrives promptly each time, so those passes are fast), then reads the body to build the + // message and aborts on the whole-read bound. Told this was permanent, a caller following the + // documented advice closes or reset()s a healthy sender and drops the buffered batch over a fault + // that was going to clear. + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.dribble(503))) { + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("127.0.0.1:" + server.port()) + .protocolVersion(Sender.PROTOCOL_VERSION_V1) + .httpTimeoutMillis(1_000) + .retryTimeoutMillis(100) + .disableAutoFlush() + .build()) { + sender.table("t").longColumn("v", 1L).atNow(); + try { + sender.flush(); + Assert.fail("expected the 503 to surface once the retry budget is spent"); + } catch (LineSenderException e) { + String msg = e.getMessage(); + // the wrapper's own shape, so a later refactor that routes this through the + // body-reading path instead cannot satisfy the assertion below by accident + Assert.assertTrue("expected the unreadable-body wrapper: " + msg, + msg.contains("could not read the error response body")); + Assert.assertTrue("the real status must survive the wrapper: " + msg, + msg.contains("http-status=503")); + Assert.assertTrue("a 503 is transient however unreadable its body: a caller told to " + + "close or reset() on it tears down a healthy sender and drops the " + + "buffered batch: " + msg, + e.isRetryable()); + } + } + } + }); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionTest.java index 4d05487e1..27e6e5d8f 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/LineSenderExceptionTest.java @@ -53,6 +53,53 @@ public void testMessage_PutAsPrintableWithNonPrintableInput() { } + @Test + public void testMessage_putAsPrintableEscapesBidiOverride() { + // U+202E RIGHT-TO-LEFT OVERRIDE is a BMP format char - regression guard for the existing behavior + LineSenderException e = new LineSenderException("char: ").putAsPrintable("a\u202Eb"); + assertEquals("char: a\\u202eb", e.getMessage()); + } + + @Test + public void testMessage_putAsPrintableEscapesLoneSurrogate() { + // a lone high surrogate has no displayable meaning and must be escaped, not passed through raw + LineSenderException e = new LineSenderException("char: ").putAsPrintable("a\uD800b"); + assertEquals("char: a\\ud800b", e.getMessage()); + } + + @Test + public void testMessage_putAsPrintableEscapesSupplementaryFormatChar() { + // U+E0001 LANGUAGE TAG is a supplementary-plane format char: it arrives as a surrogate pair and must + // be escaped (as both halves), not passed through raw, or it could hide or forge text in a log + String tagChar = new String(Character.toChars(0xE0001)); + LineSenderException e = new LineSenderException("char: ").putAsPrintable("a" + tagChar + "b"); + assertEquals("char: a\\udb40\\udc01b", e.getMessage()); + } + + @Test + public void testMessage_putAsPrintableKeepsEmoji() { + // U+1F600 GRINNING FACE is a normal supplementary char (not control or format) - emitted verbatim + String emoji = new String(Character.toChars(0x1F600)); + LineSenderException e = new LineSenderException("char: ").putAsPrintable("a" + emoji + "b"); + assertEquals("char: a" + emoji + "b", e.getMessage()); + } + + @Test + public void testMessage_putAsPrintableAgreesOnBothPaths() { + // putAsPrintable now classifies before it copies: an all-printable sequence is handed to + // put(CharSequence) in one go, and only a sequence carrying something unsafe is walked and escaped + // character by character. Two paths mean they can drift, so pin that they agree - the same text, + // with and without one unsafe code point in it, must differ only by that code point's escape. + String printable = "Could not flush buffer: table 'trades' column 'price' rejected, line 42"; + assertEquals(printable, new LineSenderException("").putAsPrintable(printable).getMessage()); + + // the escaping path over the same text, with a bidi override spliced into the middle + int at = printable.indexOf("column"); + String tampered = printable.substring(0, at) + (char) 0x202e + printable.substring(at); + assertEquals(printable.substring(0, at) + "\\u202e" + printable.substring(at), + new LineSenderException("").putAsPrintable(tampered).getMessage()); + } + @Test public void testMessage_withErrNo() { LineSenderException e = new LineSenderException("message").errno(10); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/line/interop/ClientInteropTest.java b/core/src/test/java/io/questdb/client/test/cutlass/line/interop/ClientInteropTest.java index 92dba65d1..16c7b5592 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/line/interop/ClientInteropTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/line/interop/ClientInteropTest.java @@ -36,7 +36,6 @@ import io.questdb.client.std.Numbers; import io.questdb.client.std.NumericException; import io.questdb.client.std.bytes.DirectByteSink; -import io.questdb.client.std.str.StringSink; import io.questdb.client.test.cutlass.line.tcp.ByteChannel; import io.questdb.client.test.tools.TestUtils; import org.junit.Assert; @@ -91,7 +90,6 @@ private static class JsonTestSuiteParser implements JsonParser { public static final int TAG_TEST_NAME = 0; private final ByteChannel byteChannel; private final Sender sender; - private final StringSink stringSink = new StringSink(); private int columnType = -1; private boolean encounteredError; private String name; @@ -105,7 +103,7 @@ public JsonTestSuiteParser(Sender sender, ByteChannel channel) { @Override public void onEvent(int code, CharSequence tag, int position) throws JsonException { - tag = unescape(tag, stringSink); + // JsonLexer already resolves JSON string escape sequences, so `tag` arrives fully decoded. switch (code) { case JsonLexer.EVT_NAME: if (Chars.equalsIgnoreCase(tag, "testname")) { @@ -269,70 +267,6 @@ private static boolean isTrueKeyword(CharSequence tok) { && (tok.charAt(3) | 32) == 'e'; } - private static CharSequence unescape(CharSequence tag, StringSink stringSink) { - if (tag == null) { - return null; - } - stringSink.clear(); - - for (int i = 0, n = tag.length(); i < n; i++) { - char sourceChar = tag.charAt(i); - if (sourceChar != '\\') { - // happy-path, nothing to unescape - stringSink.put(sourceChar); - } else { - // slow path. either there is a code unit sequence. think of this: foo\u0001bar - // or a simple escaping: \n, \r, \\, \", etc. - // in both cases we will consume more than 1 character from the input, - // so we have to adjust "i" accordingly - - // malformed input could throw IndexOutOfBoundsException, but given we control - // the test data then we are OK. - char nextChar = tag.charAt(i + 1); - if (nextChar == 'u') { - // code unit sequence - char ch; - try { - ch = (char) Numbers.parseHexInt(tag, i + 2, i + 6); - } catch (NumericException e) { - throw new AssertionError("cannot parse code sequence in " + tag); - } - stringSink.put(ch); - i += 5; - } else if (nextChar == '\\') { - stringSink.put('\\'); - i++; - } else if (nextChar == '\"') { - stringSink.put('\"'); - i++; - } else if (nextChar == 'b') { - // backspace - stringSink.put('\b'); - i++; - } else if (nextChar == 'f') { - // form-feed - stringSink.put('\f'); - i++; - } else if (nextChar == 'n') { - // new line - stringSink.put('\n'); - i++; - } else if (nextChar == 'r') { - // carriage return - stringSink.put('\r'); - i++; - } else if (nextChar == 't') { - // tab - stringSink.put('\t'); - i++; - } else { - throw new AssertionError("Unknown escaping sequence at " + tag); - } - } - } - return stringSink.toString(); - } - private void assertSuccessfulLine(byte[] tag) { Assert.assertTrue("Produced line does not end with a new line char", byteChannel.endWith((byte) '\n')); Assert.assertTrue("buffer base64[" + byteChannel.encodeBase64String() + "]", byteChannel.equals(tag, 0, tag.length - 1)); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/CloseDrainTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/CloseDrainTest.java index d6f8e4256..900ce108a 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/CloseDrainTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/CloseDrainTest.java @@ -29,15 +29,19 @@ import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; +import org.junit.Assume; import org.junit.Test; import java.io.IOException; +import java.lang.management.ManagementFactory; +import java.lang.management.ThreadMXBean; import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.nio.file.Paths; import java.util.Arrays; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; @@ -157,6 +161,47 @@ public void testCloseBlocksUntilAckArrives() throws Exception { } } + @Test(timeout = 30_000L) + public void testInterruptDuringSuccessfulCloseDrainIsRestoredAfterTeardown() throws Exception { + DelayingAckHandler handler = new DelayingAckHandler(800L); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + QwpWebSocketSender sender = (QwpWebSocketSender) Sender.fromConfig( + "ws::addr=localhost:" + server.getPort() + ";"); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + sender.setCloseDrainWaitingHook(() -> { + // PoolHousekeeper.stop() uses this signal after its first join budget. The close drain owns it + // while waiting for the ACK; later teardown waits must not mistake the carried signal for a new + // failure to stop their own worker. + Thread.currentThread().interrupt(); + }); + + Throwable closeFailure = null; + boolean interruptedAtReturn; + try { + sender.close(); + } catch (Throwable t) { + closeFailure = t; + } finally { + interruptedAtReturn = Thread.currentThread().isInterrupted(); + Thread.interrupted(); + } + + Assert.assertNull("an interrupt consumed by a successful ACK drain poisoned later teardown", + closeFailure); + Assert.assertTrue("close() must return its consumed cancellation signal to the caller", + interruptedAtReturn); + Assert.assertTrue("the delayed ACK must have completed the close drain", handler.nextSeq.get() >= 1L); + Assert.assertTrue("successful ACK drain must release its slot before close returns", + sender.isSlotLockReleased()); + Assert.assertTrue("all close-owned resources must be released before close returns", + sender.isCloseCleanupComplete()); + } + } + @Test public void testCloseStartedHookRunsAfterClosedStateTransition() throws Exception { QwpWebSocketSender sender = QwpWebSocketSender.createForTesting("localhost", 1); @@ -310,6 +355,73 @@ public void testCloseDrainTimesOutWhenAcksNeverArrive() throws Exception { } } + @Test(timeout = 30_000L) + public void testInterruptedCloseDrainRemainsPaced() throws Exception { + ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean(); + Assume.assumeTrue("current-thread CPU time is required for this regression", + threadMxBean.isCurrentThreadCpuTimeSupported()); + if (!threadMxBean.isThreadCpuTimeEnabled()) { + threadMxBean.setThreadCpuTimeEnabled(true); + } + + final long timeoutMillis = 750L; + try (TestWebSocketServer server = new TestWebSocketServer(new SilentHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + String cfg = "ws::addr=localhost:" + server.getPort() + + ";close_flush_timeout_millis=" + timeoutMillis + ";"; + QwpWebSocketSender sender = (QwpWebSocketSender) Sender.fromConfig(cfg); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + + AtomicBoolean drainReached = new AtomicBoolean(); + AtomicLong closeCpuNanos = new AtomicLong(-1L); + AtomicLong closeWallNanos = new AtomicLong(-1L); + AtomicReference closeFailure = new AtomicReference<>(); + sender.setCloseDrainWaitingHook(() -> { + drainReached.set(true); + // PoolHousekeeper.stop() applies this escalation after its first join budget expires. + Thread.currentThread().interrupt(); + }); + Thread closer = new Thread(() -> { + long cpuStart = threadMxBean.getCurrentThreadCpuTime(); + long wallStart = System.nanoTime(); + try { + sender.close(); + } catch (Throwable t) { + closeFailure.set(t); + } finally { + closeWallNanos.set(System.nanoTime() - wallStart); + closeCpuNanos.set(threadMxBean.getCurrentThreadCpuTime() - cpuStart); + Thread.interrupted(); + } + }, "interrupted-close-drain"); + try { + closer.start(); + closer.join(10_000L); + } finally { + if (closer.isAlive()) { + closer.interrupt(); + closer.join(10_000L); + } + sender.close(); + } + + Assert.assertFalse("interrupted close drain did not finish", closer.isAlive()); + Assert.assertTrue("close never reached a real unacknowledged drain target", drainReached.get()); + Assert.assertTrue("silent server must end in the configured drain timeout", + closeFailure.get() instanceof LineSenderException + && closeFailure.get().getMessage().contains("drain timed out")); + long wallMillis = TimeUnit.NANOSECONDS.toMillis(closeWallNanos.get()); + long cpuMillis = TimeUnit.NANOSECONDS.toMillis(closeCpuNanos.get()); + Assert.assertTrue("close drain returned before its timeout [wallMillis=" + wallMillis + ']', + wallMillis >= timeoutMillis); + Assert.assertTrue("interrupted close drain spun instead of parking [cpuMillis=" + cpuMillis + + ", wallMillis=" + wallMillis + ']', + cpuMillis < timeoutMillis / 2); + } + } + @Test public void testCloseDrainTimeoutNamesTheReconnectOutage() throws Exception { // The drain-timeout message exists to name the outage the I/O thread is diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientInterruptedCloseLeakTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientInterruptedCloseLeakTest.java new file mode 100644 index 000000000..34986a67a --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientInterruptedCloseLeakTest.java @@ -0,0 +1,93 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.util.concurrent.TimeUnit; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * {@link QwpQueryClient#close()} must free the I/O thread's native buffer pool and its WebSocket even + * when the calling thread arrives carrying an interrupt. + *

+ * {@code Thread.join(long)} throws {@code InterruptedException} the instant the caller's flag is set, + * without ever checking whether the joined thread exited. Before the fix that turned close()'s I/O-thread + * join into an immediate throw, taking the "could not join" return and skipping {@code closePool()} and + * {@code webSocketClient.close()} - a leak with no second chance, because {@code closedFlag} is CAS'd on + * entry and a pooled worker has already been removed from {@code QueryClientPool.all} by the reap that + * called it. + *

+ * The path is real rather than theoretical: {@code PoolHousekeeper.stop()} interrupts the housekeeper + * thread to break a recovery build's credential pull, and that same thread runs {@code + * queryPool.reapIdle()} immediately afterwards with the flag still set. + *

+ * {@code assertMemoryLeak} is the assertion - it compares native memory per tag around the body, so a + * skipped {@code closePool()} fails the test. The interrupt-preserved check guards the other half of the + * contract: taking the flag out of the way must not swallow the caller's cancellation. + */ +public class QwpQueryClientInterruptedCloseLeakTest { + + @Test(timeout = 30_000) + public void testCloseFreesNativeResourcesWhenTheCallerCarriesAnInterrupt() throws Exception { + try { + assertMemoryLeak(() -> { + TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + }); + server.setSendServerInfo(true); + try { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + server.getPort() + ";auth_timeout_ms=2000;"); + try { + client.connect(); + Assert.assertTrue("the client must bind the endpoint, or there is no I/O thread " + + "and no buffer pool for this test to observe", client.isConnected()); + + // Arrive at close() already interrupted, exactly as the housekeeper does after + // stop() escalates. + Thread.currentThread().interrupt(); + } finally { + client.close(); + } + + Assert.assertTrue("close() must hand the caller's cancellation back, not swallow it", + Thread.currentThread().isInterrupted()); + } finally { + server.close(); + } + }); + } finally { + // Never let the flag escape into the next test on this thread. + Thread.interrupted(); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientPostConnectGuardTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientPostConnectGuardTest.java index a220084ab..98235ed9e 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientPostConnectGuardTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientPostConnectGuardTest.java @@ -51,6 +51,8 @@ public void testAllSettersRejectAfterConnect() throws Exception { assertRejects(c -> c.withBasicAuth("u", "p"), "withBasicAuth"); // withBearerToken assertRejects(c -> c.withBearerToken("tok"), "withBearerToken"); + // withBearerTokenProvider + assertRejects(c -> c.withBearerTokenProvider(() -> "tok"), "withBearerTokenProvider"); // withBufferPoolSize assertRejects(c -> c.withBufferPoolSize(2), "withBufferPoolSize"); // withClientId diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientTokenProviderTest.java new file mode 100644 index 000000000..e66e0e5ba --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientTokenProviderTest.java @@ -0,0 +1,382 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.HandOffCharSequence; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.net.InetAddress; +import java.net.ServerSocket; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Unit coverage for {@link QwpQueryClient#withBearerTokenProvider}: header + * synthesis, re-query at each resolve (a fresh token per WebSocket upgrade), + * token validation, null rejection, and mutual exclusion with the fixed-token + * and basic-auth setters - exercised both through + * {@link QwpQueryClient#getAuthorizationHeaderForTest()} (which resolves the + * header the same way a real upgrade does) and, for the real connect path, + * against a loopback mock that captures the upgrade's {@code Authorization} + * header and confirms a throwing provider fails the connection attempt. The + * post-connect guard for the setter lives in + * {@link QwpQueryClientPostConnectGuardTest}. + *

+ * Every test runs under {@code assertMemoryLeak}: a {@link QwpQueryClient} + * mallocs native scratch in its constructor, so each case proves that scratch + * is freed on close, including on the connect/error paths. + */ +public class QwpQueryClientTokenProviderTest { + + private static final QwpColumnBatchHandler NOOP_BATCH_HANDLER = new QwpColumnBatchHandler() { + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + }; + + @Test + public void testProviderBufferMutatedDuringResolveCannotSplice() throws Exception { + assertMemoryLeak(() -> { + // resolveAuthorizationHeader snapshots the pulled value before validating it, so the bytes that + // are checked are the bytes that are sent. Without the snapshot validateToken scans the + // provider's live sequence and the "Bearer " concatenation then materialises it a second time - + // two reads of a buffer HttpTokenProvider explicitly invites a provider to reuse. A mutation + // landing between them passes the check and splices CR/LF into the upgrade header. + // + // The ILP sender's copy of this rule is pinned by + // LineHttpSenderTokenProviderTest.testTokenMutatedBetweenValidationAndTheHeaderCannotSplice; + // this is the same rule at the query client's callsite, which had no test. + final String clean = "GOODTOKEN"; + final String spliced = "abc" + (char) 0x0d + (char) 0x0a + "X-Injected: pwned"; + AtomicInteger pulls = new AtomicInteger(); + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> { + pulls.incrementAndGet(); + return new HandOffCharSequence(clean, spliced); + })) { + Assert.assertEquals("the header must carry the bytes that were validated, not a value " + + "swapped in after the scan", "Bearer " + clean, c.getAuthorizationHeaderForTest()); + Assert.assertEquals("the provider must have been queried, or this test passes for the " + + "wrong reason", 1, pulls.get()); + } + }); + } + + @Test + public void testOidcProviderFailureIsWrappedAsLineSenderException() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> { + throw new OidcAuthException("the cached token could not be refreshed"); + })) { + try { + c.getAuthorizationHeaderForTest(); + Assert.fail("an OIDC provider failure must fail token resolution"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), + e.getMessage().contains("the cached token could not be refreshed")); + Assert.assertTrue("the provider failure must be retained as the cause", + e.getCause() instanceof OidcAuthException); + } + } + }); + } + + @Test + public void testProviderConflictsWithBasicAuth() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000).withBearerTokenProvider(() -> "tok")) { + try { + c.withBasicAuth("u", "p"); + Assert.fail("withBasicAuth after withBearerTokenProvider must throw"); + } catch (IllegalStateException expected) { + // mutually exclusive + } + } + }); + } + + @Test + public void testProviderConflictsWithBearerToken() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000).withBearerTokenProvider(() -> "tok")) { + try { + c.withBearerToken("other"); + Assert.fail("withBearerToken after withBearerTokenProvider must throw"); + } catch (IllegalStateException expected) { + // mutually exclusive + } + } + }); + } + + @Test + public void testProviderNullOrBlankReturnRejected() throws Exception { + assertMemoryLeak(() -> { + // validateToken rejects a null, empty or blank token RETURNED by the provider before it reaches the + // "Bearer " header (distinct from testProviderNullRejected, which rejects a null provider at the setter) + String[] bad = {null, "", " "}; + for (String token : bad) { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> token)) { + try { + c.getAuthorizationHeaderForTest(); + Assert.fail("a null/empty/blank provider token must be rejected, was: [" + token + ']'); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("null or empty")); + } + } + } + }); + } + + @Test + public void testProviderNullRejected() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000)) { + try { + c.withBearerTokenProvider(null); + Assert.fail("a null provider must be rejected"); + } catch (IllegalArgumentException expected) { + // expected + } + } + }); + } + + @Test + public void testProviderQueriedAtEachResolve() throws Exception { + assertMemoryLeak(() -> { + int[] counter = {0}; + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> "tok-" + (counter[0]++))) { + // each resolve re-queries the provider, so a reconnect presents a fresh token + Assert.assertEquals("Bearer tok-0", c.getAuthorizationHeaderForTest()); + Assert.assertEquals("Bearer tok-1", c.getAuthorizationHeaderForTest()); + } + }); + } + + @Test + public void testProviderSynthesizesBearerHeader() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> "abc123")) { + Assert.assertEquals("Bearer abc123", c.getAuthorizationHeaderForTest()); + } + }); + } + + @Test(timeout = 20_000) + public void testProviderTokenReResolvedOnFailoverReconnect() throws Exception { + assertMemoryLeak(() -> { + // The failover reconnect path (reconnectViaTracker) resolves the Authorization header once before its + // endpoint walk, exactly as connect() does, so a rotating token reaches the reconnect upgrade. This + // pins that a regression dropping the re-resolve from the reconnect path would be caught: bind endpoint + // A on the initial connect (capturing tok-0), drop it, then run a query - the failover reconnect to + // endpoint B must upgrade with a FRESHLY resolved token, not the stale connect-time one. + AtomicInteger calls = new AtomicInteger(); + TestWebSocketServer a = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + }); + a.setSendServerInfo(true); + TestWebSocketServer b = new TestWebSocketServer(new ExecDoneQueryServer()); + b.setSendServerInfo(true); + try { + a.start(); + b.start(); + Assert.assertTrue(a.awaitStart(5, TimeUnit.SECONDS)); + Assert.assertTrue(b.awaitStart(5, TimeUnit.SECONDS)); + + try (QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + ";auth_timeout_ms=2000;") + .withBearerTokenProvider(() -> "tok-" + calls.getAndIncrement())) { + client.connect(); + Assert.assertTrue("client must bind the first endpoint on connect", client.isConnected()); + String aHeader = a.pollAuthorizationHeader(5, TimeUnit.SECONDS); + Assert.assertEquals("the initial connect upgrade must carry the first resolved token", + "Bearer tok-0", aHeader); + + // drop endpoint A so the next execute() cannot use its connection and must fail over + a.close(); + + // the query fails on the dead A connection, drives the failover loop -> reconnectViaTracker, + // which re-resolves the header and upgrades B; B answers EXEC_DONE so execute() returns + client.execute("SELECT 1", NOOP_BATCH_HANDLER, false); + + String bHeader = b.pollAuthorizationHeader(5, TimeUnit.SECONDS); + Assert.assertNotNull("the failover reconnect must upgrade endpoint B", bHeader); + Assert.assertTrue("the reconnect upgrade must carry a Bearer token, was: " + bHeader, + bHeader.startsWith("Bearer tok-")); + Assert.assertNotEquals("the failover reconnect must RE-RESOLVE the provider, not reuse the " + + "connect-time token", aHeader, bHeader); + } + } finally { + a.close(); + b.close(); + } + }); + } + + @Test(timeout = 15_000) + public void testProviderTokenSentOnRealUpgrade() throws Exception { + assertMemoryLeak(() -> { + // drive the REAL connect path (connect() -> resolveAuthorizationHeader -> runUpgradeWithTimeout), + // not the test hook: the upgrade request must carry the freshly pulled "Bearer ". The mock + // answers 404 (not auth-failed, not terminal) so connect() fails fast after the header was sent. + // MockOidcServer is the harness for this: it records the Authorization header of every request it + // reads and resurfaces a handler throwable on close(), where the hand-rolled listener this + // replaced swallowed every harness fault into `catch (Exception ignored)` - so an accept, read or + // write that broke arrived as a MISSING header, i.e. as a product regression. + try (MockOidcServer server = new MockOidcServer((method, path, body) -> MockOidcServer.json(404, "")); + QwpQueryClient client = QwpQueryClient + .fromConfig("ws::addr=127.0.0.1:" + server.port() + ";failover=off;target=any;") + .withBearerTokenProvider(() -> "tok-0")) { + try { + client.connect(); + Assert.fail("expected connect to fail on a 404 upgrade"); + } catch (HttpClientException expected) { + // 404 is neither auth-failed nor terminal: the endpoint is exhausted and connect() fails - + // but the upgrade request already carried the Bearer header captured below + } + List authHeaders = server.requestAuthHeaders(); + Assert.assertEquals("the provider's token must reach the real upgrade request", + 1, authHeaders.size()); + Assert.assertEquals("Bearer tok-0", authHeaders.get(0)); + } + }); + } + + @Test + public void testProviderTokenValidated() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(() -> "bad\ntoken")) { + try { + c.getAuthorizationHeaderForTest(); + Assert.fail("a token carrying a control character must be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("control or non-ASCII")); + } + } + }); + } + + @Test + public void testSettingBearerTokenThenProviderConflicts() throws Exception { + assertMemoryLeak(() -> { + try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000).withBearerToken("tok")) { + try { + c.withBearerTokenProvider(() -> "other"); + Assert.fail("withBearerTokenProvider after withBearerToken must throw"); + } catch (IllegalStateException expected) { + // mutually exclusive + } + } + }); + } + + @Test(timeout = 10_000) + public void testThrowingProviderFailsConnect() throws Exception { + assertMemoryLeak(() -> { + // a provider that throws must fail the connection attempt on the REAL connect path: + // resolveAuthorizationHeader runs once before the endpoint walk, so the throw propagates straight + // out of connect() as the provider's own error (not wrapped as "all endpoints unreachable") + try ( + ServerSocket listener = new ServerSocket(0, 50, InetAddress.getLoopbackAddress()); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=127.0.0.1:" + listener.getLocalPort() + ";failover=off;target=any;" + ).withBearerTokenProvider(() -> { + throw new LineSenderException("provider down"); + }) + ) { + try { + client.connect(); + Assert.fail("a throwing provider must fail the connection attempt"); + } catch (RuntimeException expected) { + // the provider's own exception propagates directly (the header is resolved before the + // endpoint walk), not wrapped as a transport "all endpoints unreachable" error + Assert.assertTrue(expected.getClass().getName(), expected instanceof LineSenderException); + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("provider down")); + Assert.assertFalse(expected.getMessage(), expected.getMessage().contains("unreachable")); + } + } + }); + } + + private static byte[] buildExecDone(byte[] queryRequest) { + int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1); // version + bb.put((byte) 0); // flags + bb.putShort((short) 0); // table_count + bb.putInt(bodyLen); // payload_length + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(queryRequest, 1, 8); // echo request_id verbatim + bb.put((byte) 0); // op_type + bb.put((byte) 0); // rows_affected = 0 + return frame; + } + + private static final class ExecDoneQueryServer implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length == 0 || data[0] != QwpEgressMsgKind.QUERY_REQUEST) { + return; + } + try { + client.sendBinary(buildExecDone(data)); + } catch (IOException e) { + // best-effort: a failed reply surfaces to the client as a transport error + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpUdpSenderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpUdpSenderTest.java index f16bc2670..0d94377d6 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpUdpSenderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpUdpSenderTest.java @@ -880,6 +880,26 @@ public void testCloseDropsInProgressRowButFlushesCommittedRows() throws Exceptio }); } + @Test + public void testColumnRejectsInvalidCharactersAreEscaped() throws Exception { + assertMemoryLeak(() -> { + CapturingNetworkFacade nf = new CapturingNetworkFacade(); + try (QwpUdpSender sender = new QwpUdpSender(nf, 0, 0, 9000, 1)) { + // a rejected COLUMN name carrying a display-unsafe char must be ESCAPED in the message, not + // spliced in raw (M6 parity with the ILP name/error render, at the QwpTableBuffer layer) + try { + sender.table("t").longColumn("bad" + (char) 0x01 + "col", 1L); + Assert.fail("expected an illegal column name to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue("the control char must be escaped: " + e.getMessage(), + e.getMessage().contains("\\u0001")); + Assert.assertTrue("a raw control char must not leak into the message", + e.getMessage().indexOf((char) 0x01) < 0); + } + } + }); + } + @Test public void testDuplicateColumnAfterSchemaFlushReplayIsRejected() throws Exception { assertMemoryLeak(() -> { @@ -1715,6 +1735,19 @@ public void testTableRejectsInvalidCharacters() throws Exception { sender.table(".leading_dot") ); + // a rejected name carrying a display-unsafe char (here a control char; also bidi/BOM/zero-width) + // must be ESCAPED in the message, not spliced in raw where it could reorder, hide or forge what a + // human reads - M6 parity with the ILP name/error render + try { + sender.table("bad" + (char) 0x01 + "name"); + Assert.fail("expected an illegal table name to be rejected"); + } catch (LineSenderException e) { + Assert.assertTrue("the control char must be escaped: " + e.getMessage(), + e.getMessage().contains("\\u0001")); + Assert.assertTrue("a raw control char must not leak into the message", + e.getMessage().indexOf((char) 0x01) < 0); + } + // Sender must remain usable after rejected names sender.table("valid") .longColumn("x", 1) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java index b1e21870c..3911f5c05 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java @@ -833,6 +833,30 @@ private static void assertClosed(Runnable r) { } } + @Test + public void testIllegalTableNameIsEscapedInTheMessage() throws Exception { + assertMemoryLeak(() -> { + // A rejected table name is attacker-influenced text on its way to a log line or a terminal, so + // it is escaped rather than concatenated. Three of the four callsites that do this are covered - + // QwpUdpSender and QwpTableBuffer by QwpUdpSenderTest, the ILP names by AbstractLineSender's + // tests - and this one, the WebSocket sender's own check, was not: its message could regress to + // a raw concatenation with the suite still green. + try (QwpWebSocketSender sender = createUnconnectedSender()) { + try { + sender.table("bad" + (char) 0x01 + "name"); + Assert.fail("an illegal table name must be rejected"); + } catch (LineSenderException e) { + final String message = e.getMessage(); + Assert.assertTrue(message, message.contains("table name contains illegal characters")); + Assert.assertTrue("the offending char must be escaped: " + message, + message.contains("\\u0001")); + Assert.assertTrue("the raw control char must not reach the message", + message.indexOf(0x01) < 0); + } + } + }); + } + private static MicrobatchBuffer getMicrobatchBuffer(QwpWebSocketSender sender, String fieldName) throws Exception { Field field = QwpWebSocketSender.class.getDeclaredField(fieldName); field.setAccessible(true); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SlotLockReleasedContractTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SlotLockReleasedContractTest.java index b777c8a87..2ae605b7f 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SlotLockReleasedContractTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SlotLockReleasedContractTest.java @@ -113,6 +113,59 @@ public void testSlotLockReleasedAfterCleanClose() throws Exception { }); } + /** + * Interrupt-neutrality: a CARRIED interrupt flag must not turn a healthy {@code close()} into a + * failed stop. + *

+ * {@code PoolHousekeeper.stop()} interrupts a housekeeper blocked in a pooled credential pull. + * {@code SenderPool.stopStartupRecoveryDriver()} has no token provider by construction, but can likewise + * interrupt an unexpected overrun in its direct recovery driver. Those threads then run + * {@code senderPool.reapIdle()} or a startup-recovery step's {@code finally}, both of which close a + * delegate. That makes a carried flag ordinary on this path rather than exotic. + *

+ * It is also fatal if unhandled: {@code CountDownLatch.await(t, u)} tests {@code Thread.interrupted()} + * before it ever consults the latch, so the shutdown await returns instantly, {@code close()} takes the + * failed-stop branch, and the slot is reported as still flocked -- the exact outcome the interrupt was + * added to prevent. The failed-stop branch re-asserts the flag, so in a reap sweep every remaining + * delegate failed the same way. + *

+ * Asserted on both halves, because clearing the flag and forgetting to restore it would pass a + * released-lock check while silently eating the caller's cancellation. + */ + @Test + public void testCarriedInterruptNeitherFailsCloseNorRetainsTheSlotLock() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (TestWebSocketServer server = new TestWebSocketServer(new AckAllHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + String cfg = "ws::addr=localhost:" + port + ";close_flush_timeout_millis=2000;"; + QwpWebSocketSender wss = (QwpWebSocketSender) Sender.fromConfig(cfg); + wss.table("t").longColumn("v", 1L).atNow(); + wss.flush(); + + final boolean flagSurvived; + Thread.currentThread().interrupt(); + try { + wss.close(); + flagSurvived = Thread.currentThread().isInterrupted(); + } finally { + // never let it escape into the next test on a reused JUnit thread + Thread.interrupted(); + } + + Assert.assertTrue( + "close() must restore the caller's interrupt flag, not consume it", + flagSurvived); + Assert.assertTrue( + "a carried interrupt must not make a healthy close() report its slot lock retained", + wss.isSlotLockReleased()); + } + }); + } + + /** * Leak path: when {@code close()} cannot wind the I/O loop down it bails * out via the {@code !ioThreadStopped} early-return and must leave the slot @@ -368,12 +421,16 @@ public void testDelegatedIoThreadEngineCloseFlipsSlotLockReleased() throws Excep wss.setCursorEngine(engine, true); wss.setCursorSendLoopForTesting(loop); - // Drive the real early-bail close() on a thread whose pending - // interrupt lands in loop.close()'s shutdownLatch.await(). + // Drive the real early-bail close() through the loop's own bounded-await backstop. + // NOT by handing the closer a pending interrupt: close() is interrupt-neutral (it clears a + // CARRIED flag and restores it on the way out), because the pool threads that close + // delegates are the same ones PoolHousekeeper.stop() interrupts. Shrinking the backstop + // reaches the same failed-stop branch deterministically and without a 30s wait, which is + // exactly what this seam exists for. + loop.setShutdownAwaitTimeoutMillis(200L); AtomicReference closeFailure = new AtomicReference<>(); QwpWebSocketSender wssRef = wss; Thread closer = new Thread(() -> { - Thread.currentThread().interrupt(); try { wssRef.close(); } catch (Throwable t) { @@ -524,9 +581,11 @@ public void testFailedIoStopReclaimsSenderResourcesAfterWorkerExit() throws Exce Assert.assertNotNull(errorDispatcherThread); Assert.assertNotNull(progressDispatcherThread); + // Bounded-await backstop rather than a pending interrupt on the closer -- see the sibling + // test above: close() is interrupt-neutral, so a carried flag no longer short-circuits it. + loop.setShutdownAwaitTimeoutMillis(200L); AtomicReference closeFailure = new AtomicReference<>(); Thread closer = new Thread(() -> { - Thread.currentThread().interrupt(); try { sender.close(); } catch (Throwable t) { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketCredentialCancellationTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketCredentialCancellationTest.java new file mode 100644 index 000000000..82cf08ea0 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketCredentialCancellationTest.java @@ -0,0 +1,366 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.FileTokenStore; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import io.questdb.client.cutlass.auth.PersistedToken; +import io.questdb.client.cutlass.auth.TokenStoreKey; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.test.tools.NoBrowserLaunch; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLock; +import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLockContentionException; +import io.questdb.client.std.ObjList; +import io.questdb.client.test.cutlass.auth.MockOidcServer; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.ClassRule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Connect cancellation against the BUILT-IN credential path — a real {@link OidcDeviceAuth} over a real + * {@link FileTokenStore} — rather than a test double. + *

+ * QWP's close() cannot reach a credential pull through {@code closeTraffic()}: the pull is caller code + * owning no socket. Its only lever is an interrupt, which + * {@code CursorWebSocketSendLoop.ConnectCancellation.cancel()} sends to the thread published as being + * inside the pull. Whether that lever WORKS depends entirely on what the pull is blocked in, and every + * existing test blocks it in an interruptible test double, so the shipped path went unchecked: it waited + * on an uninterruptible {@code ReentrantLock.lock()} and polled the lock file through {@code Os.sleep}, + * which catches {@code InterruptedException} and keeps sleeping to its own deadline. The lock-acquire + * budget caps at 30s, the same as close()'s shutdown budget, so a sender closing while another + * same-identity instance held the lock burned the whole budget and then gave up on its I/O thread, + * delegating teardown of the native client, the cursor engine and the store-and-forward slot lock. + *

+ * The token endpoint is never reached in these tests: the pull cannot get past the store lock. That is + * asserted, because reaching it would mean the wait had already been abandoned for a lock-free refresh + * and the test would no longer be exercising the blocked path. + */ +public class WebSocketCredentialCancellationTest { + private static final String DEVICE_PATH = "/device"; + // Issued lifetime and remaining life of the seeded entry. effectiveSkewMillis caps the clock-skew + // margin at half the issued lifetime, so this reads as valid for (12s - 10s) = ~2s and stale after: + // long enough for the foreground connect to be a cache hit, short enough to force the reconnect's pull + // into a refresh without stubbing the clock. + private static final long SEED_REMAINING_MILLIS = 12_000L; + private static final long SEED_TTL_MILLIS = 20_000L; + private static final String TOKEN_PATH = "/token"; + + // the credential pull can reach the device-code prompt; see NoBrowserLaunch for why this is a rule + @ClassRule + public static final NoBrowserLaunch NO_BROWSER = new NoBrowserLaunch(); + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 120_000) + public void testCloseBreaksAForegroundReconnectBlockedInTheBuiltInCredentialPull() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenEndpointCalls = new AtomicInteger(); + try (MockOidcServer idp = new MockOidcServer((method, path, body) -> { + if (TOKEN_PATH.equals(path)) { + tokenEndpointCalls.incrementAndGet(); + } + return MockOidcServer.json(200, "{}"); + })) { + Path dir = storeDir(); + Files.createDirectories(dir); + // The maximum permitted acquire budget, which is exactly QWP's close() shutdown budget: this + // is the wait an interrupt has to be able to cut short. + FileTokenStore store = new FileTokenStore(dir, 30_000, 600_000); + TokenStoreKey key = keyFor(idp); + store.save(key, seededEntry()); + + DropAfterFirstAckHandler wire = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(wire)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + AtomicBoolean armed = new AtomicBoolean(); + CountDownLatch blockedPullStarted = new CountDownLatch(1); + try (OidcDeviceAuth auth = authFor(idp, store)) { + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + // The wrapper only reports that a pull started; the blocking is all done by + // the real OidcDeviceAuth/FileTokenStore underneath. + if (armed.get()) { + blockedPullStarted.countDown(); + } + return auth.getToken(); + }) + .build(); + boolean closed = false; + try { + // the seeded entry is still valid, so the foreground connect is a cache hit and + // never touches the store lock + Assert.assertEquals("Bearer ACCESS-SEED", + server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // A peer holds the identity's lock: stamped (so the empty-lock grace does not + // apply) and far inside the staleness window (so it is never stolen). Every later + // refresh can only poll for it. + Path lock = dir.resolve(key.hash() + ".lock"); + Files.write(lock, "live-peer-nonce".getBytes(StandardCharsets.UTF_8)); + + armed.set(true); + // let the seeded token fall inside its clock-skew margin, so the reconnect's pull + // has to refresh rather than serve the cache + // stale once now >= expiresAt - skew, i.e. after (remaining - ttl/2) + Thread.sleep(SEED_REMAINING_MILLIS - SEED_TTL_MILLIS / 2 + 500L); + + // one batch: the server acks it, then drops the socket -> foreground reconnect -> + // credential pull -> refresh -> blocked polling for the store lock + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + Assert.assertTrue("the reconnect must reach the credential pull", + blockedPullStarted.await(30, TimeUnit.SECONDS)); + + long startNanos = System.nanoTime(); + sender.close(); + closed = true; + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // the shutdown budget is 30s; anything near it means close() waited the store lock + // out instead of breaking it. A generous ceiling keeps this off the CI flake line + // while still failing the pre-fix behaviour by a wide margin. + Assert.assertTrue("close() must break the store-lock wait, not sit out the shutdown " + + "budget; took " + elapsedMillis + "ms", elapsedMillis < 15_000); + Assert.assertEquals("the pull never got past the store lock, so the IdP must not " + + "have been reached", 0, tokenEndpointCalls.get()); + } finally { + if (!closed) { + sender.close(); + } + } + } + } + } + }); + } + + @Test(timeout = 120_000) + public void testCloseStopsAnOrphanDrainerBlockedInTheBuiltInCredentialPull() throws Exception { + assertMemoryLeak(() -> { + // The orphan drainer's INITIAL connect -- the one it makes before any CursorWebSocketSendLoop + // exists, so before the loop's ConnectCancellation is in play. Its lever is a different one: + // BackgroundDrainerPool.close() ends its stop grace with executor.shutdownNow(), which + // interrupts the drainer thread. That interrupt only accomplishes anything if what the thread + // is blocked in honours it, and a credential pull sat in the token store's uninterruptible + // waits -- so the drainer sat out the store's whole 30s lock-acquire budget while close() gave + // up on it, leaving the orphan slot locked by a thread nobody was waiting for any more. + String sfDir = temp.getRoot().toPath().resolve("sf").toString(); + AtomicInteger tokenEndpointCalls = new AtomicInteger(); + try (MockOidcServer idp = new MockOidcServer((method, path, body) -> { + if (TOKEN_PATH.equals(path)) { + tokenEndpointCalls.incrementAndGet(); + } + return MockOidcServer.json(200, "{}"); + })) { + // Phase 1: a ghost sender leaves un-acked frames behind, so phase 2 has an orphan to adopt. + try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) { + silent.start(); + Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS)); + try (Sender ghost = Sender.fromConfig("ws::addr=localhost:" + silent.getPort() + + ";sf_dir=" + sfDir + ";sender_id=ghost;close_flush_timeout_millis=0;")) { + ghost.table("foo").longColumn("v", 7L).atNow(); + ghost.flush(); + } + } + ObjList orphans = OrphanScanner.scan(sfDir, "primary"); + Assert.assertEquals("phase 1 must leave exactly one orphan slot", 1, orphans.size()); + + Path dir = storeDir(); + Files.createDirectories(dir); + FileTokenStore store = new FileTokenStore(dir, 30_000, 600_000); + TokenStoreKey key = keyFor(idp); + // No access token, only a refresh token: adopt() keeps the refresh token and leaves the cache + // empty, so EVERY pull -- the foreground's and the drainer's initial one alike -- goes into + // inLock rather than hitting a cache. + store.save(key, new PersistedToken(null, null, "REFRESH-SEED", 0L, 0L)); + // the peer's live lock is in place BEFORE the sender is built, so the drainer's very first + // connect blocks + Files.write(dir.resolve(key.hash() + ".lock"), "live-peer-nonce".getBytes(StandardCharsets.UTF_8)); + + try (TestWebSocketServer server = new TestWebSocketServer(new SilentHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + CountDownLatch drainerPullStarted = new CountDownLatch(1); + try (OidcDeviceAuth auth = authFor(idp, store)) { + // ASYNC so build() itself does not sit in the blocked foreground pull and never hand + // back a sender to close; the orphan drainers still start inside build(). + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("primary") + .drainOrphans(true) + .initialConnectMode(Sender.InitialConnectMode.ASYNC) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + // positively confirm the DRAINER (not just the foreground loop) reaches + // the pull; without this the test could pass on a drainer that never + // started and prove nothing about the initial-connect path + if (Thread.currentThread().getName().contains("orphan-drainer")) { + drainerPullStarted.countDown(); + } + return auth.getToken(); + }) + .build(); + boolean closed = false; + try { + Assert.assertTrue("the orphan drainer must reach its initial credential pull", + drainerPullStarted.await(30, TimeUnit.SECONDS)); + // let it settle into the store's lock wait + Thread.sleep(500L); + + long startNanos = System.nanoTime(); + sender.close(); + closed = true; + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + Assert.assertTrue("close() must stop a drainer blocked in the built-in credential " + + "pull, not leave it to the store's 30s budget; took " + elapsedMillis + + "ms", elapsedMillis < 15_000); + } finally { + if (!closed) { + sender.close(); + } + } + } + Assert.assertEquals("the pull never got past the store lock, so the IdP must not have " + + "been reached", 0, tokenEndpointCalls.get()); + // The drainer must have released the orphan slot's lock on the way out: a slot still + // locked by an abandoned drainer thread cannot be adopted by anyone, which is the durable + // cost of close() giving up on it. + Assert.assertTrue("the abandoned orphan slot must be adoptable again after close()", + awaitSlotAdoptable(sfDir + "/ghost", 10_000)); + } + } + }); + } + + private static OidcDeviceAuth authFor(MockOidcServer idp, FileTokenStore store) { + return OidcDeviceAuth.builder() + .clientId("questdb") + .deviceAuthorizationEndpoint(idp.httpUrl(DEVICE_PATH)) + .tokenEndpoint(idp.httpUrl(TOKEN_PATH)) + .scope("openid") + .allowInsecureTransport(true) + .tokenStore(store) + .prompt(challenge -> { + }) + .build(); + } + + private static TokenStoreKey keyFor(MockOidcServer idp) { + return new TokenStoreKey( + "questdb", + idp.httpUrl(TOKEN_PATH), + idp.httpUrl(DEVICE_PATH), + "openid", + null, + false); + } + + private static PersistedToken seededEntry() { + // an access token the foreground connect can serve straight from cache, plus the refresh token that + // sends the next pull into inLock once it goes stale + return new PersistedToken("ACCESS-SEED", null, "REFRESH-SEED", + System.currentTimeMillis() + SEED_REMAINING_MILLIS, SEED_TTL_MILLIS); + } + + private static boolean awaitSlotAdoptable(String slotPath, long timeoutMillis) throws Exception { + // the slot lock is an flock held by the drainer's engine, so a fresh acquire succeeds only once the + // drainer has genuinely let go; SlotLock.acquire throws SlotLockContentionException while it has not + long deadline = System.currentTimeMillis() + timeoutMillis; + while (System.currentTimeMillis() < deadline) { + try (SlotLock probe = SlotLock.acquire(slotPath)) { + Assert.assertNotNull(probe); + return true; + } catch (SlotLockContentionException e) { + Thread.sleep(50); + } + } + return false; + } + + private Path storeDir() { + return temp.getRoot().toPath().resolve("oidc-tokens"); + } + + /** Never acks, so a sender's frames stay un-acked on disk. */ + private static final class SilentHandler implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + } + } + + /** + * Acks the first binary frame, then closes the socket — a deterministic drop that drives the foreground + * loop into a reconnect, and so into a fresh credential pull. + */ + private static final class DropAfterFirstAckHandler implements TestWebSocketServer.WebSocketServerHandler { + private final AtomicInteger received = new AtomicInteger(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + int n = received.incrementAndGet(); + try { + if (n == 1) { + client.sendBinary(okFrame(0L)); + client.close(); + } + } catch (IOException ignored) { + // best-effort: the connection died under us + } + } + + private static byte[] okFrame(long wireSeq) { + ByteBuffer bb = ByteBuffer.allocate(1 + 8 + 2).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(wireSeq); + bb.putShort((short) 0); + return bb.array(); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketTokenProviderTest.java new file mode 100644 index 000000000..093a8b650 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketTokenProviderTest.java @@ -0,0 +1,855 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.auth.OidcAuthException; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.HandOffCharSequence; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Base64; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Verifies that the WebSocket (QWP) transport accepts an + * {@link Sender.LineSenderBuilder#httpTokenProvider} and presents the provider's current token as the + * {@code Authorization: Bearer} header on every upgrade handshake - the initial connect and each + * reconnect - so a long-lived WebSocket sender follows token rotation the way the HTTP transport does. + * The provider is queried at handshake time, not per data frame, because an established WebSocket is + * not re-authenticated mid-stream. The fixed-token and username/password paths are covered too as a + * regression guard for the refactor that turned the captured header string into a per-handshake supplier. + *

+ * Each test runs under {@code assertMemoryLeak} so the sender's native buffers are proven freed on close. + */ +public class WebSocketTokenProviderTest { + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testProviderBufferMutatedDuringTheHandshakeCannotSplice() throws Exception { + assertMemoryLeak(() -> { + // Sender.buildWebSocketAuthHeader's supplier applies the same snapshot-before-validate rule as + // the ILP sender and the query client, and was the one of the three with no test. Without the + // snapshot validateToken scans the provider's live sequence and the "Bearer " concatenation + // materialises it again, so a buffer that changes between those two reads ships the mutated + // bytes - CR/LF included - into the upgrade request. + final String clean = "GOODTOKEN"; + final String spliced = "abc" + (char) 0x0d + (char) 0x0a + "X-Injected: pwned"; + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + return new HandOffCharSequence(clean, spliced); + }) + .build()) { + Assert.assertNotNull(sender); + Assert.assertEquals("the upgrade must carry the bytes that were validated, not a value " + + "swapped in after the scan", + "Bearer " + clean, server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertTrue("the provider must have been queried, or this test passes for the " + + "wrong reason", pulls.get() >= 1); + } + } + }); + } + + @Test + public void testCredentialKindTaggedForTheOrphanDrainerTerminalPolicy() throws Exception { + assertMemoryLeak(() -> { + // The builder routes a CONSTANT credential through QwpWebSocketSender.fixedAuthHeader + // and an httpTokenProvider through a bare lambda. That type difference is the whole + // signal: hasDynamicCredential() reads it, and BackgroundDrainer.connectWithDurableAckRetry + // decides on it whether a 401 during an orphan drain may quarantine the slot. + // + // A mis-tag is silent at build time and asymmetric in cost. Tagging a rotating credential + // as fixed makes the first 401 of an orphan drain drop a .failed sentinel that nothing in + // production clears, permanently abandoning replayable rows over a token the next pull + // would have refreshed. The other direction only delays the operator's signal: a wrong + // fixed password rides out the attempt threshold and the dwell floor before quarantining. + // + // Nothing connected the builder half to the drainer half, so assert both on a real built + // sender: the header the server actually received (a tag asserted alone would still pass + // if the credential reached the wire by some other route) and the tag itself, read both + // directly and through the background reconnect factory the drainer is handed. + // + // This is the ONLY test of the classification itself: the drainer's own suites + // (BackgroundDrainerDurableAckRetryTest, BackgroundDrainerMidDrainAuthRejectTest) stub + // hasDynamicCredential() on a scripted factory, because what they pin is the terminal policy + // each verdict produces. Delete this test and both verdicts become assumptions. + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + assertCredentialKind(server, port, "Bearer static-token", false, + b -> b.httpToken("static-token")); + assertCredentialKind(server, port, + "Basic " + Base64.getEncoder().encodeToString( + "user:pass".getBytes(StandardCharsets.UTF_8)), + false, + b -> b.httpUsernamePassword("user", "pass")); + assertCredentialKind(server, port, "Bearer rotating-token", true, + b -> b.httpTokenProvider(() -> "rotating-token")); + // No credential at all: nothing to refresh, so a rejection is never + // a window a later pull can close. + assertCredentialKind(server, port, "", false, b -> { + }); + } + }); + } + + @Test + public void testProviderRequeriedOnEveryReconnect() throws Exception { + assertMemoryLeak(() -> { + // The handler ACKs the first frame then drops the connection, forcing the I/O loop to reconnect. + // The reconnect runs the same buildAndConnect path, so it must re-query the provider and present + // the next token on the new upgrade - proving refresh-at-handshake, not a token captured once. + AtomicInteger tokenSeq = new AtomicInteger(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .httpTokenProvider(() -> "TOKEN-" + tokenSeq.incrementAndGet()) + .build()) { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // batch 1 lands, gets ACKed, then the server drops the socket -> reconnect + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + + // the reconnect handshake must carry a freshly pulled token (blocks for the reconnect) + Assert.assertEquals("Bearer TOKEN-2", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // batch 2 goes through on the new connection, end to end + sender.table("foo").longColumn("v", 2L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 5_000); + } + } + }); + } + + @Test + public void testProviderTokenSuppliedOnInitialUpgrade() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger tokenSeq = new AtomicInteger(); + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .httpTokenProvider(() -> "TOKEN-" + tokenSeq.incrementAndGet()) + .build()) { + // the upgrade handshake runs during build(); the provider was queried exactly once for it + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertEquals(1, tokenSeq.get()); + + // sending data must NOT re-query the provider: the established socket carries no new auth + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + Assert.assertEquals(1, tokenSeq.get()); + } + } + }); + } + + @Test + public void testStaticTokenStillSuppliedOverWebSocket() throws Exception { + assertMemoryLeak(() -> { + // regression guard for the supplier refactor: a fixed httpToken still reaches the upgrade header + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .httpToken("static-token") + .build()) { + Assert.assertEquals("Bearer static-token", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + } + } + }); + } + + @Test + public void testThrowingProviderResolvedOncePerConnectRound() throws Exception { + assertMemoryLeak(() -> { + // A token-provider failure (a failed silent refresh, or not signed in) is cluster-wide, not a + // per-endpoint transport fault. The credential is resolved once before the endpoint walk, so the + // provider is queried exactly once per connect round even across a multi-endpoint failover, and the + // provider's own error reaches the caller instead of being masked as "all endpoints unreachable". + AtomicInteger calls = new AtomicInteger(); + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + // two endpoints at the same reachable server (distinct host strings, so not rejected as + // duplicates) - a pre-fix per-endpoint pull would query the provider twice for one connect + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .address("127.0.0.1:" + port) + .httpTokenProvider(() -> { + calls.incrementAndGet(); + throw new OidcAuthException("no token has been obtained yet; call signIn()"); + }) + .build(); + Assert.fail("expected build() to fail when the token provider throws"); + } catch (OidcAuthException e) { + // the provider's own error surfaces directly, not wrapped as a transport failure + String msg = e.getMessage(); + Assert.assertTrue("expected the provider's message, got: " + msg, + msg.contains("no token has been obtained yet")); + Assert.assertFalse("a provider failure must not be mislabeled as unreachable, got: " + msg, + msg.contains("unreachable")); + } catch (Exception e) { + Assert.fail("expected the provider's OidcAuthException to surface, got: " + e); + } + // queried once per connect round, not once per endpoint (pre-fix this would be 2) + Assert.assertEquals(1, calls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testThrowingProviderFailsFastInSyncInitialConnect() throws Exception { + assertMemoryLeak(() -> { + // Setting any reconnect_* knob promotes the initial connect to SYNC mode (Sender.build). In SYNC + // mode a token-provider failure (not signed in / a failed refresh) must STILL fail fast with the + // provider's own exception - exactly like OFF mode - not be treated as a transport outage and + // retried for the whole reconnect budget (which would block build() for up to that budget, then + // surface a transport-shaped wrapper). A deterministic "no token" can never recover by retrying. + AtomicInteger calls = new AtomicInteger(); + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + long budgetMillis = 10_000; // if the fix regressed, build() would block ~this long before failing + long startNanos = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .reconnectMaxDurationMillis(budgetMillis) // -> SYNC initial connect + .httpTokenProvider(() -> { + calls.incrementAndGet(); + throw new OidcAuthException("no token has been obtained yet; call signIn()"); + }) + .build(); + Assert.fail("expected build() to fail when the token provider throws"); + } catch (OidcAuthException e) { + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // the provider's own error, surfaced fast - not a wrapped transport failure after the budget + String msg = e.getMessage(); + Assert.assertTrue("expected the provider's message, got: " + msg, + msg.contains("no token has been obtained yet")); + Assert.assertTrue("build() must fail fast, not burn the reconnect budget; took " + elapsedMillis + "ms", + elapsedMillis < budgetMillis / 2); + } catch (Exception e) { + Assert.fail("SYNC-mode credential failure must surface the provider's OidcAuthException, got: " + e); + } + // one deterministic failure, not a budget's worth of retries + Assert.assertEquals(1, calls.get()); + } + }); + } + + @Test + public void testThrowingProviderOnReconnectIsRetriedAndRecovers() throws Exception { + assertMemoryLeak(() -> { + // The riskiest token-provider path: a throw on the BACKGROUND I/O thread during a reconnect. The + // server ACKs the first frame then drops the socket, forcing a reconnect; on that reconnect the + // provider throws once (a transient failed silent refresh), then succeeds. connectWithRetry must + // catch the (non-terminal) throw and retry within the reconnect budget - re-querying the provider - + // so the sender recovers and batch 2 still lands, rather than the throw killing the I/O thread or + // being silently swallowed. A regression narrowing that catch (so the throw is not retried) fails here. + AtomicInteger calls = new AtomicInteger(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .httpTokenProvider(() -> { + int n = calls.incrementAndGet(); + // n==1 initial connect (ok); n==2 first reconnect attempt (transient throw); + // n>=3 reconnect retry (ok) + if (n == 2) { + throw new OidcAuthException("transient: a silent refresh failed"); + } + return "TOKEN-" + n; + }) + .build()) { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // batch 1 lands and is ACKed, then the server drops the socket -> background reconnect + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + + // the reconnect's first pull threw; connectWithRetry retries, re-querying the provider, and + // the retry's token reaches the upgrade (the throwing attempt never connected, so TOKEN-2 is + // never seen on the wire) + Assert.assertEquals("Bearer TOKEN-3", server.pollAuthorizationHeader(10, TimeUnit.SECONDS)); + + // batch 2 goes through on the recovered connection: the reconnect throw did not terminate it + sender.table("foo").longColumn("v", 2L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 10_000); + Assert.assertTrue("the provider must be re-queried on the reconnect retry (>=3 pulls), got " + calls.get(), + calls.get() >= 3); + } + } + }); + } + + @Test(timeout = 60_000) + public void testCloseBreaksADrainerBlockedInACredentialPull() throws Exception { + assertMemoryLeak(() -> { + // The reconnect walk publishes the WebSocketClient it is about to block on so close() can break it + // (ConnectCancellation), but the credential pull that now precedes the walk is caller code owning + // no socket, so closeTraffic() cannot reach it. A pull can outlast close()'s 30s shutdown budget - + // OidcDeviceAuth.getToken() waits up to 6 x httpTimeoutMillis behind a peer's silent refresh - and + // during an IdP outage the drainer sits inside a pull for most of every retry cycle, so close() + // lands there routinely. Before the fix close() burned the whole budget and then threw + // "cursor I/O thread did not stop", delegating teardown, on what is a clean shutdown. + CountDownLatch pullEntered = new CountDownLatch(1); + CountDownLatch neverReleased = new CountDownLatch(1); + AtomicBoolean blockNextPull = new AtomicBoolean(false); + AtomicBoolean sawInterrupt = new AtomicBoolean(false); + AtomicInteger calls = new AtomicInteger(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + int n = calls.incrementAndGet(); + if (blockNextPull.get()) { + pullEntered.countDown(); + try { + // only an interrupt can free this, exactly like OidcDeviceAuth's timed + // wait for its instance lock behind a peer's silent refresh + neverReleased.await(45, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + sawInterrupt.set(true); + throw new OidcAuthException("interrupted while waiting for a token"); + } + } + return "TOKEN-" + n; + }) + .build(); + boolean closed = false; + try { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // arm the block, then let the server's drop drive the background reconnect into the pull + blockNextPull.set(true); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + Assert.assertTrue("the drainer must reach the credential pull", + pullEntered.await(15, TimeUnit.SECONDS)); + + long startNanos = System.nanoTime(); + sender.close(); + closed = true; + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + // the budget is 30s; anything near it means close() waited it out instead of breaking + // the pull. A generous ceiling keeps this off the CI flake line while still failing the + // pre-fix behaviour by a wide margin. + Assert.assertTrue("close() must break the pull, not wait out the shutdown budget; took " + + elapsedMillis + "ms", elapsedMillis < 15_000); + Assert.assertTrue("close() must interrupt the thread parked in the pull", sawInterrupt.get()); + } finally { + if (!closed) { + neverReleased.countDown(); + sender.close(); + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testPersistentCredentialOutageIsReportedToTheErrorHandler() throws Exception { + assertMemoryLeak(() -> { + // Retrying a credential outage forever (Invariant B) must not make it programmatically INVISIBLE. + // A revoked refresh token or a permanently dead IdP is not self-healing, yet the drainer keeps + // retrying and flush() keeps returning success while SF absorbs the rows; without a dispatched + // SenderError the only signal is a throttled slf4j WARN - and this library ships embedded, often + // with no binding configured - until SF fills and the failure resurfaces as ring backpressure, + // pointing the operator at disk sizing instead of at their credentials. The auth/upgrade and + // durable-ack policy failures already dispatch a RETRIABLE error for exactly this reason; the + // credential arm did not. RETRIABLE, not TERMINAL: the handler learns the wire is down while the + // producer stays alive and no data is at risk. + AtomicBoolean providerFailing = new AtomicBoolean(false); + AtomicInteger calls = new AtomicInteger(); + AtomicReference credentialError = new AtomicReference<>(); + AtomicReference terminalError = new AtomicReference<>(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .errorHandler(e -> { + if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { + terminalError.compareAndSet(null, e); + } else if (e.getServerMessage() != null + && e.getServerMessage().contains("credential-unavailable")) { + credentialError.compareAndSet(null, e); + } + }) + .httpTokenProvider(() -> { + int n = calls.incrementAndGet(); + if (providerFailing.get()) { + throw new OidcAuthException("persistent: not signed in"); + } + return "TOKEN-" + n; + }) + .build()) { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // arm the outage before the drop, so every reconnect pull throws + providerFailing.set(true); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 1, 5_000); + + // the handler must be told, by category and by message, that the CREDENTIAL is the problem + waitFor(() -> credentialError.get() != null, 15_000); + SenderError err = credentialError.get(); + Assert.assertEquals(SenderError.Category.SECURITY_ERROR, err.getCategory()); + Assert.assertEquals(SenderError.Policy.RETRIABLE, err.getAppliedPolicy()); + Assert.assertTrue("the provider's own message must reach the handler: " + err.getServerMessage(), + err.getServerMessage().contains("not signed in")); + + // and it stays RETRIABLE: no terminal, and the producer is still alive + Assert.assertNull("a credential outage must never latch a terminal", terminalError.get()); + sender.table("foo").longColumn("v", 2L).atNow(); + + // the provider recovers -> the next reconnect succeeds and the buffered rows drain + providerFailing.set(false); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 15_000); + } + } + }); + } + + @Test(timeout = 60_000) + public void testPersistentlyThrowingProviderOnReconnectDoesNotTerminateAndRecovers() throws Exception { + assertMemoryLeak(() -> { + // Invariant B: the RUNNING store-and-forward drainer must NEVER terminate on a token-provider + // failure, however long it persists. A failing provider (IdP unreachable, a silent refresh failing, + // sign-in not yet complete) is a transient outage like any other - the un-acked rows stay safe in SF + // and the sender recovers once a token is available again. Here the provider throws for FAR longer + // than the (deliberately short) reconnect budget: the sender must stay alive the whole time - no + // terminal, no exception surfaced to the producer - and then ship the buffered row once the provider + // recovers. Before the fix the drainer latched a TERMINAL SECURITY_ERROR at reconnectMaxDurationMillis + // and dropped the producer store-and-forward had promised to keep alive. + AtomicBoolean providerFailing = new AtomicBoolean(false); + AtomicInteger calls = new AtomicInteger(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + long budgetMillis = 300; // SHORT: the outage below far exceeds it, proving the budget is not consulted + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .reconnectMaxDurationMillis(budgetMillis) + .httpTokenProvider(() -> { + int n = calls.incrementAndGet(); + if (providerFailing.get()) { + throw new OidcAuthException("persistent: not signed in"); + } + return "TOKEN-" + n; + }) + .build()) { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // Arm the provider failure on the established connection, BEFORE the drop triggers a reconnect, + // so every reconnect pull throws (no race where the first reconnect succeeds first). + providerFailing.set(true); + // batch 1 lands and is ACKed on the initial connection, then the server drops the socket -> + // the background reconnect loop starts and every provider pull now throws + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 1, 5_000); + + // let the failing reconnect run for 4x the budget - the old code would have terminated at 1x + int callsAtOutageStart = calls.get(); + Thread.sleep(budgetMillis * 4); + + // the drainer kept re-querying the provider (retrying, not giving up) ... + Assert.assertTrue("the provider must be re-queried during the outage, got " + calls.get(), + calls.get() > callsAtOutageStart); + // ... and the sender is still ALIVE: buffering another row must not surface a terminal even + // though every reconnect is currently failing. Before the fix this threw a SECURITY_ERROR + // "token-provider-failed" once the budget elapsed. + try { + sender.table("foo").longColumn("v", 2L).atNow(); + } catch (Exception e) { + Assert.fail("the drainer terminated the sender on a transient provider outage: " + e.getMessage()); + } + + // the provider recovers -> the next reconnect succeeds and the buffered row drains + providerFailing.set(false); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 15_000); + } + } + }); + } + + @Test(timeout = 60_000) + public void testRepeated401sWithDynamicTokenDoNotTerminateLiveSenderAndRecover() throws Exception { + assertMemoryLeak(() -> { + // This composes the production foreground path all the way from a dynamic HTTP token provider, + // through real 401 upgrade responses, to the live store-and-forward engine. Unit tests of the + // orphan drainer's dynamic-credential policy cannot catch the running sender accidentally adopting + // the orphan-only quarantine policy: that would latch TERMINAL/DATA_LOSS, drop .failed, and kill the + // producer once the ordinary reconnect budget elapsed. + final long budgetMillis = 300; + final String senderId = "live-dynamic-401"; + AtomicInteger tokenPulls = new AtomicInteger(); + AtomicReference terminalOrDataLoss = new AtomicReference<>(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + Path sfDir = temp.newFolder("live-dynamic-401-sf").toPath(); + Path failedSentinel = sfDir.resolve(senderId).resolve(".failed"); + + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir.toString()) + .senderId(senderId) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .reconnectMaxDurationMillis(budgetMillis) + .errorHandler(error -> { + if (error.getAppliedPolicy() == SenderError.Policy.TERMINAL + || error.getCategory() == SenderError.Category.DATA_LOSS) { + terminalOrDataLoss.compareAndSet(null, error); + } + }) + .httpTokenProvider(() -> "TOKEN-" + tokenPulls.incrementAndGet()) + .build()) { + try { + Assert.assertEquals("Bearer TOKEN-1", + server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // The existing connection still accepts and ACKs batch 1, then drops. Every reconnect + // from that point receives a genuine HTTP 401 after pulling a fresh token. + server.setRejectWithStatus(401, "Unauthorized"); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 1, 5_000); + + // Do not start the dwell witness from the original connection's frame: its handler + // records that frame before ACKing and closing the socket, so a delayed close could + // consume most of the sleep before the first 401 even occurred. Two fully written + // rejects prove the reconnect loop processed the first 401 and began another attempt. + waitFor(() -> server.statusRejectCount() >= 2, 5_000); + int pullsAtOutageStart = tokenPulls.get(); + int rejectsAtOutageStart = server.statusRejectCount(); + Thread.sleep(budgetMillis * 4); + Assert.assertTrue("the live sender must keep retrying 401s beyond its reconnect budget", + tokenPulls.get() > pullsAtOutageStart); + Assert.assertTrue("the server must keep returning 401s throughout the extended outage", + server.statusRejectCount() > rejectsAtOutageStart); + Assert.assertNull("dynamic-token 401s must not become TERMINAL or DATA_LOSS", + terminalOrDataLoss.get()); + Assert.assertFalse("a live dynamic-token outage must not quarantine the active slot", + Files.exists(failedSentinel)); + + // A producer call made while the 401 outage is still active must remain usable. Once the + // server accepts the next rotated token, this buffered batch must drain normally. + sender.table("foo").longColumn("v", 2L).atNow(); + sender.flush(); + Assert.assertNull("the producer must survive the extended 401 outage", + terminalOrDataLoss.get()); + + server.setRejectWithStatus(0, null); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 15_000); + Assert.assertNull("recovery must not leave a terminal or data-loss report", + terminalOrDataLoss.get()); + Assert.assertFalse("recovery must leave no .failed sentinel", + Files.exists(failedSentinel)); + } finally { + // Let sender.close() reconnect and finish its cleanup even when an assertion above fails. + server.setRejectWithStatus(0, null); + } + } + + Assert.assertNull("close must not synthesize a terminal or data-loss report", + terminalOrDataLoss.get()); + Assert.assertFalse("the recovered sender must close without a .failed sentinel", + Files.exists(failedSentinel)); + } + }); + } + + @Test(timeout = 60_000) + public void testCredentialFailuresInterleavedWithRoleRejectsDoNotTerminateAndRecover() throws Exception { + assertMemoryLeak(() -> { + // Neither a token-provider failure nor a transient role reject may terminate the running drainer, and + // interleaving them must not either: both fall through to capped backoff and retry indefinitely + // (Invariant B). Here credential blips (even provider calls throw) alternate with 421 role rejects + // (odd calls return a token whose upgrade the server rejects) for far longer than the reconnect + // budget; the sender must survive the whole span and then ship batch 2 once both faults clear. Before + // the fix the credential blips accumulated to a budget-latched terminal and the sender was dropped. + AtomicInteger calls = new AtomicInteger(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + long budgetMillis = 300; // short: the interleaved outage below far exceeds it + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .reconnectInitialBackoffMillis(20) + .reconnectMaxBackoffMillis(20) + .reconnectMaxDurationMillis(budgetMillis) + .httpTokenProvider(() -> { + int n = calls.incrementAndGet(); + // call 1: the initial connect (must succeed). Then alternate on every reconnect + // attempt: even calls THROW (a credential blip), odd calls RETURN a token whose + // connect then hits the 421 role reject below. So credential failures and role + // rejects strictly alternate across the whole outage. + if (n > 1 && n % 2 == 0) { + throw new OidcAuthException("transient: a silent refresh failed"); + } + return "TOKEN-" + n; + }) + .build()) { + Assert.assertEquals("Bearer TOKEN-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + // reject every NEW handshake with a transient 421 role reject BEFORE the drop, so a + // token-returning reconnect attempt deterministically hits it. The already-established + // initial connection is unaffected and still ships batch 1. + server.setRejectWithRole("replica"); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 1, 5_000); + + // let the interleaved credential + role failures run for well over the budget; the sender + // must NOT terminate on either fault class or their interleaving + Thread.sleep(budgetMillis * 4); + try { + sender.table("foo").longColumn("v", 2L).atNow(); + } catch (Exception e) { + Assert.fail("the drainer terminated during interleaved credential/role failures: " + e.getMessage()); + } + + // clear the reject: the next token-returning reconnect now succeeds and batch 2 drains + server.setRejectWithRole(null); + sender.flush(); + waitFor(() -> handler.totalBinaryReceived.get() >= 2, 15_000); + Assert.assertTrue("the provider must have been re-queried across the reconnect phase, got " + calls.get(), + calls.get() >= 4); + } + } + }); + } + + @Test + public void testUsernamePasswordStillSuppliedOverWebSocket() throws Exception { + assertMemoryLeak(() -> { + // regression guard for the supplier refactor: username/password still becomes the Basic header + try (TestWebSocketServer server = new TestWebSocketServer(new AckHandler())) { + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port) + .httpUsernamePassword("user", "pass") + .build()) { + String expected = "Basic " + Base64.getEncoder().encodeToString( + "user:pass".getBytes(StandardCharsets.UTF_8)); + Assert.assertEquals(expected, server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + sender.table("foo").longColumn("v", 1L).atNow(); + sender.flush(); + } + } + }); + } + + // Mirrors WebSocketResponse STATUS_OK layout: status u8 | sequence u64 | table_count u16 + private static byte[] buildAck(long seq) { + byte[] buf = new byte[1 + 8 + 2]; + ByteBuffer bb = ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(seq); + bb.putShort((short) 0); + return buf; + } + + private static void waitFor(BoolCondition cond, long timeoutMillis) { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (System.currentTimeMillis() < deadline) { + if (cond.test()) return; + try { + Thread.sleep(20); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + Assert.fail("interrupted"); + } + } + Assert.fail("waitFor timed out after " + timeoutMillis + "ms"); + } + + @FunctionalInterface + private interface BoolCondition { + boolean test(); + } + + /** ACKs every binary frame so the sender doesn't hang. */ + private static class AckHandler implements TestWebSocketServer.WebSocketServerHandler { + private final AtomicLong nextSeq = new AtomicLong(0); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + try { + client.sendBinary(buildAck(nextSeq.getAndIncrement())); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } + + /** + * ACKs every binary frame; on the first connection's first frame it closes the socket right after + * the ACK, so the sender's I/O loop must reconnect to deliver the next batch. Later connections ACK + * normally. + */ + private static class DropAfterFirstAckHandler implements TestWebSocketServer.WebSocketServerHandler { + final AtomicInteger connectionsAccepted = new AtomicInteger(); + final AtomicLong totalBinaryReceived = new AtomicLong(); + private final AtomicLong nextSeq = new AtomicLong(0); + private TestWebSocketServer.ClientHandler firstClient; + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (firstClient == null || firstClient != client) { + connectionsAccepted.incrementAndGet(); + if (firstClient == null) { + firstClient = client; + } + } + totalBinaryReceived.incrementAndGet(); + try { + client.sendBinary(buildAck(nextSeq.getAndIncrement())); + if (totalBinaryReceived.get() == 1) { + // brief sleep so the queued ACK flushes before we close the socket under it + Thread.sleep(50); + client.close(); + } + } catch (IOException | InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + } + } + + private static void assertCredentialKind( + TestWebSocketServer server, + int port, + String expectedHeader, + boolean expectedDynamic, + Consumer credential + ) throws Exception { + Sender.LineSenderBuilder builder = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + port); + credential.accept(builder); + try (Sender sender = builder.build()) { + Assert.assertEquals("the configured credential must reach the upgrade header", + expectedHeader, server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + QwpWebSocketSender qwp = (QwpWebSocketSender) sender; + Assert.assertEquals("credential tag for [" + expectedHeader + "]", + expectedDynamic, qwp.isCredentialDynamic()); + // The value BackgroundDrainer.connectWithDurableAckRetry actually reads: + // ReconnectFactory.hasDynamicCredential() on the background factory an + // orphan drainer is handed. + Assert.assertEquals("drainer-visible credential tag for [" + expectedHeader + "]", + expectedDynamic, + qwp.newBackgroundReconnectFactory(() -> false).hasDynamicCredential()); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerCredentialOutageReportTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerCredentialOutageReportTest.java new file mode 100644 index 000000000..7cbffe821 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerCredentialOutageReportTest.java @@ -0,0 +1,418 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.http.client.WebSocketClient; +import io.questdb.client.cutlass.http.client.WebSocketClientFactory; +import io.questdb.client.cutlass.qwp.client.QwpCredentialUnavailableException; +import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.std.Files; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Credential-outage observability for an orphan {@link BackgroundDrainer}. + *

+ * A credential the client cannot ACQUIRE — the configured token provider throws + * instead of returning one, after a revocation, an IdP outage, or a sign-in the + * user has not finished — is retried indefinitely under Invariant B, exactly like + * a transport outage: the un-acked rows stay safe in store-and-forward and no + * {@code .failed} sentinel is dropped. Retrying forever is correct; retrying + * SILENTLY is not. A revoked refresh token does not heal on its own, so with no + * report the outage is invisible until SF fills and resurfaces as ring + * backpressure, which points the operator at disk sizing instead of at their + * credentials. + *

+ * The foreground sender already reports it (see + * {@code WebSocketTokenProviderTest#testPersistentCredentialOutageIsReportedToTheErrorHandler}). + * An orphan drainer rides out the very same fault and had neither half: + *

    + *
  • its drain loop's {@code SenderError} dispatcher was never wired, so the + * loop's own {@code credential-unavailable} report was dropped into a null;
    • + *
  • at initial connect the exception matched none of the typed arms and landed + * in the generic transport arm, whose WARN says "cluster unreachable" — the + * wrong condition, sending the operator after a network fault that does not + * exist.
    • + *
+ * Both halves are pinned here. + *

+ * Wire realism matches {@link BackgroundDrainerMidDrainAuthRejectTest}: a real + * {@link TestWebSocketServer} durably acks over a live socket while the scripted + * {@link CursorWebSocketSendLoop.ReconnectFactory} decides, per connect attempt, + * whether the sweep produces a client or fails to obtain a credential. + */ +public class BackgroundDrainerCredentialOutageReportTest { + + private static final long FAST_BACKOFF_MAX_MILLIS = 4L; + private static final long FAST_BACKOFF_MILLIS = 1L; + private static final String PROVIDER_FAILURE_MESSAGE = "refresh token revoked by the IdP"; + private static final long RECONNECT_MAX_DURATION_MILLIS = 25L; + private static final int SEEDED_FRAMES = 5; + private static final long SEGMENT_SIZE_BYTES = 16_384L; + private static final long SF_MAX_TOTAL_BYTES = 1L << 20; + private static final String TABLE = "trades"; + + private String slotPath; + + // one shared temp-directory mechanism instead of a per-class java.io.tmpdir path plus a hand-rolled + // recursive delete: the rule cleans up on failure and on an exception thrown out of a test too + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Before + public void setUp() { + slotPath = temp.getRoot().toPath().resolve("slot").toString(); + assertEquals("mkdir slot dir", 0, Files.mkdir(slotPath, Files.DIR_MODE_DEFAULT)); + } + + + @Test + public void testInitialConnectCredentialOutageIsNamedNotMislabelledUnreachable() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // The drain loop does not exist yet at initial connect, so the sink cannot + // carry this one -- the log is the only diagnostic, which makes naming the + // condition the whole of the fix. "cluster unreachable" is actively + // misleading here: nothing was attempted on the wire at all. + seedSlot(SEEDED_FRAMES); + AckAllHandler handler = new AckAllHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Calls 1-2: the token provider throws. Call 3+: it hands over a token. + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), 1, 2); + BackgroundDrainer drainer = newDrainer(factory); + + ch.qos.logback.classic.Logger drainerLogger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(BackgroundDrainer.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = drainerLogger.getLevel(); + drainerLogger.setLevel(ch.qos.logback.classic.Level.ALL); + drainerLogger.addAppender(appender); + try { + runToCompletion(drainer); + } finally { + drainerLogger.detachAppender(appender); + drainerLogger.setLevel(savedLevel); + appender.stop(); + } + + // Invariant B: a credential outage is transient, so it is ridden out -- + // never quarantined, and the drain completes once a token appears. + assertEquals(BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("a credential outage must never quarantine the slot", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("the drainer must have ridden out both outage sweeps, attempts=" + + factory.attempts(), factory.attempts() >= 3); + + boolean named = false; + boolean mislabelled = false; + for (ch.qos.logback.classic.spi.ILoggingEvent e : appender.list) { + String msg = e.getFormattedMessage(); + if (msg.contains("token provider failed to supply a credential") + && msg.contains(PROVIDER_FAILURE_MESSAGE)) { + named = true; + } + if (msg.contains("cluster unreachable")) { + mislabelled = true; + } + } + assertTrue("a credential outage at initial connect must name itself in the log -- " + + "it is the only diagnostic that path produces. Saw: " + appender.list, named); + assertFalse("a credential outage must not be reported as a network fault. Saw: " + + appender.list, mislabelled); + } + }); + } + + @Test + public void testMidDrainCredentialOutageReachesTheErrorSink() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // The wire drops after one durable ack; the loop's own reconnect sweeps then + // fail to obtain a credential. The loop rides that out itself (it never + // reaches the drainer's connect path), so its dispatcher is the ONLY route + // to the sink -- and it was never wired on an orphan drainer. + seedSlot(SEEDED_FRAMES); + DropFirstHandler handler = new DropFirstHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Call 1: healthy connect, drain starts. Calls 2-4: the mid-drain + // reconnect cannot obtain a credential. Call 5+: a token is available + // again and the drain finishes. + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), 2, 4); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + assertEquals("a credential outage must be ridden out, not quarantined", + BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("no .failed sentinel after a drain that recovered", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + + SenderError credentialError = null; + for (SenderError e : captured) { + if (e.getServerMessage() != null + && e.getServerMessage().contains("credential-unavailable")) { + credentialError = e; + break; + } + } + assertTrue("the credential outage must reach the drainer's error sink -- without it " + + "the only signal is a throttled slf4j WARN, a NOP in an app with no " + + "binding configured. Saw: " + captured, + credentialError != null); + assertEquals(SenderError.Category.SECURITY_ERROR, credentialError.getCategory()); + // RETRIABLE, not TERMINAL: the rows are safe in SF and the drain recovers. + assertEquals(SenderError.Policy.RETRIABLE, credentialError.getAppliedPolicy()); + assertTrue("the provider's own failure must be carried through: " + + credentialError.getServerMessage(), + credentialError.getServerMessage().contains(PROVIDER_FAILURE_MESSAGE)); + assertEquals("orphan-engine FSNs must not be exposed through the live sender's handler", + SenderError.NO_MESSAGE_SEQUENCE, credentialError.getFromFsn()); + assertEquals("orphan-engine FSNs must not be exposed through the live sender's handler", + SenderError.NO_MESSAGE_SEQUENCE, credentialError.getToFsn()); + + for (SenderError e : captured) { + assertFalse("a drain that recovered must report no data loss: " + captured, + e.getCategory() == SenderError.Category.DATA_LOSS); + // An ORPHAN loop latches TERMINAL only to hand the slot back to this + // drainer, which then decides. Forwarding it would announce a dead + // producer for a fault the very next sweep clears. + assertFalse("the loop's hand-back terminal must not reach the sink: " + captured, + e.getAppliedPolicy() == SenderError.Policy.TERMINAL); + } + } + }); + } + + private static byte[] durableAckFrame(long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x02); // STATUS_DURABLE_ACK + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + + private static byte[] okFrame(long wireSeq, long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 8 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(wireSeq); + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + + private BackgroundDrainer newDrainer(ScriptedWireFactory factory) { + return new BackgroundDrainer( + slotPath, + SEGMENT_SIZE_BYTES, + SF_MAX_TOTAL_BYTES, + factory, + RECONNECT_MAX_DURATION_MILLIS, + FAST_BACKOFF_MILLIS, + FAST_BACKOFF_MAX_MILLIS, + /* requestDurableAck */ true, + /* durableAckKeepaliveIntervalMillis */ 200L); + } + + + private static void runToCompletion(BackgroundDrainer drainer) throws InterruptedException { + Thread t = new Thread(drainer, "test-credential-outage-drainer"); + t.setDaemon(true); + t.start(); + t.join(20_000); + if (t.isAlive()) { + drainer.requestStop(); + t.join(5_000); + fail("drainer did not finish within 20s (outcome=" + drainer.outcome() + ")"); + } + } + + private void seedSlot(int frames) { + try (CursorSendEngine engine = new CursorSendEngine(slotPath, SEGMENT_SIZE_BYTES)) { + long buf = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT); + try { + byte[] payload = "frame-bytes-padd".getBytes(StandardCharsets.US_ASCII); + for (int i = 0; i < payload.length; i++) { + Unsafe.getUnsafe().putByte(buf + i, payload[i]); + } + for (int i = 0; i < frames; i++) { + engine.appendBlocking(buf, 16); + } + } finally { + Unsafe.free(buf, 16, MemoryTag.NATIVE_DEFAULT); + } + } + } + + /** + * Durably acks everything on every connection — the wire is never the fault + * under test here, only the credential the client cannot obtain to open it. + */ + private static final class AckAllHandler implements TestWebSocketServer.WebSocketServerHandler { + private final java.util.Map wireSeqByConn = + new java.util.IdentityHashMap<>(); + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long[] counter = wireSeqByConn.computeIfAbsent(client, k -> new long[1]); + long seq = counter[0]++; + try { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } catch (IOException ignored) { + // Best-effort ack: the connection died under us; the client replays. + } + } + } + + /** + * Server-side script. Connection #1 durably acks exactly one frame, then closes + * the socket — a deterministic mid-drain wire drop that forces the loop's own + * reconnect sweep. Every later connection acks all traffic, so a reconnected + * loop drains to completion. + */ + private static final class DropFirstHandler implements TestWebSocketServer.WebSocketServerHandler { + private final List arrivalOrder = new ArrayList<>(); + private final java.util.Map wireSeqByConn = + new java.util.IdentityHashMap<>(); + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long[] counter = wireSeqByConn.get(client); + if (counter == null) { + counter = new long[1]; + wireSeqByConn.put(client, counter); + arrivalOrder.add(client); + } + int connectionIndex = arrivalOrder.indexOf(client) + 1; + long seq = counter[0]++; + try { + if (connectionIndex == 1) { + if (seq == 0) { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } else if (seq == 1) { + client.close(); // mid-drain wire drop + } + // seq > 1: late buffered frames from the condemned connection; ignore. + } else { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } + } catch (IOException ignored) { + // Best-effort ack: the connection died under us; the client replays. + } + } + } + + /** + * Per-call-index scripted factory over a real wire. Call indexes inside + * {@code [throwFrom, throwTo]} (1-based, inclusive) fail to obtain a credential — + * a {@link QwpCredentialUnavailableException} wrapping the provider's own + * exception, exactly as {@code QwpWebSocketSender} wraps a throwing + * {@code httpTokenProvider}. Every other call returns a live upgraded client. + */ + private static final class ScriptedWireFactory implements CursorWebSocketSendLoop.ReconnectFactory { + private final AtomicInteger calls = new AtomicInteger(); + private final int port; + private final int throwFrom; + private final int throwTo; + + ScriptedWireFactory(int port, int throwFrom, int throwTo) { + this.port = port; + this.throwFrom = throwFrom; + this.throwTo = throwTo; + } + + int attempts() { + return calls.get(); + } + + @Override + public boolean hasDynamicCredential() { + // A token provider is by definition a rotating credential. + return true; + } + + @Override + public WebSocketClient reconnect() throws Exception { + int n = calls.incrementAndGet(); + if (n >= throwFrom && n <= throwTo) { + throw new QwpCredentialUnavailableException( + new RuntimeException(PROVIDER_FAILURE_MESSAGE)); + } + WebSocketClient c = WebSocketClientFactory.newPlainTextInstance(); + try { + c.setQwpMaxVersion(1); + c.setQwpRequestDurableAck(true); + c.setConnectTimeout(5_000); + c.connect("localhost", port); + c.upgrade("/write/v4", 5_000, null); + } catch (Throwable t) { + c.close(); + throw t; + } + return c; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerDurableAckRetryTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerDurableAckRetryTest.java index 4dde3f9e0..fb6bbfb23 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerDurableAckRetryTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerDurableAckRetryTest.java @@ -30,6 +30,8 @@ import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; import io.questdb.client.network.PlainSocketFactory; import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.std.Os; import io.questdb.client.cutlass.qwp.client.QwpDurableAckMismatchException; import io.questdb.client.cutlass.qwp.client.QwpIngressRoleRejectedException; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; @@ -41,9 +43,10 @@ import org.junit.After; import org.junit.Assert; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; -import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -54,6 +57,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; @@ -90,10 +94,14 @@ public class BackgroundDrainerDurableAckRetryTest { private String slotPath; + // one shared temp-directory mechanism instead of a per-class java.io.tmpdir path plus a hand-rolled + // recursive delete: the rule cleans up on failure and on an exception thrown out of a test too + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + @Before public void setUp() { - slotPath = Paths.get(System.getProperty("java.io.tmpdir"), - "qdb-da-retry-" + System.nanoTime()).toString(); + slotPath = temp.getRoot().toPath().resolve("slot").toString(); assertEquals("mkdir slot dir", 0, Files.mkdir(slotPath, Files.DIR_MODE_DEFAULT)); } @@ -101,28 +109,12 @@ public void setUp() { public void tearDown() { // Safety net for exits that bypass the assertMemoryLeak wrapper; // normally a no-op because the wrapper's finally already closed - // and cleared the stubs (close() is idempotent). + // and cleared the stubs (close() is idempotent). The slot directory + // itself is the TemporaryFolder rule's job. closeAllStubs(); - if (slotPath == null) return; - long find = Files.findFirst(slotPath); - if (find > 0) { - try { - int rc = 1; - while (rc > 0) { - String name = Files.utf8ToString(Files.findName(find)); - if (name != null && !".".equals(name) && !"..".equals(name)) { - Files.remove(slotPath + "/" + name); - } - rc = Files.findNext(find); - } - } finally { - Files.findClose(find); - } - } - Files.remove(slotPath); } - @Test + @Test(timeout = 60_000) public void testCallbackArgumentsCarrySlotPathAndAttemptNumber() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -141,7 +133,7 @@ public void testCallbackArgumentsCarrySlotPathAndAttemptNumber() throws Exceptio }); } - @Test + @Test(timeout = 60_000) public void testEscalatesAfterMaxAttemptsAndDropsSentinel() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -167,7 +159,7 @@ public void testEscalatesAfterMaxAttemptsAndDropsSentinel() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testListenerThrowingOnPersistentFailureStillMarksFailed() throws Exception { assertMemoryLeak(() -> { BackgroundDrainerListener throwing = new BackgroundDrainerListener() { @@ -193,7 +185,7 @@ public void onDurableAckUnavailable(String slotPath, int attemptNumber) { }); } - @Test + @Test(timeout = 60_000) public void testListenerThrowingOnUnavailableContinuesRetrying() throws Exception { assertMemoryLeak(() -> { AtomicInteger unavailableCalls = new AtomicInteger(); @@ -222,7 +214,7 @@ public void onDurableAckUnavailable(String slotPath, int attemptNumber) { }); } - @Test + @Test(timeout = 60_000) public void testNoListenerNoNullPointerOnEscalation() throws Exception { assertMemoryLeak(() -> { ScriptedFactory factory = ScriptedFactory.alwaysFailing( @@ -236,7 +228,7 @@ public void testNoListenerNoNullPointerOnEscalation() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testTerminalUpgradeMarksFailedImmediately() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -274,7 +266,502 @@ public void testTerminalUpgradeMarksFailedImmediately() throws Exception { }); } - @Test + @Test(timeout = 60_000) + public void testTerminalUpgradeWithDynamicCredentialStillQuarantinesImmediately() throws Exception { + assertMemoryLeak(() -> { + // A non-421 upgrade reject is terminal whatever the credential's nature: waiting cannot change a + // 5xx handshake refusal, so the drainer must quarantine on the first attempt exactly as it does + // without a dynamic credential (testTerminalUpgradeMarksFailedImmediately). The rotating-credential + // ride-out is gated on `e instanceof QwpAuthFailedException`, NOT on hasDynamicCredential() alone - + // WebSocketUpgradeException shares the same catch arm. Dropping that conjunct would route this + // upgrade reject into the ride-out and retry a genuine terminal up to + // MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS_PER_EPISODE (256) times before escalating, spending the + // settle budget on data the handshake will never accept. + ScriptedFactory factory = ScriptedFactory + .alwaysFailing(() -> new WebSocketUpgradeException(500, null, "server error during upgrade")) + .withDynamicCredential(); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + WebSocketClient out = drainer.connectWithDurableAckRetry(); + + assertNull(out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertEquals("a non-421 upgrade reject is terminal whatever the credential kind - the ride-out is " + + "keyed on QwpAuthFailedException, not on hasDynamicCredential() alone", + 1, factory.attempts()); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + }); + } + + @Test(timeout = 60_000) + public void testFixedCredentialAuthRejectionStillQuarantinesImmediately() throws Exception { + assertMemoryLeak(() -> { + // A 401 against a CONSTANT credential is a permanent misconfiguration: re-presenting the same + // header cannot change the answer, so the pre-existing fail-fast quarantine must stay exactly as + // it was. This is the control for the rotating-credential case below - the settle budget must + // key off the credential's nature, not relax auth handling across the board. + ScriptedFactory factory = ScriptedFactory.alwaysFailing( + () -> new QwpAuthFailedException(401, "127.0.0.1", 9000)); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + WebSocketClient out = drainer.connectWithDurableAckRetry(); + + assertNull(out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertEquals("a constant credential must not be retried", 1, factory.attempts()); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + }); + } + + @Test(timeout = 60_000) + public void testRotatingCredentialAuthRejectionQuarantinesOnceBudgetExhausted() throws Exception { + assertMemoryLeak(() -> { + // The settle budget must be BOUNDED, not "retry forever". Quarantine is permitted only once both + // the attempt threshold and the wall-clock dwell floor are exhausted; this short test budget pins + // the terminal behavior without waiting for the production five-minute default. + ScriptedFactory factory = ScriptedFactory + .alwaysFailing(() -> new QwpAuthFailedException(401, "127.0.0.1", 9000)) + .withDynamicCredential(); + // Far above what the six attempts cost on their own (six capped backoffs of 1-4ms, ~15ms + // total): at the 25ms this used to use, the assertion below cleared the floor with ~10ms of + // margin, so a machine that merely ran the attempts slowly satisfied it without the dwell being + // honoured at all. A quarter second is still a quarter second of test time, and an elapsed of + // ~15ms against it is unmistakable. + long authDwellFloorMillis = 250L; + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, authDwellFloorMillis, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + long startNanos = System.nanoTime(); + WebSocketClient out = drainer.connectWithDurableAckRetry(); + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos); + + assertNull(out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("the attempt threshold must be reached", + factory.attempts() >= BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS); + assertTrue("quarantine must not precede the auth dwell floor [elapsedMillis=" + + elapsedMillis + "]", + elapsedMillis >= authDwellFloorMillis); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + }); + } + + @Test(timeout = 60_000) + public void testRotatingCredentialAuthRejectionRidesPastAttemptThresholdBeforeDwellFloor() throws Exception { + assertMemoryLeak(() -> { + // Six fast 401s must not strand the slot while the configured self-healing window is still open. + // The seventh attempt succeeds, proving that attempt count alone cannot quarantine replayable data. + ScriptedFactory factory = ScriptedFactory + .failingTimes(BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS, + () -> new QwpAuthFailedException(401, "127.0.0.1", 9000)) + .withDynamicCredential(); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + WebSocketClient out = drainer.connectWithDurableAckRetry(); + + assertSame("the drainer must keep trying after the fast attempt threshold", + factory.successSentinel(), out); + assertEquals(BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS + 1, + factory.attempts()); + assertNotEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertFalse("a recovered credential must leave no .failed sentinel", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("no abandonment may be reported: " + captured, captured.isEmpty()); + }); + } + + @Test(timeout = 60_000) + public void testRotatingCredentialAuthRejectionRidesOutBoundedBudget() throws Exception { + assertMemoryLeak(() -> { + // With a ROTATING credential the Authorization header is re-derived from the token provider on + // every sweep, so a 401 can be a window that heals itself: a revocation landing mid-flight, an + // identity provider rotating signing keys, a token expiring during the settle so the next pull + // refreshes it. Quarantining on the first one permanently abandons replayable data - nothing in + // production clears the .failed sentinel - on a fault that repairs itself in seconds. + ScriptedFactory factory = ScriptedFactory + .failingTimes(2, () -> new QwpAuthFailedException(401, "127.0.0.1", 9000)) + .withDynamicCredential(); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + WebSocketClient out = drainer.connectWithDurableAckRetry(); + + assertSame("the drainer must recover once the credential is accepted", + factory.successSentinel(), out); + assertEquals(3, factory.attempts()); + assertNotEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertFalse("a recovered credential must leave no .failed sentinel", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("no abandonment may be reported: " + captured, captured.isEmpty()); + }); + } + + @Test(timeout = 60_000) + public void testRotatingCredentialAuthDwellIsClampedSoEscalationStaysReachable() { + // reconnect_max_duration_millis is validated only as > 0, and Long.MAX_VALUE is the documented way + // to ask a reconnect never to give up. TimeUnit saturates it, so the dwell half of the rotating-401 + // gate - an AND, unlike the capability-gap gate's OR - became unsatisfiable and the ride-out never + // ended: the drainer swept forever, never wrote the .failed sentinel, never reported DATA_LOSS, and + // pinned the slot lock plus one worker of a FIXED-size drainer pool for the life of the process. + // + // This pins the clamp as a FUNCTION. On its own that proves nothing about the connect loop, which + // could go on using the raw budget - so testConnectLoopAppliesTheClampedRotating401Dwell drives the + // loop itself against a saturated reconnect_max_duration_millis, pre-ageing the rejection anchor past + // the ceiling rather than waiting out five minutes of wall clock. Keep the two together: neither + // half is worth much alone. The third leg - that a FINITE dwell does quarantine - is + // testRotatingCredentialAuthRejectionQuarantinesOnceBudgetExhausted, with a 250ms budget. + long ceilingNanos = TimeUnit.MILLISECONDS.toNanos( + BackgroundDrainer.MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS); + + assertEquals("a saturated budget must not saturate the dwell", + ceilingNanos, BackgroundDrainer.dynamicCredentialAuthDwellNanos(Long.MAX_VALUE)); + assertTrue("and the clamped dwell must be reachable at all", + BackgroundDrainer.dynamicCredentialAuthDwellNanos(Long.MAX_VALUE) < Long.MAX_VALUE); + assertEquals("a budget above the ceiling is clamped to it", ceilingNanos, + BackgroundDrainer.dynamicCredentialAuthDwellNanos( + BackgroundDrainer.MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS * 10)); + // a smaller configured budget is honoured as-is, so tuning down still works - and this is the value + // the existing end-to-end quarantine tests rely on + assertEquals("a budget below the ceiling is used as configured", + TimeUnit.MILLISECONDS.toNanos(25L), BackgroundDrainer.dynamicCredentialAuthDwellNanos(25L)); + assertEquals("including the default, which IS the ceiling", ceilingNanos, + BackgroundDrainer.dynamicCredentialAuthDwellNanos( + CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS)); + } + + @Test(timeout = 60_000) + public void testConnectLoopAppliesTheClampedRotating401Dwell() throws Exception { + assertMemoryLeak(() -> { + // The clamp asserted above is a pure function; this asserts the CALL SITE applies it. Nothing + // else does: every other end-to-end test here configures a dwell far below the ceiling, where + // Math.min returns its first argument either way, so the connect loop reverting to the raw + // TimeUnit.MILLISECONDS.toNanos(reconnectMaxDurationMillis) leaves all of them green - and + // reconnect_max_duration_millis is validated only as > 0, with Long.MAX_VALUE the documented way + // to ask a reconnect never to give up. Saturated, the dwell conjunct can never be satisfied, so + // the ride-out never ends: no .failed sentinel, no DATA_LOSS report, and the slot lock plus one + // worker of a FIXED-size drainer pool pinned for the life of the process. + // + // Reaching the ceiling honestly costs five minutes of wall clock, so the rejection anchor is + // pre-aged past it instead and the loop runs for real against it. + final long anchorAgeMillis = BackgroundDrainer.MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS * 2; + // Bounds the counterfactual: unclamped, the loop never escalates, and without this it would run + // to the test timeout with nothing to say. Stopping it turns that into a named assertion + // failure on outcome() instead. Well clear of the six attempts a clamped run needs. + final int stopAfterAttempts = BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS * 5; + final BackgroundDrainer[] ref = new BackgroundDrainer[1]; + AtomicInteger scripted = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { + if (scripted.incrementAndGet() >= stopAfterAttempts) { + ref[0].requestStop(); + } + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + }).withDynamicCredential(); + + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, Long.MAX_VALUE, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + ref[0] = drainer; + drainer.ageDynamicCredentialAuthAnchorForTesting( + TimeUnit.MILLISECONDS.toNanos(anchorAgeMillis)); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + assertNull(drainer.connectWithDurableAckRetry()); + // FAILED, not STOPPED: STOPPED means the loop was still riding out rejections when the factory + // pulled the plug, which is precisely what an unclamped dwell does. + assertEquals("a saturated reconnect_max_duration_millis must not disable the escalation - the " + + "connect loop has to use the CLAMPED dwell [attempts=" + factory.attempts() + "]", + BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + // The dwell was already satisfied before the first sweep, so the attempt threshold is what the + // quarantine waited for - it must fire on exactly that attempt, not later. + assertEquals("quarantine must fall on the attempt threshold once the dwell is behind it", + BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS, factory.attempts()); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + }); + } + + @Test(timeout = 60_000) + public void testAlternating401AndOutageStillReachesTheEscalation() throws Exception { + assertMemoryLeak(() -> { + // The two tests below prove an unrelated outage must not let the dwell be satisfied for free, + // and they restart the dwell anchor to get it. Taken alone that leaves the AND gate + // unsatisfiable: the anchor is rewound by the capability-gap, role-reject and transport arms, + // while the attempt counter is only ever reset by real ack progress. A cluster that alternates + // - reject, blip, reject, blip - therefore re-anchors before every rejection, elapsed is always + // ~0, the second disjunct is permanently true, and connectWithDurableAckRetry() never returns: + // the slot is never quarantined, no DATA_LOSS is reported, and one of max_background_drainers + // workers is pinned for the life of the process. + // + // The attempt cap is what closes it. It cannot be rewound by an unrelated state, so it bounds + // the episode however the rejections are spaced, while leaving the dwell to decide every case + // that is not pathological. Without it this test does not fail an assertion - it never returns + // and dies on the @Test timeout. + final AtomicInteger calls = new AtomicInteger(); + CursorWebSocketSendLoop.ReconnectFactory alternating = new CursorWebSocketSendLoop.ReconnectFactory() { + @Override + public boolean hasDynamicCredential() { + return true; + } + + @Override + public WebSocketClient reconnect() { + // strict alternation: no two rejections are ever consecutive, so the dwell anchor is + // reset before each one and never accumulates + if (calls.incrementAndGet() % 2 == 1) { + throw new QwpAuthFailedException(401, "127.0.0.1", 9000); + } + throw new RuntimeException("cluster unreachable"); + } + }; + // A dwell far larger than anything this test can accumulate, so ONLY the cap can end it. + BackgroundDrainer drainer = newDrainerWithBudgets( + alternating, Long.MAX_VALUE, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + + assertNull("an alternating credential rejection must still reach the escalation", + drainer.connectWithDurableAckRetry()); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("the cap must be the backstop, not the first line: the ordinary dwell path has to " + + "get its full attempt threshold first", + calls.get() >= BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + }); + } + + @Test(timeout = 60_000) + public void testTransientOutageDoesNotCountTowardTheRotating401Dwell() throws Exception { + assertMemoryLeak(() -> { + // The dwell measures how long the REJECTION persisted, so an unrelated outage in the middle of a + // 401 run is not part of it. Anchored at the first 401 and never restarted, a 401, then an outage + // outlasting the dwell, then a sixth rejection satisfied both thresholds at once and quarantined + // the slot on a credential that had been rejected for seconds - abandoning replayable rows behind + // a .failed sentinel nothing in production clears. + final long dwellMillis = 100L; + AtomicInteger scripted = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { + if (scripted.incrementAndGet() == 6) { + // an unrelated cluster outage, longer than the whole dwell + Os.sleep(dwellMillis * 3); + return new RuntimeException("cluster unreachable"); + } + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + }).withDynamicCredential(); + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, dwellMillis, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + assertNull(drainer.connectWithDurableAckRetry()); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + + // five 401s, the outage, then the sixth 401 - attempt seven overall. Charging the outage to the + // dwell quarantines exactly there; restarting it means the sixth rejection must be followed by a + // fresh dwell of uninterrupted 401s first. + // The configured dwell here is far below MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS, so the clamp + // is inert and the dwell is unambiguously what ends the ride-out. + assertTrue("the outage must not have satisfied the dwell [attempts=" + factory.attempts() + "]", + factory.attempts() > 7); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + }); + } + + @Test(timeout = 60_000) + public void testCapabilityGapDoesNotCountTowardTheRotating401Dwell() throws Exception { + assertMemoryLeak(() -> { + // The same defect as the transient case, in the one arm that arm did not cover. A durable-ack + // capability gap means we REACHED a node and it answered - it simply cannot do durable ack - so + // it is not time the credential spent rejected. Its own settle budget can legitimately run for + // the whole reconnect budget, which is exactly the span the rotating-401 dwell is meant to + // require of an UNINTERRUPTED rejection, so charging it lets a rolling upgrade satisfy that + // floor for free and quarantine a slot over a credential rejected for seconds. + final long dwellMillis = 100L; + AtomicInteger scripted = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { + if (scripted.incrementAndGet() == 6) { + // one gap sweep, longer than the whole dwell. The first gap charges nothing to the + // capability-gap episode (lastCapabilityGapNanos is still 0), so it cannot escalate on + // its own and the rotating-401 accounting is what this observes. + Os.sleep(dwellMillis * 3); + return new QwpDurableAckMismatchException("h", 1234, "primary"); + } + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + }).withDynamicCredential(); + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, dwellMillis, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + assertNull(drainer.connectWithDurableAckRetry()); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + + // five 401s, the gap, then the sixth 401 - attempt seven overall. Charging the gap to the dwell + // quarantines exactly there; restarting it means the sixth rejection must be followed by a fresh + // dwell of uninterrupted 401s first. + assertTrue("a capability gap must not have satisfied the dwell [attempts=" + + factory.attempts() + "]", + factory.attempts() > 7); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + }); + } + + @Test(timeout = 60_000) + public void testRoleRejectDoesNotCountTowardTheRotating401Dwell() throws Exception { + assertMemoryLeak(() -> { + // The third sibling of the dwell-anchor reset, alongside the transient-outage and capability-gap + // cases above. The dwell measures how long the REJECTION persisted, so a role reject - an + // all-replica failover window in the middle of a 401 run - is not part of it. Anchored at the + // first 401 and never restarted, a 401, then a role-reject window outlasting the dwell, then a + // sixth rejection would satisfy both thresholds at once and quarantine a slot on a credential + // rejected for seconds - abandoning replayable rows behind a .failed sentinel nothing in + // production clears. + final long dwellMillis = 100L; + AtomicInteger scripted = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { + if (scripted.incrementAndGet() == 6) { + // an all-replica failover window, longer than the whole dwell + Os.sleep(dwellMillis * 3); + return new QwpIngressRoleRejectedException( + QwpIngressRoleRejectedException.ROLE_REPLICA, "127.0.0.1", 9000); + } + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + }).withDynamicCredential(); + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, dwellMillis, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + assertNull(drainer.connectWithDurableAckRetry()); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + + // five 401s, the role-reject window, then the sixth 401 - attempt seven overall. Charging the + // window to the dwell quarantines exactly there; restarting it means the sixth rejection must be + // followed by a fresh dwell of uninterrupted 401s first. The configured dwell is far below + // MAX_DYNAMIC_CREDENTIAL_AUTH_DWELL_MILLIS, so the clamp is inert and the dwell is unambiguously + // what ends the ride-out. + assertTrue("the role-reject window must not have satisfied the dwell [attempts=" + + factory.attempts() + "]", + factory.attempts() > 7); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + }); + } + + @Test(timeout = 60_000) + public void testFlappingCapabilityGapEscalatesAcrossMidDrainRecycles() throws Exception { + assertMemoryLeak(() -> { + // The capability-gap half of the counters-as-fields fix, which nothing else pins. Every other + // capability-gap test drives ONE connectWithDurableAckRetry() call whose factory rejects + // continuously, so the settle budget is spent inside that single call and the field-vs-local + // distinction never shows. Reverting capabilityGapAttempts to a method local leaves all of + // them green. + // + // The shape that needs a field is a cluster that flaps: connect accepted, drain, mid-drain + // durable-ack terminal, run() re-enters connectWithDurableAckRetry(), repeat. One gap sweep + // per call refills a local budget on every recycle, so 16 consecutive sweeps never accumulate + // and the slot is never quarantined - the drainer sweeps forever holding the slot lock and one + // of max_background_drainers workers. + // + // Only the ATTEMPT counter can escalate here, which is what makes this discriminating: the + // wall-clock half (capabilityGapElapsedNanos, lastCapabilityGapNanos) is deliberately per-call, + // and with a single gap per call lastCapabilityGapNanos is still 0 when it is charged, so the + // episode clock stays at zero however many recycles run. + final AtomicInteger calls = new AtomicInteger(); + CursorWebSocketSendLoop.ReconnectFactory flapping = new CursorWebSocketSendLoop.ReconnectFactory() { + @Override + public WebSocketClient reconnect() { + // one capability gap, then let the connect through - the recycle-forever shape + if (calls.incrementAndGet() % 2 == 1) { + throw new QwpDurableAckMismatchException("h", 1234, "primary"); + } + return stubClient(); + } + }; + BackgroundDrainer drainer = newDrainerWithBudgets( + flapping, Long.MAX_VALUE, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + + WebSocketClient out = null; + int recycles = 0; + for (; recycles < 40; recycles++) { + out = drainer.connectWithDurableAckRetry(); + if (out == null) { + break; + } + Os.sleep(2); // stand in for the drain between two mid-drain terminals + } + + assertNull("a flapping capability gap must reach the escalation instead of recycling forever", + out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("it must spend the whole settle budget, not escalate at the first recycle " + + "[recycles=" + recycles + "]", + recycles >= BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS - 1); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + }); + } + + @Test(timeout = 60_000) + public void testFlappingCredentialEscalatesAcrossMidDrainRecycles() throws Exception { + assertMemoryLeak(() -> { + // run() re-enters connectWithDurableAckRetry() after every mid-drain terminal. While the + // escalation counters were locals, each recycle refilled the budget it is meant to spend, so a + // cluster that flaps - connect accepted, drop, 401, recycle, repeat - looped forever with no ack + // progress: no quarantine, the slot lock never released, and one of max_background_drainers + // workers (four by default) pinned, starving every other orphan slot of a drainer. + // + // Driven by calling connectWithDurableAckRetry() repeatedly, which is what the recycle does. + final AtomicInteger calls = new AtomicInteger(); + CursorWebSocketSendLoop.ReconnectFactory flapping = new CursorWebSocketSendLoop.ReconnectFactory() { + @Override + public boolean hasDynamicCredential() { + return true; + } + + @Override + public WebSocketClient reconnect() { + // reject once, then let the connect through - the shape that recycles forever + if (calls.incrementAndGet() % 2 == 1) { + throw new QwpAuthFailedException(401, "127.0.0.1", 9000); + } + return stubClient(); + } + }; + BackgroundDrainer drainer = newDrainerWithBudgets( + flapping, 25L, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + + WebSocketClient out = null; + int recycles = 0; + for (; recycles < 40; recycles++) { + out = drainer.connectWithDurableAckRetry(); + if (out == null) { + break; + } + Os.sleep(2); // stand in for the drain between two mid-drain terminals + } + + assertNull("a flapping credential must reach the escalation instead of recycling forever", out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("it must escalate once both thresholds are met, not at the very first recycle " + + "[recycles=" + recycles + "]", + recycles >= BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS - 1); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + }); + } + + @Test(timeout = 60_000) public void testReturnsClientOnSuccessFirstAttempt() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -291,7 +778,7 @@ public void testReturnsClientOnSuccessFirstAttempt() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testRetriesOnDurableAckMismatchThenSucceeds() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -313,7 +800,7 @@ public void testRetriesOnDurableAckMismatchThenSucceeds() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testStopRequestedDuringRetryAbortsWithStoppedOutcome() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -348,7 +835,7 @@ public void testStopRequestedDuringRetryAbortsWithStoppedOutcome() throws Except }); } - @Test + @Test(timeout = 60_000) public void testWallTimeBudgetEscalatesBeforeAttemptCap() throws Exception { assertMemoryLeak(() -> { CountingListener listener = new CountingListener(); @@ -381,7 +868,7 @@ public void testWallTimeBudgetEscalatesBeforeAttemptCap() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testAllReplicaWindowNeverEscalatesInvariantB() throws Exception { assertMemoryLeak(() -> { // INVARIANT B (orphan drainer): a store-and-forward drainer must NEVER @@ -399,11 +886,12 @@ public void testAllReplicaWindowNeverEscalatesInvariantB() throws Exception { // problem and stays terminal. This test uses a role reject (every // endpoint is a replica right now), which must NOT be terminal. // - // Red-first: connectWithDurableAckRetry() currently lumps role rejects in - // with the durable-ack-mismatch give-up, so after the 16-attempt cap / - // the budget it markFailed()s and returns -> the helper thread dies. Goes - // green once the drainer treats an all-replica window as retry-forever - // (split the catch: role reject -> retry; capability gap -> quarantine). + // The regression this pins: lumping role rejects in with the + // durable-ack-mismatch give-up. Under that shape the 16-attempt cap or + // the wall-clock budget markFailed()s and returns, so the helper thread + // started below dies inside the observation window. The drainer keeps the + // two apart - a role reject backs off and retries, a capability gap + // quarantines - which is what the still-alive assertions rest on. CountingListener listener = new CountingListener(); AtomicInteger attempts = new AtomicInteger(); ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { @@ -452,7 +940,7 @@ public void testAllReplicaWindowNeverEscalatesInvariantB() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testTransportErrorNeverQuarantinesInvariantB() throws Exception { assertMemoryLeak(() -> { // INVARIANT B (orphan drainer): a fully-unreachable cluster (server down, @@ -464,11 +952,12 @@ public void testTransportErrorNeverQuarantinesInvariantB() throws Exception { // (CursorWebSocketSendLoop.connectLoop: a transport error backs off and // retries), which the orphan drainer must match. // - // Red-first: connectWithDurableAckRetry() currently routes any non-role, - // non-durable-ack Throwable (including "all endpoints unreachable") to an - // IMMEDIATE markFailed / .failed sentinel on the first attempt. Green once - // transport errors are retried indefinitely like connectLoop. (Genuine - // terminals -- auth / non-421 upgrade -- must still fail fast.) + // The regression this pins: routing any non-role, non-durable-ack + // Throwable - "all endpoints unreachable" included - to an IMMEDIATE + // markFailed / .failed sentinel on the first attempt. The catch-all + // retries a transport failure indefinitely, exactly as connectLoop does; + // the genuine terminals (auth, non-421 upgrade, durable-ack capability + // gap) are caught ahead of it and still fail fast. CountingListener listener = new CountingListener(); AtomicInteger attempts = new AtomicInteger(); ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { @@ -514,7 +1003,7 @@ public void testTransportErrorNeverQuarantinesInvariantB() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testJvmErrorEscapesConnectRetryLoop() throws Exception { assertMemoryLeak(() -> { // Regression (M3): catch (Throwable) in connectWithDurableAckRetry used @@ -546,7 +1035,7 @@ public void testJvmErrorEscapesConnectRetryLoop() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testRoleRejectChurnDoesNotConsumeCapabilityGapBudgetInvariantB() throws Exception { assertMemoryLeak(() -> { // Rolling-upgrade interleave: a long all-replica window (role rejects), @@ -593,7 +1082,7 @@ public void testRoleRejectChurnDoesNotConsumeCapabilityGapBudgetInvariantB() thr }); } - @Test + @Test(timeout = 60_000) public void testFailoverWindowDoesNotBurnCapabilityGapWallClockInvariantB() throws Exception { assertMemoryLeak(() -> { // The wall-clock half of the settle budget must be anchored at the @@ -636,7 +1125,7 @@ public void testFailoverWindowDoesNotBurnCapabilityGapWallClockInvariantB() thro }); } - @Test + @Test(timeout = 60_000) public void testRoleRejectResetsCapabilityGapEpisode() throws Exception { assertMemoryLeak(() -> { // An intervening role reject proves the topology changed (the node @@ -682,7 +1171,51 @@ public void testRoleRejectResetsCapabilityGapEpisode() throws Exception { }); } - @Test + @Test(timeout = 60_000) + public void testRotatingCredentialRejectionResetsCapabilityGapEpisode() throws Exception { + assertMemoryLeak(() -> { + // The reverse direction of testRoleRejectResetsCapabilityGapEpisode / + // testTransportErrorResetsCapabilityGapEpisode: a rotating-401 that is ridden out proves nothing + // about a node's batch cap, so it restarts the capability-gap settle budget exactly as those + // transient states do. 15 gap errors, one 401 (ridden out with a dynamic credential), then gaps + // again -- the second episode gets the full 16 attempts, it does not inherit the first episode's + // 15. + int cap = BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS; + CountingListener listener = new CountingListener(); + AtomicInteger sweeps = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.alwaysFailing(() -> { + if (sweeps.incrementAndGet() == cap) { // 16th sweep: a rotating-401 between the gap runs + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + } + return new QwpDurableAckMismatchException("h", 1234, "primary"); + }).withDynamicCredential(); + BackgroundDrainer drainer = newDrainer(factory); + drainer.setListener(listener); + WebSocketClient out = drainer.connectWithDurableAckRetry(); + assertNull(out); + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertEquals(1, listener.persistentFailures.get()); + assertEquals("second episode must get the full budget after the reset", + cap, listener.lastPersistentTotalAttempts.get()); + // 15 gap + 1 rotating-401 + 16 gap = 32 sweeps total. + assertEquals(2 * cap, factory.attempts()); + // The DA stream carries both episodes' per-episode numbering (1..15, then 1..15 -- the second + // episode's 16th attempt fires persistent-failure instead). A ridden-out 401 fires no + // observability callback, so unlike the role-reject reset nothing lands on the primary stream. + List expectedDaStream = new ArrayList<>(); + for (int episode = 0; episode < 2; episode++) { + for (int i = 1; i <= cap - 1; i++) { + expectedDaStream.add(i); + } + } + assertEquals(expectedDaStream, listener.unavailableAttempts); + assertTrue("a ridden-out 401 fires no primary-unavailable callback", + listener.primaryUnavailableAttempts.isEmpty()); + assertTrue(Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + }); + } + + @Test(timeout = 60_000) public void testRoleRejectAndCapabilityGapLandOnSeparateStreams() throws Exception { assertMemoryLeak(() -> { // M10 discriminator: gap -> role reject -> gap -> success. The @@ -720,7 +1253,7 @@ public void testRoleRejectAndCapabilityGapLandOnSeparateStreams() throws Excepti }); } - @Test + @Test(timeout = 60_000) public void testSaturatingCapabilityGapBudgetDoesNotQuarantineOnTheFirstSweep() throws Exception { assertMemoryLeak(() -> { // reconnect_max_duration_millis is validated only as > 0, and Long.MAX_VALUE @@ -746,7 +1279,7 @@ public void testSaturatingCapabilityGapBudgetDoesNotQuarantineOnTheFirstSweep() }); } - @Test + @Test(timeout = 60_000) public void testTransportErrorResetsCapabilityGapEpisode() throws Exception { assertMemoryLeak(() -> { // A transport state breaks a consecutive capability-gap episode. @@ -779,7 +1312,7 @@ public void testTransportErrorResetsCapabilityGapEpisode() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testTransportWindowResetsCapabilityGapWallClock() throws Exception { assertMemoryLeak(() -> { // The wall-clock half of the settle budget is anchored at gap #1. @@ -824,7 +1357,7 @@ public void testTransportWindowResetsCapabilityGapWallClock() throws Exception { }); } - @Test + @Test(timeout = 60_000) public void testRoleRejectGrantsFreshWallClockToNextGapEpisode() { // Companion to testRoleRejectResetsCapabilityGapEpisode, which pins the // ATTEMPT-counter half of the episode reset but runs under a 60s budget @@ -883,7 +1416,60 @@ public void testRoleRejectGrantsFreshWallClockToNextGapEpisode() { assertEquals(Collections.singletonList(1), listener.primaryUnavailableAttempts); } - @Test + @Test(timeout = 60_000) + public void testRotatingCredentialRejectionGrantsFreshWallClockToNextGapEpisode() { + // Companion to testRotatingCredentialRejectionResetsCapabilityGapEpisode, which pins the + // ATTEMPT-counter half of the reset but runs under a 60s budget where the wall-clock half is + // unobservable: a mutant that resets only capabilityGapAttempts (leaving capabilityGapElapsedNanos / + // lastCapabilityGapNanos ticking) passes it. This pins the WALL-CLOCK half of the ride-out arm's + // reset - the twin of testRoleRejectGrantsFreshWallClockToNextGapEpisode: gap sweeps burn most of the + // budget, a ridden-out 401 proves nothing about the node's batch cap, and the next gap episode must + // start from a zero wall clock -- under the counter-only mutant the stale elapsed (plus the + // still-anchored lastCapabilityGapNanos charging straight across the 401 window) exhausts the budget + // and quarantines a cluster that was about to settle. + long budgetMillis = 800L; + CountingListener listener = new CountingListener(); + AtomicInteger sweeps = new AtomicInteger(); + ScriptedFactory factory = ScriptedFactory.failingTimes(5, () -> { + switch (sweeps.incrementAndGet()) { + case 2: + // Burn ~600ms of the 800ms budget inside the first gap episode. + sleepQuietly(600); + return new QwpDurableAckMismatchException("h", 1234, "primary"); + case 3: + // A rotating-401 ridden out: the settle budget must restart in full. + return new QwpAuthFailedException(401, "127.0.0.1", 9000); + case 5: + // Second episode burns ~350ms -- well inside a fresh 800ms budget, but 600 + 350 > 800 + // under the mutant's carried-over wall clock. + sleepQuietly(350); + return new QwpDurableAckMismatchException("h", 1234, "primary"); + default: + return new QwpDurableAckMismatchException("h", 1234, "primary"); + } + }).withDynamicCredential(); + BackgroundDrainer drainer = newDrainerWithBudgets( + factory, budgetMillis, FAST_BACKOFF_MILLIS, FAST_BACKOFF_MAX_MILLIS); + drainer.setListener(listener); + WebSocketClient out = drainer.connectWithDurableAckRetry(); + assertSame("a ridden-out 401 restarts the episode wall clock -- the second gap episode must get the " + + "full settle budget, not the first episode's leftovers", + factory.successSentinel(), out); + // gap, gap(+600ms), 401(ridden out), gap, gap(+350ms), success = 6 sweeps. + assertEquals(6, factory.attempts()); + assertEquals(BackgroundDrainer.DrainOutcome.PENDING, drainer.outcome()); + assertEquals("a settling cluster must never see a persistent-failure escalation", + 0, listener.persistentFailures.get()); + assertFalse("no .failed sentinel: both gap episodes stayed inside their budgets", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + // DA stream: gaps 1,2 then the fresh episode's 1,2. A ridden-out 401 fires no observability callback, + // so nothing lands on the primary stream (unlike the role-reject twin). + assertEquals(Arrays.asList(1, 2, 1, 2), listener.unavailableAttempts); + assertTrue("a ridden-out 401 fires no primary-unavailable callback", + listener.primaryUnavailableAttempts.isEmpty()); + } + + @Test(timeout = 60_000) public void testRequestStopInterruptsLongBackoffParkPromptly() throws Exception { // Pins the stop-promptness contract of the backoff park: requestStop() // must break the drainer out of a LONG park (unpark, backstopped by @@ -942,7 +1528,7 @@ private BackgroundDrainer newDrainer(ScriptedFactory factory) { } private BackgroundDrainer newDrainerWithBudgets( - ScriptedFactory factory, + CursorWebSocketSendLoop.ReconnectFactory factory, long reconnectMaxDurationMillis, long backoffInitMillis, long backoffMaxMillis) { @@ -1032,6 +1618,9 @@ private static final class ScriptedFactory implements CursorWebSocketSendLoop.Re private final WebSocketClient successSentinel; private final ThrowableSupplier throwSupplier; private final int throwingTimes; + // models a sender wired to an httpTokenProvider: the Authorization header is re-derived on every + // attempt, so a 401 can be a window that heals rather than a permanent misconfiguration + private boolean dynamicCredential; ScriptedFactory(WebSocketClient successSentinel, int throwingTimes, @@ -1057,6 +1646,22 @@ int attempts() { return calls.get(); } + /** + * The signal BackgroundDrainer branches its terminal policy on. Stubbed here, deliberately: these + * tests pin the POLICY (fail fast on a constant credential, ride out the settle budget on a rotating + * one), not the classification. What decides it in production is + * {@code QwpWebSocketSender.hasDynamicCredential()} - a {@code FixedAuthHeader} identity check on the + * configured supplier - and that is pinned on a real built sender, for httpToken, + * httpUsernamePassword, httpTokenProvider and no-credential alike, by + * {@code WebSocketTokenProviderTest.testCredentialKindTaggedForTheOrphanDrainerTerminalPolicy}, + * which reads it both directly and through the background reconnect factory a drainer is handed. + * Neither half means much without the other: keep them named in each other's comments. + */ + @Override + public boolean hasDynamicCredential() { + return dynamicCredential; + } + @Override public WebSocketClient reconnect() throws Exception { int n = calls.incrementAndGet(); @@ -1073,6 +1678,11 @@ public WebSocketClient reconnect() throws Exception { WebSocketClient successSentinel() { return successSentinel; } + + ScriptedFactory withDynamicCredential() { + this.dynamicCredential = true; + return this; + } } /** diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainAuthRejectTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainAuthRejectTest.java new file mode 100644 index 000000000..db5efa91d --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainAuthRejectTest.java @@ -0,0 +1,573 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client.sf.cursor; + +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.http.client.WebSocketClient; +import io.questdb.client.cutlass.http.client.WebSocketClientFactory; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainer; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; +import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorWebSocketSendLoop; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.std.Files; +import io.questdb.client.std.MemoryTag; +import io.questdb.client.std.Unsafe; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.IntPredicate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Mid-drain rotating-credential 401/403 coverage for {@link BackgroundDrainer}. + *

+ * {@code connectWithDurableAckRetry} gives an ORPHAN drainer whose credential + * rotates ({@code hasDynamicCredential()}) a bounded ride-out requiring both + * an attempt threshold ({@link BackgroundDrainer#DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS}) + * and a wall-clock dwell floor before quarantining on a 401 — the header is + * re-derived from the token provider every attempt, so a rejection can be a self-healing window (a + * revocation landing mid-flight, the IdP rotating signing keys, clock skew) a + * freshly pulled token clears. The same rejection hit mid-drain (the + * wire drops, the loop's reconnect sweep is refused) must get the same ride-out + * rather than dropping a {@code .failed} sentinel on the first sweep — otherwise + * a token rotation during an in-progress drain permanently abandons replayable + * data on a fault that heals in seconds. The initial-connect ride-out + * ({@link BackgroundDrainerDurableAckRetryTest}) never exercised the mid-drain + * reconnect, which the ORPHAN {@link CursorWebSocketSendLoop} handles. + *

+ * A CONSTANT credential still quarantines on the first mid-drain 401: it is + * uniformly rejected across the cluster and will not heal. The sanctioned + * terminal set is otherwise unchanged. + *

+ * Wire realism: a real {@link TestWebSocketServer} durably acks over a live + * socket; the scripted {@link CursorWebSocketSendLoop.ReconnectFactory} decides, + * per connect attempt, whether the sweep reaches a healthy node or is refused + * with a 401. The mid-drain drop is deterministic — the server closes the first + * connection after durably acking exactly one frame. + */ +public class BackgroundDrainerMidDrainAuthRejectTest { + + private static final long ACK_OBSERVATION_DELAY_MILLIS = 800L; + private static final long FAST_BACKOFF_MAX_MILLIS = 4L; + private static final long FAST_BACKOFF_MILLIS = 1L; + private static final long RECONNECT_MAX_DURATION_MILLIS = 25L; + private static final int SEEDED_FRAMES = 5; + private static final long SEGMENT_SIZE_BYTES = 16_384L; + private static final long SF_MAX_TOTAL_BYTES = 1L << 20; + + private String slotPath; + + // one shared temp-directory mechanism instead of a per-class java.io.tmpdir path plus a hand-rolled + // recursive delete: the rule cleans up on failure and on an exception thrown out of a test too + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Before + public void setUp() { + slotPath = temp.getRoot().toPath().resolve("slot").toString(); + assertEquals("mkdir slot dir", 0, Files.mkdir(slotPath, Files.DIR_MODE_DEFAULT)); + } + + + @Test + public void testDeliveringBetweenTwoRotating401WindowsGrantsAFreshRideOut() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // noteAckProgress() clears dynamicCredentialAuthAttempts when the wire durably acks something + // past the watermark, and nothing failed when that line was deleted. Held per drain with no + // notion of progress, the counter instead spans every session: two rejection windows with a + // DELIVERING session between them accumulate toward one threshold, so rejections that were + // never consecutive quarantine a slot the cluster is still draining - and nothing in + // production clears the .failed sentinel, so those replayable rows are abandoned for good. + // + // Two windows of 5 calls each. Each window spends its first call inside the send loop's own + // reconnect (latched as authTerminal, deliberately not counted), leaving 4 counted per window + // - under the threshold of 6 alone, over it cumulatively. + // + // The quarantine gate is an AND of the attempt threshold and a wall-clock dwell floor, so the + // dwell is set to 1ms here: it is satisfied within one backoff either way, which leaves the + // ATTEMPT count as the only thing deciding the verdict. Its twin below does the reverse. + final long dwellMillis = 1L; + seedSlot(SEEDED_FRAMES); + Map drops = new HashMap<>(); + drops.put(1, 0L); // connection 1 acks one frame, then drops -> into window 1 + drops.put(2, 1L); // connection 2 is the delivering session -> advances, then drops into window 2 + // Keep the delivering connection alive briefly after its progress ack. Closing it immediately + // races the drainer's 50ms ack poll: the second 401 window can begin before noteAckProgress() + // observes the new watermark, making two separate windows look like one continuous episode. + try (TestWebSocketServer server = new TestWebSocketServer( + new ScriptedAckHandler(drops, 2, ACK_OBSERVATION_DELAY_MILLIS), true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), + n -> (n >= 2 && n <= 6) || (n >= 8 && n <= 12), /* dynamicCredential */ true); + BackgroundDrainer drainer = newDrainer(factory, dwellMillis); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + assertEquals("delivering between the windows ends the episode, so neither window reaches " + + "the attempt threshold and the slot must still drain [attempts=" + + factory.attempts() + "]", + BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("a slot the cluster is still draining must not be quarantined", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("a credential the next token healed must report no data loss: " + captured, + captured.isEmpty()); + assertTrue("both rejection windows must actually have been driven [attempts=" + + factory.attempts() + "]", factory.attempts() > 12); + } + }); + } + + @Test + public void testDeliveringBetweenTwoRotating401WindowsRestartsTheDwellAnchor() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // The twin of the test above, for the OTHER line noteAckProgress() clears: + // firstDynamicCredentialAuthFailureNanos, the anchor the wall-clock dwell is measured from. + // The quarantine gate is an AND, so each line needs its own discriminator - dropping the + // attempts reset leaves the dwell short and the OR still satisfied, and dropping the anchor + // reset leaves the attempt count low. This one keeps the attempt count legitimate and makes + // only the anchor decide. + // + // The dwell is 300ms here (reconnect_max_duration_millis, under the clamp). The delivering + // session sleeps 800ms before it acks, so a STALE anchor measures ~800ms+ and a restarted one + // measures only the second window's own backoff - a margin no scheduling jitter closes. + final long dwellMillis = 300L; + seedSlot(SEEDED_FRAMES); + Map drops = new HashMap<>(); + drops.put(1, 0L); + drops.put(2, 1L); + // connection 2 - the delivering session - acks its progress and then lingers before it drops, + // putting real wall clock between the two rejection windows (see ScriptedAckHandler). + try (TestWebSocketServer server = new TestWebSocketServer( + new ScriptedAckHandler(drops, 2, ACK_OBSERVATION_DELAY_MILLIS), true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Window 2 is long enough to reach the attempt threshold on its own, so the attempt + // conjunct is satisfied either way and only the dwell conjunct decides the verdict. + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), + n -> (n >= 2 && n <= 6) || (n >= 8 && n <= 14), /* dynamicCredential */ true); + BackgroundDrainer drainer = newDrainer(factory, dwellMillis); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + assertEquals("ack progress must restart the dwell anchor, so the second window is measured " + + "from its own first rejection and cannot satisfy the dwell floor " + + "[attempts=" + factory.attempts() + "]", + BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("a slot the cluster is still draining must not be quarantined", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("a credential the next token healed must report no data loss: " + captured, + captured.isEmpty()); + assertTrue("both rejection windows must actually have been driven [attempts=" + + factory.attempts() + "]", factory.attempts() > 14); + } + }); + } + + @Test + public void testMidDrainConstantCredential401QuarantinesImmediately() throws Exception { + TestUtils.assertMemoryLeak(() -> { + seedSlot(SEEDED_FRAMES); + DropFirstHandler handler = new DropFirstHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Same mid-drain 401, but the credential is CONSTANT: no ride-out, + // it must quarantine on the first sweep exactly as before the fix. + ScriptedWireFactory factory = new ScriptedWireFactory( + server.getPort(), 2, Integer.MAX_VALUE, /* dynamicCredential */ false); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("a constant-credential 401 must quarantine the slot", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("a constant credential must not consume the rotating-401 ride-out", + 2, factory.attempts()); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + } + }); + } + + @Test + public void testMidDrainPersistentRotating401ExhaustsRideOutThenQuarantines() throws Exception { + TestUtils.assertMemoryLeak(() -> { + seedSlot(SEEDED_FRAMES); + DropFirstHandler handler = new DropFirstHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // The rotation never heals: every sweep after the drop is a 401. + ScriptedWireFactory factory = new ScriptedWireFactory( + server.getPort(), 2, Integer.MAX_VALUE, /* dynamicCredential */ true); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + assertEquals(BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("a persistent rotating 401 must quarantine after the ride-out", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + // 1 healthy connect + 1 loop reconnect sweep (latches the loop's authTerminal) + enough + // re-entered sweeps to satisfy both the attempt threshold and the wall-clock dwell floor. + assertTrue("the drainer must reach the rotating-auth attempt threshold", + factory.attempts() >= 2 + BackgroundDrainer.DEFAULT_MAX_DYNAMIC_CREDENTIAL_AUTH_ATTEMPTS); + assertEquals("exactly one abandonment report: " + captured, 1, captured.size()); + assertEquals(SenderError.Category.DATA_LOSS, captured.get(0).getCategory()); + } + }); + } + + @Test + public void testMidDrainRotating401RidesOutThenDrains() throws Exception { + TestUtils.assertMemoryLeak(() -> { + seedSlot(SEEDED_FRAMES); + DropFirstHandler handler = new DropFirstHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // Call 1: healthy connect (drain starts; server durably acks one + // frame, then drops the wire). Calls 2-4: the reconnect sweep is + // refused with a 401. Call 5+: the rotation healed; the freshly + // pulled token is accepted and the drain completes. + ScriptedWireFactory factory = new ScriptedWireFactory( + server.getPort(), 2, 4, /* dynamicCredential */ true); + BackgroundDrainer drainer = newDrainer(factory); + List captured = Collections.synchronizedList(new ArrayList()); + drainer.setErrorSink(captured::add); + + runToCompletion(drainer); + + // Without the mid-drain ride-out the first 401 (call 2) latches a + // fatal terminal and the drainer quarantines: outcome FAILED, + // attempts == 2, a .failed sentinel. The fix routes it into the + // ride-out instead, so the drain survives the rotation. + assertEquals("a rotating 401 that heals within the ride-out must not quarantine", + BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("no .failed sentinel after a successful drain", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertTrue("expected the drainer to ride out the 401s, attempts=" + factory.attempts(), + factory.attempts() >= 5); + assertTrue("a healed rotation must report no data loss: " + captured, captured.isEmpty()); + } + }); + } + + private BackgroundDrainer newDrainer(ScriptedWireFactory factory) { + return newDrainer(factory, RECONNECT_MAX_DURATION_MILLIS); + } + + private BackgroundDrainer newDrainer(ScriptedWireFactory factory, long reconnectMaxDurationMillis) { + return new BackgroundDrainer( + slotPath, + SEGMENT_SIZE_BYTES, + SF_MAX_TOTAL_BYTES, + factory, + reconnectMaxDurationMillis, + FAST_BACKOFF_MILLIS, + FAST_BACKOFF_MAX_MILLIS, + /* requestDurableAck */ true, + /* durableAckKeepaliveIntervalMillis */ 200L); + } + + + private static void runToCompletion(BackgroundDrainer drainer) throws InterruptedException { + Thread t = new Thread(drainer, "test-mid-drain-auth-drainer"); + t.setDaemon(true); + t.start(); + t.join(20_000); + if (t.isAlive()) { + drainer.requestStop(); + t.join(5_000); + fail("drainer did not finish within 20s (outcome=" + drainer.outcome() + ")"); + } + } + + private void seedSlot(int frames) { + try (CursorSendEngine engine = new CursorSendEngine(slotPath, SEGMENT_SIZE_BYTES)) { + long buf = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT); + try { + byte[] payload = "frame-bytes-padd".getBytes(StandardCharsets.US_ASCII); + for (int i = 0; i < payload.length; i++) { + Unsafe.getUnsafe().putByte(buf + i, payload[i]); + } + for (int i = 0; i < frames; i++) { + engine.appendBlocking(buf, 16); + } + } finally { + Unsafe.free(buf, 16, MemoryTag.NATIVE_DEFAULT); + } + } + } + + /** + * Server-side script. Connection #1 durably acks exactly one frame, then + * closes the socket — a deterministic mid-drain wire drop. Every later + * connection acks all traffic, so a reconnected loop drains to completion. + * Keyed per {@code ClientHandler} identity; a dead connection's late + * buffered frames are ignored rather than acked with a stale counter. + */ + private static final class DropFirstHandler implements TestWebSocketServer.WebSocketServerHandler { + private static final String TABLE = "trades"; + private final List arrivalOrder = new ArrayList<>(); + private final java.util.Map wireSeqByConn = + new java.util.IdentityHashMap<>(); + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long[] counter = wireSeqByConn.get(client); + if (counter == null) { + counter = new long[1]; + wireSeqByConn.put(client, counter); + arrivalOrder.add(client); + } + int connectionIndex = arrivalOrder.indexOf(client) + 1; + long seq = counter[0]++; + try { + if (connectionIndex == 1) { + if (seq == 0) { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } else if (seq == 1) { + client.close(); // mid-drain wire drop + } + // seq > 1: late buffered frames from the condemned connection; ignore. + } else { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } + } catch (IOException ignored) { + // Best-effort ack: the connection died under us. The client replays + // on its next connection. + } + } + + private static byte[] durableAckFrame(long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x02); // STATUS_DURABLE_ACK + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + + private static byte[] okFrame(long wireSeq, long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 8 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(wireSeq); + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + } + + /** + * Per-connection scripted acks over a real wire. Connection indexes present in + * {@code dropAfterSeqByConnection} (1-based, arrival order) durably ack up to that per-connection + * seq and then close the socket - a deterministic mid-drain wire drop; every other connection acks + * whatever it is sent. One connection may additionally linger after durably acking its progress and + * before it drops, which is how {@link #testDeliveringBetweenTwoRotating401WindowsRestartsTheDwellAnchor()} + * puts real wall clock between the two rejection windows without leaning on backoff timing - and without + * racing the drainer's ack-progress poll, which a sleep before the ack did. + *

+ * Mirrors {@code BackgroundDrainerMidDrainCapabilityGapTest.GapScenarioHandler}; kept local because + * that one is private to its own class and the two scripts differ in the delay. + */ + private static final class ScriptedAckHandler implements TestWebSocketServer.WebSocketServerHandler { + private static final String TABLE = "trades"; + private final List arrivalOrder = new ArrayList<>(); + private final int delayConnectionIndex; + private final long delayMillis; + private final Map dropAfterSeqByConnection; + private final Map wireSeqByConn = + new java.util.IdentityHashMap<>(); + private boolean delayed; + + ScriptedAckHandler(Map dropAfterSeqByConnection, int delayConnectionIndex, long delayMillis) { + this.dropAfterSeqByConnection = dropAfterSeqByConnection; + this.delayConnectionIndex = delayConnectionIndex; + this.delayMillis = delayMillis; + } + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + long[] counter = wireSeqByConn.get(client); + if (counter == null) { + counter = new long[1]; + wireSeqByConn.put(client, counter); + arrivalOrder.add(client); + } + int connectionIndex = arrivalOrder.indexOf(client) + 1; + long seq = counter[0]++; + try { + Long dropAfterSeq = dropAfterSeqByConnection.get(connectionIndex); + if (dropAfterSeq != null) { + if (seq <= dropAfterSeq) { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } else if (seq == dropAfterSeq + 1) { + // The wall-clock gap between the two rejection windows goes HERE - AFTER this + // connection durably acked its progress (the seq <= dropAfterSeq branch above) and + // BEFORE it drops the wire. Sleeping before the ack instead raced the drop: on a + // loaded runner the client had not committed the durable ack - so the drainer's poll + // never observed the watermark advance and noteAckProgress never reset the counter or + // anchor - by the time the drop recycled it into the second window, and a healthy slot + // quarantined. Lingering after the ack lets the poll observe the advance first. + if (connectionIndex == delayConnectionIndex && !delayed) { + delayed = true; + try { + Thread.sleep(delayMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + client.close(); // mid-drain wire drop + } + // beyond that: late buffered frames from the condemned connection; ignore. + } else { + client.sendBinary(okFrame(seq, seq)); + client.sendBinary(durableAckFrame(seq)); + } + } catch (IOException ignored) { + // Best-effort ack: the connection died under us. The client replays on its next one. + } + } + + private static byte[] durableAckFrame(long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x02); // STATUS_DURABLE_ACK + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + + private static byte[] okFrame(long wireSeq, long seqTxn) { + byte[] name = TABLE.getBytes(StandardCharsets.UTF_8); + ByteBuffer bb = ByteBuffer.allocate(1 + 8 + 2 + 2 + name.length + 8) + .order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(wireSeq); + bb.putShort((short) 1); // tableCount + bb.putShort((short) name.length); + bb.put(name); + bb.putLong(seqTxn); + return bb.array(); + } + } + + /** + * Per-call-index scripted factory over a real wire. Call indexes inside + * {@code [throwFrom, throwTo]} (1-based, inclusive) throw a + * {@link QwpAuthFailedException} (401); every other call returns a live + * upgraded client against the test server. {@link #hasDynamicCredential()} + * reports whether the credential rotates — the signal the orphan drainer's + * terminal policy reads. + */ + private static final class ScriptedWireFactory implements CursorWebSocketSendLoop.ReconnectFactory { + private final AtomicInteger calls = new AtomicInteger(); + private final boolean dynamicCredential; + private final int port; + private final IntPredicate rejectWhen; + + ScriptedWireFactory(int port, int throwFrom, int throwTo, boolean dynamicCredential) { + this(port, n -> n >= throwFrom && n <= throwTo, dynamicCredential); + } + + ScriptedWireFactory(int port, IntPredicate rejectWhen, boolean dynamicCredential) { + this.port = port; + this.rejectWhen = rejectWhen; + this.dynamicCredential = dynamicCredential; + } + + int attempts() { + return calls.get(); + } + + @Override + public boolean hasDynamicCredential() { + return dynamicCredential; + } + + @Override + public WebSocketClient reconnect() throws Exception { + int n = calls.incrementAndGet(); + if (rejectWhen.test(n)) { + throw new QwpAuthFailedException(401, "localhost", port); + } + WebSocketClient c = WebSocketClientFactory.newPlainTextInstance(); + try { + c.setQwpMaxVersion(1); + c.setQwpRequestDurableAck(true); + c.setConnectTimeout(5_000); + c.connect("localhost", port); + c.upgrade("/write/v4", 5_000, null); + } catch (Throwable t) { + c.close(); + throw t; + } + return c; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainCapabilityGapTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainCapabilityGapTest.java index 889fd3e5f..6f2224a14 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainCapabilityGapTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/BackgroundDrainerMidDrainCapabilityGapTest.java @@ -50,6 +50,7 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.List; +import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; @@ -138,6 +139,188 @@ public void testMidDrainCapabilityGapGetsSettleBudgetNotQuarantine() throws Exce }); } + @Test + public void testFinalAckObservedAfterRecoverableTerminalStopsWithoutAnotherAttempt() throws Exception { + TestUtils.assertMemoryLeak(() -> { + seedSlot(SEEDED_FRAMES); + CountDownLatch releaseFinalAck = new CountDownLatch(1); + FinalAckThenCloseHandler handler = new FinalAckThenCloseHandler( + releaseFinalAck, SEEDED_FRAMES - 1L); + try (TestWebSocketServer server = new TestWebSocketServer(handler, true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + // Attempt 1 sends every orphan frame. The server holds its ACKs until the drainer has read + // the old watermark, then durably ACKs the target and closes. Attempt 2 is the loop's + // recoverable capability-gap terminal. The post-terminal re-read must see the completed + // drain and return; attempting connection 3 would spend a fresh settle budget and can + // quarantine rows the server has already accepted. + ScriptedWireFactory factory = new ScriptedWireFactory( + server.getPort(), 2, Integer.MAX_VALUE); + BackgroundDrainer drainer = newDrainer(factory); + CountDownLatch staleAckPolled = new CountDownLatch(1); + drainer.setAfterAckPollHookForTesting(loop -> { + staleAckPolled.countDown(); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (loop.getTerminalError() == null && System.nanoTime() < deadline) { + Thread.yield(); + } + if (loop.getTerminalError() == null) { + throw new AssertionError("recoverable terminal was not published after the final ACK"); + } + }); + + Thread runner = new Thread(drainer, "test-post-terminal-ack-drainer"); + runner.setDaemon(true); + runner.start(); + try { + assertTrue("drainer did not capture the pre-ACK watermark", + staleAckPolled.await(5, TimeUnit.SECONDS)); + releaseFinalAck.countDown(); + runner.join(10_000L); + } finally { + releaseFinalAck.countDown(); + if (runner.isAlive()) { + drainer.requestStop(); + runner.join(5_000L); + } + } + + assertFalse("drainer did not finish after observing the final ACK", runner.isAlive()); + assertEquals(BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("a fully acknowledged orphan slot must not be quarantined", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("the final ACK must stop the drainer before it starts another connect sweep", + 2, factory.attempts()); + } + }); + } + + @Test + public void testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // The settle budget counts CONSECUTIVE capability-gap sweeps. Held per drain with no notion + // of progress, it instead spans every session: two gap windows with a DELIVERING session + // between them accumulate toward one threshold, so 16 sweeps that were never consecutive + // quarantine a slot the cluster is still draining - and nothing in production clears the + // .failed sentinel, so those replayable rows are abandoned for good. That is the rolling + // upgrade this budget exists to ride out, reaching the opposite verdict. + // + // Two windows of 9 (18 cumulative, past the threshold of 16), never more than 9 in a row, + // with a session that durably acks between them. + final int windowLength = BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS - 7; + final int firstGapFrom = 2; + final int firstGapTo = firstGapFrom + windowLength - 1; // 2..10 + final int deliveringAttempt = firstGapTo + 1; // 11 + final int secondGapFrom = deliveringAttempt + 1; // 12 + final int secondGapTo = secondGapFrom + windowLength - 1; // 20 + assertTrue("the two windows must exceed the threshold that only consecutive sweeps may reach", + 2 * windowLength >= BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS); + assertTrue("neither window may reach it on its own", + windowLength < BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS); + + seedSlot(SEEDED_FRAMES); + // Connection 1 acks frame 0 then drops; connection 2 - the delivering session between the + // windows - acks frames 0 and 1, advancing the watermark, then drops too. + java.util.Map drops = new java.util.HashMap<>(); + drops.put(1, 0L); + drops.put(2, 1L); + try (TestWebSocketServer server = new TestWebSocketServer(new GapScenarioHandler(drops), true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), + n -> (n >= firstGapFrom && n <= firstGapTo) + || (n >= secondGapFrom && n <= secondGapTo)); + BackgroundDrainer drainer = newDrainer(factory); + CountingListener listener = new CountingListener(); + drainer.setListener(listener); + + runToCompletion(drainer); + + assertEquals("delivering between the windows ends the episode, so neither window reaches " + + "the threshold and the slot must still drain [attempts=" + + factory.attempts() + "]", + BackgroundDrainer.DrainOutcome.SUCCESS, drainer.outcome()); + assertFalse("a slot the cluster is still draining must not be quarantined", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("no persistent-failure escalation may be reported", 0, + listener.persistentFailures.get()); + assertTrue("both gap windows must actually have been driven [attempts=" + + factory.attempts() + "]", factory.attempts() > secondGapTo); + // The observability callback fires per gap sweep and must show the counter restarting + // rather than running on to the threshold. + assertTrue("the second window must restart the count, not continue the first " + + "[unavailableAttempts=" + listener.unavailableAttempts + "]", + java.util.Collections.max(listener.unavailableAttempts) <= windowLength); + } + }); + } + + @Test + public void testConnectingWithoutDeliveringDoesNotGrantAFreshSettleBudget() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // The negative twin of testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget, and the + // half that pins noteAckProgress()'s guard rather than its effect. A durable ack PAST the + // watermark ends the episode; merely reaching a node and drawing a session must not, because + // a session that delivers nothing is no evidence the cluster can drain this slot. run()'s + // poll loop calls noteAckProgress() on every 50ms tick, so weakening `acked <= watermark` to + // `acked < watermark` - or dropping the guard - refills all three escalation counters twenty + // times a second for as long as the drain is connected. The orphan drainer then sweeps + // forever with no ack progress: no .failed sentinel, no DATA_LOSS report, the slot lock held + // and one of max_background_drainers workers pinned for the life of the process. + // + // Same shape as the delivering twin - two windows of 9, neither reaching the threshold of 16 + // alone, 18 cumulative - with the session between them acking nothing new. + final int windowLength = BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS - 6; + final int firstGapFrom = 2; + final int firstGapTo = firstGapFrom + windowLength - 1; // 2..11 + final int silentAttempt = firstGapTo + 1; // 12 + final int secondGapFrom = silentAttempt + 1; // 13 + final int secondGapTo = secondGapFrom + windowLength - 1; // 22 + assertTrue("the two windows must exceed the threshold that only consecutive sweeps may reach", + 2 * windowLength >= BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS); + assertTrue("neither window may reach it on its own", + windowLength < BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS); + + seedSlot(SEEDED_FRAMES); + // Connection 1 acks seq 0 then drops. Connection 2 - the session between the windows - takes + // the first frame and closes without acking anything (-1 puts the handler's drop at seq 0), so + // it connects, runs, and leaves the engine's durable-ack watermark exactly where it found it. + // That is the state the guard has to recognise. + java.util.Map drops = new java.util.HashMap<>(); + drops.put(1, 0L); + drops.put(2, -1L); + try (TestWebSocketServer server = new TestWebSocketServer(new GapScenarioHandler(drops), true)) { + server.start(); + assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + ScriptedWireFactory factory = new ScriptedWireFactory(server.getPort(), + n -> (n >= firstGapFrom && n <= firstGapTo) + || (n >= secondGapFrom && n <= secondGapTo)); + BackgroundDrainer drainer = newDrainer(factory); + CountingListener listener = new CountingListener(); + drainer.setListener(listener); + + runToCompletion(drainer); + + assertEquals("a session that delivered nothing must not refill the settle budget, so the " + + "two windows accumulate and the slot is quarantined [attempts=" + + factory.attempts() + ", unavailableAttempts=" + + listener.unavailableAttempts + "]", + BackgroundDrainer.DrainOutcome.FAILED, drainer.outcome()); + assertTrue("the exhausted settle budget must drop the .failed sentinel", + Files.exists(slotPath + "/" + OrphanScanner.FAILED_SENTINEL_NAME)); + assertEquals("escalation must go through the settle budget, not the generic wire-error path", + 1, listener.persistentFailures.get()); + assertEquals(BackgroundDrainer.DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS, + listener.lastPersistentTotalAttempts.get()); + // and the count must have run ON through the silent session rather than restarting at it + assertTrue("the second window must continue the first, not restart it [unavailableAttempts=" + + listener.unavailableAttempts + "]", + java.util.Collections.max(listener.unavailableAttempts) > windowLength); + } + }); + } + @Test public void testMidDrainPersistentCapabilityGapExhaustsBudgetThenQuarantines() throws Exception { TestUtils.assertMemoryLeak(() -> { @@ -300,13 +483,22 @@ public synchronized void onDurableAckUnavailable(String slotPath, int attemptNum */ private static final class GapScenarioHandler implements TestWebSocketServer.WebSocketServerHandler { private static final String TABLE = "trades"; - private final boolean dropFirstConnection; private final List arrivalOrder = new ArrayList<>(); + // connection index (1-based, in arrival order) -> the last per-connection seq it acks before + // closing the wire. A connection absent from the map acks everything it is sent. + private final java.util.Map dropAfterSeqByConnection; private final java.util.Map wireSeqByConn = new java.util.IdentityHashMap<>(); GapScenarioHandler(boolean dropFirstConnection) { - this.dropFirstConnection = dropFirstConnection; + this.dropAfterSeqByConnection = new java.util.HashMap<>(); + if (dropFirstConnection) { + this.dropAfterSeqByConnection.put(1, 0L); + } + } + + GapScenarioHandler(java.util.Map dropAfterSeqByConnection) { + this.dropAfterSeqByConnection = dropAfterSeqByConnection; } @Override @@ -320,14 +512,15 @@ public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler clien int connectionIndex = arrivalOrder.indexOf(client) + 1; long seq = counter[0]++; try { - if (dropFirstConnection && connectionIndex == 1) { - if (seq == 0) { + Long dropAfterSeq = dropAfterSeqByConnection.get(connectionIndex); + if (dropAfterSeq != null) { + if (seq <= dropAfterSeq) { client.sendBinary(okFrame(seq, seq)); client.sendBinary(durableAckFrame(seq)); - } else if (seq == 1) { + } else if (seq == dropAfterSeq + 1) { client.close(); // mid-drain wire drop } - // seq > 1: late buffered frames from the condemned + // beyond that: late buffered frames from the condemned // connection; ignore. } else { client.sendBinary(okFrame(seq, seq)); @@ -365,6 +558,41 @@ private static byte[] okFrame(long wireSeq, long seqTxn) { } } + /** + * Gates the first ACK until the drainer has captured its old watermark, then delegates normal ACK + * framing to {@link GapScenarioHandler} and closes immediately after the target. Keeping this wrapper + * separate leaves the shared gap fixture's timing unchanged for its existing dwell tests. + */ + private static final class FinalAckThenCloseHandler implements TestWebSocketServer.WebSocketServerHandler { + private final GapScenarioHandler delegate = new GapScenarioHandler(false); + private final CountDownLatch firstAckGate; + private final long targetSequence; + private long sequence; + + private FinalAckThenCloseHandler(CountDownLatch firstAckGate, long targetSequence) { + this.firstAckGate = firstAckGate; + this.targetSequence = targetSequence; + } + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (sequence == 0) { + try { + if (!firstAckGate.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("timed out waiting to release the final ACK scenario"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("interrupted while waiting to release the final ACK scenario", e); + } + } + delegate.onBinaryMessage(client, data); + if (sequence++ == targetSequence) { + client.close(); + } + } + } + /** * Per-call-index scripted factory over a real wire. Call indexes inside * {@code [throwFrom, throwTo]} (1-based, inclusive) throw the scripted @@ -374,10 +602,9 @@ private static byte[] okFrame(long wireSeq, long seqTxn) { */ private static final class ScriptedWireFactory implements CursorWebSocketSendLoop.ReconnectFactory { private final AtomicInteger calls = new AtomicInteger(); + private final java.util.function.IntPredicate isGapAttempt; private final int port; private final ThrowableSupplier throwSupplier; - private final int throwFrom; - private final int throwTo; ScriptedWireFactory(int port, int throwFrom, int throwTo) { this(port, throwFrom, throwTo, @@ -385,9 +612,20 @@ private static final class ScriptedWireFactory implements CursorWebSocketSendLoo } ScriptedWireFactory(int port, int throwFrom, int throwTo, ThrowableSupplier throwSupplier) { + this(port, n -> n >= throwFrom && n <= throwTo, throwSupplier); + } + + // General form: an arbitrary set of gap attempts, so a scenario can script more than one + // contiguous window and put a delivering session between them. + ScriptedWireFactory(int port, java.util.function.IntPredicate isGapAttempt) { + this(port, isGapAttempt, + () -> new QwpDurableAckMismatchException("localhost", port, "primary")); + } + + ScriptedWireFactory(int port, java.util.function.IntPredicate isGapAttempt, + ThrowableSupplier throwSupplier) { this.port = port; - this.throwFrom = throwFrom; - this.throwTo = throwTo; + this.isGapAttempt = isGapAttempt; this.throwSupplier = throwSupplier; } @@ -398,7 +636,7 @@ int attempts() { @Override public WebSocketClient reconnect() throws Exception { int n = calls.incrementAndGet(); - if (n >= throwFrom && n <= throwTo) { + if (isGapAttempt.test(n)) { Throwable t = throwSupplier.get(); if (t instanceof RuntimeException) throw (RuntimeException) t; if (t instanceof Exception) throw (Exception) t; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java index d9aa508d6..ec828aae2 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoopCatchUpAlignmentTest.java @@ -1534,23 +1534,6 @@ private List captureCatchUpFramesWithOneLargeSymbol( return client.capturedFrames; } - /** - * Reassembles the frames captured since the last call through the same - * {@link QwpWireTestUtils#accumulateDeltaDictionary} the end-to-end tests' - * handler uses -- with {@code allowGap=true}, so a hole surfaces as a null - * entry here instead of raising {@code DictionaryGapException} the way a - * real server now would -- and asserts the result is the seeded dictionary, - * dense and in order. - *

- * This is what frame counting cannot do. A catch-up split ships its chunks as - * {@code [deltaStart, deltaStart+count)} ranges that must tile {@code [0, n)} - * exactly; an off-by-one in the walk's start id keeps the frame COUNT intact - * while overlapping a range (an id silently takes its neighbour's symbol) or - * skipping one (surfaced here as a null entry; against a real server that id - * would instead be REJECTED as a dictionary gap). Comparing the reassembled - * dictionary catches all three shapes -- overlap, gap and shift -- because it - * compares content per id, not just the ranges. - */ /** * As {@link #assertCatchUpReassembles(CatchUpCapturingClient, String...)}, but for * {@link #captureCatchUpFrames} / {@link #captureCatchUpFramesWithOneLargeSymbol}, @@ -1572,6 +1555,23 @@ private static void assertCatchUpReassembles(List frames, int expectedCo } } + /** + * Reassembles the frames captured since the last call through the same + * {@link QwpWireTestUtils#accumulateDeltaDictionary} the end-to-end tests' + * handler uses -- with {@code allowGap=true}, so a hole surfaces as a null + * entry here instead of raising {@code DictionaryGapException} the way a + * real server now would -- and asserts the result is the seeded dictionary, + * dense and in order. + *

+ * This is what frame counting cannot do. A catch-up split ships its chunks as + * {@code [deltaStart, deltaStart+count)} ranges that must tile {@code [0, n)} + * exactly; an off-by-one in the walk's start id keeps the frame COUNT intact + * while overlapping a range (an id silently takes its neighbour's symbol) or + * skipping one (surfaced here as a null entry; against a real server that id + * would instead be REJECTED as a dictionary gap). Comparing the reassembled + * dictionary catches all three shapes -- overlap, gap and shift -- because it + * compares content per id, not just the ranges. + */ private static void assertCatchUpReassembles(CatchUpCapturingClient client, String... expected) { List rebuilt = new ArrayList<>(); for (byte[] frame : client.capturedFrames) { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/EngineCloseSlotLockReleaseTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/EngineCloseSlotLockReleaseTest.java index 3233ac4c1..0c78e7777 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/EngineCloseSlotLockReleaseTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/EngineCloseSlotLockReleaseTest.java @@ -32,14 +32,14 @@ import io.questdb.client.cutlass.qwp.client.sf.cursor.SlotLock; import io.questdb.client.std.Files; import io.questdb.client.test.tools.TestUtils; -import org.junit.After; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; import java.lang.reflect.Field; import java.net.InetAddress; import java.net.ServerSocket; -import java.nio.file.Paths; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -67,10 +67,10 @@ * the dead engine. * *

The test injects an NPE into {@code ring.close()} by reflectively - * setting the engine's {@code ring} field to {@code null}. The current - * code propagates the NPE before reaching slotLock cleanup. After the - * fix (wrap the close steps in try/finally so slotLock.close() always - * runs), the slot is releasable by a fresh sender and the test goes green. + * setting the engine's {@code ring} field to {@code null}. A close that + * propagates that NPE before reaching slotLock cleanup is the regression + * this pins; the close steps run under try/finally so slotLock.close() + * always runs, which is what leaves the slot releasable by a fresh sender. * *

The end-to-end signal is "can a fresh {@code SlotLock.acquire} on * the same slot dir succeed?" — the user-visible consequence of a leaked @@ -80,45 +80,18 @@ public class EngineCloseSlotLockReleaseTest { private String sfDir; + // one shared temp-directory mechanism instead of a per-class java.io.tmpdir path plus a hand-rolled + // recursive delete: the rule cleans up on failure and on an exception thrown out of a test too + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + @Before public void setUp() { - sfDir = Paths.get(System.getProperty("java.io.tmpdir"), - "qdb-engine-close-leak-" + System.nanoTime()).toString(); + sfDir = temp.getRoot().toPath().resolve("slot").toString(); assertEquals(0, Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT)); } - @After - public void tearDown() { - if (sfDir == null) return; - rmDirRecursive(sfDir); - } - private static void rmDirRecursive(String dir) { - if (!Files.exists(dir)) return; - long find = Files.findFirst(dir); - if (find > 0) { - try { - int rc = 1; - while (rc > 0) { - String name = Files.utf8ToString(Files.findName(find)); - if (name != null && !".".equals(name) && !"..".equals(name)) { - String child = dir + "/" + name; - long probe = Files.findFirst(child); - if (probe > 0) { - Files.findClose(probe); - rmDirRecursive(child); - } else { - Files.remove(child); - } - } - rc = Files.findNext(find); - } - } finally { - Files.findClose(find); - } - } - Files.remove(dir); - } /** * A close driven by a caller that HOLDS the logical slot lock must not unlink it. diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java index 304fffa06..908e3bd56 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java @@ -42,8 +42,10 @@ import java.security.MessageDigest; import java.util.Base64; import java.util.List; +import java.util.concurrent.BlockingQueue; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; @@ -55,6 +57,9 @@ public class TestWebSocketServer implements Closeable { private static final Logger LOG = LoggerFactory.getLogger(TestWebSocketServer.class); private static final String WEBSOCKET_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + // Authorization header value captured from each well-formed upgrade request ("" when absent), in + // arrival order. Tests poll this to assert the token a provider supplied at each (re)handshake. + private final BlockingQueue capturedAuthHeaders = new LinkedBlockingQueue<>(); private final List clients = new CopyOnWriteArrayList<>(); private final boolean emitDurableAckHeader; private final WebSocketServerHandler handler; @@ -69,6 +74,11 @@ public class TestWebSocketServer implements Closeable { private final AtomicBoolean running = new AtomicBoolean(false); private final ServerSocket serverSocket; private final CountDownLatch startLatch = new CountDownLatch(1); + // Number of arbitrary HTTP-status rejects that were fully written to the + // client. Tests use this rather than an observed token pull: the provider + // runs before the upgrade response arrives, so a pull alone does not prove + // that the reconnect loop has entered its rejection episode. + private final AtomicInteger statusRejectCount = new AtomicInteger(); // Monotonic count of completed handshakes over the server's lifetime. Unlike // liveConnections it never decrements, so a test can confirm how many clients // connected even after they have all disconnected. @@ -232,6 +242,14 @@ public int liveConnectionCount() { return liveConnections.get(); } + /** + * Authorization header value seen on the next upgrade handshake ("" if the request carried none), + * in arrival order. Blocks up to the timeout for a handshake to arrive; returns null on timeout. + */ + public String pollAuthorizationHeader(long timeout, TimeUnit unit) throws InterruptedException { + return capturedAuthHeaders.poll(timeout, unit); + } + /** * Number of HTTP 421 role-reject responses sent over the server's lifetime. */ @@ -239,6 +257,13 @@ public int roleRejectCount() { return roleRejectCount.get(); } + /** + * Number of arbitrary HTTP-status reject responses sent over the server's lifetime. + */ + public int statusRejectCount() { + return statusRejectCount.get(); + } + /** * Advertises {@code X-QWP-Max-Batch-Size: } on subsequent * handshakes (live update). Pass {@code 0} to stop advertising a cap. @@ -557,6 +582,7 @@ private boolean performHandshake() throws IOException { } String key = null; + String authorization = ""; String[] lines = request.toString().split("\r\n"); if (lines.length > 0) { // GET HTTP/1.1 @@ -566,15 +592,18 @@ private boolean performHandshake() throws IOException { } } for (String line : lines) { - if (line.toLowerCase().startsWith("sec-websocket-key:")) { + String lower = line.toLowerCase(); + if (lower.startsWith("sec-websocket-key:")) { key = line.substring(18).trim(); - break; + } else if (lower.startsWith("authorization:")) { + authorization = line.substring("authorization:".length()).trim(); } } if (key == null) { return false; } + capturedAuthHeaders.add(authorization); // Read-path reject: drop the egress upgrade before the 101 so the // query pool's connect fails fast, while ingest write-path upgrades @@ -595,6 +624,7 @@ private boolean performHandshake() throws IOException { "\r\n"; out.write(sb.getBytes(StandardCharsets.US_ASCII)); out.flush(); + statusRejectCount.incrementAndGet(); return false; } // Role-aware reject path: emit a 421 Misdirected Request + diff --git a/core/src/test/java/io/questdb/client/test/example/OIDCAuthExample.java b/core/src/test/java/io/questdb/client/test/example/OIDCAuthExample.java new file mode 100644 index 000000000..80a1bab6d --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/example/OIDCAuthExample.java @@ -0,0 +1,97 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.example; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.FileTokenStore; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; + +import java.time.Instant; +import java.time.temporal.ChronoUnit; + +public class OIDCAuthExample { + public static void main(String[] args) { + + // Discover the client id, scope and endpoints from the QuestDB server's /settings: + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB( + "http://localhost:9000", + new OidcDeviceAuth.DiscoveryOptions() + .allowInsecureTransport(true) + .tokenStore(FileTokenStore.atDefaultLocation()) + )) { + // one-time interactive sign-in; caches token + refresh token + auth.signIn(); + + // ingress - ILP over HTTP + try (Sender sender = Sender.builder(Sender.Transport.HTTP) + .address("localhost:9000") + .httpTokenProvider(auth::getToken) + .build()) { + sender.table("abcde") + .longColumn("c0", 25) + .atNow(); + } + + // ingress - QWP + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:9000") + .httpTokenProvider(auth::getToken) + .build()) { + sender.table("abcde") + .longColumn("c0", 28) + .atNow(); + } + + // egress - QWP + CollectingHandler handler = new CollectingHandler(); + try (QwpQueryClient client = QwpQueryClient.newPlainText("localhost", 9000) + .withBearerTokenProvider(auth::getToken)) { + client.connect(); + client.execute("SELECT c0, ts FROM abcde", handler); + } + } + } + + static final class CollectingHandler implements QwpColumnBatchHandler { + public void onBatch(QwpColumnBatch batch) { + batch.forEachRow(row -> { + long c0 = row.getLongValue(0); + // QuestDB TIMESTAMP columns arrive as microseconds since the Unix epoch + Instant ts = Instant.EPOCH.plus(row.getLongValue(1), ChronoUnit.MICROS); + System.out.printf("%d %s%n", c0, ts); + }); + } + + public void onEnd(long totalRows) { + } + + public void onError(byte status, String message) { + System.err.println("query failed: status=" + status + " msg=" + message); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/ConfStringParserTest.java b/core/src/test/java/io/questdb/client/test/impl/ConfStringParserTest.java index e25a2095b..b1f93d257 100644 --- a/core/src/test/java/io/questdb/client/test/impl/ConfStringParserTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/ConfStringParserTest.java @@ -82,6 +82,13 @@ public void testInvalidCtrlCharsInValue() { pos = ConfStringParser.value(config, pos, sink); Assert.assertTrue(pos < 0); TestUtils.assertContains(sink, "invalid character"); + // ...and that the offending char is RENDERED as an escape rather than spliced in raw. This is + // the only production caller of putAsPrintable(char), and asserting the prefix alone let that + // overload emit anything at all: a config string carrying an ESC or a bidi override would then + // rewrite the terminal of whoever read the parse error. + TestUtils.assertContains(sink, String.format("\\u%04x", badChar)); + Assert.assertTrue("the raw control char must not reach the message", + sink.toString().indexOf(badChar) < 0); TestUtils.assertContains(sink, "at position 11"); assertNoNext(config, pos); } diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java index 0dd6ee756..1af8176dc 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java @@ -77,6 +77,52 @@ public void testClientGetterReturnsConstructorInstance() throws Exception { }); } + /** + * {@code shutdown()} clears the caller's interrupt for the whole teardown and hands it back at the + * end. The clear is what stops every {@code join()} below throwing on arrival instead of waiting; + * the hand-back is what keeps the flag a cancellation signal for whoever set it. + *

+ * This pins the hand-back and the thread teardown. It does NOT pin the clear itself: the only + * observable difference there is whether {@code join()} waited, and with {@code thread.interrupt()} + * fired immediately before it the dispatch thread is already on its way out, so any assertion on + * "still alive on return" is a race rather than a check. That half would need a dispatch thread with a + * controllable exit latency, which is a production seam this does not justify. + */ + @Test(timeout = 30_000) + public void testShutdownHandsBackACarriedInterrupt() throws Exception { + TestUtils.assertMemoryLeak(() -> { + // A null client is enough: shutdown()'s cancel() and close() calls NPE and are swallowed by + // their own catch (Throwable), which is the documented best-effort teardown, and the dispatch + // thread this asserts on does not touch the client while parked. + QueryWorker worker = new QueryWorker(null, null, 0); + // start()/shutdown() are package-private; reached the same way the sibling tests in this class + // reach bumpGeneration() and isCurrentThreadWorker(), rather than widening them for a test + Method start = QueryWorker.class.getDeclaredMethod("start"); + start.setAccessible(true); + Method shutdown = QueryWorker.class.getDeclaredMethod("shutdown"); + shutdown.setAccessible(true); + start.invoke(worker); + + Field threadField = QueryWorker.class.getDeclaredField("thread"); + threadField.setAccessible(true); + Thread dispatch = (Thread) threadField.get(worker); + Assert.assertTrue("the dispatch thread must be running before shutdown", dispatch.isAlive()); + + Thread.currentThread().interrupt(); + try { + shutdown.invoke(worker); + Assert.assertTrue("shutdown() must hand the caller's cancellation back, or the pool loop " + + "that set it silently loses its stop signal", Thread.currentThread().isInterrupted()); + } finally { + // do not leak the flag into the next test + Thread.interrupted(); + } + + dispatch.join(TimeUnit.SECONDS.toMillis(10)); + Assert.assertFalse("shutdown() must not leave its dispatch thread running", dispatch.isAlive()); + }); + } + /** * Regression test for the shutdown-vs-dispatch race in * {@code QueryWorker.runLoop()}. If {@code shuttingDown} flips to true diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java index 81d564fd8..24e60e49b 100644 --- a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTest.java @@ -1536,6 +1536,12 @@ public void testSharedManagerPassCompletionRecoversRetiredPoolSlot() throws Exce releaseWorker.countDown(); manager.setAfterRingCleanupHook(null); manager.setBeforeInstallSyncHook(null); + // The 50 ms timeout above is only for forcing the slot- + // retirement path under test. Restore a normal teardown + // budget before closing the recovered sender: otherwise a + // slow CI runner can defer that sender's final mmap cleanup + // past assertMemoryLeak() and bleed it into the next test. + manager.setWorkerJoinTimeoutMillis(TimeUnit.SECONDS.toMillis(60)); pool.close(); manager.close(); } @@ -3387,6 +3393,92 @@ public void testDirectRecoveryFailedAttemptEntersRetryWait() throws Throwable { } } + @Test(timeout = 30_000) + public void testDirectRecoveryCloseStillEscalatesWhenCallerIsInterruptedDuringFirstJoin() throws Exception { + createCandidateSlot("default-0"); + CountDownLatch afterJoin = new CountDownLatch(1); + CountDownLatch beforeJoin = new CountDownLatch(1); + CountDownLatch closeReturned = new CountDownLatch(1); + CountDownLatch releaseWait = new CountDownLatch(1); + CountDownLatch targetInterrupted = new CountDownLatch(1); + CountDownLatch waitEntered = new CountDownLatch(1); + AtomicBoolean callerInterruptRestored = new AtomicBoolean(); + AtomicBoolean driverAliveAfterJoin = new AtomicBoolean(); + AtomicReference closeFailure = new AtomicReference<>(); + IntFunction senderFactory = idx -> { + throw new LineSenderException("injected recovery failure"); + }; + Runnable recoveryWaiter = () -> { + waitEntered.countDown(); + try { + releaseWait.await(); + } catch (InterruptedException e) { + targetInterrupted.countDown(); + Thread.currentThread().interrupt(); + } + }; + + SenderPool pool = newPoolWithRecoveryControls( + "ws::addr=localhost:1;sf_dir=" + sfDir + ";", + 0, 1, 0, senderFactory, null, recoveryWaiter, null); + Thread recoveryThread = pool.getStartupRecoveryThreadForTesting(); + pool.setStartupRecoveryJoinHooksForTesting( + beforeJoin::countDown, + () -> { + driverAliveAfterJoin.set(recoveryThread.isAlive()); + afterJoin.countDown(); + }); + Thread closeThread = new Thread(() -> { + try { + pool.close(); + callerInterruptRestored.set(Thread.currentThread().isInterrupted()); + } catch (Throwable t) { + closeFailure.set(t); + } finally { + closeReturned.countDown(); + } + }, "test-interrupted-direct-pool-close"); + try { + Assert.assertTrue("the failed recovery must enter its retry wait", + waitEntered.await(10, TimeUnit.SECONDS)); + closeThread.start(); + Assert.assertTrue("close must reach the direct-driver join", + beforeJoin.await(10, TimeUnit.SECONDS)); + + // Do not merely arrive with a carried flag: wait until close is actually inside the first timed + // join, then interrupt it. The old single try/catch jumped straight past target.interrupt() and + // the second join in precisely this interleaving. + long stateDeadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (closeThread.getState() != Thread.State.TIMED_WAITING + && System.nanoTime() < stateDeadline) { + Thread.yield(); + } + Assert.assertEquals("close never entered the first timed join", + Thread.State.TIMED_WAITING, closeThread.getState()); + closeThread.interrupt(); + + Assert.assertTrue("close must finish after escalating to the recovery driver", + closeReturned.await(10, TimeUnit.SECONDS)); + Assert.assertTrue("the interrupted first join must not skip the target interrupt", + targetInterrupted.await(1, TimeUnit.SECONDS)); + Assert.assertTrue("close must complete its second join", afterJoin.await(1, TimeUnit.SECONDS)); + if (closeFailure.get() != null) { + throw new AssertionError("close failed", closeFailure.get()); + } + Assert.assertFalse("the recovery driver must be dead after the second join", + driverAliveAfterJoin.get()); + Assert.assertFalse("the recovery driver must remain quiescent after close", + recoveryThread.isAlive()); + Assert.assertTrue("the caller's interrupt must be restored after the shutdown protocol", + callerInterruptRestored.get()); + } finally { + releaseWait.countDown(); + recoveryThread.interrupt(); + closeThread.join(TimeUnit.SECONDS.toMillis(10)); + pool.close(); + } + } + @Test public void testDirectRecoveryThreadCreationFailureClosesPrewarmedDelegates() throws Throwable { createCandidateSlot("default-2"); diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTokenProviderTest.java new file mode 100644 index 000000000..8fa12c0bd --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSfTokenProviderTest.java @@ -0,0 +1,429 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.impl; + +import io.questdb.client.HttpTokenProvider; +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.std.Files; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; + +/** + * Token-provider wiring for POOLED senders that also run store-and-forward — + * pooled WebSocket + SF + OIDC, the configuration this client is built for and + * the one combination no test covered. + *

+ * {@code SenderPool.buildManagedSlotSender} has two legs. The + * {@code !storeAndForward} leg applies the provider inline and is exercised by + * {@code QuestDBBuilderTest#testConnectTokenProviderSuppliesBothPoolsAndPoolGrowth}, + * which configures no {@code sf_dir}. The SF leg builds the delegate through the + * slot-id / orphan-exclusion / recovery-mode chain and applies the provider at the + * end of it, for both ordinary and recovery delegates — and nothing asserted either. + *

+ * What an unwired provider costs is not a missing header in isolation. Every SF + * pooled sender's upgrade would go out unauthenticated, take a 401, and hand the + * rows to store-and-forward; the operator would not learn at connect time but much + * later, through ring backpressure or a quarantined slot. On the recovery leg it is + * worse: a recovery delegate drains the PREVIOUS run's data, so an unauthenticated + * build quarantines the slot and reports {@code DATA_LOSS} for rows that were + * replayable all along. + *

+ * Both tests are black-box through the public facade — {@code QuestDB.connect(cfg, + * provider)} against a real {@link TestWebSocketServer} — and assert on the + * Authorization header the server actually received, so they hold for any wiring + * that gets the credential onto the wire. + */ +public class SenderPoolSfTokenProviderTest { + + private String sfDir; + + // one shared temp-directory mechanism instead of a per-class java.io.tmpdir path plus a hand-rolled + // recursive delete: the rule cleans up on failure and on an exception thrown out of a test too + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Before + public void setUp() { + sfDir = temp.getRoot().toPath().resolve("slot").toString(); + } + + + @Test + public void testSfPooledSendersCarryTheProviderToken() throws Exception { + // The SF leg of buildManagedSlotSender: every pooled SF sender must pull + // its own current token, exactly as the non-SF leg does. Two prewarmed + // slots, so this also pins that the provider is consulted per sender and + // not once for the pool. + TestUtils.assertMemoryLeak(() -> { + CountingAckHandler handler = new CountingAckHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + AtomicInteger tokenCalls = new AtomicInteger(); + HttpTokenProvider provider = () -> "ROTATING-" + tokenCalls.incrementAndGet(); + // query_pool_min=0 keeps the egress pool from connecting, so every + // captured header belongs to an SF pooled sender. + String cfg = "ws::addr=localhost:" + server.getPort() + ";sf_dir=" + sfDir + ";" + + "sender_pool_min=2;sender_pool_max=2;" + + "query_pool_min=0;query_pool_max=1;"; + + try (QuestDB db = QuestDB.connect(cfg, provider)) { + assertAuthorizationHeaders(server, "Bearer ROTATING-1", "Bearer ROTATING-2"); + // The senders are genuinely usable on those credentials, not + // merely upgraded: borrow both slots and ship a row through each. + try (Sender s1 = db.borrowSender(); Sender s2 = db.borrowSender()) { + s1.table("pooled").longColumn("v", 1).atNow(); + s1.flush(); + s2.table("pooled").longColumn("v", 2).atNow(); + s2.flush(); + } + Assert.assertTrue("both pooled SF senders must reach the server", + awaitAtLeast(handler.frames, 2, 10_000)); + } + Assert.assertEquals("one token pull per pooled SF sender", 2, tokenCalls.get()); + } + }); + } + + @Test(timeout = 60_000) + public void testCloseBreaksARecoveryDelegateStuckInACredentialPull() throws Exception { + TestUtils.assertMemoryLeak(() -> assertCloseBreaksBlockingCredentialPull(CloseInterruptMode.NONE)); + } + + @Test(timeout = 60_000) + public void testCloseBreaksARecoveryDelegateStuckInACredentialPullWhenTheCallerIsInterrupted() throws Exception { + TestUtils.assertMemoryLeak(() -> + assertCloseBreaksBlockingCredentialPull(CloseInterruptMode.CARRIED)); + } + + @Test(timeout = 60_000) + public void testCloseBreaksARecoveryDelegateWhenInterruptedDuringTheJoin() throws Exception { + TestUtils.assertMemoryLeak(() -> + assertCloseBreaksBlockingCredentialPull(CloseInterruptMode.DURING_JOIN)); + } + + @Test + public void testSfStartupRecoveryDelegateCarriesTheProviderToken() throws Exception { + // The forRecovery leg of buildManagedSlotSender. A recovery delegate replays + // the user's own data from a previous run, so an unauthenticated build does + // not merely fail to connect: it quarantines the slot and reports DATA_LOSS + // for rows that were replayable. + TestUtils.assertMemoryLeak(() -> { + // Phase 1 -- a server that never acks, so three frames stay unacked on + // disk under default-0 after the pool closes. + try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) { + silent.start(); + Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS)); + String cfg = "ws::addr=localhost:" + silent.getPort() + ";sf_dir=" + sfDir + ";" + + "sender_pool_min=1;sender_pool_max=1;" + + "query_pool_min=0;query_pool_max=1;" + + "close_flush_timeout_millis=500;"; + try (QuestDB db = QuestDB.connect(cfg, () -> "PHASE1-TOKEN")) { + try (Sender s = db.borrowSender()) { + for (int i = 0; i < 3; i++) { + s.table("recover").longColumn("v", i).atNow(); + s.flush(); + } + } + } + } + Assert.assertTrue("unacked data must persist on disk for recovery to have work", + hasSegmentFile(sfDir + "/default-0")); + + // Phase 2 -- an ack-ing server and a brand-new pool over the same sf_dir. + // sender_pool_min=0 prewarms nothing, so the ONLY connect this server can + // see is the startup-recovery delegate's. + CountingAckHandler handler = new CountingAckHandler(); + try (TestWebSocketServer ack = new TestWebSocketServer(handler)) { + ack.start(); + Assert.assertTrue(ack.awaitStart(5, TimeUnit.SECONDS)); + + AtomicInteger tokenCalls = new AtomicInteger(); + HttpTokenProvider provider = () -> { + tokenCalls.incrementAndGet(); + return "RECOVERY-TOKEN"; + }; + String cfg = "ws::addr=localhost:" + ack.getPort() + ";sf_dir=" + sfDir + ";" + + "sender_pool_min=0;sender_pool_max=1;" + + "query_pool_min=0;query_pool_max=1;"; + + try (QuestDB db = QuestDB.connect(cfg, provider)) { + Assert.assertNotNull(db); + String header = ack.pollAuthorizationHeader(10, TimeUnit.SECONDS); + Assert.assertNotNull("the recovery delegate must connect", header); + Assert.assertEquals( + "a recovery delegate must present the provider's credential -- " + + "without it the replay is rejected and the slot is quarantined, " + + "reporting DATA_LOSS for replayable rows", + "Bearer RECOVERY-TOKEN", header); + // Tie that header to recovery rather than to any other connect: + // the previous run's frames actually reach the new server. + Assert.assertTrue("the recovered frames must be replayed", + awaitAtLeast(handler.frames, 1, 10_000)); + } + Assert.assertTrue("the recovery delegate must consult the provider", + tokenCalls.get() >= 1); + } + }); + } + + private static void assertAuthorizationHeaders( + TestWebSocketServer server, + String... expected + ) throws InterruptedException { + Set actual = new HashSet<>(); + for (int i = 0; i < expected.length; i++) { + String header = server.pollAuthorizationHeader(10, TimeUnit.SECONDS); + Assert.assertNotNull("timed out waiting for an Authorization header", header); + // The server records "" for an upgrade that carried no Authorization + // header at all -- the exact shape of an unwired provider. Name it, + // rather than letting two of them collide as a "duplicate". + Assert.assertFalse("an SF pooled sender upgraded with NO Authorization header", + header.isEmpty()); + Assert.assertTrue("duplicate Authorization header: " + header, actual.add(header)); + } + Set want = new HashSet<>(); + for (int i = 0; i < expected.length; i++) { + want.add(expected[i]); + } + Assert.assertEquals(want, actual); + } + + private void assertCloseBreaksBlockingCredentialPull(CloseInterruptMode interruptMode) throws Exception { + seedUnackedFrames(); + + CountDownLatch pullEntered = new CountDownLatch(1); + AtomicBoolean pullInterrupted = new AtomicBoolean(); + HttpTokenProvider blockingProvider = () -> { + pullEntered.countDown(); + try { + Thread.sleep(TimeUnit.MINUTES.toMillis(5)); + } catch (InterruptedException e) { + pullInterrupted.set(true); + Thread.currentThread().interrupt(); + throw new RuntimeException("credential pull cancelled"); + } + return "NEVER-ARRIVES"; + }; + + CountingAckHandler handler = new CountingAckHandler(); + try (TestWebSocketServer ack = new TestWebSocketServer(handler)) { + ack.start(); + Assert.assertTrue(ack.awaitStart(5, TimeUnit.SECONDS)); + String cfg = "ws::addr=localhost:" + ack.getPort() + ";sf_dir=" + sfDir + ";" + + "sender_pool_min=0;sender_pool_max=1;" + + "query_pool_min=0;query_pool_max=1;"; + + QuestDB db = QuestDB.connect(cfg, blockingProvider); + Assert.assertTrue("the recovery delegate must reach the credential pull", + pullEntered.await(20, TimeUnit.SECONDS)); + + long startNanos = System.nanoTime(); + boolean callerInterruptSurvived = false; + if (interruptMode == CloseInterruptMode.CARRIED) { + Thread.currentThread().interrupt(); + try { + db.close(); + } finally { + callerInterruptSurvived = Thread.interrupted(); + } + } else if (interruptMode == CloseInterruptMode.DURING_JOIN) { + AtomicBoolean closerInterruptSurvived = new AtomicBoolean(); + AtomicReference closeFailure = new AtomicReference<>(); + Thread closer = new Thread(() -> { + try { + db.close(); + } catch (Throwable t) { + closeFailure.set(t); + } finally { + closerInterruptSurvived.set(Thread.interrupted()); + } + }, "test-close-during-housekeeper-join"); + closer.start(); + boolean enteredJoin = awaitHousekeeperJoin(closer, 10_000L); + closer.interrupt(); + closer.join(30_000L); + Assert.assertTrue("close() must enter the housekeeper join before cancellation", enteredJoin); + Assert.assertFalse("close() did not finish after cancellation", closer.isAlive()); + if (closeFailure.get() != null) { + throw new AssertionError("close() failed", closeFailure.get()); + } + callerInterruptSurvived = closerInterruptSurvived.get(); + } else { + db.close(); + } + long elapsedMillis = (System.nanoTime() - startNanos) / 1_000_000L; + + Assert.assertTrue("close() must interrupt a recovery delegate parked in a credential pull, " + + "or it returns while that delegate still holds the slot flock", + pullInterrupted.get()); + if (interruptMode != CloseInterruptMode.NONE) { + Assert.assertTrue("close() must hand the caller's cancellation back rather than consume it", + callerInterruptSurvived); + } + Assert.assertTrue("close() must not wait out the parked pull; took " + elapsedMillis + "ms", + elapsedMillis < 30_000L); + } + } + + private static boolean awaitHousekeeperJoin(Thread closer, long timeoutMillis) throws InterruptedException { + long deadlineMillis = System.currentTimeMillis() + timeoutMillis; + while (closer.isAlive() && System.currentTimeMillis() < deadlineMillis) { + boolean hasJoin = false; + boolean hasStop = false; + for (StackTraceElement frame : closer.getStackTrace()) { + if (Thread.class.getName().equals(frame.getClassName()) + && frame.getMethodName().startsWith("join")) { + hasJoin = true; + } else if ("io.questdb.client.impl.PoolHousekeeper".equals(frame.getClassName()) + && "stop".equals(frame.getMethodName())) { + hasStop = true; + } + } + if (hasJoin && hasStop) { + return true; + } + Thread.sleep(1L); + } + return false; + } + + private static boolean awaitAtLeast(AtomicInteger counter, int target, long timeoutMillis) + throws InterruptedException { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (System.currentTimeMillis() < deadline) { + if (counter.get() >= target) { + return true; + } + Thread.sleep(10); + } + return counter.get() >= target; + } + + private static boolean hasSegmentFile(String slotPath) { + if (!Files.exists(slotPath)) { + return false; + } + long find = Files.findFirst(slotPath); + if (find <= 0) { + return false; + } + try { + int rc = 1; + while (rc > 0) { + String name = Files.utf8ToString(Files.findName(find)); + rc = Files.findNext(find); + if (name != null && name.endsWith(".sfa")) { + return true; + } + } + } finally { + Files.findClose(find); + } + return false; + } + + private void seedUnackedFrames() throws Exception { + try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) { + silent.start(); + Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS)); + String cfg = "ws::addr=localhost:" + silent.getPort() + ";sf_dir=" + sfDir + ";" + + "sender_pool_min=1;sender_pool_max=1;" + + "query_pool_min=0;query_pool_max=1;" + + "close_flush_timeout_millis=500;"; + try (QuestDB db = QuestDB.connect(cfg, () -> "PHASE1-TOKEN")) { + try (Sender s = db.borrowSender()) { + for (int i = 0; i < 3; i++) { + s.table("recover").longColumn("v", i).atNow(); + s.flush(); + } + } + } + } + Assert.assertTrue("unacked data must persist on disk for recovery to have work", + hasSegmentFile(sfDir + "/default-0")); + } + + private enum CloseInterruptMode { + CARRIED, + DURING_JOIN, + NONE + } + + + private static final class CountingAckHandler implements TestWebSocketServer.WebSocketServerHandler { + final AtomicInteger frames = new AtomicInteger(); + private final Map seqByClient = + new ConcurrentHashMap<>(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + frames.incrementAndGet(); + AtomicLong seq = seqByClient.computeIfAbsent(client, c -> new AtomicLong(0)); + try { + client.sendBinary(buildAck(seq.getAndIncrement())); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + static byte[] buildAck(long seq) { + byte[] buf = new byte[1 + 8 + 2]; + ByteBuffer bb = ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(seq); + bb.putShort((short) 0); + return buf; + } + } + + private static final class SilentHandler implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + // No ack -- the frames stay unacked on disk for phase 2 to recover. + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/std/NumbersTest.java b/core/src/test/java/io/questdb/client/test/std/NumbersTest.java index d30ff29a2..0467235da 100644 --- a/core/src/test/java/io/questdb/client/test/std/NumbersTest.java +++ b/core/src/test/java/io/questdb/client/test/std/NumbersTest.java @@ -270,6 +270,32 @@ public void testHexInt() { assertEquals(0xac, Numbers.parseHexInt("ac")); } + @Test + public void testParseHexLongWrapsOnOverflowAndCallersBoundIt() { + // Two's-complement, like parseHexInt beside it and the server-side Numbers of the same name, whose + // Long256 decoding depends on the wrap. io.questdb.client.std is an exported package, so this is a + // shipped contract and not an internal detail. + assertEquals(Long.MAX_VALUE, Numbers.parseHexLong("7fffffffffffffff")); + assertEquals(0L, Numbers.parseHexLong("0")); + assertEquals(0xacL, Numbers.parseHexLong("ac")); + // leading zeros carry no magnitude + assertEquals(1L, Numbers.parseHexLong("000000000000000000001")); + // range form + assertEquals(0xf0L, Numbers.parseHexLong("xxF0yy", 2, 4)); + + // The wrap itself, in the three shapes that break a length-prefixed format differently. Pinning the + // VALUES rather than a rejection is the point: a caller that parses a count it did not choose has + // to bound the digits before it gets here, because none of these is distinguishable afterwards - + // 10000000000000000 in particular is indistinguishable from a genuine 0. + assertEquals(Long.MIN_VALUE, Numbers.parseHexLong("8000000000000000")); // negative residue + assertEquals(0L, Numbers.parseHexLong("10000000000000000")); // zero residue + assertEquals(1L, Numbers.parseHexLong("10000000000000001")); // positive residue + assertEquals(-1L, Numbers.parseHexLong("ffffffffffffffff")); // the full-width word + + // an empty sequence is still an error rather than a zero + assertHexLongRejected(""); + } + @Test public void testIntEdge() { Numbers.append(sink, Integer.MAX_VALUE); @@ -711,4 +737,12 @@ private static void assertParseLongException(String input) { } catch (NumericException ignore) { } } + + private static void assertHexLongRejected(String hex) { + try { + long parsed = Numbers.parseHexLong(hex); + Assert.fail("expected [" + hex + "] to be rejected, got " + parsed); + } catch (NumericException expected) { + } + } } diff --git a/core/src/test/java/io/questdb/client/test/std/str/DirectUtf8SinkTest.java b/core/src/test/java/io/questdb/client/test/std/str/DirectUtf8SinkTest.java index 410f2487e..ad8860284 100644 --- a/core/src/test/java/io/questdb/client/test/std/str/DirectUtf8SinkTest.java +++ b/core/src/test/java/io/questdb/client/test/std/str/DirectUtf8SinkTest.java @@ -39,6 +39,10 @@ public class DirectUtf8SinkTest extends AbstractTest { + // DirectByteSink.implCreate allocates at least this much however small a capacity it is asked for, so a + // test that means to exercise growth has to write past it + private static final int MIN_ALLOCATED_CAPACITY = 32; + @Test public void testAsAsciiCharSequence() { try (DirectUtf8Sink sink = new DirectUtf8Sink(4)) { @@ -121,6 +125,78 @@ public void testDirectUtf8Sequence() { } } + @Test + public void testPutByteArrayRange() { + try (DirectUtf8Sink sink = new DirectUtf8Sink(4)) { + final byte[] src = "abcdefgh".getBytes(StandardCharsets.UTF_8); + + // a partial range [2, 5) copies exactly "cde" + sink.put(src, 2, 5); + Assert.assertEquals(3, sink.size()); + TestUtils.assertEquals("cde".getBytes(StandardCharsets.UTF_8), sink); + // the bulk overload sets the ascii hint to false conservatively, even for ascii bytes + Assert.assertFalse(sink.isAscii()); + + // an empty range [3, 3) is a no-op + final int sizeBefore = sink.size(); + sink.put(src, 3, 3); + Assert.assertEquals(sizeBefore, sink.size()); + + // a full range [0, len) appends the whole array + sink.clear(); + sink.put(src, 0, src.length); + Assert.assertEquals(src.length, sink.size()); + TestUtils.assertEquals(src, sink); + } + } + + @Test + public void testPutByteArrayRangeGrowsTheSink() { + // DirectByteSink's native create allocates a MINIMUM of 32 bytes however small a capacity it is + // asked for, so a handful of bytes into a new DirectUtf8Sink(4) never reallocates - the sibling test + // above used to claim it did. Cross the floor for real: a range longer than 32 bytes must reallocate + // mid-copy, and the whole payload must survive that move, contiguous and in order. + final byte[] src = new byte[MIN_ALLOCATED_CAPACITY * 4]; + for (int i = 0; i < src.length; i++) { + src[i] = (byte) ('a' + (i % 26)); + } + try (DirectUtf8Sink sink = new DirectUtf8Sink(4)) { + // seed a few bytes first, so the growing copy has existing content to preserve rather than + // starting from an empty sink + sink.put(src, 0, 3); + final int lo = 3; + final int hi = lo + MIN_ALLOCATED_CAPACITY + 17; // comfortably past the floor, and not a round number + sink.put(src, lo, hi); + + Assert.assertTrue("preconditions: the payload must exceed the 32-byte floor", + sink.size() > MIN_ALLOCATED_CAPACITY); + Assert.assertEquals(3 + (hi - lo), sink.size()); + final byte[] expected = new byte[3 + (hi - lo)]; + System.arraycopy(src, 0, expected, 0, 3); + System.arraycopy(src, lo, expected, 3, hi - lo); + TestUtils.assertEquals(expected, sink); + + // and it keeps growing across repeated appends, not just the first reallocation + for (int i = 0; i < 8; i++) { + sink.put(src, 0, src.length); + } + Assert.assertEquals(3 + (hi - lo) + 8 * src.length, sink.size()); + Assert.assertEquals((byte) src[0], sink.byteAt(3 + (hi - lo))); + } + } + + @Test + public void testPutByteArrayRangeRejectsBadBounds() { + try (DirectUtf8Sink sink = new DirectUtf8Sink(4)) { + final byte[] src = {1, 2, 3}; + // a bad range must throw rather than run an unchecked native copy (asserts are off in client apps) + assertBadRange(sink, src, -1, 2); // lo < 0 + assertBadRange(sink, src, 0, 4); // hi > len + assertBadRange(sink, src, 2, 1); // lo > hi + Assert.assertEquals("a rejected put must not advance the sink", 0, sink.size()); + } + } + @Test public void testPutUtf8Sequence() { try (DirectUtf8Sink sink = new DirectUtf8Sink(1)) { @@ -208,6 +284,15 @@ public void testUtf8Sequence() { } } + private static void assertBadRange(DirectUtf8Sink sink, byte[] src, int lo, int hi) { + try { + sink.put(src, lo, hi); + Assert.fail("expected IndexOutOfBoundsException for lo=" + lo + ", hi=" + hi); + } catch (IndexOutOfBoundsException expected) { + // ok: the public overload range-checks before the native copy + } + } + private static void assertUtf8Encoding(DirectUtf8Sink sink, String s) { sink.clear(); sink.put(s); diff --git a/core/src/test/java/io/questdb/client/test/std/str/DisplaySafeTest.java b/core/src/test/java/io/questdb/client/test/std/str/DisplaySafeTest.java new file mode 100644 index 000000000..b8e5c75b0 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/std/str/DisplaySafeTest.java @@ -0,0 +1,119 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.std.str; + +import io.questdb.client.std.str.DisplaySafe; +import org.junit.Assert; +import org.junit.Test; + +/** + * Direct coverage for {@link DisplaySafe}, the single source of truth for whether a code point may be shown + * verbatim in a terminal or a log line. Both {@code Utf16Sink.putAsPrintable} and the OIDC display sanitizer + * delegate to it, so a regression here would silently weaken every display-escaping path - yet the classifier + * was previously exercised only transitively, for the few code points the integration tests happen to use. + *

+ * The unsafe code points (controls, Unicode format chars, surrogates, bidi controls, the BOM) are written as + * hex literals so this source stays pure ASCII and carries none of the chars it asserts on. + */ +public class DisplaySafeTest { + + @Test + public void testC0C1ControlsAndDelAreUnsafe() { + // C0 (incl. TAB/LF/CR/ESC), DEL, and the C1 block: every ISO control must be escaped + int[] unsafe = {0x00, 0x07, 0x08, 0x09, 0x0A, 0x0D, 0x1B, 0x1F, 0x7F, 0x80, 0x90, 0x9F}; + for (int cp : unsafe) { + String hex = "0x" + Integer.toHexString(cp); + Assert.assertFalse("control " + hex + " must be unsafe", DisplaySafe.isDisplaySafe(cp)); + Assert.assertTrue("control " + hex + " must be unsafe", DisplaySafe.isUnsafeForDisplay(cp)); + } + } + + @Test + public void testFormatBidiAndBomAreUnsafe() { + // Cf format chars and the explicit bidi/BOM set that reorder, hide, or mark text - including the + // supplementary-plane "tag" chars that arrive as a surrogate pair and must be judged whole + int[] unsafe = { + 0x00AD, // soft hyphen + 0x200B, // zero-width space + 0x200E, 0x200F, // LRM, RLM + 0x202A, 0x202B, 0x202C, 0x202D, 0x202E, // LRE, RLE, PDF, LRO, RLO + 0x2066, 0x2067, 0x2068, 0x2069, // LRI, RLI, FSI, PDI + 0xFEFF, // BOM / zero-width no-break space + 0xE0001, // language tag + 0xE0020, 0xE007F // tag space, cancel tag + }; + for (int cp : unsafe) { + String hex = "0x" + Integer.toHexString(cp); + Assert.assertTrue("format " + hex + " must be unsafe", DisplaySafe.isUnsafeForDisplay(cp)); + Assert.assertFalse("format " + hex + " must be unsafe", DisplaySafe.isDisplaySafe(cp)); + } + } + + @Test + public void testLineAndParagraphSeparatorsAreUnsafe() { + // U+2028 LINE SEPARATOR (Zl) and U+2029 PARAGRAPH SEPARATOR (Zp) are Unicode line breaks that split a + // rendered log line in ECMAScript/GUI/JSON log consumers, yet are neither ISO control nor Cf format, + // so a tampered field could otherwise forge an apparent extra log line + int[] unsafe = {0x2028, 0x2029}; + for (int cp : unsafe) { + String hex = "0x" + Integer.toHexString(cp); + Assert.assertTrue("separator " + hex + " must be unsafe", DisplaySafe.isUnsafeForDisplay(cp)); + Assert.assertFalse("separator " + hex + " must be unsafe", DisplaySafe.isDisplaySafe(cp)); + } + } + + @Test + public void testLoneSurrogatesAreUnsafe() { + // a lone surrogate half has no displayable meaning; the code-point classifier must reject it + int[] surrogates = {0xD800, 0xDBFF, 0xDC00, 0xDFFF}; + for (int cp : surrogates) { + Assert.assertFalse("surrogate 0x" + Integer.toHexString(cp) + " must be unsafe", DisplaySafe.isDisplaySafe(cp)); + } + } + + @Test + public void testPrintableAsciiIsSafe() { + // the fast-path range 0x20..0x7e is the overwhelmingly common case and must always pass + for (int cp = 0x20; cp <= 0x7e; cp++) { + String hex = "0x" + Integer.toHexString(cp); + Assert.assertTrue("printable ASCII " + hex + " must be safe", DisplaySafe.isDisplaySafe(cp)); + Assert.assertFalse("printable ASCII " + hex + " must be safe", DisplaySafe.isUnsafeForDisplay(cp)); + } + } + + @Test + public void testPrintableSupplementaryCharsAreSafe() { + // a normal supplementary char (emoji, CJK extension) is neither control, format, nor surrogate and + // stays safe, so the classifier does not over-escape legitimate non-BMP text + int[] safe = { + 0x1F600, // grinning face emoji + 0x1F4A9, // pile of poo + 0x20000 // CJK Extension B ideograph + }; + for (int cp : safe) { + Assert.assertTrue("supplementary 0x" + Integer.toHexString(cp) + " must be safe", DisplaySafe.isDisplaySafe(cp)); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/std/str/StringSinkWipeTest.java b/core/src/test/java/io/questdb/client/test/std/str/StringSinkWipeTest.java new file mode 100644 index 000000000..70e4ae417 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/std/str/StringSinkWipeTest.java @@ -0,0 +1,157 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.std.str; + +import io.questdb.client.std.str.StringSink; +import org.junit.Assert; +import org.junit.Test; + +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.List; + +/** + * Covers {@link StringSink#wipe()}, the hygiene primitive the OIDC client uses to stop a token remaining + * legible in a reusable sink after the instance that read it is closed. + */ +public class StringSinkWipeTest { + + @Test + public void testClearLeavesTheTailLegibleAndWipeDoesNot() { + // subSequence() reads the backing array directly rather than the write position, so it can see what + // clear() left behind - which is exactly the retention wipe() exists to close, demonstrated here + // rather than asserted. + StringSink sink = new StringSink(); + sink.put("REFRESH-TOKEN-abcdef0123456789"); + final int held = sink.length(); + sink.clear(); + sink.put("ok"); // a short write after a long secret: the position rewinds, the characters do not + + Assert.assertEquals("ok", sink.toString()); + Assert.assertTrue("clear() only rewinds, so the tail is still readable: " + sink.subSequence(0, held), + sink.subSequence(0, held).toString().contains("TOKEN-abcdef")); + + sink.wipe(); + + Assert.assertEquals(0, sink.length()); + Assert.assertEquals("", sink.toString()); + Assert.assertFalse("wipe() must overwrite the whole buffer, not just rewind: " + + sink.subSequence(0, held), + sink.subSequence(0, held).toString().contains("TOKEN")); + } + + @Test + public void testGrowthZeroesTheBufferItAbandons() throws Exception { + // wipe() can only reach the buffer the sink currently holds. Growth replaces that buffer, so every + // generation left behind used to keep its contents legible on the heap - the collector is under no + // obligation to overwrite them, and a heap dump taken meanwhile shows the lot. + final Field bufferField = StringSink.class.getDeclaredField("buffer"); + bufferField.setAccessible(true); + + StringSink sink = new StringSink(16); + // exactly fills the initial buffer, so no growth has happened yet + sink.put("SECRET-0123456789".substring(0, 16)); + final char[] abandoned = (char[]) bufferField.get(sink); + Assert.assertEquals(16, abandoned.length); + Assert.assertEquals("precondition: the secret really is in this array", + "SECRET-012345678", new String(abandoned)); + + // one more character forces the grow-and-copy + sink.put('9'); + Assert.assertNotSame("precondition: the sink must have moved to a new array", + abandoned, bufferField.get(sink)); + + Assert.assertEquals("the array growth abandoned still holds the secret it carried: " + + new String(abandoned).trim(), + "", new String(abandoned).replace((char) 0, ' ').trim()); + // and the live sink is intact + Assert.assertEquals("SECRET-0123456789", sink.toString()); + } + + @Test + public void testNoGenerationKeepsTheTokenAfterAFormBodyIsBuiltAndWiped() throws Exception { + // The shape that matters: OidcDeviceAuth's formSink is a default 16-char sink that builds the + // refresh POST body. It is already holding the whole refresh token by the time the later parameters + // make it grow again, so each hand-off carried a full copy - and wipe() at close() reached only the + // last one. + final String token = "REFRESH-TOKEN-abcdef0123456789"; + final Field bufferField = StringSink.class.getDeclaredField("buffer"); + bufferField.setAccessible(true); + + StringSink formSink = new StringSink(); + final List generations = new ArrayList<>(); + generations.add((char[]) bufferField.get(formSink)); + + // Sampled after EVERY write, not at the end: once the sink has moved on, the array it abandoned is + // unreachable from it, which is precisely why wipe() cannot clean them and why this has to catch + // each one as it goes. + final String[] body = { + "grant_type=refresh_token", + "&refresh_token=", token, + "&client_id=questdb", + "&scope=openid+profile+email", + }; + for (String part : body) { + formSink.put(part); + final char[] live = (char[]) bufferField.get(formSink); + if (generations.get(generations.size() - 1) != live) { + generations.add(live); + } + } + Assert.assertTrue("precondition: the sink must have grown at least twice while holding the token, " + + "or this test is not exercising the hand-off it is about", + generations.size() >= 3); + + formSink.wipe(); + + for (int i = 0; i < generations.size(); i++) { + final String contents = new String(generations.get(i)); + Assert.assertFalse( + "generation " + i + " of " + generations.size() + " still holds the refresh token after " + + "wipe(); it was abandoned by growth, so wipe() never reached it: " + + contents.replace((char) 0, '.'), + contents.contains(token)); + } + } + + @Test + public void testWipeLeavesTheSinkUsable() { + // it is a hygiene step, not a teardown: the OIDC client wipes on clearCache() and keeps going + StringSink sink = new StringSink(); + sink.put("secret"); + sink.wipe(); + sink.put("reused"); + Assert.assertEquals("reused", sink.toString()); + Assert.assertEquals(6, sink.length()); + } + + @Test + public void testWipeOfAnEmptySinkIsANoOp() { + StringSink sink = new StringSink(); + sink.wipe(); + Assert.assertEquals(0, sink.length()); + Assert.assertEquals("", sink.toString()); + } +} diff --git a/core/src/test/java/io/questdb/client/test/std/str/Utf16SinkPrintableTest.java b/core/src/test/java/io/questdb/client/test/std/str/Utf16SinkPrintableTest.java new file mode 100644 index 000000000..61759eabe --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/std/str/Utf16SinkPrintableTest.java @@ -0,0 +1,105 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.std.str; + +import io.questdb.client.std.str.StringSink; +import org.junit.Assert; +import org.junit.Test; + +/** + * Rendering coverage for {@code Utf16Sink.putAsPrintable}, at the sink rather than through an exception + * message. + *

+ * {@code DisplaySafeTest} pins the CLASSIFIER - which code points are safe - and the + * {@code LineSenderException} tests pin the {@code CharSequence} overload through one caller. Two things + * fell between them: the single-char overload, whose only production caller + * ({@code ConfStringParser}) asserts the message prefix and the position but never the escape, and the + * code points that only the sink can show are emitted correctly - U+2028, U+2029 and the BOM, which the + * classifier rejects but which no test followed through a sink. + *

+ * Both matter for the same reason the escaping exists at all: these strings are rendered into a log line + * or a terminal, and an unescaped bidi override or ANSI escape rewrites what a human reads. + */ +public class Utf16SinkPrintableTest { + + @Test + public void testPutAsPrintableCharEscapesEveryUnsafeClass() { + // C0, DEL, C1, bidi override, BOM, and a lone surrogate - one per class the classifier rejects + assertCharRenders((char) 0x00, "\\u0000"); + assertCharRenders((char) 0x1b, "\\u001b"); + assertCharRenders((char) 0x7f, "\\u007f"); + assertCharRenders((char) 0x9f, "\\u009f"); + assertCharRenders((char) 0x202e, "\\u202e"); + assertCharRenders((char) 0xfeff, "\\ufeff"); + assertCharRenders((char) 0xd800, "\\ud800"); + } + + @Test + public void testPutAsPrintableCharKeepsPrintableAscii() { + // the boundaries of the printable range, which an off-by-one on either end would escape + assertCharRenders(' ', " "); + assertCharRenders('A', "A"); + assertCharRenders('~', "~"); + } + + @Test + public void testPutAsPrintableCharUsesFourHexDigits() { + // The escape must name the char, not its low byte. An implementation that truncates renders U+202E + // as . - a full stop - which is worse than useless: it looks like ordinary text. + StringSink sink = new StringSink(); + sink.putAsPrintable((char) 0x202e); + Assert.assertEquals("\\u202e", sink.toString()); + Assert.assertNotEquals("\\u002e", sink.toString()); + } + + @Test + public void testPutAsPrintableSequenceEscapesLineAndParagraphSeparators() { + // U+2028 and U+2029 are neither C0/C1 nor Cf, so they need their own arm in the classifier; through + // a sink they must come out escaped, because a JSON or GUI log consumer treats them as line breaks + // and a tampered field could forge an apparent extra log line. + assertSequenceRenders("a" + (char) 0x2028 + "b", "a\\u2028b"); + assertSequenceRenders("a" + (char) 0x2029 + "b", "a\\u2029b"); + assertSequenceRenders("a" + (char) 0xfeff + "b", "a\\ufeffb"); + } + + private static void assertCharRenders(char c, String expected) { + StringSink sink = new StringSink(); + sink.putAsPrintable(c); + Assert.assertEquals("rendering of char 0x" + Integer.toHexString(c), expected, sink.toString()); + } + + private static void assertSequenceRenders(CharSequence input, String expected) { + StringSink sink = new StringSink(); + sink.putAsPrintable(input); + Assert.assertEquals(expected, sink.toString()); + for (int i = 0; i < input.length(); i++) { + char c = input.charAt(i); + if (c > 0x7e) { + Assert.assertTrue("the raw char 0x" + Integer.toHexString(c) + " must not survive: " + + sink, sink.toString().indexOf(c) < 0); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/tools/HandOffCharSequence.java b/core/src/test/java/io/questdb/client/test/tools/HandOffCharSequence.java new file mode 100644 index 000000000..f35fc7a40 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/tools/HandOffCharSequence.java @@ -0,0 +1,77 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.tools; + +/** + * A {@link CharSequence} that swaps its contents the instant a full scan of it completes. + *

+ * Models the hazard {@code HttpTokenProvider} explicitly invites: a provider that returns a reused + * mutable buffer. Any reader that VALIDATES the sequence and then RE-READS it to build the header has a + * window between those two reads, and a mutation landing in it passes the check and ships the mutated + * bytes - a CR/LF among them - into an {@code Authorization} header. Handing off exactly when the first + * scan finishes puts the mutation in that window deterministically, with no threads involved. + *

+ * {@link #toString()} materialises whatever the sequence currently holds, which is what a + * {@code StringSink}- or {@code StringBuilder}-backed buffer does. That is what makes the fix + * observable: a reader that snapshots BEFORE validating gets the clean value, because the snapshot is + * taken before any scan has triggered the hand-off; a reader that validates first and stringifies after + * gets the spliced one. + */ +public final class HandOffCharSequence implements CharSequence { + private final String spliced; + private CharSequence current; + private boolean handedOff; + + public HandOffCharSequence(String clean, String spliced) { + this.current = clean; + this.spliced = spliced; + } + + @Override + public char charAt(int index) { + final char c = current.charAt(index); + if (!handedOff && index == current.length() - 1) { + handedOff = true; + current = spliced; + } + return c; + } + + @Override + public int length() { + return current.length(); + } + + @Override + public CharSequence subSequence(int start, int end) { + return current.subSequence(start, end); + } + + @Override + public String toString() { + // what a StringBuilder-backed buffer does: materialise whatever it currently holds + return current.toString(); + } +} diff --git a/core/src/test/java/io/questdb/client/test/tools/NoBrowserLaunch.java b/core/src/test/java/io/questdb/client/test/tools/NoBrowserLaunch.java new file mode 100644 index 000000000..d9a381f7f --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/tools/NoBrowserLaunch.java @@ -0,0 +1,66 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.tools; + +import org.junit.rules.ExternalResource; + +/** + * Disables the OIDC device-flow browser launch for one test class, and puts the property back afterwards. + *

+ * The default {@code DeviceCodePrompt} opens a browser when one is available, which a developer machine has, + * so any test reaching the prompt would pop a real tab. Setting + * {@code questdb.client.oidc.open.browser=false} in a static initializer stopped that, but surefire runs the + * whole module in one JVM: whichever class loaded first flipped the property for every class after it, and + * nothing ever put it back. A test that means to exercise the DEFAULT - the launch enabled - then silently + * ran against someone else's override, depending on class-load order. + *

+ * Use as a class rule, so the window is exactly the class that needs it: + * 

+ * @ClassRule
+ * public static final NoBrowserLaunch NO_BROWSER = new NoBrowserLaunch();
+ *
+ */ +public final class NoBrowserLaunch extends ExternalResource { + + private static final String PROPERTY = "questdb.client.oidc.open.browser"; + private String previous; + private boolean wasSet; + + @Override + protected void after() { + if (wasSet) { + System.setProperty(PROPERTY, previous); + } else { + System.clearProperty(PROPERTY); + } + } + + @Override + protected void before() { + previous = System.getProperty(PROPERTY); + wasSet = previous != null; + System.setProperty(PROPERTY, "false"); + } +} diff --git a/design/oidc-token-persistence.md b/design/oidc-token-persistence.md new file mode 100644 index 000000000..5942291fd --- /dev/null +++ b/design/oidc-token-persistence.md @@ -0,0 +1,705 @@ +# OIDC device-flow token persistence + +Status: **implemented** in PR #52 (`OidcDeviceAuth`, RFC 8628 device flow) on branch +`ia_oidc_device_flow` — both the `TokenStore` SPI and the default `FileTokenStore` (directory +recovery coordination, Layer 1 atomic replace and Layer 2 refresh critical section) shipped +together. This document remains the frozen cross-language on-disk contract (file name, JSON +schema, atomic-write and lock-file protocols) that other clients (e.g. Python) mirror; the design +discussion below is retained as the rationale of record. Code line references are indicative and +may drift from the current source. + +## Problem + +`OidcDeviceAuth` keeps all token state in memory (`OidcDeviceAuth.java:165-175`): +`accessToken`, `idToken`, `refreshToken`, `expiresAtMillis`, `tokenTtlMillis`. Its +own javadoc says so: *"Token state is in-memory only and does not survive a process +restart."* Every restart of the host app therefore forces the human back through the +interactive device flow (open URL, enter code, authorize), even though a long-lived +**refresh token** that could mint a new access token silently was sitting in memory +seconds earlier. + +Goal: optionally persist the token state so a restarted process resumes from the +refresh token (one silent token-endpoint round-trip) instead of re-prompting — without +weakening any of the trust/secret-handling guarantees PR #52 establishes. + +## Goals + +- **Survive restart without re-prompting.** A process that signed in, then restarted, + obtains a usable token from a persisted refresh token with no human interaction. +- **Opt-in.** Default behaviour is unchanged (in-memory only). Persisting a long-lived + credential to disk is a security trade the caller makes explicitly. +- **Pluggable.** A `TokenStore` SPI so an integrator can back persistence with an OS + keychain / KMS / vault. Ship one default `FileTokenStore` (strict-perms file). +- **Language-neutral on-disk contract.** The Java client is the reference + implementation; the Python client (and any other) will mirror it. The file location, + name, JSON schema, and the multi-writer coordination protocol are therefore a *frozen + cross-language contract*, specified below to the byte, not a Java-internal detail. +- **Correctly scoped.** A persisted entry is keyed by the identity it belongs to + (endpoints + client id + scope + audience + groups-in-token mode); a token is never + served for a different configuration. +- **Crash- and concurrency-safe at the file level.** A torn write or an overlapping + writer never yields a half-read credential. +- **Upholds PR #52's invariants.** Tokens never reach logs or exceptions; a persisted + file is treated as untrusted input and validated before any byte reaches a header. +- **Java 8 floor, zero third-party deps** (`java-questdb-client/CLAUDE.md`): reuse + `JsonLexer`/`StringSink`/`MessageDigest`/`java.nio.file` only. + +## Non-goals (this spec) + +- **Encryption at rest with a built-in key.** A key stored next to the ciphertext is + theatre; a key from an OS secret store needs native code we cannot take as a + dependency. Confidentiality at rest is delegated to (a) filesystem permissions for the + default store and (b) the `TokenStore` SPI for anyone who wants a keychain. Stated as a + residual risk below, not solved here. +- **A new credential surface in connection strings / `QDB_CLIENT_CONF`.** The README + already warns against putting tokens there; persistence is a separate, file-scoped + channel. + +## Background: the two code points that constrain the design + +1. **Single write funnel.** Both the interactive flow (`runDeviceFlow` -> `pollOnce` -> + `storeTokens`) and the silent refresh (`tryRefresh` -> `storeTokens`) commit token + state in exactly one method, `storeTokens(TokenResponseParser)` + (`OidcDeviceAuth.java:1261-1284`). That is the natural — and only — place to persist. + +2. **Refresh is gated behind a non-null cached token.** In `getToken()` + (`OidcDeviceAuth.java:444-454`) and `signIn()` (`OidcDeviceAuth.java:477-486`) the + silent-refresh branch only runs when `cachedToken != null`: + + ```java + final String cachedToken = groupsInToken ? idToken : accessToken; + if (cachedToken != null) { + if (System.currentTimeMillis() < expiresAtMillis - effectiveSkewMillis()) { + return cachedToken; + } + if (refreshToken != null && tryRefresh()) { + return selectToken(); + } + // getToken(): throw "expired, can't refresh"; signIn(): fall through to device flow + } + ``` + + Consequence: **restoring only a refresh token does not work with the current logic** — + `signIn()` would skip the refresh and run a fresh device flow; `getToken()` would throw + "no token has been obtained yet". This is the pivotal fact. Two ways out: + + - **(A) Persist the full token blob** (access + id + refresh + `expiresAtMillis` + + `tokenTtlMillis`). On restore `cachedToken != null` holds, so the *existing, audited* + expiry-then-refresh logic runs untouched: a still-valid access token is served with + zero network; an expired one triggers exactly one silent refresh. **No change to the + delicate `signIn`/`getToken`/`tryRefresh` flow.** + - **(B) Persist the refresh token only** and lift the refresh attempt out from behind + the `cachedToken != null` gate in both methods. Smaller on-disk secret footprint, but + it modifies the security-sensitive control flow. + + **Recommendation: (A).** Minimal blast radius on reviewed code, and it makes a quick + restart fully warm (no round-trip at all). The extra on-disk item is a *short-lived* + access token; the long-lived secret (refresh token) is on disk under either option, so + (A) does not change the qualitative risk. (B) is noted as a leaner alternative if we + later decide the access token must never touch disk. + +## API + +New, all in `io.questdb.client.cutlass.auth`: + +```java +public interface TokenStore { + /** Load previously persisted tokens for this identity, or null if none / unreadable. */ + PersistedToken load(TokenStoreKey key); + + /** Persist (atomically replace) the tokens for this identity. Best-effort: an + * implementation reports failure by throwing; the caller treats persistence as + * non-fatal and continues with the in-memory token. */ + void save(TokenStoreKey key, PersistedToken token); + + /** Remove any persisted tokens for this identity. */ + void clear(TokenStoreKey key); + + /** Layer 2 (optional): run action while holding the per-identity cross-process lock. + * The default just runs it unlocked, so a store with no cross-process concern stays a + * plain load/save/clear. An implementation that cannot acquire the lock within its + * budget should run action anyway (degrade to Layer 1) rather than fail a sign-in. + * NOT re-entrant: action must not call back into the owning OidcDeviceAuth. */ + default boolean inLock(TokenStoreKey key, CriticalSection action) { + return action.run(); + } + + /** The critical section; its boolean result (whether a valid token resulted) is + * returned by inLock unchanged. */ + @FunctionalInterface + interface CriticalSection { + boolean run(); + } +} +``` + +`TokenStoreKey` — the non-secret identity fingerprint, computed by `OidcDeviceAuth` from +its config so the store stays semantics-free: + +```java +public final class TokenStoreKey { + private final String clientId; + private final String tokenEndpoint; // origin + path, canonicalised + private final String deviceAuthorizationEndpoint; + private final String scope; + private final String audience; // may be null + private final boolean groupsInToken; + // getters only; identity is the hash() below plus the per-field fingerprint compare on + // load, so no equals/hashCode - the key is never used as a hash-map key + // hash(): hex SHA-256 of a canonical join of the fields, for use as a file name +} +``` + +`PersistedToken` — an immutable carrier mirroring the in-memory fields: + +```java +public final class PersistedToken { + private final String accessToken; // nullable + private final String idToken; // nullable + private final String refreshToken; // nullable + private final long expiresAtMillis; // absolute wall-clock; survives restart + private final long tokenTtlMillis; + // ctor + getters only +} +``` + +Builder wiring (default `null` => no persistence, preserving today's behaviour): + +```java +OidcDeviceAuth.builder().clientId(...)....tokenStore(store).build(); +OidcDeviceAuth.fromQuestDB(url, new DiscoveryOptions().tokenStore(store)); +``` + +Convenience: `FileTokenStore.atDefaultLocation()` and `FileTokenStore.at(Path dir)`. + +## `FileTokenStore` (default implementation) + +- **Location.** `${questdb.client.oidc.token.store.dir}` if set, else + `${user.home}/.questdb/oidc-tokens/`. The `questdb.client.oidc.*` system-property + namespace already exists (`questdb.client.oidc.open.browser`), so this matches. +- **One file per CONFIGURATION**, named `.json`. A hashed name avoids + leaking the endpoint/client id/scope through directory listings, and lets several + configurations coexist (multiple servers or providers on one host). The key names a + configuration, not a subject — no field of it identifies the human who signed in — so two + people using the same configuration address the same file and the later sign-in overwrites + the earlier: **one store holds one active login**. A client MUST document that, and point a + caller who needs several concurrent logins at separate store directories rather than letting + them read the hashed name as a per-user partition. The default location is per OS user + already, so the case that bites is one OS user signing in as several people. +- **Permissions.** Directory created `rwx------` (0700), file `rw-------` (0600), set + *at creation* via `PosixFilePermissions.asFileAttribute(...)` so there is no + world-readable window. On a non-POSIX FS (`setPosixFilePermissions`/attribute throws + `UnsupportedOperationException`) fall back to the ACL-protected user-profile dir and + log a one-line warning that OS-level perms were not enforced (Windows hardening via + `AclFileAttributeView` is a future item). +- **File format and atomic write** — flat plaintext JSON; the exact schema, file naming, + and write protocol are the frozen cross-language contract in + *On-disk interop contract* below. Parsed with the existing `JsonLexer` + a small + `JsonParser` (same pattern as `TokenResponseParser`); written by hand into a `StringSink` + with `"`/`\`/control-char escaping. +- **Bounded, defensive read.** Cap the file at a sane size (reuse the 1 MiB + `JSON_LEXER_MAX_VALUE_BYTES` rationale — an id token with many group claims is several + KB). Parse failure, size overrun, `v` mismatch, or a **fingerprint that does not match + the live config** => return `null` (treat as "no cache"), never throw into the sign-in + path. The fingerprint re-check is defence in depth against a copied/renamed/hostile file + whose name happens to collide. +- **clear():** `Files.deleteIfExists(target)`, run **under the same cross-process lock** as the + read-refresh-write, so a peer's in-flight refresh cannot resurrect the entry by renaming a fresh + file in just after the delete. It returns without creating the directory when nothing is + persisted yet. Cross-process clear stays best-effort: a peer holding a live in-memory token may + legitimately re-persist afterwards. It always forces a fresh sign-in for the calling process, + which resets its in-memory token state regardless. + + `clear()` MUST also delete the identity's **write temps** (`*.tmp`, excluding `.lock.` + captures — see *Temp-file hygiene* below), at **any age**, not just past the staleness window a + `save()`-time sweep uses. A crash between the temp create and the atomic rename orphans a file + holding the full entry — refresh token included — in plaintext, and a caller that clears and then + never signs in again would otherwise leave that credential on disk indefinitely, which + contradicts what `clear()` promises. A temp a concurrent writer is mid-rename on is a benign + loser: its rename fails, persistence is best-effort, and the caller is discarding the credential + anyway. + +## Integration into `OidcDeviceAuth` + +All four touch points sit under the existing `ReentrantLock`, so persistence I/O is +already serialised with sign-in/refresh/clear and needs no new locking. + +1. **Lazy load**, once, at the top of the locked section in `signIn()` and `getToken()`, + guarded by a `boolean storeLoadAttempted` flag: + ```java + if (tokenStore != null && !storeLoadAttempted) { + // Latch only AFTER a read that COMPLETED. A missing, corrupt or foreign-identity file + // yields null without throwing, so that answer is definitive and is not re-read. A read + // that THREW is not an answer: latching there would disable persistence for the life of + // the instance on one transient fault, sending a headless getToken() consumer back to an + // interactive flow it cannot run. + PersistedToken t = tokenStore.load(storeKey); // a throw here leaves the flag unset + storeLoadAttempted = true; + if (t != null) { + // Tell an ABSENT served kind apart from a CORRUPT one; the safe answer differs. + // absent (null) -> legitimate: a grant that returned only the other kind persists + // this shape. Keep the refresh token, leave the cache empty and + // expired, and let the refresh path do the rest. Discarding it + // would re-prompt a human where one silent refresh would do. + // present but unusable (blank, or a control/non-ASCII char) -> positive evidence + // something else wrote this file. Reject the WHOLE entry, + // refresh token included: adopting the refresh token of a + // tampered file would let whoever can write the store swap in + // their own and have the client silently sign in as them. + accessToken = t.getAccessToken(); + idToken = t.getIdToken(); + refreshToken = t.getRefreshToken(); + expiresAtMillis = t.getExpiresAtMillis(); + tokenTtlMillis = t.getTokenTtlMillis(); + } + } + ``` + Nice side effect: after a restart with a persisted refresh token, `getToken()` works as + the *first* call (no explicit `signIn()` needed) — a clean fit for the + `Sender...httpTokenProvider(auth::getToken)` pattern. It may cost one silent refresh + round-trip, which is already inside `getToken()`'s documented contract. + +2. **Save** at the end of `storeTokens(...)` (`OidcDeviceAuth.java:1261`), after the + in-memory fields are set: + ```java + persistIfConfigured(); // builds a PersistedToken from the current fields, calls tokenStore.save + ``` + - On the interactive sign-in: always write (the refresh token is new). + - On a refresh: **write only when the refresh token changed** (rotation). A + non-rotating IdP returns no new refresh token (`storeTokens` keeps the old one), so + the on-disk refresh token is still valid and we skip the write — keeping `getToken()` + cheap on the hot path. A rotating IdP issues a new refresh token, which we *must* + persist or a later restart would replay a revoked one; that write is unavoidable. + - **Best-effort:** wrap `save` so an I/O failure logs one warning and is swallowed — + a disk problem must never fail an otherwise-valid sign-in. The token is good in + memory regardless. + +3. **clear()** in `clearCache()` (`OidcDeviceAuth.java:361`): after nulling the + in-memory fields, call `tokenStore.clear(storeKey)` so the next `signIn()` genuinely + re-prompts. Leave `storeLoadAttempted = true` so we do not immediately reload the file + we just deleted. + +4. **close()** (`OidcDeviceAuth.java:390`): no change. `FileTokenStore` holds no native + resources; `TokenStore` is deliberately **not** `Closeable`. + +The `storeKey` is built once in the constructor from the already-parsed config +(`clientIdEncoded` decodes back, or capture the raw values before encoding; +`tokenEndpoint`/`deviceAuthorizationEndpoint` `Endpoint` -> canonical origin+path). + +## Threat model / security + +Persisting a refresh token widens the attack surface versus memory-only; this is the +whole reason persistence is **opt-in**. Mitigations, mapped to PR #52's existing posture: + +- **At-rest exposure.** Anyone who can read the file (the user, root, a backup) gets a + credential valid until the IdP expires/revokes it. Mitigation: 0600 file in a 0700 dir, + created with those perms (no open window). This matches what `gcloud`, `aws`, and `gh` + do. Residual risk is explicit in the README ("enabling persistence stores a long-lived + credential on disk; use a `TokenStore` backed by your OS keychain to avoid that"). +- **Tampered/forged file = untrusted input.** A file is attacker-writable, so on load we + (a) bound its size, (b) parse defensively and ignore garbage, (c) re-check the + in-file fingerprint against the live config, and (d) run `validateTokenChars` on the + served token before it can become an `Authorization: Bearer` value or a `_sso` password + — exactly the CR/LF / non-ASCII rejection PR #52 applies to IdP responses + (`OidcDeviceAuth.java:935-951`). A bad file degrades to an interactive sign-in; it never + injects into a request or throws token bytes into a message. +- **Untrusted CONTAINER = discard the whole directory.** The file checks above cover the + artefact; they say nothing about who could have put it there. On POSIX, assert the store + directory is not group/other-**writable** before adopting anything out of it (a merely + world-*readable* 0755 from a default umask is fine — no other user can create or replace an + entry, and the files are 0600). When it *was* writable, tighten it to 0700 and discard + **every** entry in it, not only the key being loaded. A client MUST do the directory-wide + version: tightening destroys the very evidence it reports, so whichever identity — or + whichever operation, load *or* save — touches the store first consumes the one observation, + and every entry left behind is one no later call can distrust. Each identity then re-signs + in. Best-effort: a delete that fails must degrade to a sign-in, never throw. +- **The distrust must outlive the chmod: an `.untrusted` sentinel.** The directory-wide discard + closes the *sequential* case (whoever touches the store first sweeps everything, so a later + call finds nothing to distrust). It does **not** by itself close the *concurrent* one: + tightening the permissions and sweeping the entries are two steps, and a second caller — + another thread, or another process — that reads the permissions in the window between them + sees an owner-only directory with the planted entries still in it, computes "trusted", and + adopts a plant. To close that window a client MUST make the distrust survive the chmod with an + on-disk marker. Before tightening the permissions, drop a sentinel file named `.untrusted` + (0600, empty, no secret) in the store directory; treat the directory as untrusted whenever + that sentinel is present, **whatever the permission bits say**; and remove it only after a + **complete** directory-wide sweep — leave it in place if the directory could not be listed or + any delete failed, so the next caller re-sweeps before anything trusts it. The sentinel's name + has no `` store prefix, so the directory sweep never mistakes it for an entry to delete; + a client sweeping the directory MUST likewise leave it alone. The Python client MUST mirror + this: same name, dropped before the chmod, cleared only after a clean sweep. + + **The presence test MUST NOT follow symlinks, and MUST treat "cannot tell" as present.** The + party this sentinel defends against is the one who can write this directory, so both defaults + fail the wrong way. A link-following test reports a *dangling* symlink planted at the sentinel's + name as absent, which disables the sentinel outright — no race to win — and an exclusive-create + mark cannot displace it, because `O_CREAT|O_EXCL` answers `EEXIST` for a symlink exactly as it + does for a peer's mark. A test that reads "absent or unreadable" as absent trusts an + indeterminate stat for the same reason. Use a no-follow test that is positive evidence of + absence — Java `Files.notExists(p, NOFOLLOW_LINKS)`, Python `os.path.lexists(p)` (negated) — + so a symlink, a directory or a failed stat all leave the directory untrusted. Only a **regular + file** (no-follow) counts as a mark a client may later clear; a client that finds any other + shape at the name MUST displace it and mark again rather than read it as a peer's mark, since + a shape its clear cannot delete would latch the directory untrusted forever. Residual: an + attacker who both planted an entry *and* actively deletes the sentinel in the sub-syscall + window between the drop and the chmod can still race a concurrent trust — a far narrower window + than the tighten-to-sweep gap this closes, and Layer 1's atomic replacement still guarantees no + torn or forged credential. +- **The recovery sweep and token operations are one directory-locked transaction.** The sentinel + prevents a peer from trusting before the sweep, but by itself allows two sweepers to act on the + same old verdict: one can complete recovery and save while the other is paused, after which the + paused sweep deletes the fresh file. Every load, save, and trust recovery therefore follows the + required `.store.lock` protocol specified below. The lock is held through the sweep and the + bounded read/write, so a successful save cannot be invalidated by an earlier distrust verdict. +- **Never log/echo secrets.** The store never logs token contents and never embeds file + contents in an exception, upholding PR #52's "tokens never leak into logs or exceptions" + rule. Only paths and `IOException` kinds appear in the one best-effort warning. +- **Wrong-identity serving.** Prevented by the `TokenStoreKey` (filename hash) plus the + in-file fingerprint re-check; a token minted for server/scope/audience A is never served + to a process configured for B. +- **Plaintext-transport interaction.** Unchanged — the IdP endpoints still require + `https` (loopback excepted), so the refresh token only ever crossed the wire encrypted; + persistence does not introduce a new cleartext path. + +## File format and confidentiality (Q1: plaintext vs encoded) + +**The file is plaintext JSON. Confidentiality at rest comes from filesystem +permissions (0600/0700), not from encoding or encryption.** Rationale: + +- **Encoding (base64 / obfuscation) is not security and would not be added as if it + were.** Anyone who can read the file can reverse base64 in one step; it protects + nothing while implying protection — the opposite of PR #52's habit of being explicit + about its trust boundaries. It would also hurt the two things plaintext buys us: + cross-language interop and debuggability. +- **Built-in encryption is a non-goal because of key management** (see Non-goals): a key + beside the ciphertext is theatre, and a key from an OS secret store needs native code + we cannot depend on. Worse for this project specifically — a shared *encrypted* format + would force the Java and Python clients to agree on a cipher *and* a key-derivation + scheme to interoperate on one file. Plaintext JSON is the only format every language + reads and writes with zero dependencies, which is exactly what "Java is the reference + for Python" needs. +- **Real at-rest encryption is delivered through the `TokenStore` SPI** — a caller who + needs it plugs in a keychain/KMS-backed store (macOS Keychain, Windows DPAPI, Linux + Secret Service, Vault). If they also need cross-language sharing, they implement the + same custom store in each client; that is their explicit choice, not our default. +- **This matches the ecosystem.** `gcloud`, `aws`, and `gh` all persist tokens as + plaintext under owner-only permissions. The README will state the residual risk plainly + ("persistence writes a long-lived credential to disk in plaintext, protected by file + permissions; back the store with your OS keychain to avoid that"). + +Writer correctness: a refresh token is an opaque IdP string, so the JSON writer **must** +escape `"`, `\`, and control characters (`< 0x20` as `\uXXXX`); the existing `JsonLexer` +already decodes escapes on read (a PR #52 change). Base64-ing token *values* would dodge +escaping, but proper escaping is trivial and keeps the file readable — not worth it. + +## On-disk interop contract (frozen cross-language spec) + +Both clients MUST agree on these to the byte, or they will not share a file (a mismatch +is benign for *correctness* — the fingerprint re-check below still prevents wrong-identity +serving — but it defeats *sharing*, leaving each client to re-prompt). + +- **Directory:** `${questdb.client.oidc.token.store.dir}` if set, else + `${user.home}/.questdb/oidc-tokens/`. Created `rwx------` (0700). +- **Directory recovery lock:** `.store.lock` in that directory, created 0600 with + `O_CREAT|O_EXCL` and carrying the same bounded owner-stamp protocol as the per-identity lock + below. Every conforming client MUST honor it; see *Directory recovery coordination*. +- **File name:** `.json`, where `` is the lowercase hex SHA-256 of the + UTF-8 **canonical identity string**, NUL-separated so no field can be confused with a + separator: + ``` + "questdb-oidc-token-v1" \0 clientId \0 canon(tokenEndpoint) \0 + canon(deviceAuthorizationEndpoint) \0 scope \0 (audience ?? "") \0 (groupsInToken?"1":"0") + ``` + `canon(endpoint)` = `lower(scheme) "://" lower(host) ":" port path`, with the port + always explicit (the device-flow default 443/80 when absent) and `path` the parsed + path (no fragment). The hash is a *bucketing* key only; correctness rests on the + in-file fingerprint, so slight normalization drift across languages costs at most a + missed share, never a wrong token. +- **Schema** (file perms `rw-------`, 0600): + ```json + { + "v": 1, + "client_id": "questdb", + "token_endpoint": "https://idp.example.com:443/as/token.oauth2", + "device_authorization_endpoint": "https://idp.example.com:443/as/device_authz.oauth2", + "scope": "openid", + "audience": "api://billing", + "groups_in_token": false, + "access_token": "...", + "id_token": "...", + "refresh_token": "...", + "expires_at_millis": 1730000000000, + "token_ttl_millis": 300000 + } + ``` + The first seven fields are the **non-secret fingerprint**; on load both clients re-check + them against the live config and ignore the file on mismatch (defence in depth against a + hash collision or a copied/renamed file). `expires_at_millis` is absolute wall-clock, so + it is portable across a restart and across machines that share a clock. + + The two endpoint fields MUST carry the same `canon(endpoint)` rendering the file name + hashes — in particular **with the port always explicit**, as in the example above. The + re-check is a byte-exact string compare, not a URL comparison, so a writer that omits the + default port produces a file every other client silently ignores: `load` returns null, the + process re-prompts, and it re-persists in its own encoding, so the two never converge. This + is the one field-level normalization that is load-bearing rather than best-effort — unlike + the hash, where drift only costs a missed share. + + A field whose value is null - an absent `audience`, or a token kind the grant did not + return (e.g. no `id_token`) - is **omitted entirely**, not written as JSON `null`. QuestDB's + `JsonLexer` reports a bare `null` and a quoted `"null"` identically, so omission is the only + encoding under which every present value round-trips verbatim (a token equal to the string + `"null"` included); a reader treats an absent field as null. The Python client MUST do the + same: omit null fields on write, and treat an absent field as null on read. + + The document MUST be a single flat JSON object. A reader rejects any other shape - an array + anywhere (for example a top-level `[ {…} ]` wrapper) or a non-object root - rather than + extract fields from a malformed structure. The Python client MUST do the same. + + At least one of `access_token` / `id_token` MUST be present. A writer MUST NOT persist an + entry carrying only a `refresh_token`, and a reader MUST reject one - the whole entry, + refresh token included. No conforming grant produces that shape (RFC 6749 5.1 requires + `access_token` in a token response, and a client stores an entry only after a grant it could + serve), so a file in that shape was not written by a conforming client. Adopting it is a + silent credential swap: an attacker who can write the store directory - without ever reading + the 0600 file - plants an entry whose fingerprint fields are all derivable from public + config, and the reader's next silent refresh presents the attacker's refresh token and + resumes as them, with no prompt and no log line recording the change of identity. The cost + of the rule when it fires on an honest file is one interactive sign-in. + + The numeric fields (`v`, `expires_at_millis`, `token_ttl_millis`) are **plain JSON integers**: + an optional leading `-` followed by bare digits. A reader MUST NOT accept its own language's + numeric extensions here — QuestDB's `Numbers.parseLong` would otherwise take `1_000` and `5L`, + and Python's `int()` takes `1_000` and surrounding whitespace. A value only one implementation + can parse is a file only that implementation can read, which is exactly the divergence this + frozen format exists to prevent; treat anything outside the plain-integer grammar as unusable + and fall back as for any other bad field. +- **Write protocol (atomicity):** write a sibling temp file created with 0600, flush, then + **atomically rename** over the target — Java `Files.move(tmp, target, ATOMIC_MOVE, + REPLACE_EXISTING)`, Python `os.replace(tmp, target)`. Both are `rename(2)` on POSIX + (atomic) and atomic on Windows. A crash or an overlapping reader sees the whole old or + whole new file, never a torn credential. This is the one *mandatory* multi-writer + guarantee and it interoperates trivially. + +## Cross-process and cross-language coordination (Q2) + +Directory recovery coordination plus two data layers; recovery and Layer 1 are mandatory, while +Layer 2 handles the rotating-refresh-token case Layer 1 cannot. + +**Directory recovery coordination (always; cross-language-safe).** The `.untrusted` sentinel +carries a distrust verdict across the chmod, but it does not serialize the directory-wide sweep +with token writes. Without serialization, two callers can both observe the sentinel: A tightens +and pauses before its sweep; B sweeps, clears the sentinel, saves another identity, and returns; +then A resumes with its stale verdict and deletes B's completed file. The save reported success, +but a headless restart finds no refresh token and requires a human sign-in. + +Every client therefore MUST treat the following as one required directory-wide critical section: + +1. Ensure the directory exists, without tightening a pre-existing directory yet. +2. Acquire `.store.lock` with `O_CREAT|O_EXCL`, 0600 permissions, the owner stamp, bounded 4 KiB + read, capture-then-verify steal, and owner-verified release specified for `.lock` below. + Once the directory is owner-only and has no `.untrusted` marker, this required lock has a separate + short lease because it then protects bounded filesystem work rather than an identity-provider + request: a stamped holder renews its mtime every 500 ms and is stale after 2 seconds without a + renewal. An empty lock is likewise reclaimable after 2 seconds. The post-chmod owner-stamp check in + step 4 makes reclaiming a creator paused before its stamp safe: it wakes without ownership and + aborts before loading, sweeping, or writing. A directory that is writable by another user, still + marked `.untrusted`, or cannot be inspected retains the configured staleness window. This prevents + a paused old sweep from resuming after displacement and deleting a new holder's completed save. +3. While holding the lock, run the permission/trust check. If the directory was group/other + writable, mark `.untrusted` before tightening it and retain that untrusted verdict for this + critical section. +4. Re-read `.store.lock` after the tighten and require the exact owner stamp. The lock was created + while the directory may still have been writable, so another local user could have removed or + replaced it before the chmod. On absence, unreadable content, or mismatch, reassert `.untrusted` + now that the directory is owner-only and abort/retry without loading, sweeping, or writing. +5. If the directory is untrusted, complete the directory-wide sweep before doing anything else. + A recovering `load` returns no entry for that call; a `save` writes its fresh entry only after + the sweep. Keep `.store.lock` held through the bounded read or the complete temp-write/flush/ + rename, then release it. + +Acquisition is **required**, unlike the per-identity refresh lock: timeout or I/O failure fails the +store operation rather than running it uncoordinated. The caller already treats load failures as +transient and save as best-effort; silent loss after a successful save is not an acceptable +degrade. A process crash stops the lease heartbeat, so the default 3-second acquisition budget can +reclaim the lock instead of inheriting the refresh lock's 10-minute stale window. The directory +lock is short-lived and is never held across token-endpoint I/O. An +`inLock` implementation may acquire and release it to prepare the directory before acquiring +`.lock`; the refresh action then briefly takes `.store.lock` inside `.lock` for its load +and save. No path may hold `.store.lock` while waiting for `.lock`, so this order has no +cross-process cycle. The Python client MUST use the same name and protocol. + +**Layer 1 — atomic replacement (always; cross-language-safe).** The write protocol above +makes every update all-or-nothing, so any mix of processes and languages sharing one file +(two notebook kernels, a restart overlapping the old process, a Java writer and a Python +reader) is *integrity-safe*: no torn reads, no partial credential. For the common case — +an IdP that does **not** rotate refresh tokens — this is fully sufficient: every process +holds the same stable refresh token, each independently refreshes to mint its own access +token, and last-writer-wins on the file is harmless because access tokens are +interchangeable and the fingerprint fields are identical for one identity. + +**Layer 2 — a lock-file critical section (for rotating refresh tokens).** When the IdP +*rotates* the refresh token on every refresh (Auth0 public clients, OAuth 2.1 BCP +guidance), bare last-writer-wins races: two processes load RT1, both refresh, the IdP +invalidates RT1, one wins and the loser's RT1 is now revoked → an unnecessary interactive +re-prompt. To eliminate it, serialise the *read-modify-write* of a refresh per identity: + +- **Use a lock *file*, not an OS advisory lock.** Java `FileLock` maps to `fcntl` POSIX + record locks on Unix while Python's `fcntl.flock` is BSD `flock`; the two **do not + interoperate on Linux**. A lock file acquired with `O_CREAT|O_EXCL` (Java + `FileChannel.open(..., CREATE_NEW, WRITE)`, Python `os.open(..., O_CREAT|O_EXCL|O_WRONLY)` + / `open(p,"x")`) is a plain filesystem primitive that interoperates trivially. The contract + mandates the lock-file scheme; OS advisory locks are out. +- **Lock file:** `.lock` beside the token file, containing a unique per-acquisition + owner stamp — a creation timestamp, a random nonce, and OPTIONALLY the holder's `pid@host`. + The nonce alone carries the uniqueness the protocol needs; `pid@host` is a debugging aid, and a + client MUST NOT obtain it in any way that can block the acquire. Python's `socket.gethostname()` + is `gethostname(2)` and is free, so the Python client includes it; Java has no cheap equivalent — + `ManagementFactory.getRuntimeMXBean().getName()` resolves the local hostname through + `InetAddress.getLocalHost()`, measured at 3.2s on a host with a cold mDNS cache, inside a 200ms + acquire budget and on a producer's flush path — so the Java client omits it rather than break the + bound below. Since no implementation parses another's stamp (see "Release verifies ownership"), + the two shapes interoperate unchanged. + Acquire by an exclusive-create (`O_CREAT|O_EXCL`) and write the owner stamp through that same + open handle — see the empty-lock note below for the window this leaves; on contention, spin + with short backoff up to a small acquire budget (~3s); if it still cannot be acquired, + **proceed without it** (degrade to Layer 1) rather than fail a sign-in. A lock older than a staleness timeout (10 minutes) + is treated as abandoned and stolen, so a crashed holder cannot wedge others. The window + must dominate the worst-case time a live holder can hold the lock. That worst case is the + refresh I/O under the lock — send + await + parse, plus a body drain on a parse failure, each + separately bounded by the HTTP timeout (capped at 120s), so up to ~4×120s = ~480s. **A client + MUST also bound the connection phase that precedes the send** — the TCP connect and the TLS + handshake — by the same HTTP timeout. Neither is bounded by it automatically: a client that + leaves the connect to the OS, or sizes the TLS handshake off a transport default, can hold + the lock far past the staleness window and have a peer steal it from under a live refresh, + at which point both replay the same rotating refresh token. Only DNS resolution is left to + the OS. So size the window above ~4×HTTP-timeout plus a DNS allowance; the interactive wait + is never held under the lock. 10 minutes clears ~480s with ample headroom; a client that + raises the HTTP timeout must raise this window in step. +- **An empty/unstamped lock is reclaimable on a short grace, not the full staleness window.** + The exclusive create and the stamp write are two operations on one open handle, so the file + **does exist empty** between them. That window is small — no I/O sits between the two — but it + is real, and a GC/safepoint pause or a descheduled thread CAN land in it, as can a crash + mid-write. Its mtime is fresh, which the staleness check would protect for the whole window, + wedging peers into lock-free refreshes; so treat a lock that carries no readable owner stamp + as stealable once it is older than a short grace — **5 seconds**, which must dominate the + create→stamp window on any implementation — instead of the full staleness window. **The grace, + not the absence of the window, is what stops a peer from stealing a lock that is mid-stamp**, so + a client MUST NOT shorten it on the assumption that create-with-stamp is atomic: it is not, in + Java (`FileChannel.open(CREATE_NEW)` then `write`) or in Python (`os.open(O_CREAT|O_EXCL)` then + `write`). A cross-machine clock skew wider than the grace (the age check compares the local clock + against the file's mtime) could still pre-empt such a partial lock, but that never forges or + tears a credential — Layer 1's atomic replacement always holds — it degrades to a concurrent + refresh (a re-prompt on a rotating-refresh-token IdP), the same best-effort residual as running + lock-free. The capture-then-verify steal below still aborts if the captured lock does not match + what was judged stale. The Python client MUST mirror the 5-second empty-lock grace. +- **Bounded lock read.** A `.lock` larger than **4 KiB** is not read; it is treated as + carrying no readable stamp, i.e. as an empty lock subject to the grace above. An owner stamp is + tens of bytes, so a client MUST keep its stamp well under that cap or its live locks will be + stolen after the grace. +- **Temp-file hygiene must not touch steal captures.** A client that sweeps its own stale + `*.tmp` files MUST skip any name containing `.lock.`: the capture-then-verify steal below + renames the lock to `.lock..tmp` while it decides, and deleting another client's + capture breaks its steal. (The exclusion applies to `clear()`'s any-age sweep too.) +- **Putting a captured lock back is not a rename.** When capture-then-verify decides the lock it + grabbed is *live* after all — a peer recreated it in the gap — the lock must be restored, and a + plain "rename back if the target is free" is the wrong primitive: Java `Files.move` without + `REPLACE_EXISTING` and Python `os.rename` both stat the target and then rename, so a third party + claiming the freed path between those two steps has its live lock silently destroyed by the very + call meant to leave it alone. Restore with a primitive that fails when the target exists and + cannot replace it — `link(2)` (Java `Files.createLink`, Python `os.link`) followed by unlinking + the capture. A filesystem without hard links may fall back to the rename, accepting that window. + A residual remains either way: if a third party did claim the path, the recreating peer's lock + file is gone while that peer still believes it holds the lock, so two holders can run for that + one refresh. No filesystem offers an atomic "rename only if the content is still X", so the + capture-verify narrows this window without closing it. +- **Release verifies ownership.** A holder releases by re-reading the lock and deleting it + **only when it still carries that holder's own owner stamp**, never by bare path. Should + a hold ever outrun the staleness window and be stolen and recreated by a peer, the + original holder must not delete the peer's live lock on release (which would admit a + third acquirer alongside the peer and break mutual exclusion). Each implementation + checks only its own stamp; it never has to parse another implementation's stamp, so the + random nonce keeps the check exact without coupling the language clients. +- **Protocol (under the existing in-process `ReentrantLock`, only when a refresh is + needed):** + 1. acquire `.lock`; + 2. **re-read the token file** — another process may have just refreshed; + 3. if the freshly read served token is now valid, adopt it (re-running + `validateTokenChars`) and **skip the network**; + 4. else POST the refresh with the current refresh token; `storeTokens()` writes the + new token atomically *inside* the lock; + 5. release (delete `.lock` only if it still carries our own owner stamp). + + The interactive device flow does **not** hold the lock file (coordinating human prompts + across processes is overkill and would hold a cross-process lock for up to 30 min); two + cold processes may each prompt once, after which later processes read the persisted + refresh token. Lock ordering is always in-process lock then lock file (leaf), so no + deadlock. + +- **SPI shape:** keep `TokenStore` simple for the no-coordination case and add one + optional hook, `default boolean inLock(TokenStoreKey, CriticalSection action)` that just + runs `action` (no lock). `FileTokenStore` overrides it with the lock-file protocol; + `OidcDeviceAuth` wraps its refresh step in `inLock` and does the re-read-then-decide + (steps 2–4) as the action body. A store with no cross-process concern stays a plain + load/save/clear. + +**Staging.** Layer 1 is required and small; Layer 2 is only needed for rotating IdPs. +Both can ship together, or Layer 1 first with Layer 2 as a fast-follow — but the lock-file +protocol above should be frozen into the spec now so the Python client implements the +same one. (See Decisions.) + +## Decisions + +Resolved: +- **Full token blob (option A)** — persist access + id + refresh + expiry; no change to + the audited `signIn`/`getToken`/`tryRefresh` gate. +- **Plaintext JSON**, confidentiality via file permissions; encryption only via the SPI + (Q1). +- **SLF4J at `WARN`** for the one best-effort persistence-failure warning (the Java client ships + `slf4j-api` only, so an application without a binding sees nothing; a client in another language + should use whatever its own ecosystem's equivalent warning channel is). +- **Opt-in** (no store unless the caller sets one). +- **Ship `FileTokenStore`** as the default; keychain/KMS via the SPI. +- **Frozen on-disk contract** (path, hash, schema, atomic write, lock-file protocol), + because the Python client will mirror it. + +Resolved (shipped in PR #52): +- **Layer 2 (lock file) shipped together with Layer 1.** `FileTokenStore` implements both the + mandatory atomic-replace integrity layer and the `O_CREAT|O_EXCL` lock-file critical section + for rotating-refresh-token IdPs, as recommended — the rotating case is realistic and the + lock-file code is modest. + +## Testing strategy + +- `FileTokenStore`: round-trip save/load; perms are 0600/0700 (skip on non-POSIX); + ATOMIC_MOVE leaves no `.tmp`; corrupt/oversized/garbage file -> `load` returns null, no + throw; fingerprint mismatch -> null; a token with CR/LF/non-ASCII -> rejected on load; + deterministically pause one caller between tightening and its distrust sweep, prove a + concurrent load/save waits on `.store.lock`, and prove a save that returns survives the sweep. +- `OidcDeviceAuth` against a fake `TokenStore` + the existing `MockOidcServer`: + - sign in -> a second *new* instance with the same store skips the device flow and only + hits the token endpoint (silent refresh) — assert the device-auth endpoint is never + called. + - quick restart with an unexpired persisted access token -> zero network. + - rotating refresh token -> file rewritten each refresh; non-rotating -> written once. + - `clearCache()` deletes the file -> next `signIn()` re-runs the device flow. + - `save` throwing -> sign-in still returns a valid token (best-effort), warning emitted. + - `getToken()` as the first call after a restore (no `signIn()`), refresh path only. +- `assertMemoryLeak` around tests that build a real `OidcDeviceAuth` (native lexer). + +## Open questions + +- **Default location on Windows** — `${user.home}/.questdb` is fine functionally, but the + ACL hardening story there is unfinished: POSIX perms do not apply, so the file relies on + the user-profile directory's default ACL. Tightening via `AclFileAttributeView` + (owner-only) is a possible follow-up; the Python client will face the same gap. +- **Windows lock-file interop** — the `O_EXCL` lock-file scheme works on Windows + (`CREATE_NEW`), but the staleness/steal heuristic must tolerate Windows' stricter + delete-while-open semantics; verify before relying on Layer 2 cross-platform. + +Notes carried from the discussion (not open): +- Python persistence does not exist yet and will be built **after** the Java client, using + this as the base — hence the frozen contract. The single most important thing Python + must copy verbatim is both **lock-file** protocols (not OS advisory locks): `.store.lock` + for directory recovery/read/write ordering and `.lock` for refresh ordering, since + Java `FileLock` (`fcntl`) and Python `flock` do not interoperate. diff --git a/examples/src/main/java/com/example/sender/OidcDeviceFlowExample.java b/examples/src/main/java/com/example/sender/OidcDeviceFlowExample.java new file mode 100644 index 000000000..fe0cc414d --- /dev/null +++ b/examples/src/main/java/com/example/sender/OidcDeviceFlowExample.java @@ -0,0 +1,55 @@ +package com.example.sender; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.OidcDeviceAuth; + +/** + * Signs in to an OIDC-secured QuestDB Enterprise from code that has no local browser + * (a remote notebook kernel, a container, a headless job) using the OAuth 2.0 Device + * Authorization Grant, then shows the three ways to use the resulting token. + *

+ * On first use this prints a verification URL and a short code and, on a machine with a + * browser, opens the URL for you; otherwise open it on any device (your laptop or your + * phone) and enter the code. The token is then cached in memory and refreshed silently, + * so re-running this does not prompt again. + */ +public class OidcDeviceFlowExample { + public static void main(String[] args) { + // Discover client id, scope, endpoints and the groups-in-token mode from the server. + // Alternatively, configure the identity provider explicitly with OidcDeviceAuth.builder(). + // The default prompt prints the URL and code AND opens the URL in your browser when one is + // available (best-effort; skipped on a headless host). To print only, pass options: + // import io.questdb.client.cutlass.auth.DeviceCodePrompt; + // OidcDeviceAuth.fromQuestDB(url, new OidcDeviceAuth.DiscoveryOptions().prompt(DeviceCodePrompt.SYSTEM_OUT)) + // To survive a restart without prompting again, persist the token with a TokenStore - the restarted + // process resumes from the saved refresh token instead of re-running the device flow: + // import io.questdb.client.cutlass.auth.FileTokenStore; + // OidcDeviceAuth.fromQuestDB(url, new OidcDeviceAuth.DiscoveryOptions().tokenStore(FileTokenStore.atDefaultLocation())) + try (OidcDeviceAuth auth = OidcDeviceAuth.fromQuestDB("https://questdb.example.com:9000")) { + auth.signIn(); // sign in once (prompts on first use, then caches and refreshes silently) + + // 1. Use one pooled QWP handle for ingest and queries. The provider is shared by both + // pools and queried again on each reconnect, so long-lived clients follow silent refreshes. + try (QuestDB db = QuestDB.connect( + "wss::addr=questdb.example.com:9000;", + auth::getToken)) { + try (Sender sender = db.borrowSender()) { + sender.table("trades") + .symbol("symbol", "ETH-USD") + .doubleColumn("price", 2615.54) + .atNow(); + } + // db.borrowQuery() uses the same rotating bearer-token provider. + } + + // 2. Query the REST API directly: send the token in the Authorization header. + // String header = auth.getAuthorizationHeaderValue(); // "Bearer " + // GET https://questdb.example.com:9000/exec?query=... with header Authorization:

+ + // 3. Connect over PG-wire with any JDBC or psql client: user "_sso", password = the token + // (requires acl.oidc.pg.token.as.password.enabled=true on the server). + // jdbc:postgresql://questdb.example.com:8812/qdb user=_sso password= + } + } +}