fix: never leak or persist a user API credential by timzsu · Pull Request #137 · mlsys-io/FlowMesh

timzsu · 2026-09-16T05:54:54Z

Purpose

Two ways a user API credential could escape:

main injects NEBULA_API_TOKEN into any request, including one aimed at a caller-supplied spec.api.url — so pointing a task at a third-party endpoint sends it our Nebula credential.
A credential supplied by the caller is persisted to Redis in plaintext, twice over, where REDIS_URL defaults to an unauthenticated redis:// with no TLS.

Changes

src/worker/executors/api_executor.py — the Nebula token is injected only when the request is going to the Nebula endpoint. A caller-supplied spec.api.url must carry its own Authorization header and fails closed without one.
src/server/task/redact.py — new. Recursive, key-name-based redaction of credentials, through nested mappings and lists, applied to both the parsed api spec and the raw workflow YAML.
src/server/task/models.py — a serializer on TaskRecord applies the redaction, so the in-memory record keeps the real credential for dispatch while every dump is redacted. This covers all four existing Redis dump sites and any added later.
src/server/dispatcher/base.py — a task rehydrated from a redacted dump cannot authenticate, so it fails with credential_not_retained rather than dispatching [REDACTED] as a bearer token.
tests/worker/test_api_executor.py, tests/server/test_redact.py — cover the credential matrix and the redaction, including the three defects below.

Design

Redaction happens at serialization, not construction. Redacting on the way in would mean the dispatcher no longer has the credential; redacting on the way out means one hook covers every dump site, current and future, instead of an enumeration that can miss one.

raw_yaml is parsed, redacted as a tree with the same recursive rule as the spec, and re-emitted. A line-based scan was tried first and is structurally unable to see a value written as a block (|) or folded (>-) scalar — both ordinary YAML, and both natural ways to write a long bearer token. Re-emitting drops comments and key order; raw_yaml is a stored record that is never re-parsed, so that cost is accepted deliberately. YAML that fails to parse is redacted wholesale rather than stored un-analysed.

Three defects were found and fixed during review, each reproduced against the code rather than reasoned about:

defect	symptom
nested credential survived	`json.auth.token` stayed in plaintext while the header was redacted
substring key matching	`monkey`, `turkey`, `keyword`, `keys` all matched, destroying unrelated stored config
line-based YAML scan	`Authorization: \|` and `>-` leaked the value on the following line

Known limit, stated rather than implied: redaction is by key name. A credential stored under a name that does not look like one is not detected.

Behaviour change worth calling out: an API task carrying a user credential can no longer be re-dispatched after a server restart, because the credential is deliberately not retained. It fails with a clear resubmit error.

Test Plan

uv run pytest tests/server tests/worker/test_api_executor.py
pre-commit run --all-files

Test Result

tests/server — 676 passed
tests/server/test_redact.py — 19 passed
tests/worker/test_api_executor.py — 5 passed
pre-commit run --all-files — all hooks pass, mypy included

kaiitunnz

Several comments. PTAL.

kaiitunnz · 2026-09-17T13:27:34Z

@@ -0,0 +1,84 @@
+"""Redaction of API credentials before a task record is serialized.


This module can be moved to src/shared/utils.

kaiitunnz · 2026-09-17T13:32:58Z

+        api = spec.api
+        if api is None:
+            return False
+        headers = api.get("headers")


This method detects only redacted credentials in "headers", but redact_api redacts credentials in ("headers", "params", "body", "json", "data"). Also add unit tests to guard against this error.

kaiitunnz · 2026-09-17T13:40:22Z

+    @model_serializer(mode="wrap")
+    def _serialize(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:
+        data = handler(self)
+        data["raw_yaml"] = redact_raw_yaml(self.raw_yaml)


raw_yaml always get redacted. n8n json gets converted into yaml, and yaml comments are removed, making it not raw anymore.

kaiitunnz · 2026-09-17T13:40:53Z

+        data = handler(self)
+        data["raw_yaml"] = redact_raw_yaml(self.raw_yaml)
+        spec = self.task.spec
+        api = getattr(spec, "api", None)


Use isinstance instead of getattr.

kaiitunnz · 2026-09-17T13:46:58Z

+            if not has_credential:
+                raise ExecutionError(
+                    "spec.api.url names a custom endpoint but no credential was "
+                    "supplied; set an Authorization header. The Nebula token is "
+                    "never sent to an endpoint the caller chose."
+                )


Why must we enforce auth on custom URL? Can we allow opting out?

kaiitunnz · 2026-09-17T13:47:59Z


-        token = os.getenv("NEBULA_API_TOKEN")
-        if token and not any(k.lower() == "authorization" for k in headers):
+        has_credential = any(k.lower() == "authorization" for k in headers)


For custom URL, we only check the Authorization header, but some endpoints use different headers, e.g., "X-API-Key". I think we should be more comprehensive.

kaiitunnz · 2026-09-17T13:54:51Z

        return self.failed_workers[-1] if self.failed_workers else None

+    @model_serializer(mode="wrap")
+    def _serialize(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:


This redacting logic runs on every TaskRecord serialization, which occurs very often, e.g., when task state changes. I suggest redacting only once when persisting the record.

…erence Extend the API executor with a general, endpoint-agnostic credential path while keeping the legacy Nebula path intact. spec.api.url wins over NEBULA_API_BASE_URL; spec.api.auth.credential_env wins over NEBULA_API_TOKEN. The secret never travels in the workflow spec — only the env-var reference does. A call with no credential fails closed unless spec.api.auth.mode: none opts out explicitly, so a forgotten credential errors instead of silently going out anonymous. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

…dential Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

…om url Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

Moving the Nebula token injection inside the "no custom url" branch left headers read before it was assigned, so every Nebula-path call raised UnboundLocalError. Restore the ordering and gate on a custom_url flag. A caller-supplied spec.api.url now requires its own Authorization header and fails closed without one, rather than calling the endpoint anonymously. The Nebula token is never sent to an endpoint the caller chose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

kaiitunnz · 2026-09-17T14:02:18Z

Also, two more issues that are worth follow-up PRs to sweep:

There are other task specs that contain secret credentials, e.g., DataRetrievalSpec, RagSpec, and ServeSpec. We may need to do a full sweep to redact them.
Some workflow templates still set "apiVersion": "mloc/v1", which is legacy naming. FlowMesh is now using "apiVersion": "flowmesh/v1".

timzsu changed the title ~~feat: make the api task executor endpoint-agnostic with env-var credential reference~~ fix(security): never leak or persist a user API credential Sep 16, 2026

timzsu changed the title ~~fix(security): never leak or persist a user API credential~~ fix: never leak or persist a user API credential Sep 16, 2026

timzsu marked this pull request as ready for review September 16, 2026 11:32

timzsu requested a review from kaiitunnz as a code owner September 16, 2026 11:32

timzsu force-pushed the api-endpoint-agnostic branch from 28593db to 5f492a2 Compare September 17, 2026 02:17

kaiitunnz requested changes Sep 17, 2026

View reviewed changes

timzsu and others added 6 commits September 17, 2026 21:58

fix: default to lum.id/llm + deepseek and drop the worker env-var cre…

af2a39d

…dential Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

fix: keep Nebula the default route and never send its token to a cust…

51e9236

…om url Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

fix(security): never persist an API credential in plaintext

bfd686a

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

refactor: narrow the redacted-credential check to ApiSpecStrict

000af3e

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

timzsu force-pushed the api-endpoint-agnostic branch from 5f492a2 to 000af3e Compare September 17, 2026 13:58

Navigation Menu

fix: never leak or persist a user API credential - #137

fix: never leak or persist a user API credential#137
timzsu wants to merge 6 commits into
mainfrom
api-endpoint-agnostic

timzsu commented Sep 16, 2026 •

edited

Loading

Uh oh!

kaiitunnz left a comment

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz Sep 17, 2026

Uh oh!

kaiitunnz commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,84 @@
		"""Redaction of API credentials before a task record is serialized.

Conversation

timzsu commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

Design

Test Plan

Test Result

Uh oh!

kaiitunnz left a comment

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

kaiitunnz commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

timzsu commented Sep 16, 2026 •

edited

Loading