feat(aicore): transparent TLS mode and reactive credential reload

[tiagoek]

Disclaimer: Do not include SAP-internal or customer-specific information in this PR (e.g. internal system URLs, customer names, tenant IDs, or confidential configurations). This is a public repository.

Description

This PR addresses two security concerns with how the aicore module handles credentials at runtime.

Tracking: AFSDK-4412 (transparent TLS) / HASI2026203 (CVE 9.9)

---

1. Reactive credential reload on AuthenticationError ✅ complete

completion() and acompletion() now intercept litellm.AuthenticationError, re-read credentials from the mounted secret volume via reload_aicore_credentials(), and retry the call once. This covers credential rotation scenarios (client secret rotation by the platform) without requiring a pod restart. If the retry also fails, the error propagates normally — no retry loop.

reload_aicore_credentials() is exported as a public function for callers that need to trigger a manual reload.

This feature is independently complete and works today with no additional changes.

---

2. Transparent TLS mode (AICORE_TRANSPARENT_TLS) ⚠️ SDK complete — pending LiteLLM upstream

Adds opt-in support for infrastructure-managed mTLS authentication. The mechanism:

1. The deployer sets AICORE_TRANSPARENT_TLS=true in the pod environment
2. set_aicore_config() skips writing AICORE_CLIENT_SECRET to os.environ and actively removes any stale value already present
3. The Kyma infrastructure sidecar intercepts the outbound XSUAA token request and adds the mTLS client certificate at the TLS layer — the agent process never holds a shared secret

Important: This is a sidecar-based mechanism. It does not require an X.509 service binding (credential-type: x509) and does not require LiteLLM to use certificate material directly. The agent calls XSUAA with only client_id; the sidecar adds the mTLS cert transparently.

This is the same authentication pattern already used by the agentgateway module (introduced in #220).

LiteLLM upstream dependency: The current public litellm validate_credentials() requires exactly one credential mode (client_secret, cert_str+key_str, or cert_file_path+key_file_path). A 4th mode (transparent_tls=True) is needed to allow a no-credential token request where the sidecar provides the cert. Until that upstream change lands and litellm minimum version is bumped in pyproject.toml, AICORE_TRANSPARENT_TLS=true will result in a ValueError from LiteLLM on the first completion call.

Alternative today: Use proxy mode or destination mode from PR #271, which do not require the LiteLLM upstream change and address CVE 9.9 for the majority of agent deployments.

Any stale AICORE_CLIENT_SECRET already present in the environment is explicitly removed when transparent TLS mode is active, preventing accidental reuse.

---

Related Issues

• AFSDK-4412 — transparent TLS mode BLI
• HASI2026203 (CVE 9.9 — internal tracking)
• Stacked on: this PR is the base; feat(aicore): clear AICORE_CLIENT_SECRET after token acquisition (AFSDK-4291) #257 and feat(aicore): transparent proxy routing and BTP Destination Service mode (Option 3) #271 stack on top

Type of Change

• New feature (non-breaking change that adds functionality)
• Bug fix (non-breaking change that fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Code refactoring
• Dependency update

How to Test

Reactive credential reload (works today):

1. Call set_aicore_config() followed by completion() successfully
2. Simulate credential rotation: update the mounted secret file with new credentials
3. Force an AuthenticationError (e.g. revoke the current token or wait for expiry)
4. Verify the next completion() call succeeds without a pod restart

Transparent TLS mode (SDK side only — LiteLLM upstream pending):

1. Set AICORE_TRANSPARENT_TLS=true in the environment before calling set_aicore_config()
2. Verify AICORE_CLIENT_SECRET is not present in os.environ after the call
3. Verify AICORE_CLIENT_ID, AICORE_AUTH_URL, AICORE_BASE_URL are still set normally
4. Full end-to-end test requires the LiteLLM upstream change — until that PR lands, litellm.completion() will still raise ValueError in transparent TLS mode

Unit tests:

python -m pytest tests/aicore/unit/ -v
# Expected: 65 passed

Checklist

• I have read the Contributing Guidelines
• I have verified that my changes solve the issue
• I have added/updated automated tests to cover my changes
• All tests pass locally
• I have verified that my code follows the Code Guidelines
• I have updated documentation (if applicable)
• I have added type hints for all public APIs
• My code does not contain sensitive information (credentials, tokens, etc.)
• I have followed Conventional Commits for commit messages

Breaking Changes

None. All changes are additive or opt-in:

• AICORE_TRANSPARENT_TLS — requires explicit opt-in; default behavior is unchanged
• reload_aicore_credentials() — new public function, no existing callers affected
• Retry on AuthenticationError — same exception type propagates if retry also fails; callers that catch AuthenticationError may observe a slight delay before receiving it (one additional attempt), but the contract is unchanged

Additional Notes

Dependency on LiteLLM upstream: validate_credentials() in litellm/llms/sap/credentials.py needs a 4th bypass mode for transparent TLS. The code change is minimal — add an optional transparent_tls: bool = False parameter that skips the credential requirement check when the sidecar handles authentication. A separate PR to BerriAI/litellm will be submitted for this.

Stacked PRs:

• feat(aicore): clear AICORE_CLIENT_SECRET after token acquisition (AFSDK-4291) #257 (feat/aicore-clear-client-secret) — clears AICORE_CLIENT_SECRET from env after first token acquisition
• feat(aicore): transparent proxy routing and BTP Destination Service mode (Option 3) #271 (feat/aicore-proxy-routing) — Option 3: proxy routing + BTP Destination Service mode (does not require the LiteLLM upstream change)

[tiagoek]

Production validation — proactive credential reload (watch_aicore_config)

Test suite (cloud-sdk-python)

All 70 unit tests pass on branch feat/aicore-transparent-tls:

tests/aicore/unit/test_aicore_watcher.py          8 passed
tests/aicore/unit/test_credential_rotation_flow.py 6 passed
tests/aicore/unit/test_aicore.py                  (existing, green)
tests/aicore/unit/test_completion.py              (existing, green)

Integration into autonomous-documentation-org

The vendor shim (vendor/sap_cloud_sdk_aicore/) was synced with watch_aicore_config + _get_secret_dir_mtime.

Finding (critical): Starlette 1.3.1 does NOT propagate lifespan events to mounted sub-apps — verified with a live ASGI simulation:

# Starlette 1.3.1 — Mount does not trigger FastAPI sub-app lifespan
outer = Starlette(routes=[Mount('/sub', app=fastapi_sub_app)])
# result: started = []  (sub-app _lifespan never called)

watch_aicore_config() was therefore moved to app/main.py at module level alongside set_aicore_config() — the same proven path that loads credentials on every boot.

Production proof (Kyma managed runtime, auto-doc-dev-eu12)

Thread count in PID 1 before and after deploying commit a72f5e0:

Pod	Image	Threads in PID 1
Pre-fix (6ad5b0854983)	0.3.3-20260825... (no watcher call)	14
Post-fix (1858f9060d95)	0.3.3-20260825200836_fd446f2+1	15

The extra thread is aicore-secret-watcher (daemon, polls /etc/secrets/appfnd/aicore/aicore-instance/ mtime every 30 s). It will call set_aicore_config() on the next kubelet symlink-swap rotation without requiring a pod restart.

autonomous-documentation-org unit tests: 286 passed, 1 skipped — coverage 90.51%

[tiagoek]

E2E validation — reactive credential reload (live pod test)

Environment: Kyma managed runtime, namespace auto-doc-dev-eu12, pod autonomous-documentation-ee13df69-1858f9060d95 (image 0.3.3-20260825)

Test script

tests/integration/e2e_llm_reactive_reload.py — run via kubectl exec in the pod where the secret volume is mounted.

Execution output

[1] Real secret loaded from file: 263a99c5...
[2] Env poisoned with invalid secret
INFO sap_cloud_sdk_aicore.completion: AI Core credentials reloading after authentication failure
INFO sap_cloud_sdk_aicore: Loaded AICORE_CLIENT_SECRET from file: /etc/secrets/appfnd/aicore/aicore-instance/clientsecret
INFO sap_cloud_sdk_aicore: AI Core configuration has been set successfully
[3] Post-call secret: (cleared by _clear_client_secret — expected PR#257)
[4] LLM response: 'OK'
[OK] Test 1 PASSED — reactive reload intercepted the failure, retried, LLM responded
[5] LiteLLM aicore/client namespace attrs: [...in_memory_llm_clients_cache...]
[6] AICORE_CLIENT_SECRET in env: (absent — cleared after token acquisition)
[OK] Test 2 PASSED — os.environ is the sole credential store; no per-model client object
INFO sap_cloud_sdk_aicore.completion: AICORE_CLIENT_SECRET cleared from environment after token acquisition (AFSDK-4291)

What this proves

Scenario	Result
Invalid AICORE_CLIENT_SECRET in env → LiteLLM call	AuthenticationError caught by handler
reload_aicore_credentials() reads from mounted file	All 5 AICORE_* vars reloaded from /etc/secrets/appfnd/aicore/aicore-instance/
Retry with restored credentials	LLM responded 'OK'
PR#257 _clear_client_secret() behaviour	Secret cleared from env after token acquisition — confirmed expected
LiteLLM in_memory_llm_clients_cache	Memory cache keyed by model identifier, not by credential values — os.environ read fresh on every OAuth token request

[pedro-hca]