feat(service-settings): the namespace read lists what is stored beneath an env pin

.changeset/30823-settings-stored-beneath-pin.md

@@ -0,0 +1,13 @@
+---
+'@objectstack/service-settings': patch
+---
+
+feat(service-settings): the namespace read lists what is stored beneath a field an `OS_*` variable pins, so Setup can tell whether "clear the saved value" has anything to clear
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+- **Now.** `getNamespace()`, the read `GET /api/settings/:namespace` serves, answers a key whose `OS_*` override is in force with the same `value`, `source: 'env'`, `locked: true` and `lockedReason` as before, and its `cascadeChain` now lists, below the `env` entry, the entries the key's read carries with no override in force: the stored rows the caller's cascade draws on (`global`, `tenant`, `user`, each with its own lock), then `default`. The `env` entry stays first and is still the only one flagged `effective`. An entry at the key's declared scope with a non-null value is the value a clear (`null`, accepted through a pin since the ADR-0007 D6 amendment) removes; after that clear the entry reads `null`.
+- **Secrets.** A secret key's entries beneath a pin carry the read mask when a value is stored and `null` when none is, in process too; the stored secret is never decrypted for them, so a key pinned because its stored secret no longer decrypts still reads.
+- **Unchanged.** `get()` and `getMany()` still answer an override in force from the env alone, without reading the store: they are the reads request paths call. An override whose value coerces to `null` keeps its single `env` entry, because any non-null entry listed below it would read as the effective one.
+- **Cost.** A pinned key joins the namespace read's existing grouped row load, so the read costs one more load only when every key of a group is pinned.

content/docs/protocol/kernel/config-resolution.mdx

@@ -262,6 +262,19 @@ OS_ALLOWED_ORIGINS=["a","b","c"]   # default is array/object → JSON.parse
 
 If the raw value cannot be parsed for the expected type, the raw string is used.
 
+**What is stored beneath a pin is listed by the namespace read.** An override in
+force answers `source: 'env'`, `locked: true`, and `get()` / `getMany()` build
+that answer from the env alone, without reading the store. `getNamespace()` — the
+read `GET /api/settings/:namespace` serves to the Setup page — also lists, below
+the `env` entry of `cascadeChain`, the entries the key's read would carry with no
+override in force: its stored rows, then the default. The `env` entry stays the
+effective one. An entry at the key's declared scope with a non-null value is the
+value a clear (`null`) removes, so the page can tell whether "clear the saved
+value" has anything to clear. A secret's entries beneath the pin carry the read
+mask or `null`, never its value. An override whose value coerces to `null` keeps
+its single `env` entry, since any non-null entry below it would read as the
+effective one.
+
 **A value outside a declared option table is ignored, not repaired.** When the
 specifier declares `options` and the env value matches none of them, the env
 layer contributes nothing at all — no value and no `cascadeChain` entry — and

packages/services/service-settings/README.md

@@ -34,7 +34,11 @@ Such fields are returned with `source: 'env', locked: true`. Every write
 (service or REST) but a clear (`null`) fails with HTTP 409. A clear removes
 the value stored beneath the override and changes nothing in force: the key
 still answers the env value, and falls back to its default once the variable
-is removed.
+is removed. The namespace read (`getNamespace`, `GET /api/settings/:namespace`)
+lists the stored entries beneath such a field's `env` entry in `cascadeChain`,
+so a client can tell whether a clear has a value to remove (a secret's entry
+shows the mask, never the value); `get()` and `getMany()` answer it from the
+env alone.
 
 ## Encryption
 

packages/services/service-settings/src/settings-getmany.test.ts

@@ -242,3 +242,24 @@ describe('[#10826] SettingsService.getMany', () => {
     expect((await svc.get('localization', 'timezone', ctx)).value).toBe('America/New_York');
   });
 });
+
+describe('[#30823] getNamespace lists a pinned key\'s stored rows from the same grouped load', () => {
+  it('a pin over a global row lists that row beneath its env entry, at no more finds than the read without the pin', async () => {
+    const { svc, engine } = await makeService();
+    const ctx = { userId: 'u1' };
+    engine.find.mockClear();
+    await svc.getNamespace('localization', ctx);
+    const unpinnedFinds = engine.find.mock.calls.length;
+
+    process.env.OS_LOCALIZATION_TIMEZONE = 'Asia/Tokyo';
+    engine.find.mockClear();
+    const ns = await svc.getNamespace('localization', ctx);
+    expect(engine.find).toHaveBeenCalledTimes(unpinnedFinds);
+    expect(ns.values.timezone).toMatchObject({ value: 'Asia/Tokyo', source: 'env', locked: true });
+    expect(ns.values.timezone.cascadeChain).toEqual([
+      expect.objectContaining({ scope: 'env', value: 'Asia/Tokyo', effective: true }),
+      { scope: 'global', value: 'America/New_York', locked: false, lockedReason: undefined },
+      { scope: 'default', value: 'UTC' },
+    ]);
+  });
+});

packages/services/service-settings/src/settings-routes.test.ts

@@ -614,8 +614,10 @@ describe('settings-routes — #7522 encrypted values are redacted at the REST bo
     expect(r1.state.body.data.values.smtp_password.source).toBe('env');
     expect(r1.state.body.data.values.smtp_password.locked).toBe(true);
     expect(r1.state.body.data.values.smtp_password.lockedReason).toContain('OS_SECRETSNS_SMTP_PASSWORD');
+    // Beneath the env entry, the stored cascade: nothing stored, so the default.
     expect(r1.state.body.data.values.smtp_password.cascadeChain).toEqual([
       expect.objectContaining({ scope: 'env', value: SETTINGS_SECRET_MASK, locked: true }),
+      { scope: 'default', value: null },
     ]);
 
     const write = http.routes.get('PUT /api/settings/:namespace')!;
@@ -921,3 +923,78 @@ describe('settings-routes — the two retired Branding keys (#24766)', () => {
     expect(await storedRetiredRows()).toEqual(RETIRED_BRANDING_ROWS);
   });
 });
+
+describe('settings-routes — GET /:ns lists what is stored beneath an env pin', () => {
+  /** The secret stack with an env the test pins after storing. */
+  function pinnableSecretStack() {
+    const env: Record<string, string | undefined> = {};
+    const secretRows = new Map<string, any>();
+    const cryptoProvider = new LocalCryptoProvider();
+    const svc = new SettingsService({
+      env,
+      cryptoProvider,
+      secretStore: {
+        async insert(row) { secretRows.set(row.id, row); return { id: row.id }; },
+        async get(id) { return secretRows.get(id) ?? null; },
+        async update(id, patch) { secretRows.set(id, { ...secretRows.get(id), ...patch }); },
+      },
+    });
+    svc.registerManifest(secretsManifest);
+    const http = new MockHttp();
+    registerSettingsRoutes(http, svc, { contextFromRequest: secretAdmin });
+    const read = async () => {
+      const r = makeReqRes({ params: { namespace: 'secretsns' } });
+      await http.routes.get('GET /api/settings/:namespace')!(r.req, r.res);
+      return r.state;
+    };
+    return { env, svc, http, cryptoProvider, read };
+  }
+
+  it('a pinned key\'s stored row rides beneath its env entry; a pinned secret\'s shows the mask, never the value', async () => {
+    const { env, svc, cryptoProvider, read } = pinnableSecretStack();
+    await svc.set('secretsns', 'smtp_host', 'stored.example.com', { userId: 'usr_1' });
+    await svc.set('secretsns', 'smtp_password', SMTP_PLAINTEXT, { userId: 'usr_1' });
+    env.OS_SECRETSNS_SMTP_HOST = 'env.example.com';
+    env.OS_SECRETSNS_SMTP_PASSWORD = 'env-supplied-secret';
+    env.OS_SECRETSNS_WEBHOOK_TOKEN = 'env-supplied-token';
+    const decrypt = vi.spyOn(cryptoProvider, 'decrypt');
+
+    const state = await read();
+    expect(state.status).toBe(200);
+    const values = state.body.data.values;
+    expect(values.smtp_host).toMatchObject({ value: 'env.example.com', source: 'env', locked: true });
+    expect(values.smtp_host.cascadeChain).toEqual([
+      expect.objectContaining({ scope: 'env', value: 'env.example.com', effective: true }),
+      { scope: 'user', value: 'stored.example.com' },
+      { scope: 'default', value: 'smtp.example.com' },
+    ]);
+    expect(values.smtp_password.cascadeChain).toEqual([
+      expect.objectContaining({ scope: 'env', value: SETTINGS_SECRET_MASK, effective: true }),
+      { scope: 'user', value: SETTINGS_SECRET_MASK },
+      { scope: 'default', value: null },
+    ]);
+    // A pinned secret with nothing stored lists no stored entry.
+    expect(values.webhook_token.cascadeChain.slice(1)).toEqual([{ scope: 'default', value: null }]);
+    const wire = JSON.stringify(state.body);
+    expect(wire).not.toContain(SMTP_PLAINTEXT);
+    expect(wire).not.toContain('env-supplied-secret');
+    expect(decrypt).not.toHaveBeenCalled();
+  });
+
+  it('after PUT clears the stored value through the pin, the next read lists it null', async () => {
+    const { env, svc, http, read } = pinnableSecretStack();
+    await svc.set('secretsns', 'smtp_host', 'stored.example.com', { userId: 'usr_1' });
+    env.OS_SECRETSNS_SMTP_HOST = 'env.example.com';
+
+    const put = makeReqRes({ params: { namespace: 'secretsns' }, body: { smtp_host: null } });
+    await http.routes.get('PUT /api/settings/:namespace')!(put.req, put.res);
+    expect(put.state.status).toBe(200);
+
+    const values = (await read()).body.data.values;
+    expect(values.smtp_host).toMatchObject({ value: 'env.example.com', source: 'env', locked: true });
+    expect(values.smtp_host.cascadeChain.slice(1)).toEqual([
+      { scope: 'user', value: null },
+      { scope: 'default', value: 'smtp.example.com' },
+    ]);
+  });
+});

packages/services/service-settings/src/settings-service.test.ts

@@ -18,6 +18,7 @@ import { companySettingsManifest } from './manifests/company.manifest.js';
 import { brandingSettingsManifest } from './manifests/branding.manifest.js';
 import { builtinSettingsManifests } from './manifests/index.js';
 import { evaluateVisibility, visibilitySource } from './visibility-eval.js';
+import { SETTINGS_SECRET_MASK } from './settings-secret-redaction.js';
 import { SettingsManifestSchema } from '@objectstack/spec/system';
 
 describe('reference manifests are spec-valid', () => {
@@ -2969,3 +2970,161 @@ describe('SettingsService — a reset clears the value stored beneath an env pin
     expect(r.message).toBe('Cleared 1 saved value(s); environment/default configuration is back in effect.');
   });
 });
+
+describe('SettingsService — the namespace read lists what is stored beneath an env pin', () => {
+  // A pin on each rung a clear can land at, a secret, and a JSON key whose env
+  // value can coerce to null.
+  const BENEATH = {
+    namespace: 'beneath',
+    version: 1,
+    label: 'Beneath',
+    scope: 'tenant',
+    specifiers: [
+      { type: 'text', key: 'motto', label: 'Motto', scope: 'tenant', default: 'none' },
+      { type: 'text', key: 'banner', label: 'Banner', scope: 'global', default: 'none' },
+      { type: 'text', key: 'nickname', label: 'Nickname', scope: 'user', default: 'none' },
+      { type: 'password', key: 'api_secret', label: 'API secret', scope: 'tenant' },
+      { type: 'password', key: 'spare_secret', label: 'Spare secret', scope: 'tenant' },
+      { type: 'json', key: 'rules', label: 'Rules', scope: 'tenant', default: [] },
+    ],
+  } as any;
+  const envEntry = (value: unknown, envName: string) => ({
+    scope: 'env', value, locked: true, lockedReason: `Set via env: ${envName}`, effective: true,
+  });
+
+  /** A crypto adapter whose decrypt is watched, so a read can prove it never opened a secret. */
+  function watchedCrypto(): CryptoAdapter & { decrypt: ReturnType<typeof vi.fn> } {
+    return {
+      confidential: true,
+      encrypt: async (plain: string) => 'kms:' + Buffer.from(plain, 'utf8').toString('base64'),
+      decrypt: vi.fn(async (cipher: string) => Buffer.from(cipher.replace(/^kms:/, ''), 'base64').toString('utf8')),
+      digest: () => 'fnv32:deadbeef',
+    };
+  }
+
+  /** `motto` stored then pinned: the operator's recovery, before the clear. */
+  async function pinnedOverStored() {
+    const env: Record<string, string | undefined> = {};
+    const crypto = watchedCrypto();
+    const svc = new SettingsService({ env, crypto });
+    svc.registerManifest(BENEATH);
+    await svc.set('beneath', 'motto', 'Stored motto');
+    env.OS_BENEATH_MOTTO = 'Pinned motto';
+    return { env, svc, crypto };
+  }
+
+  it('a pinned key over a stored row lists that row beneath its env entry; the answer is otherwise the pin\'s', async () => {
+    const { svc } = await pinnedOverStored();
+    const { values } = await svc.getNamespace('beneath');
+
+    expect(values.motto).toEqual({
+      value: 'Pinned motto',
+      source: 'env',
+      locked: true,
+      lockedReason: 'Set via env: OS_BENEATH_MOTTO',
+      cascadeChain: [
+        envEntry('Pinned motto', 'OS_BENEATH_MOTTO'),
+        { scope: 'tenant', value: 'Stored motto', locked: false, lockedReason: undefined },
+        { scope: 'default', value: 'none' },
+      ],
+    });
+    // The env entry is the only effective one, and the first non-null one.
+    expect(values.motto.cascadeChain!.filter((e) => e.effective)).toHaveLength(1);
+  });
+
+  it('after the clear through the pin the stored entry reads null; a pin with no row lists none', async () => {
+    const { env, svc } = await pinnedOverStored();
+    await svc.set('beneath', 'motto', null);
+    env.OS_BENEATH_BANNER = 'Pinned banner';
+
+    const { values } = await svc.getNamespace('beneath');
+    expect(values.motto.cascadeChain).toEqual([
+      envEntry('Pinned motto', 'OS_BENEATH_MOTTO'),
+      { scope: 'tenant', value: null, locked: false, lockedReason: undefined },
+      { scope: 'default', value: 'none' },
+    ]);
+    expect(values.banner.cascadeChain).toEqual([
+      envEntry('Pinned banner', 'OS_BENEATH_BANNER'),
+      { scope: 'default', value: 'none' },
+    ]);
+  });
+
+  it('a global key lists its global row, and an upper row\'s lock rides along, so a refused clear is visible', async () => {
+    const { env, svc } = await pinnedOverStored();
+    await svc.set('beneath', 'banner', 'Stored banner');
+    env.OS_BENEATH_BANNER = 'Pinned banner';
+    await (svc as any).upsertRow({
+      namespace: 'beneath', key: 'motto', scope: 'global', user_id: null, value: 'Platform motto',
+      value_enc: null, encrypted: false, locked: true, locked_reason: 'Set by the platform.',
+    });
+
+    const { values } = await svc.getNamespace('beneath');
+    expect(values.banner.cascadeChain).toEqual([
+      envEntry('Pinned banner', 'OS_BENEATH_BANNER'),
+      { scope: 'global', value: 'Stored banner', locked: false, lockedReason: undefined },
+      { scope: 'default', value: 'none' },
+    ]);
+    expect(values.motto.cascadeChain!.slice(1, 3)).toEqual([
+      { scope: 'global', value: 'Platform motto', locked: true, lockedReason: 'Set by the platform.' },
+      { scope: 'tenant', value: 'Stored motto', locked: false, lockedReason: undefined },
+    ]);
+    // The pin still answers: an upper row's lock does not take the lock over.
+    expect(values.motto).toMatchObject({ source: 'env', lockedReason: 'Set via env: OS_BENEATH_MOTTO' });
+  });
+
+  it('a user key lists the caller\'s own row only', async () => {
+    const env: Record<string, string | undefined> = {};
+    const svc = new SettingsService({ env });
+    svc.registerManifest(BENEATH);
+    await svc.set('beneath', 'nickname', 'Ada', { userId: 'usr_a' });
+    await svc.set('beneath', 'nickname', 'Bob', { userId: 'usr_b' });
+    env.OS_BENEATH_NICKNAME = 'Pinned';
+
+    const forA = (await svc.getNamespace('beneath', { userId: 'usr_a' })).values.nickname.cascadeChain!;
+    expect(forA.slice(1)).toEqual([{ scope: 'user', value: 'Ada' }, { scope: 'default', value: 'none' }]);
+    const forNobody = (await svc.getNamespace('beneath')).values.nickname.cascadeChain!;
+    expect(forNobody.slice(1)).toEqual([{ scope: 'default', value: 'none' }]);
+  });
+
+  it('a secret beneath a pin shows that a value is stored, never the value, and is never decrypted', async () => {
+    const { env, svc, crypto } = await pinnedOverStored();
+    await svc.set('beneath', 'api_secret', 'stored-secret-plaintext');
+    env.OS_BENEATH_API_SECRET = 'env-secret';
+    env.OS_BENEATH_SPARE_SECRET = 'env-spare';
+    crypto.decrypt.mockClear();
+
+    const { values } = await svc.getNamespace('beneath');
+    expect(values.api_secret.cascadeChain).toEqual([
+      envEntry('env-secret', 'OS_BENEATH_API_SECRET'),
+      { scope: 'tenant', value: SETTINGS_SECRET_MASK, locked: false, lockedReason: undefined },
+      { scope: 'default', value: null },
+    ]);
+    expect(values.spare_secret.cascadeChain!.slice(1)).toEqual([{ scope: 'default', value: null }]);
+    expect(JSON.stringify(values)).not.toContain('stored-secret-plaintext');
+    expect(crypto.decrypt).not.toHaveBeenCalled();
+
+    // Cleared through the pin, the secret's stored entry reads null.
+    await svc.set('beneath', 'api_secret', null);
+    const after = (await svc.getNamespace('beneath')).values.api_secret.cascadeChain!;
+    expect(after[1]).toEqual({ scope: 'tenant', value: null, locked: false, lockedReason: undefined });
+  });
+
+  it('an override whose value coerces to null keeps its single env entry', async () => {
+    const env: Record<string, string | undefined> = {};
+    const svc = new SettingsService({ env });
+    svc.registerManifest(BENEATH);
+    await svc.set('beneath', 'rules', ['stored']);
+    env.OS_BENEATH_RULES = 'null';
+
+    const { values } = await svc.getNamespace('beneath');
+    expect(values.rules).toMatchObject({ value: null, source: 'env', locked: true });
+    expect(values.rules.cascadeChain).toEqual([envEntry(null, 'OS_BENEATH_RULES')]);
+  });
+
+  it('get() and getMany() still answer a pin from the env alone', async () => {
+    const { svc } = await pinnedOverStored();
+    const pinOnly = [envEntry('Pinned motto', 'OS_BENEATH_MOTTO')];
+    expect((await svc.get('beneath', 'motto')).cascadeChain).toEqual(pinOnly);
+    expect((await svc.getMany('beneath', ['motto'])).motto.cascadeChain).toEqual(pinOnly);
+  });
+});