v1.0.14

CLAUDE.md

@@ -17,7 +17,7 @@ node bin/workflow.js   # run the CLI directly (same as `npm run dev` / `npm star
 ```
 
 - **`build` is a no-op** (`echo 'Build not needed for now'`) — nothing to compile.
-- **There is no test suite, linter, or formatter configured.** Do not invent `npm test`/`npm run lint` commands; they will fail. Verify changes by running the CLI against a real vNext project directory.
+- **Index SQL tests**: `node --test test/indexes.test.js` (Node 18+); real PostgreSQL tests use `VNEXT_INDEX_TEST_URL=... node --test test/indexes.postgres.test.js` against a disposable database. No npm test/lint script is configured. Also verify the CLI against a real vNext project directory.
 
 The CLI always treats `process.cwd()` as the project root and requires at least one solution file (`vnext.config.json` or `vnext.{domain}.config.json`) in that directory. To exercise it, `cd` into a vNext workspace (not this repo) before running. A two-solution testbed lives at `../vnext-example` (`core` + `partner`); copy it into a scratch directory before running write-mode commands against it.
 
@@ -63,3 +63,13 @@ Every workspace command runs the table above **once per solution, sequentially**
 - Library functions (`discover.js`, `workflow.js`, `csx.js`) take a **solution object** (`projectRoot`, `componentsRoot`, `componentTypes`, …) rather than a bare `projectRoot` or reading cwd directly; commands receive it from `runForEachSolution`. Only `config.get('PROJECT_ROOT')` (inside `solutions.loadWorkspace`) reads cwd.
 - Commands are split into a thin `xxxCommand(options)` (header, prompts that must happen once, then `runForEachSolution`) and an `xxxSolution(solution, options)` body that returns `{ success, failed, errors }` so the workspace summary can aggregate.
 - DB and API helpers swallow connection errors and return `false`/`null` rather than throwing — callers treat a missing instance as "not in DB".
+
+## Manual attribute-index SQL
+
+`wf indexes generate` is strictly offline: `src/lib/indexes/` resolves local Master
+references and emits SQL; `src/commands/indexes.js` writes immutable batches. Never add DB execution
+or automatic sync/publish hooks. DBA execution owns the maintenance window. Keep physical keys/columns
+compatible with runtime `AttributeIndexDefinition` (`v1:latest` SHA-256 first 24 hex); version matching
+follows runtime `InstanceDataVersionComparer`, including package revisions. SQL compares actual index
+structure, skips equivalent indexes and only rebuilds/removes owned indexes. Projection retirement
+requires explicit `--retire-obsolete` and a complete local inventory of active workflow versions.

README.md

@@ -171,6 +171,74 @@ wf reset
 
 All workspace commands (`check`, `csx`, `sync`, `update`, `reset`) accept the global `--domain <name>` option. Without it they run once per solution file found in the project root.
 
+### `wf indexes generate`
+
+**Purpose**: Generate offline attribute-index SQL for planned execution by your DBA team.
+
+From a domain workspace containing `vnext.config.json`:
+
+```bash
+wf indexes generate --output ./index-sql
+wf indexes generate --flow money-transfer --output ./index-sql
+```
+
+Each invocation writes a **new batch folder** containing one `.sql` per workflow, a `manifest.json`
+(source paths, versions, SHA-256 checksums, physical index definitions), and execution notes. Earlier
+batches are preserved. The command never connects to API/DB and never executes SQL; `sync`, `update`
+and Master publication do not invoke it.
+
+The generator resolves local workflow → Master references from the configured component directories.
+All local versions of a workflow contribute requirements. Full `-pkg.` revisions remain pinned; `latest`,
+artifact, minor and major selectors use runtime version ordering. Build metadata is ignored. Referenced
+schemas must exist locally; incomplete references, duplicate versions and incompatible types fail.
+
+Indexed fields use `x-indexed: true`. Nested scalar string/number/integer/boolean fields are supported;
+dates use `format: "date-time"`. Filtering/sorting permissions remain `x-filterOperators`/`x-sortable`.
+The generator preserves the runtime `v1:latest` column/key contract; readable index names include a hash
+of their physical definition.
+
+The DBA reviews and executes each SQL file in a maintenance window:
+
+```bash
+psql -X -v ON_ERROR_STOP=1 --dbname=vNext_MyDomainDb --file=index-sql/<batch>/money_transfer.sql
+```
+
+SQL takes an advisory lock plus an **ACCESS EXCLUSIVE table lock**.
+It validates current/history data, adds stored generated columns in one table rewrite, reuses structurally
+matching indexes (including legacy names), rebuilds changed owned indexes, and updates the runtime ready
+catalog atomically. Index comparison covers keys, includes, ordering, collation, access method, operator
+classes and partial predicates. Unmanaged collisions abort; no `CASCADE` is issued. Replaying an unchanged
+batch preserves index OIDs and skips data validation, rewrites and ANALYZE. Missing/invalid values are not
+silently coerced; conversion errors identify the field and roll back all changes.
+
+This is **not concurrent DDL**. Plan disk/WAL/replica capacity for generated-column rewrites; the script's
+5-second lock timeout may be adjusted during DBA review. Text-search indexes require `pg_trgm` in `public`
+and `tr-TR-x-icu`. Runtime routing uses `AttributeIndexes:Enabled`; catalog refresh defaults to 30 seconds.
+
+Obsolete projections are retained by default for other deployed versions. To explicitly retire them:
+
+```bash
+wf indexes generate --flow money-transfer --retire-obsolete
+```
+
+Include **every still-active workflow version** locally and drain readers/writers for this maintenance.
+Retirement removes obsolete expressions/owned indexes and marks catalog entries not ready; columns and
+stored values remain. This is required when an old numeric/date cast would reject values after a type
+change. The offline tool cannot verify deployed versions: compare the source manifest before execution.
+Roll back query routing through the runtime's `AttributeIndexes:DisabledFlows`; schedule physical cleanup
+separately. See the runtime [maintenance runbook](https://github.com/burgan-tech/vnext/blob/main/docs/runtime/manual-attribute-index-maintenance.md).
+
+**Tests (Node 18+ test runner):**
+
+```bash
+node --test test/indexes.test.js
+# Use only a disposable PostgreSQL database; the test owns uniquely named fixture schemas.
+VNEXT_INDEX_TEST_URL=postgresql://user:password@localhost:5432/disposable_test \
+  node --test test/indexes.postgres.test.js
+```
+
+---
+
 ### `wf check`
 
 **Purpose**: System health check
@@ -198,7 +266,7 @@ wf check --domain partner  # one solution
    - Checks if it exists in DB (by key)
    - If **exists** → Skip (already synced)
    - If **not exists** → Publish to API
-3. Re-initializes the system
+3. Signals `definitions/publish/completed` once — the runtime's post-deployment hook
 
 **Use when**: Initial setup, adding new components without affecting existing ones
 
@@ -208,6 +276,39 @@ wf sync
 
 ---
 
+### Publish-completed
+
+`sync`, `update` and `reset` each end with one call to the runtime:
+
+```
+POST {baseUrl}/api/{version}/definitions/publish/completed
+```
+
+It runs the runtime's post-deployment hooks once per command. Today that means forcing a re-read of
+the domain discovery registry — and that matters more than it looks: the runtime's discovery endpoint
+cache carries **no TTL**, so this call is its only automatic invalidation. Without it a runtime keeps
+resolving cross-domain calls by whatever it learned at startup, until its pod restarts.
+
+The response is always HTTP 200; a failed hook is reported in the body, and the CLI prints each
+hook's outcome:
+
+```
+✔ Publish-completed ran 1 hook(s)
+    discovery-cache: Refreshed
+```
+
+`Refreshed` is the normal outcome. `SkippedNotOwner` is also fine — another replica is reading the
+registry and its result applies cluster-wide. `Disabled` means that runtime has no discovery cache
+(`ServiceDiscovery:Provider=dapr`, or the cache switched off). `Failed` means the registry could not
+be read and nothing else will retry it.
+
+> **Requires a runtime that exposes `definitions/publish/completed`.** It replaces
+> `definitions/re-initialize`, which older runtimes served as a no-op and current ones have removed.
+> Against an older runtime this call 404s and the CLI prints the warning above without failing the
+> command.
+
+---
+
 ### `wf update [options]`
 
 **Purpose**: Update changed components (delete + re-add)
@@ -218,7 +319,7 @@ wf sync
    - Checks if it exists in DB (by key)
    - If **exists** → Delete from DB, then publish to API
    - If **not exists** → Publish to API
-3. Re-initializes the system
+3. Signals `definitions/publish/completed` once — the runtime's post-deployment hook
 
 **Use when**: You modified existing components and want to update them
 
@@ -255,7 +356,7 @@ wf update -d Workflows/person      # Only Workflows/person
    - Checks if it exists in DB (by key)
    - If **exists** → Delete from DB, then publish to API
    - If **not exists** → Publish to API
-3. Re-initializes the system
+3. Signals `definitions/publish/completed` once — the runtime's post-deployment hook
 
 **Use when**: You need to force reset components regardless of changes
 
@@ -782,7 +883,7 @@ vnext-workflow-cli/
 │   │   ├── sync.js
 │   │   └── update.js
 │   └── lib/                 # Library modules
-│       ├── api.js           # API client (publish, reinitialize)
+│       ├── api.js           # API client (publish, publish-completed)
 │       ├── config.js        # CLI configuration
 │       ├── csx.js           # CSX processing
 │       ├── db.js            # Database operations
@@ -820,3 +921,36 @@ npm run dev
 ## 📝 License
 
 MIT License - see [LICENSE](LICENSE) for details.
+
+### Schema purpose for index generation
+
+Schema components use the existing `attributes.type` string. Its values are not restricted to an enum;
+legacy and custom types remain valid. Only the exact value `master` permits `x-indexed` metadata
+(including `false`) and contributes index SQL. Missing, null, blank or other values never mean master.
+The existing publication requirement for a non-empty schema type is unchanged. JSON Schema `type`
+keywords within `attributes.schema` retain their existing meaning.
+A latest reference resolving to a non-master schema does not fall back to an older master version.
+No matching masters means no SQL batch. There is no separate component root `type` field.
+
+```json
+{
+  "key": "order-master",
+  "domain": "sales",
+  "flow": "sys-schemas",
+  "version": "1.0.0",
+  "flowVersion": "1.0.0",
+  "tags": ["orders"],
+  "attributes": {
+    "type": "master",
+    "schema": {
+      "type": "object",
+      "properties": {
+        "amount": { "type": "number", "x-indexed": true, "x-filterOperators": ["gt"] }
+      }
+    }
+  }
+}
+```
+
+The schema validator reads `SchemaDefinition.Type` (`attributes.type`) during publication, including
+schema seed items. No extra envelope metadata is required.

bin/workflow.js

@@ -82,6 +82,15 @@ program
   .description('Reset workflows (force update)')
   .action(resetCommand);
 
+// Offline index maintenance artifacts; execution is owned by the DBA team.
+program.command('indexes').description('Generate DBA-reviewed attribute index SQL')
+  .command('generate')
+  .description('Read local workflow/Master definitions and write SQL files without contacting API/DB')
+  .option('--flow <key>', 'Generate for one workflow key (all its local versions)')
+  .option('-o, --output <directory>', 'Parent folder for a new immutable SQL batch', 'index-sql')
+  .option('--retire-obsolete', 'Retire obsolete projections; requires ALL active workflow versions locally')
+  .action(require('../src/commands/indexes'));
+
 // Config command
 program
   .command('config')

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@burgan-tech/vnext-workflow-cli",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "vNext Workflow Manager - CLI tool for managing workflows, tasks, schemas and more",
   "main": "dist/index.js",
   "bin": {

src/commands/indexes.js

@@ -0,0 +1,46 @@
+const fs = require('fs');
+const path = require('path');
+const { LOG } = require('../lib/ui');
+const { loadPlans, hash } = require('../lib/indexes/definitions');
+const { generateSql, indexes } = require('../lib/indexes/sql');
+
+async function generate(options, projectRoot = process.cwd()) {
+  const plans = await loadPlans(projectRoot, options.flow);
+  if (!plans.length) throw new Error("No workflows referencing attributes.type: master schemas found; no SQL files generated.");
+  // Validate/render the whole batch before creating any output; never overwrite an earlier batch.
+  const rendered = plans.map(plan => ({ plan, sql: generateSql(plan, options) }));
+  const output = path.resolve(projectRoot, options.output || 'index-sql');
+  fs.mkdirSync(output, { recursive: true });
+  const batch = fs.mkdtempSync(path.join(output, new Date().toISOString().replace(/[:.]/g, '-') + '-'));
+  const manifest = { formatVersion: 1, physicalContract: 'v1:latest', generatorVersion: require('../../package.json').version,
+    domain: plans[0].domain, retireObsolete: !!options.retireObsolete,
+    generatedAt: new Date().toISOString(), flows: [] };
+  for (const { plan, sql } of rendered) {
+    const file = plan.schema + '.sql';
+    fs.writeFileSync(path.join(batch, file), sql, { flag: 'wx' });
+    manifest.flows.push({ ...plan, projections: plan.projections.map(p => ({ ...p, indexes: indexes(p) })), file, sha256: hash(sql) });
+  }
+  fs.writeFileSync(path.join(batch, 'manifest.json'), JSON.stringify(manifest, null, 2) + '\n', { flag: 'wx' });
+  fs.writeFileSync(path.join(batch, 'README.txt'), [
+    `Domain: ${manifest.domain}. Generated offline; nothing was executed.`,
+    'DBA: review manifest/source versions and each SQL file. Execute files individually in a maintenance window.',
+    'Example: psql -X -v ON_ERROR_STOP=1 --dbname=<target-database> --file=<flow.sql>',
+    'Scripts lock InstancesData ACCESS EXCLUSIVE and use one transaction per flow. Plan disk/WAL/replica capacity for table rewrites.',
+    'A 5s lock_timeout is included; adjust it during DBA review if necessary. Re-run the SAME script after rollback on failure.',
+    'Existing correct indexes are adopted; changed CLI-owned indexes are rebuilt. Unmanaged name collisions fail.',
+    'No obsolete projection is retired unless --retire-obsolete was explicitly passed. Retirement requires all active versions locally and drained runtime readers/writers.',
+    'Stored values/columns are retained on retirement; physical cleanup remains a separate DBA operation.',
+    'Runtime reads use AttributeIndexes:Enabled and refresh the catalog after CatalogCacheSeconds (default 30).',
+    'Rollback routing with AttributeIndexes:DisabledFlows. Do not drop columns under running readers.', ''
+  ].join('\n'), { flag: 'wx' });
+  return { batch, count: rendered.length };
+}
+async function command(options) {
+  try {
+    const { batch, count } = await generate(options);
+    LOG.success(`Generated ${count} SQL file(s): ${batch}`);
+    LOG.info('No API or database connection was made. Give this batch to your DBA for review and execution.');
+  } catch (error) { LOG.error(error.message); process.exitCode = 1; }
+}
+module.exports = command;
+module.exports.generate = generate;

src/commands/reset.js

@@ -7,7 +7,7 @@ const { buildDbConfig, buildApiConfig } = require('../lib/config');
 const { discoverComponents, toGlobPattern, JSON_IGNORE_PATTERNS } = require('../lib/discover');
 const { loadWorkspace, runForEachSolution } = require('../lib/solutions');
 const { getJsonMetadata, findAllJson, detectComponentType, checkComponentDomain } = require('../lib/workflow');
-const { publishComponent, reinitializeSystem } = require('../lib/api');
+const { publishComponent, publishCompleted } = require('../lib/api');
 const { getInstanceId, deleteWorkflow } = require('../lib/db');
 const { LOG, printApiError, printErrorSummaryTable } = require('../lib/ui');
 
@@ -189,18 +189,26 @@ async function resetSolution(solution) {
     }
   }
 
-  // Re-initialize
+  // Post-deployment hook. One call per command, after the publish loop: the runtime's discovery
+  // endpoint cache has no TTL, so this is its only automatic invalidation.
   const totalSuccess = Object.values(componentStats).reduce((sum, s) => sum + s.success, 0);
 
   if (totalSuccess > 0) {
     console.log();
-    const reinitSpinner = ora('  Re-initializing system...').start();
-    const reinitSuccess = await reinitializeSystem(apiConfig.baseUrl, apiConfig.version);
+    const hookSpinner = ora('Signalling publish-completed...').start();
+    const hookResult = await publishCompleted(apiConfig.baseUrl, apiConfig.version);
 
-    if (reinitSuccess) {
-      reinitSpinner.succeed(chalk.green('  System re-initialized'));
+    if (hookResult.success) {
+      hookSpinner.succeed(chalk.green(`Publish-completed ran ${hookResult.hooks.length} hook(s)`));
     } else {
-      reinitSpinner.warn(chalk.yellow('  System re-initialization failed (continuing)'));
+      hookSpinner.warn(chalk.yellow(`Publish-completed reported a problem: ${hookResult.error} (continuing)`));
+    }
+
+    // Printed per hook rather than summarised: a hook that did not run means cross-domain
+    // endpoints stay as they were until the next deployment, and nothing else will say so.
+    for (const hook of hookResult.hooks) {
+      const line = `${hook.name}: ${hook.outcome}${hook.message ? ` - ${hook.message}` : ''}`;
+      console.log(hook.outcome === 'Failed' ? chalk.yellow(`    ${line}`) : chalk.dim(`    ${line}`));
     }
   }
 

src/commands/sync.js

@@ -4,7 +4,7 @@ const path = require('path');
 const { buildDbConfig, buildApiConfig } = require('../lib/config');
 const { discoverComponents, findAllJsonFiles } = require('../lib/discover');
 const { runForEachSolution } = require('../lib/solutions');
-const { publishComponent, reinitializeSystem } = require('../lib/api');
+const { publishComponent, publishCompleted } = require('../lib/api');
 const { getInstanceId, deleteWorkflow } = require('../lib/db');
 const { getJsonMetadata, detectComponentType, checkComponentDomain } = require('../lib/workflow');
 const { processCsxFile, findAllCsx } = require('../lib/csx');
@@ -151,18 +151,26 @@ async function syncSolution(solution) {
     }
   }
 
-  // Re-initialize
+  // Post-deployment hook. One call per command, after the publish loop: the runtime's discovery
+  // endpoint cache has no TTL, so this is its only automatic invalidation.
   const totalSuccess = Object.values(componentStats).reduce((sum, s) => sum + s.success, 0);
 
   if (totalSuccess > 0) {
     console.log();
-    const reinitSpinner = ora('Re-initializing system...').start();
-    const reinitSuccess = await reinitializeSystem(apiConfig.baseUrl, apiConfig.version);
+    const hookSpinner = ora('Signalling publish-completed...').start();
+    const hookResult = await publishCompleted(apiConfig.baseUrl, apiConfig.version);
 
-    if (reinitSuccess) {
-      reinitSpinner.succeed(chalk.green('System re-initialized'));
+    if (hookResult.success) {
+      hookSpinner.succeed(chalk.green(`Publish-completed ran ${hookResult.hooks.length} hook(s)`));
     } else {
-      reinitSpinner.warn(chalk.yellow('System re-initialization failed'));
+      hookSpinner.warn(chalk.yellow(`Publish-completed reported a problem: ${hookResult.error}`));
+    }
+
+    // Printed per hook rather than summarised: a hook that did not run means cross-domain
+    // endpoints stay as they were until the next deployment, and nothing else will say so.
+    for (const hook of hookResult.hooks) {
+      const line = `${hook.name}: ${hook.outcome}${hook.message ? ` - ${hook.message}` : ''}`;
+      console.log(hook.outcome === 'Failed' ? chalk.yellow(`    ${line}`) : chalk.dim(`    ${line}`));
     }
   }