Investigate fail open on audit log failure to write.

[david-rocca]

Prevent organization mutations from silently succeeding when audit persistence fails

Issue metadata

• Type: Reliability / data integrity
• Affected area: Organization and user repositories, audit persistence, transactional API operations

Summary

Organization audit-write errors are caught and logged without being propagated to the calling repository or controller. As a result, an organization mutation can be committed and reported as successful even though its corresponding audit entry was not persisted.

The affected operations already pass a MongoDB session to both the canonical mutation and audit write. If audit history is required, propagating the audit error would allow the existing controller transaction handling to abort both operations atomically.

If audit history is intentionally best-effort, the current console-only failure handling should be replaced with a durable retry mechanism so audit records are not silently lost.

Current behavior

The shared audit helper catches every audit exception and then resolves normally:

async function createAuditLogEntry (
  registryOrg,
  originalRegistryOrgObject,
  requestingUserUUID,
  options
) {
  if (!requestingUserUUID) return

  try {
    // Seed the original state and append the changed state.
    await auditRepo.seedAuditHistoryForOrg(/* ... */)
    await auditRepo.appendToAuditHistoryForOrg(/* ... */)
  } catch (auditError) {
    console.error('Audit entry creation failed:', auditError)
  }
}

Because the exception is not rethrown, callers cannot distinguish a successful audit write from a failed one.

Separate organization create and partial-update paths contain equivalent inline try/catch blocks that also suppress audit errors.

Concrete failure scenario

Given the following existing BaseOrg:

{
  "UUID": "10000000-0000-4000-8000-000000000002",
  "short_name": "window_1",
  "long_name": "Booker LLC"
}

And an audit history whose latest entry also records "long_name": "Booker LLC":

1. A Secretariat user submits a valid organization update that changes the long name to Booker Security LLC.
2. The registry organization controller starts a MongoDB transaction.
3. updateOrgFull modifies the BaseOrg in memory.
4. createAuditLogEntry attempts to append the new organization state.
5. The audit repository rejects with Failed to save audit history entry.
6. createAuditLogEntry catches the exception, writes a console error, and resolves normally.
7. updateOrgFull saves the BaseOrg.
8. The controller commits the transaction and returns HTTP 200.

The resulting data can be inconsistent:

BaseOrg.long_name:            Booker Security LLC
Latest audited long_name:     Booker LLC
API response:                 200 Success
Durable audit retry scheduled: No

If creation of the initial audit document failed, the organization could be mutated without any corresponding audit history.

Some MongoDB errors may independently invalidate the transaction and cause the later save or commit to fail. However, the application does not guarantee fail-closed behavior because its explicit control flow suppresses every audit exception.

Impact

• Audit history may not accurately represent the current canonical BaseOrg state.
• Membership and administrator changes can be committed without an audit record.
• Clients receive a successful response and cannot know that audit persistence failed.
• Operators receive only a console error; there is no durable retry, dead-letter record, or explicit failure metric.
• Investigations or compliance reviews may incorrectly conclude that an organization change never occurred.

Known affected paths

The shared fail-open helper is used by operations including:

• full organization updates;
• adding and removing organization users;
• granting and revoking organization administrators;
• changing organization hierarchy;
• moving users between organizations;
• deleting users and removing their organization relationships.

Equivalent inline audit-error suppression exists in:

• organization creation;
• partial organization updates.

Relevant implementation locations:

• src/repositories/baseOrgRepositoryHelpers.js — createAuditLogEntry
• src/repositories/baseOrgRepository.js — createOrg, updateOrg, updateOrgFull, membership, administrator, and hierarchy mutations
• src/repositories/baseUserRepository.js — user operations that mutate BaseOrg relationships
• src/controller/registry.controller/org.registry.controller.js — transaction commit/abort behavior
• src/repositories/auditRepository.js — audit seed and append operations

Product decision required

Before implementation, confirm whether audit persistence is mandatory for organization mutations.

Option A: Audit is mandatory — recommended

Treat an audit-write failure as a failure of the entire mutation. Log the error and rethrow it so the surrounding transaction aborts.

Expected behavior:

Audit succeeds + canonical mutation succeeds -> commit
Audit fails                               -> abort both
Canonical mutation fails                  -> abort both

This option fits the current architecture because the controllers already use transactions and pass the same MongoDB session to canonical and audit operations.

Option B: Audit is best-effort

Allow the canonical mutation to commit, but atomically write an audit-outbox record in the same transaction. A worker must retry audit persistence and move repeatedly failing entries to a dead-letter state with monitoring and alerting.

Console-only error handling is not sufficient because it provides no delivery guarantee or recovery mechanism.

Recommended implementation

Assuming audit persistence is mandatory:

1. Update createAuditLogEntry to log and rethrow audit exceptions.
2. Remove or replace the inline fail-open audit try/catch blocks in organization create and partial-update paths.
3. Centralize audit creation through the shared helper where practical so all paths use the same failure policy.
4. Preserve the MongoDB session in every audit repository call.
5. Return a non-success API response when audit persistence prevents the transaction from committing.
6. Use structured application logging while retaining the original error as the cause.

Illustrative behavior:

try {
  await auditRepo.seedAuditHistoryForOrg(/* ... */)
  await auditRepo.appendToAuditHistoryForOrg(/* ... */)
} catch (auditError) {
  logger.error({ err: auditError }, 'Audit entry creation failed')
  throw auditError
}

Acceptance criteria

• The product owner confirms whether organization audit persistence is mandatory or best-effort.
• Audit failures are no longer silently reduced to a console error.
• All organization create/update and relationship-mutation paths use the selected failure policy consistently.
• If audit is mandatory, an audit seed or append failure aborts the surrounding transaction.
• If audit is mandatory, the failed operation does not change BaseOrg, membership, administrator, or hierarchy data.
• If audit is mandatory, the API returns a non-success response and does not report the mutation as successful.
• If audit is best-effort, a durable outbox entry is committed atomically with the canonical mutation.
• If audit is best-effort, retries, dead-letter handling, metrics, and alerting are implemented and documented.
• Structured logs include the target organization UUID, requesting user UUID, request UUID when available, operation, and original error.
• Existing successful audit behavior remains unchanged: a baseline is seeded when needed, and a changed state is appended only when the organization actually changes.

Testing requirements

• an injected audit failure calls abortTransaction;
• commitTransaction is not called;
• the BaseOrg remains unchanged;
• no partial membership or administrator mutation remains;
• the API does not return HTTP 200.

Definition of done

• The audit criticality decision is documented.
• All known fail-open audit call sites follow the chosen policy.
• Unit and integration tests cover audit failures and partial-write prevention or durable retry.
• Operational behavior, including alerts and recovery steps, is documented.
• No successful API response can correspond to an untracked audit loss.