diff --git a/backend/docs/data-migration.md b/backend/docs/data-migration.md new file mode 100644 index 0000000..33ae3d9 --- /dev/null +++ b/backend/docs/data-migration.md @@ -0,0 +1,513 @@ +# Data Migration and Backfill Runbook + +## Purpose + +This runbook defines the safe procedure for MongoDB schema migrations, legacy-data normalization, EventLog backfills, index maintenance, and demo-data reseeding in SwiftChain Backend. + +The migration workflow is: + +```text +Backup → Dry Run → Throttle → Apply → Verify → Monitor → Rollback if required +``` + +> **Repository note:** the requested documentation location is `backend/docs/`. The application models and seed script currently live under the repository-root `src/` directory, including `src/models/Delivery.ts`, `src/models/EventLog.ts`, and `src/seed.ts`. + +--- + +## 1. Pre-Migration Checklist + +Before changing production data: + +* [ ] Confirm the migration target and affected collection. +* [ ] Confirm the migration has been reviewed and approved. +* [ ] Confirm the deployment window. +* [ ] Confirm a recent MongoDB backup exists and is restorable. +* [ ] Record current document counts. +* [ ] Record current index definitions. +* [ ] Run the migration in dry-run mode against a staging copy. +* [ ] Estimate the number of documents to modify. +* [ ] Define batch size and throttle interval. +* [ ] Define expected post-migration invariants. +* [ ] Define rollback criteria. +* [ ] Confirm application compatibility with both old and new data formats. +* [ ] Confirm no destructive operation is being executed without a verified backup. + +### Baseline queries + +```javascript +db.deliveries.countDocuments({}) +db.eventlogs.countDocuments({}) + +db.deliveries.aggregate([ + { $group: { _id: "$status", count: { $sum: 1 } } }, + { $sort: { _id: 1 } } +]) + +db.eventlogs.aggregate([ + { $group: { _id: "$eventType", count: { $sum: 1 } } }, + { $sort: { _id: 1 } } +]) +``` + +Capture these values before and after the migration. + +--- + +## 2. Backup + +Create a backup before production migration work. + +Example: + +```bash +mongodump \ + --uri="$MONGODB_URI" \ + --db="$(echo "$MONGODB_URI" | sed -E 's/.*\/([^/?]+).*/\1/')" \ + --out="./backup/$(date +%Y%m%d-%H%M%S)" +``` + +Verify that the backup contains the expected collections before continuing. + +For production environments, use the organization's managed MongoDB backup/snapshot mechanism where applicable. + +--- + +## 3. Legacy Delivery Status Migration + +The current canonical delivery statuses are defined in: + +```text +src/models/Delivery.ts +``` + +Canonical values include: + +```text +pending +funded +assigned +in_progress +completed +cancelled +``` + +### Legacy-to-canonical mapping + +| Legacy status | Canonical status | Migration note | +| ------------- | ---------------- | ----------------------------------------------------------- | +| `picked_up` | `in_progress` | Delivery has left pickup and is actively being transported. | +| `in_transit` | `in_progress` | Delivery is actively being transported. | +| `delivered` | `completed` | Delivery has reached its completed state. | + +No legacy value should be silently converted to `pending`, `funded`, `assigned`, or `cancelled`. + +### Dry-run query + +```javascript +db.deliveries.aggregate([ + { + $match: { + status: { + $in: ["picked_up", "in_transit", "delivered"] + } + } + }, + { + $group: { + _id: "$status", + count: { $sum: 1 } + } + } +]) +``` + +### Migration outline + +Use a bounded batch operation rather than modifying an unbounded production collection in one request. + +```javascript +const statusMap = { + picked_up: "in_progress", + in_transit: "in_progress", + delivered: "completed" +}; + +for (const [legacyStatus, canonicalStatus] of Object.entries(statusMap)) { + const result = await db.deliveries.updateMany( + { status: legacyStatus }, + { $set: { status: canonicalStatus } } + ); + + console.log({ + legacyStatus, + canonicalStatus, + modifiedCount: result.modifiedCount + }); +} +``` + +For large collections, process `_id` ranges or limited batches and pause between batches. + +### Post-migration validation + +```javascript +db.deliveries.countDocuments({ + status: { + $in: ["picked_up", "in_transit", "delivered"] + } +}) +``` + +Expected result: + +```text +0 +``` + +Then verify the canonical distribution: + +```javascript +db.deliveries.aggregate([ + { $group: { _id: "$status", count: { $sum: 1 } } }, + { $sort: { _id: 1 } } +]) +``` + +--- + +## 4. EventLog Backfill + +The EventLog model is: + +```text +src/models/EventLog.ts +``` + +The model stores: + +* `eventType` +* `transactionHash` +* `ledgerSequence` +* `contractId` +* `eventData` +* `processedAt` +* `status` +* timestamps + +EventLog has a unique compound index on: + +```javascript +{ transactionHash: 1, eventType: 1 } +``` + +This uniqueness constraint is the primary database-level deduplication mechanism for backfills. + +### Backfill flow + +```text +Determine ledger range + ↓ +Read on-chain events + ↓ +Normalize event type/data + ↓ +Check transactionHash + eventType + ↓ +Insert missing EventLog record + ↓ +Mark processed when successfully handled + ↓ +Record failed events for retry +``` + +### Determine the starting ledger + +Use the latest successfully processed ledger for the relevant event type: + +```typescript +const lastProcessedLedger = + await EventLog.getLastProcessedLedger(eventType); +``` + +For an initial backfill, define an explicit starting ledger rather than assuming the current ledger. + +### Event insertion + +Backfill records should preserve the original transaction hash and ledger sequence. + +```typescript +await EventLog.updateOne( + { + transactionHash: event.transactionHash, + eventType: event.eventType, + }, + { + $setOnInsert: { + transactionHash: event.transactionHash, + eventType: event.eventType, + ledgerSequence: event.ledgerSequence, + contractId: event.contractId, + eventData: event.eventData, + status: 'pending', + processedAt: null, + }, + }, + { upsert: true }, +); +``` + +Using `upsert` with `$setOnInsert` allows the migration to be safely retried without replacing an existing event record. + +### Dedupe verification + +```javascript +db.eventlogs.aggregate([ + { + $group: { + _id: { + transactionHash: "$transactionHash", + eventType: "$eventType" + }, + count: { $sum: 1 } + } + }, + { + $match: { + count: { $gt: 1 } + } + } +]) +``` + +Expected result: + +```text +No duplicate groups +``` + +--- + +## 5. Index Maintenance + +Inspect indexes before rebuilding: + +```javascript +db.deliveries.getIndexes() +db.eventlogs.getIndexes() +db.driverlocations.getIndexes() +``` + +### Sparse indexes + +Example: + +```javascript +db.deliveries.createIndex( + { trackingNumber: 1 }, + { unique: true, sparse: true, name: "trackingNumber_1_sparse" } +) +``` + +Sparse indexes should only be used when documents without the indexed field must be excluded from the index. + +### Partial indexes + +Example pattern: + +```javascript +db.deliveries.createIndex( + { deletedAt: -1 }, + { + partialFilterExpression: { + isDeleted: true + }, + name: "deletedAt_1_deleted_partial" + } +) +``` + +Verify the existing application query semantics before replacing an index with a partial index. + +### TTL indexes + +Example: + +```javascript +db.collection.createIndex( + { createdAt: 1 }, + { + expireAfterSeconds: 86400, + name: "createdAt_1_ttl" + } +) +``` + +TTL indexes must only be applied to collections where automatic expiration is explicitly intended. + +### 2dsphere indexes + +For GeoJSON location data: + +```javascript +db.collection.createIndex( + { location: "2dsphere" }, + { name: "location_2dsphere" } +) +``` + +Validate existing documents before creating the index because malformed GeoJSON can prevent index creation. + +### Safe index replacement + +```javascript +db.collection.getIndexes() +db.collection.dropIndex("old_index_name") +db.collection.createIndex( + { field: 1 }, + { name: "new_index_name" } +) +``` + +For large production collections, schedule index maintenance during an approved maintenance window and monitor database load. + +--- + +## 6. Demo Environment Reseeding + +The repository seed script is: + +```text +src/seed.ts +``` + +Run: + +```bash +pnpm run seed +``` + +The current seed implementation removes existing delivery documents before inserting its demo delivery: + +```typescript +await Delivery.deleteMany({}); +await Delivery.create(delivery); +``` + +Therefore, the current seed process is **repeatable for its demo dataset but destructive to existing delivery records in the target database**. + +It must only be used against disposable/demo environments unless the implementation is explicitly changed to use scoped, idempotent upserts. + +### Recommended safe demo workflow + +```bash +export NODE_ENV=development +export MONGODB_URI="mongodb://localhost:27017/swiftchain_demo" +pnpm run seed +``` + +Verify: + +```javascript +db.deliveries.countDocuments({ + deliveryId: "DEL-001" +}) +``` + +Expected: + +```text +1 +``` + +Do not run the current seed command against production. + +--- + +## 7. Verification Checklist + +After every migration: + +* [ ] Migration process completed without unhandled errors. +* [ ] Expected number of records changed. +* [ ] No legacy delivery statuses remain. +* [ ] EventLog transaction/event uniqueness is preserved. +* [ ] EventLog ledger sequence continuity was checked for the migration range. +* [ ] Indexes exist with the expected definitions. +* [ ] Application queries continue to return expected results. +* [ ] Application health check succeeds. +* [ ] Error logs show no migration-related failures. +* [ ] Relevant API smoke tests pass. +* [ ] Migration results have been recorded. + +--- + +## 8. Rollback + +Rollback depends on the migration type. + +### Data rollback + +For destructive transformations, restore the affected collection/database from the verified backup. + +```bash +mongorestore \ + --uri="$MONGODB_URI" \ + --drop \ + ./backup/ +``` + +Do not use `--drop` against production without confirming the exact target database and restore scope. + +### Status rollback + +If the migration only changed delivery status values, reverse only the affected records using a migration-specific backup or audit dataset. + +Do not infer the original status after the fact when multiple legacy states mapped to the same canonical state. + +### EventLog rollback + +For backfilled EventLog records, identify the exact migration window and event source before deletion. Never delete EventLog records globally to undo a failed backfill. + +### Index rollback + +Recreate the previous index definition before dropping a newly introduced index where possible. + +--- + +## 9. Deployment Integration + +Database migrations should be treated as a deployment dependency: + +```text +Build + ↓ +Backup + ↓ +Deploy compatible application + ↓ +Dry-run migration + ↓ +Apply migration + ↓ +Verify database + ↓ +Health check + ↓ +Enable normal traffic +``` + +For backwards-incompatible migrations, use an expand/contract approach: + +```text +Expand schema + ↓ +Deploy compatible application + ↓ +Backfill + ↓ +Verify parity + ↓ +Switch reads/writes + ↓ +Remove legacy representation +``` diff --git a/backend/docs/environment-matrix.md b/backend/docs/environment-matrix.md new file mode 100644 index 0000000..9534e69 --- /dev/null +++ b/backend/docs/environment-matrix.md @@ -0,0 +1,222 @@ +# Environment Configuration Matrix + +## Purpose + +This document defines how SwiftChain Backend configuration differs between development, test/CI, staging, and production. + +The canonical environment schema is: + +```text +src/config/env.ts +``` + +The example configuration is: + +```text +.env.example +``` + +CI configuration is defined in: + +```text +.github/workflows/ci.yml +``` + +--- + +## 1. Environment Matrix + +| Configuration | Development | Test / CI | Staging | Production | +| ---------------------------- | -------------------------- | -------------------------------------------------------- | ------------------------ | --------------------------- | +| `NODE_ENV` | `development` | `test` | `production` | `production` | +| `MONGODB_URI` | Local MongoDB | CI MongoDB | Managed staging MongoDB | Managed production MongoDB | +| `JWT_SECRET` | Local-only secret | CI secret/test value | Deployment secret | Deployment secret | +| `CORS_ORIGIN` | Local frontend | Test origin | Staging frontend | Production frontend | +| `RATE_LIMIT_MAX_REQUESTS` | Development value | Disabled/bypassed by test configuration where applicable | Staging limit | Production limit | +| `REDIS_URL` | Local Redis | CI/test Redis | Managed Redis | Managed Redis | +| `UPLOAD_STORAGE_DRIVER` | `local` | `local` | `s3` | `s3` | +| `AWS_REGION` | Local/default | Test value | Deployment configuration | Deployment configuration | +| `AWS_ACCESS_KEY_ID` | Provider/local credentials | CI secret if required | Secret manager | Secret manager | +| `AWS_SECRET_ACCESS_KEY` | Provider/local credentials | CI secret if required | Secret manager | Secret manager | +| `FCM_PROJECT_ID` | Optional | Test/blank | Staging Firebase project | Production Firebase project | +| `FCM_CLIENT_EMAIL` | Optional | Test/blank | Secret | Secret | +| `FCM_PRIVATE_KEY` | Optional | Test/blank | Secret | Secret | +| `STELLAR_NETWORK` | `testnet` | `testnet`/test configuration | `testnet` | `mainnet` | +| `SOROBAN_RPC_URL` | Testnet/default | Testnet/default | Staging RPC | Production RPC | +| `STELLAR_NETWORK_PASSPHRASE` | Testnet | Testnet | Staging network | Mainnet | +| `SOROBAN_ESCROW_CONTRACT_ID` | Development contract | Test contract | Staging contract | Production contract | +| `CB_SOROBAN_*` | Fast feedback values | Stable test values | Production-like | Production tuning | +| `INDEXER_LAG_*` | Development values | Test-safe values | Alerting enabled | Alerting enabled | +| `WEBHOOK_*` | Local/test endpoints | CI-safe endpoints | Staging endpoints | Production endpoints | +| `GOOGLE_MAPS_API_KEY` | Optional | Optional | Secret | Secret | +| `OPENWEATHER_API_KEY` | Optional | Optional | Secret | Secret | +| `APP_BASE_URL` | `http://localhost:3000` | CI URL/test value | Staging URL | Production URL | + +Example values are intentionally non-secret. Actual deployment values must be supplied by the environment rather than committed to source control. + +--- + +## 2. CI Environment + +The current workflow is: + +```text +.github/workflows/ci.yml +``` + +It currently uses: + +```yaml +strategy: + matrix: + node-version: [22.x] + mongodb-version: ['6.0'] +``` + +MongoDB is started inside the GitHub Actions runner: + +```yaml +- name: Start MongoDB + uses: supercharge/mongodb-github-action@1.11.0 + with: + mongodb-version: ${{ matrix.mongodb-version }} +``` + +Tests receive CI-specific values: + +```yaml +env: + CI: true + MONGO_URI: mongodb://localhost:27017/swiftchain_test + JWT_SECRET: test-secret-key-16chars +``` + +Do not copy CI credentials or test database URLs into staging or production configuration. + +--- + +## 3. NODE_ENV Behaviour + +`NODE_ENV` is validated by `src/config/env.ts`: + +```text +development +test +production +``` + +Use: + +| Value | Intended use | +| ------------- | ------------------------------------ | +| `development` | Local development and debugging | +| `test` | Automated tests and CI | +| `production` | Staging/production runtime behaviour | + +Configuration-dependent behaviour should be determined from `NODE_ENV` rather than hard-coding environment names throughout the application. + +Typical differences include: + +* error detail/verbosity +* logger formatting +* CORS policy +* background-job startup +* external-service configuration +* security defaults + +--- + +## 4. Secret Management + +The following values must never contain real credentials in Git: + +* `JWT_SECRET` +* `AWS_ACCESS_KEY_ID` +* `AWS_SECRET_ACCESS_KEY` +* `FCM_PRIVATE_KEY` +* `FCM_CLIENT_EMAIL` +* `FCM_PROJECT_ID` where treated as confidential by deployment policy +* `GOOGLE_MAPS_API_KEY` +* `OPENWEATHER_API_KEY` +* private webhook credentials +* database credentials embedded in `MONGODB_URI` +* private Stellar/Soroban credentials + +`.env.example` must contain placeholders or safe development values only. + +Production secrets should be injected by the deployment platform's secret/environment-variable mechanism. + +--- + +## 5. Adding a New Environment Variable + +Every new configuration value must follow this sequence: + +```text +1. Add validation/default to src/config/env.ts + ↓ +2. Add safe example to .env.example + ↓ +3. Add the variable to this matrix + ↓ +4. Determine whether CI requires it + ↓ +5. Add CI injection only when required + ↓ +6. Document production secret injection + ↓ +7. Run lint/build/tests +``` + +### Example + +For a new variable: + +```text +NEW_FEATURE_TIMEOUT_MS +``` + +Update: + +```text +src/config/env.ts +.env.example +backend/docs/environment-matrix.md +.github/workflows/ci.yml +``` + +Only modify CI when the variable is required for tests or builds. + +--- + +## 6. Environment Safety Rules + +* Never commit production secrets. +* Never use production MongoDB credentials locally. +* Never point CI at production infrastructure. +* Never point the demo seed script at production. +* Keep Stellar network and passphrase values consistent. +* Use separate storage credentials per environment. +* Use separate Firebase projects/credentials where environment isolation is required. +* Use environment-specific webhook endpoints. +* Verify `NODE_ENV` before running administrative commands. +* Review environment changes as part of deployment review. + +--- + +## 7. Configuration Validation + +The application validates environment variables through Zod during startup. + +A missing or invalid value should cause startup failure rather than allowing the application to run with an invalid configuration. + +Before deployment: + +```bash +pnpm run lint +pnpm run build +pnpm test +pnpm run test:coverage +``` + +Then verify the deployed service health endpoint and relevant external integrations. diff --git a/backend/docs/releases.md b/backend/docs/releases.md new file mode 100644 index 0000000..5341c2b --- /dev/null +++ b/backend/docs/releases.md @@ -0,0 +1,384 @@ +# Release, Versioning, and Changelog Process + +## Purpose + +This document defines how SwiftChain Backend changes are grouped into releases, versioned, documented, tagged, deployed, and communicated. + +The repository currently declares its package version in: + +```text +package.json +``` + +Current release conventions should be applied consistently with the existing `main` and `develop` workflow. + +--- + +## 1. Semantic Versioning + +SwiftChain Backend follows: + +```text +MAJOR.MINOR.PATCH +``` + +### MAJOR + +Increment MAJOR for backwards-incompatible API or contract changes. + +Examples: + +* Removing a supported endpoint. +* Removing or renaming a required response field. +* Changing an API contract in a way that breaks existing clients. +* Removing a supported model or integration contract. + +Example: + +```text +1.4.2 → 2.0.0 +``` + +### MINOR + +Increment MINOR for backwards-compatible functionality. + +Examples: + +* Adding a new endpoint. +* Adding an optional request parameter. +* Adding a new non-breaking integration. +* Adding a new optional response field. + +Example: + +```text +1.4.2 → 1.5.0 +``` + +### PATCH + +Increment PATCH for backwards-compatible fixes. + +Examples: + +* Bug fixes. +* Security fixes that do not change the public contract. +* Documentation-only changes when the release process includes them. +* Internal performance improvements without contract changes. + +Example: + +```text +1.4.2 → 1.4.3 +``` + +--- + +## 2. Changelog + +Maintain release history in: + +```text +CHANGELOG.md +``` + +Each release should include: + +```markdown +## [1.5.0] - YYYY-MM-DD + +### Added +- New functionality. + +### Changed +- Backwards-compatible changes. + +### Fixed +- Bug fixes. + +### Deprecated +- Features scheduled for removal. + +### Removed +- Features removed in this release. + +### Security +- Security-related changes. +``` + +Only include categories that contain actual changes. + +--- + +## 3. PR Labels and Changelog Entries + +PR labels should communicate the release impact. + +Suggested mapping: + +| PR label | Changelog category | Version impact | +| ------------- | ------------------ | ----------------------------------------- | +| `breaking` | Removed / Changed | MAJOR | +| `feature` | Added | MINOR | +| `enhancement` | Changed | MINOR | +| `bug` / `fix` | Fixed | PATCH | +| `security` | Security | PATCH unless breaking | +| `deprecation` | Deprecated | Usually MINOR | +| `docs` | Documentation | No API version impact by default | +| `refactor` | Changed/internal | No version impact unless contract changes | + +The final release version is determined from the merged changes, not from the label alone. + +--- + +## 4. Release Checklist + +### Before release + +* [ ] Review all merged PRs since the previous release. +* [ ] Confirm semantic-version impact. +* [ ] Update `CHANGELOG.md`. +* [ ] Confirm required database migrations. +* [ ] Confirm rollback procedures. +* [ ] Confirm deprecated features are documented. +* [ ] Confirm environment configuration changes. +* [ ] Run lint. +* [ ] Run build. +* [ ] Run tests. +* [ ] Run coverage. +* [ ] Review the final Git diff. + +### Validation commands + +```bash +pnpm run lint +pnpm run build +pnpm test +pnpm run test:coverage +``` + +### Database + +Before deployment: + +```text +Review migration + ↓ +Backup database + ↓ +Run dry run + ↓ +Apply migration + ↓ +Verify data parity + ↓ +Verify indexes +``` + +### Deployment + +```text +Prepare release + ↓ +Create release branch + ↓ +Run CI + ↓ +Create version/tag + ↓ +Deploy + ↓ +Run health check + ↓ +Monitor logs and metrics +``` + +--- + +## 5. Branch and Tag Conventions + +### Feature branches + +Use the existing repository convention: + +```text +feat/ +``` + +### Bug fixes + +```text +fix/ +``` + +### Documentation + +```text +docs/ +``` + +### Release branches + +Use: + +```text +release/v +``` + +Example: + +```text +release/v1.5.0 +``` + +### Release tags + +Create annotated tags: + +```bash +git tag -a v1.5.0 -m "Release v1.5.0" +git push origin v1.5.0 +``` + +### Hotfixes + +Create hotfix branches from the production release line: + +```text +hotfix/ +``` + +After validation: + +```text +hotfix → main +hotfix → develop +``` + +Ensure the changelog and version are updated consistently in both branches. + +--- + +## 6. Worked Release Example + +Assume the merged changes contain: + +```text +PR A: Add delivery search endpoint +PR B: Fix escrow timeout +PR C: Improve delivery documentation +``` + +Release impact: + +```text +PR A → MINOR +PR B → PATCH +PR C → Documentation +``` + +The release therefore requires a MINOR version increment. + +If the previous release was: + +```text +1.4.2 +``` + +the release becomes: + +```text +1.5.0 +``` + +Update `CHANGELOG.md`: + +```markdown +## [1.5.0] - YYYY-MM-DD + +### Added +- Added delivery search functionality. + +### Fixed +- Corrected escrow timeout handling. + +### Documentation +- Updated delivery documentation. +``` + +Create: + +```bash +git checkout main +git pull origin main + +git checkout -b release/v1.5.0 + +# Update package version and CHANGELOG.md + +pnpm run lint +pnpm run build +pnpm test +pnpm run test:coverage + +git add package.json CHANGELOG.md +git commit -m "chore(release): prepare v1.5.0" + +git tag -a v1.5.0 -m "Release v1.5.0" +git push origin release/v1.5.0 +git push origin v1.5.0 +``` + +After deployment, perform the health check and monitor application logs. + +--- + +## 7. Deprecation Policy + +Deprecated endpoints and models must be announced before removal. + +A deprecation should document: + +* affected endpoint/model +* current version +* replacement endpoint/model +* reason for deprecation +* migration instructions +* planned removal version +* relevant changelog entry + +Example: + +```markdown +### Deprecated + +`GET /api/v1/legacy-deliveries` + +Use: + +`GET /api/v1/deliveries` + +The legacy endpoint remains available for the documented deprecation period and is +scheduled for removal in the next major API version. +``` + +Do not remove a documented public endpoint solely because a replacement exists. + +--- + +## 8. Release Verification + +After deployment: + +* [ ] Deployment completed successfully. +* [ ] Health endpoint responds successfully. +* [ ] Database connectivity is healthy. +* [ ] Redis connectivity is healthy where enabled. +* [ ] Stellar/Soroban connectivity is healthy where enabled. +* [ ] Background jobs are running as expected. +* [ ] No migration errors are present. +* [ ] Error rate is normal. +* [ ] Release tag points to the deployed commit. +* [ ] Changelog matches the release. +* [ ] Deprecated functionality is behaving according to the documented policy.