feat(dlr): replace MVStore with PostgreSQL persistence

[lykakis]

Summary

This PR replaces Sendium's MVStore-based DLR persistence with a PostgreSQL-backed lifecycle for provider correlation and downstream HTTP/SMPP receipt delivery. Startup and HTTP ingress remain fail-closed; downstream SMPP uses early UUID acceptance with internal persistence retries.

It adds:

• One durable lifecycle row per gateway message.
• Provider-scoped correlation using the exact (provider_name, provider_message_id) pair.
• Final-only provider receipt resolution.
• Durable HTTP callback scheduling and retries.
• Acknowledgement-driven downstream SMPP delivery and bind-driven replay.
• Attempt-number fencing so stale callbacks cannot mutate a newer delivery attempt.
• PostgreSQL readiness, storage-operation metrics, retention cleanup, Flyway migration, and Quick Start integration.

It removes MVStore, H2, backend selectors, local-file configuration, and volatile persistence fallback. Sendium-owned DLR persistence remains disabled by default in reusable sendium-core and is enabled explicitly by sendium-app.

Downstream delivery metrics and a new structured DLR event schema are intentionally deferred to a separate observability design and PR. Existing PostgreSQL storage metrics and message.deliver.* events remain.

Why

Sendium must retain DLR state long enough to correlate provider receipts with the original gateway message and complete downstream HTTP or SMPP delivery after process restarts and transient failures.

The previous local persistence model tied state to one application filesystem and could fall back to volatile storage. It also removed resolved state before downstream delivery was acknowledged, leaving a crash window where a provider receipt could be accepted and then lost before reaching the originating client.

PostgreSQL provides an externally managed durability boundary with transactional transitions, versioned migrations, standard backup and monitoring options, and consistent behavior across application restarts.

Architecture

Component Responsibilities

Component	Responsibility
DlrMessageStorage / DlrStorage	Persistence boundary for lifecycle operations.
DlrService	Application-facing API and terminal-DLR classification.
ManagedDlrStorage	Runtime activation, datasource validation, and storage metrics.
PostgresqlDlrStorage	Transactions, SQL locking, provider correlation, retries, and retention.
MessageState	Persisted ingress, provider outcome, and downstream-delivery state.
StandardMessageTracker	Connects upstream SMPP responses and receipts to DLR storage.
ForwardDlrService	Polls and delivers pending HTTP callbacks.
StandardSmppServerMessageStore	Persists SMPP ingress and coordinates pending SMPP DLR replay.
DlrDeliveryBatch	Tracks all deliver_sm parts belonging to one delivery attempt.
DlrStorageReadinessCheck	Reports whether the PostgreSQL DLR schema is available.

Persistence is a build-time application boundary controlled by sendium.dlr.persistence.enabled. It is enabled in standalone sendium-app; applications embedding sendium-core must opt in and provide the named datasource and Flyway configuration.

Persisted Model

PostgreSQL contains two main tables:

• sendium_dlr.dlr_message stores one lifecycle row per Sendium gateway message ID. It contains ingress metadata, downstream target, provider outcome, delivery status, retry schedule, timestamps, and fenced attempt number.
• sendium_dlr.provider_correlation maps (provider_name, provider_message_id) to the gateway message ID. Provider scoping allows different providers to reuse the same message ID independently.

Provider outcome and downstream delivery are separate concerns. A provider can have completed the message while Sendium still has a pending HTTP callback or SMPP deliver_sm to deliver.

The first terminal receipt stores the exact provider outcome and consumes every correlation for the gateway message in one transaction. Concurrent terminal receipts can therefore resolve the lifecycle only once.

End-to-End Flows

HTTP Submission to HTTP Callback

sequenceDiagram
    participant Client as HTTP client
    participant Kannel as KannelResource
    participant DlrService
    participant DB as PostgreSQL
    participant Router
    participant Provider as Upstream SMPP provider
    participant Tracker as StandardMessageTracker
    participant Delivery as ForwardDlrService
    participant Callback as HTTP callback endpoint

    Client->>Kannel: GET /sendsms
    Kannel->>Kannel: validate, assign UUID, create MessageState
    Kannel->>DlrService: save initial state
    DlrService->>DB: insert WAITING_PROVIDER row
    alt Initial persistence fails
        DB-->>DlrService: unavailable
        DlrService-->>Kannel: persistence failure
        Kannel-->>Client: HTTP 503, not routed
    else Initial state committed
        DB-->>DlrService: persisted
        DlrService-->>Kannel: persistence succeeded
        Kannel->>Router: enqueue message
        alt Router admission is interrupted
            Router-->>Kannel: admission failure
            Kannel-->>Client: HTTP 503
        else Router admission succeeds
            Router-->>Kannel: queued
            Kannel-->>Client: HTTP 202 + gateway UUID
            Router->>Provider: submit_sm
            Provider-->>Tracker: submit_sm_resp + provider message ID
            Tracker->>DlrService: link provider correlation
            DlrService->>DB: persist provider-scoped correlation
            Provider->>Tracker: terminal provider DLR
            Tracker->>DlrService: resolve terminal outcome
            DlrService->>DB: move HTTP delivery to PENDING
            DB-->>DlrService: resolution committed
            Tracker-->>Provider: deliver_sm_resp STATUS_OK
            loop Up to 10 due delivery attempts
                Delivery->>DlrService: poll and start fenced attempt
                DlrService->>DB: claim delivery attempt
                Delivery->>Callback: GET resolved dlr-url
                alt HTTP 200-399
                    Callback-->>Delivery: success
                    Delivery->>DlrService: complete attempt
                    DlrService->>DB: delete lifecycle row
                else Timeout or non-success response
                    Callback-->>Delivery: failure
                    Delivery->>DlrService: retry in 120s or mark FAILED
                    DlrService->>DB: update lifecycle row
                end
            end
        end
    end

Loading

1. KannelResource receives /sendsms, validates the request, and creates the message and initial MessageState.
2. The state is persisted before router admission. A nonblank dlr-url selects the HTTP delivery channel; otherwise the channel is NONE.
3. A persistence failure returns retryable HTTP 503 and the message is not queued.
4. After successful persistence and router admission, Sendium returns HTTP 202 with the gateway message ID.
5. The upstream submit_sm_resp links the gateway message to the provider-scoped message ID.
6. Intermediate provider receipts are acknowledged without consuming correlation.
7. The first terminal provider receipt atomically stores the outcome and consumes all correlations.
8. Channel NONE deletes the lifecycle row. Channel HTTP moves it to PENDING.
9. ForwardDlrService polls up to 100 due rows every second and performs callbacks serially.
10. HTTP 200 through 399 completes delivery and deletes the row. Redirects are not followed.
11. Attempts 1 through 9 retry after 120 seconds. Attempt 10 becomes FAILED and remains until retention cleanup.

SMPP Ingress Architecture

sequenceDiagram
    participant Client as Downstream SMPP client
    participant Server as SMPP server worker
    participant DlrService as DLR service
    participant DB as PostgreSQL
    participant Router
    participant Upstream as Upstream SMPP worker
    participant Provider as Upstream SMPP provider

    Client->>Server: submit_sm
    Server->>Server: fence ingress, validate, and assign gateway UUID
    alt Rejected before acceptance
        Server-->>Client: error submit_sm_resp
    else Accepted
        Server-->>Client: submit_sm_resp STATUS_OK + UUID
        loop Until initial state is persisted
            Server->>DlrService: save accepted event
            DlrService->>DB: save initial state
            alt PostgreSQL unavailable
                DB-->>DlrService: persistence failure
                DlrService-->>Server: schedule internal retry
                Note over Client,Server: No second submit_sm_resp
            else Commit succeeds
                DB-->>DlrService: persisted
                DlrService-->>Server: continue accepted event
            end
        end
        Server->>Server: assemble multipart parts if needed
        Server->>Router: admit persisted message
        Router->>Upstream: route message
        Upstream->>Provider: submit_sm
        Provider-->>Upstream: submit_sm_resp + provider message ID
        Upstream->>DlrService: link provider correlation
        DlrService->>DB: persist provider-scoped correlation

        Provider->>Upstream: deliver_sm with terminal provider DLR
        Upstream->>DlrService: resolve terminal outcome
        DlrService->>DB: consume correlation and set SMPP delivery PENDING
        alt Terminal resolution fails
            DB-->>DlrService: persistence failure
            DlrService-->>Upstream: resolution failed
            Upstream-->>Provider: deliver_sm_resp STATUS_SYSERR
        else Terminal resolution committed
            DB-->>DlrService: resolved SMPP delivery
            DlrService-->>Upstream: durable outcome
            Upstream-->>Provider: deliver_sm_resp STATUS_OK
            Upstream->>Router: enqueue internal MSG_DLR
            Router->>Server: route to matching SMPP server worker
            alt Original system_id is bound
                Server->>DlrService: start fenced delivery attempt
                DlrService->>DB: mark active attempt
                loop One deliver_sm per original submission ID
                    Server->>Client: deliver_sm
                    Client-->>Server: deliver_sm_resp
                end
                alt Every deliver_sm_resp is STATUS_OK
                    Server->>DlrService: complete fenced attempt
                    DlrService->>DB: delete lifecycle row
                else Timeout, disconnect, NACK, or non-OK response
                    Server->>DlrService: return attempt to PENDING
                    DlrService->>DB: persist retryable state
                    Note over Client,Server: Replay on next bind for the same system_id
                end
            else No matching bound session
                Note over Client,DB: Delivery remains PENDING and replays oldest-first on bind
            end
        end
    end

Loading

Downstream SMPP Submission to SMPP DLR

1. A downstream SMPP client sends submit_sm; registered_delivery determines whether a receipt is requested.
2. InTask gathers up to 100 ingress events for up to 100 ms.
3. After validation, Sendium assigns the gateway UUID and queues a successful submit_sm_resp before persistence.
4. Persistence or router admission failures after acceptance produce no second response; Sendium retries them internally and does not route before persistence succeeds.
5. Graceful shutdown fences new submissions, drains queued responses and persistence, flushes multipart assembly, and closes sessions afterward.
6. The upstream submit_sm_resp links the provider message ID, and the first terminal provider DLR resolves the lifecycle.
7. When forwarding is requested, StandardMessageTracker creates an internal MSG_DLR and routes it to the matching SMPP server worker.
8. The server creates one deliver_sm, or one per original multipart submission ID.
9. DlrDeliveryBatch treats all generated PDUs as one fenced delivery attempt.
10. The lifecycle row is deleted only after every matching deliver_sm_resp succeeds.
11. Timeout, disconnect, generic_nack, wrong response type, non-OK response, or send failure releases the entire attempt back to PENDING.
12. Pending SMPP deliveries replay oldest-first when the same system_id binds again.

Provider Receipt Processing

1. ACCEPTD and ENROUTE receipts are acknowledged but do not consume correlation.
2. DELIVRD and SEEN normalize to DELIVERED; other terminal outcomes normalize to FAILED while retaining the exact DLR state and error code.
3. Unknown or expired correlations are logged, acknowledged, and do not create downstream delivery.
4. A terminal persistence failure returns SMPP STATUS_SYSERR, allowing the provider to retry the receipt.
5. Successful provider acknowledgement confirms durable resolution only; it does not wait for downstream HTTP or SMPP delivery.

Provider IDs are protected by transaction-scoped advisory locks and canonical gateway-row lock ordering. Reusing an ID within one provider transfers ownership to the newest gateway message; different providers remain isolated. msg.hash.prefix can make multiple workers share one provider namespace and must remain stable while correlations are outstanding.

Acknowledgement and Failure Contract

Failure point	External response	Persisted result and retry behavior
Initial HTTP persistence	HTTP 503	Not routed; caller can retry.
Initial SMPP persistence after acceptance	No second submit_sm_resp	Accepted event remains in memory and retries internally; routed only after persistence.
Router admission after persistence	HTTP 503; no second SMPP response	The HTTP caller can retry; accepted SMPP work retries internally.
Provider correlation link after upstream response	No upstream response remains available to reject	Failure is logged; the initial row remains but later provider DLRs may be unresolvable.
Terminal provider resolution	deliver_sm_resp with STATUS_SYSERR	Provider can retry the receipt.
HTTP callback failure	No success completion	Persisted retry schedule survives restart.
SMPP delivery failure or disconnect	No success completion	Attempt returns to PENDING and replays on a later matching bind.
Receiver accepts a callback/receipt but completion persistence fails	Receiver saw success	Delivery can be repeated because semantics are at-least-once.

For downstream SMPP submission, STATUS_OK means the validated message was accepted into Sendium's in-memory ingress pipeline. Persistence and routing happen asynchronously afterward; the message is never routed before its initial state is persisted.

Important Boundaries

• Delivery is at-least-once, not exactly-once. Consumers must be idempotent using the gateway or receipted message ID.
• Multipart SMPP replay can repeat parts acknowledged before another part failed.
• SMPP replay is bind-driven rather than periodic. A failed attempt remains pending until the matching client binds again or retention removes it.
• HTTP delivery uses non-overlapping serial batches. A slow batch delays later due callbacks.
• PostgreSQL makes persisted DLR lifecycle state durable; it does not make pre-persistence SMPP ingress, router queues, worker queues, or multipart submission assembly durable.
• A process crash can lose an SMPP submission acknowledged before persistence. A sustained PostgreSQL outage retains accepted submissions in an unbounded in-memory backlog and requires prompt operational intervention.
• Active-attempt exclusion is process-local. Multiple active Sendium replicas sharing one database can start duplicate deliveries.
• Provider namespaces must remain stable while correlations are outstanding.
• Correlations expire after three days. Waiting and resolved lifecycle rows expire according to the documented seven-day retention policy.
• The V1 Flyway migration is edited directly because PostgreSQL DLR persistence has not shipped or been applied to a production installation.
• Existing MVStore files are not imported.

See docs/13-dlr-persistence.md for the operational durability contract, retention, migration behavior, and remaining crash windows.

Deployment and Migration

• Uses a versioned Flyway migration and named Agroal dlr datasource.
• Adds PostgreSQL to Quick Start with a private service, persistent volume, generated credentials, health-gated startup, and password preservation across regeneration.
• Supports externally managed PostgreSQL through explicit JDBC configuration.
• Fails startup when persistence is enabled without an active datasource and Flyway migration.
• Exposes the sendium-dlr-storage readiness check and storage-operation metrics.
• Does not import existing MVStore data.

Verification

• mvnw -pl sendium-core verify
• 259 unit tests passed.
• 35 integration tests passed, including PostgreSQL migration, storage, outage, recovery, retention, provider isolation, and concurrency coverage.
• 18 focused HTTP dispatcher and SMPP output-task tests passed.
• All 16 Quick Start tests passed, including local/external PostgreSQL password preservation and Docker Compose parsing.
• Packaged JVM PostgreSQL outage and restart recovery passed during the branch acceptance gates.
• Container-built native PostgreSQL restart recovery passed during the branch acceptance gates.
• External-consumer startup with Sendium-owned persistence disabled passed.
• Checkstyle and git diff --check pass.

Review Guide

Review the final behavior by concern rather than treating this as only a storage-adapter replacement. Compatibility wiring introduced in earlier stages is removed by finalization stages; the final tree is PostgreSQL-only.

Pass	Focus	Current commit range
1	Schema, storage boundary, PostgreSQL message state, correlation, and receipt storage	d0ab195 through b58faf2
2	Runtime wiring, storage metrics, health, and fail-closed HTTP/SMPP submission behavior	0126df5 through e433984
3	Quick Start deployment, JVM/native restart testing, and operational documentation	f39679b through efa5086
4	PostgreSQL default, MVStore removal, and reusable-core persistence boundary	c442ae2 through 19de0c8
5	Provider-scoped correlation hardening, adapter naming, and architecture flow	d13655e through 8f91844
6	Unified durable lifecycle, HTTP retries, SMPP acknowledgement, and final documentation	144ff19 through 04a18b1

Suggested review order:

1. Architecture, persisted model, state transitions, and acknowledgement contract.
2. PostgreSQL schema, locking, correlation consumption, retention, and direct storage tests.
3. HTTP persistence-before-acceptance, SMPP early acknowledgement/retry behavior, and provider receipt handling.
4. Durable HTTP delivery and attempt fencing.
5. Durable SMPP multipart acknowledgement and bind-driven replay.
6. Runtime activation, Quick Start, CI, MVStore removal, and reusable-core compatibility.