Skip to content

fix: implement SQS partial batch failure handling - #23

Merged
ngoyal16 merged 1 commit into
mainfrom
fix/partial-batch-failure
Sep 29, 2026
Merged

ngoyal16 merged 1 commit into
mainfrom
fix/partial-batch-failure

Conversation

@ngoyal16

Copy link
Copy Markdown
Contributor

Fixes #3

Summary of Changes

  • Partial Batch Failure Reporting: Modified the Lambda handler signature to return events.SQSEventResponse containing BatchItemFailures.
  • Per-Record Processing: Extracted message processing logic into processMessage and processSQSEvent. If an individual record fails, its MessageId is appended to BatchItemFailures while allowing other records in the SQS batch to proceed without reprocessing previously written records.
  • CloudFormation Configuration: Updated the EventSourceMapping in deploy/cloudformation/lambda.yaml with FunctionResponseTypes: ['ReportBatchItemFailures'] so AWS Lambda properly respects partial failure responses.
  • Mockable Client Interfaces: Defined S3Client and KMSClient interfaces to decouple AWS SDK calls and enable clean unit test mocking.
  • Parquet Tag Fix: Fixed conflicting dict,plain encoding tags on DatabaseActivityEvent fields which were causing a runtime panic in parquet-go.
  • Unit Test Coverage: Added comprehensive unit tests in main_test.go covering:
    • 2 records where first succeeds and second fails -> only second message in BatchItemFailures
    • All records succeed -> empty BatchItemFailures
    • All records fail -> all message IDs in BatchItemFailures
    • Empty batch handling
    • S3 and KMS error paths
    • End-to-end payload processing with encryption and decompression

- Return events.SQSEventResponse with BatchItemFailures to avoid reprocessing succeeded records
- Extract processMessage and processSQSEvent for granular per-message failure tracking
- Configure FunctionResponseTypes: ['ReportBatchItemFailures'] in CloudFormation EventSourceMapping
- Introduce S3Client and KMSClient interfaces for mockability
- Fix conflicting dict,plain parquet tags on DatabaseActivityEvent
- Add unit tests covering partial batch failure, all success, all fail, and error cases
@ngoyal16
ngoyal16 merged commit 029989b into main Sep 29, 2026
9 checks passed
@ngoyal16
ngoyal16 deleted the fix/partial-batch-failure branch September 29, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partial-batch failure: a single bad SQS record causes loss of all subsequent records

1 participant