Problem
The Lambda function has no observability beyond slog.Info/slog.Error log lines:
-
No metrics — There is no way to track:
- Number of records processed per invocation
- Number of records filtered/dropped
- Time spent in each pipeline stage (S3 fetch, KMS decrypt, decompress, Parquet convert, S3 write)
- Error rates by stage
-
No structured error aggregation — When a batch has multiple failures, each error is logged individually but there is no summary of batch-level outcomes.
-
No Dead-Letter Queue (DLQ) handling — If SQS retries are exhausted, messages go to the DLQ silently with no alerting or reprocessing path.
-
No tracing — No AWS X-Ray or OpenTelemetry integration to trace the full pipeline across S3 → KMS → Parquet → S3.
Proposed Solution
Metrics (using embedded metric format via aws-lambda-go or CloudWatch EMF)
Add structured metrics for:
RecordsProcessed (count)
RecordsFiltered (count)
RecordsDropped (count)
ProcessingDuration (per stage, in milliseconds)
Errors (count, by stage)
Tracing
- Add
aws-lambda-go X-Ray tracing wrapper
- Trace segments for: S3 GetObject, KMS Decrypt, ESDK Decrypt, Decompress, Parquet Convert, S3 PutObject
DLQ
- Document DLQ configuration in README
- Add a reprocessing handler or document manual reprocessing steps
Acceptance Criteria
Problem
The Lambda function has no observability beyond
slog.Info/slog.Errorlog lines:No metrics — There is no way to track:
No structured error aggregation — When a batch has multiple failures, each error is logged individually but there is no summary of batch-level outcomes.
No Dead-Letter Queue (DLQ) handling — If SQS retries are exhausted, messages go to the DLQ silently with no alerting or reprocessing path.
No tracing — No AWS X-Ray or OpenTelemetry integration to trace the full pipeline across S3 → KMS → Parquet → S3.
Proposed Solution
Metrics (using embedded metric format via
aws-lambda-goor CloudWatch EMF)Add structured metrics for:
RecordsProcessed(count)RecordsFiltered(count)RecordsDropped(count)ProcessingDuration(per stage, in milliseconds)Errors(count, by stage)Tracing
aws-lambda-goX-Ray tracing wrapperDLQ
Acceptance Criteria