Skip to content

Add observability: metrics, tracing, and DLQ support #6

Description

@ngoyal16

Problem

The Lambda function has no observability beyond slog.Info/slog.Error log lines:

  1. No metrics — There is no way to track:

    • Number of records processed per invocation
    • Number of records filtered/dropped
    • Time spent in each pipeline stage (S3 fetch, KMS decrypt, decompress, Parquet convert, S3 write)
    • Error rates by stage
  2. No structured error aggregation — When a batch has multiple failures, each error is logged individually but there is no summary of batch-level outcomes.

  3. No Dead-Letter Queue (DLQ) handling — If SQS retries are exhausted, messages go to the DLQ silently with no alerting or reprocessing path.

  4. No tracing — No AWS X-Ray or OpenTelemetry integration to trace the full pipeline across S3 → KMS → Parquet → S3.

Proposed Solution

Metrics (using embedded metric format via aws-lambda-go or CloudWatch EMF)

Add structured metrics for:

  • RecordsProcessed (count)
  • RecordsFiltered (count)
  • RecordsDropped (count)
  • ProcessingDuration (per stage, in milliseconds)
  • Errors (count, by stage)

Tracing

  • Add aws-lambda-go X-Ray tracing wrapper
  • Trace segments for: S3 GetObject, KMS Decrypt, ESDK Decrypt, Decompress, Parquet Convert, S3 PutObject

DLQ

  • Document DLQ configuration in README
  • Add a reprocessing handler or document manual reprocessing steps

Acceptance Criteria

  • CloudWatch metrics are emitted for record counts and processing durations
  • X-Ray tracing wraps the handler with sub-segments per pipeline stage
  • DLQ setup is documented
  • Metrics are testable (not hardcoded to CloudWatch SDK)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestobservabilityLogging, metrics, tracing, and alerting

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions