Skip to content

Support EventBridge S3 event notification format and gracefully handle s3:TestEvent #19

Description

@ngoyal16

Problem

In main.go, the event parsing logic only handles two event formats:

  1. SNS-wrapped legacy S3 Event Notifications
  2. Direct legacy S3 Event Notifications (Records[].s3.bucket.name)

Limitations

  1. Amazon EventBridge format unsupported:
    AWS recommends Amazon EventBridge for S3 event notifications (EventBridge S3 notifications). EventBridge uses a different JSON structure:

    {
      "version": "0",
      "id": "...",
      "detail-type": "Object Created",
      "source": "aws.s3",
      "detail": {
        "bucket": { "name": "my-bucket" },
        "object": { "key": "das/log.json.gz" }
      }
    }

    When SQS receives EventBridge notifications, fails to parse them, logging an error and skipping the event.

  2. s3:TestEvent causes false alarm error logs:
    When configuring S3 event notifications in the AWS Console, AWS sends an s3:TestEvent payload ("Event": "s3:TestEvent"). In the current code, this fails parsing and logs an error with the full message body.

Proposed Solution

  1. Detect and parse EventBridge aws.s3 event format if legacy S3 records are not present.
  2. Gracefully identify s3:TestEvent and log an informational message instead of an error:
    slog.Info("Received S3 test event notification, skipping")
  3. Extract common bucket and key extraction into a helper function extractS3Events(body string) ([]S3Record, error).

Acceptance Criteria

  • Direct S3 Event Notifications supported
  • SNS-wrapped S3 Event Notifications supported
  • EventBridge S3 notifications supported
  • s3:TestEvent handled gracefully without error logs
  • Unit tests for all 4 notification types

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions