Skip to content

Write processed Parquet to DAS_OUTPUT_BUCKET instead of source S3 bucket #13

Description

@ngoyal16

Problem

In main.go, the destination S3 bucket is hardcoded to the source S3 bucket:

bucket := record.S3.Bucket.Name
key := record.S3.Object.Key
...
destKey := fmt.Sprintf("das/%s/%s-processed.parquet", filterName, key)
slog.Info("Writing processed data back to S3", "DestBucket", bucket, "DestKey", destKey)

_, err = s3Client.PutObject(ctx, &s3.PutObjectInput{
    Bucket: aws.String(bucket), // <-- Source bucket!
    Key:    aws.String(destKey),
    Body:   bytes.NewReader(parquetBytes),
})

However, in deploy/cloudformation/lambda.yaml, a dedicated DestinationBucket is provisioned and passed via environment variable:

Environment:
  Variables:
    DAS_OUTPUT_BUCKET: !Ref DestinationBucket

And the Lambda execution IAM role only grants s3:PutObject permission to the destination bucket (!Sub '${DestinationBucket.Arn}/*'), NOT the source bucket (DasSourceBucketArn).

Consequences

  1. AccessDenied in production: Lambda will crash with AccessDenied when attempting to write Parquet output to the source bucket.
  2. Infinite loop risk: If the source bucket has event notifications for s3:ObjectCreated:* without strict prefix filtering, writing processed files back into the source bucket will fire a new S3 event notification to SQS, creating an infinite recursive processing loop and incurring massive AWS costs.
  3. DAS_OUTPUT_BUCKET is completely ignored in code.

Proposed Solution

  1. Read DAS_OUTPUT_BUCKET from environment variables (fallback to source bucket only if not set, or make it required).
  2. Write processed Parquet files to outputBucket.
  3. Support optional DAS_OUTPUT_PREFIX (defaulting to das/).

Acceptance Criteria

  • Lambda writes processed Parquet files to DAS_OUTPUT_BUCKET
  • Falls back gracefully or errors clearly if DAS_OUTPUT_BUCKET is not set
  • Prevents writing back to the source bucket when DAS_OUTPUT_BUCKET is defined
  • Unit test verifies destination bucket configuration

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsecuritySecurity hardening and vulnerabilities

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions