Problem
In main.go, the destination S3 bucket is hardcoded to the source S3 bucket:
bucket := record.S3.Bucket.Name
key := record.S3.Object.Key
...
destKey := fmt.Sprintf("das/%s/%s-processed.parquet", filterName, key)
slog.Info("Writing processed data back to S3", "DestBucket", bucket, "DestKey", destKey)
_, err = s3Client.PutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(bucket), // <-- Source bucket!
Key: aws.String(destKey),
Body: bytes.NewReader(parquetBytes),
})
However, in deploy/cloudformation/lambda.yaml, a dedicated DestinationBucket is provisioned and passed via environment variable:
Environment:
Variables:
DAS_OUTPUT_BUCKET: !Ref DestinationBucket
And the Lambda execution IAM role only grants s3:PutObject permission to the destination bucket (!Sub '${DestinationBucket.Arn}/*'), NOT the source bucket (DasSourceBucketArn).
Consequences
- AccessDenied in production: Lambda will crash with
AccessDenied when attempting to write Parquet output to the source bucket.
- Infinite loop risk: If the source bucket has event notifications for
s3:ObjectCreated:* without strict prefix filtering, writing processed files back into the source bucket will fire a new S3 event notification to SQS, creating an infinite recursive processing loop and incurring massive AWS costs.
DAS_OUTPUT_BUCKET is completely ignored in code.
Proposed Solution
- Read
DAS_OUTPUT_BUCKET from environment variables (fallback to source bucket only if not set, or make it required).
- Write processed Parquet files to
outputBucket.
- Support optional
DAS_OUTPUT_PREFIX (defaulting to das/).
Acceptance Criteria
Problem
In main.go, the destination S3 bucket is hardcoded to the source S3 bucket:
However, in deploy/cloudformation/lambda.yaml, a dedicated
DestinationBucketis provisioned and passed via environment variable:And the Lambda execution IAM role only grants
s3:PutObjectpermission to the destination bucket (!Sub '${DestinationBucket.Arn}/*'), NOT the source bucket (DasSourceBucketArn).Consequences
AccessDeniedwhen attempting to write Parquet output to the source bucket.s3:ObjectCreated:*without strict prefix filtering, writing processed files back into the source bucket will fire a new S3 event notification to SQS, creating an infinite recursive processing loop and incurring massive AWS costs.DAS_OUTPUT_BUCKETis completely ignored in code.Proposed Solution
DAS_OUTPUT_BUCKETfrom environment variables (fallback to source bucket only if not set, or make it required).outputBucket.DAS_OUTPUT_PREFIX(defaulting todas/).Acceptance Criteria
DAS_OUTPUT_BUCKETDAS_OUTPUT_BUCKETis not setDAS_OUTPUT_BUCKETis defined