Skip to content

Improve README: architecture diagram, filter DSL docs, Parquet schema, deployment guide #11

Description

@ngoyal16

Problem

The README.md is minimal and missing several important sections for a production Lambda project:

Missing:

  1. Architecture diagram — No visual representation of the data pipeline (SQS → Lambda → S3 → KMS → Parquet → S3)
  2. Deployment instructions — No Terraform/SAM/CDK examples or manual deployment steps
  3. Filter configuration examples — The filters/ directory and filter query DSL (the not, selector, and, or, contains, dimensionExists node types in filter.go) are completely undocumented
  4. Parquet schema documentation — The DatabaseActivityEvent struct in main.go defines 20+ fields but there is no schema reference
  5. Input/Output format — No documentation of the expected S3 event format and the output Parquet structure
  6. Troubleshooting — No common error scenarios and resolutions
  7. Environment variable examples — No .env.example file

Proposed Solution

  1. Add a Mermaid architecture diagram to README
  2. Add a "Filter Configuration" section documenting the query DSL with examples
  3. Add a "Deployment" section with at least manual steps
  4. Add a "Parquet Schema" table documenting all fields in DatabaseActivityEvent
  5. Create a .env.example file with all supported env vars
  6. Add a "Troubleshooting" section

Acceptance Criteria

  • README contains an architecture diagram (Mermaid)
  • Filter query DSL is documented with at least one example per node type
  • All 20+ Parquet fields are documented in a schema table
  • .env.example file exists with all env vars
  • Deployment instructions are present

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions