Skip to content

Support configurable filter path (LAMBDA_TASK_ROOT) and cache loaded filter in memory #20

Description

@ngoyal16

Problem

In filter.go and main.go, filter configuration loading has several issues:

  1. Relative path failure in Lambda:
    LoadFilterConfig hardcodes relative path filepath.Join("filters", ...). In AWS Lambda custom runtimes, the working directory may not be the task root or where the binary is executed. In Dockerfile.lambda, filters are copied to ${LAMBDA_TASK_ROOT}/filters/. If the process is invoked from another working directory, os.Stat fails to find the filter files.
  2. Disk I/O on every SQS invocation:
    filterConfig, err := LoadFilterConfig(filterName) is called on every Lambda invocation in the handler loop instead of during initialization or cached in memory.
  3. Silent fallback hides configuration errors:
    If a non-existent DAS_FILTER_NAME is specified and no default files exist, LoadFilterConfig silently returns an empty FilterConfig (Drop: [], Query: {}) with err == nil. The operator has no visibility that their custom filter was never loaded.

Proposed Solution

  1. Support path resolution: check DAS_FILTER_DIR environment variable first, then filepath.Join(os.Getenv("LAMBDA_TASK_ROOT"), "filters"), then relative filters/.
  2. Cache the loaded *FilterConfig in memory so warm Lambda invocations don't perform disk I/O on each message.
  3. Return an explicit error or structured log warning if a specified filter name cannot be found and fallback to default occurs.

Acceptance Criteria

  • Filter loader correctly locates filters under ${LAMBDA_TASK_ROOT}/filters
  • Warm invocations reuse cached filter configuration
  • Missing filter produces an explicit warning or error instead of silent no-op
  • Unit tests verify loading order and fallback behavior

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    configurationConfiguration management and env varsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions