Skip to content

Logger calls eagerly construct large payload objects on publish/failure #27

Description

@iifawzi

Summary

Multiple log sites pass the entire message body as part of the structured log fields. Even when the configured log level discards the line, the payload reference is captured and most structured-log transports will JSON.stringify it. For high-throughput publishers / heavy retry traffic, this is real GC pressure.

Where

  • src/core/RunMQ.ts:76-80publish() logs { topic, correlationId, message } at info level
  • src/core/consumer/processors/RunMQFailureLoggerProcessor.ts:13-16 — failure logs include message: message.message
  • src/core/consumer/processors/RunMQRetriesCheckerProcessor.ts:38-45 — max-retries log also includes message: message.message

Failure mode

For an MB-sized payload at 1k msg/s, this is GB/s of allocations the GC has to clean up — independent of whether the line is actually written.

Proposed fix

  • Drop message from the publish info log (correlationId is enough for tracing).
  • For failure logs, log a truncated preview or just the size:
    this.logger.error('Message processing failed', {
        correlationId: message.correlationId,
        size: typeof message.message === 'string' ? message.message.length : undefined,
    }, e instanceof Error ? e.stack : undefined);
  • Optional: lazy-evaluate via a () => ({ ... }) thunk if the logger supports it.

Acceptance criteria

  • No log site captures the full message payload by default.
  • Optional opt-in (e.g., logFullMessage: true in processor config) for debugging.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance / hot-path issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions