Skip to content

Create Public OTLP Collector and Victoria Stack for external pushes - #38

Merged
mamoutou-diarra merged 10 commits into
mainfrom
mamoutou/add_external_otlp_collector
Jun 29, 2026
Merged

mamoutou-diarra merged 10 commits into
mainfrom
mamoutou/add_external_otlp_collector

Conversation

@mamoutou-diarra

@mamoutou-diarra mamoutou-diarra commented Jun 26, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

This PR introduces a dedicated ingestion path for OTLP metrics and logs coming from external nodes.
To reduce the risk of metric poisoning and denial-of-service attacks against our main cluster, external traffic is isolated through:

  • a separate public OTLP collector,
  • a separate VictoriaMetrics stack, and a separate VictoriaLogs stack.
  • L7 request-rate control and L3 bandwidth limits on the public resources

Changes

  • Create a dedicated public OTLP collector for external pushes.
  • Add resource label dst_source: external to every received metric/log in the external collector.
  • Expose secure and insecure public gRPC OTLP endpoints:
    • public.otlp.lab.vac.dev:443 (TLS)
    • public.otlp.lab.vac.dev:80 (insecure)
  • Create a dedicated bearer token for the external OTLP endpoint.
  • Set OTLP max_recv_msg_size_mib to 2 (2 MB max receive message size).
  • Expose a public VictoriaLogs VMUI endpoint (no auth): https://public.log.lab.vac.dev
  • Disable scraping and external exposure on the public VictoriaMetrics stack where not required.
  • Apply L3 bandwidth limits (Cilium annotations) and Traefik rate limiting on:
    • public vlselect / VMUI endpoint
    • public OTLP collector endpoint
  • Update the original VictoriaMetrics stack to include the new public datasource(s).
  • Copy the External Blockchain Nodes dashboard and switch its datasource to VictoriaMetrics-Public.
  • Publish a read-only Grafana public dashboard link (no authentication required): https://grafana.lab.vac.dev/public-dashboards/e778dd4396ea43939e09ce111f378511

@mamoutou-diarra mamoutou-diarra added the ift IFT commitments label Jun 26, 2026
@mamoutou-diarra mamoutou-diarra self-assigned this Jun 26, 2026
@mamoutou-diarra mamoutou-diarra linked an issue Jun 26, 2026 that may be closed by this pull request
@mamoutou-diarra
mamoutou-diarra merged commit 2673c13 into main Jun 29, 2026
@github-project-automation github-project-automation Bot moved this to Done in DST Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ift IFT commitments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analyze current stack (recurring)

2 participants