🎯 The objective

We should have monitoring, alerting and dashboards setup around our infrastructure so we can check the live and historical health of our systems, and so we can be alerted when things go bad. We should also aggregate logs to help us understand these failures.

Currently, we have some CloudWatch metrics for our AWS resources, but I don’t think we have any monitoring or alerting from non-AWS resources like Neon and B2. We need to know the health of these critical systems.

🚗 How we’re getting there

Based on the preliminary discussion, let's focus on OpenTelemetry as our foundation (Stalwart, AWS and Neon support emitting them), and let's just go with betterstack so we can leverage croth's team's experience if we need to, and we'll have more in-house knowledge on the same tech. It seems like the quickest solution to get started.

We need a timeseries database for the telemetry. could use prometheus or whatever we're most comfortable with.

So: Let's try BetterStack with:

• Availability:Successful requests: ex: At ALB, track % of successful requestsError rate: (this is to account for any internal retries within our stack), % of requests that returned 500 or timed out/exhausted retriesConnection checks: ex: can stalwart talk to keycloak?Heart beats: ex: is a stalwart node still responsive?
• Latency: ex: latency from ECS to Neon db
• resource load: cpu/ram for spikes (useful usually after deployments and helps diagnose memory leaks)

Early bird: Focus on Availability and Errors. Latency can follow, cache hit/miss rate, etc.

• Hook up all sources to BetterStack using services-dev account (damn, they have a lot of docs: https://betterstack.com/docs/logs/aws-fargate/ for AWS Fargate (have another for ECS, which we should use for CPU/Mem), https://betterstack.com/docs/logs/aws-cloudwatch/ for CW, https://betterstack.com/docs/logs/cloudflare/logpush/ for CloudFlare). Neon: https://neon.com/docs/guides/opentelemetry?utm_source=chatgpt.com
• Note: Autoscaling in Fargate reacts only to Cloudwatch metrics, so we’ll need to find a solution to bridge this gap

We’ll determine our SLO/SLIs in the meantime.

Aggregating logs will follow.

🗒️ What are we measuring?

🎒 Resources

ℹ️ List all relevant resources like BugZilla meta bugs, Figma files, Notion links and other files. These can be filled in later.

• Lisa Wess’ proposal on Incident Management
• User Metrics for Early Bird will have app metrics which may be tied into this effort as well

💲Pricing estimate

Free tier includes: 10 monitors, 10 heartbeats, 1 public Status page, 3GB logs (3 day retention), 2B metrics (30 day retention)

“Responder” package:

Incident Management:

Telemetry:

Monitoring:

🦮 Rules of the Road

ℹ️ Initial definition of requirements and limitations, the scope of this feature and other potential considerations in order to keep the project narrow and focused.

🟢 In Scope

• Determine how we are going to monitor our services, eg: CloudWatch? https://signoz.io/ (RyanJ recommends)? BetterStack (croth recommends) and set it up.
  • Sept 8, 2025: We’re going to do our first attempt with BetterStack
• Create live dashboards with health check statuses and graphs tracking usage.
• Determine the critical monitoring metrics for our services and propagate them up to the live dashboards. Set these up as SLOs with related SLIs.
• Identify all services we use and integrating health checks on them, like B2, CloudFront, Neon, etc.
• Determine the properties to monitor across infrastructure we raise and maintain
• Determine alerting system and set up alarms

🔴 Out of Scope

• Nice-to-have metrics. Let’s aim for implementing mission-critical metrics like server uptime and mail deliverability.

🚧 Technical and/or Business Requirements

🏃 Milestones

ℹ️ If the project is so large that requires to be split into smaller body of works, list all the relevant milestones with a description of requirements for each. Also consider if it makes sense to split into two different projects. You can remove this section if there are no milestones.

Engineering Tasks

Cross-Functional Tasks

ℹ️ The project may require work from other teams, who are tracking their work individually. List all of the tasks required to complete this project. This list doesn’t need to be complete on a first draft and it will be expanded during the initial project kick-off.

📊 MarComm

Owner: Natalie Ivanova

Goal: Determine what to do when things go bad. Do we make an announcement? where, and how?

🧭 Services

Owner: Malini Das/Chris Aquino

Goal: Identifying any overlap between application/user metrics with this initiative. There may not be much overlap, but good to keep in touch.

🤝 Support - Product

Owner: Lisa Wess

Goal: Support should be flagged when we have a significant incident, and should be enmeshed with us as we work to resolve the issue. Work with Support to determine what they need from us and how to make their lives easier(ie: tying the uuid from User Metrics for Early Bird to determine which users are affected, etc).