From 0fc3f93d40d7e9214b7dc793d1afd18a9ce45fdf Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 15:04:51 +0000 Subject: [PATCH 1/6] docs(metrics): review PR #38 code headed upstream in daeuniverse/dae#1015 Review of the metrics code merged by PR #38, scoped to what is shared with the upstream PR (40 files byte-identical to #1015 head 8573436). Findings: node metrics export proxy credentials through the link label; duplicate node names in a group fail the whole scrape with HTTP 500 (reproduced with client_golang v1.19.1); tcp4(DNS)/tcp6(DNS) series alias tcp4/tcp6 under v2.1.1; plus HELP, BasicAuth, example bind address, TLS permission and health-check counting fixes, each with a minimal patch sketch and the rollout order for #1015 and fork main. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- .plan/metrics/pr38-upstream-1015-review.md | 413 +++++++++++++++++++++ 1 file changed, 413 insertions(+) create mode 100644 .plan/metrics/pr38-upstream-1015-review.md diff --git a/.plan/metrics/pr38-upstream-1015-review.md b/.plan/metrics/pr38-upstream-1015-review.md new file mode 100644 index 0000000000..fcbdb4c034 --- /dev/null +++ b/.plan/metrics/pr38-upstream-1015-review.md @@ -0,0 +1,413 @@ +# PR #38 审查报告：将推向上游（daeuniverse/dae#1015）的 metrics 代码 + +- 审查对象：MaurUppi/dae#38（已合并，merge commit `a25e50e`，head `fe27ea6`） +- 关注范围：PR #38 中**会推送到上游 PR daeuniverse/dae#1015** 的 metrics 代码（#1015 head = `feat/metrics-endpoint-clean` @ `8573436`，base = 上游 `17cc1de`） +- 审查日期：2026-10-02 +- 目标：找出“最佳但不过度优化”的修复方案 + +--- + +## 1. 摘要与结论 + +**结论：#1015 在当前状态下不建议合并。需要先合入一个小型修复提交（约 8 个文件），覆盖 2 个 P0 和 4 个 P1 问题。** + +| 级别 | 数量 | 一句话概括 | +|---|---|---| +| **P0** | 2 | ① `link` 标签把代理节点的密码/UUID 导出到 `/metrics`；② 同组内节点重名会让整个 `/metrics` 返回 HTTP 500 | +| **P1** | 4 | TCP 健康序列重复导出；`dae_dns_cache_hit_total` 的 HELP 与实际语义不符；只设密码时 BasicAuth 被静默关闭；示例配置默认监听 `0.0.0.0` | +| **P2** | 3 | TLS 权限精确匹配会拒绝更严格的权限，另有一个函数是死代码；`dae_health_check_total` 把“无结论”的检查也计入；PR 需要 rebase，面板位置也不合适 | +| **P3** | 5 | 只记录、不建议在本轮修改（避免过度优化） | + +另外，审查确认 PR #38 的 reload 生命周期重构本身是正确的：`adoptPreparedGeneration` 是唯一的发布点，`managementServers` 只由 Run 所在的 goroutine 访问，reload 前会先预校验配置。审查中没有发现并发或生命周期缺陷。 + +--- + +## 2. 审查范围与方法 + +### 2.1 fork main 与 #1015 是否为同一份代码？ + +方法：用 GitHub API 读取 #1015 head（`8573436`）下各文件的 blob SHA，再与本地 `git rev-parse a25e50e:` 逐个比对。 + +| 结果 | 文件 | +|---|---| +| **相同（40 个）** | `cmd/{endpoint_config,management_servers,management_servers_test,reload_adoption_test,reload_manager,run,run_reload_worker,run_test}.go`；`control/{connection_metrics,connection_metrics_test,control_plane,dns_cache_metrics_test,dns_control,dns_controller_handle,dns_controller_response,dns_metrics,dns_metrics_test,dns_preference_wait_test,dns_singleflight_test,node_latency,tcp,udp,udp_task_pool}.go`；`pkg/metrics/*`（7 个）；`pkg/metricshttp/*`（3 个）；`common/file_permission{,_test}.go`；`config/config.go`；`example.dae`；`go.mod`；`go.sum`；`component/outbound/dialer_group.go` | +| **不同（1 个）** | `component/outbound/dialer/connectivity_check.go`：fork 额外带有 `e956321`（动态 CheckOpts / 移除 IPv6 probe skip）。#1015 的提交说明写明 “The probe list (CheckOpts) is unchanged”，只包含 metrics 访问器与计数器 | + +**推论**：本报告列出的问题**同时存在于 fork main 和 #1015**。同一个修复补丁可以直接应用到两边；`check()` 函数在两边也完全一致。 + +### 2.2 #1015 的提交结构（5 个） + +| 提交 | 内容 | +|---|---| +| `78ea02c` | `pkg/metricshttp`、TLS 权限检查、`endpoint_*` 配置 | +| `1d9686f` | `pkg/metrics` 各采集器，以及 control/dialer 的只读访问器 | +| `f6b01e8` | TCP/UDP/DNS 埋点 | +| `cbf59fc` | `cmd`：management server 生命周期、reload 发布点 | +| `8573436` | Grafana 面板（`.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json`） | + +### 2.3 方法说明 + +- 静态审查上述全部 Go 代码，并沿调用链追到 v2.1.1 的 dialer、health 和 reload 子系统。 +- 读取 outbound 依赖（`olicesx/outbound@cc86ced2e683`）中 `ExportToURL()` 的源码，确认 `Property.Link` 里有什么。 +- **实证实验**：在独立的 scratchpad 模块中，用与仓库相同版本的 `client_golang v1.19.1` 复现了 P0-2（输出见 §3.2）。 +- 本轮只做审查，没有改动仓库中的任何代码。 + +--- + +## 3. 逐条发现 + +> 行号以 fork main `a25e50e` 为准。由于文件逐字节相同，这些行号同样适用于 #1015 head `8573436`。 + +### 3.1 【P0-1 安全】`link` 标签泄露代理凭据 + +**现象** +`pkg/metrics/collector_runtime.go:50-61` 定义了 `dae_node_latency_seconds` 和 `dae_node_alive`，标签为 `{group, name, link}`；`:89-100` 把 `node.Link` 原样写入标签。`node.Link` 来自 `control/node_latency.go:71-79` 的 `d.Property().Link`。 + +**`Link` 的实际内容**（已读取 outbound 源码确认） + +| 协议 | 源码位置 | 包含的敏感信息 | +|---|---|---| +| Trojan | `dialer/trojan/trojan.go:133` 处 `Link: s.ExportToURL()`；`:176` 处 `User: url.User(t.Password)` | **明文密码** | +| Shadowsocks | `dialer/shadowsocks/shadowsocks.go:165,455` | `base64(cipher:password)`，可直接解码 | +| VLESS / VMess | `dialer/v2ray/v2ray.go:315,449` | VLESS 是 `url.User(s.ID)`（UUID）；VMess 是整个 JSON 的 base64，其中含 ID | + +**影响** +- 凡是能访问 `/metrics` 的人，都能拿到**所有代理节点的完整凭据**。默认没有 BasicAuth，而 `example.dae` 示例的监听地址是 `0.0.0.0:5556`（见 P1-4）。 +- 凭据还会进入 Prometheus TSDB、远程存储和 Grafana，且会在其中长期保留。 +- 风险对象是路由器或网关上的 dae，暴露面很大。 + +**推荐方案：删除这两个指标，而不是改造它们** + +理由： +1. 它们与 `dae_dialer_alive` / `dae_dialer_latency_last_seconds{network="tcp4"|"tcp6"}` 的信息**冗余**。 +2. #1015 自己的 Grafana 面板**没有用到** `dae_node_*`（已在 `8573436` 的面板 JSON 中检索确认）。 +3. 如果只删掉 `link`、保留 `{group, name}`，又会落入 P0-2 的重名问题。 +4. 删除可以让上游 diff 变小：`control/node_latency.go` 中新增的 `Name`/`Group` 字段可以一并回退。 + +```diff +--- a/pkg/metrics/collector_runtime.go ++++ b/pkg/metrics/collector_runtime.go +@@ type RuntimeCollector struct { + uploadRateBytesPerSecond *prometheus.Desc + downloadRateBytesPerSecond *prometheus.Desc +- nodeLatencySeconds *prometheus.Desc +- nodeAlive *prometheus.Desc + } +@@ func NewRuntimeCollector +- nodeLatencySeconds: prometheus.NewDesc("dae_node_latency_seconds", ..., []string{"group", "name", "link"}, nil), +- nodeAlive: prometheus.NewDesc("dae_node_alive", ..., []string{"group", "name", "link"}, nil), +@@ func (c *RuntimeCollector) Describe +- ch <- c.nodeLatencySeconds +- ch <- c.nodeAlive +@@ func (c *RuntimeCollector) Collect +- for _, node := range cp.SnapshotNodeLatencies() { +- ... +- } +--- a/control/node_latency.go ++++ b/control/node_latency.go +- Name string // human-readable dialer name, e.g. "香港标准 IEPL 专线 1" +- Group string // outbound group name, e.g. "FC_HK" +... +- snapshot.Name = d.Property().Name +- snapshot.Group = group.Name +``` + +测试同步修改：把 `pkg/metrics/collector_describe_test.go:98` 的 `TestRuntimeCollectorDescribeIncludesRuntimeAndNodeDescriptors` 改为只断言 4 个 runtime 描述符，并增加一条断言：没有任何描述符带有 `link` 标签。 + +### 3.2 【P0-2 可用性】同组内节点重名导致整个 `/metrics` 返回 500 + +**现象** +- `pkg/metrics/collector_dialer.go:125` 的标签是 `{group.Name, prop.Name, typ.String()}`。 +- `component/outbound/filter.go:150` 的 `NewDialerSetFromLinksContext` **不对节点名去重**。多订阅场景下（例如两个机场都有“香港 01”），配合 `filter: name(keyword: 香港)`，同一组内会出现同名 dialer。 +- `pkg/metricshttp/server.go:44` 使用 `promhttp.HandlerOpts{}`，其 `ErrorHandling` 默认值是 `HTTPErrorOnError`。 + +**实证**（独立模块，`client_golang v1.19.1`，构造两个 `{group="HK",dialer="香港 01",network="tcp4"}` 序列加一个正常 counter）： + +```text +HandlerOpts{}: status=500 body="An error has occurred while serving metrics: + collected metric "dae_dialer_alive" { ... dialer="香港 01" group="HK" network="tcp4" ... } + was collected before with the same name and label values" +ContinueOnError: status=200 body="... dae_dialer_alive{dialer="香港 01",group="HK",network="tcp4"} 1 + ... dae_dns_query_total 42" +``` + +**影响** +只要配置里出现一次重名，**整个 endpoint 就不返回任何指标**：DNS、连接、runtime 指标全部丢失，Prometheus 判定 target down。在多订阅用户中，这是高概率触发的问题（初步判断，数据未充分确认：触发比例没有统计数据）。 + +**推荐方案：两处改动，都很小** + +(a) 在采集器内按组为重名追加后缀，让标签保持唯一： + +```go +// dialerMetricName keeps label sets unique when a group holds several nodes +// with the same name (common with multiple subscriptions): a duplicate label +// set fails the whole scrape. +func dialerMetricName(seen map[string]int, name string) string { + seen[name]++ + if n := seen[name]; n > 1 { + return fmt.Sprintf("%s #%d", name, n) + } + return name +} +``` + +(b) 用 `ContinueOnError` 兜底。这样以后任何一个采集器出现标签冲突，都不会再让整个 endpoint 失效： + +```diff +--- a/pkg/metricshttp/server.go ++++ b/pkg/metricshttp/server.go +- promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), ++ promhttp.HandlerFor(registry, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}), +``` + +权衡说明： +- 后缀依赖 `group.Dialers` 的顺序，而该顺序来自 map 遍历，所以 reload 之后 “#2” 可能指向另一个节点。同一代 generation 内是稳定的，可以接受。 +- 如果要做到跨 reload 稳定，就得引入订阅 tag 或链接哈希这类新标签，并改动标签 schema，属于过度设计，本轮不建议。 +- 仅靠 (b) 不够：重复的那一条会被静默丢弃，而且每次抓取都会产生一次错误。所以 (a) 才是主修复，(b) 是防线。 + +### 3.3 【P1-1 正确性】`tcp4(DNS)` / `tcp6(DNS)` 与 `tcp4` / `tcp6` 重复导出 + +**现象** +- v2.1.1 把 TCP 统一为一个健康域。`component/outbound/dialer/dialer.go:296-297` 中 `collections[IdxDnsTcp4] = collections[IdxTcp4]`、`collections[IdxDnsTcp6] = collections[IdxTcp6]`。 +- `component/outbound/dialer_group.go:481-483` 对 `aliveDialerSets` 也做了同样的别名。上游自己的 `uniqueAliveDialerSets`（`dialer_group.go:610`）正是为了处理这种别名而存在的。 +- 但 `collector_dialer.go:17-26` 仍沿用 v2.1.1 之前的 8 类型表，于是每个 dialer 的 6 个指标中，TCP 部分以 `network="tcp4(DNS)"` 和 `"tcp4"` 两种标签**各导出一次**，`dae_group_alive_dialers_total` 也一样。 + +**影响** +- 跨 network 聚合会把 TCP 重复计算，例如 `sum by (group) (increase(dae_health_check_total[1h]))` 中 TCP 部分被算两次。 +- 每个 dialer 多出 2/8 = 25% 的序列。 +- #1015 的面板因为用了 `network!~".*DNS.*"` 过滤，没有直接受影响；但用户自己写的查询会受影响。 + +**推荐方案** +改为遍历上游的权威列表 `dialer.StandardHealthKeys()`（6 个互不相同的健康域），并改用已有的 `MustGetAliveDialerSet`。这样新增的 `DialerGroup.AliveDialerSets()` 就可以删除，进一步缩小上游 diff： + +```go +// dialerMetricNetworkTypes lists each distinct health collection once. +// tcp4(DNS)/tcp6(DNS) share the tcp4/tcp6 collection and alive set +// (see NewDialerContext and buildSelectionState), so they are not exported. +var dialerMetricNetworkTypes = func() (types [6]*dialer.NetworkType) { + for i, key := range dialer.StandardHealthKeys() { + types[i] = key.NetworkType() + } + return types +}() + +// in Collect, per group: + seen := make(map[string]int, len(group.Dialers)) + for _, d := range group.Dialers { + // ... nil checks unchanged ... + name := dialerMetricName(seen, prop.Name) + for _, typ := range dialerMetricNetworkTypes { + alive, lastLatency, avg10, movingAvg, hasLastLatency := d.GetCollectionState(typ) + labels := []string{group.Name, name, typ.String()} + // ... emits unchanged ... + } + } + for _, typ := range dialerMetricNetworkTypes { + if set := group.MustGetAliveDialerSet(typ); set != nil { + ch <- prometheus.MustNewConstMetric(c.groupAliveDialers, prometheus.GaugeValue, + float64(set.Len()), group.Name, typ.String()) + } + } +``` + +```diff +--- a/component/outbound/dialer_group.go ++++ b/component/outbound/dialer_group.go +-func (g *DialerGroup) AliveDialerSets() [8]*dialer.AliveDialerSet { +- return g.currentSelectionState().aliveDialerSets +-} +``` + +兼容性：保留下来的 6 个 label 值（`tcp4`、`tcp6`、`udp4(DNS)`、`udp6(DNS)`、`udp4`、`udp6`）与现状**逐字相同**。现有查询只要不显式依赖 `tcp4(DNS)` / `tcp6(DNS)`，就不受影响。 + +### 3.4 【P1-2 语义】`dae_dns_cache_hit_total` 的 HELP 与实际语义不符 + +**现象** +- `pkg/metrics/collector_dns.go:67` 的 HELP 写的是 “Total number of **fresh** DNS cache hits”。 +- 但 `control/dns_metrics.go:234-244` 的 `noteDNSCacheServed` 对 lazy（陈旧）命中也会执行 `dnsCacheHitTotal.Add(1)`。测试 `TestDNSStaleServedRefreshCountsLazyHit` 明确断言了 `hit=1, lazy=1`。 + +**影响** +- 指标一旦进入上游就成为公共 API，HELP 写错会误导所有使用者。 +- #1015 的面板已经按 “hit 包含 lazy” 来计算（新鲜命中 = `hit - lazy`），是正确的。 +- **fork 仓库**中的旧面板（`.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json` 和 `.plan/metrics/dae-dashboard.json`）却用 `hit + lazy` 计算“总命中率”，**重复计入了 lazy**；“Fresh Hit Ratio = hit/query” 实际也包含陈旧命中。 + +**推荐方案** + +```diff +- "Total number of fresh DNS cache hits", ++ "Total number of DNS queries answered from the response cache, including stale (lazy) hits also counted in dae_dns_cache_lazy_hit_total", +``` + +fork 侧：用 #1015 的面板文件替换上述两个旧面板。 + +### 3.5 【P1-3 安全】只设 `endpoint_password` 时 BasicAuth 被静默关闭 + +**现象** +`pkg/metricshttp/auth.go:14`：`if username == "" { return handler }`。如果用户只写了密码，以为已经开启了保护，实际上 endpoint 是完全开放的。反过来，只写用户名时，密码会被要求为空字符串，同样不符合预期。 + +**推荐方案**：在 `cmd/management_servers.go` 的 `resolveManagementServers` 中，构建 `cfg` 之后增加校验。由于 reload 前已经会预校验，这个改动也自动覆盖了 reload 路径： + +```go +if (cfg.Username == "") != (cfg.Password == "") { + return managementPlan{}, fmt.Errorf("endpoint_username and endpoint_password must be configured together") +} +``` + +测试：在 `TestResolveManagementServers` 的表中增加两条用例（只有用户名、只有密码），都期望返回错误。 + +### 3.6 【P1-4 安全默认值】`example.dae` 示例监听 `0.0.0.0:5556` + +**现象** +`example.dae:39` 是 `#endpoint_listen_address: '0.0.0.0:5556'`。dae 常部署在路由器上，用户照抄示例就会把 endpoint 绑定到所有接口，可能包括 WAN。即使修复了 P0-1，指标里仍有节点名、组名和 DNS 上游地址。 + +**推荐方案** + +```diff +- # Set a listen address to enable the endpoint server (disabled by default). +- #endpoint_listen_address: '0.0.0.0:5556' ++ # Set a listen address to enable the endpoint server (disabled by default). ++ # Metrics reveal node, group and DNS upstream names: keep it on loopback or a ++ # LAN address, and set endpoint_username/endpoint_password (and TLS) first. ++ #endpoint_listen_address: '127.0.0.1:5556' +``` + +### 3.7 【P2-1】TLS 权限是精确匹配，会拒绝更严格的权限；另有一个函数是死代码 + +**现象** +- `cmd/endpoint_config.go:57` 规定证书只能是 `0640` 或 `0644`；`:70` 规定私钥只能是 `0600`。于是 `0400` 的私钥会被拒绝，尽管它更安全；`0600` 或 `0444` 的证书也会被拒绝，而 Caddy、acme.sh 等工具常生成这类权限（初步判断，数据未充分确认：各工具的默认权限未逐一核实）。 +- `common/file_permission.go:15` 的 `ValidateFilePermissionNotTooOpen` **没有任何调用者**（`grep` 确认），是死代码。它与上游已有的 `config/config_merger.go:78`、`common/subscription/subscription.go:128` 中的内联检查重复。 + +**推荐方案**：改为“禁止位”检查，与上游现有的 `0037` 掩码风格保持一致。两个函数合并为一个： + +```go +// ValidateFilePermissionForbidden rejects a directory, or a file whose +// permission bits include any of forbidden. +func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.FileMode) error { + if fi.IsDir() { + return fmt.Errorf("cannot read a directory: %v", path) + } + if perm := fi.Mode().Perm(); perm&forbidden != 0 { + return fmt.Errorf("permissions %04o for '%v' are too open; bits %04o must not be set", perm, path, forbidden.Perm()) + } + return nil +} +``` + +调用方式：证书用 `0o022`（同组和其他用户不可写），私钥用 `0o077`（同组和其他用户不可访问）。同时更新 `example.dae:42` 的注释和 `common/file_permission_test.go`。 + +更小的备选方案：保留 `ValidateFilePermissionAllowed`，只扩充允许列表（证书加 `0600/0444/0440/0400`，私钥加 `0400`）。改动更少，但枚举式写法较笨拙，死代码问题也没有解决，因此不推荐。 + +### 3.8 【P2-2】`dae_health_check_total` 把“无结论”的检查也计入 + +**现象** +`component/outbound/dialer/connectivity_check.go:1413` 在 `check()` 入口无条件执行 `CheckTotal.Add(1)`。以下几种情况没有健康结论，却也被计入总数： +- context canceled（dialer 退役） +- `errCheckOptionUnavailable`（probe 基础设施失败） +- `ok=false, err=nil`（跳过） + +它们不会计入 `CheckFailureTotal`（`:1486`），于是面板里的 `1 - failure/total` 成功率被系统性抬高。 + +**推荐方案**：把计数移到两个有结论的分支里，各 1 行： + +```diff +- d.mustGetCollection(opts.networkType).CheckTotal.Add(1) + ... + case ok && err == nil: + d.collectionFineMu.Lock() + collection := d.mustGetCollection(opts.networkType) ++ collection.CheckTotal.Add(1) + ... + case err != nil && !d.isLifecycleTeardownError(err) && !stderrors.Is(err, errCheckOptionUnavailable): + ... ++ collection.CheckTotal.Add(1) + collection.CheckFailureTotal.Add(1) +``` + +另外，HELP 文本可以相应改为 “...health checks that produced a verdict”。 + +### 3.9 【P2-3 流程】rebase 与面板位置 + +- 上游 main 当前为 `e3fee8f`，比 #1015 的 base `17cc1de` 多 6 个提交：#1111、#1117（config Marshaller）、#1122、#1124（config 校验）、#1130、#1125。其中 #1117 和 #1124 涉及 `config/`，可能与 `config/config.go` 中新增的 `endpoint_*` 字段冲突（初步判断，数据未充分确认：尚未实际 rebase 验证）。 +- 上游仓库（`dbae2e8`）**没有** `.plan/` 目录。把 Grafana 面板放进 `.plan/metrics/` 会引入一个上游原本不存在的内部目录。建议移到 `docs/`（上游自 #1100 起维护中英文手册），或者把 JSON 作为 PR 附件或单独的文档 PR 提交，由维护者决定。 +- 上游文档目前没有 `endpoint_*` 的配置说明。建议至少在 PR 描述里给出配置示例，以及“对外暴露前需要认证和 TLS”的提示。 + +### 3.10 【P3】仅记录，本轮不建议修改 + +| 项 | 说明 | 不修改的理由 | +|---|---|---| +| `endpoint_prometheus_enabled` 默认 `false` | 只设置监听地址时，得到一个没有任何 handler 的 server（访问 `/metrics` 返回 404） | 属于配置语义选择。可以在 PR 中询问维护者：是改为默认 `true`，还是在 `start` 时打印一条警告 | +| DNS upstream 序列不清理 | `dnsUpstreamMetrics` 存放在跨 reload 共享的 store 中，上游配置被移除后，旧序列仍会继续导出 | 基数受配置约束；`asis` 上游统一记为 `"asis"`，不会膨胀 | +| `RouteDialTcpContext` 埋点 | 该导出函数在仓库内没有调用者（`control/tcp.go:384`） | 无害，只服务外部嵌入方；也可以删掉以缩小 diff | +| `pprof_port is deprecated` 警告 | `endpointConfigFromGlobal` 在 reload 预校验和 `apply` 时各执行一次，所以每次 reload 打印 2 次 | 只是日志噪声 | +| 端点地址冲突检测 | 只拒绝字符串完全等于 `localhost:` 的地址；`127.0.0.1:`、`:` 不会被拦截，失败时只记日志 | 绑定失败有日志可查，补全需要做地址规范化，收益低 | + +--- + +## 4. 修复落地策略 + +### 4.1 推荐流程 + +1. 在 `feat/metrics-endpoint-clean` 上追加**一个**提交 `fix(metrics): address review findings`，覆盖 §3.1 至 §3.8。 +2. 将该分支 rebase 到上游最新 main（`e3fee8f`），解决冲突，然后重新跑 `go build`、`go vet`、golangci-lint、gofmt 和 SPDX 检查，并 force-push 到 PR 分支（这是 PR 作者自己的分支，force-push 可以接受）。 +3. 把同一个修复提交 cherry-pick 到 fork 的新分支，向 fork main 提 PR。相关文件逐字节相同，预期可以干净应用（初步判断，数据未充分确认）。fork 侧另外用 #1015 的面板替换 `.plan/metrics/` 下的两个旧面板。 + +为什么选择“追加 1 个提交”，而不是把修复折进原来的 5 个提交： +- 对评审者来说，增量最清楚。 +- 上游通常 squash 合并，提交历史的“洁癖”收益很低（初步判断，数据未充分确认：上游合并策略未逐一核实）。 + +### 4.2 预计改动规模（初步判断，数据未充分确认） + +| 文件 | 改动 | +|---|---| +| `pkg/metrics/collector_runtime.go` | 删除 2 个 node 指标（约 −30 行） | +| `control/node_latency.go` | 回退 `Name`/`Group`（−4 行） | +| `pkg/metrics/collector_dialer.go` | 改用 6 个健康域，加入重名后缀（约 ±30 行） | +| `component/outbound/dialer_group.go` | 删除 `AliveDialerSets()`（−4 行） | +| `pkg/metricshttp/server.go` | 改用 `ContinueOnError`（1 行） | +| `pkg/metrics/collector_dns.go` | 修正 HELP（1 行） | +| `cmd/management_servers.go` | 用户名和密码成对校验（+3 行） | +| `cmd/endpoint_config.go` + `common/file_permission.go` | 改为禁止位检查，删除死代码（约 −30 行） | +| `component/outbound/dialer/connectivity_check.go` | 移动 `CheckTotal` 计数（±3 行） | +| `example.dae` | 改示例地址和注释（±4 行） | +| 测试 | 见 §4.3（约 +60 行） | + +### 4.3 测试建议（适度即可，不为测试而重构） + +- `dialerMetricName`：表驱动测试，覆盖不重名、重名两次、重名三次。 +- `dialerMetricNetworkTypes`：断言共 6 个，`Index()` 两两不同，`String()` 恰好是 `{tcp4, tcp6, udp4(DNS), udp6(DNS), udp4, udp6}`。 +- `pkg/metricshttp`：注册一个故意产生重复标签的采集器，断言 `/metrics` 返回 200 且其余指标仍在。这相当于把 §3.2 的实证固化为回归测试。 +- `collector_describe_test.go`：断言没有任何 Desc 带 `link` 标签，runtime 采集器只有 4 个描述符。 +- `TestResolveManagementServers`：增加用户名和密码只配其一的用例。 +- `TestValidateEndpointTLSFilesChecksPermissions`：增加“私钥 0400 通过”“证书 0600 通过”“私钥 0640 拒绝”。 +- **不建议**为测试 `Collect` 而给 `ControlPlane` 增加测试构造器。那需要导出内部字段，代价大于收益。 + +--- + +## 5. 风险与权衡：为什么不做更多 + +| 可选的更大改动 | 不做的理由 | +|---|---| +| 给 dialer 指标加 `subtag` 或链接哈希标签，做跨 reload 稳定的唯一标识 | 要改标签 schema，增加基数，还要解释哈希来源；重名后缀加 ContinueOnError 已经能消除故障 | +| 把连接计数器挪到共享 store，避免 reload 归零 | Prometheus 的 `rate()` 原生能处理计数器重置；PR #38 也已在说明中把它列为已知行为 | +| DNS upstream 序列按配置做 GC | 基数有上限，收益低 | +| 端点地址规范化与冲突预检 | 绑定失败有日志可查，规范化逻辑容易出现平台差异 | +| 在 fork 侧为 dae_node_* 做兼容层 | 冗余指标，fork 面板改用 #1015 的面板即可 | + +--- + +## 6. 未核实事项清单 + +1. #1015 rebase 到 `e3fee8f` 是否有冲突，以及冲突范围。（初步判断，数据未充分确认） +2. 修复提交能否直接 cherry-pick 到 fork main。理论上可以，因为文件相同，但未实际执行。 +3. 多订阅用户中同组节点重名的实际比例。（此信息/数据暂未核实完毕） +4. Caddy、acme.sh 等工具默认生成的证书和私钥权限。（此信息/数据暂未核实完毕） +5. 上游维护者对 `endpoint_prometheus_enabled` 默认值和面板存放位置的偏好。 +6. 本报告的 P0 和 P1 都基于代码阅读和一次独立实证，**没有**在真实 dae 进程上做端到端抓取。建议修复后在测试机上用两个含重名节点的订阅实际抓一次 `/metrics` 验证。 + +--- + +## 7. 后续思考 + +- **Q1：** 如果 `/metrics` 已经被暴露过一段时间，凭据可能已经写入 Prometheus 或远程存储。是否需要提醒已部署 fork 的用户轮换节点密码或 UUID，并清理 TSDB 中的 `dae_node_*` 序列？ +- **Q2：** 健康指标的 `network` 维度应该直接对齐 v2.1.1 的“健康域”（`tcp`、`dns_udp`、`data_udp`），还是继续沿用 `tcp4(DNS)` 这类旧字符串以保证兼容？这个选择会影响上游长期的指标契约。 +- **Q3：** fork 独有的动态 CheckOpts（`e956321`）与 #1015 分叉。上游合并 #1015 之后，fork 每次同步都会在 `connectivity_check.go` 上产生冲突。是把它单独提交为上游 PR，还是在 fork 中长期作为补丁维护？ From 3c026570df26a920c320715dafd1cd8c7a2fa87a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 15:19:11 +0000 Subject: [PATCH 2/6] fix(metrics): address review findings - Drop dae_node_latency_seconds and dae_node_alive: their link label is the node share link (Trojan password, SS cipher:password, VLESS/VMess ID), and they duplicate the per-dialer health metrics. Revert the NodeLatencySnapshot Name/Group fields they needed. - Keep dialer label sets unique: same-named nodes in one group (common with several subscriptions) get a " #N" suffix. A duplicate label set made promhttp return HTTP 500 for the whole scrape; serve with ContinueOnError so one collector error cannot blank the endpoint. - Export each distinct health collection once via StandardHealthKeys. tcp4(DNS)/tcp6(DNS) alias tcp4/tcp6 in v2.1.1 and doubled every TCP series. Remove the DialerGroup.AliveDialerSets accessor. - Fix the dae_dns_cache_hit_total help: it includes lazy hits. - Require endpoint_username and endpoint_password together; a password alone left the endpoint open. - example.dae: bind the sample endpoint to 127.0.0.1. - TLS files: reject group/other write on the certificate and any group/other access on the key instead of matching exact modes, so 0400 keys and 0600 certificates pass. Remove the unused ValidateFilePermissionNotTooOpen. - Count dae_health_check_total only for checks with a verdict; skips and probe-infrastructure failures diluted the failure ratio. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- cmd/endpoint_config.go | 6 +- cmd/management_servers.go | 5 ++ cmd/management_servers_test.go | 23 +++++++ cmd/run_test.go | 18 ++++++ common/file_permission.go | 31 ++------- common/file_permission_test.go | 61 +++++++++--------- .../outbound/dialer/check_counters_test.go | 49 +++++++++++++++ .../outbound/dialer/connectivity_check.go | 5 +- component/outbound/dialer_group.go | 4 -- control/node_latency.go | 4 -- example.dae | 8 ++- pkg/metrics/collector_describe_test.go | 63 +++++++++++++++++-- pkg/metrics/collector_dialer.go | 53 +++++++++------- pkg/metrics/collector_dialer_test.go | 51 +++++++++++++++ pkg/metrics/collector_dns.go | 2 +- pkg/metrics/collector_runtime.go | 30 --------- pkg/metricshttp/endpoint_server_test.go | 32 ++++++++++ pkg/metricshttp/server.go | 4 +- 18 files changed, 322 insertions(+), 127 deletions(-) create mode 100644 component/outbound/dialer/check_counters_test.go create mode 100644 pkg/metrics/collector_dialer_test.go diff --git a/cmd/endpoint_config.go b/cmd/endpoint_config.go index b7ce8dcc34..1e23bfaafb 100644 --- a/cmd/endpoint_config.go +++ b/cmd/endpoint_config.go @@ -54,7 +54,8 @@ func validateEndpointTLSFiles(cfg metricshttp.EndpointConfig) error { if err != nil { return fmt.Errorf("cannot stat endpoint_tls_certificate '%s': %w", cfg.TlsCertificate, err) } - if err = common.ValidateFilePermissionAllowed(cfg.TlsCertificate, certFi, 0o640, 0o644); err != nil { + // The certificate is public: reject only group or other write access. + if err = common.ValidateFilePermissionForbidden(cfg.TlsCertificate, certFi, 0o022); err != nil { return fmt.Errorf("invalid endpoint_tls_certificate: %w", err) } @@ -67,7 +68,8 @@ func validateEndpointTLSFiles(cfg metricshttp.EndpointConfig) error { if err != nil { return fmt.Errorf("cannot stat endpoint_tls_key '%s': %w", cfg.TlsKey, err) } - if err = common.ValidateFilePermissionAllowed(cfg.TlsKey, keyFi, 0o600); err != nil { + // The private key must not be accessible by group or others. + if err = common.ValidateFilePermissionForbidden(cfg.TlsKey, keyFi, 0o077); err != nil { return fmt.Errorf("invalid endpoint_tls_key: %w", err) } return nil diff --git a/cmd/management_servers.go b/cmd/management_servers.go index 6a528bd3dc..c13e54bf2b 100644 --- a/cmd/management_servers.go +++ b/cmd/management_servers.go @@ -54,6 +54,11 @@ func resolveManagementServers(conf *config.Config, log *logrus.Logger) (manageme } cfg := endpointConfigFromGlobal(conf, log) + // BasicAuth is keyed on the username; a password alone would leave the + // endpoint open while looking protected. + if (cfg.Username == "") != (cfg.Password == "") { + return managementPlan{}, fmt.Errorf("endpoint_username and endpoint_password must be configured together") + } if err := validateEndpointTLSFiles(cfg); err != nil { return managementPlan{}, fmt.Errorf("invalid endpoint tls config: %w", err) } diff --git a/cmd/management_servers_test.go b/cmd/management_servers_test.go index a95b48099b..df3d3f2fdb 100644 --- a/cmd/management_servers_test.go +++ b/cmd/management_servers_test.go @@ -125,6 +125,29 @@ func TestResolveManagementServers(t *testing.T) { } }) + t.Run("username and password must be configured together", func(t *testing.T) { + for _, tc := range []struct{ username, password string }{ + {username: "", password: "secret"}, + {username: "admin", password: ""}, + } { + conf := &config.Config{} + conf.Global.EndpointListenAddress = "127.0.0.1:5556" + conf.Global.EndpointUsername = tc.username + conf.Global.EndpointPassword = tc.password + _, err := resolveManagementServers(conf, log) + if err == nil || !strings.Contains(err.Error(), "endpoint_username and endpoint_password") { + t.Fatalf("username=%q password=%q: err = %v, want pairing error", tc.username, tc.password, err) + } + } + conf := &config.Config{} + conf.Global.EndpointListenAddress = "127.0.0.1:5556" + conf.Global.EndpointUsername = "admin" + conf.Global.EndpointPassword = "secret" + if _, err := resolveManagementServers(conf, log); err != nil { + t.Fatalf("paired credentials: %v", err) + } + }) + t.Run("tls validation error", func(t *testing.T) { conf := &config.Config{} conf.Global.EndpointListenAddress = "127.0.0.1:5556" diff --git a/cmd/run_test.go b/cmd/run_test.go index d3fa07b847..1ac0de77ae 100644 --- a/cmd/run_test.go +++ b/cmd/run_test.go @@ -99,4 +99,22 @@ func TestValidateEndpointTLSFilesChecksPermissions(t *testing.T) { if err == nil { t.Fatal("expected too-open certificate permissions to fail") } + + // Stricter modes than the common defaults must keep passing. + strictCert := writeEndpointFile(t, "cert-strict.pem", 0o600) + strictKey := writeEndpointFile(t, "key-strict.pem", 0o400) + if err := validateEndpointTLSFiles(metricshttp.EndpointConfig{ + TlsCertificate: strictCert, + TlsKey: strictKey, + }); err != nil { + t.Fatalf("expected 0600 certificate and 0400 key to pass: %v", err) + } + + groupReadableKey := writeEndpointFile(t, "key-open.pem", 0o640) + if err := validateEndpointTLSFiles(metricshttp.EndpointConfig{ + TlsCertificate: cert, + TlsKey: groupReadableKey, + }); err == nil { + t.Fatal("expected group-readable key to fail") + } } diff --git a/common/file_permission.go b/common/file_permission.go index 4165c098d0..3d17a6f83c 100644 --- a/common/file_permission.go +++ b/common/file_permission.go @@ -8,37 +8,16 @@ package common import ( "fmt" "os" - "sort" - "strings" ) -func ValidateFilePermissionNotTooOpen(path string, fi os.FileInfo) error { +// ValidateFilePermissionForbidden rejects a directory, or a file whose +// permission bits include any of forbidden. +func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.FileMode) error { if fi.IsDir() { return fmt.Errorf("cannot read a directory: %v", path) } - if fi.Mode()&0o037 > 0 { - return fmt.Errorf("permissions %04o for '%v' are too open; requires the file is NOT writable by the same group and NOT accessible by others; suggest 0640 or 0600", fi.Mode()&0o777, path) + if perm := fi.Mode().Perm(); perm&forbidden != 0 { + return fmt.Errorf("permissions %04o for '%v' are too open; bits %04o must not be set", perm, path, forbidden.Perm()) } return nil } - -func ValidateFilePermissionAllowed(path string, fi os.FileInfo, allowedModes ...os.FileMode) error { - if fi.IsDir() { - return fmt.Errorf("cannot read a directory: %v", path) - } - perm := fi.Mode().Perm() - for _, mode := range allowedModes { - if perm == mode.Perm() { - return nil - } - } - if len(allowedModes) == 0 { - return fmt.Errorf("permissions %04o for '%v' are invalid", perm, path) - } - allowed := make([]string, 0, len(allowedModes)) - for _, mode := range allowedModes { - allowed = append(allowed, fmt.Sprintf("%04o", mode.Perm())) - } - sort.Strings(allowed) - return fmt.Errorf("permissions %04o for '%v' are invalid; allowed: %s", perm, path, strings.Join(allowed, ", ")) -} diff --git a/common/file_permission_test.go b/common/file_permission_test.go index 4ecaee355a..854462d3bf 100644 --- a/common/file_permission_test.go +++ b/common/file_permission_test.go @@ -28,39 +28,38 @@ func writeTempFile(t *testing.T, mode os.FileMode) (string, os.FileInfo) { return path, fi } -func TestValidateFilePermissionNotTooOpen(t *testing.T) { - _, fi0600 := writeTempFile(t, 0o600) - if err := ValidateFilePermissionNotTooOpen("test-0600", fi0600); err != nil { - t.Fatalf("0600 should pass: %v", err) +func TestValidateFilePermissionForbidden(t *testing.T) { + for _, tc := range []struct { + mode os.FileMode + forbidden os.FileMode + wantErr bool + }{ + // Private key: no group/other access. + {mode: 0o600, forbidden: 0o077}, + {mode: 0o400, forbidden: 0o077}, + {mode: 0o640, forbidden: 0o077, wantErr: true}, + {mode: 0o604, forbidden: 0o077, wantErr: true}, + // Certificate: no group/other write. + {mode: 0o644, forbidden: 0o022}, + {mode: 0o640, forbidden: 0o022}, + {mode: 0o600, forbidden: 0o022}, + {mode: 0o444, forbidden: 0o022}, + {mode: 0o664, forbidden: 0o022, wantErr: true}, + {mode: 0o646, forbidden: 0o022, wantErr: true}, + } { + path, fi := writeTempFile(t, tc.mode) + err := ValidateFilePermissionForbidden(path, fi, tc.forbidden) + if (err != nil) != tc.wantErr { + t.Fatalf("mode %04o forbidden %04o: err = %v, wantErr %v", tc.mode, tc.forbidden, err, tc.wantErr) + } } - _, fi0640 := writeTempFile(t, 0o640) - if err := ValidateFilePermissionNotTooOpen("test-0640", fi0640); err != nil { - t.Fatalf("0640 should pass: %v", err) - } - - _, fi0644 := writeTempFile(t, 0o644) - if err := ValidateFilePermissionNotTooOpen("test-0644", fi0644); err == nil { - t.Fatal("0644 should fail as too open") - } -} - -func TestValidateFilePermissionAllowed(t *testing.T) { - _, fi0600 := writeTempFile(t, 0o600) - if err := ValidateFilePermissionAllowed("key", fi0600, 0o600); err != nil { - t.Fatalf("0600 should pass for key: %v", err) - } - if err := ValidateFilePermissionAllowed("key", fi0600, 0o640, 0o644); err == nil { - t.Fatal("0600 should fail for cert-only allowed modes") - } - - _, fi0640 := writeTempFile(t, 0o640) - if err := ValidateFilePermissionAllowed("cert", fi0640, 0o640, 0o644); err != nil { - t.Fatalf("0640 should pass for cert: %v", err) + dir := t.TempDir() + fi, err := os.Stat(dir) + if err != nil { + t.Fatalf("stat dir: %v", err) } - - _, fi0644 := writeTempFile(t, 0o644) - if err := ValidateFilePermissionAllowed("cert", fi0644, 0o640, 0o644); err != nil { - t.Fatalf("0644 should pass for cert: %v", err) + if err := ValidateFilePermissionForbidden(dir, fi, 0o077); err == nil { + t.Fatal("a directory should be rejected") } } diff --git a/component/outbound/dialer/check_counters_test.go b/component/outbound/dialer/check_counters_test.go new file mode 100644 index 0000000000..8d485ae974 --- /dev/null +++ b/component/outbound/dialer/check_counters_test.go @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: AGPL-3.0-only + * Copyright (c) 2022-2026, daeuniverse Organization + */ + +package dialer + +import ( + "context" + "fmt" + "testing" +) + +// TestCheck_CountersCountOnlyVerdicts pins the health-check counters exported +// as dae_health_check_total / dae_health_check_failure_total: a skip or a +// probe-infrastructure failure carries no node evidence and must not dilute +// the failure ratio. +func TestCheck_CountersCountOnlyVerdicts(t *testing.T) { + d := newNamedTestDialer(t, "counter-node") + typ := newTestNetworkType() + + steps := []struct { + name string + result func() (bool, error) + wantTotal uint64 + wantFailure uint64 + }{ + {"success", func() (bool, error) { return true, nil }, 1, 0}, + {"node failure", func() (bool, error) { return false, fmt.Errorf("connection refused") }, 2, 1}, + {"check option unavailable", func() (bool, error) { + return false, wrapCheckOptionError(fmt.Errorf("resolve refused")) + }, 2, 1}, + {"plain skip", func() (bool, error) { return false, nil }, 2, 1}, + } + for _, step := range steps { + result := step.result + opts := &CheckOption{ + networkType: typ, + CheckFunc: func(context.Context, *NetworkType) (bool, error) { + return result() + }, + } + _, _ = d.check(opts, false, nil) + total, failure := d.GetCollectionCounters(typ) + if total != step.wantTotal || failure != step.wantFailure { + t.Fatalf("after %s: total=%d failure=%d, want %d and %d", step.name, total, failure, step.wantTotal, step.wantFailure) + } + } +} diff --git a/component/outbound/dialer/connectivity_check.go b/component/outbound/dialer/connectivity_check.go index 42ba4318c8..110918357c 100644 --- a/component/outbound/dialer/connectivity_check.go +++ b/component/outbound/dialer/connectivity_check.go @@ -1410,7 +1410,6 @@ func (d *Dialer) check(opts *CheckOption, isResuscitation bool, cycle *cycleResu const maxAttempts = 2 var bestLatency time.Duration checkedAt := time.Now() - d.mustGetCollection(opts.networkType).CheckTotal.Add(1) for range maxAttempts { ctx, cancel := context.WithTimeout(d.ctx, Timeout) @@ -1440,6 +1439,9 @@ func (d *Dialer) check(opts *CheckOption, isResuscitation bool, cycle *cycleResu case ok && err == nil: d.collectionFineMu.Lock() collection := d.mustGetCollection(opts.networkType) + // CheckTotal counts only checks that produced a verdict; skips, + // teardown and probe-infrastructure failures carry no health evidence. + collection.CheckTotal.Add(1) collection.LastProbe = DialerProbeObservationSnapshot{ CheckedAt: checkedAt, Alive: true, @@ -1483,6 +1485,7 @@ func (d *Dialer) check(opts *CheckOption, isResuscitation bool, cycle *cycleResu Alive: false, Message: err.Error(), } + collection.CheckTotal.Add(1) collection.CheckFailureTotal.Add(1) d.collectionFineMu.Unlock() diff --git a/component/outbound/dialer_group.go b/component/outbound/dialer_group.go index 8ed0f7cee2..11206131ae 100644 --- a/component/outbound/dialer_group.go +++ b/component/outbound/dialer_group.go @@ -176,10 +176,6 @@ func (g *DialerGroup) MinCheckInterval() time.Duration { return min } -func (g *DialerGroup) AliveDialerSets() [8]*dialer.AliveDialerSet { - return g.currentSelectionState().aliveDialerSets -} - func (d *DialerGroup) MustGetAliveDialerSet(typ *dialer.NetworkType) *dialer.AliveDialerSet { return d.currentSelectionState().aliveDialerSets[typ.Index()] } diff --git a/control/node_latency.go b/control/node_latency.go index 568d50b37c..f9152287a7 100644 --- a/control/node_latency.go +++ b/control/node_latency.go @@ -15,8 +15,6 @@ import ( // NodeLatencySnapshot describes the latest observable latency status for one node link. type NodeLatencySnapshot struct { Link string - Name string // human-readable dialer name, e.g. "香港标准 IEPL 专线 1" - Group string // outbound group name, e.g. "FC_HK" LatencyMs *int32 Alive bool Message string @@ -73,8 +71,6 @@ func (c *ControlPlane) SnapshotNodeLatencies() []NodeLatencySnapshot { } snapshot := bestNodeLatencySnapshotForDialer(d) - snapshot.Name = d.Property().Name - snapshot.Group = group.Name if existing, ok := latenciesByLink[snapshot.Link]; !ok || preferNodeLatencySnapshot(snapshot, existing) { latenciesByLink[snapshot.Link] = snapshot } diff --git a/example.dae b/example.dae index 078672b046..326f1472fa 100644 --- a/example.dae +++ b/example.dae @@ -36,10 +36,14 @@ global { # Endpoint configuration for metrics and diagnostics. # Set a listen address to enable the endpoint server (disabled by default). - #endpoint_listen_address: '0.0.0.0:5556' + # Metrics reveal node, group and DNS upstream names: keep it on loopback or a + # LAN address, and set endpoint_username/endpoint_password (and TLS) first. + #endpoint_listen_address: '127.0.0.1:5556' + # Username and password must be set together. #endpoint_username: '' #endpoint_password: '' - # TLS permission policy: certificate must be 0640 or 0644; private key must be 0600. + # TLS permission policy: the certificate must not be writable by group or + # others; the private key must not be accessible by group or others (e.g. 0600). #endpoint_tls_certificate: '' #endpoint_tls_key: '' #endpoint_prometheus_enabled: true diff --git a/pkg/metrics/collector_describe_test.go b/pkg/metrics/collector_describe_test.go index a5cd8bc4a6..dd2f2a94e3 100644 --- a/pkg/metrics/collector_describe_test.go +++ b/pkg/metrics/collector_describe_test.go @@ -7,12 +7,40 @@ package metrics import ( "regexp" + "slices" + "strings" "testing" "github.com/prometheus/client_golang/prometheus" ) -var descNamePattern = regexp.MustCompile(`fqName: "([^"]+)"`) +var ( + descNamePattern = regexp.MustCompile(`fqName: "([^"]+)"`) + descLabelsPattern = regexp.MustCompile(`variableLabels: \{([^}]*)\}`) +) + +// descriptorLabels maps each descriptor name to its variable label names. +func descriptorLabels(collector prometheus.Collector) map[string][]string { + ch := make(chan *prometheus.Desc, 64) + collector.Describe(ch) + close(ch) + + labels := make(map[string][]string) + for desc := range ch { + s := desc.String() + name := descNamePattern.FindStringSubmatch(s) + vars := descLabelsPattern.FindStringSubmatch(s) + if len(name) != 2 || len(vars) != 2 { + continue + } + if vars[1] == "" { + labels[name[1]] = nil + } else { + labels[name[1]] = strings.Split(vars[1], ",") + } + } + return labels +} func descriptorNames(collector prometheus.Collector) map[string]struct{} { ch := make(chan *prometheus.Desc, 64) @@ -95,14 +123,41 @@ func TestConnCollectorDescribeIncludesPhase2Descriptors(t *testing.T) { }) } -func TestRuntimeCollectorDescribeIncludesRuntimeAndNodeDescriptors(t *testing.T) { +func TestRuntimeCollectorDescribeIncludesRuntimeDescriptors(t *testing.T) { names := descriptorNames(NewRuntimeCollector(nil)) requireDescriptors(t, names, []string{ "dae_runtime_upload_bytes_total", "dae_runtime_download_bytes_total", "dae_runtime_upload_rate_bytes_per_second", "dae_runtime_download_rate_bytes_per_second", - "dae_node_latency_seconds", - "dae_node_alive", }) + // The node metrics exported the share link, which carries credentials. + requireNoDescriptor(t, names, "dae_node_latency_seconds") + requireNoDescriptor(t, names, "dae_node_alive") +} + +// TestNoDescriptorExportsLinkLabel guards against exporting node share links: +// they embed proxy passwords and UUIDs. +func TestNoDescriptorExportsLinkLabel(t *testing.T) { + collectors := map[string]prometheus.Collector{ + "dns": NewDnsCollector(nil), + "dialer": NewDialerCollector(nil), + "conn": NewConnCollector(nil), + "runtime": NewRuntimeCollector(nil), + } + for collectorName, collector := range collectors { + labels := descriptorLabels(collector) + if len(labels) == 0 { + t.Fatalf("%s collector: no descriptor labels parsed; descriptorLabels is out of date", collectorName) + } + for desc, names := range labels { + if slices.Contains(names, "link") { + t.Fatalf("%s collector: %s exports a link label", collectorName, desc) + } + } + } + // Keep the check non-vacuous: a labelled descriptor must parse. + if got := descriptorLabels(NewDialerCollector(nil))["dae_dialer_alive"]; !slices.Equal(got, []string{"group", "dialer", "network"}) { + t.Fatalf("dae_dialer_alive labels = %v", got) + } } diff --git a/pkg/metrics/collector_dialer.go b/pkg/metrics/collector_dialer.go index 31057f0b68..4885ff595f 100644 --- a/pkg/metrics/collector_dialer.go +++ b/pkg/metrics/collector_dialer.go @@ -6,23 +6,32 @@ package metrics import ( - "github.com/daeuniverse/dae/common/consts" + "fmt" + "github.com/daeuniverse/dae/component/outbound/dialer" "github.com/prometheus/client_golang/prometheus" ) -// dialerMetricNetworkTypes must be indexed by the Idx* constants in -// component/outbound/dialer (IdxDnsTcp4=0 … IdxUdp6=7) so that -// AliveDialerSets()[i] and dialerMetricNetworkTypes[i] stay in sync. -var dialerMetricNetworkTypes = [8]dialer.NetworkType{ - {L4Proto: consts.L4ProtoStr_TCP, IpVersion: consts.IpVersionStr_4, IsDns: true}, // [0] IdxDnsTcp4 - {L4Proto: consts.L4ProtoStr_TCP, IpVersion: consts.IpVersionStr_6, IsDns: true}, // [1] IdxDnsTcp6 - {L4Proto: consts.L4ProtoStr_UDP, IpVersion: consts.IpVersionStr_4, IsDns: true, UdpHealthDomain: dialer.UdpHealthDomainDns}, // [2] IdxDnsUdp4 - {L4Proto: consts.L4ProtoStr_UDP, IpVersion: consts.IpVersionStr_6, IsDns: true, UdpHealthDomain: dialer.UdpHealthDomainDns}, // [3] IdxDnsUdp6 - {L4Proto: consts.L4ProtoStr_TCP, IpVersion: consts.IpVersionStr_4, IsDns: false}, // [4] IdxTcp4 - {L4Proto: consts.L4ProtoStr_TCP, IpVersion: consts.IpVersionStr_6, IsDns: false}, // [5] IdxTcp6 - {L4Proto: consts.L4ProtoStr_UDP, IpVersion: consts.IpVersionStr_4, IsDns: false}, // [6] IdxUdp4 - {L4Proto: consts.L4ProtoStr_UDP, IpVersion: consts.IpVersionStr_6, IsDns: false}, // [7] IdxUdp6 +// dialerMetricNetworkTypes lists each distinct health collection once. +// tcp4(DNS)/tcp6(DNS) share the tcp4/tcp6 collection and alive set (see +// dialer.NewDialerContext and DialerGroup.buildSelectionState), so exporting +// them would duplicate the TCP series. +var dialerMetricNetworkTypes = func() (types [6]*dialer.NetworkType) { + for i, key := range dialer.StandardHealthKeys() { + types[i] = key.NetworkType() + } + return types +}() + +// dialerMetricName keeps label sets unique when a group holds several nodes +// with the same name (common with multiple subscriptions): a duplicate label +// set fails the whole scrape. +func dialerMetricName(seen map[string]int, name string) string { + seen[name]++ + if n := seen[name]; n > 1 { + return fmt.Sprintf("%s #%d", name, n) + } + return name } type DialerCollector struct { @@ -66,7 +75,7 @@ func NewDialerCollector(state *State) *DialerCollector { ), healthCheckTotal: prometheus.NewDesc( "dae_health_check_total", - "Total number of dialer connectivity health checks", + "Total number of dialer connectivity health checks that produced a verdict (success or failure)", []string{"group", "dialer", "network"}, nil, ), @@ -107,6 +116,7 @@ func (c *DialerCollector) Collect(ch chan<- prometheus.Metric) { if group == nil { continue } + seen := make(map[string]int, len(group.Dialers)) for _, d := range group.Dialers { if d == nil { continue @@ -115,26 +125,27 @@ func (c *DialerCollector) Collect(ch chan<- prometheus.Metric) { if prop == nil { continue } - for i := range dialerMetricNetworkTypes { - typ := dialerMetricNetworkTypes[i] - alive, lastLatency, avg10, movingAvg, hasLastLatency := d.GetCollectionState(&typ) + name := dialerMetricName(seen, prop.Name) + for _, typ := range dialerMetricNetworkTypes { + alive, lastLatency, avg10, movingAvg, hasLastLatency := d.GetCollectionState(typ) aliveFloat := 0.0 if alive { aliveFloat = 1 } - labels := []string{group.Name, prop.Name, typ.String()} + labels := []string{group.Name, name, typ.String()} ch <- prometheus.MustNewConstMetric(c.dialerAlive, prometheus.GaugeValue, aliveFloat, labels...) if hasLastLatency { ch <- prometheus.MustNewConstMetric(c.dialerLatencyLast, prometheus.GaugeValue, lastLatency.Seconds(), labels...) } ch <- prometheus.MustNewConstMetric(c.dialerLatencyAvg10, prometheus.GaugeValue, avg10.Seconds(), labels...) ch <- prometheus.MustNewConstMetric(c.dialerLatencyMovingAvg, prometheus.GaugeValue, movingAvg.Seconds(), labels...) - checkTotal, checkFailureTotal := d.GetCollectionCounters(&typ) + checkTotal, checkFailureTotal := d.GetCollectionCounters(typ) ch <- prometheus.MustNewConstMetric(c.healthCheckTotal, prometheus.CounterValue, float64(checkTotal), labels...) ch <- prometheus.MustNewConstMetric(c.healthCheckFailure, prometheus.CounterValue, float64(checkFailureTotal), labels...) } } - for i, set := range group.AliveDialerSets() { + for _, typ := range dialerMetricNetworkTypes { + set := group.MustGetAliveDialerSet(typ) if set == nil { continue } @@ -143,7 +154,7 @@ func (c *DialerCollector) Collect(ch chan<- prometheus.Metric) { prometheus.GaugeValue, float64(set.Len()), group.Name, - dialerMetricNetworkTypes[i].String(), + typ.String(), ) } } diff --git a/pkg/metrics/collector_dialer_test.go b/pkg/metrics/collector_dialer_test.go new file mode 100644 index 0000000000..0f81b20a6c --- /dev/null +++ b/pkg/metrics/collector_dialer_test.go @@ -0,0 +1,51 @@ +/* + * SPDX-License-Identifier: AGPL-3.0-only + * Copyright (c) 2022-2026, daeuniverse Organization + */ + +package metrics + +import ( + "testing" +) + +func TestDialerMetricNameSuffixesDuplicates(t *testing.T) { + seen := make(map[string]int) + got := []string{ + dialerMetricName(seen, "HK 01"), + dialerMetricName(seen, "JP 01"), + dialerMetricName(seen, "HK 01"), + dialerMetricName(seen, "HK 01"), + } + want := []string{"HK 01", "JP 01", "HK 01 #2", "HK 01 #3"} + for i := range want { + if got[i] != want[i] { + t.Fatalf("name %d = %q, want %q (all: %q)", i, got[i], want[i], got) + } + } +} + +// TestDialerMetricNetworkTypesAreDistinct pins one series per health +// collection: tcp4(DNS)/tcp6(DNS) alias tcp4/tcp6 and must not be exported. +func TestDialerMetricNetworkTypesAreDistinct(t *testing.T) { + want := map[string]bool{ + "tcp4": true, "tcp6": true, + "udp4(DNS)": true, "udp6(DNS)": true, + "udp4": true, "udp6": true, + } + indexes := make(map[int]string) + for _, typ := range dialerMetricNetworkTypes { + label := typ.String() + if !want[label] { + t.Fatalf("unexpected network label %q", label) + } + delete(want, label) + if prev, ok := indexes[typ.Index()]; ok { + t.Fatalf("%q and %q share collection index %d", prev, label, typ.Index()) + } + indexes[typ.Index()] = label + } + if len(want) != 0 { + t.Fatalf("missing network labels: %v", want) + } +} diff --git a/pkg/metrics/collector_dns.go b/pkg/metrics/collector_dns.go index 519d169ad0..5084f7d8d9 100644 --- a/pkg/metrics/collector_dns.go +++ b/pkg/metrics/collector_dns.go @@ -64,7 +64,7 @@ func NewDnsCollector(state *State) *DnsCollector { ), cacheHitTotal: prometheus.NewDesc( "dae_dns_cache_hit_total", - "Total number of fresh DNS cache hits", + "Total number of DNS queries answered from the response cache, including stale (lazy) hits also counted in dae_dns_cache_lazy_hit_total", nil, nil, ), diff --git a/pkg/metrics/collector_runtime.go b/pkg/metrics/collector_runtime.go index d71d7aa486..629ba71e07 100644 --- a/pkg/metrics/collector_runtime.go +++ b/pkg/metrics/collector_runtime.go @@ -16,8 +16,6 @@ type RuntimeCollector struct { downloadBytesTotal *prometheus.Desc uploadRateBytesPerSecond *prometheus.Desc downloadRateBytesPerSecond *prometheus.Desc - nodeLatencySeconds *prometheus.Desc - nodeAlive *prometheus.Desc } func NewRuntimeCollector(state *State) *RuntimeCollector { @@ -47,18 +45,6 @@ func NewRuntimeCollector(state *State) *RuntimeCollector { nil, nil, ), - nodeLatencySeconds: prometheus.NewDesc( - "dae_node_latency_seconds", - "Best known per-node latency snapshot exported from runtime latency probing", - []string{"group", "name", "link"}, - nil, - ), - nodeAlive: prometheus.NewDesc( - "dae_node_alive", - "Whether the node is currently considered alive by runtime latency probing", - []string{"group", "name", "link"}, - nil, - ), } } @@ -67,8 +53,6 @@ func (c *RuntimeCollector) Describe(ch chan<- *prometheus.Desc) { ch <- c.downloadBytesTotal ch <- c.uploadRateBytesPerSecond ch <- c.downloadRateBytesPerSecond - ch <- c.nodeLatencySeconds - ch <- c.nodeAlive } func (c *RuntimeCollector) Collect(ch chan<- prometheus.Metric) { @@ -85,18 +69,4 @@ func (c *RuntimeCollector) Collect(ch chan<- prometheus.Metric) { ch <- prometheus.MustNewConstMetric(c.downloadBytesTotal, prometheus.CounterValue, float64(snapshot.DownloadTotal)) ch <- prometheus.MustNewConstMetric(c.uploadRateBytesPerSecond, prometheus.GaugeValue, float64(snapshot.UploadRate)) ch <- prometheus.MustNewConstMetric(c.downloadRateBytesPerSecond, prometheus.GaugeValue, float64(snapshot.DownloadRate)) - - for _, node := range cp.SnapshotNodeLatencies() { - if node.Link == "" { - continue - } - alive := 0.0 - if node.Alive { - alive = 1 - } - ch <- prometheus.MustNewConstMetric(c.nodeAlive, prometheus.GaugeValue, alive, node.Group, node.Name, node.Link) - if node.LatencyMs != nil { - ch <- prometheus.MustNewConstMetric(c.nodeLatencySeconds, prometheus.GaugeValue, float64(*node.LatencyMs)/1000.0, node.Group, node.Name, node.Link) - } - } } diff --git a/pkg/metricshttp/endpoint_server_test.go b/pkg/metricshttp/endpoint_server_test.go index f3c83a973e..a9cb1aff54 100644 --- a/pkg/metricshttp/endpoint_server_test.go +++ b/pkg/metricshttp/endpoint_server_test.go @@ -98,3 +98,35 @@ func TestNewEndpointServerServesPrometheusAndPprofWithAuth(t *testing.T) { t.Fatalf("unexpected pprof status: got=%d want=%d", pprofRec.Code, http.StatusOK) } } + +// dupCollector emits the same label set twice, as two same-named nodes in one +// group would. +type dupCollector struct{ desc *prometheus.Desc } + +func (c dupCollector) Describe(ch chan<- *prometheus.Desc) { ch <- c.desc } +func (c dupCollector) Collect(ch chan<- prometheus.Metric) { + ch <- prometheus.MustNewConstMetric(c.desc, prometheus.GaugeValue, 1, "dup") + ch <- prometheus.MustNewConstMetric(c.desc, prometheus.GaugeValue, 0, "dup") +} + +func TestPrometheusHandlerSurvivesCollectorError(t *testing.T) { + reg := prometheus.NewRegistry() + counter := prometheus.NewCounter(prometheus.CounterOpts{ + Name: "dae_test_metric_total", + Help: "test metric", + }) + reg.MustRegister(counter, dupCollector{prometheus.NewDesc("dae_test_dup", "dup", []string{"name"}, nil)}) + + server := NewEndpointServer(EndpointConfig{ + ListenAddress: "127.0.0.1:0", + PrometheusEnabled: true, + }, reg) + rec := httptest.NewRecorder() + server.Handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil)) + if rec.Code != http.StatusOK { + t.Fatalf("one bad collector must not fail the scrape: status=%d body=%q", rec.Code, rec.Body.String()) + } + if body := rec.Body.String(); !strings.Contains(body, "dae_test_metric_total") { + t.Fatalf("healthy metrics missing from body: %q", body) + } +} diff --git a/pkg/metricshttp/server.go b/pkg/metricshttp/server.go index 2763c507ef..fea3a83105 100644 --- a/pkg/metricshttp/server.go +++ b/pkg/metricshttp/server.go @@ -41,7 +41,9 @@ func NewEndpointServer(cfg EndpointConfig, registry *prometheus.Registry) *http. if cfg.PrometheusEnabled && registry != nil { mux.Handle(NormalizePrometheusPath(cfg.PrometheusPath), BasicAuthMiddleware( - promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), + // ContinueOnError: one collector's error must not blank the + // whole scrape. + promhttp.HandlerFor(registry, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}), cfg.Username, cfg.Password, ), ) From ea9f7060a3e79d897fe81e161d350339572ec596 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 15:19:23 +0000 Subject: [PATCH 3/6] docs(metrics): replace stale fork dashboards with the upstream PR dashboard Both fork dashboards queried the removed dae_node_* metrics and treated dae_dns_cache_hit_total as fresh-only (hit + lazy double-counted stale hits). Use the daeuniverse/dae#1015 dashboard (blob 21ce121), which derives fresh hits as hit - lazy and does not use dae_node_*. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- ...e Transparent Proxy-Grafana_dashboard.json | 3421 ----------------- ..._Transparent_Proxy-Grafana_dashboard.json} | 720 ++-- 2 files changed, 361 insertions(+), 3780 deletions(-) delete mode 100644 .plan/metrics/dae Transparent Proxy-Grafana_dashboard.json rename .plan/metrics/{dae-dashboard.json => dae_Transparent_Proxy-Grafana_dashboard.json} (84%) diff --git a/.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json b/.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json deleted file mode 100644 index 5320e32cd8..0000000000 --- a/.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json +++ /dev/null @@ -1,3421 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": { - "type": "grafana", - "uid": "-- Grafana --" - }, - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "description": "dae eBPF transparent proxy — full metrics dashboard (Phase 1 gauges + Phase 2 counters/histograms)", - "editable": true, - "fiscalYearStartMonth": 0, - "graphTooltip": 1, - "id": 33, - "links": [], - "panels": [ - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 100, - "panels": [], - "title": "Overview", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Number of alive proxy dialers (network=tcp4, all groups)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "green", - "value": 1 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 0, - "y": 1 - }, - "id": 101, - "options": { - "colorMode": "background", - "graphMode": "none", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(dae_dialer_alive{network=\"$network\", group=~\"$group\"})", - "instant": true, - "legendFormat": "Alive Dialers", - "refId": "A" - } - ], - "title": "Alive Dialers", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Current active TCP connections being proxied through dae", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 3, - "y": 1 - }, - "id": 104, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_tcp_connections_active", - "instant": true, - "legendFormat": "Active", - "refId": "A" - } - ], - "title": "Active TCP Conn", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS queries per second handled by dae", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "decimals": 2, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "blue", - "value": null - } - ] - }, - "unit": "reqps" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 6, - "y": 1 - }, - "id": 102, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_query_total[$__rate_interval])", - "legendFormat": "DNS qps", - "refId": "A" - } - ], - "title": "DNS Query Rate", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Fraction of DNS queries served from cache (fresh hits only). Stale/lazy hits excluded.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "decimals": 1, - "mappings": [], - "max": 1, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "yellow", - "value": 0.3 - }, - { - "color": "green", - "value": 0.7 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 9, - "y": 1 - }, - "id": 103, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval]) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", - "legendFormat": "Hit Ratio", - "refId": "A" - } - ], - "title": "DNS Cache Fresh Hit Ratio", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Current active UDP endpoint associations and task queues", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "blue", - "value": null - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 12, - "y": 1 - }, - "id": 105, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_endpoints_active", - "instant": true, - "legendFormat": "UDP Endpoints", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_task_queues_active", - "instant": true, - "legendFormat": "UDP Task Queues", - "refId": "B" - } - ], - "title": "Active UDP", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Process resident memory (RSS)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "yellow", - "value": 134217728 - }, - { - "color": "red", - "value": 536870912 - } - ] - }, - "unit": "bytes" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 16, - "y": 1 - }, - "id": 106, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "process_resident_memory_bytes", - "instant": true, - "legendFormat": "RSS", - "refId": "A" - } - ], - "title": "Memory (RSS)", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Cumulative bytes uploaded and downloaded since last process start. Resets on restart.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "blue", - "value": null - } - ] - }, - "unit": "bytes" - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "Upload Total" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "green", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "Download Total" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "blue", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 20, - "y": 1 - }, - "id": 702, - "options": { - "colorMode": "background", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "vertical", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_runtime_upload_bytes_total", - "instant": true, - "legendFormat": "Upload Total", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_runtime_download_bytes_total", - "instant": true, - "legendFormat": "Download Total", - "refId": "B" - } - ], - "title": "Cumulative Traffic", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Real-time upload and download throughput sampled by the runtime traffic statistics engine (250 ms buckets). These are direct gauge values — no rate() needed.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 15, - "gradientMode": "opacity", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "smooth", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "Bps" - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "Upload" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "green", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "Download" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "blue", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 5, - "w": 24, - "x": 0, - "y": 4 - }, - "id": 701, - "options": { - "legend": { - "calcs": [ - "mean", - "max", - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_runtime_upload_rate_bytes_per_second", - "legendFormat": "Upload", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_runtime_download_rate_bytes_per_second", - "legendFormat": "Download", - "refId": "B" - } - ], - "title": "Network Throughput", - "type": "timeseries" - }, - { - "id": 703, - "title": "Node Alive", - "type": "timeseries", - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 9 - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_node_alive", - "legendFormat": "{{link}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "min": 0, - "max": 1, - "custom": { - "lineWidth": 2, - "fillOpacity": 20, - "drawStyle": "line" - }, - "color": { - "mode": "palette-classic" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "green", - "value": 1 - } - ] - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "list", - "placement": "bottom" - }, - "tooltip": { - "mode": "multi" - } - } - }, - { - "id": 704, - "title": "Node Latency (ms)", - "type": "timeseries", - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 9 - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_node_latency_seconds * 1000", - "legendFormat": "{{link}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "ms", - "custom": { - "lineWidth": 2, - "fillOpacity": 10 - }, - "color": { - "mode": "palette-classic" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "yellow", - "value": 100 - }, - { - "color": "red", - "value": 500 - } - ] - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "list", - "placement": "bottom" - }, - "tooltip": { - "mode": "multi" - } - } - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 9 - }, - "id": 200, - "panels": [], - "title": "Dialer Health", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 1, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "yellow", - "value": 0.8 - }, - { - "color": "green", - "value": 0.95 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 0, - "y": 10 - }, - "id": 202, - "options": { - "colorMode": "background", - "graphMode": "none", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "expr": "1 - sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) / sum(increase(dae_health_check_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range]))", - "legendFormat": "__auto", - "range": true, - "refId": "A" - } - ], - "title": "健康检查成功率", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "decimals": 0, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "yellow", - "value": 1 - }, - { - "color": "red", - "value": 5 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 3, - "x": 3, - "y": 10 - }, - "id": 203, - "options": { - "colorMode": "background", - "graphMode": "none", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "expr": "sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range]))", - "legendFormat": "__auto", - "refId": "A" - } - ], - "title": "检查失败数", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Number of alive dialers per group for the selected network type. Drop to 0 = group is unusable.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 3, - "w": 18, - "x": 6, - "y": 10 - }, - "id": 201, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "auto", - "wideLayout": true - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(dae_dialer_alive{network=\"$network\", group=~\"$group\"}) by (group)", - "legendFormat": "{{group}}", - "refId": "A" - } - ], - "title": "Alive Dialers by Group (network=$network)", - "type": "stat" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 13 - }, - "id": 300, - "panels": [], - "title": "Dialer Latency", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Most recent health check round-trip latency per dialer, displayed in milliseconds. Zero means no health check data yet (e.g., block/direct built-ins).", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "fieldMinMax": false, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "#EAB839", - "value": 200 - }, - { - "color": "red", - "value": 300 - } - ] - }, - "unit": "ms" - }, - "overrides": [] - }, - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 14 - }, - "id": 301, - "options": { - "displayMode": "gradient", - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": false - }, - "maxVizHeight": 300, - "minVizHeight": 40, - "minVizWidth": 8, - "namePlacement": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true, - "sizing": "auto", - "text": { - "titleSize": 10 - }, - "valueMode": "color" - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sort(dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", - "instant": true, - "legendFormat": "{{group}} / {{dialer}}", - "range": false, - "refId": "A" - } - ], - "title": "Last Health Check Latency (network=$network)", - "transparent": true, - "type": "bargauge" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "EWMA (thick line) smooths out noise from individual health checks. Instantaneous (thin) shows raw last-check latency. EWMA = current best estimate of dialer health.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "ms" - }, - "overrides": [ - { - "matcher": { - "id": "byFrameRefID", - "options": "B" - }, - "properties": [] - } - ] - }, - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 14 - }, - "id": 302, - "options": { - "displayMode": "gradient", - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": false - }, - "maxVizHeight": 300, - "minVizHeight": 16, - "minVizWidth": 8, - "namePlacement": "auto", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true, - "sizing": "auto", - "text": { - "titleSize": 10 - }, - "valueMode": "color" - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sort(dae_dialer_latency_moving_avg_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", - "instant": true, - "legendFormat": "{{group}} / {{dialer}}", - "range": false, - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "expr": "sort(dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", - "hide": true, - "instant": true, - "legendFormat": "{{group}} / {{dialer}} (instant)", - "range": false, - "refId": "B" - } - ], - "title": "Latency: EWMA vs Instantaneous (network=$network)", - "transparent": true, - "type": "bargauge" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 24 - }, - "id": 400, - "panels": [], - "title": "DNS", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "custom": { - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "scaleDistribution": { - "type": "linear" - } - } - }, - "overrides": [] - }, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 25 - }, - "id": 409, - "interval": "30s", - "maxDataPoints": 50, - "options": { - "calculate": false, - "cellGap": 1, - "color": { - "exponent": 0.5, - "fill": "dark-orange", - "mode": "scheme", - "reverse": false, - "scale": "exponential", - "scheme": "RdYlGn", - "steps": 64 - }, - "exemplars": { - "color": "rgba(255,0,255,0.7)" - }, - "filterValues": { - "le": 1e-09 - }, - "legend": { - "show": true - }, - "rowsFrame": { - "layout": "auto" - }, - "tooltip": { - "mode": "single", - "showColorScale": false, - "yHistogram": false - }, - "yAxis": { - "axisPlacement": "left", - "decimals": 1, - "reverse": false, - "unit": "s" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "expr": "ceil(increase(dae_dns_response_latency_seconds_bucket{le!=\"+Inf\"}[$__interval]))", - "format": "heatmap", - "legendFormat": "{{le}}", - "range": true, - "refId": "A" - } - ], - "title": "DNS Response Latency Heatmap", - "type": "heatmap" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS upstream forwarding attempt and error rates per upstream server. High error rates indicate upstream availability issues.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [ - { - "matcher": { - "id": "byFrameRefID", - "options": "B" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "red", - "mode": "fixed" - } - }, - { - "id": "custom.lineWidth", - "value": 2 - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 34 - }, - "id": 405, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_upstream_query_total[$__rate_interval])", - "legendFormat": "Query: {{upstream}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_upstream_err_total[$__rate_interval])", - "legendFormat": "Error: {{upstream}}", - "refId": "B" - } - ], - "title": "DNS Upstream Query & Error Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS upstream round-trip latency percentiles per upstream server. Measures time from forwarding request to receiving response.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "ms" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 34 - }, - "id": 406, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum(rate(dae_dns_upstream_latency_seconds_bucket[$__rate_interval])) by (le, upstream)) * 1000", - "legendFormat": "p50 {{upstream}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum(rate(dae_dns_upstream_latency_seconds_bucket[$__rate_interval])) by (le, upstream)) * 1000", - "legendFormat": "p95 {{upstream}}", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum(rate(dae_dns_upstream_latency_seconds_bucket[$__rate_interval])) by (le, upstream)) * 1000", - "legendFormat": "p99 {{upstream}}", - "refId": "C" - } - ], - "title": "DNS Upstream Latency (per upstream)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "End-to-end DNS handling latency percentiles (p50/p95/p99). Measured from query receipt to response sent, including cache lookup, forwarding, and response processing.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "ms" - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "p99" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "red", - "mode": "fixed" - } - }, - { - "id": "custom.lineWidth", - "value": 2 - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "p95" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "orange", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "p50" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "green", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 34 - }, - "id": 403, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum(rate(dae_dns_response_latency_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p50", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum(rate(dae_dns_response_latency_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p95", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum(rate(dae_dns_response_latency_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p99", - "refId": "C" - } - ], - "title": "DNS Response Latency (end-to-end)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS query rate broken down by cache outcome: fresh hit (served from valid cache), lazy/stale hit (reserved for future stale-while-revalidate — currently always 0), and miss (derived from per-upstream forwarded query rate).", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 20, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "Total Queries" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "blue", - "mode": "fixed" - } - }, - { - "id": "custom.lineWidth", - "value": 2 - }, - { - "id": "custom.fillOpacity", - "value": 0 - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "Cache Miss" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "orange", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "Cache Hit" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "green", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byName", - "options": "Lazy (Stale) Hit" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "yellow", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 42 - }, - "id": 401, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_query_total[$__rate_interval])", - "legendFormat": "Total Queries", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval])", - "legendFormat": "Cache Hit", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_cache_lazy_hit_total[$__rate_interval])", - "hide": true, - "legendFormat": "Lazy (Stale) Hit", - "refId": "C" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(dae_dns_upstream_query_total[$__rate_interval]))", - "legendFormat": "Cache Miss", - "refId": "D" - } - ], - "title": "DNS Query & Cache Breakdown", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Fresh hit ratio = cache_hit / total_queries. Total hit ratio includes stale (lazy) hits served during background refresh. Lower fresh hit ratio with higher total hit ratio indicates active background refresh. Stale-while-revalidate is not yet implemented upstream; the lazy-hit term is reserved and always 0, so Total and Fresh overlap today.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "max": 1, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 42 - }, - "id": 402, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval]) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", - "hide": false, - "legendFormat": "Fresh Hit Ratio", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "(rate(dae_dns_cache_hit_total[$__rate_interval]) + rate(dae_dns_cache_lazy_hit_total[$__rate_interval])) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", - "hide": true, - "legendFormat": "Total Hit Ratio (incl. stale)", - "refId": "B" - } - ], - "title": "DNS Cache Hit Ratio", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS cache entries currently cached, forwarder connections cached, and refused/rejected query counts.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [ - { - "matcher": { - "id": "byFrameRefID", - "options": "C" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "red", - "mode": "fixed" - } - } - ] - }, - { - "matcher": { - "id": "byFrameRefID", - "options": "D" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "orange", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 42 - }, - "id": 407, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_dns_cache_entries", - "legendFormat": "Cache Entries", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_dns_forwarder_cache_entries", - "legendFormat": "Forwarder Cache", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_rejected_total[$__rate_interval])", - "legendFormat": "Rejected/s", - "refId": "C" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_dns_refused_total[$__rate_interval])", - "legendFormat": "Refused/s (overload)", - "refId": "D" - } - ], - "title": "DNS Cache State & Error Events", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "DNS query concurrency in use — saturation gauge counting client DNS queries currently being handled by dae (inc on `HandleWithResponseWriter_` entry, dec on return). No hard limit is enforced upstream; persistent non-zero values indicate slow upstream or head-of-line queueing.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green" - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "In Use" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "orange", - "mode": "fixed" - } - } - ] - } - ] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 50 - }, - "id": 404, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_dns_concurrency_in_use", - "legendFormat": "In Use", - "refId": "A" - } - ], - "title": "DNS Concurrency In Use", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Number of in-flight DNS requests currently being forwarded to each upstream. Persistently high values suggest slow upstream or high concurrency.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 20, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green" - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 50 - }, - "id": 408, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_dns_forwarder_in_flight", - "legendFormat": "{{upstream}}", - "refId": "A" - } - ], - "title": "DNS Forwarder In-Flight (per upstream)", - "type": "timeseries" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 50 - }, - "id": 500, - "panels": [], - "title": "Connections", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "TCP proxy connection establishment rate per outbound group and protocol. Shows proxy throughput — each increment is a new successfully established proxied connection.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 51 - }, - "id": 501, - "options": { - "legend": { - "calcs": [ - "mean", - "last" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_tcp_connections_total[$__rate_interval])", - "legendFormat": "{{group}} / {{protocol}}", - "refId": "A" - } - ], - "title": "TCP Connection Establishment Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Current active TCP connections, UDP endpoint associations, and UDP task queues. These are instantaneous gauges, not rates.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 51 - }, - "id": 502, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_tcp_connections_active", - "legendFormat": "TCP Active", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_endpoints_active", - "legendFormat": "UDP Endpoints", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_task_queues_active", - "legendFormat": "UDP Task Queues", - "refId": "C" - } - ], - "title": "Active Connection Pools", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "UDP new endpoint association rate per group and protocol. Each increment is a new NAT mapping created for a UDP flow.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 59 - }, - "id": 503, - "options": { - "legend": { - "calcs": [ - "mean", - "last" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(dae_udp_connections_total[$__rate_interval])", - "legendFormat": "{{group}} / {{protocol}}", - "refId": "A" - } - ], - "title": "UDP New Endpoint Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Cumulative total TCP and UDP connections proxied since last restart (or SIGUSR1 reload). Resets on reload — use rate() for throughput.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 59 - }, - "id": 504, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_tcp_connections_total", - "legendFormat": "TCP {{group}}/{{protocol}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_connections_total", - "legendFormat": "UDP {{group}}/{{protocol}}", - "refId": "B" - } - ], - "title": "Cumulative Connection Totals (resets on reload)", - "type": "timeseries" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 67 - }, - "id": 600, - "panels": [], - "title": "Go Runtime & Process", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "CPU time consumed per second. Value of 1.0 = one full CPU core. Values > number of cores indicate compute saturation.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "percentunit" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 68 - }, - "id": 603, - "options": { - "legend": { - "calcs": [ - "lastNotNull" - ], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "none" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(process_cpu_seconds_total[$__rate_interval])", - "legendFormat": "CPU Usage", - "refId": "A" - } - ], - "title": "CPU Usage", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Go heap memory: allocated (in-use objects) vs. in-use heap spans. Sustained growth between GC cycles indicates memory pressure.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "bytes" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 68 - }, - "id": 602, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "go_memstats_heap_alloc_bytes", - "legendFormat": "Heap Alloc", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "go_memstats_heap_inuse_bytes", - "legendFormat": "Heap In-Use", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "process_resident_memory_bytes", - "legendFormat": "RSS", - "refId": "C" - } - ], - "title": "Memory", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Number of goroutines. Sudden spikes may indicate goroutine leak. Gradual growth under load is normal.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 68 - }, - "id": 601, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "none" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "go_goroutines", - "legendFormat": "Goroutines", - "refId": "A" - } - ], - "title": "Goroutines", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "GC pause duration percentiles. p99 > 10ms may cause latency spikes in proxy handling. Consider GOGC tuning if consistently high.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 5, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green" - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "ms" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 76 - }, - "id": 604, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum(rate(go_gc_duration_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p50", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum(rate(go_gc_duration_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p99", - "refId": "B" - } - ], - "title": "GC Pause Duration", - "type": "timeseries" - } - ], - "preload": false, - "refresh": "30s", - "schemaVersion": 40, - "tags": [ - "dae", - "proxy", - "ebpf" - ], - "templating": { - "list": [ - { - "current": { - "text": "prometheus", - "value": "cfa1zpgywwiyod" - }, - "includeAll": false, - "label": "Datasource", - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "type": "datasource" - }, - { - "current": { - "text": [ - "FC-Netflix", - "FC-TW", - "FC-HK" - ], - "value": [ - "FC-Netflix", - "FC-TW", - "FC-HK" - ] - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "definition": "label_values(dae_dialer_alive, group)", - "includeAll": true, - "label": "Group", - "multi": true, - "name": "group", - "options": [], - "query": { - "query": "label_values(dae_dialer_alive, group)", - "refId": "StandardVariableQuery" - }, - "refresh": 2, - "regex": "", - "sort": 1, - "type": "query" - }, - { - "current": { - "text": "tcp4", - "value": "tcp4" - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "definition": "label_values(dae_dialer_latency_last_seconds{network!~\".*DNS.*\"}, network)", - "includeAll": false, - "label": "Network", - "name": "network", - "options": [], - "query": { - "query": "label_values(dae_dialer_latency_last_seconds{network!~\".*DNS.*\"}, network)", - "refId": "StandardVariableQuery" - }, - "refresh": 2, - "regex": "", - "sort": 1, - "type": "query" - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": {}, - "timezone": "browser", - "title": "dae Transparent Proxy-v3", - "uid": "dae-metrics-v3", - "version": 3, - "weekStart": "" -} diff --git a/.plan/metrics/dae-dashboard.json b/.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json similarity index 84% rename from .plan/metrics/dae-dashboard.json rename to .plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json index 2132963326..21ce121539 100644 --- a/.plan/metrics/dae-dashboard.json +++ b/.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json @@ -19,7 +19,6 @@ "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 1, - "id": 32, "links": [], "panels": [ { @@ -40,7 +39,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Number of alive proxy dialers (network=tcp4, all groups)", + "description": "Number of alive proxy dialers for the selected network ($network) and groups ($group). An IPv6 family (tcp6/udp6) whose check target has no IPv6 address fails every check (ErrNoApplicableIP) and reports alive=0; this is not a node outage.", "fieldConfig": { "defaults": { "color": { @@ -101,7 +100,7 @@ "refId": "A" } ], - "title": "Alive Dialers", + "title": "有效节点", "type": "stat" }, { @@ -166,7 +165,7 @@ "refId": "A" } ], - "title": "Active TCP Conn", + "title": "TCP活跃数", "type": "stat" }, { @@ -231,7 +230,7 @@ "refId": "A" } ], - "title": "DNS Query Rate", + "title": "DNS 查询频率", "type": "stat" }, { @@ -272,7 +271,7 @@ }, "gridPos": { "h": 3, - "w": 3, + "w": 4, "x": 9, "y": 1 }, @@ -301,12 +300,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval]) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", + "expr": "(rate(dae_dns_cache_hit_total[$__rate_interval]) - rate(dae_dns_cache_lazy_hit_total[$__rate_interval])) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", "legendFormat": "Hit Ratio", "refId": "A" } ], - "title": "DNS Cache Fresh Hit Ratio", + "title": "DNS 缓存命中率", "type": "stat" }, { @@ -336,8 +335,8 @@ }, "gridPos": { "h": 3, - "w": 4, - "x": 12, + "w": 5, + "x": 13, "y": 1 }, "id": 105, @@ -419,8 +418,8 @@ }, "gridPos": { "h": 3, - "w": 4, - "x": 16, + "w": 5, + "x": 18, "y": 1 }, "id": 106, @@ -448,7 +447,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "process_resident_memory_bytes", + "expr": "process_resident_memory_bytes{job=\"dae\"}", "instant": true, "legendFormat": "RSS", "refId": "A" @@ -462,7 +461,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Cumulative bytes uploaded and downloaded since last process start. Resets on restart.", + "description": "Cumulative bytes uploaded and downloaded, kept per control plane. Resets on process restart and on config reload.", "fieldConfig": { "defaults": { "color": { @@ -514,10 +513,10 @@ ] }, "gridPos": { - "h": 3, - "w": 4, - "x": 20, - "y": 1 + "h": 5, + "w": 6, + "x": 0, + "y": 4 }, "id": 702, "options": { @@ -560,7 +559,7 @@ "refId": "B" } ], - "title": "Cumulative Traffic", + "title": "累计流量（重启或重载后重新计算）", "type": "stat" }, { @@ -568,7 +567,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Real-time upload and download throughput sampled by the runtime traffic statistics engine (250 ms buckets). These are direct gauge values — no rate() needed.", + "description": "Real-time upload and download throughput from the runtime traffic statistics engine: bytes over the last 1 s, aggregated from 250 ms buckets. These are direct gauge values — no rate() needed.", "fieldConfig": { "defaults": { "color": { @@ -655,8 +654,8 @@ }, "gridPos": { "h": 5, - "w": 24, - "x": 0, + "w": 17, + "x": 6, "y": 4 }, "id": 701, @@ -780,7 +779,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "1 - sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) / sum(increase(dae_health_check_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range]))", + "expr": "clamp_min(1 - sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) / clamp_min(sum(increase(dae_health_check_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])), 1), 0)", "legendFormat": "__auto", "range": true, "refId": "A" @@ -794,6 +793,7 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "时间范围内所选 network（$network）健康检查失败的累计次数（排除 block/direct 组）。失败包括连接错误，以及 tcp6/udp6 检查目标无 IPv6 地址时的 ErrNoApplicableIP（每次检查都计为失败，节点 alive=0）。尚未完成首次探测的节点（0ms）没有检查记录，不计入。", "fieldConfig": { "defaults": { "color": { @@ -894,7 +894,7 @@ }, "gridPos": { "h": 3, - "w": 18, + "w": 17, "x": 6, "y": 10 }, @@ -928,28 +928,15 @@ "refId": "A" } ], - "title": "Alive Dialers by Group (network=$network)", + "title": "有效节点 分组明细 (network=$network)", "type": "stat" }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 13 - }, - "id": 300, - "panels": [], - "title": "Dialer Latency", - "type": "row" - }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, - "description": "Most recent health check round-trip latency per dialer, displayed in milliseconds. Zero means no health check data yet (e.g., block/direct built-ins).", + "description": "每个存活 dialer 最近一次健康检查的实际延迟（network=$network）。0ms（灰色）= 节点存活但尚未完成首次探测（check_interval=900s）；非 0ms 为真实探测延迟。已失败（alive=0）的节点不显示。", "fieldConfig": { "defaults": { "color": { @@ -962,9 +949,13 @@ "mode": "absolute", "steps": [ { - "color": "green", + "color": "gray", "value": null }, + { + "color": "green", + "value": 1 + }, { "color": "#EAB839", "value": 200 @@ -983,7 +974,7 @@ "h": 10, "w": 12, "x": 0, - "y": 14 + "y": 13 }, "id": 301, "options": { @@ -1020,7 +1011,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "sort(dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", + "expr": "sort_desc(\n (\n (dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)\n and (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1)\n )\n or (0 * (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1))\n)", "instant": true, "legendFormat": "{{group}} / {{dialer}}", "range": false, @@ -1036,7 +1027,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "EWMA (thick line) smooths out noise from individual health checks. Instantaneous (thin) shows raw last-check latency. EWMA = current best estimate of dialer health.", + "description": "EWMA（指数加权移动平均，α=0.5）与最新瞬时延迟对比。0ms（灰色）= 节点存活但尚未完成首次探测，EWMA 尚未初始化；非 0ms 为路由决策使用的真实 EWMA 值。", "fieldConfig": { "defaults": { "color": { @@ -1048,9 +1039,13 @@ "mode": "absolute", "steps": [ { - "color": "green", + "color": "gray", "value": null }, + { + "color": "green", + "value": 1 + }, { "color": "red", "value": 80 @@ -1071,9 +1066,9 @@ }, "gridPos": { "h": 10, - "w": 12, + "w": 11, "x": 12, - "y": 14 + "y": 13 }, "id": 302, "options": { @@ -1110,7 +1105,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "sort(dae_dialer_latency_moving_avg_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", + "expr": "sort_desc(\n (\n (dae_dialer_latency_moving_avg_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)\n and (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1)\n )\n or (0 * (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1))\n)", "instant": true, "legendFormat": "{{group}} / {{dialer}}", "range": false, @@ -1122,7 +1117,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "sort(dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)", + "expr": "sort_desc(\n (\n (dae_dialer_latency_last_seconds{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} * 1000)\n and (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1)\n )\n or (0 * (dae_dialer_alive{network=\"$network\", group=~\"$group\", group!~\"block|direct\"} == 1))\n)", "hide": true, "instant": true, "legendFormat": "{{group}} / {{dialer}} (instant)", @@ -1140,7 +1135,7 @@ "h": 1, "w": 24, "x": 0, - "y": 24 + "y": 23 }, "id": 400, "panels": [], @@ -1169,9 +1164,9 @@ }, "gridPos": { "h": 9, - "w": 24, + "w": 23, "x": 0, - "y": 25 + "y": 24 }, "id": 409, "interval": "30s", @@ -1192,7 +1187,7 @@ "color": "rgba(255,0,255,0.7)" }, "filterValues": { - "le": 1e-09 + "le": 1e-9 }, "legend": { "show": true @@ -1228,6 +1223,7 @@ } ], "title": "DNS Response Latency Heatmap", + "transparent": true, "type": "heatmap" }, { @@ -1235,7 +1231,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "DNS upstream forwarding attempt and error rates per upstream server. High error rates indicate upstream availability issues.", + "description": "DNS upstream exchange and error rates per upstream server. A query is counted once an upstream dialer has been selected (selection failures are not counted) and includes singleflight leaders and background cache refreshes. Errors are failed exchanges plus replies that do not echo the request question (dropped as possible spoofing or cross-talk).", "fieldConfig": { "defaults": { "color": { @@ -1315,9 +1311,9 @@ }, "gridPos": { "h": 8, - "w": 8, + "w": 7, "x": 0, - "y": 34 + "y": 33 }, "id": 405, "options": { @@ -1364,7 +1360,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "DNS upstream round-trip latency percentiles per upstream server. Measures time from forwarding request to receiving response.", + "description": "DNS upstream round-trip latency percentiles per upstream server, observed for every upstream exchange (including failed ones) from forwarding the request to receiving the response or error, fallback attempts included.", "fieldConfig": { "defaults": { "color": { @@ -1425,8 +1421,8 @@ "gridPos": { "h": 8, "w": 8, - "x": 8, - "y": 34 + "x": 7, + "y": 33 }, "id": 406, "options": { @@ -1593,8 +1589,8 @@ "gridPos": { "h": 8, "w": 8, - "x": 16, - "y": 34 + "x": 15, + "y": 33 }, "id": 403, "options": { @@ -1650,7 +1646,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "DNS query rate broken down by cache outcome: fresh hit (served from valid cache), lazy/stale hit (reserved for future stale-while-revalidate — currently always 0), and miss (derived from per-upstream forwarded query rate).", + "description": "DNS query rate by cache outcome. Cache Hit = fresh hits only, cache_hit − lazy_hit (served from a valid cache entry); Lazy (Stale) Hit = stale entry served while a background refresh runs; Cache Miss = total − cache_hit (not answered from cache: upstream-resolved, rejected, refused, or a suppressed non-preferred cached answer). dae_dns_cache_hit_total already includes lazy hits and counts each request at most once.", "fieldConfig": { "defaults": { "color": { @@ -1779,9 +1775,9 @@ }, "gridPos": { "h": 8, - "w": 8, + "w": 7, "x": 0, - "y": 42 + "y": 41 }, "id": 401, "options": { @@ -1815,7 +1811,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval])", + "expr": "(rate(dae_dns_cache_hit_total[$__rate_interval]) - rate(dae_dns_cache_lazy_hit_total[$__rate_interval]))", "legendFormat": "Cache Hit", "refId": "B" }, @@ -1825,6 +1821,7 @@ "uid": "${datasource}" }, "expr": "rate(dae_dns_cache_lazy_hit_total[$__rate_interval])", + "hide": true, "legendFormat": "Lazy (Stale) Hit", "refId": "C" }, @@ -1833,7 +1830,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "sum(rate(dae_dns_upstream_query_total[$__rate_interval]))", + "expr": "clamp_min(rate(dae_dns_query_total[$__rate_interval]) - rate(dae_dns_cache_hit_total[$__rate_interval]), 0)", "legendFormat": "Cache Miss", "refId": "D" } @@ -1846,7 +1843,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Fresh hit ratio = cache_hit / total_queries. Total hit ratio includes stale (lazy) hits served during background refresh. Lower fresh hit ratio with higher total hit ratio indicates active background refresh. Stale-while-revalidate is not yet implemented upstream; the lazy-hit term is reserved and always 0, so Total and Fresh overlap today.", + "description": "Fresh hit ratio = (cache_hit − lazy_hit) / total_queries. Total hit ratio = cache_hit / total_queries; cache_hit already includes stale (lazy) hits served during background refresh. The gap between the two lines is the share of queries answered stale while a refresh runs.", "fieldConfig": { "defaults": { "color": { @@ -1903,18 +1900,46 @@ }, "unit": "percentunit" }, - "overrides": [] + "overrides": [ + { + "__systemRef": "hideSeriesFrom", + "matcher": { + "id": "byNames", + "options": { + "mode": "exclude", + "names": [ + "Fresh Hit Ratio" + ], + "prefix": "All except:", + "readOnly": true + } + }, + "properties": [ + { + "id": "custom.hideFrom", + "value": { + "legend": false, + "tooltip": false, + "viz": true + } + } + ] + } + ] }, "gridPos": { "h": 8, "w": 8, - "x": 8, - "y": 42 + "x": 7, + "y": 41 }, "id": 402, "options": { "legend": { - "calcs": [], + "calcs": [ + "lastNotNull", + "mean" + ], "displayMode": "list", "placement": "bottom", "showLegend": true @@ -1932,7 +1957,8 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(dae_dns_cache_hit_total[$__rate_interval]) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", + "expr": "(rate(dae_dns_cache_hit_total[$__rate_interval]) - rate(dae_dns_cache_lazy_hit_total[$__rate_interval])) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", + "hide": false, "legendFormat": "Fresh Hit Ratio", "refId": "A" }, @@ -1941,7 +1967,8 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "(rate(dae_dns_cache_hit_total[$__rate_interval]) + rate(dae_dns_cache_lazy_hit_total[$__rate_interval])) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", + "expr": "rate(dae_dns_cache_hit_total[$__rate_interval]) / clamp_min(rate(dae_dns_query_total[$__rate_interval]), 0.001)", + "hide": true, "legendFormat": "Total Hit Ratio (incl. stale)", "refId": "B" } @@ -1954,7 +1981,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "DNS cache entries currently cached, forwarder connections cached, and refused/rejected query counts.", + "description": "DNS response cache entries, cached forwarder connections, and per-second rates of rejected queries (routing reject verdict, empty answer) and refused queries (concurrency limit exceeded, REFUSED).", "fieldConfig": { "defaults": { "color": { @@ -2046,8 +2073,8 @@ "gridPos": { "h": 8, "w": 8, - "x": 16, - "y": 42 + "x": 15, + "y": 41 }, "id": 407, "options": { @@ -2107,25 +2134,12 @@ "title": "DNS Cache State & Error Events", "type": "timeseries" }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 50 - }, - "id": 500, - "panels": [], - "title": "Connections", - "type": "row" - }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, - "description": "TCP proxy connection establishment rate per outbound group and protocol. Shows proxy throughput — each increment is a new successfully established proxied connection.", + "description": "DNS query concurrency in use — client DNS queries currently being handled by dae (inc on `HandleWithResponseWriter_` entry, dec on return). A concurrency limit is enforced (ConcurrencyLimit, default 16384); queries beyond it are answered REFUSED and counted in dae_dns_refused_total. Persistent non-zero values indicate slow upstream or head-of-line queueing.", "fieldConfig": { "defaults": { "color": { @@ -2140,7 +2154,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 10, + "fillOpacity": 5, "gradientMode": "none", "hideFrom": { "legend": false, @@ -2179,22 +2193,37 @@ } ] }, - "unit": "reqps" + "unit": "short" }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "In Use" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + } + ] }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 51 + "y": 49 }, - "id": 501, + "id": 404, "options": { "legend": { "calcs": [ - "mean", - "last" + "lastNotNull" ], "displayMode": "list", "placement": "bottom", @@ -2213,12 +2242,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(dae_tcp_connections_total[$__rate_interval])", - "legendFormat": "{{group}} / {{protocol}}", + "expr": "dae_dns_concurrency_in_use", + "legendFormat": "In Use", "refId": "A" } ], - "title": "TCP Connection Establishment Rate", + "title": "DNS Concurrency In Use", "type": "timeseries" }, { @@ -2226,7 +2255,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Current active TCP connections, UDP endpoint associations, and UDP task queues. These are instantaneous gauges, not rates.", + "description": "Number of in-flight DNS requests currently being forwarded to each upstream. Persistently high values suggest slow upstream or high concurrency.", "fieldConfig": { "defaults": { "color": { @@ -2241,7 +2270,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 10, + "fillOpacity": 20, "gradientMode": "none", "hideFrom": { "legend": false, @@ -2250,7 +2279,7 @@ }, "insertNulls": false, "lineInterpolation": "linear", - "lineWidth": 2, + "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" @@ -2286,11 +2315,11 @@ }, "gridPos": { "h": 8, - "w": 12, + "w": 11, "x": 12, - "y": 51 + "y": 49 }, - "id": 502, + "id": 408, "options": { "legend": { "calcs": [ @@ -2313,38 +2342,33 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "dae_tcp_connections_active", - "legendFormat": "TCP Active", + "expr": "dae_dns_forwarder_in_flight", + "legendFormat": "{{upstream}}", "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_endpoints_active", - "legendFormat": "UDP Endpoints", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "dae_udp_task_queues_active", - "legendFormat": "UDP Task Queues", - "refId": "C" } ], - "title": "Active Connection Pools", + "title": "DNS Forwarder In-Flight (per upstream)", "type": "timeseries" }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 57 + }, + "id": 500, + "panels": [], + "title": "Connections", + "type": "row" + }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, - "description": "UDP new endpoint association rate per group and protocol. Each increment is a new NAT mapping created for a UDP flow.", + "description": "TCP proxy connection establishment rate per outbound group and protocol. Shows proxy throughput — each increment is a new successfully established proxied connection.", "fieldConfig": { "defaults": { "color": { @@ -2406,9 +2430,9 @@ "h": 8, "w": 12, "x": 0, - "y": 59 + "y": 58 }, - "id": 503, + "id": 501, "options": { "legend": { "calcs": [ @@ -2432,12 +2456,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(dae_udp_connections_total[$__rate_interval])", + "expr": "rate(dae_tcp_connections_total[$__rate_interval])", "legendFormat": "{{group}} / {{protocol}}", "refId": "A" } ], - "title": "UDP New Endpoint Rate", + "title": "TCP Connection Establishment Rate", "type": "timeseries" }, { @@ -2445,7 +2469,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Cumulative total TCP and UDP connections proxied since last restart (or SIGUSR1 reload). Resets on reload — use rate() for throughput.", + "description": "Current active TCP connections, UDP endpoint associations, and UDP task queues. These are instantaneous gauges, not rates.", "fieldConfig": { "defaults": { "color": { @@ -2460,7 +2484,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 5, + "fillOpacity": 10, "gradientMode": "none", "hideFrom": { "legend": false, @@ -2469,7 +2493,7 @@ }, "insertNulls": false, "lineInterpolation": "linear", - "lineWidth": 1, + "lineWidth": 2, "pointSize": 5, "scaleDistribution": { "type": "linear" @@ -2505,14 +2529,16 @@ }, "gridPos": { "h": 8, - "w": 12, + "w": 11, "x": 12, - "y": 59 + "y": 58 }, - "id": 504, + "id": 502, "options": { "legend": { - "calcs": [], + "calcs": [ + "lastNotNull" + ], "displayMode": "list", "placement": "bottom", "showLegend": true @@ -2530,8 +2556,8 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "dae_tcp_connections_total", - "legendFormat": "TCP {{group}}/{{protocol}}", + "expr": "dae_tcp_connections_active", + "legendFormat": "TCP Active", "refId": "A" }, { @@ -2539,33 +2565,29 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "dae_udp_connections_total", - "legendFormat": "UDP {{group}}/{{protocol}}", + "expr": "dae_udp_endpoints_active", + "legendFormat": "UDP Endpoints", "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "dae_udp_task_queues_active", + "legendFormat": "UDP Task Queues", + "refId": "C" } ], - "title": "Cumulative Connection Totals (resets on reload)", + "title": "Active Connection Pools", "type": "timeseries" }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 67 - }, - "id": 600, - "panels": [], - "title": "Go Runtime & Process", - "type": "row" - }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, - "description": "CPU time consumed per second. Value of 1.0 = one full CPU core. Values > number of cores indicate compute saturation.", + "description": "UDP new endpoint rate per group and protocol. Each increment is one UDP endpoint created whose first datagram was accepted (counted once per endpoint, not per packet).", "fieldConfig": { "defaults": { "color": { @@ -2619,21 +2641,22 @@ } ] }, - "unit": "percentunit" + "unit": "reqps" }, "overrides": [] }, "gridPos": { "h": 8, - "w": 8, + "w": 12, "x": 0, - "y": 68 + "y": 66 }, - "id": 603, + "id": 503, "options": { "legend": { "calcs": [ - "lastNotNull" + "mean", + "last" ], "displayMode": "list", "placement": "bottom", @@ -2642,7 +2665,7 @@ "tooltip": { "hideZeros": false, "mode": "multi", - "sort": "none" + "sort": "desc" } }, "pluginVersion": "11.5.2", @@ -2652,12 +2675,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "rate(process_cpu_seconds_total[$__rate_interval])", - "legendFormat": "CPU Usage", + "expr": "rate(dae_udp_connections_total[$__rate_interval])", + "legendFormat": "{{group}} / {{protocol}}", "refId": "A" } ], - "title": "CPU Usage", + "title": "UDP New Endpoint Rate", "type": "timeseries" }, { @@ -2665,7 +2688,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Go heap memory: allocated (in-use objects) vs. in-use heap spans. Sustained growth between GC cycles indicates memory pressure.", + "description": "Cumulative total TCP and UDP connections proxied since last restart (or SIGUSR1 reload). Resets on reload — use rate() for throughput.", "fieldConfig": { "defaults": { "color": { @@ -2680,7 +2703,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 10, + "fillOpacity": 5, "gradientMode": "none", "hideFrom": { "legend": false, @@ -2689,7 +2712,7 @@ }, "insertNulls": false, "lineInterpolation": "linear", - "lineWidth": 2, + "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" @@ -2719,17 +2742,17 @@ } ] }, - "unit": "bytes" + "unit": "short" }, "overrides": [] }, "gridPos": { "h": 8, - "w": 8, - "x": 8, - "y": 68 + "w": 11, + "x": 12, + "y": 66 }, - "id": 602, + "id": 504, "options": { "legend": { "calcs": [], @@ -2750,8 +2773,8 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "go_memstats_heap_alloc_bytes", - "legendFormat": "Heap Alloc", + "expr": "dae_tcp_connections_total", + "legendFormat": "TCP {{group}}/{{protocol}}", "refId": "A" }, { @@ -2759,119 +2782,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "go_memstats_heap_inuse_bytes", - "legendFormat": "Heap In-Use", + "expr": "dae_udp_connections_total", + "legendFormat": "UDP {{group}}/{{protocol}}", "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "process_resident_memory_bytes", - "legendFormat": "RSS", - "refId": "C" } ], - "title": "Memory", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "description": "Number of goroutines. Sudden spikes may indicate goroutine leak. Gradual growth under load is normal.", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 68 - }, - "id": 601, - "options": { - "legend": { - "calcs": [], - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "none" - } - }, - "pluginVersion": "11.5.2", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "go_goroutines", - "legendFormat": "Goroutines", - "refId": "A" - } - ], - "title": "Goroutines", + "title": "Cumulative Connection Totals (resets on reload)", "type": "timeseries" }, { @@ -2880,16 +2796,16 @@ "h": 1, "w": 24, "x": 0, - "y": 76 + "y": 74 }, - "id": 605, + "id": 600, "panels": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, - "description": "DNS query concurrency in use — saturation gauge counting client DNS queries currently being handled by dae (inc on `HandleWithResponseWriter_` entry, dec on return). No hard limit is enforced upstream; persistent non-zero values indicate slow upstream or head-of-line queueing.", + "description": "CPU time consumed per second. Value of 1.0 = one full CPU core. Values > number of cores indicate compute saturation.", "fieldConfig": { "defaults": { "color": { @@ -2904,7 +2820,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 5, + "fillOpacity": 10, "gradientMode": "none", "hideFrom": { "legend": false, @@ -2934,8 +2850,7 @@ "mode": "absolute", "steps": [ { - "color": "green", - "value": null + "color": "green" }, { "color": "red", @@ -2943,33 +2858,17 @@ } ] }, - "unit": "short" + "unit": "percentunit" }, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "In Use" - }, - "properties": [ - { - "id": "color", - "value": { - "fixedColor": "orange", - "mode": "fixed" - } - } - ] - } - ] + "overrides": [] }, "gridPos": { "h": 8, - "w": 8, + "w": 12, "x": 0, - "y": 81 + "y": 75 }, - "id": 404, + "id": 603, "options": { "legend": { "calcs": [ @@ -2979,6 +2878,103 @@ "placement": "bottom", "showLegend": true }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.5.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "rate(process_cpu_seconds_total{job=\"dae\"}[$__rate_interval])", + "legendFormat": "CPU Usage", + "refId": "A" + } + ], + "title": "CPU Usage", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Go heap memory: allocated (in-use objects) vs. in-use heap spans. Sustained growth between GC cycles indicates memory pressure.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 11, + "x": 12, + "y": 75 + }, + "id": 602, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, "tooltip": { "hideZeros": false, "mode": "multi", @@ -2992,12 +2988,30 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "dae_dns_concurrency_in_use", - "legendFormat": "In Use", + "expr": "go_memstats_heap_alloc_bytes{job=\"dae\"}", + "legendFormat": "Heap Alloc", "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "go_memstats_heap_inuse_bytes{job=\"dae\"}", + "legendFormat": "Heap In-Use", + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "expr": "process_resident_memory_bytes{job=\"dae\"}", + "legendFormat": "RSS", + "refId": "C" } ], - "title": "DNS Concurrency In Use", + "title": "Memory", "type": "timeseries" }, { @@ -3005,7 +3019,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "Number of in-flight DNS requests currently being forwarded to each upstream. Persistently high values suggest slow upstream or high concurrency.", + "description": "Number of goroutines. Sudden spikes may indicate goroutine leak. Gradual growth under load is normal.", "fieldConfig": { "defaults": { "color": { @@ -3020,7 +3034,7 @@ "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", - "fillOpacity": 20, + "fillOpacity": 10, "gradientMode": "none", "hideFrom": { "legend": false, @@ -3029,7 +3043,7 @@ }, "insertNulls": false, "lineInterpolation": "linear", - "lineWidth": 1, + "lineWidth": 2, "pointSize": 5, "scaleDistribution": { "type": "linear" @@ -3050,8 +3064,7 @@ "mode": "absolute", "steps": [ { - "color": "green", - "value": null + "color": "green" }, { "color": "red", @@ -3065,16 +3078,14 @@ }, "gridPos": { "h": 8, - "w": 8, - "x": 8, - "y": 81 + "w": 12, + "x": 0, + "y": 83 }, - "id": 408, + "id": 601, "options": { "legend": { - "calcs": [ - "lastNotNull" - ], + "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true @@ -3082,7 +3093,7 @@ "tooltip": { "hideZeros": false, "mode": "multi", - "sort": "desc" + "sort": "none" } }, "pluginVersion": "11.5.2", @@ -3092,12 +3103,12 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "dae_dns_forwarder_in_flight", - "legendFormat": "{{upstream}}", + "expr": "go_goroutines{job=\"dae\"}", + "legendFormat": "Goroutines", "refId": "A" } ], - "title": "DNS Forwarder In-Flight (per upstream)", + "title": "Goroutines", "type": "timeseries" }, { @@ -3105,7 +3116,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "GC pause duration percentiles. p99 > 10ms may cause latency spikes in proxy handling. Consider GOGC tuning if consistently high.", + "description": "GC pause duration: p50 and max (quantile 1 / 1.0 is the maximum pause in the summary window, not p99). Max > 10ms may cause latency spikes in proxy handling. Consider GOGC tuning if consistently high.", "fieldConfig": { "defaults": { "color": { @@ -3150,8 +3161,7 @@ "mode": "absolute", "steps": [ { - "color": "green", - "value": null + "color": "green" }, { "color": "red", @@ -3165,11 +3175,11 @@ }, "gridPos": { "h": 8, - "w": 6, - "x": 16, - "y": 81 + "w": 11, + "x": 12, + "y": 83 }, - "id": 604, + "id": 705, "options": { "legend": { "calcs": [], @@ -3190,7 +3200,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "histogram_quantile(0.50, sum(rate(go_gc_duration_seconds_bucket[$__rate_interval])) by (le)) * 1000", + "expr": "go_gc_duration_seconds{job=\"dae\", quantile=\"0.5\"} * 1000", "legendFormat": "p50", "refId": "A" }, @@ -3199,8 +3209,8 @@ "type": "prometheus", "uid": "${datasource}" }, - "expr": "histogram_quantile(0.99, sum(rate(go_gc_duration_seconds_bucket[$__rate_interval])) by (le)) * 1000", - "legendFormat": "p99", + "expr": "go_gc_duration_seconds{job=\"dae\", quantile=~\"1|1.0\"} * 1000", + "legendFormat": "max", "refId": "B" } ], @@ -3208,7 +3218,7 @@ "type": "timeseries" } ], - "title": "temp", + "title": "Go Runtime & Process", "type": "row" } ], @@ -3223,10 +3233,7 @@ "templating": { "list": [ { - "current": { - "text": "prometheus", - "value": "cfa1zpgywwiyod" - }, + "current": {}, "includeAll": false, "label": "Datasource", "name": "datasource", @@ -3239,14 +3246,10 @@ { "current": { "text": [ - "FC-Netflix", - "FC-TW", - "FC-HK" + "All" ], "value": [ - "FC-Netflix", - "FC-TW", - "FC-HK" + "$__all" ] }, "datasource": { @@ -3277,13 +3280,13 @@ "type": "prometheus", "uid": "${datasource}" }, - "definition": "label_values(dae_dialer_latency_last_seconds{network!~\".*DNS.*\"}, network)", + "definition": "label_values(dae_dialer_alive{network!~\".*DNS.*\"}, network)", "includeAll": false, "label": "Network", "name": "network", "options": [], "query": { - "query": "label_values(dae_dialer_latency_last_seconds{network!~\".*DNS.*\"}, network)", + "query": "label_values(dae_dialer_alive{network!~\".*DNS.*\"}, network)", "refId": "StandardVariableQuery" }, "refresh": 2, @@ -3294,13 +3297,12 @@ ] }, "time": { - "from": "now-15m", + "from": "now-1h", "to": "now" }, "timepicker": {}, "timezone": "browser", - "title": "dae Transparent Proxy-NEW", - "uid": "dae-metrics-v2", - "version": 5, + "title": "dae Transparent Proxy-v6", + "uid": "dae-metrics-v6", "weekStart": "" } From d0eebc5e79e257583afc13ed97336f675aba5cd4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 2 Oct 2026 15:21:30 +0000 Subject: [PATCH 4/6] docs(metrics): record implementation status of the PR #38 review Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- .plan/metrics/pr38-upstream-1015-review.md | 44 ++++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/.plan/metrics/pr38-upstream-1015-review.md b/.plan/metrics/pr38-upstream-1015-review.md index fcbdb4c034..49d00e5995 100644 --- a/.plan/metrics/pr38-upstream-1015-review.md +++ b/.plan/metrics/pr38-upstream-1015-review.md @@ -397,8 +397,8 @@ func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.F ## 6. 未核实事项清单 -1. #1015 rebase 到 `e3fee8f` 是否有冲突，以及冲突范围。（初步判断，数据未充分确认） -2. 修复提交能否直接 cherry-pick 到 fork main。理论上可以，因为文件相同，但未实际执行。 +1. ~~#1015 rebase 到 `e3fee8f` 是否有冲突~~ → **已核实：无冲突**，两边改动的文件零重叠，见 §7。 +2. ~~修复提交能否直接 cherry-pick~~ → **已核实**：同一个修复提交已干净应用到 fork 和 #1015 分支，见 §7。 3. 多订阅用户中同组节点重名的实际比例。（此信息/数据暂未核实完毕） 4. Caddy、acme.sh 等工具默认生成的证书和私钥权限。（此信息/数据暂未核实完毕） 5. 上游维护者对 `endpoint_prometheus_enabled` 默认值和面板存放位置的偏好。 @@ -406,7 +406,45 @@ func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.F --- -## 7. 后续思考 +## 7. 实施状态（2026-10-02 更新） + +本节记录 P0–P2 修复的落地情况。按要求，P3 项本轮不处理。 + +### 7.1 各项发现的处理结果 + +| 发现 | 状态 | 实施要点 | +|---|---|---| +| §3.1 P0-1 `link` 凭据泄露 | 已修复 | 删除 `dae_node_latency_seconds` / `dae_node_alive`；`control/node_latency.go` 回退后与上游 v2.1.1 完全一致 | +| §3.2 P0-2 重名导致 500 | 已修复 | 新增 `dialerMetricName` 按组追加 ` #N` 后缀；`promhttp` 改用 `ContinueOnError` | +| §3.3 P1-1 TCP 序列重复 | 已修复 | 改为遍历 `dialer.StandardHealthKeys()`（6 个）；删除 `DialerGroup.AliveDialerSets()`，`dialer_group.go` 回退后与上游一致 | +| §3.4 P1-2 HELP 语义 | 已修复 | HELP 改为“包含 lazy 命中”；fork 的两个旧面板已替换为 #1015 的面板（blob `21ce121`） | +| §3.5 P1-3 认证被关闭 | 已修复 | 用户名和密码必须成对配置，否则启动或 reload 报错 | +| §3.6 P1-4 示例监听地址 | 已修复 | 示例改为 `127.0.0.1:5556`，并补充暴露风险说明 | +| §3.7 P2-1 TLS 权限 | 已修复 | 改用禁止位检查：证书为 `0o022`，私钥为 `0o077`；删除两个旧函数，新增 `ValidateFilePermissionForbidden` | +| §3.8 P2-2 检查计数 | 已修复 | `CheckTotal` 只在得出结论的成功或失败分支中计数 | +| §3.9 P2-3 rebase | 本地已完成，待推送 | 与上游 6 个新提交**没有任何文件重叠**，rebase 无冲突。面板位置仍留给维护者决定 | + +### 7.2 提交 + +- fork（分支 `claude/laughing-sagan-ndu7a8`）： + - `3c02657 fix(metrics): address review findings`：18 个文件，+322/−127。不计测试和 `example.dae`，生产代码净减约 42 行。 + - `ea9f706 docs(metrics): replace stale fork dashboards ...`：仅 fork 侧。 +- 上游 PR 分支：本地分支 `upstream-prep/metrics-endpoint-clean`，已 rebase 到上游 `e3fee8f`，修复提交为 `5c9df67`。**尚未推送**，因为需要 force-push 到 `feat/metrics-endpoint-clean`。 + +### 7.3 验证 + +验证环境与 CI 等价：go1.26.0、CI 的 GOEXPERIMENT、`-tags dae_stub_ebpf`、golangci-lint v2.11.0。 + +| 检查 | fork 分支 | 上游 rebase 分支 | +|---|---|---| +| `go build ./...` 与 `go vet` | 通过 | 6 个提交**逐个**通过 | +| 相关包测试（pkg、common、component/outbound、dialer、cmd、config、control 中与 metrics 相关的部分） | 通过 | 通过 | +| golangci-lint | 0 issues | 0 issues | +| gofmt 与 `go mod tidy` | 无差异 | 无差异 | + +新增的回归测试已在**修复前的代码**上确认会失败：`TestPrometheusHandlerSurvivesCollectorError` 得到 `status=500`，`TestCheck_CountersCountOnlyVerdicts` 得到 `total=3`。修复后两者都通过。 + +## 8. 后续思考 - **Q1：** 如果 `/metrics` 已经被暴露过一段时间，凭据可能已经写入 Prometheus 或远程存储。是否需要提醒已部署 fork 的用户轮换节点密码或 UUID，并清理 TSDB 中的 `dae_node_*` 序列？ - **Q2：** 健康指标的 `network` 维度应该直接对齐 v2.1.1 的“健康域”（`tcp`、`dns_udp`、`data_udp`），还是继续沿用 `tcp4(DNS)` 这类旧字符串以保证兼容？这个选择会影响上游长期的指标契约。 From 3006758ae01a290dce673212cea8940926bad3c5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 01:50:10 +0000 Subject: [PATCH 5/6] docs(metrics): base the fork dashboard on v6 with the review fixes Rename it back to "dae Transparent Proxy-Grafana_dashboard.json" and use the same revision as daeuniverse/dae#1015: the sanitized v6 export (no id/version, datasource or group selection), with the network variable listing udp4(DNS)/udp6(DNS), the health check success rate showing No data when no checks ran, and the health check descriptions matching the verdict-only counters. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- ...json => dae Transparent Proxy-Grafana_dashboard.json} | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) rename .plan/metrics/{dae_Transparent_Proxy-Grafana_dashboard.json => dae Transparent Proxy-Grafana_dashboard.json} (98%) diff --git a/.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json b/.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json similarity index 98% rename from .plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json rename to .plan/metrics/dae Transparent Proxy-Grafana_dashboard.json index 21ce121539..4b8b6486a8 100644 --- a/.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json +++ b/.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json @@ -718,6 +718,7 @@ "type": "prometheus", "uid": "${datasource}" }, + "description": "时间范围内所选 network（$network）健康检查的成功率（排除 block/direct 组），只统计得出结论的检查（成功或失败；跳过与探测基础设施故障不计入）。只有 tcp4/tcp6 与 udp4(DNS)/udp6(DNS) 会被主动探测；数据 UDP（udp4/udp6）没有主动检查，此处显示 No data。", "fieldConfig": { "defaults": { "color": { @@ -779,7 +780,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "clamp_min(1 - sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) / clamp_min(sum(increase(dae_health_check_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])), 1), 0)", + "expr": "clamp_min(1 - sum(increase(dae_health_check_failure_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) / (sum(increase(dae_health_check_total{network=\"$network\", group=~\"$group\", group!~\"block|direct\"}[$__range])) > 0), 0)", "legendFormat": "__auto", "range": true, "refId": "A" @@ -793,7 +794,7 @@ "type": "prometheus", "uid": "${datasource}" }, - "description": "时间范围内所选 network（$network）健康检查失败的累计次数（排除 block/direct 组）。失败包括连接错误，以及 tcp6/udp6 检查目标无 IPv6 地址时的 ErrNoApplicableIP（每次检查都计为失败，节点 alive=0）。尚未完成首次探测的节点（0ms）没有检查记录，不计入。", + "description": "时间范围内所选 network（$network）健康检查失败的累计次数（排除 block/direct 组）。失败包括连接错误，以及 tcp6/udp6(DNS) 检查目标无 IPv6 地址时的 ErrNoApplicableIP（每次检查都计为失败，节点 alive=0）；跳过与探测基础设施故障不计入。数据 UDP（udp4/udp6）没有主动检查，恒为 0。尚未完成首次探测的节点（0ms）没有检查记录，不计入。", "fieldConfig": { "defaults": { "color": { @@ -3280,13 +3281,13 @@ "type": "prometheus", "uid": "${datasource}" }, - "definition": "label_values(dae_dialer_alive{network!~\".*DNS.*\"}, network)", + "definition": "label_values(dae_dialer_alive, network)", "includeAll": false, "label": "Network", "name": "network", "options": [], "query": { - "query": "label_values(dae_dialer_alive{network!~\".*DNS.*\"}, network)", + "query": "label_values(dae_dialer_alive, network)", "refId": "StandardVariableQuery" }, "refresh": 2, From b0fd81039b1d85682ba6d367460fd949dc867101 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 01:51:08 +0000 Subject: [PATCH 6/6] docs(metrics): record the #1015 push and dashboard revision Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BYykYRdT6p8qLLe4RMiV7s --- .plan/metrics/pr38-upstream-1015-review.md | 24 ++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/.plan/metrics/pr38-upstream-1015-review.md b/.plan/metrics/pr38-upstream-1015-review.md index 49d00e5995..f8c4e3a37b 100644 --- a/.plan/metrics/pr38-upstream-1015-review.md +++ b/.plan/metrics/pr38-upstream-1015-review.md @@ -422,14 +422,17 @@ func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.F | §3.6 P1-4 示例监听地址 | 已修复 | 示例改为 `127.0.0.1:5556`，并补充暴露风险说明 | | §3.7 P2-1 TLS 权限 | 已修复 | 改用禁止位检查：证书为 `0o022`，私钥为 `0o077`；删除两个旧函数，新增 `ValidateFilePermissionForbidden` | | §3.8 P2-2 检查计数 | 已修复 | `CheckTotal` 只在得出结论的成功或失败分支中计数 | -| §3.9 P2-3 rebase | 本地已完成，待推送 | 与上游 6 个新提交**没有任何文件重叠**，rebase 无冲突。面板位置仍留给维护者决定 | +| §3.9 P2-3 rebase 与面板 | 已完成并推送 | 与上游 6 个新提交**没有任何文件重叠**，rebase 无冲突。面板保留在 `.plan/metrics/`，内容见 §7.4 | ### 7.2 提交 - fork（分支 `claude/laughing-sagan-ndu7a8`）： - `3c02657 fix(metrics): address review findings`：18 个文件，+322/−127。不计测试和 `example.dae`，生产代码净减约 42 行。 - `ea9f706 docs(metrics): replace stale fork dashboards ...`：仅 fork 侧。 -- 上游 PR 分支：本地分支 `upstream-prep/metrics-endpoint-clean`，已 rebase 到上游 `e3fee8f`，修复提交为 `5c9df67`。**尚未推送**，因为需要 force-push 到 `feat/metrics-endpoint-clean`。 +- 上游 PR 分支 `feat/metrics-endpoint-clean`（daeuniverse/dae#1015 的 head）： + - 用 `--force-with-lease`（基准 `8573436`）推送：`8573436...5c9df67`。5 个原提交已 rebase 到上游 `e3fee8f`，修复提交为 `5c9df67`。 + - 面板修订作为快进提交追加：`5c9df67..8502e56`。 +- fork 分支：`3006758` 依据 v6 修订面板，并把文件名改回 `dae Transparent Proxy-Grafana_dashboard.json`。 ### 7.3 验证 @@ -444,6 +447,23 @@ func ValidateFilePermissionForbidden(path string, fi os.FileInfo, forbidden os.F 新增的回归测试已在**修复前的代码**上确认会失败：`TestPrometheusHandlerSurvivesCollectorError` 得到 `status=500`，`TestCheck_CountersCountOnlyVerdicts` 得到 `total=3`。修复后两者都通过。 +### 7.4 Grafana 面板（基于 v6 修订） + +- 基准文件：用户上传的 `dae_Transparent_Proxy-v6-1790143652539.json`。经深度比对，#1015 原有面板正是 v6 去除环境信息后的版本，差异只有 `id`、`version`、数据源 uid 和组选择这 4 处。v6 本身不使用 `dae_node_*`，DNS 缓存公式也已按“hit 含 lazy”编写。 +- 修订内容（文本 diff 共 5 行）： + - **D1**：`network` 变量去掉 `network!~".*DNS.*"` 过滤。重复的 `tcp4(DNS)` 序列已不存在，这个过滤器现在只会隐藏唯一被主动探测的 `udp4(DNS)`/`udp6(DNS)`。 + - **D2**：健康检查成功率的分母改为 `(sum(...) > 0)`。数据 UDP 没有检查，原写法会显示误导性的 100%，现在显示 No data。 + - **D3**：面板 202 与 203 的描述改为与“只统计得出结论的检查”一致。 +- 文件位置：fork 侧为 `.plan/metrics/dae Transparent Proxy-Grafana_dashboard.json`；#1015 侧保留 `.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json`。两者内容逐字节相同。 +- 校验结果： + - 程序化比对确认“结果等于对 v6 去环境化后再应用 D1–D3”； + - 引用的 27 个 `dae_*` 指标全部由修复后的采集器导出； + - 文件中不出现 `dae_node_`、`tcp4(DNS)` 或 `link`。 + +### 7.5 #1015 说明评论 + +本会话无法直接在 daeuniverse/dae 发评论：`add_repo` 因同名仓库目录冲突被拒，这是检出布局的限制，与权限无关。评论正文已交给仓库所有者手动发布。 + ## 8. 后续思考 - **Q1：** 如果 `/metrics` 已经被暴露过一段时间，凭据可能已经写入 Prometheus 或远程存储。是否需要提醒已部署 fork 的用户轮换节点密码或 UUID，并清理 TSDB 中的 `dae_node_*` 序列？