diff --git a/services/libs/tinybird/README.md b/services/libs/tinybird/README.md index d015832333..e7d5383886 100644 --- a/services/libs/tinybird/README.md +++ b/services/libs/tinybird/README.md @@ -339,6 +339,34 @@ tb sql "SELECT count() FROM activities_backup FINAL" If both pairs match, the backup is **logically consistent** with the source dataset. +--- + +## Health Score v2 Configuration + +### Excluding Repositories from Health Score + +Repositories can be marked as excluded from health scoring by setting the `repositories.excluded` flag to `1`. This is the official mechanism for marking experimental, sandbox, meta, or otherwise out-of-scope repositories. + +**Use cases for exclusion:** +- `.github` meta repositories (configuration-only, no product code) +- Archived experiments or prototypes +- Forks created for testing purposes +- Any repository where activity metrics are meaningless or undesired + +**Behavior when excluded:** +- Health Score v2 categorizes excluded repos as `unavailable` for responsiveness scoring (never scored 0 for missing PR/issue data) +- Other category signals continue to be scored normally +- Project-level health scores (when a project contains multiple repos) exclude the marked repo from aggregation + +**Setting the flag:** +```sql +UPDATE repositories SET excluded = TRUE WHERE url = 'https://github.com/org/repo-meta'; +``` + +The flag is read by `project_insights_copy.pipe` and `health_score_v2_raw_inputs_snapshot.pipe` as part of their graceful-degradation and audit logic. + +--- + ## Glossary - **CDP (Community Data Platform)**: Customer data operations and management pipelines diff --git a/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource index daaa218dc9..b94fb1bcbb 100644 --- a/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource @@ -38,7 +38,8 @@ SCHEMA > `prMergeAvailable` UInt8, `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource index 84e81a7998..054f606b1a 100644 --- a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource @@ -2,13 +2,14 @@ DESCRIPTION > - `health_score_v2_lifecycle_ds` holds the per-repo Lifecycle state for Health Score v2. Populated by `health_score_v2_lifecycle.pipe`. - `repoUrl` is the repository URL — the join key back to `repositories`. - - `lifecycleLabelV2` — one of active/stable/declining/abandoned/archived, or NULL when the - repo has no usable activity signal at all (2026-07-24, IN-1196 — see + - `lifecycleLabelV2` — one of active/stable/declining/inert/abandoned/archived, or NULL when + the repo has no usable activity signal at all (2026-07-24, IN-1196 — see health_score_v2_lifecycle.pipe for the exact condition). SCHEMA > `repoUrl` String, - `lifecycleLabelV2` Nullable(String) + `lifecycleLabelV2` Nullable(String), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource index 46def9226d..6ed97564f8 100644 --- a/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource @@ -33,7 +33,8 @@ SCHEMA > `medianIssueResponseS` Nullable(Float64), `isGerrit` UInt8, `isExcluded` UInt8, - `coveredWeight` UInt8 + `coveredWeight` UInt8, + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource new file mode 100644 index 0000000000..d3d3b97aa9 --- /dev/null +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -0,0 +1,69 @@ +DESCRIPTION > + - `health_score_v2_raw_inputs_snapshot_ds` captures raw measured inputs (not computed scores) for + Health Score v2 at periodic intervals for validation and historical analysis. + - Populated by `health_score_v2_raw_inputs_snapshot.pipe` (append-mode, monthly cadence). + - Each row is keyed by `repoUrl + snapshotDate + methodologyVersion` and captures all the atomic + measurement signals used to compute the three health score categories: + - Maintainer signals: bus factor (curated maintainer count), observed review/merge actors, + organization diversity (distinct org count), PR/issue response times (medians). + - Security signals: open CVE counts (critical/high/moderate), OpenSSF Scorecard score, security + practices flags (policy/branch-protection state), vulnerable dependencies count. + - Development signals: release recency/cadence, commit counts (6m windows), issue counts/closure + times, PR counts/merge times. + - Responsiveness/lifecycle signals: unanswered aged issue/PR counts and 18mo opened totals + (the per-repo unanswered count/ratio the compliance spec item 6 requires, and the historical + input the abandoned-threshold-sweep validation analysis runs against). Definitions mirror + health_score_v2_lifecycle.pipe: open, no non-author response, aged 90+ days, opened within + the 18mo window. + - Does NOT include computed category/overall scores — those are derived from these raw inputs via + the scoring logic in the respective pipes. This datasource is for tracing where scores come from. + - snapshotId = `toStartOfInterval(now(), INTERVAL 1 day)` at pipe execution time. + - Partitioned by year(snapshotDate) and month(snapshotDate) for efficient time-based queries. + - TTL set to 24 months (enough for trend analysis and validation studies). + - ReplacingMergeTree keyed on (snapshotDate, repoUrl, methodologyVersion): a same-day manual + re-run of the copy converges to one row per key after background merges instead of + accumulating duplicates (UNIQUE_KEY is not a thing in Tinybird — attempted and reverted in + cdf6cf263). Validation queries must still read with FINAL (or dedupe via argMax/LIMIT BY) + since replacement is asynchronous. + +SCHEMA > + `repoUrl` String, + `snapshotDate` DateTime, + `methodologyVersion` String, + `busFactorCuratedCount` Nullable(UInt64), + `busFactorObservedActorsCount` Nullable(UInt64), + `orgDiversityCount` Nullable(UInt64), + `medianPrResponseSeconds` Nullable(Float64), + `medianIssueResponseSeconds` Nullable(Float64), + `openCriticalVulns` Nullable(UInt64), + `openHighVulns` Nullable(UInt64), + `openModerateVulns` Nullable(UInt64), + `scorecardScore` Nullable(String), + `securityPolicyEnabled` Nullable(UInt8), + `branchProtectionEnabled` Nullable(UInt8), + `branchProtectionRequiredReviews` Nullable(Int32), + `branchProtectionRequiresStatusChecks` Nullable(UInt8), + `branchProtectionAllowsForcePush` Nullable(UInt8), + `vulnerableDeps` Nullable(UInt64), + `daysSinceLatestRelease` Nullable(Int64), + `daysBetweenRecentReleases` Nullable(Int64), + `commitsLast6m` Nullable(UInt64), + `commitsPrior6m` Nullable(UInt64), + `lastCommitAt` Nullable(DateTime64(3)), + `issuesClosedLast12m` Nullable(UInt64), + `issuesOpenedLast12m` Nullable(UInt64), + `medianIssueCloseSeconds` Nullable(Float64), + `prsMergedLast12m` Nullable(UInt64), + `prsClosedUnmergedLast12m` Nullable(UInt64), + `medianPrMergeSeconds` Nullable(Float64), + `excluded` Nullable(UInt8), + `trackedPackageCount` Nullable(UInt64), + `unansweredIssuesAged90d` Nullable(UInt64), + `unansweredPrsAged90d` Nullable(UInt64), + `issuesOpenedLast18m` Nullable(UInt64), + `prsOpenedLast18m` Nullable(UInt64) + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY toYYYYMM(snapshotDate) +ENGINE_SORTING_KEY (snapshotDate, repoUrl, methodologyVersion) +ENGINE_TTL snapshotDate + INTERVAL 24 MONTH diff --git a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource index 3414e2aa07..c935def4fa 100644 --- a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource @@ -13,15 +13,24 @@ DESCRIPTION > across unrelated packages), supply chain integrity (hardcoded 0 — provenance/2FA data not yet piped). - `developmentActivityScoreV2` (0-25) — release cadence, commit activity, issue resolution, PR merge health. - `healthScoreV2` (0-100) is the sum of the three categories above, clamped to 100. - - `lifecycleLabelV2` — per-repo lifecycle state (active/stable/declining/abandoned/archived), computed - via the spec's decision tree (archived flag > abandoned > declining > stable > active, first match - wins), or NULL when the repo has zero commits and zero issues/PRs in every window checked (2026-07-24, - IN-1196 — no usable activity signal at all). Project-level rollup uses best-state-wins precedence in - project_insights_copy.pipe. + - `lifecycleLabelV2` — per-repo lifecycle state (active/stable/declining/inert/abandoned/archived), + computed via the spec's decision tree (archived flag > abandoned > inert > declining > stable > + active, first match wins), or NULL when the repo has zero commits and zero issues/PRs in every window + checked (2026-07-24, IN-1196 — no usable activity signal at all). Project-level rollup uses + best-state-wins precedence in project_insights_copy.pipe. - `impactScore` (0-100) — MAX(packages.impact) * 100 over packages published by this repo. NULL when the repo has no linked packages. - - No graceful-degradation/signal-coverage redistribution yet — a repo with zero signal in a category - scores 0 there rather than having points redistributed from available categories. + - Graceful degradation (spec Layer 1+2): per-category pipes emit NULL when covered sub-signal weight + is <40% of that category's max pts (Layer 1); the aggregate health_score_v2 applies Layer 2 rescaling + (`SUM(available categories) * (100 / SUM(available category weights))`) so only actually-available + scores are summed. If all three categories are unavailable, healthScoreV2 itself is NULL (see + health_score_v2.pipe DESCRIPTION for the full spec). + - Known temporary gap: security-practices score's spec-stated max is 8 points (includes +1 for + `security_contact_email IS NOT NULL` flag), but no `security_contact_email` column exists yet in + the GitHub enrichment datasource, so the current maximum achievable is 7 points. This gap is + intentional and temporary — do not fabricate the missing column or change scoring. When the email + field becomes available in repos, security-practices score will automatically rise to its full 8-point + potential. SCHEMA > `repoUrl` String, @@ -30,7 +39,8 @@ SCHEMA > `developmentActivityScoreV2` Nullable(UInt8), `healthScoreV2` Nullable(UInt8), `lifecycleLabelV2` Nullable(String), - `impactScore` Nullable(UInt8) + `impactScore` Nullable(UInt8), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource index 09fee28c01..2a33f852a2 100644 --- a/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource @@ -37,7 +37,8 @@ SCHEMA > `branchProtectionAllowsForcePush` Nullable(UInt8), `dependencyHealthScore` UInt8, `dependencyHealthAvailable` UInt8, - `vulnerableDeps` Nullable(UInt64) + `vulnerableDeps` Nullable(UInt64), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource index f6bf42ff0e..c0a380bbec 100644 --- a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource @@ -71,7 +71,8 @@ SCHEMA > `prMergeAvailable` Nullable(UInt8), `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource b/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource index 00efad93af..dee8306698 100644 --- a/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource +++ b/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource @@ -12,7 +12,8 @@ DESCRIPTION > - Maintainer signals: `busFactorScore` (avg), `busFactorAvailable` (max), `busFactorCount` (max), `orgDiversityScore` (avg), `orgDiversityAvailable` (max), `orgCount` (max), `responsivenessScore` (avg), `responsivenessAvailable` (max), `medianPrResponseS` (avg), - `medianIssueResponseS` (avg), `isGerrit` (min), `isExcluded` (min). + `medianIssueResponseS` (avg), `isGerrit` (min), `isExcluded` (min), + `busFactorScoreActivityWeightedMean` (activity-weighted mean of bus-factor score across repos). - Security signals: `openVulnScore` (avg), `openCriticals`/`openHighs`/`openModerates` (max), `scorecardScorePts` (avg), `scorecardAvailable` (max), `scorecardScore` (avg, cast from the raw String), `securityPracticesScore` (avg), `securityPracticesAvailable` (max), @@ -75,7 +76,8 @@ SCHEMA > `prMergeAvailable` Nullable(UInt8), `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `busFactorScoreActivityWeightedMean` Nullable(Float64) ENGINE MergeTree ENGINE_SORTING_KEY slug, projectId diff --git a/services/libs/tinybird/datasources/vulnerabilities.datasource b/services/libs/tinybird/datasources/vulnerabilities.datasource index 684d5c69be..1a55220ef3 100644 --- a/services/libs/tinybird/datasources/vulnerabilities.datasource +++ b/services/libs/tinybird/datasources/vulnerabilities.datasource @@ -7,7 +7,7 @@ DESCRIPTION > - `cveIds` is an array of CVE identifiers associated with this vulnerability (empty array default). - `ghsaIds` is an array of GitHub Security Advisory identifiers. - `otherIds` is an array of other vulnerability identifiers. - - `severity` is the vulnerability severity level (UNKNOWN, LOW, MODERATE, HIGH, CRITICAL). + - `severity` is the vulnerability severity level (UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL). - `cvssScore` is the raw CVSS score (nullable, null when score is unavailable). - `summary` is a short description of the vulnerability. - `details` contains detailed information about the vulnerability. diff --git a/services/libs/tinybird/pipes/health_score_v2.pipe b/services/libs/tinybird/pipes/health_score_v2.pipe index 96812a9021..c2ce47bd11 100644 --- a/services/libs/tinybird/pipes/health_score_v2.pipe +++ b/services/libs/tinybird/pipes/health_score_v2.pipe @@ -40,7 +40,8 @@ SQL > lifecycleLabelV2, if( impactScoreRaw IS NULL, NULL, toNullable(toUInt8(round(least(impactScoreRaw, 100)))) - ) AS impactScore + ) AS impactScore, + '2.0.0' AS methodologyVersion FROM ( SELECT diff --git a/services/libs/tinybird/pipes/health_score_v2_development.pipe b/services/libs/tinybird/pipes/health_score_v2_development.pipe index e9e63bbd78..3f79e90194 100644 --- a/services/libs/tinybird/pipes/health_score_v2_development.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_development.pipe @@ -36,7 +36,8 @@ SQL > prMergeAvailable, merged12m, closedUnmerged12m, - medianMergeS + medianMergeS, + '2.0.0' AS methodologyVersion FROM ( SELECT diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 60e2837227..f99eecc1b3 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -1,6 +1,7 @@ DESCRIPTION > Per-repo Lifecycle state, computed via the spec's decision tree (first match wins): - archived (repos.archived) > abandoned (no commits in 18mo AND no issue/PR activity in 18mo) > + archived (repos.archived) > abandoned (an issue or PR open and unanswered for 90+ days AND no + commits in 12mo) > inert (no commits in 18mo AND no issue/PR activity in 18mo) > declining (commits down >50% vs prior 6mo AND issues opened up vs prior 6mo) > stable (release within 12mo, <50 open issues, no open critical vulns, commits down >50% vs prior 6mo) > active (fallback) — but NULL overrides all of the above when there is no usable signal at all (see @@ -27,13 +28,51 @@ DESCRIPTION > commitsLast6m=0 (down from 1895 prior), because it still has 3 real PRs in the 18mo window — real activity, just not enough to clear any of the other branches' thresholds. That's a legitimate 'active' classification under the existing spec, not the bug being fixed here. + - `r.lastCommitAt IS NULL` added to the NULL-guard condition (2026-08-11, per Joana's PR + review): the guard as written (zero commits in both 6mo windows + zero issue/PR activity in + 18mo) was a strict superset of the `inert` branch's own condition (no commits in 18mo + zero + issue/PR activity), since zero commits across the trailing 12mo already implies + `lastCommitAt` is either NULL or older than 18mo. That made the guard fire first on every repo + that could otherwise have matched `inert`, so `inert` was dead code — never actually emitted. + Restricting the guard to the `lastCommitAt IS NULL` case (matching the blocknetdx/blocknet + example this branch was originally written for) lets repos with a known, stale last-commit + timestamp fall through into the real decision tree, where `inert` can now actually fire. NODE health_score_v2_lifecycle_calc +DESCRIPTION > + - Precedence (first match wins): archived > abandoned > inert > declining > stable > active. + - Threshold note (provisional, pending validation analysis): abandoned requires at least one + issue OR PR that has sat unanswered for 90+ days, plus a 12-month no-commit gate. The age + gate is a single threshold, NOT a closed 90-180 band: a band would let a repo exit + 'abandoned' once its ignored issues age past the upper bound (they still block 'inert' via + issuesInWindow18m, so the repo would flip back to 'active'). The spec's "~90-180 days" is + the tuning range for this one threshold — the abandoned-threshold-sweep validation analysis + picks the final value. The no-commit gate requires BOTH commit signals to be silent: zero + authored-commit activity in both trailing 6mo windows (covers repos where lastCommitAt is + NULL/stale from the upstream `repos` data gap but real commits exist in activityRelations) + AND repos.lastCommitAt NULL or older than 12mo (covers the reverse gap). Both signals are + last-commit-by-anyone, not maintainer-specific activity; whether maintainer review/comment + activity should also block 'abandoned' is an open question on the PR. + - Inert: lastCommitAt older than 18mo AND zero issues/PRs opened in the 18mo window. + lastCommitAt IS NULL deliberately does NOT qualify: a NULL-with-no-signal repo is already + caught by the NULL guard above, and a NULL-with-recent-commits repo (upstream data gap in + `repos`) must not be mislabeled inert just because one timestamp field is missing. + - Unanswered issue: opened in the 18mo window, no non-author comment, still open, and has been + open 90+ days. + - Unanswered PR: opened in the 18mo window, no reviewer activity (reviewedAt IS NULL), still + open (closedAt IS NULL), and has been open 90+ days. Mirrors the issue-side signal using + pull_requests_analyzed.reviewedAt, which is already the first-response timestamp for PRs + (first review / changes-requested / Gerrit patchset approval) — no new datasource field needed. + - Declining: commit count down >50% vs prior 6mo AND issues opened up vs prior 6mo. + - Stable: release within 12mo, <50 open issues, no open critical vulns, AND commits down + >50% vs prior 6mo. + SQL > SELECT r.url AS repoUrl, if( r.archived != 1 + AND r.lastCommitAt IS NULL AND coalesce(c.commitsLast6m, 0) = 0 AND coalesce(c.commitsPrior6m, 0) = 0 AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, @@ -42,9 +81,14 @@ SQL > multiIf( r.archived = 1, 'archived', - (r.lastCommitAt < now() - INTERVAL 18 MONTH) - AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, + (coalesce(w.unansweredAged90d, 0) > 0 OR coalesce(pw.unansweredAged90d, 0) > 0) + AND coalesce(c.commitsLast6m, 0) = 0 + AND coalesce(c.commitsPrior6m, 0) = 0 + AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', + r.lastCommitAt < now() - INTERVAL 18 MONTH + AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, + 'inert', coalesce(c.commitsLast6m, 0) < coalesce(c.commitsPrior6m, 0) * 0.5 AND coalesce(w.issuesOpenedLast6m, 0) > coalesce(w.issuesOpenedPrior6m, 0), 'declining', @@ -56,7 +100,8 @@ SQL > 'active' ) ) - ) AS lifecycleLabelV2 + ) AS lifecycleLabelV2, + '2.0.0' AS methodologyVersion FROM ( SELECT base.url AS url, base.archived AS archived, rc.lastCommitAt AS lastCommitAt @@ -95,7 +140,13 @@ SQL > countIf( openedAt <= now() - INTERVAL 6 MONTH AND openedAt > now() - INTERVAL 12 MONTH ) AS issuesOpenedPrior6m, - countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m + countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m, + countIf( + openedAt > now() - INTERVAL 18 MONTH + AND respondedInSeconds IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d FROM issues_analyzed GROUP BY channel ) AS w @@ -107,6 +158,21 @@ SQL > GROUP BY channel ) AS p ON p.repoUrl = r.url + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + openedAt > now() - INTERVAL 18 MONTH + AND reviewedAt IS NULL + AND approvedAt IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d + FROM pull_requests_analyzed + GROUP BY channel + ) AS pw + ON pw.repoUrl = r.url LEFT JOIN ( SELECT repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals diff --git a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe index 1c2dc02eaa..73dfc398cb 100644 --- a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe @@ -28,6 +28,10 @@ DESCRIPTION > repo within the same trailing 12-month window used elsewhere in this pipe, not just an unexpired role record. This was the direct cause of dead repos hitting healthScoreV2=100 after Layer 2 rescaling (see LFX/cdp/health-score-v2-renormalization-bug.md). + - Bus factor enhancement (IN-1226): Added parallel count of observed review/merge actors in the + trailing 12-month window. Final busFactorCount = max(curated_maintainers_count, observed_actors_count). + busFactorAvailable = 1 if either source has data. This ensures teams relying on informal review + duties aren't penalized for lack of formal role curation. NODE health_score_v2_maintainer_calc SQL > @@ -50,7 +54,8 @@ SQL > base.medianIssueResponseS AS medianIssueResponseS, base.isGerrit AS isGerrit, base.isExcluded AS isExcluded, - coveredWeight AS coveredWeight + coveredWeight AS coveredWeight, + '2.0.0' AS methodologyVersion FROM ( SELECT @@ -77,16 +82,18 @@ SQL > allRepos.repoUrl AS repoUrl, allRepos.isGerrit AS isGerrit, allRepos.isExcluded AS isExcluded, - bf.busFactorCount AS busFactorCount, - bf.repoUrl != '' AS busFactorAvailable, + greatest( + coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0) + ) AS busFactorCount, + (bf.repoUrl != '' OR obs.repoUrl != '') AS busFactorAvailable, multiIf( - bf.busFactorCount >= 5, + busFactorCount >= 5, 18, - bf.busFactorCount >= 3, + busFactorCount >= 3, 15, - bf.busFactorCount = 2, + busFactorCount = 2, 6, - bf.busFactorCount = 1, + busFactorCount = 1, 3, 0 ) AS busFactorScore, @@ -181,12 +188,14 @@ SQL > ) AS allRepos LEFT JOIN ( - SELECT mr.repoUrl AS repoUrl, count(DISTINCT mr.memberId) AS busFactorCount + SELECT + mr.repoUrl AS repoUrl, + count(DISTINCT mr.memberId) AS curatedBusFactorCount FROM maintainers_roles_copy_ds mr INNER JOIN ( SELECT DISTINCT memberId, channel AS repoUrl - FROM activityRelations + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH ) AS recentActivity ON recentActivity.memberId = mr.memberId @@ -197,10 +206,29 @@ SQL > GROUP BY mr.repoUrl ) AS bf ON bf.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, count(DISTINCT memberId) AS observedActorsCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE + timestamp > now() - INTERVAL 12 MONTH + AND memberId != '' + AND ( + type = 'pull_request-reviewed' + OR type = 'pull_request-merged' + OR type = 'merge_request-review-approved' + OR type = 'merge_request-merged' + OR type = 'patchset_approval-created' + OR type = 'merge_request-review-unapproved' + OR type = 'merge_request-review-commented' + ) + GROUP BY channel + ) AS obs + ON obs.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT channel AS repoUrl, count(DISTINCT organizationId) AS orgCount - FROM activityRelations + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH AND organizationId != '' GROUP BY channel ) AS od diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe new file mode 100644 index 0000000000..54a3d0cbfd --- /dev/null +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -0,0 +1,294 @@ +DESCRIPTION > + - Periodic snapshot of raw measured inputs (not computed scores) for Health Score v2 validation + and historical trend analysis. Runs monthly (1st of each month) and appends one row per repo. + - Each execution creates a snapshotDate = toStartOfInterval(now(), INTERVAL 1 day), tagged with + the current methodologyVersion ('2.0.0'). + - Sources the same base data as the individual category pipes (health_score_v2_maintainer, + health_score_v2_security, health_score_v2_development) but captures RAW measured values only — + not computed scores. This allows downstream validation queries to reconstruct or audit scoring + without needing access to the category pipes' intermediate logic. + - Append-mode COPY, partitioned by date, with 24-month TTL. The target is a + ReplacingMergeTree keyed on (snapshotDate, repoUrl, methodologyVersion), so a same-day + re-run converges instead of duplicating — see the datasource description. + - Also captures the per-repo unanswered issue/PR counts (open, no non-author response, aged + 90+ days, opened within 18mo — same definitions as health_score_v2_lifecycle.pipe) plus the + 18mo opened totals, giving the unanswered ratio required by compliance spec item 6 and the + historical input for the abandoned-threshold-sweep validation analysis. + - Issue/PR subqueries scope their population by `openedAt > 12 MONTH` first, then count/quantile + within that set (2026-08-11, per Joana's PR review) — matching `health_score_v2_development.pipe` + exactly, so the snapshot's raw counts and medians reconstruct the same population the development + category score was computed from. The prior version scoped `opened12m`/quantiles independently + per-column (via a `FILTER`/inline date check on `closedAt`/`mergedAt` instead of `openedAt`), + which measured a different, unbounded-by-openedAt population than the category pipe — the two + could never be reconciled. + +TAGS "Validation", "Health Score v2" + +NODE health_score_v2_raw_inputs_snapshot_calc +DESCRIPTION > + Combines raw input signals from all three categories (maintainer, security, development) + and tags with snapshot metadata. Inlined as one node (rather than three joined nodes) + because Tinybird cannot resolve column references across multiple independently-defined + subqueries once inlined together — each category's raw-input subquery is joined directly + against a single shared repo base here, matching the pattern already proven in the + single-node health_score_v2_maintainer/_security/_development pipes. + +SQL > + SELECT + allRepos.repoUrl AS repoUrl, + toStartOfInterval(now(), INTERVAL 1 day) AS snapshotDate, + '2.0.0' AS methodologyVersion, + bf.curatedBusFactorCount AS busFactorCuratedCount, + obs.observedActorsCount AS busFactorObservedActorsCount, + od.orgCount AS orgDiversityCount, + r.medianPrResponseS AS medianPrResponseSeconds, + ir.medianIssueResponseS AS medianIssueResponseSeconds, + coalesce(v.openCriticals, 0) AS openCriticalVulns, + coalesce(v.openHighs, 0) AS openHighVulns, + coalesce(v.openModerates, 0) AS openModerateVulns, + repoMeta.scorecardScore, + repoMeta.securityPolicyEnabled, + repoMeta.branchProtectionEnabled, + repoMeta.branchProtectionRequiredReviews, + repoMeta.branchProtectionRequiresStatusChecks, + repoMeta.branchProtectionAllowsForcePush, + deps.vulnerableDeps, + rl.daysSinceLatest AS daysSinceLatestRelease, + rl.daysBetweenRecent AS daysBetweenRecentReleases, + c.commitsLast6m, + c.commitsPrior6m, + rc.lastCommitAt, + w.closed12m AS issuesClosedLast12m, + w.opened12m AS issuesOpenedLast12m, + w.medianCloseS AS medianIssueCloseSeconds, + p.merged12m AS prsMergedLast12m, + p.closedUnmerged12m AS prsClosedUnmergedLast12m, + p.medianMergeS AS medianPrMergeSeconds, + rf.excluded, + pkgs.trackedPackageCount, + uw.unansweredAged90d AS unansweredIssuesAged90d, + upw.unansweredAged90d AS unansweredPrsAged90d, + uw.openedLast18m AS issuesOpenedLast18m, + upw.openedLast18m AS prsOpenedLast18m + FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE deletedAt IS NULL) AS allRepos + LEFT JOIN + (SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL) AS rf + ON rf.url = allRepos.repoUrl + LEFT JOIN + ( + SELECT mr.repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount + FROM maintainers_roles_copy_ds mr + INNER JOIN + ( + SELECT DISTINCT memberId, channel AS repoUrl + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE timestamp > now() - INTERVAL 12 MONTH + ) AS recentActivity + ON recentActivity.memberId = mr.memberId + AND recentActivity.repoUrl = mr.repoUrl + WHERE mr.endDate = '1970-01-01 00:00:00' OR mr.endDate > now() - INTERVAL 12 MONTH + GROUP BY mr.repoUrl + ) AS bf + ON bf.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, count(DISTINCT memberId) AS observedActorsCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE + timestamp > now() - INTERVAL 12 MONTH + AND memberId != '' + AND ( + type = 'pull_request-reviewed' + OR type = 'pull_request-merged' + OR type = 'merge_request-review-approved' + OR type = 'merge_request-merged' + OR type = 'patchset_approval-created' + OR type = 'merge_request-review-unapproved' + OR type = 'merge_request-review-commented' + ) + GROUP BY channel + ) AS obs + ON obs.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, count(DISTINCT organizationId) AS orgCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE timestamp > now() - INTERVAL 12 MONTH AND organizationId != '' + GROUP BY channel + ) AS od + ON od.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, quantile(0.5)(reviewedInSeconds) AS medianPrResponseS + FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH AND reviewedInSeconds IS NOT NULL + GROUP BY channel + ) AS r + ON r.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, quantile(0.5)(respondedInSeconds) AS medianIssueResponseS + FROM issues_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH AND respondedInSeconds IS NOT NULL + GROUP BY channel + ) AS ir + ON ir.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + repoUrl, + countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, + countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, + countIf(status = 'OPEN' AND severity = 'MEDIUM') AS openModerates + FROM vulnerabilities FINAL + GROUP BY repoUrl + ) AS v + ON v.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + url AS repoUrl, + argMax(scorecardScore, updatedAt) AS scorecardScore, + argMax(securityPolicyEnabled, updatedAt) AS securityPolicyEnabled, + argMax(branchProtectionEnabled, updatedAt) AS branchProtectionEnabled, + argMax(branchProtectionRequiredReviews, updatedAt) AS branchProtectionRequiredReviews, + argMax( + branchProtectionRequiresStatusChecks, updatedAt + ) AS branchProtectionRequiresStatusChecks, + argMax(branchProtectionAllowsForcePush, updatedAt) AS branchProtectionAllowsForcePush + FROM repos + GROUP BY url + ) AS repoMeta + ON repoMeta.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT r2.url AS repoUrl, count(DISTINCT pd.dependsOnId) AS vulnerableDeps + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN packageDependencies pd ON pd.packageId = pr.packageId + WHERE + pd.dependsOnId IN ( + SELECT ap.packageId + FROM advisoryPackages ap + INNER JOIN advisories a ON a.id = ap.advisoryId + WHERE a.severity IN ('HIGH', 'CRITICAL') + ) + GROUP BY r2.url + ) AS deps + ON deps.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + repoUrl, + dateDiff('day', top2[1], today()) AS daysSinceLatest, + if(length(top2) >= 2, dateDiff('day', top2[2], top2[1]), 9999) AS daysBetweenRecent + FROM + ( + SELECT + r2.url AS repoUrl, + arraySlice( + arraySort(x -> - toInt64(x), arrayDistinct(groupArray(rd.releaseDay))), 1, 2 + ) AS top2 + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN + ( + SELECT DISTINCT packageId, toDate(publishedAt) AS releaseDay + FROM versions + WHERE publishedAt IS NOT NULL + ) rd + ON rd.packageId = pr.packageId + GROUP BY r2.url + ) + ) AS rl + ON rl.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m, + countIf( + timestamp <= now() - INTERVAL 6 MONTH AND timestamp > now() - INTERVAL 12 MONTH + ) AS commitsPrior6m + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE type = 'authored-commit' + GROUP BY channel + ) AS c + ON c.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT url AS repoUrl, argMax(lastCommitAt, updatedAt) AS lastCommitAt + FROM repos + GROUP BY url + ) AS rc + ON rc.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(closedAt IS NOT NULL) AS closed12m, + count() AS opened12m, + quantileIf(0.5) + (closedInSeconds, closedAt > now() - INTERVAL 12 MONTH) AS medianCloseS + FROM issues_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH + GROUP BY channel + ) AS w + ON w.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(mergedAt IS NOT NULL) AS merged12m, + countIf(closedAt IS NOT NULL AND mergedAt IS NULL) AS closedUnmerged12m, + quantileIf(0.5) + (mergedInSeconds, mergedAt > now() - INTERVAL 12 MONTH) AS medianMergeS + FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH + GROUP BY channel + ) AS p + ON p.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + respondedInSeconds IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d, + count() AS openedLast18m + FROM issues_analyzed + WHERE openedAt > now() - INTERVAL 18 MONTH + GROUP BY channel + ) AS uw + ON uw.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + reviewedAt IS NULL + AND approvedAt IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d, + count() AS openedLast18m + FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 18 MONTH + GROUP BY channel + ) AS upw + ON upw.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT r2.url AS repoUrl, count(DISTINCT pr.packageId) AS trackedPackageCount + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + GROUP BY r2.url + ) AS pkgs + ON pkgs.repoUrl = allRepos.repoUrl + +TYPE COPY +TARGET_DATASOURCE health_score_v2_raw_inputs_snapshot_ds +COPY_MODE append +COPY_SCHEDULE 30 0 1 * * diff --git a/services/libs/tinybird/pipes/health_score_v2_security.pipe b/services/libs/tinybird/pipes/health_score_v2_security.pipe index d1f303b60f..1521e560b8 100644 --- a/services/libs/tinybird/pipes/health_score_v2_security.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_security.pipe @@ -37,7 +37,8 @@ SQL > branchProtectionAllowsForcePush, dependencyHealthScore, dependencyHealthAvailable, - vulnerableDeps + vulnerableDeps, + '2.0.0' AS methodologyVersion FROM ( SELECT @@ -143,7 +144,7 @@ SQL > repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, - countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates + countIf(status = 'OPEN' AND severity = 'MEDIUM') AS openModerates FROM vulnerabilities GROUP BY repoUrl ) AS vc diff --git a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe index 87b91acd96..ff51cf9cfc 100644 --- a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe @@ -67,7 +67,8 @@ SQL > d.prMergeAvailable AS prMergeAvailable, d.merged12m AS merged12m, d.closedUnmerged12m AS closedUnmerged12m, - d.medianMergeS AS medianMergeS + d.medianMergeS AS medianMergeS, + coalesce(m.methodologyVersion, '2.0.0') AS methodologyVersion FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE isNull (deletedAt)) AS base LEFT JOIN health_score_v2_maintainer_ds AS m ON m.repoUrl = base.repoUrl LEFT JOIN health_score_v2_security_ds AS s ON s.repoUrl = base.repoUrl diff --git a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe index 341fb64d80..297fc0bcbf 100644 --- a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe @@ -22,12 +22,19 @@ SQL > WHERE type = 'issues-closed' AND sourceParentId != '' AND toYear(timestamp) >= 1971 GROUP BY sourceParentId -NODE issues_comment +NODE issues_comment_non_author SQL > - SELECT sourceParentId, MIN(timestamp) AS commentedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'issue-comment' AND sourceParentId != '' AND toYear(timestamp) >= 1971 - GROUP BY sourceParentId + SELECT c.sourceParentId, MIN(c.timestamp) AS respondedAt + FROM activityRelations_deduplicated_cleaned_bucket_union c + INNER JOIN issues_opened opened ON c.sourceParentId = opened.sourceId + WHERE + c.type = 'issue-comment' + AND c.sourceParentId != '' + AND c.memberId != '' + AND opened.memberId != '' + AND c.memberId != opened.memberId + AND toYear(c.timestamp) >= 1971 + GROUP BY c.sourceParentId NODE issue_analysis_results_merged SQL > @@ -40,7 +47,9 @@ SQL > opened.memberId, opened.organizationId, opened.openedAt, - IF(comment.commentedAt = toDateTime(0), NULL, comment.commentedAt) AS commentedAt, + IF( + comment_non_author.respondedAt = toDateTime(0), NULL, comment_non_author.respondedAt + ) AS commentedAt, IF(closed.closedAt = toDateTime(0), NULL, closed.closedAt) AS closedAt, IF( closedAt IS NULL, @@ -50,12 +59,14 @@ SQL > IF( commentedAt IS NULL, NULL, - toUnixTimestamp(toDateTime(comment.commentedAt)) + toUnixTimestamp(toDateTime(comment_non_author.respondedAt)) - toUnixTimestamp(toDateTime(opened.openedAt)) ) AS respondedInSeconds FROM issues_opened opened LEFT JOIN issues_closed AS closed ON opened.sourceId = closed.sourceParentId - LEFT JOIN issues_comment AS comment ON opened.sourceId = comment.sourceParentId + LEFT JOIN + issues_comment_non_author AS comment_non_author + ON opened.sourceId = comment_non_author.sourceParentId TYPE COPY TARGET_DATASOURCE issues_analyzed diff --git a/services/libs/tinybird/pipes/project_insights_copy.pipe b/services/libs/tinybird/pipes/project_insights_copy.pipe index be1d6b8e81..b5348dfb10 100644 --- a/services/libs/tinybird/pipes/project_insights_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_copy.pipe @@ -91,7 +91,7 @@ DESCRIPTION > per-repo values in health_score_v2_repo_copy_ds (materialized by health_score_v2.pipe — the expensive per-signal joins run there as their own copy job, not inline here). Per spec section 5/6: - healthScoreV2: straight mean across the project's enabled, non-excluded repos. - - lifecycleLabelV2: best-state-wins (active > stable > declining > abandoned > archived), with + - lifecycleLabelV2: best-state-wins (active > stable > declining > inert > abandoned > archived), with NULL (no usable signal, see health_score_v2_lifecycle.pipe) as the most-cautious outcome. `groupArray` silently drops NULL entries, so a project with at least one repo in a real state already rolls up correctly for free — NULL only surfaces at the project level when EVERY repo @@ -127,7 +127,9 @@ SQL > toNullable( arrayElement( arraySort( - x -> indexOf(['active', 'stable', 'declining', 'abandoned', 'archived'], x), + x -> indexOf( + ['active', 'stable', 'declining', 'inert', 'abandoned', 'archived'], x + ), groupArray(hv2.lifecycleLabelV2) ), 1 diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index 0f5da451a3..8435878c0a 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -122,7 +122,16 @@ SQL > max(sd.prMergeAvailable) AS prMergeAvailable, max(sd.merged12m) AS merged12m, max(sd.closedUnmerged12m) AS closedUnmerged12m, - avg(sd.medianMergeS) AS medianMergeS + avg(sd.medianMergeS) AS medianMergeS, + sumIf( + coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), + sd.busFactorAvailable + ) / nullIf( + sumIf( + coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable AND sd.commitsLast6m IS NOT NULL + ), + 0 + ) AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN repositories rep FINAL diff --git a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe index 960dc9f19d..83658857a7 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe @@ -1,6 +1,17 @@ DESCRIPTION > Compacts activities from same PR into one, keeping necessary information in a single row. Uses existing pull_requests_analyzed data as baseline and merges new events on top. + - reviewedAt/approvedAt count only NON-AUTHOR activity (2026-08-11, IN-1226): this incremental + path feeds the minute-0 snapshot merger that replaces pull_requests_analyzed right before the + 02:00-02:15 health score jobs read it. See the non-author filter node below. + - `pull_request_analysis_copy_pipe.pipe` (2026-08-11/12, IN-1226): deleted — confirmed obsolete + by Anıl Bostancı (2026-08-12): "prs now run through MV + merger copy pipe, this was the old one + that copied everything at once." It referenced a datasource deleted in January's bucketing + migration (`activityRelations_deduplicated_cleaned_ds` -> `..._bucket_union`) and had been + disabled/not running since 2026-08-06, which is why the reference had gone stale unnoticed. + This MV + pull_request_analysis_snapshot_merger_copy.pipe are the real hourly path; this MV + + pull_request_analysis_initial_snapshot.pipe (bootstrap) are the only two places the non-author + filter needs to live. NODE snapshot_resolver DESCRIPTION > @@ -50,6 +61,85 @@ SQL > 'changeset-merged' ) +NODE pr_authors +DESCRIPTION > + PR author per PR touched by a review/approval event in this batch: baseline authors from + pull_requests_analyzed (semi-join, same pattern as baseline_filtered) plus authors from open + events arriving in the same batch. + +SQL > + SELECT sourceId AS prSourceId, memberId + FROM pull_requests_analyzed + WHERE + memberId != '' + AND sourceId IN ( + SELECT DISTINCT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) + FROM new_pull_request_related_activity + WHERE + type IN ( + 'pull_request-reviewed', + 'merge_request-review-changes-requested', + 'merge_request-review-approved', + 'patchset_approval-created' + ) + ) + UNION ALL + SELECT sourceId AS prSourceId, memberId + FROM new_pull_request_related_activity + WHERE + type IN ('pull_request-opened', 'merge_request-opened', 'changeset-created') AND memberId != '' + +NODE new_pull_request_related_activity_non_author +DESCRIPTION > + Drops review/approval events performed by the PR author (strict both-IDs-non-empty rule, + matching issue_analysis_copy_pipe.pipe). Non-review event types pass through untouched. + A review whose PR has no known author yet is dropped here — it was already discarded + downstream (the merge node requires an open event), so behavior for those rows is unchanged. + +SQL > + SELECT + a.id, + a.sourceId, + a.sourceParentId, + a.channel, + a.ts, + a.segmentId, + a.gitChangedLinesBucket, + a.memberId, + a.organizationId, + a.platform, + a.updatedAt, + a.type, + a.pullRequestReviewState + FROM + ( + SELECT + *, + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prKey + FROM new_pull_request_related_activity + ) AS a + ANY + LEFT JOIN pr_authors AS auth ON auth.prSourceId = a.prKey + WHERE + NOT ( + a.type IN ( + 'pull_request-reviewed', + 'merge_request-review-changes-requested', + 'merge_request-review-approved', + 'patchset_approval-created' + ) + ) + OR (a.memberId != '' AND auth.memberId != '' AND a.memberId != auth.memberId) + NODE new_events_aggregated DESCRIPTION > Aggregates new events from the latest snapshot by PR source ID. @@ -231,7 +321,7 @@ SQL > NULL ) ) AS resolvedAt - FROM new_pull_request_related_activity + FROM new_pull_request_related_activity_non_author GROUP BY prSourceId HAVING prSourceId != '' diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe deleted file mode 100644 index 4663d111b1..0000000000 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ /dev/null @@ -1,173 +0,0 @@ -DESCRIPTION > - Compacts activities from same PR into one, keeping necessary information in a single row. Helps to serve PR-wide widgets in the development tab. - -NODE pull_request_opened -SQL > - SELECT - activityId as id, - sourceId, - channel, - timestamp AS openedAt, - segmentId, - gitChangedLinesBucket, - memberId, - organizationId, - platform - FROM activityRelations_deduplicated_cleaned_ds - WHERE type = 'pull_request-opened' OR type = 'merge_request-opened' OR type = 'changeset-created' - -NODE pull_request_first_assigned -SQL > - SELECT sourceParentId, MIN(timestamp) AS assignedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE type = 'pull_request-assigned' OR type = 'merge_request-assigned' - GROUP BY sourceParentId - order by min(timestamp) desc - -NODE pull_request_first_review_requested -SQL > - SELECT sourceParentId, MIN(timestamp) AS reviewRequestedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE type = 'pull_request-review-requested' OR type = 'merge_request-review-requested' - GROUP BY sourceParentId - -NODE pull_request_first_reviewed -SQL > - SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS reviewedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - type = 'pull_request-reviewed' - OR type = 'merge_request-review-changes-requested' - OR type = 'patchset_approval-created' - GROUP BY sourceParentId - -NODE pull_request_first_review_approved -SQL > - SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS approvedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - GROUP BY sourceParentId - -NODE pull_request_first_closed -SQL > - SELECT - if(type = 'changeset-abandoned', sourceId, sourceParentId) AS sourceParentId, - MIN(timestamp) AS closedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - type = 'pull_request-closed' - OR type = 'merge_request-closed' - OR type = 'changeset-closed' - OR type = 'changeset-abandoned' - GROUP BY sourceParentId - -NODE pull_request_first_merged -DESCRIPTION > - Resolved PRs are the ones that are either closed or merged - -SQL > - SELECT - if(type = 'changeset-merged', sourceId, sourceParentId) AS sourceParentId, - MIN(timestamp) AS mergedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE type = 'pull_request-merged' OR type = 'merge_request-merged' OR type = 'changeset-merged' - GROUP BY sourceParentId - -NODE pull_request_first_resolved -SQL > - SELECT - if( - type IN ('changeset-abandoned', 'changeset-merged'), sourceId, sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS resolvedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - type = 'pull_request-closed' - OR type = 'pull_request-merged' - OR type = 'merge_request-closed' - OR type = 'merge_request-merged' - OR type = 'changeset-merged' - OR type = 'changeset-closed' - OR type = 'changeset-abandoned' - GROUP BY sourceParentId - -NODE patchsets_count -DESCRIPTION > - Count the number of patchsets for each Gerrit changeset - -SQL > - SELECT sourceParentId, toInt64(COUNT(*)) AS numberOfPatchsets - FROM activityRelations_deduplicated_cleaned_ds - WHERE type = 'patchset-created' - GROUP BY sourceParentId - -NODE pull_request_analysis_results_merged -SQL > - SELECT - pr_opened.id, - pr_opened.sourceId, - pr_opened.openedAt, - pr_opened.segmentId, - pr_opened.channel, - pr_opened.memberId, - pr_opened.organizationId, - pr_opened.gitChangedLinesBucket, - IF(assigned.assignedAt = toDateTime(0), NULL, assigned.assignedAt) AS assignedAt, - IF( - review_requested.reviewRequestedAt = toDateTime(0), NULL, review_requested.reviewRequestedAt - ) AS reviewRequestedAt, - IF(reviewed.reviewedAt = toDateTime(0), NULL, reviewed.reviewedAt) AS reviewedAt, - IF(approved.approvedAt = toDateTime(0), NULL, approved.approvedAt) AS approvedAt, - IF(closed.closedAt = toDateTime(0), NULL, closed.closedAt) AS closedAt, - IF(merged.mergedAt = toDateTime(0), NULL, merged.mergedAt) AS mergedAt, - IF(resolved.resolvedAt = toDateTime(0), NULL, resolved.resolvedAt) AS resolvedAt, - IF( - assignedAt IS NULL, NULL, toUnixTimestamp(assignedAt) - toUnixTimestamp(openedAt) - ) AS assignedInSeconds, - IF( - reviewRequestedAt IS NULL, - NULL, - toUnixTimestamp(reviewRequestedAt) - toUnixTimestamp(openedAt) - ) AS reviewRequestedInSeconds, - IF( - reviewedAt IS NULL, NULL, toUnixTimestamp(reviewedAt) - toUnixTimestamp(openedAt) - ) AS reviewedInSeconds, - IF( - closedAt IS NULL, NULL, toUnixTimestamp(closedAt) - toUnixTimestamp(openedAt) - ) AS closedInSeconds, - IF( - mergedAt IS NULL, NULL, toUnixTimestamp(mergedAt) - toUnixTimestamp(openedAt) - ) AS mergedInSeconds, - IF( - resolvedAt IS NULL, NULL, toUnixTimestamp(resolvedAt) - toUnixTimestamp(openedAt) - ) AS resolvedInSeconds, - pr_opened.platform, - patchsets.numberOfPatchsets - FROM pull_request_opened pr_opened - LEFT JOIN pull_request_first_assigned AS assigned ON pr_opened.sourceId = assigned.sourceParentId - LEFT JOIN - pull_request_first_review_requested AS review_requested - ON pr_opened.sourceId = review_requested.sourceParentId - LEFT JOIN pull_request_first_reviewed AS reviewed ON pr_opened.sourceId = reviewed.sourceParentId - LEFT JOIN - pull_request_first_review_approved AS approved ON pr_opened.sourceId = approved.sourceParentId - LEFT JOIN pull_request_first_closed AS closed ON pr_opened.sourceId = closed.sourceParentId - LEFT JOIN pull_request_first_merged AS merged ON pr_opened.sourceId = merged.sourceParentId - LEFT JOIN pull_request_first_resolved as resolved on pr_opened.sourceId = resolved.sourceParentId - LEFT JOIN patchsets_count AS patchsets ON pr_opened.sourceId = patchsets.sourceParentId - -TYPE COPY -TARGET_DATASOURCE pull_requests_analyzed -COPY_MODE replace -COPY_SCHEDULE 20 * * * * diff --git a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe index fdc2d3116f..d5b933268a 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe @@ -78,66 +78,96 @@ SQL > GROUP BY sourceParentId NODE pull_request_first_reviewed +DESCRIPTION > + Non-author reviews only (2026-08-11, IN-1226) — same rule as the baseline-merge MV. + SQL > % SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - argMin(updatedAt, timestamp) AS updatedAt, - MIN(timestamp) AS reviewedAt - FROM activityRelations_enriched_deduplicated_bucket_union - WHERE - sourceParentId <> '' - and ( - type = 'pull_request-reviewed' - OR type = 'merge_request-review-changes-requested' - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - ) - {% if defined(bucket_id) %} - AND cityHash64(segmentId) - % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} - = {{ - UInt8( - bucket_id, - 0, - description="Bucket index to process (0 to num_buckets-1)", - required=False, + r.prSourceId AS sourceParentId, + argMin(r.updatedAt, r.timestamp) AS updatedAt, + MIN(r.timestamp) AS reviewedAt + FROM + ( + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + updatedAt, + memberId + FROM activityRelations_enriched_deduplicated_bucket_union + WHERE + sourceParentId <> '' + and ( + type = 'pull_request-reviewed' + OR type = 'merge_request-review-changes-requested' + OR type = 'merge_request-review-approved' + OR type = 'patchset_approval-created' ) - }} - {% end %} - GROUP BY sourceParentId + {% if defined(bucket_id) %} + AND cityHash64(segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_review_approved +DESCRIPTION > + Non-author approvals only (2026-08-11, IN-1226) — same rule as the baseline-merge MV. + SQL > % SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - argMin(updatedAt, timestamp) AS updatedAt, - MIN(timestamp) AS approvedAt - FROM activityRelations_enriched_deduplicated_bucket_union - WHERE + r.prSourceId AS sourceParentId, + argMin(r.updatedAt, r.timestamp) AS updatedAt, + MIN(r.timestamp) AS approvedAt + FROM ( - (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - ) - {% if defined(bucket_id) %} - AND cityHash64(segmentId) - % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} - = {{ - UInt8( - bucket_id, - 0, - description="Bucket index to process (0 to num_buckets-1)", - required=False, + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + updatedAt, + memberId + FROM activityRelations_enriched_deduplicated_bucket_union + WHERE + ( + (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') + OR type = 'merge_request-review-approved' + OR type = 'patchset_approval-created' ) - }} - {% end %} - GROUP BY sourceParentId + {% if defined(bucket_id) %} + AND cityHash64(segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_closed SQL >