From 76015bec6e744580a929d2d032bebb1a39d5a385 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 20:33:23 +0100 Subject: [PATCH 01/25] feat: align health score v2 with community feedback commitments (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 - Issue responsiveness fix: - issue_analysis_copy_pipe: Filter comments to exclude author self-responses - This ensures 'responded' signal reflects genuine external feedback, not self-commentary Phase 2 - v2 compliance & documentation: - health_score_v2_lifecycle: Split 'abandoned' (unanswered 90-180d + no maintainer 12m) from 'inert' (no commits 18m + zero activity) - health_score_v2_maintainer: Add observed review/merge actor count (parallel to curated roster) via greatest(curated, observed) for bus-factor - health_score_v2_maintainer/lifecycle: Repoint org-diversity & bus-factor joins to use cleaned deduplicated streams (activityRelations_deduplicated_cleaned_bucket_union) instead of raw activityRelations - All v2 datasources/pipes: Add methodologyVersion='2.0.0' constant - Create health_score_v2_raw_inputs_snapshot pipe+datasource (monthly append-mode capture of raw signals for validation) - README: Document repositories.excluded as official experimental-repo mechanism - project_insights_health_breakdown_copy: Add busFactorScoreActivityWeightedMean for validation/comparison alongside existing coverage-filtered average - health_score_v2_repo_copy_ds: Fix description (graceful-degradation IS implemented, document security_contact_email spec gap as temporary) Item 7 verification: All 9 per-signal *Available flags confirmed present and emitted (busFactorAvailable, orgDiversityAvailable, responsivenessAvailable, scorecardAvailable, securityPracticesAvailable, dependencyHealthAvailable, releaseCadenceAvailable, issueResolutionAvailable, prMergeAvailable). Note: Day thresholds (90/180/18mo) are provisional pending validation analysis. No changes to IN-1212 branch. Postgres writeback deferred (Tinybird-only per decision). Signed-off-by: Gašper Grom --- services/libs/tinybird/README.md | 28 ++ .../health_score_v2_development_ds.datasource | 3 +- .../health_score_v2_lifecycle_ds.datasource | 3 +- .../health_score_v2_maintainer_ds.datasource | 3 +- ...score_v2_raw_inputs_snapshot_ds.datasource | 52 ++++ .../health_score_v2_repo_copy_ds.datasource | 16 +- .../health_score_v2_security_ds.datasource | 3 +- ...ealth_score_v2_signal_detail_ds.datasource | 3 +- .../libs/tinybird/pipes/health_score_v2.pipe | 3 +- .../pipes/health_score_v2_development.pipe | 3 +- .../pipes/health_score_v2_lifecycle.pipe | 25 +- .../pipes/health_score_v2_maintainer.pipe | 42 ++- .../health_score_v2_raw_inputs_snapshot.pipe | 284 ++++++++++++++++++ .../pipes/health_score_v2_security.pipe | 3 +- .../pipes/health_score_v2_signal_detail.pipe | 3 +- .../pipes/issue_analysis_copy_pipe.pipe | 21 +- ...roject_insights_health_breakdown_copy.pipe | 8 +- 17 files changed, 470 insertions(+), 33 deletions(-) create mode 100644 services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource create mode 100644 services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe diff --git a/services/libs/tinybird/README.md b/services/libs/tinybird/README.md index d015832333..4bb209eeff 100644 --- a/services/libs/tinybird/README.md +++ b/services/libs/tinybird/README.md @@ -339,6 +339,34 @@ tb sql "SELECT count() FROM activities_backup FINAL" If both pairs match, the backup is **logically consistent** with the source dataset. +--- + +## Health Score v2 Configuration + +### Excluding Repositories from Health Score + +Repositories can be marked as excluded from health scoring by setting the `repositories.excluded` flag to `1`. This is the official mechanism for marking experimental, sandbox, meta, or otherwise out-of-scope repositories. + +**Use cases for exclusion:** +- `.github` meta repositories (configuration-only, no product code) +- Archived experiments or prototypes +- Forks created for testing purposes +- Any repository where activity metrics are meaningless or undesired + +**Behavior when excluded:** +- Health Score v2 categorizes excluded repos as `unavailable` for responsiveness scoring (never scored 0 for missing PR/issue data) +- Other category signals continue to be scored normally +- Project-level health scores (when a project contains multiple repos) exclude the marked repo from aggregation + +**Setting the flag:** +```sql +UPDATE repositories SET excluded = 1 WHERE url = 'https://github.com/org/repo-meta'; +``` + +The flag is read by `health_score_v2_maintainer.pipe` and `health_score_v2_lifecycle.pipe` as part of their graceful-degradation logic. + +--- + ## Glossary - **CDP (Community Data Platform)**: Customer data operations and management pipelines diff --git a/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource index daaa218dc9..b94fb1bcbb 100644 --- a/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_development_ds.datasource @@ -38,7 +38,8 @@ SCHEMA > `prMergeAvailable` UInt8, `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource index 84e81a7998..6137eedbbf 100644 --- a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource @@ -8,7 +8,8 @@ DESCRIPTION > SCHEMA > `repoUrl` String, - `lifecycleLabelV2` Nullable(String) + `lifecycleLabelV2` Nullable(String), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource index 46def9226d..6ed97564f8 100644 --- a/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource @@ -33,7 +33,8 @@ SCHEMA > `medianIssueResponseS` Nullable(Float64), `isGerrit` UInt8, `isExcluded` UInt8, - `coveredWeight` UInt8 + `coveredWeight` UInt8, + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource new file mode 100644 index 0000000000..a85d38fd28 --- /dev/null +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -0,0 +1,52 @@ +DESCRIPTION > + - `health_score_v2_raw_inputs_snapshot_ds` captures raw measured inputs (not computed scores) for + Health Score v2 at periodic intervals for validation and historical analysis. + - Populated by `health_score_v2_raw_inputs_snapshot.pipe` (append-mode, monthly cadence). + - Each row is keyed by `repoUrl + snapshotDate + methodologyVersion` and captures all the atomic + measurement signals used to compute the three health score categories: + - Maintainer signals: bus factor (curated maintainer count), observed review/merge actors, + organization diversity (distinct org count), PR/issue response times (medians). + - Security signals: open CVE counts (critical/high/moderate), OpenSSF Scorecard score, security + practices flags (policy/branch-protection state), vulnerable dependencies count. + - Development signals: release recency/cadence, commit counts (6m windows), issue counts/closure + times, PR counts/merge times. + - Does NOT include computed category/overall scores — those are derived from these raw inputs via + the scoring logic in the respective pipes. This datasource is for tracing where scores come from. + - snapshotId = `toStartOfInterval(now(), INTERVAL 1 day)` at pipe execution time. + - Partitioned by year(snapshotDate) and month(snapshotDate) for efficient time-based queries. + - TTL set to 24 months (enough for trend analysis and validation studies). + +SCHEMA > + `repoUrl` String, + `snapshotDate` DateTime, + `methodologyVersion` String, + `busFactorCuratedCount` Nullable(UInt64), + `busFactorObservedActorsCount` Nullable(UInt64), + `orgDiversityCount` Nullable(UInt64), + `medianPrResponseSeconds` Nullable(Float64), + `medianIssueResponseSeconds` Nullable(Float64), + `openCriticalVulns` Nullable(UInt64), + `openHighVulns` Nullable(UInt64), + `openModerateVulns` Nullable(UInt64), + `scorecardScore` Nullable(String), + `securityPolicyEnabled` Nullable(UInt8), + `branchProtectionEnabled` Nullable(UInt8), + `branchProtectionRequiredReviews` Nullable(Int32), + `branchProtectionRequiresStatusChecks` Nullable(UInt8), + `branchProtectionAllowsForcePush` Nullable(UInt8), + `vulnerableDeps` Nullable(UInt64), + `daysSinceLatestRelease` Nullable(Int64), + `daysBetweenRecentReleases` Nullable(Int64), + `commitsLast6m` Nullable(UInt64), + `commitsPrior6m` Nullable(UInt64), + `lastCommitAt` Nullable(DateTime64(3)), + `issuesClosedLast12m` Nullable(UInt64), + `issuesOpenedLast12m` Nullable(UInt64), + `medianIssueCloseSeconds` Nullable(Float64), + `prsMergedLast12m` Nullable(UInt64), + `prsClosedUnmergedLast12m` Nullable(UInt64), + `medianPrMergeSeconds` Nullable(Float64) + +ENGINE MergeTree +ENGINE_SORTING_KEY (snapshotDate, repoUrl) +ENGINE_TTL snapshotDate + INTERVAL 24 MONTH diff --git a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource index 3414e2aa07..99d10644c8 100644 --- a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource @@ -20,8 +20,17 @@ DESCRIPTION > project_insights_copy.pipe. - `impactScore` (0-100) — MAX(packages.impact) * 100 over packages published by this repo. NULL when the repo has no linked packages. - - No graceful-degradation/signal-coverage redistribution yet — a repo with zero signal in a category - scores 0 there rather than having points redistributed from available categories. + - Graceful degradation (spec Layer 1+2): per-category pipes emit NULL when covered sub-signal weight + is <40% of that category's max pts (Layer 1); the aggregate health_score_v2 applies Layer 2 rescaling + (`SUM(available categories) * (100 / SUM(available category weights))`) so only actually-available + scores are summed. If all three categories are unavailable, healthScoreV2 itself is NULL (see + health_score_v2.pipe DESCRIPTION for the full spec). + - Known temporary gap: security-practices score's spec-stated max is 8 points (includes +1 for + `security_contact_email IS NOT NULL` flag), but no `security_contact_email` column exists yet in + the GitHub enrichment datasource, so the current maximum achievable is 7 points. This gap is + intentional and temporary — do not fabricate the missing column or change scoring. When the email + field becomes available in repos, security-practices score will automatically rise to its full 8-point + potential. SCHEMA > `repoUrl` String, @@ -30,7 +39,8 @@ SCHEMA > `developmentActivityScoreV2` Nullable(UInt8), `healthScoreV2` Nullable(UInt8), `lifecycleLabelV2` Nullable(String), - `impactScore` Nullable(UInt8) + `impactScore` Nullable(UInt8), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource index 09fee28c01..2a33f852a2 100644 --- a/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_security_ds.datasource @@ -37,7 +37,8 @@ SCHEMA > `branchProtectionAllowsForcePush` Nullable(UInt8), `dependencyHealthScore` UInt8, `dependencyHealthAvailable` UInt8, - `vulnerableDeps` Nullable(UInt64) + `vulnerableDeps` Nullable(UInt64), + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource index f6bf42ff0e..b160dfa138 100644 --- a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource @@ -71,7 +71,8 @@ SCHEMA > `prMergeAvailable` Nullable(UInt8), `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `methodologyVersion` Nullable(String) ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/pipes/health_score_v2.pipe b/services/libs/tinybird/pipes/health_score_v2.pipe index 96812a9021..c2ce47bd11 100644 --- a/services/libs/tinybird/pipes/health_score_v2.pipe +++ b/services/libs/tinybird/pipes/health_score_v2.pipe @@ -40,7 +40,8 @@ SQL > lifecycleLabelV2, if( impactScoreRaw IS NULL, NULL, toNullable(toUInt8(round(least(impactScoreRaw, 100)))) - ) AS impactScore + ) AS impactScore, + '2.0.0' AS methodologyVersion FROM ( SELECT diff --git a/services/libs/tinybird/pipes/health_score_v2_development.pipe b/services/libs/tinybird/pipes/health_score_v2_development.pipe index e9e63bbd78..3f79e90194 100644 --- a/services/libs/tinybird/pipes/health_score_v2_development.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_development.pipe @@ -36,7 +36,8 @@ SQL > prMergeAvailable, merged12m, closedUnmerged12m, - medianMergeS + medianMergeS, + '2.0.0' AS methodologyVersion FROM ( SELECT diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 60e2837227..a6034b8a20 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -29,6 +29,17 @@ DESCRIPTION > legitimate 'active' classification under the existing spec, not the bug being fixed here. NODE health_score_v2_lifecycle_calc +DESCRIPTION > + - Precedence (first match wins): archived > abandoned > inert > declining > stable > active. + - Threshold note (provisional, pending validation analysis): abandoned uses 90-180 day + unanswered-issue window and 12-month no-maintainer-activity gates; inert uses 18-month + no-commits AND zero open issues/PRs gates. These day counts are under review via the + abandoned-threshold-sweep validation analysis and may shift in a follow-up tuning pass. + - Unanswered: minimum one issue opened in 18mo window + has unanswered issues now. + - Declining: commit count down >50% vs prior 6mo AND issues opened up vs prior 6mo. + - Stable: release within 12mo, <50 open issues, no open critical vulns, AND commits down + >50% vs prior 6mo. + SQL > SELECT r.url AS repoUrl, @@ -42,9 +53,13 @@ SQL > multiIf( r.archived = 1, 'archived', + coalesce(w.issuesInWindow18m, 0) > 0 + AND coalesce(w.unansweredCount, 0) > 0 + AND (r.lastCommitAt < now() - INTERVAL 12 MONTH), + 'abandoned', (r.lastCommitAt < now() - INTERVAL 18 MONTH) AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, - 'abandoned', + 'inert', coalesce(c.commitsLast6m, 0) < coalesce(c.commitsPrior6m, 0) * 0.5 AND coalesce(w.issuesOpenedLast6m, 0) > coalesce(w.issuesOpenedPrior6m, 0), 'declining', @@ -56,7 +71,8 @@ SQL > 'active' ) ) - ) AS lifecycleLabelV2 + ) AS lifecycleLabelV2, + '2.0.0' AS methodologyVersion FROM ( SELECT base.url AS url, base.archived AS archived, rc.lastCommitAt AS lastCommitAt @@ -95,7 +111,10 @@ SQL > countIf( openedAt <= now() - INTERVAL 6 MONTH AND openedAt > now() - INTERVAL 12 MONTH ) AS issuesOpenedPrior6m, - countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m + countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m, + countIf( + openedAt > now() - INTERVAL 18 MONTH AND respondedInSeconds IS NULL + ) AS unansweredCount FROM issues_analyzed GROUP BY channel ) AS w diff --git a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe index 1c2dc02eaa..8f05a286c0 100644 --- a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe @@ -28,6 +28,10 @@ DESCRIPTION > repo within the same trailing 12-month window used elsewhere in this pipe, not just an unexpired role record. This was the direct cause of dead repos hitting healthScoreV2=100 after Layer 2 rescaling (see LFX/cdp/health-score-v2-renormalization-bug.md). + - Bus factor enhancement (IN-1226): Added parallel count of observed review/merge actors in the + trailing 12-month window. Final busFactorCount = max(curated_maintainers_count, observed_actors_count). + busFactorAvailable = 1 if either source has data. This ensures teams relying on informal review + duties aren't penalized for lack of formal role curation. NODE health_score_v2_maintainer_calc SQL > @@ -50,7 +54,8 @@ SQL > base.medianIssueResponseS AS medianIssueResponseS, base.isGerrit AS isGerrit, base.isExcluded AS isExcluded, - coveredWeight AS coveredWeight + coveredWeight AS coveredWeight, + '2.0.0' AS methodologyVersion FROM ( SELECT @@ -77,16 +82,16 @@ SQL > allRepos.repoUrl AS repoUrl, allRepos.isGerrit AS isGerrit, allRepos.isExcluded AS isExcluded, - bf.busFactorCount AS busFactorCount, - bf.repoUrl != '' AS busFactorAvailable, + greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) AS busFactorCount, + (bf.repoUrl != '' OR obs.repoUrl != '') AS busFactorAvailable, multiIf( - bf.busFactorCount >= 5, + greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) >= 5, 18, - bf.busFactorCount >= 3, + greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) >= 3, 15, - bf.busFactorCount = 2, + greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) = 2, 6, - bf.busFactorCount = 1, + greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) = 1, 3, 0 ) AS busFactorScore, @@ -181,12 +186,12 @@ SQL > ) AS allRepos LEFT JOIN ( - SELECT mr.repoUrl AS repoUrl, count(DISTINCT mr.memberId) AS busFactorCount + SELECT mr.repoUrl AS repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount FROM maintainers_roles_copy_ds mr INNER JOIN ( SELECT DISTINCT memberId, channel AS repoUrl - FROM activityRelations + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH ) AS recentActivity ON recentActivity.memberId = mr.memberId @@ -197,10 +202,27 @@ SQL > GROUP BY mr.repoUrl ) AS bf ON bf.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + count(DISTINCT memberId) AS observedActorsCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE + timestamp > now() - INTERVAL 12 MONTH + AND ( + type = 'pull_request-reviewed' + OR type = 'pull_request-merged' + OR type = 'merge_request-review-approved' + OR type = 'merge_request-merged' + ) + GROUP BY channel + ) AS obs + ON obs.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT channel AS repoUrl, count(DISTINCT organizationId) AS orgCount - FROM activityRelations + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH AND organizationId != '' GROUP BY channel ) AS od diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe new file mode 100644 index 0000000000..ec4e23f656 --- /dev/null +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -0,0 +1,284 @@ +DESCRIPTION > + - Periodic snapshot of raw measured inputs (not computed scores) for Health Score v2 validation + and historical trend analysis. Runs monthly (1st of each month) and appends one row per repo. + - Each execution creates a snapshotDate = toStartOfInterval(now(), INTERVAL 1 day), tagged with + the current methodologyVersion ('2.0.0'). + - Sources the same base data as the individual category pipes (health_score_v2_maintainer, + health_score_v2_security, health_score_v2_development) but captures RAW measured values only — + not computed scores. This allows downstream validation queries to reconstruct or audit scoring + without needing access to the category pipes' intermediate logic. + - Append-mode COPY, partitioned by date, with 24-month TTL. + +TAGS "Validation", "Health Score v2" + +NODE raw_inputs_maintainer +DESCRIPTION > + Raw maintainer-health measurement signals: bus-factor counts (curated and observed), + org diversity, and response times. + +SQL > + SELECT + allRepos.repoUrl, + bf.curatedBusFactorCount, + obs.observedActorsCount, + od.orgCount, + r.medianPrResponseS, + ir.medianIssueResponseS + FROM + ( + SELECT DISTINCT url AS repoUrl + FROM repositories + WHERE deletedAt IS NULL + ) AS allRepos + LEFT JOIN + ( + SELECT mr.repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount + FROM maintainers_roles_copy_ds mr + INNER JOIN + ( + SELECT DISTINCT memberId, channel AS repoUrl + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE timestamp > now() - INTERVAL 12 MONTH + ) AS recentActivity + ON recentActivity.memberId = mr.memberId + AND recentActivity.repoUrl = mr.repoUrl + WHERE + mr.endDate = '1970-01-01 00:00:00' + OR mr.endDate > now() - INTERVAL 12 MONTH + GROUP BY mr.repoUrl + ) AS bf + ON bf.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + count(DISTINCT memberId) AS observedActorsCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE + timestamp > now() - INTERVAL 12 MONTH + AND ( + type = 'pull_request-reviewed' + OR type = 'pull_request-merged' + OR type = 'merge_request-review-approved' + OR type = 'merge_request-merged' + ) + GROUP BY channel + ) AS obs + ON obs.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT channel AS repoUrl, count(DISTINCT organizationId) AS orgCount + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE timestamp > now() - INTERVAL 12 MONTH AND organizationId != '' + GROUP BY channel + ) AS od + ON od.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + quantile(0.5)(reviewedInSeconds) AS medianPrResponseS + FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH AND reviewedInSeconds IS NOT NULL + GROUP BY channel + ) AS r + ON r.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + quantile(0.5)(respondedInSeconds) AS medianIssueResponseS + FROM issues_analyzed + WHERE + openedAt > now() - INTERVAL 12 MONTH AND respondedInSeconds IS NOT NULL + GROUP BY channel + ) AS ir + ON ir.repoUrl = allRepos.repoUrl + +NODE raw_inputs_security +DESCRIPTION > + Raw security-health measurement signals: vulnerability counts, scorecard score, and branch + protection / security policy flags. + +SQL > + SELECT + allRepos.repoUrl, + coalesce(v.openCriticals, 0) AS openCriticals, + coalesce(v.openHighs, 0) AS openHighs, + coalesce(v.openModerates, 0) AS openModerates, + repos.scorecardScore, + repos.securityPolicyEnabled, + repos.branchProtectionEnabled, + repos.branchProtectionRequiredReviews, + repos.branchProtectionRequiresStatusChecks, + repos.branchProtectionAllowsForcePush, + deps.vulnerableDeps + FROM + ( + SELECT DISTINCT url AS repoUrl + FROM repositories + WHERE deletedAt IS NULL + ) AS allRepos + LEFT JOIN + ( + SELECT repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, + countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, + countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates + FROM vulnerabilities + GROUP BY repoUrl + ) AS v + ON v.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + url AS repoUrl, + scorecardScore, + securityPolicyEnabled, + branchProtectionEnabled, + branchProtectionRequiredReviews, + branchProtectionRequiresStatusChecks, + branchProtectionAllowsForcePush + FROM repos + ) AS repos + ON repos.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + r2.url AS repoUrl, + count(DISTINCT ad.id) AS vulnerableDeps + FROM repos r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN dependencies ad ON ad.packageId = pr.packageId + WHERE ad.advisoryCountCritical > 0 OR ad.advisoryCountHigh > 0 + GROUP BY r2.url + ) AS deps + ON deps.repoUrl = allRepos.repoUrl + +NODE raw_inputs_development +DESCRIPTION > + Raw development-activity measurement signals: release cadence, commit counts, issue/PR + resolution metrics. + +SQL > + SELECT + allRepos.repoUrl, + rl.daysSinceLatest, + rl.daysBetweenRecent, + c.commitsLast6m, + c.commitsPrior6m, + rc.lastCommitAt, + w.closed12m, + w.opened12m, + w.medianCloseS, + p.merged12m, + p.closedUnmerged12m, + p.medianMergeS + FROM + ( + SELECT DISTINCT url AS repoUrl + FROM repositories + WHERE deletedAt IS NULL + ) AS allRepos + LEFT JOIN + ( + SELECT + r2.url AS repoUrl, + dateDiff('day', max(rd.releaseDay), today()) AS daysSinceLatest, + quantile(0.5)(dateDiff('day', rd.releaseDay, max(rd.releaseDay) OVER (PARTITION BY r2.url))) AS daysBetweenRecent + FROM repos r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN + ( + SELECT DISTINCT packageId, toDate(publishedAt) AS releaseDay + FROM versions + WHERE publishedAt IS NOT NULL + ) rd + ON rd.packageId = pr.packageId + GROUP BY r2.url + ) AS rl + ON rl.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m, + countIf( + timestamp <= now() - INTERVAL 6 MONTH AND timestamp > now() - INTERVAL 12 MONTH + ) AS commitsPrior6m + FROM activityRelations_deduplicated_cleaned_bucket_union + WHERE type = 'authored-commit' + GROUP BY channel + ) AS c + ON c.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT url AS repoUrl, lastCommitAt + FROM repos + ) AS rc + ON rc.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(closedAt IS NOT NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closed12m, + countIf(openedAt > now() - INTERVAL 12 MONTH) AS opened12m, + quantile(0.5)(closedInSeconds) AS medianCloseS + FROM issues_analyzed + GROUP BY channel + ) AS w + ON w.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf(mergedAt IS NOT NULL AND mergedAt > now() - INTERVAL 12 MONTH) AS merged12m, + countIf(closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closedUnmerged12m, + quantile(0.5)(mergedInSeconds) AS medianMergeS + FROM pull_requests_analyzed + GROUP BY channel + ) AS p + ON p.repoUrl = allRepos.repoUrl + +NODE health_score_v2_raw_inputs_snapshot_calc +DESCRIPTION > + Combines raw input signals from all three categories and tags with snapshot metadata. + +SQL > + SELECT + m.repoUrl, + toStartOfInterval(now(), INTERVAL 1 day) AS snapshotDate, + '2.0.0' AS methodologyVersion, + m.curatedBusFactorCount, + m.observedActorsCount, + m.orgCount, + m.medianPrResponseS, + m.medianIssueResponseS, + s.openCriticals, + s.openHighs, + s.openModerates, + s.scorecardScore, + s.securityPolicyEnabled, + s.branchProtectionEnabled, + s.branchProtectionRequiredReviews, + s.branchProtectionRequiresStatusChecks, + s.branchProtectionAllowsForcePush, + s.vulnerableDeps, + d.daysSinceLatest, + d.daysBetweenRecent, + d.commitsLast6m, + d.commitsPrior6m, + d.lastCommitAt, + d.closed12m, + d.opened12m, + d.medianCloseS, + d.merged12m, + d.closedUnmerged12m, + d.medianMergeS + FROM raw_inputs_maintainer m + LEFT JOIN raw_inputs_security s ON s.repoUrl = m.repoUrl + LEFT JOIN raw_inputs_development d ON d.repoUrl = m.repoUrl + +TYPE COPY +TARGET_DATASOURCE health_score_v2_raw_inputs_snapshot_ds +COPY_MODE append +COPY_SCHEDULE 0 1 1 * * diff --git a/services/libs/tinybird/pipes/health_score_v2_security.pipe b/services/libs/tinybird/pipes/health_score_v2_security.pipe index d1f303b60f..0bdeaf4edc 100644 --- a/services/libs/tinybird/pipes/health_score_v2_security.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_security.pipe @@ -37,7 +37,8 @@ SQL > branchProtectionAllowsForcePush, dependencyHealthScore, dependencyHealthAvailable, - vulnerableDeps + vulnerableDeps, + '2.0.0' AS methodologyVersion FROM ( SELECT diff --git a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe index 87b91acd96..6318aeb6aa 100644 --- a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe @@ -67,7 +67,8 @@ SQL > d.prMergeAvailable AS prMergeAvailable, d.merged12m AS merged12m, d.closedUnmerged12m AS closedUnmerged12m, - d.medianMergeS AS medianMergeS + d.medianMergeS AS medianMergeS, + m.methodologyVersion AS methodologyVersion FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE isNull (deletedAt)) AS base LEFT JOIN health_score_v2_maintainer_ds AS m ON m.repoUrl = base.repoUrl LEFT JOIN health_score_v2_security_ds AS s ON s.repoUrl = base.repoUrl diff --git a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe index 341fb64d80..274e2ef64c 100644 --- a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe @@ -24,10 +24,17 @@ SQL > NODE issues_comment SQL > - SELECT sourceParentId, MIN(timestamp) AS commentedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'issue-comment' AND sourceParentId != '' AND toYear(timestamp) >= 1971 - GROUP BY sourceParentId + SELECT + c.sourceParentId, + MIN(c.timestamp) AS commentedAt + FROM activityRelations_deduplicated_cleaned_bucket_union c + INNER JOIN issues_opened opened ON c.sourceParentId = opened.sourceId + WHERE + c.type = 'issue-comment' + AND c.sourceParentId != '' + AND c.memberId != opened.memberId + AND toYear(c.timestamp) >= 1971 + GROUP BY c.sourceParentId NODE issue_analysis_results_merged SQL > @@ -40,7 +47,7 @@ SQL > opened.memberId, opened.organizationId, opened.openedAt, - IF(comment.commentedAt = toDateTime(0), NULL, comment.commentedAt) AS commentedAt, + IF(comment_non_author.respondedAt = toDateTime(0), NULL, comment_non_author.respondedAt) AS commentedAt, IF(closed.closedAt = toDateTime(0), NULL, closed.closedAt) AS closedAt, IF( closedAt IS NULL, @@ -50,12 +57,12 @@ SQL > IF( commentedAt IS NULL, NULL, - toUnixTimestamp(toDateTime(comment.commentedAt)) + toUnixTimestamp(toDateTime(comment_non_author.respondedAt)) - toUnixTimestamp(toDateTime(opened.openedAt)) ) AS respondedInSeconds FROM issues_opened opened LEFT JOIN issues_closed AS closed ON opened.sourceId = closed.sourceParentId - LEFT JOIN issues_comment AS comment ON opened.sourceId = comment.sourceParentId + LEFT JOIN issues_comment_non_author AS comment_non_author ON opened.sourceId = comment_non_author.sourceParentId TYPE COPY TARGET_DATASOURCE issues_analyzed diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index 0f5da451a3..6262e86365 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -122,7 +122,13 @@ SQL > max(sd.prMergeAvailable) AS prMergeAvailable, max(sd.merged12m) AS merged12m, max(sd.closedUnmerged12m) AS closedUnmerged12m, - avg(sd.medianMergeS) AS medianMergeS + avg(sd.medianMergeS) AS medianMergeS, + sumIf( + coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), + sd.busFactorAvailable + ) + / nullIf(sumIf(coalesce(sd.commitsLast6m, 1), sd.busFactorAvailable), 0) + AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN repositories rep FINAL From 9c0e94df843e50845b34d642c068dd7c8fd659f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 20:51:45 +0100 Subject: [PATCH 02/25] fix: correct node/schema/table mismatches in health score v2 pipes (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - issue_analysis_copy_pipe: fix node name mismatch (issues_comment -> issues_comment_non_author) and column alias (commentedAt -> respondedAt) that would have failed at Tinybird push time - project_insights_health_breakdown_ds: add missing busFactorScoreActivityWeightedMean schema column to match the pipe output that already emitted it - health_score_v2_raw_inputs_snapshot: fix vulnerable-deps join to use the real advisory schema (packageDependencies/advisoryPackages/advisories), matching the pattern already proven in health_score_v2_security.pipe -- the pipe previously referenced a nonexistent 'dependencies' table Found while dispatching the Tinybird deploy for the IN-1226 commit; 8 of 9 pipes and all datasources deployed clean on the first pass, this fixes the one that didn't. Signed-off-by: Gašper Grom --- ...score_v2_raw_inputs_snapshot_ds.datasource | 12 +-- ...ct_insights_health_breakdown_ds.datasource | 6 +- .../pipes/health_score_v2_maintainer.pipe | 36 ++++++-- .../health_score_v2_raw_inputs_snapshot.pipe | 84 ++++++++----------- .../pipes/issue_analysis_copy_pipe.pipe | 14 ++-- ...roject_insights_health_breakdown_copy.pipe | 6 +- 6 files changed, 82 insertions(+), 76 deletions(-) diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource index a85d38fd28..92fa4d8dff 100644 --- a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -4,12 +4,12 @@ DESCRIPTION > - Populated by `health_score_v2_raw_inputs_snapshot.pipe` (append-mode, monthly cadence). - Each row is keyed by `repoUrl + snapshotDate + methodologyVersion` and captures all the atomic measurement signals used to compute the three health score categories: - - Maintainer signals: bus factor (curated maintainer count), observed review/merge actors, - organization diversity (distinct org count), PR/issue response times (medians). - - Security signals: open CVE counts (critical/high/moderate), OpenSSF Scorecard score, security - practices flags (policy/branch-protection state), vulnerable dependencies count. - - Development signals: release recency/cadence, commit counts (6m windows), issue counts/closure - times, PR counts/merge times. + - Maintainer signals: bus factor (curated maintainer count), observed review/merge actors, + organization diversity (distinct org count), PR/issue response times (medians). + - Security signals: open CVE counts (critical/high/moderate), OpenSSF Scorecard score, security + practices flags (policy/branch-protection state), vulnerable dependencies count. + - Development signals: release recency/cadence, commit counts (6m windows), issue counts/closure + times, PR counts/merge times. - Does NOT include computed category/overall scores — those are derived from these raw inputs via the scoring logic in the respective pipes. This datasource is for tracing where scores come from. - snapshotId = `toStartOfInterval(now(), INTERVAL 1 day)` at pipe execution time. diff --git a/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource b/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource index 00efad93af..dee8306698 100644 --- a/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource +++ b/services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource @@ -12,7 +12,8 @@ DESCRIPTION > - Maintainer signals: `busFactorScore` (avg), `busFactorAvailable` (max), `busFactorCount` (max), `orgDiversityScore` (avg), `orgDiversityAvailable` (max), `orgCount` (max), `responsivenessScore` (avg), `responsivenessAvailable` (max), `medianPrResponseS` (avg), - `medianIssueResponseS` (avg), `isGerrit` (min), `isExcluded` (min). + `medianIssueResponseS` (avg), `isGerrit` (min), `isExcluded` (min), + `busFactorScoreActivityWeightedMean` (activity-weighted mean of bus-factor score across repos). - Security signals: `openVulnScore` (avg), `openCriticals`/`openHighs`/`openModerates` (max), `scorecardScorePts` (avg), `scorecardAvailable` (max), `scorecardScore` (avg, cast from the raw String), `securityPracticesScore` (avg), `securityPracticesAvailable` (max), @@ -75,7 +76,8 @@ SCHEMA > `prMergeAvailable` Nullable(UInt8), `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), - `medianMergeS` Nullable(Float64) + `medianMergeS` Nullable(Float64), + `busFactorScoreActivityWeightedMean` Nullable(Float64) ENGINE MergeTree ENGINE_SORTING_KEY slug, projectId diff --git a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe index 8f05a286c0..135c7150b8 100644 --- a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe @@ -82,16 +82,34 @@ SQL > allRepos.repoUrl AS repoUrl, allRepos.isGerrit AS isGerrit, allRepos.isExcluded AS isExcluded, - greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) AS busFactorCount, + greatest( + coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0) + ) AS busFactorCount, (bf.repoUrl != '' OR obs.repoUrl != '') AS busFactorAvailable, multiIf( - greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) >= 5, + greatest( + coalesce(bf.curatedBusFactorCount, 0), + coalesce(obs.observedActorsCount, 0) + ) + >= 5, 18, - greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) >= 3, + greatest( + coalesce(bf.curatedBusFactorCount, 0), + coalesce(obs.observedActorsCount, 0) + ) + >= 3, 15, - greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) = 2, + greatest( + coalesce(bf.curatedBusFactorCount, 0), + coalesce(obs.observedActorsCount, 0) + ) + = 2, 6, - greatest(coalesce(bf.curatedBusFactorCount, 0), coalesce(obs.observedActorsCount, 0)) = 1, + greatest( + coalesce(bf.curatedBusFactorCount, 0), + coalesce(obs.observedActorsCount, 0) + ) + = 1, 3, 0 ) AS busFactorScore, @@ -186,7 +204,9 @@ SQL > ) AS allRepos LEFT JOIN ( - SELECT mr.repoUrl AS repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount + SELECT + mr.repoUrl AS repoUrl, + count(DISTINCT mr.memberId) AS curatedBusFactorCount FROM maintainers_roles_copy_ds mr INNER JOIN ( @@ -204,9 +224,7 @@ SQL > ON bf.repoUrl = allRepos.repoUrl LEFT JOIN ( - SELECT - channel AS repoUrl, - count(DISTINCT memberId) AS observedActorsCount + SELECT channel AS repoUrl, count(DISTINCT memberId) AS observedActorsCount FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index ec4e23f656..7573390f71 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -24,12 +24,7 @@ SQL > od.orgCount, r.medianPrResponseS, ir.medianIssueResponseS - FROM - ( - SELECT DISTINCT url AS repoUrl - FROM repositories - WHERE deletedAt IS NULL - ) AS allRepos + FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( SELECT mr.repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount @@ -42,17 +37,13 @@ SQL > ) AS recentActivity ON recentActivity.memberId = mr.memberId AND recentActivity.repoUrl = mr.repoUrl - WHERE - mr.endDate = '1970-01-01 00:00:00' - OR mr.endDate > now() - INTERVAL 12 MONTH + WHERE mr.endDate = '1970-01-01 00:00:00' OR mr.endDate > now() - INTERVAL 12 MONTH GROUP BY mr.repoUrl ) AS bf ON bf.repoUrl = allRepos.repoUrl LEFT JOIN ( - SELECT - channel AS repoUrl, - count(DISTINCT memberId) AS observedActorsCount + SELECT channel AS repoUrl, count(DISTINCT memberId) AS observedActorsCount FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH @@ -75,9 +66,7 @@ SQL > ON od.repoUrl = allRepos.repoUrl LEFT JOIN ( - SELECT - channel AS repoUrl, - quantile(0.5)(reviewedInSeconds) AS medianPrResponseS + SELECT channel AS repoUrl, quantile(0.5)(reviewedInSeconds) AS medianPrResponseS FROM pull_requests_analyzed WHERE openedAt > now() - INTERVAL 12 MONTH AND reviewedInSeconds IS NOT NULL GROUP BY channel @@ -85,12 +74,9 @@ SQL > ON r.repoUrl = allRepos.repoUrl LEFT JOIN ( - SELECT - channel AS repoUrl, - quantile(0.5)(respondedInSeconds) AS medianIssueResponseS + SELECT channel AS repoUrl, quantile(0.5)(respondedInSeconds) AS medianIssueResponseS FROM issues_analyzed - WHERE - openedAt > now() - INTERVAL 12 MONTH AND respondedInSeconds IS NOT NULL + WHERE openedAt > now() - INTERVAL 12 MONTH AND respondedInSeconds IS NOT NULL GROUP BY channel ) AS ir ON ir.repoUrl = allRepos.repoUrl @@ -113,17 +99,14 @@ SQL > repos.branchProtectionRequiresStatusChecks, repos.branchProtectionAllowsForcePush, deps.vulnerableDeps - FROM - ( - SELECT DISTINCT url AS repoUrl - FROM repositories - WHERE deletedAt IS NULL - ) AS allRepos + FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( - SELECT repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, - countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, - countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates + SELECT + repoUrl, + countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, + countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, + countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates FROM vulnerabilities GROUP BY repoUrl ) AS v @@ -143,13 +126,17 @@ SQL > ON repos.repoUrl = allRepos.repoUrl LEFT JOIN ( - SELECT - r2.url AS repoUrl, - count(DISTINCT ad.id) AS vulnerableDeps + SELECT r2.url AS repoUrl, count(DISTINCT pd.dependsOnId) AS vulnerableDeps FROM repos r2 INNER JOIN packageRepos pr ON pr.repoId = r2.id - INNER JOIN dependencies ad ON ad.packageId = pr.packageId - WHERE ad.advisoryCountCritical > 0 OR ad.advisoryCountHigh > 0 + INNER JOIN packageDependencies pd ON pd.packageId = pr.packageId + WHERE + pd.dependsOnId IN ( + SELECT ap.packageId + FROM advisoryPackages ap + INNER JOIN advisories a ON a.id = ap.advisoryId + WHERE a.severity IN ('HIGH', 'CRITICAL') + ) GROUP BY r2.url ) AS deps ON deps.repoUrl = allRepos.repoUrl @@ -173,18 +160,16 @@ SQL > p.merged12m, p.closedUnmerged12m, p.medianMergeS - FROM - ( - SELECT DISTINCT url AS repoUrl - FROM repositories - WHERE deletedAt IS NULL - ) AS allRepos + FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( SELECT r2.url AS repoUrl, dateDiff('day', max(rd.releaseDay), today()) AS daysSinceLatest, - quantile(0.5)(dateDiff('day', rd.releaseDay, max(rd.releaseDay) OVER (PARTITION BY r2.url))) AS daysBetweenRecent + quantile(0.5) + ( + dateDiff('day', rd.releaseDay, max(rd.releaseDay) OVER (PARTITION BY r2.url)) + ) AS daysBetweenRecent FROM repos r2 INNER JOIN packageRepos pr ON pr.repoId = r2.id INNER JOIN @@ -210,19 +195,15 @@ SQL > GROUP BY channel ) AS c ON c.repoUrl = allRepos.repoUrl - LEFT JOIN - ( - SELECT url AS repoUrl, lastCommitAt - FROM repos - ) AS rc - ON rc.repoUrl = allRepos.repoUrl + LEFT JOIN (SELECT url AS repoUrl, lastCommitAt FROM repos) AS rc ON rc.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT channel AS repoUrl, countIf(closedAt IS NOT NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closed12m, countIf(openedAt > now() - INTERVAL 12 MONTH) AS opened12m, - quantile(0.5)(closedInSeconds) AS medianCloseS + quantile(0.5) + (closedInSeconds) AS medianCloseS FROM issues_analyzed GROUP BY channel ) AS w @@ -232,8 +213,11 @@ SQL > SELECT channel AS repoUrl, countIf(mergedAt IS NOT NULL AND mergedAt > now() - INTERVAL 12 MONTH) AS merged12m, - countIf(closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closedUnmerged12m, - quantile(0.5)(mergedInSeconds) AS medianMergeS + countIf( + closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH + ) AS closedUnmerged12m, + quantile(0.5) + (mergedInSeconds) AS medianMergeS FROM pull_requests_analyzed GROUP BY channel ) AS p diff --git a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe index 274e2ef64c..9158a31495 100644 --- a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe @@ -22,11 +22,9 @@ SQL > WHERE type = 'issues-closed' AND sourceParentId != '' AND toYear(timestamp) >= 1971 GROUP BY sourceParentId -NODE issues_comment +NODE issues_comment_non_author SQL > - SELECT - c.sourceParentId, - MIN(c.timestamp) AS commentedAt + SELECT c.sourceParentId, MIN(c.timestamp) AS respondedAt FROM activityRelations_deduplicated_cleaned_bucket_union c INNER JOIN issues_opened opened ON c.sourceParentId = opened.sourceId WHERE @@ -47,7 +45,9 @@ SQL > opened.memberId, opened.organizationId, opened.openedAt, - IF(comment_non_author.respondedAt = toDateTime(0), NULL, comment_non_author.respondedAt) AS commentedAt, + IF( + comment_non_author.respondedAt = toDateTime(0), NULL, comment_non_author.respondedAt + ) AS commentedAt, IF(closed.closedAt = toDateTime(0), NULL, closed.closedAt) AS closedAt, IF( closedAt IS NULL, @@ -62,7 +62,9 @@ SQL > ) AS respondedInSeconds FROM issues_opened opened LEFT JOIN issues_closed AS closed ON opened.sourceId = closed.sourceParentId - LEFT JOIN issues_comment_non_author AS comment_non_author ON opened.sourceId = comment_non_author.sourceParentId + LEFT JOIN + issues_comment_non_author AS comment_non_author + ON opened.sourceId = comment_non_author.sourceParentId TYPE COPY TARGET_DATASOURCE issues_analyzed diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index 6262e86365..bf0da207c5 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -126,9 +126,9 @@ SQL > sumIf( coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), sd.busFactorAvailable - ) - / nullIf(sumIf(coalesce(sd.commitsLast6m, 1), sd.busFactorAvailable), 0) - AS busFactorScoreActivityWeightedMean + ) / nullIf( + sumIf(coalesce(sd.commitsLast6m, 1), sd.busFactorAvailable), 0 + ) AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN repositories rep FINAL From 7cd4c4769bf09126fe6cd69f2dc4c1a816252b8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 21:03:15 +0100 Subject: [PATCH 03/25] fix: collapse snapshot pipe to single node, align schema aliases (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit health_score_v2_raw_inputs_snapshot.pipe previously split its base data into three NODEs (maintainer/security/development) joined together in a final calc node. Tinybird cannot resolve column references once it inlines multiple independently-defined subqueries together at that depth ("Identifier 's.repoUrl' cannot be resolved from subquery with name s"). The three proven, already-deployed category pipes (health_score_v2_maintainer/ _security/_development) never join across NODEs this way - each does its base + joins in a single node. Collapsed this pipe to match that pattern. Also fixes 13 SELECT aliases that didn't match the target datasource's schema column names (e.g. curatedBusFactorCount vs busFactorCuratedCount) - same class of reference bug as 9c0e94df8, only surfaces at actual deploy. Signed-off-by: Gašper Grom --- .../health_score_v2_raw_inputs_snapshot.pipe | 127 +++++------------- 1 file changed, 37 insertions(+), 90 deletions(-) diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 7573390f71..f9bf69714d 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -11,19 +11,46 @@ DESCRIPTION > TAGS "Validation", "Health Score v2" -NODE raw_inputs_maintainer +NODE health_score_v2_raw_inputs_snapshot_calc DESCRIPTION > - Raw maintainer-health measurement signals: bus-factor counts (curated and observed), - org diversity, and response times. + Combines raw input signals from all three categories (maintainer, security, development) + and tags with snapshot metadata. Inlined as one node (rather than three joined nodes) + because Tinybird cannot resolve column references across multiple independently-defined + subqueries once inlined together — each category's raw-input subquery is joined directly + against a single shared repo base here, matching the pattern already proven in the + single-node health_score_v2_maintainer/_security/_development pipes. SQL > SELECT allRepos.repoUrl, - bf.curatedBusFactorCount, - obs.observedActorsCount, - od.orgCount, - r.medianPrResponseS, - ir.medianIssueResponseS + toStartOfInterval(now(), INTERVAL 1 day) AS snapshotDate, + '2.0.0' AS methodologyVersion, + bf.curatedBusFactorCount AS busFactorCuratedCount, + obs.observedActorsCount AS busFactorObservedActorsCount, + od.orgCount AS orgDiversityCount, + r.medianPrResponseS AS medianPrResponseSeconds, + ir.medianIssueResponseS AS medianIssueResponseSeconds, + coalesce(v.openCriticals, 0) AS openCriticalVulns, + coalesce(v.openHighs, 0) AS openHighVulns, + coalesce(v.openModerates, 0) AS openModerateVulns, + repoMeta.scorecardScore, + repoMeta.securityPolicyEnabled, + repoMeta.branchProtectionEnabled, + repoMeta.branchProtectionRequiredReviews, + repoMeta.branchProtectionRequiresStatusChecks, + repoMeta.branchProtectionAllowsForcePush, + deps.vulnerableDeps, + rl.daysSinceLatest AS daysSinceLatestRelease, + rl.daysBetweenRecent AS daysBetweenRecentReleases, + c.commitsLast6m, + c.commitsPrior6m, + rc.lastCommitAt, + w.closed12m AS issuesClosedLast12m, + w.opened12m AS issuesOpenedLast12m, + w.medianCloseS AS medianIssueCloseSeconds, + p.merged12m AS prsMergedLast12m, + p.closedUnmerged12m AS prsClosedUnmergedLast12m, + p.medianMergeS AS medianPrMergeSeconds FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( @@ -80,26 +107,6 @@ SQL > GROUP BY channel ) AS ir ON ir.repoUrl = allRepos.repoUrl - -NODE raw_inputs_security -DESCRIPTION > - Raw security-health measurement signals: vulnerability counts, scorecard score, and branch - protection / security policy flags. - -SQL > - SELECT - allRepos.repoUrl, - coalesce(v.openCriticals, 0) AS openCriticals, - coalesce(v.openHighs, 0) AS openHighs, - coalesce(v.openModerates, 0) AS openModerates, - repos.scorecardScore, - repos.securityPolicyEnabled, - repos.branchProtectionEnabled, - repos.branchProtectionRequiredReviews, - repos.branchProtectionRequiresStatusChecks, - repos.branchProtectionAllowsForcePush, - deps.vulnerableDeps - FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( SELECT @@ -122,8 +129,8 @@ SQL > branchProtectionRequiresStatusChecks, branchProtectionAllowsForcePush FROM repos - ) AS repos - ON repos.repoUrl = allRepos.repoUrl + ) AS repoMeta + ON repoMeta.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT r2.url AS repoUrl, count(DISTINCT pd.dependsOnId) AS vulnerableDeps @@ -140,27 +147,6 @@ SQL > GROUP BY r2.url ) AS deps ON deps.repoUrl = allRepos.repoUrl - -NODE raw_inputs_development -DESCRIPTION > - Raw development-activity measurement signals: release cadence, commit counts, issue/PR - resolution metrics. - -SQL > - SELECT - allRepos.repoUrl, - rl.daysSinceLatest, - rl.daysBetweenRecent, - c.commitsLast6m, - c.commitsPrior6m, - rc.lastCommitAt, - w.closed12m, - w.opened12m, - w.medianCloseS, - p.merged12m, - p.closedUnmerged12m, - p.medianMergeS - FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( SELECT @@ -223,45 +209,6 @@ SQL > ) AS p ON p.repoUrl = allRepos.repoUrl -NODE health_score_v2_raw_inputs_snapshot_calc -DESCRIPTION > - Combines raw input signals from all three categories and tags with snapshot metadata. - -SQL > - SELECT - m.repoUrl, - toStartOfInterval(now(), INTERVAL 1 day) AS snapshotDate, - '2.0.0' AS methodologyVersion, - m.curatedBusFactorCount, - m.observedActorsCount, - m.orgCount, - m.medianPrResponseS, - m.medianIssueResponseS, - s.openCriticals, - s.openHighs, - s.openModerates, - s.scorecardScore, - s.securityPolicyEnabled, - s.branchProtectionEnabled, - s.branchProtectionRequiredReviews, - s.branchProtectionRequiresStatusChecks, - s.branchProtectionAllowsForcePush, - s.vulnerableDeps, - d.daysSinceLatest, - d.daysBetweenRecent, - d.commitsLast6m, - d.commitsPrior6m, - d.lastCommitAt, - d.closed12m, - d.opened12m, - d.medianCloseS, - d.merged12m, - d.closedUnmerged12m, - d.medianMergeS - FROM raw_inputs_maintainer m - LEFT JOIN raw_inputs_security s ON s.repoUrl = m.repoUrl - LEFT JOIN raw_inputs_development d ON d.repoUrl = m.repoUrl - TYPE COPY TARGET_DATASOURCE health_score_v2_raw_inputs_snapshot_ds COPY_MODE append From 629db04d895cbf024c38bd9ea21f0206e5056983 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 21:18:31 +0100 Subject: [PATCH 04/25] fix: avoid nested window fn in release-cadence subquery (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit quantile(0.5)(dateDiff(..., max(...) OVER (PARTITION BY ...))) is rejected by ClickHouse - window functions can't nest inside an aggregate. Replaced with the same groupArray/arraySort/arraySlice top-2-dates approach already proven in health_score_v2_development.pipe's release-cadence subquery. Signed-off-by: Gašper Grom --- .../health_score_v2_raw_inputs_snapshot.pipe | 37 +++++++++++-------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index f9bf69714d..2e9983e425 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -22,7 +22,7 @@ DESCRIPTION > SQL > SELECT - allRepos.repoUrl, + allRepos.repoUrl AS repoUrl, toStartOfInterval(now(), INTERVAL 1 day) AS snapshotDate, '2.0.0' AS methodologyVersion, bf.curatedBusFactorCount AS busFactorCuratedCount, @@ -150,22 +150,27 @@ SQL > LEFT JOIN ( SELECT - r2.url AS repoUrl, - dateDiff('day', max(rd.releaseDay), today()) AS daysSinceLatest, - quantile(0.5) - ( - dateDiff('day', rd.releaseDay, max(rd.releaseDay) OVER (PARTITION BY r2.url)) - ) AS daysBetweenRecent - FROM repos r2 - INNER JOIN packageRepos pr ON pr.repoId = r2.id - INNER JOIN + repoUrl, + dateDiff('day', top2[1], today()) AS daysSinceLatest, + if(length(top2) >= 2, dateDiff('day', top2[2], top2[1]), 9999) AS daysBetweenRecent + FROM ( - SELECT DISTINCT packageId, toDate(publishedAt) AS releaseDay - FROM versions - WHERE publishedAt IS NOT NULL - ) rd - ON rd.packageId = pr.packageId - GROUP BY r2.url + SELECT + r2.url AS repoUrl, + arraySlice( + arraySort(x -> - toInt64(x), arrayDistinct(groupArray(rd.releaseDay))), 1, 2 + ) AS top2 + FROM repos r2 + INNER JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN + ( + SELECT DISTINCT packageId, toDate(publishedAt) AS releaseDay + FROM versions + WHERE publishedAt IS NOT NULL + ) rd + ON rd.packageId = pr.packageId + GROUP BY r2.url + ) ) AS rl ON rl.repoUrl = allRepos.repoUrl LEFT JOIN From f0807ebf35a58e0c1436d089d7492a8df814edbb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 22:50:15 +0100 Subject: [PATCH 05/25] fix: dedupe repos table joins in snapshot pipe to prevent row fan-out (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../health_score_v2_raw_inputs_snapshot.pipe | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 2e9983e425..828d944b24 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -122,19 +122,22 @@ SQL > ( SELECT url AS repoUrl, - scorecardScore, - securityPolicyEnabled, - branchProtectionEnabled, - branchProtectionRequiredReviews, - branchProtectionRequiresStatusChecks, - branchProtectionAllowsForcePush + argMax(scorecardScore, updatedAt) AS scorecardScore, + argMax(securityPolicyEnabled, updatedAt) AS securityPolicyEnabled, + argMax(branchProtectionEnabled, updatedAt) AS branchProtectionEnabled, + argMax(branchProtectionRequiredReviews, updatedAt) AS branchProtectionRequiredReviews, + argMax( + branchProtectionRequiresStatusChecks, updatedAt + ) AS branchProtectionRequiresStatusChecks, + argMax(branchProtectionAllowsForcePush, updatedAt) AS branchProtectionAllowsForcePush FROM repos + GROUP BY url ) AS repoMeta ON repoMeta.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT r2.url AS repoUrl, count(DISTINCT pd.dependsOnId) AS vulnerableDeps - FROM repos r2 + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 INNER JOIN packageRepos pr ON pr.repoId = r2.id INNER JOIN packageDependencies pd ON pd.packageId = pr.packageId WHERE @@ -160,7 +163,7 @@ SQL > arraySlice( arraySort(x -> - toInt64(x), arrayDistinct(groupArray(rd.releaseDay))), 1, 2 ) AS top2 - FROM repos r2 + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 INNER JOIN packageRepos pr ON pr.repoId = r2.id INNER JOIN ( @@ -186,7 +189,10 @@ SQL > GROUP BY channel ) AS c ON c.repoUrl = allRepos.repoUrl - LEFT JOIN (SELECT url AS repoUrl, lastCommitAt FROM repos) AS rc ON rc.repoUrl = allRepos.repoUrl + LEFT JOIN + (SELECT url AS repoUrl, argMax(lastCommitAt, updatedAt) AS lastCommitAt FROM repos GROUP BY url) + AS rc + ON rc.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT From 310860e5d8ea6140851bfd080d813a107bccbaad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Mon, 10 Aug 2026 23:37:07 +0100 Subject: [PATCH 06/25] fix: correct vulnerability severity filter from MODERATE to MEDIUM (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The vulnerabilities datasource has always stored medium-severity CVEs as severity='MEDIUM' (see extractSeverity.ts), but health_score_v2_security and health_score_v2_raw_inputs_snapshot filtered on severity='MODERATE', a value that never existed in the data. openModerateVulns/openModerates were silently 0 for every repo, so the openVulnScore penalty term for medium-severity vulnerabilities never applied, inflating securitySupplyChainScoreV2 for any repo with real medium-severity CVEs. Found via the Part 3 CVE data sanity validation analysis (0/31,710 repos ever showed a moderate CVE, which is not a plausible population characteristic). Also corrects the same stale MODERATE reference in the vulnerabilities datasource's severity enum docstring. Signed-off-by: Gašper Grom --- .../libs/tinybird/datasources/vulnerabilities.datasource | 2 +- .../pipes/health_score_v2_raw_inputs_snapshot.pipe | 9 ++++++--- .../libs/tinybird/pipes/health_score_v2_security.pipe | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/services/libs/tinybird/datasources/vulnerabilities.datasource b/services/libs/tinybird/datasources/vulnerabilities.datasource index 684d5c69be..1a55220ef3 100644 --- a/services/libs/tinybird/datasources/vulnerabilities.datasource +++ b/services/libs/tinybird/datasources/vulnerabilities.datasource @@ -7,7 +7,7 @@ DESCRIPTION > - `cveIds` is an array of CVE identifiers associated with this vulnerability (empty array default). - `ghsaIds` is an array of GitHub Security Advisory identifiers. - `otherIds` is an array of other vulnerability identifiers. - - `severity` is the vulnerability severity level (UNKNOWN, LOW, MODERATE, HIGH, CRITICAL). + - `severity` is the vulnerability severity level (UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL). - `cvssScore` is the raw CVSS score (nullable, null when score is unavailable). - `summary` is a short description of the vulnerability. - `details` contains detailed information about the vulnerability. diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 828d944b24..0757a2aa7d 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -113,7 +113,7 @@ SQL > repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, - countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates + countIf(status = 'OPEN' AND severity = 'MEDIUM') AS openModerates FROM vulnerabilities GROUP BY repoUrl ) AS v @@ -190,8 +190,11 @@ SQL > ) AS c ON c.repoUrl = allRepos.repoUrl LEFT JOIN - (SELECT url AS repoUrl, argMax(lastCommitAt, updatedAt) AS lastCommitAt FROM repos GROUP BY url) - AS rc + ( + SELECT url AS repoUrl, argMax(lastCommitAt, updatedAt) AS lastCommitAt + FROM repos + GROUP BY url + ) AS rc ON rc.repoUrl = allRepos.repoUrl LEFT JOIN ( diff --git a/services/libs/tinybird/pipes/health_score_v2_security.pipe b/services/libs/tinybird/pipes/health_score_v2_security.pipe index 0bdeaf4edc..1521e560b8 100644 --- a/services/libs/tinybird/pipes/health_score_v2_security.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_security.pipe @@ -144,7 +144,7 @@ SQL > repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, - countIf(status = 'OPEN' AND severity = 'MODERATE') AS openModerates + countIf(status = 'OPEN' AND severity = 'MEDIUM') AS openModerates FROM vulnerabilities GROUP BY repoUrl ) AS vc From 6a22872e0c9da418992e214df1859659f74ad75f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 10:28:43 +0100 Subject: [PATCH 07/25] fix: address PR #4460 review comments on Health Score v2 pipelines (IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- services/libs/tinybird/README.md | 2 +- ...score_v2_raw_inputs_snapshot_ds.datasource | 6 +++- .../pipes/health_score_v2_lifecycle.pipe | 10 +++--- .../pipes/health_score_v2_maintainer.pipe | 4 +++ .../health_score_v2_raw_inputs_snapshot.pipe | 33 +++++++++++++++---- .../pipes/issue_analysis_copy_pipe.pipe | 2 ++ .../tinybird/pipes/project_insights_copy.pipe | 2 +- ...roject_insights_health_breakdown_copy.pipe | 2 +- 8 files changed, 47 insertions(+), 14 deletions(-) diff --git a/services/libs/tinybird/README.md b/services/libs/tinybird/README.md index 4bb209eeff..c230e99a76 100644 --- a/services/libs/tinybird/README.md +++ b/services/libs/tinybird/README.md @@ -363,7 +363,7 @@ Repositories can be marked as excluded from health scoring by setting the `repos UPDATE repositories SET excluded = 1 WHERE url = 'https://github.com/org/repo-meta'; ``` -The flag is read by `health_score_v2_maintainer.pipe` and `health_score_v2_lifecycle.pipe` as part of their graceful-degradation logic. +The flag is read by `project_insights_copy.pipe` and `health_score_v2_raw_inputs_snapshot.pipe` as part of their graceful-degradation and audit logic. --- diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource index 92fa4d8dff..7157b42894 100644 --- a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -45,8 +45,12 @@ SCHEMA > `medianIssueCloseSeconds` Nullable(Float64), `prsMergedLast12m` Nullable(UInt64), `prsClosedUnmergedLast12m` Nullable(UInt64), - `medianPrMergeSeconds` Nullable(Float64) + `medianPrMergeSeconds` Nullable(Float64), + `excluded` Nullable(UInt8), + `trackedPackageCount` Nullable(UInt64) ENGINE MergeTree ENGINE_SORTING_KEY (snapshotDate, repoUrl) +ENGINE_PARTITION_KEY toYYYYMM(snapshotDate) ENGINE_TTL snapshotDate + INTERVAL 24 MONTH +UNIQUE_KEY (repoUrl, snapshotDate, methodologyVersion) diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index a6034b8a20..4fe031cc96 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -47,7 +47,8 @@ SQL > r.archived != 1 AND coalesce(c.commitsLast6m, 0) = 0 AND coalesce(c.commitsPrior6m, 0) = 0 - AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, + AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0 + AND r.lastCommitAt IS NOT NULL, NULL, toNullable( multiIf( @@ -55,9 +56,10 @@ SQL > 'archived', coalesce(w.issuesInWindow18m, 0) > 0 AND coalesce(w.unansweredCount, 0) > 0 - AND (r.lastCommitAt < now() - INTERVAL 12 MONTH), + AND coalesce(w.issuesOpenNow, 0) > 0 + AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', - (r.lastCommitAt < now() - INTERVAL 18 MONTH) + (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 18 MONTH) AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, 'inert', coalesce(c.commitsLast6m, 0) < coalesce(c.commitsPrior6m, 0) * 0.5 @@ -113,7 +115,7 @@ SQL > ) AS issuesOpenedPrior6m, countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m, countIf( - openedAt > now() - INTERVAL 18 MONTH AND respondedInSeconds IS NULL + openedAt > now() - INTERVAL 18 MONTH AND respondedInSeconds IS NULL AND closedAt IS NULL ) AS unansweredCount FROM issues_analyzed GROUP BY channel diff --git a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe index 135c7150b8..b44d96c5fc 100644 --- a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe @@ -228,11 +228,15 @@ SQL > FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH + AND memberId != '' AND ( type = 'pull_request-reviewed' OR type = 'pull_request-merged' OR type = 'merge_request-review-approved' OR type = 'merge_request-merged' + OR type = 'patchset_approval-created' + OR type = 'merge_request-review-unapproved' + OR type = 'merge_request-review-commented' ) GROUP BY channel ) AS obs diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 0757a2aa7d..b173a29cf8 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -50,8 +50,15 @@ SQL > w.medianCloseS AS medianIssueCloseSeconds, p.merged12m AS prsMergedLast12m, p.closedUnmerged12m AS prsClosedUnmergedLast12m, - p.medianMergeS AS medianPrMergeSeconds - FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS allRepos + p.medianMergeS AS medianPrMergeSeconds, + r.excluded, + pkgs.trackedPackageCount + FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE deletedAt IS NULL) AS allRepos + LEFT JOIN + ( + SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL + ) AS r + ON r.url = allRepos.repoUrl LEFT JOIN ( SELECT mr.repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount @@ -74,11 +81,15 @@ SQL > FROM activityRelations_deduplicated_cleaned_bucket_union WHERE timestamp > now() - INTERVAL 12 MONTH + AND memberId != '' AND ( type = 'pull_request-reviewed' OR type = 'pull_request-merged' OR type = 'merge_request-review-approved' OR type = 'merge_request-merged' + OR type = 'patchset_approval-created' + OR type = 'merge_request-review-unapproved' + OR type = 'merge_request-review-commented' ) GROUP BY channel ) AS obs @@ -202,8 +213,9 @@ SQL > channel AS repoUrl, countIf(closedAt IS NOT NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closed12m, countIf(openedAt > now() - INTERVAL 12 MONTH) AS opened12m, - quantile(0.5) - (closedInSeconds) AS medianCloseS + quantile(0.5)( + closedInSeconds + ) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianCloseS FROM issues_analyzed GROUP BY channel ) AS w @@ -216,12 +228,21 @@ SQL > countIf( closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH ) AS closedUnmerged12m, - quantile(0.5) - (mergedInSeconds) AS medianMergeS + quantile(0.5)( + mergedInSeconds + ) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianMergeS FROM pull_requests_analyzed GROUP BY channel ) AS p ON p.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT r2.url AS repoUrl, count(DISTINCT pr.packageId) AS trackedPackageCount + FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 + LEFT JOIN packageRepos pr ON pr.repoId = r2.id + GROUP BY r2.url + ) AS pkgs + ON pkgs.repoUrl = allRepos.repoUrl TYPE COPY TARGET_DATASOURCE health_score_v2_raw_inputs_snapshot_ds diff --git a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe index 9158a31495..297fc0bcbf 100644 --- a/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/issue_analysis_copy_pipe.pipe @@ -30,6 +30,8 @@ SQL > WHERE c.type = 'issue-comment' AND c.sourceParentId != '' + AND c.memberId != '' + AND opened.memberId != '' AND c.memberId != opened.memberId AND toYear(c.timestamp) >= 1971 GROUP BY c.sourceParentId diff --git a/services/libs/tinybird/pipes/project_insights_copy.pipe b/services/libs/tinybird/pipes/project_insights_copy.pipe index be1d6b8e81..6b5ae220ff 100644 --- a/services/libs/tinybird/pipes/project_insights_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_copy.pipe @@ -127,7 +127,7 @@ SQL > toNullable( arrayElement( arraySort( - x -> indexOf(['active', 'stable', 'declining', 'abandoned', 'archived'], x), + x -> indexOf(['active', 'stable', 'declining', 'abandoned', 'inert', 'archived'], x), groupArray(hv2.lifecycleLabelV2) ), 1 diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index bf0da207c5..39e7b22151 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -127,7 +127,7 @@ SQL > coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), sd.busFactorAvailable ) / nullIf( - sumIf(coalesce(sd.commitsLast6m, 1), sd.busFactorAvailable), 0 + sumIf(coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable), 0 ) AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN From 315ed5512ec4a9b180b7e7641756e71094b35ac9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 11:03:09 +0100 Subject: [PATCH 08/25] fix: resolve Tinybird query compilation and logic errors (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename repository alias from 'r' to 'rf' in snapshot to avoid collision - Change snapshot schedule from 0 1 1 * * to 30 0 1 * * to avoid quota collision - Fix lifecycle: use 'n.issuesOpenNow' instead of 'w.issuesOpenNow' - Remove incorrect 'lastCommitAt IS NOT NULL' guard from NULL branch - Fix packageRepos join from LEFT to INNER for correct NULL handling - Fix busFactorScoreActivityWeightedMean denominator NULL filter - Standardize methodologyVersion from Nullable(String) to String Signed-off-by: Gašper Grom --- .../health_score_v2_signal_detail_ds.datasource | 2 +- .../libs/tinybird/pipes/health_score_v2_lifecycle.pipe | 5 ++--- .../pipes/health_score_v2_raw_inputs_snapshot.pipe | 10 +++++----- .../pipes/project_insights_health_breakdown_copy.pipe | 2 +- 4 files changed, 9 insertions(+), 10 deletions(-) diff --git a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource index b160dfa138..c0a380bbec 100644 --- a/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource @@ -72,7 +72,7 @@ SCHEMA > `merged12m` Nullable(UInt64), `closedUnmerged12m` Nullable(UInt64), `medianMergeS` Nullable(Float64), - `methodologyVersion` Nullable(String) + `methodologyVersion` String ENGINE MergeTree ENGINE_SORTING_KEY repoUrl diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 4fe031cc96..46cef60215 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -47,8 +47,7 @@ SQL > r.archived != 1 AND coalesce(c.commitsLast6m, 0) = 0 AND coalesce(c.commitsPrior6m, 0) = 0 - AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0 - AND r.lastCommitAt IS NOT NULL, + AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, NULL, toNullable( multiIf( @@ -56,7 +55,7 @@ SQL > 'archived', coalesce(w.issuesInWindow18m, 0) > 0 AND coalesce(w.unansweredCount, 0) > 0 - AND coalesce(w.issuesOpenNow, 0) > 0 + AND coalesce(n.issuesOpenNow, 0) > 0 AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 18 MONTH) diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index b173a29cf8..61b0a26599 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -51,14 +51,14 @@ SQL > p.merged12m AS prsMergedLast12m, p.closedUnmerged12m AS prsClosedUnmergedLast12m, p.medianMergeS AS medianPrMergeSeconds, - r.excluded, + rf.excluded, pkgs.trackedPackageCount FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE deletedAt IS NULL) AS allRepos LEFT JOIN ( SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL - ) AS r - ON r.url = allRepos.repoUrl + ) AS rf + ON rf.url = allRepos.repoUrl LEFT JOIN ( SELECT mr.repoUrl, count(DISTINCT mr.memberId) AS curatedBusFactorCount @@ -239,7 +239,7 @@ SQL > ( SELECT r2.url AS repoUrl, count(DISTINCT pr.packageId) AS trackedPackageCount FROM (SELECT url, argMax(id, updatedAt) AS id FROM repos GROUP BY url) r2 - LEFT JOIN packageRepos pr ON pr.repoId = r2.id + INNER JOIN packageRepos pr ON pr.repoId = r2.id GROUP BY r2.url ) AS pkgs ON pkgs.repoUrl = allRepos.repoUrl @@ -247,4 +247,4 @@ SQL > TYPE COPY TARGET_DATASOURCE health_score_v2_raw_inputs_snapshot_ds COPY_MODE append -COPY_SCHEDULE 0 1 1 * * +COPY_SCHEDULE 30 0 1 * * diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index 39e7b22151..b0296a2bd0 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -127,7 +127,7 @@ SQL > coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), sd.busFactorAvailable ) / nullIf( - sumIf(coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable), 0 + sumIf(coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable AND sd.commitsLast6m IS NOT NULL), 0 ) AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN From cdf6cf2635fda994d3f37fcd64673b1f80453885 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 11:17:22 +0100 Subject: [PATCH 09/25] fix: remove invalid UNIQUE_KEY from snapshot datasource (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../health_score_v2_raw_inputs_snapshot_ds.datasource | 1 - 1 file changed, 1 deletion(-) diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource index 7157b42894..63217e74c0 100644 --- a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -53,4 +53,3 @@ ENGINE MergeTree ENGINE_SORTING_KEY (snapshotDate, repoUrl) ENGINE_PARTITION_KEY toYYYYMM(snapshotDate) ENGINE_TTL snapshotDate + INTERVAL 24 MONTH -UNIQUE_KEY (repoUrl, snapshotDate, methodologyVersion) From 35f10090cd75c946e83c3ce01ec4c0e074e85413 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 11:40:56 +0100 Subject: [PATCH 10/25] fix: add age gate to abandoned lifecycle branch, cover PRs (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per Joana's PR review feedback: - abandoned previously fired whenever ANY unanswered issue existed, ever, with no 90-180 day age gate - add unansweredAged90to180d on both the issue and PR side of health_score_v2_lifecycle.pipe and require it for the abandoned branch instead of the old unbounded unansweredCount. - abandoned only ever looked at issues; PRs are now covered via pull_requests_analyzed.reviewedAt (already the first-response signal for PRs) mirroring the issue-side unanswered logic, no new datasource field needed. - also add a pre-commit hook step that runs tb fmt on staged Tinybird pipe/datasource files (the tbf alias equivalent) so formatting is enforced automatically instead of manually. format.sh's own exit code is not meaningful in --sequential mode (its last statement is an unrelated parallel-mode branch test), and husky runs hooks under sh -e, so a bare non-zero return there aborted the whole commit before git add ran - explicitly neutralize it and let git add fail loudly instead if formatting genuinely broke something. Third item from the same feedback (observed-reviewer filter should cover GitLab/Gerrit) required no code change - health_score_v2_maintainer.pipe's obs subquery already includes merge_request-review-* and patchset_approval-created types. Signed-off-by: Gašper Grom --- .husky/pre-commit | 15 ++++++ .../health_score_v2_lifecycle_ds.datasource | 4 +- .../pipes/health_score_v2_lifecycle.pipe | 50 +++++++++++++++---- 3 files changed, 56 insertions(+), 13 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index 4947849428..0747aa9222 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -6,6 +6,8 @@ wc -l` # grep -E "backend\/.+\.(?:js|ts|vue|scss|html)$" | # wc -l` +tinybird_files=`git --no-pager diff --name-only --cached | +grep -E "services/libs/tinybird/(pipes|datasources)/.+\.(pipe|datasource)$"` if [ $frontend_files -gt 0 ] then @@ -16,3 +18,16 @@ fi # then # cd backend && npx lint-staged # fi + +if [ -n "$tinybird_files" ] +then + (cd services/libs/tinybird && source .venv/bin/activate && pip install -q -r requirements.txt) || exit 1 + ( + cd services/libs/tinybird/scripts + source ../.venv/bin/activate + for f in $tinybird_files; do + ./format.sh --match "$(basename "$f")" --sequential || true + done + ) + git add $tinybird_files || exit 1 +fi diff --git a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource index 6137eedbbf..054f606b1a 100644 --- a/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_lifecycle_ds.datasource @@ -2,8 +2,8 @@ DESCRIPTION > - `health_score_v2_lifecycle_ds` holds the per-repo Lifecycle state for Health Score v2. Populated by `health_score_v2_lifecycle.pipe`. - `repoUrl` is the repository URL — the join key back to `repositories`. - - `lifecycleLabelV2` — one of active/stable/declining/abandoned/archived, or NULL when the - repo has no usable activity signal at all (2026-07-24, IN-1196 — see + - `lifecycleLabelV2` — one of active/stable/declining/inert/abandoned/archived, or NULL when + the repo has no usable activity signal at all (2026-07-24, IN-1196 — see health_score_v2_lifecycle.pipe for the exact condition). SCHEMA > diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 46cef60215..5050e3d0bb 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -1,6 +1,7 @@ DESCRIPTION > Per-repo Lifecycle state, computed via the spec's decision tree (first match wins): - archived (repos.archived) > abandoned (no commits in 18mo AND no issue/PR activity in 18mo) > + archived (repos.archived) > abandoned (an issue or PR unanswered 90-180 days AND no maintainer + commits in 12mo) > inert (no commits in 18mo AND no issue/PR activity in 18mo) > declining (commits down >50% vs prior 6mo AND issues opened up vs prior 6mo) > stable (release within 12mo, <50 open issues, no open critical vulns, commits down >50% vs prior 6mo) > active (fallback) — but NULL overrides all of the above when there is no usable signal at all (see @@ -31,11 +32,18 @@ DESCRIPTION > NODE health_score_v2_lifecycle_calc DESCRIPTION > - Precedence (first match wins): archived > abandoned > inert > declining > stable > active. - - Threshold note (provisional, pending validation analysis): abandoned uses 90-180 day - unanswered-issue window and 12-month no-maintainer-activity gates; inert uses 18-month - no-commits AND zero open issues/PRs gates. These day counts are under review via the - abandoned-threshold-sweep validation analysis and may shift in a follow-up tuning pass. - - Unanswered: minimum one issue opened in 18mo window + has unanswered issues now. + - Threshold note (provisional, pending validation analysis): abandoned now requires at least + one issue OR PR that has sat unanswered for 90-180 days (per Joana's 2026-08-11 PR feedback — + previously this branch only checked whether ANY unanswered issue existed, ever, with no age + gate at all) plus 12-month no-maintainer-activity; inert uses 18-month no-commits AND zero + open issues/PRs gates. The 90/180-day bounds are under review via the abandoned-threshold-sweep + validation analysis and may shift in a follow-up tuning pass. + - Unanswered issue: opened in the 18mo window, no non-author comment, still open, and has been + open 90-180 days. + - Unanswered PR: opened in the 18mo window, no reviewer activity (reviewedAt IS NULL), still + open (closedAt IS NULL), and has been open 90-180 days. Mirrors the issue-side signal using + pull_requests_analyzed.reviewedAt, which is already the first-response timestamp for PRs + (first review / changes-requested / Gerrit patchset approval) — no new datasource field needed. - Declining: commit count down >50% vs prior 6mo AND issues opened up vs prior 6mo. - Stable: release within 12mo, <50 open issues, no open critical vulns, AND commits down >50% vs prior 6mo. @@ -53,9 +61,10 @@ SQL > multiIf( r.archived = 1, 'archived', - coalesce(w.issuesInWindow18m, 0) > 0 - AND coalesce(w.unansweredCount, 0) > 0 - AND coalesce(n.issuesOpenNow, 0) > 0 + ( + coalesce(w.unansweredAged90to180d, 0) > 0 + OR coalesce(pw.unansweredAged90to180d, 0) > 0 + ) AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 18 MONTH) @@ -114,8 +123,12 @@ SQL > ) AS issuesOpenedPrior6m, countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m, countIf( - openedAt > now() - INTERVAL 18 MONTH AND respondedInSeconds IS NULL AND closedAt IS NULL - ) AS unansweredCount + openedAt > now() - INTERVAL 18 MONTH + AND respondedInSeconds IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + AND openedAt > now() - INTERVAL 180 DAY + ) AS unansweredAged90to180d FROM issues_analyzed GROUP BY channel ) AS w @@ -127,6 +140,21 @@ SQL > GROUP BY channel ) AS p ON p.repoUrl = r.url + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + openedAt > now() - INTERVAL 18 MONTH + AND reviewedAt IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + AND openedAt > now() - INTERVAL 180 DAY + ) AS unansweredAged90to180d + FROM pull_requests_analyzed + GROUP BY channel + ) AS pw + ON pw.repoUrl = r.url LEFT JOIN ( SELECT repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals From 1f6e6f8ed6e7ad255db75ac1ed8b14d74d6b566f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 12:41:04 +0100 Subject: [PATCH 11/25] fix: apply canonical tb fmt to Tinybird files (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's Datafiles check runs tb fmt --diff over every Tinybird file changed on this branch, not just the last commit's. These four were formatted with an older/different tb fmt output before the new pre-commit hook existed, so they drifted from the CLI version CI validates against - fixing directly: - health_score_v2_raw_inputs_snapshot_ds.datasource: ENGINE_PARTITION_KEY and ENGINE_SORTING_KEY were in the wrong order relative to canonical tb fmt output. - health_score_v2_raw_inputs_snapshot.pipe, project_insights_copy.pipe, project_insights_health_breakdown_copy.pipe: line-wrapping only, no logic change (collapsed/re-wrapped subqueries and long expressions to match tb fmt's canonical style). No SQL logic changed in any of these files. Signed-off-by: Gašper Grom --- ...alth_score_v2_raw_inputs_snapshot_ds.datasource | 2 +- .../pipes/health_score_v2_raw_inputs_snapshot.pipe | 14 +++++--------- .../libs/tinybird/pipes/project_insights_copy.pipe | 4 +++- .../project_insights_health_breakdown_copy.pipe | 5 ++++- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource index 63217e74c0..bf414cd9a8 100644 --- a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -50,6 +50,6 @@ SCHEMA > `trackedPackageCount` Nullable(UInt64) ENGINE MergeTree -ENGINE_SORTING_KEY (snapshotDate, repoUrl) ENGINE_PARTITION_KEY toYYYYMM(snapshotDate) +ENGINE_SORTING_KEY (snapshotDate, repoUrl) ENGINE_TTL snapshotDate + INTERVAL 24 MONTH diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 61b0a26599..f7206e48c7 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -55,9 +55,7 @@ SQL > pkgs.trackedPackageCount FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE deletedAt IS NULL) AS allRepos LEFT JOIN - ( - SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL - ) AS rf + (SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL) AS rf ON rf.url = allRepos.repoUrl LEFT JOIN ( @@ -213,9 +211,8 @@ SQL > channel AS repoUrl, countIf(closedAt IS NOT NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closed12m, countIf(openedAt > now() - INTERVAL 12 MONTH) AS opened12m, - quantile(0.5)( - closedInSeconds - ) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianCloseS + quantile(0.5) + (closedInSeconds) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianCloseS FROM issues_analyzed GROUP BY channel ) AS w @@ -228,9 +225,8 @@ SQL > countIf( closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH ) AS closedUnmerged12m, - quantile(0.5)( - mergedInSeconds - ) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianMergeS + quantile(0.5) + (mergedInSeconds) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianMergeS FROM pull_requests_analyzed GROUP BY channel ) AS p diff --git a/services/libs/tinybird/pipes/project_insights_copy.pipe b/services/libs/tinybird/pipes/project_insights_copy.pipe index 6b5ae220ff..a2edf8e01d 100644 --- a/services/libs/tinybird/pipes/project_insights_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_copy.pipe @@ -127,7 +127,9 @@ SQL > toNullable( arrayElement( arraySort( - x -> indexOf(['active', 'stable', 'declining', 'abandoned', 'inert', 'archived'], x), + x -> indexOf( + ['active', 'stable', 'declining', 'abandoned', 'inert', 'archived'], x + ), groupArray(hv2.lifecycleLabelV2) ), 1 diff --git a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe index b0296a2bd0..8435878c0a 100644 --- a/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_health_breakdown_copy.pipe @@ -127,7 +127,10 @@ SQL > coalesce(sd.commitsLast6m, 0) * coalesce(sd.busFactorScore, 0), sd.busFactorAvailable ) / nullIf( - sumIf(coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable AND sd.commitsLast6m IS NOT NULL), 0 + sumIf( + coalesce(sd.commitsLast6m, 0), sd.busFactorAvailable AND sd.commitsLast6m IS NOT NULL + ), + 0 ) AS busFactorScoreActivityWeightedMean FROM insightsProjects ip FINAL LEFT JOIN From ae0703e12b816a723da04091fd03ae31b7ca2e5c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 12:48:56 +0100 Subject: [PATCH 12/25] fix: prevent pre-commit hook from failing on unrelated commits (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .husky/pre-commit | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index 0747aa9222..5cc0dbfb2c 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -7,7 +7,7 @@ wc -l` # wc -l` tinybird_files=`git --no-pager diff --name-only --cached | -grep -E "services/libs/tinybird/(pipes|datasources)/.+\.(pipe|datasource)$"` +grep -E "services/libs/tinybird/(pipes|datasources)/.+\.(pipe|datasource)$" || true` if [ $frontend_files -gt 0 ] then @@ -21,7 +21,12 @@ fi if [ -n "$tinybird_files" ] then - (cd services/libs/tinybird && source .venv/bin/activate && pip install -q -r requirements.txt) || exit 1 + ( + cd services/libs/tinybird + [ -d .venv ] || python3 -m venv .venv + source .venv/bin/activate + pip install -q -r requirements.txt + ) || exit 1 ( cd services/libs/tinybird/scripts source ../.venv/bin/activate @@ -31,3 +36,5 @@ then ) git add $tinybird_files || exit 1 fi + +exit 0 From d27053fe9cc8c47e5149634f3e6c14588b9facc6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 12:54:34 +0100 Subject: [PATCH 13/25] fix: skip tb fmt in pre-commit when tinybird venv is not set up (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .husky/pre-commit | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index 5cc0dbfb2c..8cd61929a7 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -21,20 +21,24 @@ fi if [ -n "$tinybird_files" ] then - ( - cd services/libs/tinybird - [ -d .venv ] || python3 -m venv .venv - source .venv/bin/activate - pip install -q -r requirements.txt - ) || exit 1 - ( - cd services/libs/tinybird/scripts - source ../.venv/bin/activate - for f in $tinybird_files; do - ./format.sh --match "$(basename "$f")" --sequential || true - done - ) - git add $tinybird_files || exit 1 + if [ ! -d services/libs/tinybird/.venv ] + then + echo "tinybird .venv not found, skipping tb fmt (see services/libs/tinybird/README.md to set it up)" + else + ( + cd services/libs/tinybird + source .venv/bin/activate + pip install -q -r requirements.txt + ) || echo "tinybird pip install failed, skipping tb fmt" + ( + cd services/libs/tinybird/scripts + source ../.venv/bin/activate + for f in $tinybird_files; do + ./format.sh --match "$(basename "$f")" --sequential || true + done + ) + git add $tinybird_files || exit 1 + fi fi exit 0 From 5f8cbc366055048105d700a0d8ebda13c64ac002 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 12:56:57 +0100 Subject: [PATCH 14/25] fix: gate tb fmt on pip install actually succeeding (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .husky/pre-commit | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index 8cd61929a7..bf2c3056b7 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -24,12 +24,10 @@ then if [ ! -d services/libs/tinybird/.venv ] then echo "tinybird .venv not found, skipping tb fmt (see services/libs/tinybird/README.md to set it up)" + elif ! (cd services/libs/tinybird && source .venv/bin/activate && pip install -q -r requirements.txt) + then + echo "tinybird pip install failed, skipping tb fmt" else - ( - cd services/libs/tinybird - source .venv/bin/activate - pip install -q -r requirements.txt - ) || echo "tinybird pip install failed, skipping tb fmt" ( cd services/libs/tinybird/scripts source ../.venv/bin/activate From 107c00531f0d7bbf003ec4b9d02b204376c368f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 13:49:29 +0100 Subject: [PATCH 15/25] fix: address PR #4460 review findings (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .husky/pre-commit | 5 ++--- services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe | 1 + .../libs/tinybird/pipes/health_score_v2_signal_detail.pipe | 2 +- services/libs/tinybird/pipes/project_insights_copy.pipe | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index bf2c3056b7..d38af5cace 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -24,18 +24,17 @@ then if [ ! -d services/libs/tinybird/.venv ] then echo "tinybird .venv not found, skipping tb fmt (see services/libs/tinybird/README.md to set it up)" - elif ! (cd services/libs/tinybird && source .venv/bin/activate && pip install -q -r requirements.txt) + elif ! (cd services/libs/tinybird && . .venv/bin/activate && pip install -q -r requirements.txt) then echo "tinybird pip install failed, skipping tb fmt" else ( cd services/libs/tinybird/scripts - source ../.venv/bin/activate + . ../.venv/bin/activate for f in $tinybird_files; do ./format.sh --match "$(basename "$f")" --sequential || true done ) - git add $tinybird_files || exit 1 fi fi diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 5050e3d0bb..27c3bf9a54 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -147,6 +147,7 @@ SQL > countIf( openedAt > now() - INTERVAL 18 MONTH AND reviewedAt IS NULL + AND approvedAt IS NULL AND closedAt IS NULL AND openedAt <= now() - INTERVAL 90 DAY AND openedAt > now() - INTERVAL 180 DAY diff --git a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe index 6318aeb6aa..ff51cf9cfc 100644 --- a/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_signal_detail.pipe @@ -68,7 +68,7 @@ SQL > d.merged12m AS merged12m, d.closedUnmerged12m AS closedUnmerged12m, d.medianMergeS AS medianMergeS, - m.methodologyVersion AS methodologyVersion + coalesce(m.methodologyVersion, '2.0.0') AS methodologyVersion FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE isNull (deletedAt)) AS base LEFT JOIN health_score_v2_maintainer_ds AS m ON m.repoUrl = base.repoUrl LEFT JOIN health_score_v2_security_ds AS s ON s.repoUrl = base.repoUrl diff --git a/services/libs/tinybird/pipes/project_insights_copy.pipe b/services/libs/tinybird/pipes/project_insights_copy.pipe index a2edf8e01d..b6d9531cdc 100644 --- a/services/libs/tinybird/pipes/project_insights_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_copy.pipe @@ -91,7 +91,7 @@ DESCRIPTION > per-repo values in health_score_v2_repo_copy_ds (materialized by health_score_v2.pipe — the expensive per-signal joins run there as their own copy job, not inline here). Per spec section 5/6: - healthScoreV2: straight mean across the project's enabled, non-excluded repos. - - lifecycleLabelV2: best-state-wins (active > stable > declining > abandoned > archived), with + - lifecycleLabelV2: best-state-wins (active > stable > declining > inert > abandoned > archived), with NULL (no usable signal, see health_score_v2_lifecycle.pipe) as the most-cautious outcome. `groupArray` silently drops NULL entries, so a project with at least one repo in a real state already rolls up correctly for free — NULL only surfaces at the project level when EVERY repo From 85b568ac008a2679a53e125d246019d572bc948f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 16:29:37 +0100 Subject: [PATCH 16/25] fix: lifecycle age threshold, snapshot idempotency and unanswered counts (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- ...score_v2_raw_inputs_snapshot_ds.datasource | 20 +++++- .../pipes/health_score_v2_lifecycle.pipe | 48 ++++++++----- .../pipes/health_score_v2_maintainer.pipe | 24 ++----- .../health_score_v2_raw_inputs_snapshot.pipe | 72 +++++++++++++++---- .../tinybird/pipes/project_insights_copy.pipe | 2 +- 5 files changed, 112 insertions(+), 54 deletions(-) diff --git a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource index bf414cd9a8..d3d3b97aa9 100644 --- a/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_raw_inputs_snapshot_ds.datasource @@ -10,11 +10,21 @@ DESCRIPTION > practices flags (policy/branch-protection state), vulnerable dependencies count. - Development signals: release recency/cadence, commit counts (6m windows), issue counts/closure times, PR counts/merge times. + - Responsiveness/lifecycle signals: unanswered aged issue/PR counts and 18mo opened totals + (the per-repo unanswered count/ratio the compliance spec item 6 requires, and the historical + input the abandoned-threshold-sweep validation analysis runs against). Definitions mirror + health_score_v2_lifecycle.pipe: open, no non-author response, aged 90+ days, opened within + the 18mo window. - Does NOT include computed category/overall scores — those are derived from these raw inputs via the scoring logic in the respective pipes. This datasource is for tracing where scores come from. - snapshotId = `toStartOfInterval(now(), INTERVAL 1 day)` at pipe execution time. - Partitioned by year(snapshotDate) and month(snapshotDate) for efficient time-based queries. - TTL set to 24 months (enough for trend analysis and validation studies). + - ReplacingMergeTree keyed on (snapshotDate, repoUrl, methodologyVersion): a same-day manual + re-run of the copy converges to one row per key after background merges instead of + accumulating duplicates (UNIQUE_KEY is not a thing in Tinybird — attempted and reverted in + cdf6cf263). Validation queries must still read with FINAL (or dedupe via argMax/LIMIT BY) + since replacement is asynchronous. SCHEMA > `repoUrl` String, @@ -47,9 +57,13 @@ SCHEMA > `prsClosedUnmergedLast12m` Nullable(UInt64), `medianPrMergeSeconds` Nullable(Float64), `excluded` Nullable(UInt8), - `trackedPackageCount` Nullable(UInt64) + `trackedPackageCount` Nullable(UInt64), + `unansweredIssuesAged90d` Nullable(UInt64), + `unansweredPrsAged90d` Nullable(UInt64), + `issuesOpenedLast18m` Nullable(UInt64), + `prsOpenedLast18m` Nullable(UInt64) -ENGINE MergeTree +ENGINE ReplacingMergeTree ENGINE_PARTITION_KEY toYYYYMM(snapshotDate) -ENGINE_SORTING_KEY (snapshotDate, repoUrl) +ENGINE_SORTING_KEY (snapshotDate, repoUrl, methodologyVersion) ENGINE_TTL snapshotDate + INTERVAL 24 MONTH diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 27c3bf9a54..535da4601f 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -1,6 +1,6 @@ DESCRIPTION > Per-repo Lifecycle state, computed via the spec's decision tree (first match wins): - archived (repos.archived) > abandoned (an issue or PR unanswered 90-180 days AND no maintainer + archived (repos.archived) > abandoned (an issue or PR open and unanswered for 90+ days AND no commits in 12mo) > inert (no commits in 18mo AND no issue/PR activity in 18mo) > declining (commits down >50% vs prior 6mo AND issues opened up vs prior 6mo) > stable (release within 12mo, <50 open issues, no open critical vulns, commits down >50% vs prior 6mo) > active @@ -28,20 +28,36 @@ DESCRIPTION > commitsLast6m=0 (down from 1895 prior), because it still has 3 real PRs in the 18mo window — real activity, just not enough to clear any of the other branches' thresholds. That's a legitimate 'active' classification under the existing spec, not the bug being fixed here. + - `r.lastCommitAt IS NULL` added to the NULL-guard condition (2026-08-11, per Joana's PR + review): the guard as written (zero commits in both 6mo windows + zero issue/PR activity in + 18mo) was a strict superset of the `inert` branch's own condition (no commits in 18mo + zero + issue/PR activity), since zero commits across the trailing 12mo already implies + `lastCommitAt` is either NULL or older than 18mo. That made the guard fire first on every repo + that could otherwise have matched `inert`, so `inert` was dead code — never actually emitted. + Restricting the guard to the `lastCommitAt IS NULL` case (matching the blocknetdx/blocknet + example this branch was originally written for) lets repos with a known, stale last-commit + timestamp fall through into the real decision tree, where `inert` can now actually fire. NODE health_score_v2_lifecycle_calc DESCRIPTION > - Precedence (first match wins): archived > abandoned > inert > declining > stable > active. - - Threshold note (provisional, pending validation analysis): abandoned now requires at least - one issue OR PR that has sat unanswered for 90-180 days (per Joana's 2026-08-11 PR feedback — - previously this branch only checked whether ANY unanswered issue existed, ever, with no age - gate at all) plus 12-month no-maintainer-activity; inert uses 18-month no-commits AND zero - open issues/PRs gates. The 90/180-day bounds are under review via the abandoned-threshold-sweep - validation analysis and may shift in a follow-up tuning pass. + - Threshold note (provisional, pending validation analysis): abandoned requires at least one + issue OR PR that has sat unanswered for 90+ days, plus a 12-month no-commit gate. The age + gate is a single threshold, NOT a closed 90-180 band: a band would let a repo exit + 'abandoned' once its ignored issues age past the upper bound (they still block 'inert' via + issuesInWindow18m, so the repo would flip back to 'active'). The spec's "~90-180 days" is + the tuning range for this one threshold — the abandoned-threshold-sweep validation analysis + picks the final value. The no-commit gate uses repos.lastCommitAt — last commit by anyone, + not maintainer-specific activity; whether maintainer review/comment activity should also + block 'abandoned' is an open question on the PR. + - Inert: lastCommitAt older than 18mo AND zero issues/PRs opened in the 18mo window. + lastCommitAt IS NULL deliberately does NOT qualify: a NULL-with-no-signal repo is already + caught by the NULL guard above, and a NULL-with-recent-commits repo (upstream data gap in + `repos`) must not be mislabeled inert just because one timestamp field is missing. - Unanswered issue: opened in the 18mo window, no non-author comment, still open, and has been - open 90-180 days. + open 90+ days. - Unanswered PR: opened in the 18mo window, no reviewer activity (reviewedAt IS NULL), still - open (closedAt IS NULL), and has been open 90-180 days. Mirrors the issue-side signal using + open (closedAt IS NULL), and has been open 90+ days. Mirrors the issue-side signal using pull_requests_analyzed.reviewedAt, which is already the first-response timestamp for PRs (first review / changes-requested / Gerrit patchset approval) — no new datasource field needed. - Declining: commit count down >50% vs prior 6mo AND issues opened up vs prior 6mo. @@ -53,6 +69,7 @@ SQL > r.url AS repoUrl, if( r.archived != 1 + AND r.lastCommitAt IS NULL AND coalesce(c.commitsLast6m, 0) = 0 AND coalesce(c.commitsPrior6m, 0) = 0 AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, @@ -61,13 +78,10 @@ SQL > multiIf( r.archived = 1, 'archived', - ( - coalesce(w.unansweredAged90to180d, 0) > 0 - OR coalesce(pw.unansweredAged90to180d, 0) > 0 - ) + (coalesce(w.unansweredAged90d, 0) > 0 OR coalesce(pw.unansweredAged90d, 0) > 0) AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', - (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 18 MONTH) + r.lastCommitAt < now() - INTERVAL 18 MONTH AND coalesce(w.issuesInWindow18m, 0) + coalesce(p.prsInWindow18m, 0) = 0, 'inert', coalesce(c.commitsLast6m, 0) < coalesce(c.commitsPrior6m, 0) * 0.5 @@ -127,8 +141,7 @@ SQL > AND respondedInSeconds IS NULL AND closedAt IS NULL AND openedAt <= now() - INTERVAL 90 DAY - AND openedAt > now() - INTERVAL 180 DAY - ) AS unansweredAged90to180d + ) AS unansweredAged90d FROM issues_analyzed GROUP BY channel ) AS w @@ -150,8 +163,7 @@ SQL > AND approvedAt IS NULL AND closedAt IS NULL AND openedAt <= now() - INTERVAL 90 DAY - AND openedAt > now() - INTERVAL 180 DAY - ) AS unansweredAged90to180d + ) AS unansweredAged90d FROM pull_requests_analyzed GROUP BY channel ) AS pw diff --git a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe index b44d96c5fc..73dfc398cb 100644 --- a/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_maintainer.pipe @@ -87,29 +87,13 @@ SQL > ) AS busFactorCount, (bf.repoUrl != '' OR obs.repoUrl != '') AS busFactorAvailable, multiIf( - greatest( - coalesce(bf.curatedBusFactorCount, 0), - coalesce(obs.observedActorsCount, 0) - ) - >= 5, + busFactorCount >= 5, 18, - greatest( - coalesce(bf.curatedBusFactorCount, 0), - coalesce(obs.observedActorsCount, 0) - ) - >= 3, + busFactorCount >= 3, 15, - greatest( - coalesce(bf.curatedBusFactorCount, 0), - coalesce(obs.observedActorsCount, 0) - ) - = 2, + busFactorCount = 2, 6, - greatest( - coalesce(bf.curatedBusFactorCount, 0), - coalesce(obs.observedActorsCount, 0) - ) - = 1, + busFactorCount = 1, 3, 0 ) AS busFactorScore, diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index f7206e48c7..6f41ec0ec7 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -7,7 +7,20 @@ DESCRIPTION > health_score_v2_security, health_score_v2_development) but captures RAW measured values only — not computed scores. This allows downstream validation queries to reconstruct or audit scoring without needing access to the category pipes' intermediate logic. - - Append-mode COPY, partitioned by date, with 24-month TTL. + - Append-mode COPY, partitioned by date, with 24-month TTL. The target is a + ReplacingMergeTree keyed on (snapshotDate, repoUrl, methodologyVersion), so a same-day + re-run converges instead of duplicating — see the datasource description. + - Also captures the per-repo unanswered issue/PR counts (open, no non-author response, aged + 90+ days, opened within 18mo — same definitions as health_score_v2_lifecycle.pipe) plus the + 18mo opened totals, giving the unanswered ratio required by compliance spec item 6 and the + historical input for the abandoned-threshold-sweep validation analysis. + - Issue/PR subqueries scope their population by `openedAt > 12 MONTH` first, then count/quantile + within that set (2026-08-11, per Joana's PR review) — matching `health_score_v2_development.pipe` + exactly, so the snapshot's raw counts and medians reconstruct the same population the development + category score was computed from. The prior version scoped `opened12m`/quantiles independently + per-column (via a `FILTER`/inline date check on `closedAt`/`mergedAt` instead of `openedAt`), + which measured a different, unbounded-by-openedAt population than the category pipe — the two + could never be reconciled. TAGS "Validation", "Health Score v2" @@ -52,7 +65,11 @@ SQL > p.closedUnmerged12m AS prsClosedUnmergedLast12m, p.medianMergeS AS medianPrMergeSeconds, rf.excluded, - pkgs.trackedPackageCount + pkgs.trackedPackageCount, + uw.unansweredAged90d AS unansweredIssuesAged90d, + upw.unansweredAged90d AS unansweredPrsAged90d, + uw.openedLast18m AS issuesOpenedLast18m, + upw.openedLast18m AS prsOpenedLast18m FROM (SELECT DISTINCT url AS repoUrl FROM repositories FINAL WHERE deletedAt IS NULL) AS allRepos LEFT JOIN (SELECT url, excluded FROM repositories FINAL WHERE deletedAt IS NULL) AS rf @@ -209,11 +226,12 @@ SQL > ( SELECT channel AS repoUrl, - countIf(closedAt IS NOT NULL AND closedAt > now() - INTERVAL 12 MONTH) AS closed12m, - countIf(openedAt > now() - INTERVAL 12 MONTH) AS opened12m, - quantile(0.5) - (closedInSeconds) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianCloseS + countIf(closedAt IS NOT NULL) AS closed12m, + count() AS opened12m, + quantileIf(0.5) + (closedInSeconds, closedAt > now() - INTERVAL 12 MONTH) AS medianCloseS FROM issues_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH GROUP BY channel ) AS w ON w.repoUrl = allRepos.repoUrl @@ -221,16 +239,46 @@ SQL > ( SELECT channel AS repoUrl, - countIf(mergedAt IS NOT NULL AND mergedAt > now() - INTERVAL 12 MONTH) AS merged12m, - countIf( - closedAt IS NOT NULL AND mergedAt IS NULL AND closedAt > now() - INTERVAL 12 MONTH - ) AS closedUnmerged12m, - quantile(0.5) - (mergedInSeconds) FILTER (WHERE openedAt > now() - INTERVAL 12 MONTH) AS medianMergeS + countIf(mergedAt IS NOT NULL) AS merged12m, + countIf(closedAt IS NOT NULL AND mergedAt IS NULL) AS closedUnmerged12m, + quantileIf(0.5) + (mergedInSeconds, mergedAt > now() - INTERVAL 12 MONTH) AS medianMergeS FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 12 MONTH GROUP BY channel ) AS p ON p.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + respondedInSeconds IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d, + count() AS openedLast18m + FROM issues_analyzed + WHERE openedAt > now() - INTERVAL 18 MONTH + GROUP BY channel + ) AS uw + ON uw.repoUrl = allRepos.repoUrl + LEFT JOIN + ( + SELECT + channel AS repoUrl, + countIf( + reviewedAt IS NULL + AND approvedAt IS NULL + AND closedAt IS NULL + AND openedAt <= now() - INTERVAL 90 DAY + ) AS unansweredAged90d, + count() AS openedLast18m + FROM pull_requests_analyzed + WHERE openedAt > now() - INTERVAL 18 MONTH + GROUP BY channel + ) AS upw + ON upw.repoUrl = allRepos.repoUrl LEFT JOIN ( SELECT r2.url AS repoUrl, count(DISTINCT pr.packageId) AS trackedPackageCount diff --git a/services/libs/tinybird/pipes/project_insights_copy.pipe b/services/libs/tinybird/pipes/project_insights_copy.pipe index b6d9531cdc..b5348dfb10 100644 --- a/services/libs/tinybird/pipes/project_insights_copy.pipe +++ b/services/libs/tinybird/pipes/project_insights_copy.pipe @@ -128,7 +128,7 @@ SQL > arrayElement( arraySort( x -> indexOf( - ['active', 'stable', 'declining', 'abandoned', 'inert', 'archived'], x + ['active', 'stable', 'declining', 'inert', 'abandoned', 'archived'], x ), groupArray(hv2.lifecycleLabelV2) ), From 7b940b8fd95160078d36f48303dbc37dd22c1972 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 16:36:18 +0100 Subject: [PATCH 17/25] fix: gate abandoned on both commit signals to cover lastCommitAt data gap (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../tinybird/pipes/health_score_v2_lifecycle.pipe | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe index 535da4601f..f99eecc1b3 100644 --- a/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe @@ -47,9 +47,12 @@ DESCRIPTION > 'abandoned' once its ignored issues age past the upper bound (they still block 'inert' via issuesInWindow18m, so the repo would flip back to 'active'). The spec's "~90-180 days" is the tuning range for this one threshold — the abandoned-threshold-sweep validation analysis - picks the final value. The no-commit gate uses repos.lastCommitAt — last commit by anyone, - not maintainer-specific activity; whether maintainer review/comment activity should also - block 'abandoned' is an open question on the PR. + picks the final value. The no-commit gate requires BOTH commit signals to be silent: zero + authored-commit activity in both trailing 6mo windows (covers repos where lastCommitAt is + NULL/stale from the upstream `repos` data gap but real commits exist in activityRelations) + AND repos.lastCommitAt NULL or older than 12mo (covers the reverse gap). Both signals are + last-commit-by-anyone, not maintainer-specific activity; whether maintainer review/comment + activity should also block 'abandoned' is an open question on the PR. - Inert: lastCommitAt older than 18mo AND zero issues/PRs opened in the 18mo window. lastCommitAt IS NULL deliberately does NOT qualify: a NULL-with-no-signal repo is already caught by the NULL guard above, and a NULL-with-recent-commits repo (upstream data gap in @@ -79,6 +82,8 @@ SQL > r.archived = 1, 'archived', (coalesce(w.unansweredAged90d, 0) > 0 OR coalesce(pw.unansweredAged90d, 0) > 0) + AND coalesce(c.commitsLast6m, 0) = 0 + AND coalesce(c.commitsPrior6m, 0) = 0 AND (r.lastCommitAt IS NULL OR r.lastCommitAt < now() - INTERVAL 12 MONTH), 'abandoned', r.lastCommitAt < now() - INTERVAL 18 MONTH From ac5672bc241108cab417f1958b4e56e93851945e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 16:48:06 +0100 Subject: [PATCH 18/25] fix: exclude PR author self-reviews from first-response timestamps (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- services/libs/tinybird/README.md | 2 +- .../health_score_v2_repo_copy_ds.datasource | 10 +-- .../pull_request_analysis_copy_pipe.pipe | 68 +++++++++++++------ 3 files changed, 52 insertions(+), 28 deletions(-) diff --git a/services/libs/tinybird/README.md b/services/libs/tinybird/README.md index c230e99a76..e7d5383886 100644 --- a/services/libs/tinybird/README.md +++ b/services/libs/tinybird/README.md @@ -360,7 +360,7 @@ Repositories can be marked as excluded from health scoring by setting the `repos **Setting the flag:** ```sql -UPDATE repositories SET excluded = 1 WHERE url = 'https://github.com/org/repo-meta'; +UPDATE repositories SET excluded = TRUE WHERE url = 'https://github.com/org/repo-meta'; ``` The flag is read by `project_insights_copy.pipe` and `health_score_v2_raw_inputs_snapshot.pipe` as part of their graceful-degradation and audit logic. diff --git a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource index 99d10644c8..c935def4fa 100644 --- a/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource +++ b/services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource @@ -13,11 +13,11 @@ DESCRIPTION > across unrelated packages), supply chain integrity (hardcoded 0 — provenance/2FA data not yet piped). - `developmentActivityScoreV2` (0-25) — release cadence, commit activity, issue resolution, PR merge health. - `healthScoreV2` (0-100) is the sum of the three categories above, clamped to 100. - - `lifecycleLabelV2` — per-repo lifecycle state (active/stable/declining/abandoned/archived), computed - via the spec's decision tree (archived flag > abandoned > declining > stable > active, first match - wins), or NULL when the repo has zero commits and zero issues/PRs in every window checked (2026-07-24, - IN-1196 — no usable activity signal at all). Project-level rollup uses best-state-wins precedence in - project_insights_copy.pipe. + - `lifecycleLabelV2` — per-repo lifecycle state (active/stable/declining/inert/abandoned/archived), + computed via the spec's decision tree (archived flag > abandoned > inert > declining > stable > + active, first match wins), or NULL when the repo has zero commits and zero issues/PRs in every window + checked (2026-07-24, IN-1196 — no usable activity signal at all). Project-level rollup uses + best-state-wins precedence in project_insights_copy.pipe. - `impactScore` (0-100) — MAX(packages.impact) * 100 over packages published by this repo. NULL when the repo has no linked packages. - Graceful degradation (spec Layer 1+2): per-category pipes emit NULL when covered sub-signal weight diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe index 4663d111b1..a5dcd9a248 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe @@ -1,5 +1,11 @@ DESCRIPTION > Compacts activities from same PR into one, keeping necessary information in a single row. Helps to serve PR-wide widgets in the development tab. + - reviewedAt/approvedAt count only NON-AUTHOR activity (2026-08-11, IN-1226): review/approval + events where the actor is the PR author (GitHub COMMENTED self-reviews, GitLab self-approval, + Gerrit self +2) no longer set the first-response timestamps. Mirrors the non-author fix in + issue_analysis_copy_pipe.pipe — the compliance commitment is that responsiveness counts a + response from someone other than the author. Affects every consumer of reviewedAt/ + reviewedInSeconds/approvedAt, including the health score v2 lifecycle "unanswered PR" signal. NODE pull_request_opened SQL > @@ -33,31 +39,49 @@ SQL > NODE pull_request_first_reviewed SQL > - SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS reviewedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - type = 'pull_request-reviewed' - OR type = 'merge_request-review-changes-requested' - OR type = 'patchset_approval-created' - GROUP BY sourceParentId + SELECT r.prSourceId AS sourceParentId, MIN(r.timestamp) AS reviewedAt + FROM + ( + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + memberId + FROM activityRelations_deduplicated_cleaned_ds + WHERE + type = 'pull_request-reviewed' + OR type = 'merge_request-review-changes-requested' + OR type = 'patchset_approval-created' + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_review_approved SQL > - SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS approvedAt - FROM activityRelations_deduplicated_cleaned_ds - WHERE - (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - GROUP BY sourceParentId + SELECT r.prSourceId AS sourceParentId, MIN(r.timestamp) AS approvedAt + FROM + ( + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + memberId + FROM activityRelations_deduplicated_cleaned_ds + WHERE + (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') + OR type = 'merge_request-review-approved' + OR type = 'patchset_approval-created' + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_closed SQL > From d108a0713ccda873e05d63a1e548401a807cc01c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 16:48:11 +0100 Subject: [PATCH 19/25] revert: drop tb fmt pre-commit hook (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .husky/pre-commit | 23 ----------------------- 1 file changed, 23 deletions(-) diff --git a/.husky/pre-commit b/.husky/pre-commit index d38af5cace..4947849428 100644 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -6,8 +6,6 @@ wc -l` # grep -E "backend\/.+\.(?:js|ts|vue|scss|html)$" | # wc -l` -tinybird_files=`git --no-pager diff --name-only --cached | -grep -E "services/libs/tinybird/(pipes|datasources)/.+\.(pipe|datasource)$" || true` if [ $frontend_files -gt 0 ] then @@ -18,24 +16,3 @@ fi # then # cd backend && npx lint-staged # fi - -if [ -n "$tinybird_files" ] -then - if [ ! -d services/libs/tinybird/.venv ] - then - echo "tinybird .venv not found, skipping tb fmt (see services/libs/tinybird/README.md to set it up)" - elif ! (cd services/libs/tinybird && . .venv/bin/activate && pip install -q -r requirements.txt) - then - echo "tinybird pip install failed, skipping tb fmt" - else - ( - cd services/libs/tinybird/scripts - . ../.venv/bin/activate - for f in $tinybird_files; do - ./format.sh --match "$(basename "$f")" --sequential || true - done - ) - fi -fi - -exit 0 From afecdfb29b37b404a3823f5643945e527e70b3fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 17:01:42 +0100 Subject: [PATCH 20/25] docs: document non-author guard convention with prod evidence (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe index a5dcd9a248..caa784e445 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe @@ -6,6 +6,11 @@ DESCRIPTION > issue_analysis_copy_pipe.pipe — the compliance commitment is that responsiveness counts a response from someone other than the author. Affects every consumer of reviewedAt/ reviewedInSeconds/approvedAt, including the health score v2 lifecycle "unanswered PR" signal. + - The both-IDs-non-empty guard is deliberate and mirrors the issue-side convention (strict + identity verification was explicitly requested in review there): authorship can't be verified + when either identity is unresolved, so such events don't count as a response. Empirically + vacuous today — 0 of 38.5M PR openers and 0 of 16.5M issue openers have an empty memberId in + prod (checked 2026-08-11) — so no PR loses a real review to this guard in practice. NODE pull_request_opened SQL > From 3bb299e64971ca282ac236f856718cb05a0a75a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 17:11:35 +0100 Subject: [PATCH 21/25] fix: apply non-author review filter to incremental and bootstrap PR analysis paths (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- ...ll_request_analysis_baseline_merge_MV.pipe | 86 +++++++++++- ...ull_request_analysis_initial_snapshot.pipe | 130 +++++++++++------- 2 files changed, 166 insertions(+), 50 deletions(-) diff --git a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe index 960dc9f19d..bf2311d791 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe @@ -1,6 +1,11 @@ DESCRIPTION > Compacts activities from same PR into one, keeping necessary information in a single row. Uses existing pull_requests_analyzed data as baseline and merges new events on top. + - reviewedAt/approvedAt count only NON-AUTHOR activity (2026-08-11, IN-1226): this incremental + path feeds the minute-0 snapshot merger that replaces pull_requests_analyzed right before the + 02:00-02:15 health score jobs read it, so it must apply the same non-author rule as the + minute-20 full recompute in pull_request_analysis_copy_pipe.pipe — otherwise self-review + timestamps reappear every hour. See the non-author filter node below. NODE snapshot_resolver DESCRIPTION > @@ -50,6 +55,85 @@ SQL > 'changeset-merged' ) +NODE pr_authors +DESCRIPTION > + PR author per PR touched by a review/approval event in this batch: baseline authors from + pull_requests_analyzed (semi-join, same pattern as baseline_filtered) plus authors from open + events arriving in the same batch. + +SQL > + SELECT sourceId AS prSourceId, memberId + FROM pull_requests_analyzed + WHERE + memberId != '' + AND sourceId IN ( + SELECT DISTINCT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) + FROM new_pull_request_related_activity + WHERE + type IN ( + 'pull_request-reviewed', + 'merge_request-review-changes-requested', + 'merge_request-review-approved', + 'patchset_approval-created' + ) + ) + UNION ALL + SELECT sourceId AS prSourceId, memberId + FROM new_pull_request_related_activity + WHERE + type IN ('pull_request-opened', 'merge_request-opened', 'changeset-created') AND memberId != '' + +NODE new_pull_request_related_activity_non_author +DESCRIPTION > + Drops review/approval events performed by the PR author (strict both-IDs-non-empty rule, + matching issue_analysis_copy_pipe.pipe). Non-review event types pass through untouched. + A review whose PR has no known author yet is dropped here — it was already discarded + downstream (the merge node requires an open event), so behavior for those rows is unchanged. + +SQL > + SELECT + a.id, + a.sourceId, + a.sourceParentId, + a.channel, + a.ts, + a.segmentId, + a.gitChangedLinesBucket, + a.memberId, + a.organizationId, + a.platform, + a.updatedAt, + a.type, + a.pullRequestReviewState + FROM + ( + SELECT + *, + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prKey + FROM new_pull_request_related_activity + ) AS a + ANY + LEFT JOIN pr_authors AS auth ON auth.prSourceId = a.prKey + WHERE + NOT ( + a.type IN ( + 'pull_request-reviewed', + 'merge_request-review-changes-requested', + 'merge_request-review-approved', + 'patchset_approval-created' + ) + ) + OR (a.memberId != '' AND auth.memberId != '' AND a.memberId != auth.memberId) + NODE new_events_aggregated DESCRIPTION > Aggregates new events from the latest snapshot by PR source ID. @@ -231,7 +315,7 @@ SQL > NULL ) ) AS resolvedAt - FROM new_pull_request_related_activity + FROM new_pull_request_related_activity_non_author GROUP BY prSourceId HAVING prSourceId != '' diff --git a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe index fdc2d3116f..f7ffe049e6 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe @@ -78,66 +78,98 @@ SQL > GROUP BY sourceParentId NODE pull_request_first_reviewed +DESCRIPTION > + Non-author reviews only (2026-08-11, IN-1226) — same rule as + pull_request_analysis_copy_pipe.pipe and the baseline-merge MV. + SQL > % SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - argMin(updatedAt, timestamp) AS updatedAt, - MIN(timestamp) AS reviewedAt - FROM activityRelations_enriched_deduplicated_bucket_union - WHERE - sourceParentId <> '' - and ( - type = 'pull_request-reviewed' - OR type = 'merge_request-review-changes-requested' - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - ) - {% if defined(bucket_id) %} - AND cityHash64(segmentId) - % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} - = {{ - UInt8( - bucket_id, - 0, - description="Bucket index to process (0 to num_buckets-1)", - required=False, + r.prSourceId AS sourceParentId, + argMin(r.updatedAt, r.timestamp) AS updatedAt, + MIN(r.timestamp) AS reviewedAt + FROM + ( + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + updatedAt, + memberId + FROM activityRelations_enriched_deduplicated_bucket_union + WHERE + sourceParentId <> '' + and ( + type = 'pull_request-reviewed' + OR type = 'merge_request-review-changes-requested' + OR type = 'merge_request-review-approved' + OR type = 'patchset_approval-created' ) - }} - {% end %} - GROUP BY sourceParentId + {% if defined(bucket_id) %} + AND cityHash64(segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_review_approved +DESCRIPTION > + Non-author approvals only (2026-08-11, IN-1226) — same rule as + pull_request_analysis_copy_pipe.pipe and the baseline-merge MV. + SQL > % SELECT - if( - type = 'patchset_approval-created', splitByChar('-', sourceParentId)[1], sourceParentId - ) AS sourceParentId, - argMin(updatedAt, timestamp) AS updatedAt, - MIN(timestamp) AS approvedAt - FROM activityRelations_enriched_deduplicated_bucket_union - WHERE + r.prSourceId AS sourceParentId, + argMin(r.updatedAt, r.timestamp) AS updatedAt, + MIN(r.timestamp) AS approvedAt + FROM ( - (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - ) - {% if defined(bucket_id) %} - AND cityHash64(segmentId) - % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} - = {{ - UInt8( - bucket_id, - 0, - description="Bucket index to process (0 to num_buckets-1)", - required=False, + SELECT + if( + type = 'patchset_approval-created', + splitByChar('-', sourceParentId)[1], + sourceParentId + ) AS prSourceId, + timestamp, + updatedAt, + memberId + FROM activityRelations_enriched_deduplicated_bucket_union + WHERE + ( + (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') + OR type = 'merge_request-review-approved' + OR type = 'patchset_approval-created' ) - }} - {% end %} - GROUP BY sourceParentId + {% if defined(bucket_id) %} + AND cityHash64(segmentId) + % {{ UInt8(num_buckets, 5, description="Total number of buckets", required=False) }} + = {{ + UInt8( + bucket_id, + 0, + description="Bucket index to process (0 to num_buckets-1)", + required=False, + ) + }} + {% end %} + ) AS r + INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId + WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId + GROUP BY r.prSourceId NODE pull_request_first_closed SQL > From 0b77c682435f0dd96eff4d40938685ec56718972 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 23:01:08 +0100 Subject: [PATCH 22/25] fix: point pull_request_analysis_copy_pipe at bucket_union datasource (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit activityRelations_deduplicated_cleaned_ds was deleted from production and no longer resolves. This pipe was the only one in the PR set still referencing the old singular name instead of the bucketed union pattern (matching issue_analysis_copy_pipe.pipe and health_score_v2_maintainer.pipe). Straight rename, no logic change — the non-author review/approval filter from ac5672bc24 is unaffected. Signed-off-by: Gašper Grom --- .../pipes/pull_request_analysis_copy_pipe.pipe | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe index caa784e445..9ad0b8fde8 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe @@ -24,13 +24,13 @@ SQL > memberId, organizationId, platform - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-opened' OR type = 'merge_request-opened' OR type = 'changeset-created' NODE pull_request_first_assigned SQL > SELECT sourceParentId, MIN(timestamp) AS assignedAt - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-assigned' OR type = 'merge_request-assigned' GROUP BY sourceParentId order by min(timestamp) desc @@ -38,7 +38,7 @@ SQL > NODE pull_request_first_review_requested SQL > SELECT sourceParentId, MIN(timestamp) AS reviewRequestedAt - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-review-requested' OR type = 'merge_request-review-requested' GROUP BY sourceParentId @@ -55,7 +55,7 @@ SQL > ) AS prSourceId, timestamp, memberId - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-reviewed' OR type = 'merge_request-review-changes-requested' @@ -78,7 +78,7 @@ SQL > ) AS prSourceId, timestamp, memberId - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') OR type = 'merge_request-review-approved' @@ -93,7 +93,7 @@ SQL > SELECT if(type = 'changeset-abandoned', sourceId, sourceParentId) AS sourceParentId, MIN(timestamp) AS closedAt - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-closed' OR type = 'merge_request-closed' @@ -109,7 +109,7 @@ SQL > SELECT if(type = 'changeset-merged', sourceId, sourceParentId) AS sourceParentId, MIN(timestamp) AS mergedAt - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-merged' OR type = 'merge_request-merged' OR type = 'changeset-merged' GROUP BY sourceParentId @@ -120,7 +120,7 @@ SQL > type IN ('changeset-abandoned', 'changeset-merged'), sourceId, sourceParentId ) AS sourceParentId, MIN(timestamp) AS resolvedAt - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'pull_request-closed' OR type = 'pull_request-merged' @@ -137,7 +137,7 @@ DESCRIPTION > SQL > SELECT sourceParentId, toInt64(COUNT(*)) AS numberOfPatchsets - FROM activityRelations_deduplicated_cleaned_ds + FROM activityRelations_deduplicated_cleaned_bucket_union WHERE type = 'patchset-created' GROUP BY sourceParentId From 87ba957da5628ee20d2902816ea67558e681d62e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Tue, 11 Aug 2026 23:02:38 +0100 Subject: [PATCH 23/25] fix: disable pull_request_analysis_copy_pipe schedule pending investigation (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Job history shows a 5-day gap (Aug 6-11) with zero executions before this deploy, preceded by three consecutive 3600s copy timeouts. That stoppage is unexplained and this deploy did not resolve it. Set COPY_SCHEDULE to @on-demand so the corrected pipe definition (bucket_union rename + non-author filter) is live without running automatically or being triggered by accident, until the gap is understood and a full recompute is deliberately decided on. Signed-off-by: Gašper Grom --- .../pipes/pull_request_analysis_copy_pipe.pipe | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe index 9ad0b8fde8..c10a090b7b 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe @@ -11,6 +11,17 @@ DESCRIPTION > when either identity is unresolved, so such events don't count as a response. Empirically vacuous today — 0 of 38.5M PR openers and 0 of 16.5M issue openers have an empty memberId in prod (checked 2026-08-11) — so no PR loses a real review to this guard in practice. + - `activityRelations_deduplicated_cleaned_ds` -> `activityRelations_deduplicated_cleaned_bucket_union` + rename (2026-08-11, IN-1226): the old singular datasource no longer exists in production (gone by + the time of this deploy); this pipe was the only one still referencing it instead of the bucketed + union pattern already used by issue_analysis_copy_pipe.pipe and health_score_v2_maintainer.pipe. + Straight rename, no logic change. + - COPY_SCHEDULE set to `@on-demand` (2026-08-11, IN-1226), not restored to its prior hourly cron: + this pipe's job history shows a 5-day gap (2026-08-06 to 2026-08-11) with zero executions of any + kind before this deploy, preceded by three consecutive `Copy operation timed out after 3600s` + failures — an unexplained prior stoppage this deploy did not investigate or resolve. Schedule + stays disabled until that gap is understood and a full recompute is deliberately triggered; do not + re-enable the cron or run this pipe without that context. NODE pull_request_opened SQL > @@ -199,4 +210,4 @@ SQL > TYPE COPY TARGET_DATASOURCE pull_requests_analyzed COPY_MODE replace -COPY_SCHEDULE 20 * * * * +COPY_SCHEDULE @on-demand From 23a2a56847ffc46fc105fccc9e0784073f9b9c00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 12 Aug 2026 09:51:45 +0100 Subject: [PATCH 24/25] chore: delete obsolete pull_request_analysis_copy_pipe (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Confirmed obsolete by Anıl Bostancı in team chat (2026-08-12): PRs now flow through the MV + merger copy pipe path; this was the old copy-everything-at-once pipe it replaced. It had been disabled since 2026-08-06 and referenced a datasource deleted in January's bucketing migration, which is why the stale reference went unnoticed until this session tried to push it. Signed-off-by: Gašper Grom --- ...ll_request_analysis_baseline_merge_MV.pipe | 12 +- .../pull_request_analysis_copy_pipe.pipe | 213 ------------------ ...ull_request_analysis_initial_snapshot.pipe | 6 +- 3 files changed, 11 insertions(+), 220 deletions(-) delete mode 100644 services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe diff --git a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe index bf2311d791..83658857a7 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe @@ -3,9 +3,15 @@ DESCRIPTION > Uses existing pull_requests_analyzed data as baseline and merges new events on top. - reviewedAt/approvedAt count only NON-AUTHOR activity (2026-08-11, IN-1226): this incremental path feeds the minute-0 snapshot merger that replaces pull_requests_analyzed right before the - 02:00-02:15 health score jobs read it, so it must apply the same non-author rule as the - minute-20 full recompute in pull_request_analysis_copy_pipe.pipe — otherwise self-review - timestamps reappear every hour. See the non-author filter node below. + 02:00-02:15 health score jobs read it. See the non-author filter node below. + - `pull_request_analysis_copy_pipe.pipe` (2026-08-11/12, IN-1226): deleted — confirmed obsolete + by Anıl Bostancı (2026-08-12): "prs now run through MV + merger copy pipe, this was the old one + that copied everything at once." It referenced a datasource deleted in January's bucketing + migration (`activityRelations_deduplicated_cleaned_ds` -> `..._bucket_union`) and had been + disabled/not running since 2026-08-06, which is why the reference had gone stale unnoticed. + This MV + pull_request_analysis_snapshot_merger_copy.pipe are the real hourly path; this MV + + pull_request_analysis_initial_snapshot.pipe (bootstrap) are the only two places the non-author + filter needs to live. NODE snapshot_resolver DESCRIPTION > diff --git a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe b/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe deleted file mode 100644 index c10a090b7b..0000000000 --- a/services/libs/tinybird/pipes/pull_request_analysis_copy_pipe.pipe +++ /dev/null @@ -1,213 +0,0 @@ -DESCRIPTION > - Compacts activities from same PR into one, keeping necessary information in a single row. Helps to serve PR-wide widgets in the development tab. - - reviewedAt/approvedAt count only NON-AUTHOR activity (2026-08-11, IN-1226): review/approval - events where the actor is the PR author (GitHub COMMENTED self-reviews, GitLab self-approval, - Gerrit self +2) no longer set the first-response timestamps. Mirrors the non-author fix in - issue_analysis_copy_pipe.pipe — the compliance commitment is that responsiveness counts a - response from someone other than the author. Affects every consumer of reviewedAt/ - reviewedInSeconds/approvedAt, including the health score v2 lifecycle "unanswered PR" signal. - - The both-IDs-non-empty guard is deliberate and mirrors the issue-side convention (strict - identity verification was explicitly requested in review there): authorship can't be verified - when either identity is unresolved, so such events don't count as a response. Empirically - vacuous today — 0 of 38.5M PR openers and 0 of 16.5M issue openers have an empty memberId in - prod (checked 2026-08-11) — so no PR loses a real review to this guard in practice. - - `activityRelations_deduplicated_cleaned_ds` -> `activityRelations_deduplicated_cleaned_bucket_union` - rename (2026-08-11, IN-1226): the old singular datasource no longer exists in production (gone by - the time of this deploy); this pipe was the only one still referencing it instead of the bucketed - union pattern already used by issue_analysis_copy_pipe.pipe and health_score_v2_maintainer.pipe. - Straight rename, no logic change. - - COPY_SCHEDULE set to `@on-demand` (2026-08-11, IN-1226), not restored to its prior hourly cron: - this pipe's job history shows a 5-day gap (2026-08-06 to 2026-08-11) with zero executions of any - kind before this deploy, preceded by three consecutive `Copy operation timed out after 3600s` - failures — an unexplained prior stoppage this deploy did not investigate or resolve. Schedule - stays disabled until that gap is understood and a full recompute is deliberately triggered; do not - re-enable the cron or run this pipe without that context. - -NODE pull_request_opened -SQL > - SELECT - activityId as id, - sourceId, - channel, - timestamp AS openedAt, - segmentId, - gitChangedLinesBucket, - memberId, - organizationId, - platform - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'pull_request-opened' OR type = 'merge_request-opened' OR type = 'changeset-created' - -NODE pull_request_first_assigned -SQL > - SELECT sourceParentId, MIN(timestamp) AS assignedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'pull_request-assigned' OR type = 'merge_request-assigned' - GROUP BY sourceParentId - order by min(timestamp) desc - -NODE pull_request_first_review_requested -SQL > - SELECT sourceParentId, MIN(timestamp) AS reviewRequestedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'pull_request-review-requested' OR type = 'merge_request-review-requested' - GROUP BY sourceParentId - -NODE pull_request_first_reviewed -SQL > - SELECT r.prSourceId AS sourceParentId, MIN(r.timestamp) AS reviewedAt - FROM - ( - SELECT - if( - type = 'patchset_approval-created', - splitByChar('-', sourceParentId)[1], - sourceParentId - ) AS prSourceId, - timestamp, - memberId - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE - type = 'pull_request-reviewed' - OR type = 'merge_request-review-changes-requested' - OR type = 'patchset_approval-created' - ) AS r - INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId - WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId - GROUP BY r.prSourceId - -NODE pull_request_first_review_approved -SQL > - SELECT r.prSourceId AS sourceParentId, MIN(r.timestamp) AS approvedAt - FROM - ( - SELECT - if( - type = 'patchset_approval-created', - splitByChar('-', sourceParentId)[1], - sourceParentId - ) AS prSourceId, - timestamp, - memberId - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE - (type = 'pull_request-reviewed' and pullRequestReviewState = 'APPROVED') - OR type = 'merge_request-review-approved' - OR type = 'patchset_approval-created' - ) AS r - INNER JOIN pull_request_opened opened ON r.prSourceId = opened.sourceId - WHERE r.memberId != '' AND opened.memberId != '' AND r.memberId != opened.memberId - GROUP BY r.prSourceId - -NODE pull_request_first_closed -SQL > - SELECT - if(type = 'changeset-abandoned', sourceId, sourceParentId) AS sourceParentId, - MIN(timestamp) AS closedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE - type = 'pull_request-closed' - OR type = 'merge_request-closed' - OR type = 'changeset-closed' - OR type = 'changeset-abandoned' - GROUP BY sourceParentId - -NODE pull_request_first_merged -DESCRIPTION > - Resolved PRs are the ones that are either closed or merged - -SQL > - SELECT - if(type = 'changeset-merged', sourceId, sourceParentId) AS sourceParentId, - MIN(timestamp) AS mergedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'pull_request-merged' OR type = 'merge_request-merged' OR type = 'changeset-merged' - GROUP BY sourceParentId - -NODE pull_request_first_resolved -SQL > - SELECT - if( - type IN ('changeset-abandoned', 'changeset-merged'), sourceId, sourceParentId - ) AS sourceParentId, - MIN(timestamp) AS resolvedAt - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE - type = 'pull_request-closed' - OR type = 'pull_request-merged' - OR type = 'merge_request-closed' - OR type = 'merge_request-merged' - OR type = 'changeset-merged' - OR type = 'changeset-closed' - OR type = 'changeset-abandoned' - GROUP BY sourceParentId - -NODE patchsets_count -DESCRIPTION > - Count the number of patchsets for each Gerrit changeset - -SQL > - SELECT sourceParentId, toInt64(COUNT(*)) AS numberOfPatchsets - FROM activityRelations_deduplicated_cleaned_bucket_union - WHERE type = 'patchset-created' - GROUP BY sourceParentId - -NODE pull_request_analysis_results_merged -SQL > - SELECT - pr_opened.id, - pr_opened.sourceId, - pr_opened.openedAt, - pr_opened.segmentId, - pr_opened.channel, - pr_opened.memberId, - pr_opened.organizationId, - pr_opened.gitChangedLinesBucket, - IF(assigned.assignedAt = toDateTime(0), NULL, assigned.assignedAt) AS assignedAt, - IF( - review_requested.reviewRequestedAt = toDateTime(0), NULL, review_requested.reviewRequestedAt - ) AS reviewRequestedAt, - IF(reviewed.reviewedAt = toDateTime(0), NULL, reviewed.reviewedAt) AS reviewedAt, - IF(approved.approvedAt = toDateTime(0), NULL, approved.approvedAt) AS approvedAt, - IF(closed.closedAt = toDateTime(0), NULL, closed.closedAt) AS closedAt, - IF(merged.mergedAt = toDateTime(0), NULL, merged.mergedAt) AS mergedAt, - IF(resolved.resolvedAt = toDateTime(0), NULL, resolved.resolvedAt) AS resolvedAt, - IF( - assignedAt IS NULL, NULL, toUnixTimestamp(assignedAt) - toUnixTimestamp(openedAt) - ) AS assignedInSeconds, - IF( - reviewRequestedAt IS NULL, - NULL, - toUnixTimestamp(reviewRequestedAt) - toUnixTimestamp(openedAt) - ) AS reviewRequestedInSeconds, - IF( - reviewedAt IS NULL, NULL, toUnixTimestamp(reviewedAt) - toUnixTimestamp(openedAt) - ) AS reviewedInSeconds, - IF( - closedAt IS NULL, NULL, toUnixTimestamp(closedAt) - toUnixTimestamp(openedAt) - ) AS closedInSeconds, - IF( - mergedAt IS NULL, NULL, toUnixTimestamp(mergedAt) - toUnixTimestamp(openedAt) - ) AS mergedInSeconds, - IF( - resolvedAt IS NULL, NULL, toUnixTimestamp(resolvedAt) - toUnixTimestamp(openedAt) - ) AS resolvedInSeconds, - pr_opened.platform, - patchsets.numberOfPatchsets - FROM pull_request_opened pr_opened - LEFT JOIN pull_request_first_assigned AS assigned ON pr_opened.sourceId = assigned.sourceParentId - LEFT JOIN - pull_request_first_review_requested AS review_requested - ON pr_opened.sourceId = review_requested.sourceParentId - LEFT JOIN pull_request_first_reviewed AS reviewed ON pr_opened.sourceId = reviewed.sourceParentId - LEFT JOIN - pull_request_first_review_approved AS approved ON pr_opened.sourceId = approved.sourceParentId - LEFT JOIN pull_request_first_closed AS closed ON pr_opened.sourceId = closed.sourceParentId - LEFT JOIN pull_request_first_merged AS merged ON pr_opened.sourceId = merged.sourceParentId - LEFT JOIN pull_request_first_resolved as resolved on pr_opened.sourceId = resolved.sourceParentId - LEFT JOIN patchsets_count AS patchsets ON pr_opened.sourceId = patchsets.sourceParentId - -TYPE COPY -TARGET_DATASOURCE pull_requests_analyzed -COPY_MODE replace -COPY_SCHEDULE @on-demand diff --git a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe index f7ffe049e6..d5b933268a 100644 --- a/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe +++ b/services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe @@ -79,8 +79,7 @@ SQL > NODE pull_request_first_reviewed DESCRIPTION > - Non-author reviews only (2026-08-11, IN-1226) — same rule as - pull_request_analysis_copy_pipe.pipe and the baseline-merge MV. + Non-author reviews only (2026-08-11, IN-1226) — same rule as the baseline-merge MV. SQL > % @@ -127,8 +126,7 @@ SQL > NODE pull_request_first_review_approved DESCRIPTION > - Non-author approvals only (2026-08-11, IN-1226) — same rule as - pull_request_analysis_copy_pipe.pipe and the baseline-merge MV. + Non-author approvals only (2026-08-11, IN-1226) — same rule as the baseline-merge MV. SQL > % From 42f442814d44b75d191342b611f2a4649829d23c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C5=A1per=20Grom?= Date: Wed, 12 Aug 2026 10:11:53 +0100 Subject: [PATCH 25/25] fix: add FINAL to vulnerabilities read in raw inputs snapshot (CM-IN-1226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gašper Grom --- .../tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe index 6f41ec0ec7..54a3d0cbfd 100644 --- a/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe +++ b/services/libs/tinybird/pipes/health_score_v2_raw_inputs_snapshot.pipe @@ -140,7 +140,7 @@ SQL > countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals, countIf(status = 'OPEN' AND severity = 'HIGH') AS openHighs, countIf(status = 'OPEN' AND severity = 'MEDIUM') AS openModerates - FROM vulnerabilities + FROM vulnerabilities FINAL GROUP BY repoUrl ) AS v ON v.repoUrl = allRepos.repoUrl