Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 20 additions & 11 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,39 +259,48 @@ jinja-rendered from the same `var('client_id')` in sources.yml (M1), so every
```

The pipeline runs dbt once per active client with only `client_id` in `--vars`
(the model `--select` comes from the plan in `scripts/pipeline.sh`):
(the model `--select` is dynamically determined by `src/agency_analytics/pipeline_plan.py`):

```bash
dbt run --select '<plan models>' --vars '{"client_id": "acme"}' --profiles-dir .
```

## Orchestration: Nightly Pipeline

The pipeline runs from the host via cron. Before any step, it validates container health.
The pipeline runs from the host via cron or directly via `make pipeline`. The host wrapper (`scripts/pipeline.sh`) validates container health and delegates execution to the Python orchestrator (`multitenant-elt pipeline` / `pipeline_runner.py`) running inside the container.

### Execution Flow

```
cron (02:00 AM)
cron (02:00 AM) or 'make pipeline'
│
▼
scripts/pipeline.sh
scripts/pipeline.sh (Host wrapper)
│
├── [1] Validate containers (postgres, pipeline_worker)
├── [1] Validate containers (agency_postgres, agency_pipeline)
│ │
│ ├── OK ──────────────────────────────────┐
│ └── FAIL ──► Telegram alert ──► ABORT │
│ │
▼ │
├── [2] For each active client in clients/*.yml │◄──┘
docker exec agency_pipeline multitenant-elt pipeline │◄──┘
│
▼
pipeline_runner.py (Python orchestrator)
│
├── [2] Discover & filter active clients in clients/*.yml
│ │
│ ├── Compute dynamic plan (pipeline_plan.py)
│ │
│ ├── Record pipeline run start (public.pipeline_runs)
│ │
│ ├── dlt: extract enabled domains
│ │ └── FAIL ──► Telegram alert ──► next client
│ ├── Execute connectors in-process / subprocess per tenant
│ │ └── Record step status (public.pipeline_run_steps)
│ │
│ └── dbt: run --vars '{client_id: ...}' via docker exec
│ └── FAIL ──► Telegram alert ──► next client
│ └── Run dbt per client with dynamic selection
│ └── Record dbt step status
│
└── [3] Telegram: execution summary (successes + failures)
└── [3] Telegram: execution summary (successes + failures) & DB finish update
```

### Container Health Check
Expand Down
14 changes: 11 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,25 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [2.0.0] - 2026-11-05

### Breaking Changes

- **Native Python Pipeline Orchestrator**: Migrated end-to-end pipeline orchestration from a 356-line bash script to a native Python engine (`multitenant-elt pipeline` / `src/agency_analytics/pipeline_runner.py`). `scripts/pipeline.sh` has been converted into a slim 30-line host wrapper. Eliminates 40+ container spawns per client, bash YAML parsing, and SQL string interpolation in favor of safe parameterized database records (`%s`), native PyYAML loading, and dynamic plan execution.
- **Sunset Google Ads Developer Token**: Completely removed all references to `developer_token` / `GOOGLE_ADS_DEVELOPER_TOKEN` across code, schemas, and documentation following Google Ads API developer token deprecation. Authorization is now bound directly to Google Cloud projects and OAuth 2.0 consent.
- **Diagnostic Tool Relocation**: Relocated the 713-line standalone CLI probe from `src/agency_analytics/ig_probe.py` to `tools/diagnostics/ig_probe.py` to decouple developer diagnostics from core pipeline runtime imports. A 19-line backward compatibility module is retained at `src/agency_analytics/ig_probe.py`.

### Added

- **Centralized Resilient HTTP Client (`src/agency_analytics/http_client.py`)**: Unified Meta Graph API client with exponential backoff on HTTP 429 and Meta rate limit error codes (4, 17, 80000, 80004, 613), with fast-fail on token expiration (code 190).
- **Unified Google OAuth Renewal Wizard**: Added `multitenant-elt auth google` CLI command and `make auth-google` target to obtain and store permanent OAuth refresh tokens for all Google integrations (Google Ads, GA4, GTM, and YouTube). Supports both global agency credentials (via MCC / manager permissions) and client-isolated credentials (`_CLIENT` suffix).
- **Automated Token Management**: Added `GoogleTokenManager` with automatic caching, proactive refresh before expiry, and automatic token invalidation and refresh on HTTP 401 Unauthorized responses across Google Ads, GTM, and YouTube connectors.
- **Automated Token Management**: Added `GoogleTokenManager` with automatic caching, proactive refresh before expiry, and automatic token invalidation and refresh on HTTP 401 Unauthorized responses across Google Ads, GTM, and YouTube connectors. Includes `.credentials` property returning official `google.oauth2.credentials.Credentials`.
- **Pipeline Test Suite Expansion**: Expanded automated unit and integration tests to 504 passing tests (505 total, 1 skipped) with global test coverage reaching 94% (all modules $\ge 85\%$).

### Changed

- **Sunset Google Ads Developer Token**: Completely removed all references to `developer_token` / `GOOGLE_ADS_DEVELOPER_TOKEN` across code, schemas, and documentation following Google Ads API developer token sunset on September 9, 2026. Developer tokens are now ignored by the API and authorization is bound directly to Google Cloud projects.
- **Hierarchical Credential Resolution**: Google connectors now look up client-specific OAuth refresh keys first (`GOOGLE_OAUTH_*_{CLIENT}`), then fall back to global agency keys (`GOOGLE_OAUTH_*`), and finally legacy connector-specific variables for backwards compatibility.
- **Dynamic Connector Execution**: Replaced hardcoded bash execution blocks with dynamic connector discovery driven by `pipeline_plan.py`.

## [1.3.0] - 2026-11-04

Expand Down
26 changes: 16 additions & 10 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,24 +100,30 @@ Add the connector definition block to `clients/_template.yml` with `enabled: fal
token_env: CUSTOM_ACCESS_TOKEN_NEW_CLIENT
```

### 3. Update Pipeline Script (`scripts/pipeline.sh`)
### 3. Pipeline Discovery & Execution

Register the connector execution block in `scripts/pipeline.sh` inside the per-client iteration loop:
Connectors matching `src/connectors/run_<name>.py` and configured in `clients/<client_id>.yml` are dynamically resolved by `src/agency_analytics/pipeline_plan.py` and executed by the Python pipeline engine (`src/agency_analytics/pipeline_runner.py`). No bash script edits are required.

To test the connector within the pipeline:

```bash
if [ "$CUSTOM_ENABLED" = "true" ]; then
log_info "Running Custom connector for $CLIENT_ID..."
run_step "$CLIENT_ID" "custom" "python src/connectors/run_custom.py --client $CLIENT_ID"
fi
# Direct Python CLI run:
uv run multitenant-elt pipeline --client <client_id>

# Or via Make:
make pipeline
```

The host wrapper `scripts/pipeline.sh` also calls `multitenant-elt pipeline` inside the container for cron automation.

### 4. Create Unit Tests (`tests/test_run_<name>.py`)

Write comprehensive mock-based unit tests covering:
- Successful extraction and pagination.
- Rate-limiting (HTTP 429) backoff.
- Authentication/token expiry (HTTP 401).
- Missing configuration / invalid input fail-fast.
- Maintain test coverage $\ge 85\%$ across all modules.

---

Expand Down Expand Up @@ -156,12 +162,12 @@ Transformations are managed with dbt Core in `src/dbt_project/`:
Before opening a pull request, run the complete local quality gate:

```bash
make quality
make release-ready
```

This runs sequentially:
1. `make lint`: `uv run ruff check src/ tests/ main.py`
2. `make typecheck`: `uv run mypy src/ main.py`
3. `make test`: `uv run pytest tests/ -v`
1. `make lint-all`: ruff check, ruff format check, mypy typecheck, actionlint, hadolint, zizmor
2. `make test`: pytest suite (504+ passed tests with $\ge 85\%$ coverage)
3. `make test-security`: Trivy vulnerability scan against `uv.lock`

All tests must be deterministic and mock-based. Live network requests during automated tests are strictly prohibited.
79 changes: 51 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@

*Eliminate manual reporting across 10 marketing platforms - one pipeline, one dashboard, zero spreadsheets.*

[![Version](https://img.shields.io/badge/Version-v1.3.0-blue?style=for-the-badge)](CHANGELOG.md)
[![Version](https://img.shields.io/badge/Version-v2.0.0-blue?style=for-the-badge)](CHANGELOG.md)
[![Python](https://img.shields.io/badge/Python_3.12-3776AB?style=for-the-badge&logo=python&logoColor=white)](pyproject.toml)
[![Docker](https://img.shields.io/badge/Docker_|_READY-2496ED?style=for-the-badge&logo=docker&logoColor=white)](Dockerfile)
[![CI](https://img.shields.io/badge/CI-Passing-brightgreen?style=for-the-badge)](.github/workflows)
[![Tests](https://img.shields.io/badge/Tests-361%20passed-success?style=for-the-badge)](tests)
[![Tests](https://img.shields.io/badge/Tests-504%20passed-success?style=for-the-badge)](tests)
[![FinOps](https://img.shields.io/badge/FinOps-98%25%20OpEx%20Reduction-2ea44f?style=for-the-badge)](#-finops--cost-efficiency)
[![Security](https://img.shields.io/badge/Security-Trivy%20%7C%20Cosign%20%7C%20SLSA-blue?style=for-the-badge)](SECURITY.md)
[![Status](https://img.shields.io/badge/Status-Production_Active-brightgreen?style=for-the-badge)](#-features)
Expand Down Expand Up @@ -49,20 +49,35 @@ mindmap

## Table of contents

- [Features](#-features)
- [FinOps & Cost Efficiency](#-finops--cost-efficiency)
- [Connector status](#-connector-status)
- [Quick start](#-quick-start)
- [Architecture](#-architecture)
- [Docker deployment](#-docker-deployment)
- [Configuration](#-configuration)
- [Tests](#-tests)
- [Security](#-security)
- [Changelog](#-changelog)
- [Roadmap](#-roadmap)
- [Contributing](#-contributing)
- [License](#-license)
- [Contact & support](#-contact--support)

- [Multitenant ELT | Developmi](#multitenant-elt--developmi)
- [Table of contents](#table-of-contents)
- [🎯 Features](#-features)
- [💰 FinOps \& Cost Efficiency](#-finops--cost-efficiency)
- [🔌 Connector status](#-connector-status)
- [Instagram diagnostics probe (read-only)](#instagram-diagnostics-probe-read-only)
- [⚡ Quick start](#-quick-start)
- [Prerequisites](#prerequisites)
- [1. Clone and configure](#1-clone-and-configure)
- [2. Start infrastructure](#2-start-infrastructure)
- [3. Run the pipeline](#3-run-the-pipeline)
- [4. Access Metabase (opt-in)](#4-access-metabase-opt-in)
- [🏗️ Architecture](#️-architecture)
- [Project structure](#project-structure)
- [🐳 Docker deployment](#-docker-deployment)
- [Build the pipeline image](#build-the-pipeline-image)
- [Start all services](#start-all-services)
- [Service isolation](#service-isolation)
- [Use a published image (`vX` tags)](#use-a-published-image-vx-tags)
- [🔧 Configuration](#-configuration)
- [🧪 Tests \& Quality Gate](#-tests--quality-gate)
- [🔑 Authentication \& Token Management](#-authentication--token-management)
- [🔒 Security](#-security)
- [📋 Changelog](#-changelog)
- [🗺️ Roadmap](#️-roadmap)
- [🤝 Contributing](#-contributing)
- [📄 License](#-license)
- [🤝 Contact \& support](#-contact--support)

---

Expand All @@ -72,9 +87,9 @@ mindmap
- **Honest status tracking**: See [Connector status](#-connector-status) for which integrations are tested live vs. code-complete only
- **Multi-tenant by design**: per-client raw namespaces and dbt output schemas (`raw_<connector>_<client_id>` / `client_<client_id>`), YAML client config, shared observability
- **No-code connector config**: Add/remove platforms per client via YAML, no code changes
- **Pipeline orchestration**: Health checks, per-client loops, Telegram alerts
- **Pipeline orchestration**: Native Python engine (`pipeline_runner.py`), dynamic client execution plans, database run step metrics, health checks, and Telegram alerts
- **dbt transformations**: 30 staging models → intermediate → marts (35 models total)
- **Quality gate**: 362 tests (361 passed, 1 skipped), ruff linting, mypy type checking
- **Quality gate**: 505 tests (504 passed, 1 skipped), ruff linting, mypy type checking, Trivy security scan
- **Docker native**: Isolated services, two Docker networks, no vendor lock-in

---
Expand Down Expand Up @@ -112,8 +127,9 @@ mindmap

### Instagram diagnostics probe (read-only)

`agency_analytics/ig_probe.py` is a **read-only** diagnostic CLI for the
Instagram Business connector (never part of the scheduled pipeline). It answers
`tools/diagnostics/ig_probe.py` is a **read-only** diagnostic CLI for the
Instagram Business connector (never part of the scheduled pipeline). A
backward-compatible shim is maintained at `agency_analytics.ig_probe`. It answers
which account-insights metrics respond today and with which `metric_type`, the
real retention horizon per trailing window, which breakdown axes work per
metric, and eligibility below 100 followers — the data that drives the
Expand All @@ -123,15 +139,15 @@ metric, and eligibility below 100 followers — the data that drives the
docker exec -w /app/src agency_pipeline \
python -m agency_analytics.ig_probe --client example --read-only

# Local run, write the JSON report to a file, skip breakdown calls:
uv run python -m agency_analytics.ig_probe --client example --read-only \
# Local run with tools/diagnostics/ig_probe.py:
uv run python tools/diagnostics/ig_probe.py --client example --read-only \
--skip-breakdowns --output /tmp/ig_probe_report.json
```

It makes no destructive calls and writes to no table (spec IG4). Full CLI
reference, report schema and call budget live in the module docstring
(`src/agency_analytics/ig_probe.py`). `scripts/pipeline.sh` is intentionally
unchanged: the probe is a manual diagnostics entry point, not a pipeline step.
(`tools/diagnostics/ig_probe.py`). The probe is a manual diagnostics entry point,
not a pipeline step.

---

Expand Down Expand Up @@ -162,8 +178,15 @@ make docker-up-all # Start Postgres, Pipeline, and Metabase (opt-in profile)

```bash
make setup-env # Ensure .env exists
make test # 361 tests should pass
./scripts/pipeline.sh # Full E2E pipeline (or use cron)
make test # 504 tests should pass

# Run full Python E2E pipeline:
make pipeline
# Or execute via the native CLI for a single client:
uv run multitenant-elt pipeline --client acme

# For host cron automation (thin wrapper around container pipeline):
./scripts/pipeline.sh
```

### 4. Access Metabase (opt-in)
Expand All @@ -184,7 +207,7 @@ user.
- **Storage**: PostgreSQL 16 - per-client raw namespaces (`raw_meta_acme`, `raw_google_nike`, ...) plus shared observability (`public`, `staging`), transformed per client into `client_<id>` schemas
- **Transformation**: dbt with multi-tenant routing - `generate_schema_name` sends tenant models to `client_<id>` and sources.yml jinja resolves each client's own raw namespace
- **Visualization**: Metabase (opt-in compose profile) connected as `metabase_reader`, a read-only Postgres role that only exists when `METABASE_READER_ENABLED=true`
- **Orchestration**: `./scripts/pipeline.sh` runs nightly via cron, validates Docker health, loops over active clients, sends Telegram summary
- **Orchestration**: Native Python orchestrator (`multitenant-elt pipeline`) executed directly, via `make pipeline`, or nightly via cron wrapper `./scripts/pipeline.sh`; validates Docker health, dynamically executes client plans, tracks run steps in PostgreSQL, and sends Telegram summaries

See [ARCHITECTURE.md](ARCHITECTURE.md) for full design.

Expand Down Expand Up @@ -298,7 +321,7 @@ See [.env.example](.env.example) for the full list of environment variables.
## 🧪 Tests & Quality Gate

```bash
make test # Run all tests (361 passed, 1 skipped)
make test # Run all tests (504 passed, 1 skipped)
make lint # ruff check
make typecheck # mypy
make test-security # Trivy vulnerability & secret scan
Expand Down
36 changes: 31 additions & 5 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,24 +92,34 @@

---

## Phase 2 - Post-MVP
## Phase 2 - Post-MVP (v1.x -> v2.0.0)

Improvements planned for after the initial release is stable in production.
Improvements planned for after the initial release and delivered in v2.0.0.

### Observability
### Observability & Orchestration

- [x] Table `pipeline_runs` + `pipeline_run_steps` in Postgres (auto-created via init SQL)
- [x] dbt staging + intermediate + mart models for pipeline monitoring data
- [x] Native Python pipeline orchestrator (`src/agency_analytics/pipeline_runner.py`) replacing monolithic shell
- [x] Dynamic connector plan execution (`src/agency_analytics/pipeline_plan.py`) from client YAML
- [x] Centralized resilient HTTP client (`src/agency_analytics/http_client.py`) with exponential backoff on HTTP 429
- [x] Diagnostic CLI tool relocated to `tools/diagnostics/ig_probe.py` with backward compatibility shim
- [ ] Internal Metabase dashboard for pipeline monitoring
- [ ] Telegram watchdog alert if pipeline hasn't run in 26 hours
- [ ] Data volume metrics per connector and client

### Authentication & Security

- [x] Automatic OAuth token refresh (Meta, Google)
- [x] Unified Google OAuth renewal wizard (`multitenant-elt auth google`) with `GoogleTokenManager`
- [x] Meta Graph API token renewal wizard (`multitenant-elt auth meta`)
- [x] Sunset Google Ads developer token across all models, schemas, and credentials

### More Connectors

- [ ] LinkedIn Ads
- [ ] Pinterest Ads (separate from organic)
- [ ] Shopify (e-commerce clients)
- [ ] Automatic OAuth token refresh (Meta, Google)

### Client Management

Expand Down Expand Up @@ -145,13 +155,29 @@ Evaluate when client count or data volume justifies it.

---

## Technical Debt & Deferrals

Tracked intentional trade-offs and deferrals documented under senior and Ponytail engineering principles.

### 1. `archiver.py` Migration vs Native `dlt` Merge
- **Status:** ⏳ Deferred / Deliberate Technical Debt (`ponytail debt`).
- **Context:** `archiver.py` uses custom SQL (`INSERT ... ON CONFLICT`) for Instagram Business and TikTok Organic metrics snapshots.
- **Evaluation:**
- *Performance:* Execution takes <0.05s nightly per client (~4 rows archived per tenant). Zero pipeline bottleneck.
- *Security:* Uses strictly parameterized SQL (`%s`) and safe identifier quotes; zero SQL injection vulnerability.
- *Cost/Benefit:* Migrating to native `dlt` merge would break live production PostgreSQL tables (which lack `_dlt_id` metadata columns and staging tables) and require rewriting 60+ unit tests with zero business or runtime performance gain.
- **Decision:** Retain current stable implementation until a broader raw schema redesign is warranted.

---

## Design Decisions

| Decision | Outcome | Rationale |
|---|---|---|
| NocoDB | ❌ Not used | Metabase covers data inspection needs |
| dbt Cloud | ❌ Not used | Free tier risk, prefer full control |
| Prefect / Airflow | ⏳ Postponed to Phase 3 | Overengineering for MVP |
| Prefect / Airflow | ⏳ Postponed to Phase 3 | Overengineering for current scale |
| Email alerts | ❌ Not used | Telegram API is simpler than email OAuth |
| Per-client database | ❌ Not used | Schema isolation is sufficient for MVP |
| Config in Postgres | ⏳ Postponed to Phase 3 | YAML + Git is more auditable for now |
| `archiver.py` custom SQL vs native `dlt` merge | ⏳ Postponed (Ponytail Debt) | Stable (<0.05s/run, 4 rows/night), zero security risk, avoids breaking production tables without user value |
Loading
Loading