Skip to content

developers.md

Latest commit

 

History

History
534 lines (374 loc) · 15.4 KB

developers.md

File metadata and controls

534 lines (374 loc) · 15.4 KB
Raw

Chitral Developer Guide

This guide covers everything you need to run and deploy the Chitral platform locally and in production.

Table of Contents

Repository Structure

chitral/
├── backend/         # FastAPI backend (Python, SQLite, uv)
├── frontend/        # React + Vite + TypeScript frontend
└── spec/            # OpenSemantics Interface (OSI) spec files

Prerequisites

Tool Version Purpose
Python ≥ 3.12 Backend runtime
uv latest Python package & project manager
Node.js ≥ 18 Frontend runtime
npm ≥ 9 Frontend package manager

Install uv (if not already installed):

curl -LsSf https://astral.sh/uv/install.sh | sh

Backend

The backend is a FastAPI application with a SQLite database, managed with uv.

Backend Local Setup

cd backend

# Install all dependencies (including optional connectors)
uv sync --extra connectors

# Or, for core dependencies only (no Snowflake connector):
uv sync

This creates a .venv virtual environment automatically.

Running the Backend in Development

cd backend
uv run run.py

The server starts at http://localhost:8000 with hot-reload enabled.

Note: On first run, chitral.db (SQLite) is created automatically in the backend/ directory.

Optional: Run with Uvicorn directly

uv run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Database Migration (YAML → SQLite)

If you have an existing spec.yaml and need to migrate it into the database:

cd backend
uv run scripts/migrate.py

Backend Configuration

Create a .env file in the backend/ directory to override defaults:

# backend/.env

DATABASE_URL=sqlite:///./chitral.db
API_PREFIX=/api
APP_NAME=Chitral
APP_VERSION=0.1.0
# Localhost frontends only by default. JSON array. Do not use ["*"] without auth.
CORS_ORIGINS=["http://localhost:5173","http://127.0.0.1:5173","http://localhost:3000","http://127.0.0.1:3000"]

# Optional: pin Fernet key for encrypting saved connector credentials.
# Generate: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# If unset, the backend creates backend/.datasource_key on first encrypt.
# DATASOURCE_ENCRYPTION_KEY=

Security: Chitral has no API authentication. Saved data-source credentials are encrypted at rest (Fernet) in chitral.db. Protect DATASOURCE_ENCRYPTION_KEY / backend/.datasource_key. Use only on a trusted local machine. See the root README security section.

If you don't set DATASOURCE_ENCRYPTION_KEY, a key is auto-generated at backend/.datasource_key. Saved datasource credentials are not portable across machines or fresh clones unless you pin that env var (or copy the key file along with the database).

For a Snowflake-connected workflow, add your Snowflake credentials:

SNOWFLAKE_ACCOUNT=your_account
SNOWFLAKE_USER=your_user
SNOWFLAKE_PASSWORD=your_password
SNOWFLAKE_WAREHOUSE=your_warehouse
SNOWFLAKE_DATABASE=your_database
SNOWFLAKE_SCHEMA=your_schema

API Documentation

Once the backend is running, interactive docs are available at:

Interface URL
Swagger UI http://localhost:8000/docs
ReDoc http://localhost:8000/redoc
Health Check http://localhost:8000/health
OSI YAML Export http://localhost:8000/api/export/osi

Running Backend Tests

cd backend
uv run pytest

To run with verbose output:

uv run pytest -v

Deploying the Backend

Option 1: Run Directly on a Server

# On the remote server
cd backend
uv sync --extra connectors
uv run uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4

Use a process manager like systemd or supervisord to keep the process running.

Option 2: Docker

Create a Dockerfile in backend/:

FROM python:3.12-slim

WORKDIR /app

# Install uv
RUN pip install uv

# Copy project files
COPY pyproject.toml uv.lock ./
RUN uv sync --extra connectors --no-dev

COPY . .

EXPOSE 8000
CMD ["uv", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

Build and run:

docker build -t chitral-backend ./backend
docker run -p 8000:8000 --env-file backend/.env chitral-backend

Option 3: Cloud (e.g., Railway, Render, Fly.io)

Point the platform to the backend/ directory and set the start command:

uv sync --extra connectors && uv run uvicorn app.main:app --host 0.0.0.0 --port 8000

Set all required environment variables in the platform's dashboard.

Note on SQLite in production: SQLite is fine for small deployments. For concurrent multi-user production workloads, consider migrating to PostgreSQL by updating DATABASE_URL to a Postgres connection string. No other code changes are required — SQLAlchemy handles the rest.

Frontend

The frontend is a React 19 application built with Vite, TypeScript, and Tailwind CSS v4.

Frontend Local Setup

cd frontend
npm install

Running the Frontend in Development

cd frontend
npm run dev

The app starts at http://localhost:5173 by default. It expects the backend API to be running at http://localhost:8000.

Linting

From the repo root:

make lint          # backend (ruff) + frontend (eslint)
make lint-backend  # ruff only
cd frontend && npm run lint   # eslint only

Backend lint/format:

cd backend
uv run ruff check app tests
uv run ruff format app tests

Frontend Unit Tests

cd frontend
npm run test:unit

Uses Vitest for fast component/hook tests (e.g. src/hooks/*.test.ts).

Building the Frontend

Compile TypeScript and bundle for production:

cd frontend
npm run build

Output is written to frontend/dist/. Preview the production build locally:

npm run preview

Deploying the Frontend

Option 1: Static Hosting (Netlify, Vercel, GitHub Pages)

  1. Build the project: 
    cd frontend && npm run build
  2. Deploy the frontend/dist/ directory to your static host.
  3. Set the environment variable VITE_API_URL (if used) to point at your deployed backend URL.

For Vercel, add a vercel.json at the repo root to handle SPA routing:

{
  "rewrites": [{ "source": "/(.*)", "destination": "/index.html" }]
}

Option 2: Docker / Nginx

# frontend/Dockerfile
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80

Build and run:

docker build -t chitral-frontend ./frontend
docker run -p 80:80 chitral-frontend

Option 3: Cloud (Railway, Render)

Set the build command and publish directory:

Setting Value
Build command npm run build
Publish directory dist

Running Both Together

For local full-stack development, run the backend and frontend in separate terminals:

Terminal 1 — Backend:

cd backend && uv run run.py

Terminal 2 — Frontend:

cd frontend && npm run dev

The frontend calls the backend API directly at http://localhost:8000 by default (override with VITE_API_URL in frontend/.env). Ensure the backend is running and that CORS_ORIGINS includes your frontend origin.

Contributing

Chitral is open source under the MIT License.

Branching and pull requests

  1. Fork the repository and create a feature branch from main (e.g. feat/postgres-sink or fix/import-warnings).
  2. Keep changes focused — one logical change per PR when possible.
  3. Run the full check suite locally before opening a PR (see Testing and make ci below).
  4. Open a pull request with:
    • A short summary of why the change is needed
    • A test plan checklist (what you ran, what you could not run)
  5. Expect review feedback on correctness, tests, and consistency with existing patterns. We do not require a CLA.

What we look for

  • Tests for behavior changes (backend pytest and/or frontend Vitest/Playwright as appropriate)
  • No committed secrets (.env, backend/.datasource_key, credentials)
  • Google-style docstrings on new public Python APIs
  • Ruff-clean backend code (make lint-backend)

Architecture

Before changing backend behavior, read backend/ARCHITECTURE.md — it explains the layered layout (api/ → crud/ → models/, connector ABCs, generation engine) and common extension points.

For connector design history and Snowflake-specific details, see also spec/connectors.md.

How to add a new connector

  1. Implement the ABCs in backend/app/connectors/<name>/:
    • client.py — BaseConnector (schema introspection, test_connection)
    • profiler.py — extends BaseProfiler (aggregate SQL profiling)
    • sampler.py — row sampling for JSON/LLM few-shot (SQL connectors can extend SqlTableSampler in connectors/sql_sampler.py)
    • inferrer.py — BaseRelationshipInferrer (FK metadata inferrers can extend InformationSchemaFkInferrer in connectors/fk_inferrer.py)
  2. Add Pydantic credential/request schemas in backend/app/schemas/connector.py.
  3. Register API routes in backend/app/api/connectors.py (/test, /introspect, /infer-relationships, /import).
  4. Wire saved sources — extend build_credentials / dispatch in backend/app/api/datasources.py if the connector supports predefined connections.
  5. Frontend — add a form tab in ConnectorsSidebar, hooks in useSnowflakeIntrospect.ts (or a sibling hook), and types in frontend/src/types/api.ts.
  6. Tests — unit tests under backend/tests/connectors/; Docker integration tests via make test-db-up when the warehouse has a container recipe in docker-compose.test.yml.

Planned refactors (not required for new connectors)

These are known duplication hotspots — safe to leave as-is for now:

  • backend/app/api/connectors.py — repeated per-connector route handlers (/test, /introspect, /import) could become a generic registry; deferred to avoid a large risky refactor before OSS.
  • MongoDB profiler — shares patterns with SQL profilers but document-model stats differ enough that a shared base has not been extracted yet.

Testing

Backend unit tests

make test
# or
cd backend && uv run pytest -v

Frontend unit tests

cd frontend && npm run test:unit

End-to-end browser tests (Playwright)

Playwright auto-starts backend + frontend dev servers (see frontend/playwright.config.ts).

Marketing seed database (required): Several specs (datasources.spec.ts, synthesis-panel.spec.ts, and others) read data/marketing_seed.db, which is gitignored. Generate it once before running E2E:

cd backend
uv run python scripts/seed_marketing.py --source-only

This writes data/marketing_seed.db at the repo root. Omit --source-only if you also want the script to reset canvas metadata in the app DB.

make test:e2e
# or
cd frontend && npx playwright test

Headed mode for debugging: npx playwright test --headed

Shared helpers live in frontend/e2e/helpers.ts. Core specs: canvas.spec.ts, context-menu.spec.ts.

Connector integration tests (Docker)

Postgres, MySQL, and MongoDB integration tests require local containers:

make test-db-up          # Postgres :5433, MySQL :3307, MongoDB :27018
make test-postgres       # or test-mysql / test-mongodb / test-*-profiler / test-*-api
make test-db-down

Without these services, make test skips ~88 integration-marked tests by design.

Full CI pipeline (local)

make ci

Runs, in order: make install (backend + frontend deps) → make test (backend pytest) → make lint (ruff + eslint) → make build (frontend production build).

Coding Conventions

Python docstrings

Use Google-style docstrings on public classes and functions in backend/app/:

def sample_table(self, database: str, schema: str, table: str) -> TableSample:
    """Fetch random rows from a single table.

    Args:
        database: Catalog/database name.
        schema: Schema namespace.
        table: Table to sample.

    Returns:
        Sampled rows packaged as a ``TableSample``.

    Raises:
        ConnectorError: When the table is empty or sampling fails.
    """

Python linting (Ruff)

Configuration lives in backend/pyproject.toml under [tool.ruff]. Before committing backend changes:

cd backend
uv run ruff check app tests --fix
uv run ruff format app tests

Or from the repo root: make lint-backend.

TypeScript / frontend

ESLint 9 flat config in frontend/eslint.config.js. Run npm run lint or make lint.

Environment Variables Reference

Backend (backend/.env)

Variable Default Description
DATABASE_URL sqlite:///./chitral.db SQLAlchemy database URL
API_PREFIX /api URL prefix for all API routes
APP_NAME Chitral App name shown in docs
APP_VERSION 0.1.0 App version shown in docs
CORS_ORIGINS localhost Vite/CRA origins (JSON array) Allowed browser origins. Override for other hosts; never use ["*"] without auth.
DATASOURCE_ENCRYPTION_KEY — Optional Fernet key (url-safe base64) for encrypting saved data-source credentials at rest. Generate: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())". If unset, the backend writes backend/.datasource_key on first encrypt.
MONGODB_URL — Optional dev URI; prevents pydantic-settings from rejecting the env var. Connector requests still pass credentials explicitly.
OPENAI_API_KEY — Required for LLM-assisted JSON/VARIANT column synthesis
OPENAI_MODEL gpt-4o-mini OpenAI model name for JSON synthesis
SNOWFLAKE_ACCOUNT — Snowflake account identifier
SNOWFLAKE_USER — Snowflake username
SNOWFLAKE_PASSWORD — Snowflake password
SNOWFLAKE_WAREHOUSE — Snowflake warehouse name
SNOWFLAKE_DATABASE — Snowflake database name
SNOWFLAKE_SCHEMA — Snowflake schema name

Frontend (frontend/.env)

Variable Default Description
VITE_API_URL http://localhost:8000 Base URL for the backend API