Give Clea a list of websites. It crawls them politely, keeps them up to date on its own, and gives you a fast, clean search page for just those sites.
Cléa, from clé (the key). Self-hosted. No ads, no tracking, no API keys.
Clea is a small, self-hosted search engine with its own crawler. Point it at documentation sites, blogs or shops, and
it builds a full-text index you can search with phrases, exclusions, site: filters, typo suggestions, and a built-in
calculator and unit converter. It runs as one Python process with a React interface, in Docker, or as a fully static
copy on GitHub Pages where search happens in your browser.
Features · Quick start · Automatic crawling · Configuration · Docker · GitHub Pages · API
![]() |
![]() |
| Light mode | Sites, crawls and automatic crawling |
Search
- Full-text search with BM25 ranking (SQLite FTS5), stemming, and title/URL boosting
- Query-aware snippets with highlighted matches
- Search syntax:
"exact phrase",-exclude,prefix*,site:example.com,intitle:word - Falls back to "any word" when nothing matches every word, and suggests spelling corrections ("Did you mean …")
- Autocomplete as you type
- Instant answers in the browser: a calculator (
15% of 240,sqrt(2)*10,5!) and unit conversion (10 km in miles,72 f to c,1.5 gib in mb) - Shareable result URLs (
?q=…&p=2), pagination,/to focus search, light/dark theme - OpenSearch, so you can add Clea as a search engine in your browser
Crawling
- Crawls each site breadth-first within its host and path (e.g. only
/docs/), with page and depth limits - Respects robots.txt (including
Crawl-delayandSitemap),<meta name="robots">andrel="nofollow", waits between requests to the same host, and sends an identifying User-Agent - Reads sitemaps, follows canonical URLs and same-site redirects, normalises URLs and skips duplicates and non-HTML files
- Recrawls use
ETag/Last-Modified, so unchanged pages cost almost nothing; pages that disappear leave the index - Parallel: different sites are crawled at the same time (4 by default), while each site still gets one request at a time
- Automatic crawling: look for new pages every N minutes, and refresh every page every N hours or days, set from the admin panel. Sites can be left out individually
- Crawls run in the background with live progress, an optional time limit, and can be cancelled
- Pages that are mostly links (generated indexes, sitemaps) are ranked below the pages they point to
Running it
- One process: the Python server serves the API and the built web app
- An admin token protects site management whenever the server is reachable from the network
- Docker image and Compose file
- Static GitHub Pages build updated by a scheduled GitHub Action
Requirements: Python 3.11+ and Node.js 20+ (macOS or Linux; on Windows use Docker or the manual steps below).
git clone https://github.com/dsameendra/clea.git
cd clea
./start.sh # first run sets everything up, then Clea runs in the background
./stop.sh # stop itOpen http://localhost:6942, click the sitemap icon, add a site (for example
https://docs.python.org/3/tutorial/) and press Crawl new pages. Results become searchable as pages are indexed.
To load the demo's site list instead, run .venv/bin/python -m clea sites import sites.toml.
./start.sh creates a virtualenv in .venv, installs the Python packages, builds the web interface when its sources
changed, and starts the server, with the log in data/clea.log. It accepts the same CLEA_* settings as the server,
for example CLEA_PORT=8080 ./start.sh.
Manual setup
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cd frontend && npm ci && npm run build && cd ..
python -m clea serveRun the backend (python -m clea serve) and, in another terminal, the Vite dev server (it proxies /api to the
backend):
cd frontend
npm run dev # http://localhost:5173python -m clea serve [--host 127.0.0.1] [--port 6942]
python -m clea sites add https://example.com/docs/ --max-pages 200 --max-depth 3
python -m clea sites list
python -m clea sites remove 3
python -m clea sites import sites.toml [--prune]
python -m clea crawl [--site 1] [--recrawl] [--workers 4] [--max-seconds 600]
python -m clea schedule --discover-every 30m --refresh-every 7d [--max-run 30m] [--workers 4]
python -m clea schedule --no-discover --no-refresh # turn automatic crawling off
python -m clea search "flask blueprints"
python -m clea stats
python -m clea export frontend/public/data # static index for the browserAll settings are environment variables:
| Variable | Default | Meaning |
|---|---|---|
CLEA_DB |
data/clea.db |
SQLite database path |
CLEA_HOST / CLEA_PORT |
127.0.0.1 / 6942 |
Where the server listens |
CLEA_ADMIN_TOKEN |
(empty) | Protects site management and crawling. Required when CLEA_HOST is not a loopback address |
CLEA_ALLOWED_HOSTS |
Extra Host names to accept when no token is set (DNS-rebinding protection) |
|
CLEA_CORS_ORIGINS |
Browser origins allowed to call the API from another site | |
CLEA_CRAWLER_CONTACT |
URL or email added to the crawler's User-Agent so site owners can reach you | |
CLEA_MIN_DELAY |
1.0 |
Minimum seconds between requests to the same host |
CLEA_CRAWL_WORKERS |
4 |
Sites (hosts) crawled in parallel, until changed in the admin panel |
CLEA_DISCOVER_MINUTES |
0 |
Look for new pages this often (0 = off), until changed in the admin panel |
CLEA_RECRAWL_HOURS |
0 |
Refresh every page this often (0 = off), until changed in the admin panel |
CLEA_REQUEST_TIMEOUT |
15 |
Per-request timeout in seconds |
CLEA_MAX_PAGE_BYTES |
5242880 |
Pages larger than this are skipped |
CLEA_STATIC_DIR |
frontend/dist |
Built web app to serve |
echo "CLEA_ADMIN_TOKEN=$(openssl rand -hex 24)" > .env
docker compose up -dOpen http://localhost:6942 and enter the token under Settings (or when the admin panel asks). The index is
kept in the clea-data volume. By default Compose turns on automatic crawling: new pages hourly and a full refresh
weekly (CLEA_DISCOVER_MINUTES, CLEA_RECRAWL_HOURS). You can change both in the admin panel.
The live demo at https://dsameendra.github.io/clea/ searches the Python tutorial and the Flask, React and SQLite docs. GitHub Pages only serves static files, so the demo works differently from a self-hosted Clea:
.github/workflows/pages.ymlruns the Python crawler on the sites insites.toml(on every push tomain, weekly, or on demand).python -m clea exportwrites the index as JSON next to the built frontend.- The frontend is built with
VITE_SEARCH_MODE=static. It loads the index in a Web Worker and searches it in the browser with MiniSearch: same query syntax, typo tolerance, snippets and suggestions. The admin panel is hidden.
To enable it, go to Settings → Pages → Build and deployment and set Source to GitHub Actions. Then push to
main or run the workflow from the Actions tab. Keep sites.toml small, because every visitor downloads the
index (a few hundred pages is a few MB).
To try the static build locally:
python -m clea export frontend/public/data
cd frontend && npm run dev:staticOpen the admin panel (sitemap icon) → Automatic crawling:
- Look for new pages every … (minimum 5 minutes) re-reads each site's start page, plus its sitemaps at most every
6 hours. Newly linked pages are crawled, and pages whose sitemap
<lastmod>is newer than Clea's copy are fetched again. Unchanged pages answer304 Not Modified, so a check of an unchanged site usually costs one small request. - Refresh every page every … (minimum 1 hour) re-checks every known page, so edits and deleted pages are picked up.
- Stop a scheduled run after … keeps runs bounded. Sites crawled longest ago go first, so every site gets its turn.
- Sites crawled in parallel sets how many websites are crawled at once. Requests to any one website are never
parallel, and each one waits
CLEA_MIN_DELAY(or the site's robots.txtCrawl-delay) between requests.
The schedule runs inside python -m clea serve. Intervals count from the end of the previous run, and the run
history is stored in the database. A restart therefore doesn't trigger extra runs, and a run that came due while
Clea was off happens once, shortly after it starts. Use a site's ⋯ → Include in automatic crawls to leave it
out.
When a site reaches its Max pages, Clea keeps its existing pages up to date but queues newly found pages without downloading them. The site shows page limit reached; raise the limit to index the waiting pages.
On the GitHub Pages demo there is no server, so the equivalent is the workflow's schedule: cron.
clea/
glaneur.py crawler: frontier, robots.txt, politeness, redirects, sitemaps (French: "gatherer")
classeur.py HTML → title/description/main text/links; writes documents ("organiser")
servir.py query parsing, FTS5/BM25 search, snippets, suggestions, stats ("to serve")
sites.py site management jobs.py background crawl jobs + scheduler
schedule.py automatic crawling settings and "is a run due?"
server.py Flask API + web app cli.py command line
db.py schema & migrations export.py static index for GitHub Pages
frontend/src/
lib/staticEngine.js browser search engine (static mode)
lib/instant/ calculator and unit converter (no eval)
components/ search UI and admin panel
Documents are stored in SQLite and mirrored into an FTS5 index by triggers, so updates and deletions never leave stale entries behind.
Upgrading from Clea 1.x: the database moved from clea_db.db to data/clea.db. The first time Clea 2 creates
its database it imports the sites from an existing clea_db.db in the project folder. Their pages are crawled again,
because the old index can't be converted. The old file is left untouched.
| Method & path | |
|---|---|
GET /api/search?q=&page=&per_page= |
Search. Returns results[{url, title, snippet, host}], where title and snippet are [{text, match}] segments, plus total, suggestion and relaxed |
GET /api/suggest?q= |
Autocomplete |
GET /api/stats |
Index statistics |
GET/POST /api/sites, PATCH/DELETE /api/sites/<id> |
Manage sites (admin) |
GET /api/sites/<id>/urls?status=failed |
Crawl problems for a site (admin) |
POST /api/crawl {site_ids?, mode?: "new" | "recrawl"} |
Start a crawl job (admin) |
GET/PUT /api/crawl/settings |
Automatic crawling and parallelism settings, plus next/last run times (admin) |
GET /api/jobs, GET /api/jobs/<id>, POST /api/jobs/<id>/cancel |
Job progress (admin) |
pip install -r requirements-dev.txt
pytest # backend tests (spins up a local test website)
ruff check clea tests
cd frontend && npm test && npm run lintContributions are welcome! Fork the repository, create a branch, and open a pull request. Please run the tests first.
MIT, see LICENSE.
Flask · SQLite FTS5 · Beautiful Soup · Protego · React · Mantine · MiniSearch


