subscan

The absurdly fast, billion-scale subdomain intelligence engine you can run on one ordinary machine.

5.3 billion names. About 9 GiB on disk. 17 µs warm queries. Approximately zero index RAM.

subscan turns the public internet's scattered subdomain exhaust—Certificate Transparency, passive DNS, web archives, code search, threat intelligence, and active probing—into one brutally compact, self-hosted search engine.

No Elasticsearch cluster. No database server. No API tokens. No premium tier. No artificial scarcity. Just one Rust binary, your data, and answers fast enough to feel instantaneous.

The headline numbers
Extrapolated capacity	5.3B names ≈ 9 GiB disk
Measured index density	0.27 bytes/name on multi-level-heavy data
Warm query latency	17 µs p50 · 29 µs p99
Cold query latency	39 µs p50
Ingest throughput	~260k names/sec with bloom dedup
Certificate parsing	~730k certs/sec/core
Data sources	25+ free sources
Index memory model	mmap-backed · ~0 GiB heap

Internet-scale subdomain intelligence should not require internet-scale infrastructure.

Search the internet's attack surface like it is a local file

Give subscan an apex. Get every indexed subdomain in microseconds. Open a discovered host directly, copy the entire result set, or export it as raw text, JSON, or CSV.

The same index is available everywhere you need it:

a minimal dark-mode web interface;
a scriptable CLI;
a streaming HTTP API with text, JSON, and CSV responses;
an MCP server that gives AI agents direct access to the index;
an SSE firehose for newly discovered names;
OpenAPI documentation and Prometheus metrics out of the box.

$ subscan search example.com
api
cdn
mail
www
...

This is not a thin UI wrapped around somebody else's paid API. subscan owns the entire pipeline: collection, normalization, lossless deduplication, compression, indexing, querying, streaming, and serving.

Quick Start

1. Build

cargo build --release -p subscan-api -p subscan-ingest

2. Ingest data

mkdir data

# Tail live CT logs (starts filling index immediately)
./target/release/subscan-ingest ./data ./cursors.db --max-logs 4

# Or bulk-import from a free source
./target/release/subscan-backfill hagezi --data-dir ./data --list heavy

3. Serve

./target/release/subscan-api --data-dir ./data --listen 127.0.0.1:8080

Open http://127.0.0.1:8080. Instant search, live ticker, dark mode.

4. Query

# API
curl "http://127.0.0.1:8080/v1/search?apex=example.com"
curl "http://127.0.0.1:8080/v1/find?q=api"
curl "http://127.0.0.1:8080/v1/top?n=100"

# CLI
cargo install --path crates/subscan-cli
SUBSCAN_API_URL=http://localhost:8080 subscan search example.com

API Reference

Endpoint	Method	Description
`/v1/search?apex=`	GET	All subdomains for an apex (text/json/csv)
`/v1/find?q=`	GET	Substring search across all names
`/v1/top?n=`	GET	Most-subdomain-rich apexes
`/v1/stream?since=`	GET	SSE feed of newly indexed names
`/mcp`	POST	MCP server for AI agents
`/healthz`	GET	Health check
`/metrics`	GET	Prometheus metrics
`/openapi.json`	GET	OpenAPI spec

Rate limits: 100 requests/day/IP (abuse protection only, service is free).

Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  CT Logs     │     │  Bulk        │     │  Active      │
│  (live tail) │     │  (backfill)  │     │  (DNS probe) │
└──────┬───────┘     └──────┬───────┘     └──────┬───────┘
       │                    │                    │
       └────────────────────┼────────────────────┘
                            │
                     ┌──────▼──────┐
                     │  Segment    │  ← FrontZstdRev codec
                     │  Writer     │  ← bloom dedup
                     └──────┬──────┘
                            │
                     ┌──────▼──────┐
                     │  idx v3     │  ← mmap'd, binary-searched
                     │  (0.27 B/n) │  ← apex-level cache
                     └──────┬──────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
       ┌──────▼──────┐ ┌───▼────┐ ┌──────▼──────┐
       │  HTTP API   │ │  MCP   │ │  Web UI     │
       │  (axum)     │ │ server │ │  (askama)   │
       └─────────────┘ └────────┘ └─────────────┘

Performance

Metric	Value
Query latency (warm)	17 µs p50, 29 µs p99
Query latency (cold)	39 µs p50
Index density	0.27 bytes/name (multi-level-heavy)
Capacity (measured)	250M names = 1.4 GiB disk, ~0 GiB RAM
Capacity (extrapolated)	5.3B names ≈ 9 GiB disk
Cert parsing	~730k certs/sec/core
Writer throughput	~260k names/sec with bloom dedup

Data Sources

subscan aggregates 25+ free sources across CT logs, passive DNS, web archives, code search, and threat intelligence.

CT Logs & Certificate Transparency

Source	Type	Yield	Auth	Command
Chrome CT logs (live)	Real-time	~500M+ names	None	`subscan-ingest`
Chrome log list replay	Bulk CT	~1B names	None	`backfill ct-replay`
crt.sh	Bulk CT	~700M names	None	`backfill crtsh-bulk`
CertStream	Real-time CT	Continuous	None	`backfill certstream`
CT.FYI static tiles	Bulk CT	Varies	None	(built into ctlog crate)
xReverseLabs Open Data	Daily dump	Varies	Free account	`backfill xreverselabs --input FILE`

Passive DNS & Bulk DNS

Source	Type	Yield	Auth	Command
Rapid7 FDNS	Bulk DNS	~3B names	None	`backfill sonar-fdns`
OpenINTEL	Bulk DNS	~500M names	Academic	(manual download)
mnemonic PassiveDNS	API	Varies	None	`backfill mnemonic`
Robtex	API	Varies	None	`backfill robtex`
AlienVault OTX	API	~200M names	Free key	`backfill otx`
ThreatMiner	API	Varies	None	`backfill threatminer`
CZDS (ICANN zone files)	Bulk zone	~400M names	Credentials	`backfill czds`

Web Archives & Crawls

Source	Type	Yield	Auth	Command
Common Crawl	Bulk WARC	~500M names	None	`backfill commoncrawl`
Wayback Machine CDX	API	~300M names	None	`backfill wayback-cdx`
Internet Archive replay	Bulk WARC	Varies	None	`backfill ia-replay`
URLScan.io	API	~100M names	Free key	`backfill urlscan`

Code Search

Source	Type	Yield	Auth	Command
GitHub code search	API	~100M names	Free token	`backfill github-dork`
GitLab code search	API	~50M names	Free token	`backfill github-dork`
SourceGraph	API	~50M names	Free token	`backfill github-dork`

Threat Intelligence & OSINT

Source	Type	Yield	Auth	Command
Anubis DB (JLDC.me)	API	Varies	None	`backfill anubis`
Hudson Rock (Cavalier)	API	Varies	None	`backfill hudsonrock`
BeVigil	API	Varies	Free key	`backfill bevigil`
FOFA	API	Varies	Free key	`backfill fofa`
ZoomEye	API	Varies	Free key	`backfill zoomeye`
Netlas.io	API	Varies	Free key	`backfill netlas`

DNS Probing (Active)

Source	Type	Yield	Auth	Command
massdns + wordlist	Active DNS	~500M names	None	`backfill dns-probe`
HaGeZi lists	Curated	Varies	None	`backfill hagezi`
ProjectDiscovery Chaos	API	Varies	Free key	`backfill chaos`

Additional Free Sources (Catalogued)

The following sources are documented and available for future integration:

Source	Type	Free Limit	URL
MerkleMap	DNS DB	4B+ records	merklemap.info
DomainsProject.org	Bulk list	3.2B+ domains	domainsproject.org
CIRCL passiveDNS	API	Free	passive.circl.lu
DNSArchive	Search	509M+ domains	dnsarchive.com
cert.sh PostgreSQL	DB	Open	cert.sh (port 5432)
CertDB (Intrusix)	CT+Scan	1000/day	certdb.io
crtlog.com	CT+IP	20/day	crtlog.com
PhishStats	Feed	150/day	phishstats.info
Enterno.io	CT+DNS	100/day	enterno.io
OTI Labs	TLS	1000/mo	otilabs.io
EdgeDNS CT	CT	200/mo	edgedns.com
C99.nl	CT	Unlimited	c99.nl
Reconeer	Recon	10/day	reconeer.io
DomScan	Multi-source	10K credits/mo	domscan.io
Spamhaus DQS	DNS	Non-commercial	spamhaus.org
CAIDA Datasets	Bulk	Free (1yr old)	caida.org
ip.thc.org	rDNS	Complete IPv4	ip.thc.org
n0kovo wordlist	Wordlist	3M entries	github.com/n0kovo
All-In-One DNS WL	Wordlist	20.7M entries	github.com/danielmiessler

Backfill Commands

# Build all backfill tools
cargo build --release -p subscan-backfill

# CT & Certificate sources
subscan-backfill ct-replay --data-dir ./data --politeness 1500
subscan-backfill crtsh-bulk --data-dir ./data --politeness 2000
subscan-backfill certstream --data-dir ./data

# Bulk DNS
subscan-backfill sonar-fdns --data-dir ./data --dataset /path/to/file.json.gz
subscan-backfill czds --data-dir ./data          # needs CZDS_USERNAME/PASSWORD

# API-based sources (no auth)
subscan-backfill hagezi --data-dir ./data --list heavy
subscan-backfill anubis --data-dir ./data
subscan-backfill robtex --data-dir ./data
subscan-backfill mnemonic --data-dir ./data
subscan-backfill hudsonrock --data-dir ./data
subscan-backfill threatminer --data-dir ./data

# API-based sources (free keys)
OTX_API_KEY=...            subscan-backfill otx --data-dir ./data
URLSCAN_API_KEY=...        subscan-backfill urlscan --data-dir ./data
BEVIGIL_API_KEY=...        subscan-backfill bevigil --data-dir ./data
FOFA_EMAIL=... FOFA_KEY=.. subscan-backfill fofa --data-dir ./data
ZOOMEYE_API_KEY=...        subscan-backfill zoomeye --data-dir ./data
NETLAS_API_KEY=...         subscan-backfill netlas --data-dir ./data
CHAOS_API_KEY=...          subscan-backfill chaos --data-dir ./data

# Web & code search
subscan-backfill wayback-cdx --data-dir ./data
subscan-backfill github-dork --data-dir ./data --github-token ghp_...

# Active DNS
subscan-backfill dns-probe --data-dir ./data --qps 500
subscan-backfill commoncrawl --data-dir ./data --crawl CC-MAIN-2024-10

# Utilities
subscan-backfill top-walk --data-dir ./data     # regenerate /v1/top artifact
subscan-backfill ia-replay --data-dir ./data --log-url <url>
subscan-backfill xreverselabs --data-dir ./data --input /path/to/extract.txt

All commands support --politeness MS to control request delay and --domains FILE for custom target lists.

Deployment

See deploy/DEPLOY.md for systemd units and cloudflared tunnel setup.

# Quick local deploy
cargo build --release
./target/release/subscan-api --data-dir ./data --listen 0.0.0.0:8080

Configuration

Env Variable	Description
`SUBSCAN_API_URL`	API server URL for CLI
`CHAOS_API_KEY`	ProjectDiscovery Chaos API key
`CZDS_USERNAME` / `CZDS_PASSWORD`	ICANN CZDS credentials
`OTX_API_KEY`	AlienVault OTX API key
`URLSCAN_API_KEY`	URLScan.io API key
`BEVIGIL_API_KEY`	BeVigil API key
`FOFA_EMAIL` / `FOFA_KEY`	FOFA credentials
`ZOOMEYE_API_KEY`	ZoomEye API key
`NETLAS_API_KEY`	Netlas.io API key
`GITHUB_TOKEN`	GitHub personal access token

Crate Structure

Crate	Purpose
`subscan-normalize`	Hostname normalization and splitting
`subscan-parse`	Subdomain label parsing and validation
`subscan-codec`	Five-codec compression family (FrontZstd, FrontZstdRev, Fsst, Dict, RevAware)
`subscan-index`	Ultra-dense segment index with bloom filter
`subscan-ctlog`	Certificate Transparency log parsing (RFC 6962, Chrome log list, CT.FYI)
`subscan-query`	Query index with apex-level cache and decode_apex
`subscan-api`	HTTP API, web UI, MCP server, rate limiting
`subscan-cli`	Command-line client
`subscan-ingest`	Live CT log ingestion pipeline
`subscan-backfill`	25 backfill data source modules
`subscan-bench`	Benchmark harness

License

Licensed under the MIT license.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
crates		crates
deploy		deploy
docs/screenshots		docs/screenshots
testdata/fixtures		testdata/fixtures
.gitignore		.gitignore
Cargo.toml		Cargo.toml
LICENSE-MIT		LICENSE-MIT
README.md		README.md

Navigation Menu

Repository files navigation

subscan

The absurdly fast, billion-scale subdomain intelligence engine you can run on one ordinary machine.

Search the internet's attack surface like it is a local file

Quick Start

1. Build

2. Ingest data

3. Serve

4. Query

API Reference

Architecture

Performance

Data Sources

CT Logs & Certificate Transparency

Passive DNS & Bulk DNS

Web Archives & Crawls

Code Search

Threat Intelligence & OSINT

DNS Probing (Active)

Additional Free Sources (Catalogued)

Backfill Commands

Deployment

Configuration

Crate Structure

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

subscan

The absurdly fast, billion-scale subdomain intelligence engine you can run on one ordinary machine.

Search the internet's attack surface like it is a local file

Quick Start

1. Build

2. Ingest data

3. Serve

4. Query

API Reference

Architecture

Performance

Data Sources

CT Logs & Certificate Transparency

Passive DNS & Bulk DNS

Web Archives & Crawls

Code Search

Threat Intelligence & OSINT

DNS Probing (Active)

Additional Free Sources (Catalogued)

Backfill Commands

Deployment

Configuration

Crate Structure

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages