smart-email-tracker

Privacy-aware email open tracking with intelligent filtering for proxy prefetches, Apple Mail Privacy Protection, security scanners, and sender self-opens.

Features

• 1x1 transparent tracking pixel (GIF or PNG) served with strict no-cache headers
• Per-message stats with a real signal model: distinct reads, repeat views, sender self-opens, proxy refreshes, and ignored prefetches — not just "opens"
• Heuristic classification of every request: Gmail image proxy, Apple MPP, Microsoft Defender / Exchange Online Protection, mail-security scanners, generic bots, humans
• Fingerprint + time-window deduplication to separate distinct reads from repeats
• Sender self-open exclusion by IP/CIDR, user-agent substring, or fingerprint
• Optional GeoIP (MaxMind mmdb) with CDN header fallback
• Simple REST API + self-contained HTML dashboard
• Optional API key authentication

Architecture

POST /messages        → create a tracked message, get back a pixel URL
GET  /pixel/{id}.{ext} → record the request, serve the pixel
GET  /stats/{id}       → detailed stats + event list (JSON)
GET  /stats            → paginated summary of all messages
GET  /dashboard/{id}   → minimal HTML dashboard
GET  /health           → liveness check

Pipeline for each pixel request (service.py), in order:

1. Unknown tracking id → pixel still returned, event recorded with counted=False, reason=unknown_tracking_id (existence is never leaked).
2. Sender exclusion → is_sender=True, counted=False.
3. Classification (proxy / scanner / human heuristics).
4. Prefetch heuristic → non-human fetches within SET_PREFETCH_WINDOW_SECONDS of sent_at are is_prefetch=True, counted=False.
5. Dedupe → a counted event from the same fingerprint inside SET_DEDUPE_WINDOW_SECONDS makes this one is_duplicate=True, counted=False.

One-click cloud deployment (no technical setup)

The repo ships a render.yaml blueprint, so you can run your own private tracker in a few minutes without installing anything:

1. Click the Deploy to Render button above (create a free Render account if asked).
2. Give the blueprint any name and click Apply. Render builds the Docker image and starts the service — wait until it shows Live (usually 2–5 minutes).
3. Open the smart-email-tracker service. Near the top you'll see your public URL, e.g. https://smart-email-tracker-xxxx.onrender.com. Click the copy icon next to it.
4. Still on the service page, open the Environment tab and copy the value of SET_API_KEY (Render generated a random secret for you).
5. In Chrome, right-click the extension icon → Options. Paste the public URL into API base URL and the key into API key, then Save.
6. Click the extension icon — the setup notice disappears and the popup is ready to use.

To check the server yourself, visit <your URL>/health — it should return {"status":"ok"}.

What the blueprint sets up

• A Docker web service built from the repo's Dockerfile, health-checked on /health.
• A 1 GB persistent disk mounted at /data; SQLite is stored at /data/tracker.db so your data survives deploys and restarts. Persistent disks need a paid instance (the blueprint uses starter, Render's smallest). If you prefer the free tier, remove the disk block and note that data is lost on every restart.
• SET_API_KEY generated automatically (generateValue: true) — you never have to invent or commit a secret.
• SET_TRUST_PROXY_HEADERS=true (correct behind Render's proxy).

Configuration & security notes

• Everything else is configured through environment variables (see Configuration); edit them in Render's Environment tab and the service redeploys.
• Keep SET_API_KEY secret. Anyone with the URL and key can read your stats and create tracked messages. Rotate it by editing the variable in Render and updating the extension options.
• /pixel/* and /health are intentionally public — a tracking pixel must be loadable by any mail client.
• The extension stores the URL and key only in your browser profile (chrome.storage.local) and sends the key only to the URL you configured.
• Open tracking may require consent/disclosure under GDPR, ePrivacy and similar laws — see Privacy & security.

Quick start

uv sync
uv run uvicorn smart_email_tracker.main:app --reload

Docker:

docker build -t smart-email-tracker .
docker run -p 8000:8000 -e SET_API_KEY=secret smart-email-tracker

Tests & lint:

uv run pytest -q
uv run ruff check .

Configuration

All settings are environment variables prefixed SET_ (see .env.example).

Variable	Default	Description
SET_DATABASE_URL	sqlite:///./tracker.db	SQLAlchemy database URL
SET_API_KEY	unset	If set, required as X-API-Key header on all endpoints except /pixel/* and /health
SET_PREFETCH_WINDOW_SECONDS	10	Non-human fetches within this window after sent_at are ignored as prefetches
SET_DEDUPE_WINDOW_SECONDS	300	Same tracking id + fingerprint within this window = repeat view
SET_TREAT_EARLY_HUMAN_OPENS_AS_PREFETCH	false	Also ignore human-classified opens inside the prefetch window
SET_EXCLUDED_SENDER_IPS	empty	Comma-separated IPs or CIDRs to exclude as sender self-opens
SET_EXCLUDED_SENDER_USER_AGENTS	empty	Comma-separated UA substrings (case-insensitive) for sender exclusion
SET_EXCLUDED_SENDER_FINGERPRINTS	empty	Comma-separated fingerprints for sender exclusion
SET_EXTRA_BOT_USER_AGENTS	empty	Additional bot UA markers appended to the built-in list
SET_GEOIP_DB_PATH	unset	Path to MaxMind GeoLite2-City .mmdb (requires geoip extra)
SET_TRUST_PROXY_HEADERS	true	Use first X-Forwarded-For hop as client IP
SET_PIXEL_FORMAT	gif	Pixel body format: gif or png
SET_CORS_ORIGINS	empty	Comma-separated origins allowed to call the API from browsers (enables Access-Control-Allow-Origin for GET requests with the X-API-Key header). Not needed for the bundled Chrome extension — it uses host_permissions instead of CORS

API usage

# Create a tracked message (id auto-generated if omitted)
curl -X POST localhost:8000/messages \
  -H 'Content-Type: application/json' \
  -d '{"recipient": "user@example.com", "subject": "Hello"}'
# → {"tracking_id": "abc123...", "pixel_url": "http://localhost:8000/pixel/abc123....gif"}

curl localhost:8000/stats/abc123...
curl 'localhost:8000/stats?limit=20&offset=0'
curl localhost:8000/dashboard/abc123...   # open in browser instead

With SET_API_KEY configured, add -H 'X-API-Key: secret' to /messages and /stats calls. /pixel/* and /health are always public (a pixel must be fetchable by any mail client).

Embedding the pixel

<img src="https://tracker.example.com/pixel/abc123....png" width="1" height="1" alt="">

The path extension (.png/.gif) is cosmetic — the served format is set by SET_PIXEL_FORMAT.

How classification works

Every pixel request is classified into one of: google_image_proxy, apple_mpp, microsoft_defender, security_scanner, bot, human.

Google Image Proxy (google_image_proxy)

• UA contains GoogleImageProxy — canonical: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
• Via header contains ggpht.com or google
• IP inside Google netblocks: 66.102.0.0/20, 66.249.80.0/20, 64.233.160.0/19, 72.14.192.0/18, 209.85.128.0/17, 74.125.0.0/16, 35.190.247.0/24, 35.191.0.0/16

Gmail fetches through this proxy roughly at open time but may cache images, so treat these as proxied opens with unreliable timing. They are counted as reads but shown separately in by_classification.

Apple Mail Privacy Protection (apple_mpp)

• UA contains an Apple token (other than AppleWebKit) with Mozilla/5.0 and no Version/ token, or
• IP in Apple/iCloud Private Relay ranges (17.0.0.0/8, 104.28.0.0/16, 172.224.0.0/12) with a bare Mozilla/5.0 UA

MPP pre-fetches remote images when mail arrives — regardless of whether the recipient ever opens it. Inside the prefetch window these are ignored; later hits are counted as proxied reads and separated in by_classification so you can decide how much to trust them.

Microsoft Defender / Exchange Online Protection (microsoft_defender)

• UA containing Microsoft-Defender, MSOffice, or Safe Links
• X-MS-Exchange-* headers present
• IP in EOP ranges (40.94.0.0/16, 40.107.0.0/16, 52.100.0.0/14, 104.47.0.0/17, 2a01:111::/32) — higher confidence when the UA is empty or a generic MSIE/Trident UA with no Accept-Language

Note: Outlook-iOS/ and Microsoft Outlook user agents are real mail clients and are deliberately not classified as Defender.

Security scanners & bots (security_scanner, bot)

• UA matches (case-insensitive): bot, crawler, spider, scanner, Proofpoint, Mimecast, Barracuda, Symantec, Cisco, IronPort, Sophos, TrendMicro, FireEye, Palo Alto, Zscaler, curl, wget, python-requests, Go-http-client, HeadlessChrome, PhantomJS (extend via SET_EXTRA_BOT_USER_AGENTS)
• Accept header present but contains no image/ type → bot (low confidence)
• Empty UA, or generic UA with neither Accept-Language nor Referer → bot

Human

Anything matching none of the above is human.

Dedup semantics

Stats distinguish:

• distinct_reads — counted events (not sender, not prefetch, not duplicate)
• repeat_views — same fingerprint again within the dedupe window
• sender_views — excluded self-opens
• proxy_refreshes — repeat views where classification isn't human (a proxy re-fetching, not a person re-reading)
• prefetch_ignored — automated fetches inside the prefetch window

A fingerprint is sha256(ip | user_agent | accept_language)[:32] — stable enough for dedupe, not reversible to identity.

Known limitations — read before trusting the numbers

No system can perfectly identify email opens. These counts are estimates:

• Apple MPP pre-fetches every remote image, so Apple Mail users appear to "open" mail they never saw.
• Gmail's proxy hides the real client IP and may cache the pixel — a fetch is not guaranteed to correspond to a human open at that moment.
• Defender and mail-security scanners actively fetch images and links; heuristics catch most but not all of them.
• Images may be blocked entirely — many clients and privacy tools never load the pixel, so "no open" does not mean "not read".
• IP-based signals depend on published netblocks that change over time.

The value of this tracker is that it separates these categories instead of lumping them into one misleading "opens" number.

Chrome extension

A Manifest V3 companion extension lives in extension/ — plain HTML/CSS/JS, no build step.

Load it

1. Open chrome://extensions
2. Enable Developer mode
3. Load unpacked → select the extension/ directory

Configure

Right-click the extension icon → Options (or use the options link on chrome://extensions). Set:

• API base URL — your backend URL (e.g. the Render URL from the one-click deployment, or http://localhost:8000 when running locally); validated to http/https, trailing slashes stripped. Until a URL is saved, the popup shows a setup notice with a button to open this page instead of silently calling localhost.
• API key — optional; stored in chrome.storage.local in the browser profile, sent only as an X-API-Key header to the configured base URL, never logged.

Permissions

• storage — persist the base URL, API key, and last-viewed tracking id.
• host_permissions (http://localhost/*, http://127.0.0.1/*, https://*/*) — lets the popup fetch() your backend. https://*/* is broad because the backend URL is user-configured; if you self-host on a fixed domain, narrow it in manifest.json to that origin.

CORS is not required for the extension (host_permissions grants the popup its own fetch privileges). For other browser-based clients on real web pages, set SET_CORS_ORIGINS on the backend.

Usage

Click the toolbar icon, paste a tracking id, hit Load. The popup shows summary cards (distinct reads, repeat views, sender views, proxy refreshes, ignored prefetches, first/last read), classification chips, derived alerts (e.g. "Apple MPP detected — open timing unreliable", "No genuine reads yet"), and a table of the last 20 events. A link opens the full /dashboard/{id} page. All API data is rendered via textContent — nothing from the server is injected as HTML.

Extension tests

cd extension && npm test

Uses Node's built-in node:test runner (Node 20+) against the pure logic in extension/src/lib.js — no npm dependencies.

Privacy & security

• Fingerprints are one-way SHA-256 hashes; raw IP/UA are stored per event — configure retention and purge open_events as your policy requires.
• Collect only what you need: recipient, subject, and metadata are optional. Do not put personal data in metadata unless you intend to store it.
• Open tracking may require consent or disclosure under GDPR/ePrivacy and similar laws — check your obligations and disclose tracking in your privacy policy.
• Set SET_API_KEY in any deployment reachable by others; without it, /stats and /messages are unauthenticated.
• SET_TRUST_PROXY_HEADERS should be false if the app is not behind a proxy that overwrites X-Forwarded-For, or clients can spoof their IP.

License

MIT — see LICENSE. Copyright CommunityPokeOrg.