Tracepass

Autonomous Web Scraping and Media Extraction Framework

Tracepass is a hierarchical, multi-agent framework for intelligent, fingerprint-free web scraping, media extraction, and deep-crawl automation. It accepts natural language instructions and autonomously navigates websites to retrieve images, videos, audio files, documents, and structured data — without leaving browser fingerprints or triggering bot-detection systems.

Overview

Tracepass abstracts the complexity of web scraping into a single natural language interface. You describe what you want, and the agent pipeline resolves the URL, generates scraper code tailored to the target page, executes it, parses the result, and drills deeper until the target resource is found and downloaded.

It is designed for:

Researchers and data engineers who need repeatable, large-scale media collection workflows.
Developers building data pipelines that consume content from the open web.
Teams that require an autonomous scraping layer plugged into a larger LLM-based system.

Example invocation:

"Go to xyz.com and get all audio files related to classical music"

[Main Agent] -> [Researcher Sub-Agent] -> [Code Generator] -> [Executor] -> Results

Architecture

Tracepass/
├── agents/
│   ├── main_agent.py           # Orchestrator: routes tasks, manages crawl state
│   ├── researcher_agent.py     # Sub-agent: resolves URLs, maps site structure
│   └── code_generator.py       # Generates Scrapling scraper code dynamically per URL
├── tools/
│   ├── scraper_executor.py     # Executes generated Scrapling code in a sandboxed environment
│   ├── html_parser.py          # Strips CSS and JS, returns a clean semantic tag tree
│   └── media_extractor.py      # Handles downloading of images, video, audio, and documents
├── core/
│   ├── agent_loop.py           # Deep-crawl loop: iterates across pages until target is found
│   ├── url_resolver.py         # Validates and resolves relative URLs to absolute form
│   └── state_manager.py        # Tracks crawl state, visited nodes, and recursion depth
├── config/
│   └── settings.py             # LLM provider, depth limits, output paths, and timeouts
├── outputs/                    # Downloaded media and extracted data are written here
├── tests/
│   └── ...
├── .github/
│   ├── workflows/
│   │   ├── ci.yml              # Ruff lint, format checks, and pytest
│   │   └── gemini-pr-summary.yml  # Automated PR review via Gemini 2.5 Flash
│   └── scripts/
│       └── gemini_pr_summary.py   # Gemini PR summary generation script
├── pyproject.toml
└── README.md

How It Works

Step 1 — Natural Language Instruction

The user provides a plain-English request describing the target website and desired content:

"Get all product images from store.example.com for the shoes category"

Step 2 — Researcher Sub-Agent

The Researcher Sub-Agent is invoked first. It:

Resolves the target URL, running a web search if the user provides a partial or ambiguous reference.
Crawls the site's top-level navigation to map its structural entry points.
Returns the most relevant starting URL to the Main Agent.

Step 3 — Code Generation

The Main Agent receives the resolved URL and passes it to the Code Generator, which:

Inspects the URL structure and any available metadata.
Produces a Scrapling scraper script tailored to the specific page layout and content type.
Delivers the generated script to the Scraper Executor tool.

Step 4 — Scraper Executor

The Scraper Executor:

Runs the generated Scrapling code inside a sandboxed environment.
Returns raw HTML with all CSS and JavaScript stripped, yielding a clean semantic tag tree.
Passes the processed content back to the Main Agent for analysis.

Step 5 — Deep-Crawl Loop

The Main Agent analyzes the tag tree for links, src attributes, and URL patterns that lead to the target resource. If the resource is not present on the current page, it:

Generates a new scraper for the next-level URL.
Re-executes the pipeline.
Repeats this loop until the desired resource is located, then downloads and returns it.

[Instruction]
     |
[Researcher Agent] --> URL
     |
[Main Agent]
     |
[Code Generator] --> Scrapling Code
     |
[Executor] --> Raw HTML / Tag Tree
     |
[Main Agent analyzes] --> Target found?
     |                         |
     |                    YES  --> Download and Return
     |
    NO --> Go deeper (repeat from Code Generator)

Tech Stack

Component	Technology
Scraping Engine	Scrapling
Agent Framework	LangGraph / LangChain
LLM Backend	Configurable: OpenAI, Google Gemini, Bedrock, Ollama
HTML Parsing	BeautifulSoup4 + lxml
Media Downloading	yt-dlp (video and audio), httpx (images and files)
Execution Sandbox	RestrictedPython / subprocess isolation
State Management	LangGraph StateGraph
Package Management	uv
Linting/Formatting	Ruff
Testing	pytest
Automated PR Review	Gemini 2.5 Flash + CodeRabbit

Quickstart

Prerequisites

Python 3.12 or higher
uv (recommended) or pip
An API key for your chosen LLM provider (OpenAI, Gemini, etc.)

1. Clone the Repository

git clone https://github.com/Edge-Explorer/Tracepass.git
cd Tracepass

2. Install Dependencies

Using uv (recommended):

uv sync

Or using pip:

pip install -e .
playwright install   # Required by Scrapling for JavaScript-rendered pages

3. Set Up Environment Variables

cp .env.example .env

Edit .env and fill in your API credentials:

GEMINI_API_KEY=your_gemini_api_key_here
OPENAI_API_KEY=your_openai_api_key_here
LLM_PROVIDER=gemini
MODEL_NAME=gemini-2.5-flash
MAX_DEPTH=5
OUTPUT_DIR=./outputs

4. Run

from agents.main_agent import Tracepass

agent = Tracepass()
result = agent.run("Go to unsplash.com and download 10 images of mountains")
print(result)

Configuration

All runtime parameters are managed through config/settings.py or via environment variables defined in .env.

Parameter	Default	Description
`LLM_PROVIDER`	`gemini`	LLM backend to use: `gemini`, `openai`, `ollama`
`MODEL_NAME`	`gemini-2.5-flash`	Model identifier for the chosen provider
`MAX_DEPTH`	`5`	Maximum crawl recursion depth before aborting
`OUTPUT_DIR`	`./outputs`	Directory where downloaded media and data are saved
`LOG_LEVEL`	`INFO`	Logging verbosity: `DEBUG`, `INFO`, `WARNING`

Usage Examples

# Download images matching a query
agent.run("Get all images of sunsets from pexels.com")

# Download a video from a specific page
agent.run("Download the intro video from apple.com/macbook-pro")

# Extract structured tabular data
agent.run("Get all product names and prices from the electronics section of example-store.com")

# Collect audio files from a podcast directory
agent.run("Find all podcast audio files on xyz.com/podcasts")

# Locate and download a PDF document via deep crawl
agent.run("Find and download the annual report PDF from investor.example.com")

Why Scrapling

Tracepass is built on Scrapling because it provides a scraping foundation that is production-ready for adversarial environments:

Zero browser fingerprints. Scrapling mimics legitimate browser behavior at the TLS and HTTP/2 header level, making requests indistinguishable from real users.
Cloudflare and CAPTCHA bypass. Through Camoufox and stealth-patched Playwright, Scrapling successfully handles bot-detection layers that block conventional scrapers.
Static and dynamic page support. Scrapling selects the appropriate fetcher automatically — a lightweight HTTP client for static pages and a full browser engine for JavaScript-rendered content — minimizing overhead.
High-speed execution. Playwright with stealth patches provides a fast and reliable execution environment for complex, JS-heavy websites.

CI/CD Pipeline

Tracepass uses a fully automated CI/CD pipeline enforced on every pull request to main.

Automated Checks (GitHub Actions)

Every pull request triggers two mandatory workflow jobs defined in .github/workflows/ci.yml:

Lint and Format — runs ruff check . and ruff format --check . to enforce code style and catch common errors.
Tests — installs all development dependencies via uv sync --extra dev and runs the full test suite with pytest.

Both checks must pass before a pull request is eligible for merging.

Automated Pull Request Review

Every pull request automatically receives a structured code review comment generated by Gemini 2.5 Flash, delivered by the workflow in .github/workflows/gemini-pr-summary.yml. The review covers:

Executive Summary — plain-English overview of the changes.
Motivation and Root Cause Analysis — why the change was needed.
Step-by-Step Technical Solution — precise breakdown of the implementation.
File-by-File Breakdown — a detailed table documenting every changed file, the action taken, and the specific logic or configuration introduced.
Architecture, Reliability, and Security Considerations — analysis of design quality and potential risks.
Risk Assessment — overall risk rating with a list of edge cases to verify.
Reviewer Checklist — actionable verification steps for the code reviewer.

In addition, CodeRabbit is configured via .coderabbit.yaml to provide automated inline code review comments directly on the diff.

Branch Protection

The main branch is protected by the following ruleset:

Direct pushes to main are blocked for all collaborators, including repository owners.
All changes must be submitted as pull requests.
At least one approving review from a collaborator with write access is required before merging.
The Lint and Format and Tests status checks must pass before merging is permitted.
Force pushes and branch deletion are blocked.
Stale approvals are dismissed when new commits are pushed to an open pull request.

Development Setup

Install the development dependencies:

uv sync --extra dev

Run the linter:

uvx ruff check .

Run the formatter:

uvx ruff format .

Run the test suite:

uv run pytest -v

Requirements

Core dependencies are declared in pyproject.toml. Key packages include:

scrapling[all,fetchers]>=0.4.15
langchain
langgraph
langchain-openai
beautifulsoup4
lxml
httpx
yt-dlp
playwright
restrictedpython

Development dependencies:

pytest>=8.0.0
ruff>=0.9.0

Authentication and Login Engine

Tracepass is being built with a Login Engine designed to handle the full range of website authentication patterns without manual intervention for v1 supported flows. The following capabilities are planned for v1 and are currently in active design. Flows that require human involvement (2FA, CAPTCHA, new device verification) are handled via interactive prompts or plugin interfaces, not silently skipped.

Planned Supported Login Flow Types (v1)

Type	Description	Examples
Standard Single-Step	All fields visible simultaneously on page load	Reddit, Stack Overflow
Multi-Step / Split	Email on step 1, password revealed on step 2	Google, Microsoft, LinkedIn
Modal and Overlay	Login form triggered by a button, rendered in a popup	Twitter/X, Medium
iFrame and Shadow DOM	Inputs embedded in cross-origin frames	Enterprise SSO portals
OAuth / SSO	Delegation to a third-party identity provider	"Continue with Google / GitHub"
API-Based (SPA)	JavaScript-driven form with no traditional `<form>`	Most modern React/Vue applications

Planned Key Capabilities (v1)

Semantic field detection — identifies username, password, submit, and next-step fields using attribute patterns, ARIA labels, and placeholder text rather than hardcoded CSS selectors. Works on any site without prior configuration.
Human-realistic input timing — randomized inter-keystroke delays and simulated mouse paths to avoid bot-detection systems that measure interaction patterns.
Session persistence (planned) — after a successful login, cookies and localStorage are saved to ~/.tracepass/sessions/ with owner-only file permissions. Re-authentication for the same domain will be skipped on subsequent runs. Session cleanup, expiry, and revocation lifecycle are defined in the design document.
Secure credential storage — credentials are stored in the OS keyring (Windows Credential Manager, macOS Keychain, Linux Secret Service) with an AES-256-GCM encrypted file as fallback for headless environments. Credentials are never written to plaintext files or passed to the LLM.
Failure-aware retry policy — each failure type (CredentialRejected, PageLoadTimeout, AccountLocked, CaptchaRequired, etc.) has a specific retry or abort behavior to protect against account lockouts.
CAPTCHA solver integration — opt-in support for 2captcha and anticaptcha solver APIs, activated by setting CAPTCHA_SOLVER_API_KEY in .env.
2FA / OTP support — blocks on a terminal prompt for TOTP and SMS codes. For headless CI environments, a shell-safe environment variable TRACEPASS_OTP_<SAFE_DOMAIN>=<code> can be pre-set (domain normalized to uppercase with dots and dashes replaced by underscores, e.g., TRACEPASS_OTP_EXAMPLE_COM=123456).

What is Out of Scope

First-time account registration and sign-up flows are not supported in v1. The Login Engine requires an existing account. If no credentials or session are found for a domain, the engine surfaces a clear error directing the user to create an account manually and provide credentials.

For the complete design specification, architecture diagrams, edge case analysis, worst-case scenarios, complexity analysis, and implementation checklist, see docs/login-engine.md.

Design Documentation

All major architectural components of Tracepass have dedicated design documents in the docs/ directory. These documents serve as the authoritative reference for implementation decisions, edge case handling, and team alignment before coding begins.

Document	Component	Status
docs/login-engine.md	Autonomous Login Engine — authentication, session management, credential storage	In Progress

Design documents follow a standard structure covering: problem statement, architecture, execution strategy, edge cases, worst-case scenarios, time and space complexity, finalized design decisions, and an implementation checklist. Contributors working on a new component are expected to produce a design document and have it reviewed before writing any implementation code.

Roadmap

Multi-threaded parallel crawling across multiple URLs simultaneously
Browser session caching for faster repeat visits to the same domain
Structured output formats: JSON, CSV, and SQLite export
REST API wrapper for remote agent invocation
Dashboard UI for real-time crawl monitoring and history
Docker deployment support with pre-configured Playwright environment

Legal and Ethical Notice

Tracepass is intended for legitimate use cases only, including academic research, personal data archiving, and data collection from websites that explicitly permit it. Before using this tool against any website, you must review its robots.txt file and Terms of Service. The authors and contributors assume no responsibility for any misuse of this framework. Scraping websites in violation of their terms of service or applicable law is solely the responsibility of the operator.

Contributing

Contributions are welcome. Please open an issue first to discuss the scope and design of any significant changes. All pull requests must pass the CI checks and receive at least one approving review before they are eligible for merging.

See the CI/CD Pipeline section for details on the automated review and quality gate processes applied to every contribution.

License

MIT License. See LICENSE for the full text.

Name		Name	Last commit message	Last commit date
.github		.github
core		core
docs		docs
experiments		experiments
tests		tests
.coderabbit.yaml		.coderabbit.yaml
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
LICENSE		LICENSE
README.md		README.md
main.py		main.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation