Filter, clean, and sample URLs for web crawling, with language, spam, and tracker filters, link extraction, and a URL store for crawl scheduling. Python and CLI
-
Updated
Nov 16, 2026 - Python
Filter, clean, and sample URLs for web crawling, with language, spam, and tracker filters, link extraction, and a URL store for crawl scheduling. Python and CLI
A tool for extracting and filtering URLs, IPs, domains, and subdomains from text or web pages, with built-in web scraping capabilities.
Django UFilter provides a safe way to filter data via human-friendly URLs - A fork from django-url-filter
Deploy a firewall that should allow or drop traffic based on customisable rules
Rule-based phishing URL detector using SSL, domain age, and URL pattern analysis
A Bash script to filter and refine URL lists by applying various criteria like extensions, query parameters, and patterns.
panw-app-id-resources
.NET client for AI Agent Allowlist
Asynchronous HTTP/HTTPS forward proxy in Python (asyncio) with domain filtering, TLS SNI inspection, and an administrative web console.
Elixir client for AI Agent Allowlist
Java client for AI Agent Allowlist
VueJS 2.x - Ag-grid filter in query params
Go client for the AI agent allow list: page-type verdicts before an agent opens a URL
A simple Python-based HTTP proxy server with URL filtering. Blocks access to predefined URLs and forwards allowed HTTP requests. Perfect for learning proxy fundamentals and request filtering. Does not support HTTPS tunneling.
AI Agent Allow List API client: page-type verdicts (allow, deny, flag) before an AI agent opens a URL
Blocks and unblocks URLs, domains and IPs in a Forcepoint Web Security custom URL category through Policy Server transactions with rollback.
PHP client for the AI Agent Allow List API: allow, deny or flag verdicts before an AI agent opens a URL
A simple yet powerful Python-based web scraping tool designed to collect HTML content from a list of URLs. This utility supports clean error logging, URL filtering, and JSON output—great for collecting articles, product data, or testing site availability, this project offers a solid foundation for custom scraping workflows.
CRM system to manage leads, assign sales agents, track lead status, and analyze performance through dashboards and reports
Crawler foundation for Inelastic Search with URL filtering, trap detection, SimHash deduplication, and crawl statistics.
To associate your repository with the url-filtering topic, visit your repo's landing page and select "manage topics."