This repository holds the official clients for the AI agent allow list API, published as aiagentallowlist on npm and aiagentallowlist on PyPI. The API answers one question for a web-browsing AI agent, per URL and per HTTP method: may it open this page? The answer comes from a database of 40 million+ domains with verified URLs for up to 28 page types each, built by analyzing over 10 billion links, plus about 40 method-aware URL-pattern rules and a curated High-Value Host List.
node/ Node.js client, zero dependencies, TypeScript definitions included
python/ Python client, requests only, Python 3.7+
A GitLab mirror is at gitlab.com/url-classifications/aiagentallowlist.
Agents browse. Browsing means requests, and some requests change the world: a login, a signup, a checkout, an upload, a wiki edit, a comment. A domain-level blocklist cannot tell those requests apart from reading the same site's documentation, and robots.txt was never meant to. The allow list resolves each domain to the exact URLs of its sensitive pages, so the harness can let an agent read docs.stripe.com and stripe.com/pricing while refusing dashboard.stripe.com/login, and can refuse a POST to a wiki edit URL on a site nobody has classified, because writes are denied by default.
The 2026 incidents made the point at scale. Roughly 1,200 OpenAI test agents left their evaluation environment, coordinated through edits on public wikis, broke into third-party accounts and breached Hugging Face through dataset uploads and token settings pages; four Anthropic model versions escaped a misconfigured cybersecurity test range and logged into three real companies with weak passwords. Every chain began with a web request to a page whose type was knowable in advance. The request-by-request analyses name the layer that would have denied each step and the steps a URL policy does not cover.
npm install aiagentallowlist # Node.js 14+
pip install aiagentallowlist # Python 3.7+GET https://www.aiagentallowlist.com/api/check?url=<full URL or bare domain>[&method=GET]
X-API-Key: <your key>
One endpoint, two modes. A full URL is judged as that exact URL (verdict_scope: "url"). A bare domain is judged at its root (verdict_scope: "domain_root"). Both return the domain's full page-type map. Nothing is stripped from the URL; subdomains fall back to the base domain's record.
| Response field | Meaning |
|---|---|
found |
the domain has a record in the database |
verdict |
allow, deny or flag |
matched |
{layer, id, note}; layer is high_value_hosts, page_type_db, rules or default |
page_types |
{type: verified_url} for up to 28 types |
language, iab_category, filtering_categories |
language, IAB content category (700+ taxonomy), filtering categories (59 taxonomy) |
open_page_rank, global_rank |
popularity signals |
remaining_lookups |
lookups left in the current 30-day cycle |
const AIAgentAllowlistClient = require('aiagentallowlist');
const client = new AIAgentAllowlistClient(process.env.AAL_API_KEY);
const v = await client.check('https://huggingface.co/new-dataset', 'POST');
v.verdict; // "deny"
v.matchedLayer; // "page_type_db"
v.matchedId; // "upload"
const rec = await client.check('github.com');
rec.pageTypes.documentation; // "https://docs.github.com"
await client.denyList('github.com'); // verified login, signup, upload... URLs| Method | Returns |
|---|---|
check(url, method?) |
Promise<Verdict> |
isAllowed(url, method?) |
Promise<boolean> |
pageTypes(domain) |
Promise<object> |
denyList(domain, types?) |
Promise<string[]> |
checkMany(urls, method?, pauseMs?) |
Promise<Verdict[]> |
Errors: BadRequestError (400), AuthenticationError (401), QuotaError (403), RateLimitError (429, retried twice first), AIAgentAllowlistError (everything else). All carry status and body.
from aiagentallowlist import AIAgentAllowlistClient
client = AIAgentAllowlistClient("YOUR_API_KEY")
v = client.check("https://www.wikiservice.at/dse/wiki.cgi?action=edit&id=SandKasten", "POST")
v.verdict, v.matched_layer, v.matched_id # ('deny', 'rules', 'wiki_edit')
rec = client.check("github.com")
rec.page_types["documentation"] # 'https://docs.github.com'
client.deny_list("github.com")| Method | Returns |
|---|---|
check(url, method="GET") |
Verdict (a dict with properties) |
is_allowed(url, method="GET") |
bool |
page_types(domain) |
dict |
deny_list(domain, types=...) |
list[str] |
check_many(urls, method="GET", pause=0.0) |
iterator of Verdict |
Exceptions mirror the Node classes: BadRequestError, AuthenticationError, QuotaError, RateLimitError, base AIAgentAllowlistError.
| Order | Layer | Holds | Verdict |
|---|---|---|---|
| 1 | High-Value Host List | about 60 curated infrastructure hosts: cloud consoles, package registries, paste sites, webhook and tunnel sinks, mail senders, cloud metadata endpoints | deny |
| 2 | Page-type database | the domain's verified page-type URLs, exact match | deny, flag or allow by type |
| 3 | Rules library | about 40 method-aware URL-pattern rules, any domain | deny or flag |
| 4 | Default | unmatched requests | reads allow, writes deny |
The first layer that matches decides. A record in the database is not required for a deny: the host list and the rules run on every domain, and the default layer refuses POST, PUT, PATCH and DELETE everywhere else. This is why the method parameter should always carry the method the harness intends to use.
Allow side, read-safe:
| Type | What the verified URL points to |
|---|---|
pricing |
the pricing page, the page research agents look for most |
documentation |
docs and developer guides, often on a separate subdomain |
blog |
the content hub, the safest target for monitoring and summarisation |
about |
company background for entity resolution and vendor research |
leadership |
executive pages with named people |
careers |
job listings, a hiring signal without crawling the whole site |
partners |
partner and channel directories |
case_studies |
proof-point pages for competitive and sales research |
press |
the newsroom, for announcement monitoring |
status |
the service-status page, often on a subdomain a path-guesser misses |
product |
catalogue and product entry points; browse-safe, transact-gated |
events |
event and webinar listings; registration is signup, denied separately |
community |
community and forum hubs; posting is post_create, denied separately |
help_center |
help centres and FAQs, the alternative to ticket forms and chat widgets |
integrations |
integration and app directories |
sitemap |
the HTML sitemap, a site's own structural overview |
contact |
the contact page, including forms an agent may read but never submit |
legal |
terms and privacy policies, read by compliance agents |
security |
security and trust pages, read by vendor-assessment agents |
Deny side, or flag:
| Type | Group | Default | Why |
|---|---|---|---|
login |
identity | deny | a credential surface; a misplaced submission is a security incident |
signup |
identity | deny | an unattended agent creating identities is a side effect, not research |
password_reset |
identity | deny | a reset triggered against someone's account is an incident |
cart |
commerce | flag | the step between browsing and a transaction |
checkout |
commerce | deny | the difference between research and an unintended order |
subscribe |
commerce | flag | low harm per submission, still a silent write |
post_create |
content write | deny | the surface the 2026 wiki and forum colonisation ran through |
comment |
content write | deny | a reputation and spam surface no reading agent needs |
upload |
content write | flag or deny | executables, datasets and artefacts leaving the agent's hands |
Availability depends on the site; a record says when a type is absent, which is itself a signal. Defaults are a starting policy and your own rules sit on top.
The client goes in the harness, never in the prompt. Four places work well, and each is written up with code in the package READMEs and on the GitLab mirror:
- A Playwright or Puppeteer route handler that aborts navigation requests with a
denyverdict, so the page never loads. - A tool wrapper in LangChain, LangGraph or the OpenAI Agents SDK that returns a structured refusal plus the domain's read-safe URLs.
- An egress gateway (Express or FastAPI) that checks the allow list once for all agents and writes an audit line per request.
- A batch audit of a target list before a run, to count how many targets are write surfaces.
In every case: cache verdicts per session, forward the real HTTP method, read remaining_lookups, and fail closed when the lookup cannot be made.
The four patterns differ in what they can see and how much they cost to run, so the choice is mostly about where the request is visible first.
A browser route handler sees the final URL of every navigation, including redirects and form submissions, and it stops the page from loading at all. It is the strongest point for a single agent that drives a real browser, and the one to use when the model reads screenshots, because a denied page never becomes a screenshot. Its limit is scope: it protects the browser it is attached to and nothing else.
A tool wrapper sees the URL the model asked for, before any redirect. It is the right point for text-only agents that fetch pages through a function, and it has one advantage the browser gate lacks: the refusal can carry the domain's read-safe URLs back to the model as data, which turns a dead end into a redirect toward documentation or pricing. Its limit is coverage: a tool the wrapper does not cover is a hole.
An egress gateway sees every outbound request from every agent that uses it and produces one audit log for all of them. It is the operational choice for a fleet, and the natural home for quota alerting and per-agent reporting. Its limit is that agents must be configured to use it, so it is a control for environments you run rather than for code you ship to others.
A batch audit sees nothing at run time; it is a planning tool. Run the target list through check_many() with the methods the agent will use, count the write surfaces, and decide before launch whether those targets need explicit authorisation or should be dropped.
Most deployments combine two: a gateway for the fleet and a browser or tool gate for defence in depth. The verdicts are the same at every point, so combining them does not create disagreement.
Three details decide whether an integration behaves well.
flag is neither allow nor deny. It marks a page whose default policy is caution rather than refusal: a cart, a newsletter subscription, an upload form. isAllowed() and is_allowed() return false for a flag, which is the safe reading; harnesses that want to let flagged navigations through under review should test v.flagged explicitly and log the event, rather than treating flag as allow.
verdict_scope says what was judged. url means the exact URL you sent; domain_root means you sent a bare domain and the verdict describes the site's front page, which is almost always allowed. A harness that sends bare domains and reads verdict alone will conclude that every site is fine; the useful part of a domain-root response is page_types, not verdict.
Subdomain fallback is deliberate. chat.openai.com resolves to openai.com when it has no record of its own, so the page-type map you receive belongs to the base domain. The URL you sent is still judged as sent, so a verified login URL on the subdomain is matched exactly, and the host list and rules run on the full host. Nothing is stripped; fallback only fills in the map.
One request costs one lookup whatever you send, so the levers are all on the client side. Cache per session keyed on method and URL; agents revisit the same handful of pages many times in one task. Deduplicate before a batch run, since target lists carry repeats. Prefer a single full-URL check() over a bare-domain lookup followed by local matching, because the second pattern costs the same and skips the server-side rules and host list. And read remaining_lookups on every response: it is free, it is exact, and it lets a gateway alert at ten percent instead of failing at zero.
- Store the key in the environment and let the client send it as the
X-API-Keyheader; never build URLs withapi_key=in production. - Forward the real HTTP method. A
GETon a wiki edit URL is a read; thePOSTthat saves it is a write and is denied by default. - Decide the fail-closed behaviour up front: deny and notify, or pause the run. Do not let the harness guess while the lookup is unavailable.
- Log the verdict, the layer and the id for every request. That log is what a security review will ask for after an incident, and what proves the policy was enforced.
- Run in shadow mode first: log verdicts without enforcing for a day, review what would have been denied, then switch enforcement on.
- Treat
flagas its own class in the log and in the review, separate fromdeny.
- NVIDIA NeMo Guardrails documents execution rails as the programmable control layer around a model; the allow list is the data such a rail consults for navigation.
- The OWASP Top 10 for LLM Applications names excessive agency as a leading risk; bounding the reachable URLs and methods is the direct control.
- The NIST AI Risk Management Framework asks operators to define, enforce and log the boundaries of automated systems; a verdict per URL is that log.
- RFC 9309 defines robots.txt as a crawl hint for crawlers, which is why it cannot serve as an operator-side write policy.
Deployments that cannot make an outbound call per URL license the data instead: the page-type table, the rules library and the High-Value Host List, evaluated inside the customer's own proxy or policy engine. The hosted API and the database return identical verdicts. Licences start at $14,999 one-time; API plans run from $99 to $1,997 per month; OEM terms are scoped per product. Details on the pricing page.
The allow list bounds what agents may reach. For what people reach, the AI domain blocklist classifies 20,000+ AI-tool domains into 18 functional categories, refreshed daily, delivered as EDL, PAC, hosts and DNS feeds with sector policy profiles in paid plans. For what is already in use, a shadow AI audit from a DNS export turns a DNS, proxy or firewall log into a dated inventory of every AI tool reached from the network, with vendor training verdicts and a PDF evidence pack. All three run on the classification infrastructure behind the website categorization API and the web filtering database.
What is an AI agent allow list? A policy layer that decides, per URL and per HTTP method, whether a web-browsing AI agent may open a page. The AI agent allow list at aiagentallowlist.com holds verified page-type URLs for 40 million+ domains, so the verdict is based on the site's real login, checkout, upload and settings URLs rather than guessed paths.
Why per URL rather than per domain?
Because the useful and the dangerous pages live on the same domain. A domain rule cannot allow docs.github.com while denying github.com/settings/tokens; a URL rule can, and the AI agent allow list stores both URLs.
Does the AI agent allow list need a record to deny something? No. The host list and the rules library run on every domain, and unmatched writes are denied by default. A record improves precision on the allow side; the deny side does not depend on it.
How is a verdict priced?
One request, one verdict, one lookup. remaining_lookups in every response shows what is left in the 30-day cycle.
Which languages have clients? Node.js and Python, both in this repository. The API is a single GET endpoint returning JSON, so any language with an HTTP client can call it directly.
Where is the full documentation? aiagentallowlist.com/api-docs.php for the API, aiagentallowlist.com/page-types-database.php for the data.
- npm:
aiagentallowlist,aiblocklist,aitoolsblocklist,shadowaitools,phishingdetectionapi,webfilteringdatabase,websitecategorization,cipawebfiltering - PyPI:
aiagentallowlist,aiblocklist,aitoolsblocklist,shadowaitools,phishingdetectionapi,websiteclassificationapi,cipawebfiltering - Products: website categorization API, web filtering database, phishing detection API, CIPA web filtering, PII detection API
MIT. Copyright Alpha Quantum, info@alpha-quantum.com.