API.md

API Documentation

Complete REST API reference for CiberWebScan (Beta).

Quick Start

Starting the API

ciberwebscan api
# Server runs on http://localhost:8000
# API docs: http://localhost:8000/docs
# ReDoc docs: http://localhost:8000/redoc

Authentication

The API requires an API key via the X-API-Key header:

curl -X POST "http://localhost:8000/api/analyze" \
  -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Response Format

All responses follow a consistent structure:

{
  "success": true,
  "data": { ... },
  "error": null,
  "timestamp": "2026-01-01T00:00:00Z",
  "download_token": "token-uuid",
  "download_url": "/api/download/token-uuid"
}

Endpoints

Health & Status

GET /health

Basic health check endpoint (no authentication required).

Response:

{
  "status": "healthy",
  "version": "2.4.0",
  "message": "CiberWebScan API is running",
  "timestamp": "2026-01-01T00:00:00Z"
}

GET /health/ready

Readiness check for container orchestration (no authentication required).

Response:

{
  "status": "ready",
  "version": "2.4.0",
  "message": "CiberWebScan API is ready to accept requests",
  "timestamp": "2026-01-01T00:00:00Z"
}

Security Analysis

POST /api/analyze

Perform comprehensive security analysis on a URL.

Includes:

• SSL/TLS certificate analysis
• Technology fingerprinting
• Security headers evaluation
• CVE lookup and vulnerability intelligence

Request:

{
  "url": "https://example.com",
  "ssl": true,
  "fingerprint": true,
  "analyze_headers": true,
  "cve": true,
  "ssl_verify": true,
  "timeout": 30.0,
  "ssl_timeout": 10.0,
  "deep_scan": false,
  "cve_sources": ["nvd"],
  "cve_limit": 100,
  "cve_severity": null,
  "headers": {},
  "cookies": {}
}

Parameters:

Parameter	Type	Default	Description
url	string	required	Target URL
ssl	boolean	true	Perform SSL/TLS analysis
fingerprint	boolean	true	Detect technologies
analyze_headers	boolean	true	Analyze security headers
cve	boolean	true	Lookup CVEs
ssl_verify	boolean	true	Verify SSL certificates
timeout	float	30.0	Request timeout (1.0-300.0)
ssl_timeout	float	10.0	SSL analysis timeout (1.0-120.0)
deep_scan	boolean	false	Deeper fingerprinting
cve_sources	array	[]	CVE sources: nvd, vulners, circl
cve_limit	integer	100	Max CVEs to retrieve (1-1000)
cve_severity	string	null	Filter by severity: critical, high, medium, low, info
headers	object	{}	Custom HTTP headers
cookies	object	{}	Cookies
proxy	string	null	HTTP proxy URL
user_agent	string	null	Custom User-Agent
check_robots	boolean	false	Respect robots.txt
enrich_exploits	boolean	false	Enrich CVEs with exploit info
export	string	null	Export file path
export_format	string	json	json, jsonl, csv, or html

Response:

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "timestamp": "2026-01-01T00:00:00Z",
    "ssl_analysis": {
      "protocol_version": "TLSv1.3",
      "certificate": {
        "subject": "CN=example.com",
        "issuer": "...",
        "valid_from": "2026-01-01T00:00:00Z",
        "valid_until": "2026-01-01T00:00:00Z"
      },
      "ciphers": [...],
      "vulnerabilities": []
    },
    "fingerprint": {
      "technologies": [
        {
          "name": "nginx",
          "version": "1.24.0",
          "category": "Web Servers"
        }
      ]
    },
    "headers_analysis": {
      "headers": {...},
      "security_issues": [...]
    },
    "cve_results": [
      {
        "id": "CVE-2026-1234",
        "severity": "HIGH",
        "description": "...",
        "cvss_score": 7.5
      }
    ]
  },
  "download_token": "uuid-token",
  "download_url": "/api/download/uuid-token",
  "timestamp": "2026-01-01T00:00:00Z"
}

Scraping

POST /api/scrape

Scrape a single URL with flexible extraction options.

Supports:

• Static scraping (BeautifulSoup)
• Dynamic scraping (Playwright for JavaScript-rendered content)
• CSS selector extraction
• Structured data extraction with schemas
• Pagination

Request:

{
  "url": "https://example.com",
  "dynamic": false,
  "wait_for": null,
  "selector": ".product",
  "attributes": ["id", "title", "price"],
  "extraction_schema": null,
  "timeout": 30.0,
  "headers": {},
  "cookies": {}
}

Parameters:

Parameter	Type	Default	Description
url	string	required	Target URL
dynamic	boolean	false	Use Playwright for JS rendering
wait_for	string	null	CSS selector to wait for (dynamic mode)
selector	string	null	CSS selector for focused extraction
attributes	array	[]	Attributes to extract from matched elements
extraction_schema	object	null	Structured extraction schema
pagination_selector	string	null	Selector for pagination links
pagination_limit	integer	1	Max pages to traverse (1-1000)
timeout	float	30.0	Request timeout in seconds (1.0-120.0)
headers	object	{}	Custom HTTP headers
cookies	object	{}	Cookies to include
proxy	string	null	HTTP/HTTPS proxy URL
user_agent	string	null	Custom User-Agent
check_robots	boolean	true	Respect robots.txt
extract_forms	boolean	false	Extract HTML forms and their fields
export	string	null	Export file path
export_format	string	json	json, jsonl, csv, or html

Response:

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "timestamp": "2026-01-01T00:00:00Z",
    "title": "Page Title",
    "links": [...],
    "images": [...],
    "forms": [...],
    "scripts": [...],
    "extracted_data": [...]
  },
  "download_token": "uuid-token",
  "download_url": "/api/download/uuid-token",
  "timestamp": "2026-01-01T00:00:00Z"
}

Error behavior:

Real scraping failures (HTTP 4xx/5xx responses, connection errors, robots.txt blocks, browser timeouts) are returned as 500 Internal Server Error — never as HTTP 200 with success: true. The detail message carries the original failure reason (for example HTTP 404).

Scenario	Status	detail
Scraping failure (404, connection, robots)	500	Original reason (e.g. HTTP 404)
SSRF-protected target (metadata/private IP)	500	SSRF protection reason
Malformed request body	422	FastAPI validation error
Missing or invalid API key	401	Authentication error

POST /api/scrape/batch

Scrape multiple URLs in batch.

Request:

{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "dynamic": false,
  "selector": ".item",
  "timeout": 30.0,
  "headers": {},
  "cookies": {}
}

Response:

{
  "success": true,
  "data": {
    "job_id": "uuid-string",
    "results": [
      {
        "url": "https://example.com/page1",
        "success": true,
        "extracted_data": [...]
      }
    ],
    "failed_urls": [
      {
        "url": "https://example.com/page3",
        "error": "Timeout"
      }
    ],
    "total_success": 2,
    "total_failed": 1
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

Batch error behavior:

• Mixed batches (some URLs succeed, some fail) return HTTP 200 with success: true; failed URLs are listed individually in failed_urls.
• If every URL fails, the endpoint returns HTTP 500 and detail summarizes the per-URL errors.

Attack Simulation

POST /api/attack

Perform controlled security attack simulations.

IMPORTANT: Only test systems you own or have explicit written authorization to test. Unauthorized security testing is illegal.

Supports:

• XSS (Cross-Site Scripting) detection
• SQL Injection testing
• Path Traversal vulnerability testing
• Directory Enumeration
• CSRF (Cross-Site Request Forgery) detection
• Subdomain enumeration (DNS brute force)
• OS Command Injection detection (GET, POST forms, and POST/JSON bodies)
• Custom payloads and wordlists

Request:

{
  "url": "https://testsite.example.com",
  "xss": true,
  "sqli": true,
  "traversal": true,
  "enumeration": true,
  "csrf": true,
  "subdomain": true,
  "command_injection": true,
  "json_body": { "cmd": "id" },
  "all_attacks": false,
  "intensity": "medium",
  "max_payloads": 50,
  "timeout": 10.0,
  "delay_between_requests": 0.1,
  "concurrent_requests": 1,
  "skip_dangerous_payloads": true,
  "user_consent": true
}

Parameters:

Parameter	Type	Default	Description
url	string	required	Target URL
xss	boolean	null	Test XSS vulnerabilities
sqli	boolean	null	Test SQL injection
traversal	boolean	null	Test path traversal
enumeration	boolean	null	Directory enumeration
csrf	boolean	null	Test CSRF vulnerabilities
subdomain	boolean	null	Enumerate active subdomains
command_injection	boolean	null	Test OS command injection
json_body	object	null	JSON body template for POST/JSON injection testing (each key is tested one at a time)
all_attacks	boolean	false	Enable all attack types
intensity	string	medium	low, medium, or high
max_payloads	integer	null	Max payloads per attack (1-1000)
custom_payloads_file	string	null	Path to custom payloads
custom_wordlist	string	null	Custom wordlist for enumeration
timeout	float	10.0	Request timeout (1.0-300.0)
delay_between_requests	float	0.1	Delay between requests (seconds)
concurrent_requests	integer	1	Concurrent requests (1-10)
skip_dangerous_payloads	boolean	true	Skip dangerous payloads
scope_urls	array	[]	URLs to scope attack to
export	string	null	Export file path
export_format	string	json	json, jsonl, csv, or html
headers	object	{}	Custom HTTP headers
cookies	object	{}	Cookies
proxy	string	null	HTTP proxy URL
user_agent	string	null	Custom User-Agent
verbose	boolean	false	Verbose output
user_consent	boolean	required	Must be true to proceed

Response:

{
  "success": true,
  "data": {
    "url": "https://testsite.example.com",
    "timestamp": "2026-01-01T00:00:00Z",
    "attack_results": {
      "xss": {
        "vulnerabilities_found": 2,
        "payloads_tested": 50,
        "results": [
          {
            "type": "XSS",
            "parameter": "search",
            "payload": "<script>alert(1)</script>",
            "status": "vulnerable",
            "severity": "HIGH"
          }
        ]
      },
      "sqli": {
        "vulnerabilities_found": 0,
        "payloads_tested": 50,
        "results": []
      },
      "traversal": {
        "vulnerabilities_found": 1,
        "payloads_tested": 30,
        "results": [...]
      },
      "enumeration": {
        "paths_found": 15,
        "common_paths_tested": 1000,
        "results": [...]
      }
    },
    "total_vulnerabilities": 3,
    "risk_score": 72
  },
  "download_token": "uuid-token",
  "download_url": "/api/download/uuid-token",
  "timestamp": "2026-01-01T00:00:00Z"
}

Error Response (Missing Consent):

{
  "success": false,
  "error": "Attack simulation requires user_consent=true. Only test systems you own or have explicit permission to test.",
  "error_code": "VALIDATION_ERROR",
  "timestamp": "2026-01-01T00:00:00Z"
}

Quick Scan

POST /api/quick/scan

Perform a combined scan using presets: analysis + attacks + scraping in a single request.

Presets:

Preset	Analysis	Attacks	Consent Required
low	SSL, fingerprint, headers	None	No
medium	+ CVE lookup	XSS, SQLi (medium intensity)	Yes
high	+ exploit enrichment, deeper	All types (high intensity)	Yes

Request:

{
  "url": "https://example.com",
  "preset": "low",
  "user_consent": false,
  "timeout": 30.0,
  "selector": null,
  "dynamic": false,
  "headers": {},
  "cookies": {},
  "proxy": null,
  "user_agent": null,
  "export": null,
  "output_format": "json"
}

Parameters:

Parameter	Type	Default	Description
url	string	required	Target URL
preset	string	low	Scan preset: low, medium, high
user_consent	boolean	false	Must be true for medium/high presets
timeout	float	null	Request timeout in seconds (overrides preset default)
selector	string	null	CSS selector to extract (enables scraping)
dynamic	boolean	false	Use Playwright for JS rendering
headers	object	{}	Custom HTTP headers
cookies	object	{}	Cookies
proxy	string	null	HTTP/HTTPS proxy URL
user_agent	string	null	Custom User-Agent
export	string	null	Optional output file path (enables download token)
output_format	string	json	Export format: json, jsonl, csv, html

Response:

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "timestamp": "2026-01-01T00:00:00Z",
    "ssl_analysis": { ... },
    "fingerprint": { ... },
    "headers_analysis": { ... },
    "cve_results": [ ... ],
    "attack_results": { ... },
    "scrape_result": { ... },
    "risk_score": 40,
    "summary": {
      "critical": 0,
      "high": 0,
      "medium": 4,
      "low": 0
    }
  },
  "preset": "low",
  "duration_seconds": 5.89,
  "warnings": [],
  "download_token": "uuid-token",
  "download_url": "/api/download/uuid-token",
  "timestamp": "2026-01-01T00:00:00Z"
}

Error Response (Missing Consent):

{
  "success": false,
  "error": "User consent is required for 'medium' preset. Set user_consent=true to confirm you have permission to test this system.",
  "error_code": "CONSENT_REQUIRED",
  "timestamp": "2026-01-01T00:00:00Z"
}

GET /api/quick/presets

List available scan presets and their configurations (no authentication required).

Response:

{
  "presets": {
    "low": {
      "analyze": {
        "ssl": true,
        "fingerprint": true,
        "cve": false,
        "analyze_headers": true
      },
      "has_attacks": false,
      "attack_types": [],
      "intensity": null,
      "scrape_dynamic": false
    },
    "medium": {
      "analyze": {
        "ssl": true,
        "fingerprint": true,
        "cve": true,
        "analyze_headers": true
      },
      "has_attacks": true,
      "attack_types": ["xss", "sqli"],
      "intensity": "medium",
      "scrape_dynamic": false
    },
    "high": {
      "analyze": {
        "ssl": true,
        "fingerprint": true,
        "cve": true,
        "analyze_headers": true,
        "enrich_exploits": true
      },
      "has_attacks": true,
      "attack_types": ["xss", "sqli", "traversal", "enumeration", "csrf"],
      "intensity": "high",
      "scrape_dynamic": true
    }
  }
}

Downloads

GET /api/download/{token}

Download previously exported results using a time-limited token.

URL Parameters:

Parameter	Type	Description
token	string	Download token from a POST response

Headers:

X-API-Key: your-api-key-here

Response: File stream (application/octet-stream). The Content-Disposition header carries a filename matching the export format: attachment; filename="export.json" (also .jsonl, .csv or .html).

Status Codes:

Code	Description
200	File downloaded successfully
400	Invalid token
401	Unauthorized (different user or max retries exceeded)
404	Token not found
410	Token expired
429	Too many retry attempts
503	Download service unavailable

Attempt and token semantics: every request consumes one of the token's retry attempts (download.max_retries, default 3) when the token passes validation — including requests that later fail. What happens next depends on the outcome:

• Completed download (200): the token is deleted; it cannot be downloaded again.
• Server-side failure before streaming (500): the consumed attempt is returned and the token remains usable, so the same request can be retried.
• Client disconnect mid-download: the server stops streaming, the consumed attempt is not returned, and the token is kept until it expires — later retries are allowed while attempts remain.

Two limits of this contract: the server cannot verify that the client stored every byte (a disconnect that happens after the last chunk was sent is indistinguishable from a completed download), and a disconnect is detected asynchronously, so retry-safe cleanup is guaranteed by the attempt accounting rather than by an immediate disconnect signal.

Same-key requirement: by default a token can only be redeemed with the same X-API-Key that issued it. The 401 case above (token belongs to different user) is controlled by download.require_same_user (default true) — keep it true unless you intentionally want to share tokens across API keys, because false disables that access control entirely. See CONFIGURATION.md — Download.

When no token is issued: export-enabled POST responses return 200 with download_token: null (and no download_url) while the export itself stays available on disk. This happens when downloads are disabled (download.enabled=false) or when the registry budget is exhausted (download.max_tokens / download.max_total_bytes — see CONFIGURATION.md — Download). A GET /api/download/{token} against a disabled service returns 503.

Example:

curl -H "X-API-Key: your-key" \
  "http://localhost:8000/api/download/uuid-token" \
  -o results.json

Authentication

The API uses API key authentication via the X-API-Key header.

Setting up API keys

API keys can be configured via the config file or environment variables:

Config file (~/.ciberwebscan/config.yaml):

api:
  auth:
    api_keys:
      - "my-secret-key-1"
      - "my-secret-key-2"

Environment variable:

CIBERWEBSCAN_API_AUTH_API_KEYS="my-secret-key-1,my-secret-key-2"

Optional server_secret (log obfuscation only):

api:
  auth:
    server_secret: "" # default; auto-generated once per process, never persisted

CIBERWEBSCAN_API_AUTH_SERVER_SECRET=""

api.auth.server_secret is used only to obfuscate API keys in logs (HMAC-SHA256). It does not take part in authentication or download-token authorization: identity is derived from the API key itself, and changing the secret only changes the identifiers shown in logs. Full details: CONFIGURATION.md — api.auth.server_secret.

Auth endpoints

GET /api/auth/me

Get information about the current authenticated user.

Response:

{
  "identifier": "user-id",
  "auth_method": "api_key",
  "scopes": ["full_access"],
  "authenticated": true
}

Identity model: for API-key requests, identifier is apikey: followed by the first 8 characters of the key that authenticated the request. It is stable across requests — the same key always yields the same identifier — and it does not depend on api.auth.server_secret. Download tokens are bound to this identifier, which is why redeeming a token requires the same X-API-Key that issued it (see Downloads).

POST /api/auth/generate-key

Generate a new API key (requires admin/full_access scope).

Response:

{
  "api_key": "new-key-uuid",
  "message": "Store this key securely. It cannot be retrieved again."
}

Configuration

GET /api/config

Retrieve all configuration settings.

Response:

{
  "success": true,
  "data": {
    "api": {...},
    "scraping": {...},
    "analysis": {...},
    "attacks": {...}
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

GET /api/config/sections/{section}

Retrieve a specific configuration section.

URL Parameters:

Parameter	Type	Description
section	string	Section name (api, scraping, analysis, attacks)

Response:

{
  "success": true,
  "data": {
    "key1": "value1",
    "key2": "value2"
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

PUT /api/config

Update a single configuration value.

Request:

{
  "path": "scraping.timeout",
  "value": 45.0,
  "save": false
}

Field	Type	Description
path	string	Configuration key in dot-notation
value	any	New value (string, number, boolean, list, or object)
save	boolean	Persist the change to the config file immediately (default: false)

Response:

{
  "success": true,
  "data": {
    "key": "scraping.timeout",
    "value": 45.0,
    "default": 30.0,
    "source": "runtime"
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

POST /api/config/export

Export the current configuration to a file inside the configuration base directory (~/.ciberwebscan).

Request:

{
  "path": "backup.yaml",
  "format": "yaml"
}

Field	Type	Description
path	string	Output file path (resolved inside ~/.ciberwebscan)
format	string	yaml (default) or json

Response:

{
  "success": true,
  "data": {
    "file_path": "/home/user/.ciberwebscan/backup.yaml",
    "operation": "export",
    "format": "yaml"
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

POST /api/config/load

Load configuration from a file (values are merged into the running configuration; sensitive values are masked in the response).

Request:

{
  "path": "backup.yaml"
}

Field	Type	Description
path	string	Input file path (resolved inside ~/.ciberwebscan)

A missing file returns 404; invalid paths return 400.

POST /api/config/reset

Reset to default configuration.

Request:

{
  "path": "scraping",
  "save": false
}

Field	Type	Description
path	string	Key or section to reset; omit the field to reset everything
save	boolean	Persist the reset to the config file immediately (default: false)

POST /api/config/save

Save the current configuration to a file.

Request (body optional):

{
  "path": "snapshot.yaml"
}

Field	Type	Description
path	string	Destination inside ~/.ciberwebscan; omit to use the default file

Response:

{
  "success": true,
  "data": {
    "file_path": "/home/user/.ciberwebscan/snapshot.yaml",
    "operation": "save",
    "format": null
  },
  "timestamp": "2026-01-01T00:00:00Z"
}

Config file paths (API)

Every per-call file path accepted by POST /api/config/export, POST /api/config/load, and POST /api/config/save is resolved against the configuration base directory ~/.ciberwebscan:

• ~ is expanded first, then the result must still lie inside the base (~/.ciberwebscan/x.yaml succeeds, ~/x.yaml is rejected).
• Relative paths such as backup.yaml resolve to ~/.ciberwebscan/backup.yaml, never to the server's working directory.
• Parent traversal (../), absolute paths outside the base, and symlinks that resolve outside the base are rejected with 400 (Invalid configuration path).
• Only .yaml, .yml, and .json extensions are accepted; a missing file on load returns 404.

The CLI uses different rules (relative to the current working directory, no sandbox) — see CLI.md.

Runtime vs startup configuration

PUT /api/config, POST /api/config/reset, and POST /api/config/load modify the in-process configuration object in place (when the request sets "save": true, the change is also persisted to the config file). Changes take effect immediately for components that read the configuration on each use:

• API authentication (api.auth.api_keys): every request re-reads the key list, so adding or revoking a key applies to the very next request, without restarting the server.
• Download access control (download.require_same_user): checked on every GET /api/download/{token}, so toggling it applies to the next download request.
• API auth server_secret (api.auth.server_secret): read per request but used only for log identifiers; the auto-generated default is fixed for the lifetime of the process.
• Trusted proxies (api.trusted_proxies): re-read on every request when resolving the client IP for rate limiting and auth log attribution, so proxies can be declared or removed at runtime.
• HTTP client settings (timeouts, retries, rate limits): apply to HTTP clients created after the change; operations already in flight keep the settings they started with.
• Scan defaults (scraping, analysis, attack options): apply to operations started after the change.

The following settings are read once at startup and require a server restart to take effect:

Setting	Read when
api.cors.*	App creation (create_app())
api.rate_limit.enabled, requests_per_minute	Middleware registration at app creation
download.cleanup_interval_seconds	Cleanup scheduler loop start
logging.* (handlers, level, file)	Logging setup at process start
uvicorn host / port / workers	Server launch

A restart re-reads the config file and environment variables (environment variables keep higher precedence than the file).

Error Handling

All error responses include standardized error information:

Validation Error

{
  "success": false,
  "error": "Validation error",
  "error_code": "VALIDATION_ERROR",
  "details": [
    {
      "field": "url",
      "message": "Invalid URL format",
      "value": "not-a-url"
    }
  ],
  "timestamp": "2026-01-01T00:00:00Z"
}

General Error

{
  "success": false,
  "error": "Internal server error",
  "error_code": "INTERNAL_ERROR",
  "timestamp": "2026-01-01T00:00:00Z"
}

Common HTTP Status Codes

Status	Meaning
200	Success
400	Bad request (validation error)
401	Unauthorized (missing/invalid API key)
403	Forbidden (insufficient permissions)
404	Not found
429	Too many requests (rate limited)
500	Internal server error
503	Service unavailable

Configuration

Environment Variables

Configure the API via environment variables:

# Server
CIBERWEBSCAN_API_HOST=0.0.0.0
CIBERWEBSCAN_API_PORT=8000

# Authentication
CIBERWEBSCAN_API_AUTH_API_KEYS="key1,key2,key3"
CIBERWEBSCAN_API_AUTH_SERVER_SECRET=""

# Rate Limiting
CIBERWEBSCAN_API_RATE_LIMIT_ENABLED=true
CIBERWEBSCAN_API_RATE_LIMIT_REQUESTS_PER_MINUTE=60

# Client identification (reverse proxies allowed to set X-Forwarded-For)
CIBERWEBSCAN_API_TRUSTED_PROXIES="127.0.0.1"

# CORS
CIBERWEBSCAN_API_CORS_ORIGINS="http://localhost:3000,https://example.com"
CIBERWEBSCAN_API_CORS_ALLOW_CREDENTIALS=true
CIBERWEBSCAN_API_CORS_ALLOW_METHODS="GET,POST"
CIBERWEBSCAN_API_CORS_ALLOW_HEADERS="Authorization,Content-Type,X-API-Key"
CIBERWEBSCAN_API_CORS_EXPOSE_HEADERS="X-Custom-Header"

# Downloads
CIBERWEBSCAN_DOWNLOAD_ENABLED=true
CIBERWEBSCAN_DOWNLOAD_RETENTION_SECONDS=1800
CIBERWEBSCAN_DOWNLOAD_MAX_RETRIES=3
CIBERWEBSCAN_DOWNLOAD_REQUIRE_SAME_USER=true
CIBERWEBSCAN_DOWNLOAD_MAX_TOKENS=100
CIBERWEBSCAN_DOWNLOAD_MAX_TOTAL_BYTES=1073741824

Rate Limiting

The API includes rate limiting to prevent abuse:

• Default: 60 requests per minute
• Configurable via CIBERWEBSCAN_API_RATE_LIMIT_REQUESTS_PER_MINUTE
• 429 status code returned when exceeded
• Retry-After header included in rate limit responses
• Per-client counter: each client identity has its own bucket (see Client identification below)

Client identification and trusted proxies

Clients are identified by their connection peer address. The X-Forwarded-For header is ignored by default (secure default) and is only honoured when the peer address is declared in api.trusted_proxies:

CIBERWEBSCAN_API_TRUSTED_PROXIES="127.0.0.1"          # single proxy
CIBERWEBSCAN_API_TRUSTED_PROXIES="10.0.0.0/8,::1"     # CIDR and IPv6 accepted

Rules applied (ciberwebscan.api.client_ip.resolve_client_ip, shared by the rate limiter and auth log attribution):

1. Peer missing or not parseable as an IP (e.g. unix socket) → that fallback identity is used and X-Forwarded-For is never consulted.
2. api.trusted_proxies empty (default) → the header is ignored.
3. Peer outside the trusted list → the header is ignored, even when the list is non-empty.
4. Peer is a trusted proxy → the chain is walked from right to left; the first hop not vouching for by a trusted proxy is the client. Every hop is validated with ipaddress (IPv4, IPv6 and IPv4-mapped forms are normalized); a malformed hop stops the walk and the last trusted hop wins. Entries are validated as IP addresses or CIDR networks at configuration time and stored in canonical network form.

Reverse proxy requirements for correct attribution:

• The proxy must overwrite X-Forwarded-For or append (rightmost) the address it observed. Transparent/pass-through proxies that do not touch the header must not be declared.
• Declare only the proxy's own endpoint addresses (/32 / /128), never a range that also contains end clients — a client inside a trusted range could forge a chain that the walk would believe.

Effect of not configuring it behind a reverse proxy: all clients share the proxy's peer address, so they share a single rate-limit bucket (60 rpm global) and logs show the proxy IP. Spoofing is not possible either way.

Interaction with Uvicorn's proxy handling: both launch paths of this API (ciberwebscan api and python -m ciberwebscan.api.app) start Uvicorn with proxy_headers=False, so Uvicorn never rewrites request.client from forwarded headers and FORWARDED_ALLOW_IPS is irrelevant — the resolver above is the single authority for client identity. Deployments that start the app under a different ASGI server (or a custom Uvicorn invocation) with forwarded-header trust enabled — especially FORWARDED_ALLOW_IPS=*, which installs the leftmost, client-controlled header value as the peer — void this policy and are not supported. The Forwarded header (RFC 7239) is not honoured by this API.

Invalid values: api.trusted_proxies entries must be valid IP addresses or CIDR networks. Catch-all networks (0.0.0.0/0, ::/0, and IPv4-mapped ranges covering the whole mapped IPv4 space), IPv4-mapped networks wider than /96, and IPv6 zone identifiers (fe80::1%eth0) are rejected: they would either trust every peer as a proxy or over-match silently. Any invalid value at startup is reported loudly and, like any invalid configuration value, makes the loader fall back to the default configuration (see Configuration); PUT /api/config and ciberwebscan config set validate without mutating.

CORS

Cross-Origin Resource Sharing is fully configurable via config.yaml or environment variables:

Setting	Default	Description
cors_origins	[]	Allowed origins (empty = no cross-origin)
cors_allow_credentials	false	Allow credentials (API keys, cookies)
cors_allow_methods	GET, POST, PUT, DELETE, PATCH	Allowed HTTP methods
cors_allow_headers	Authorization, Content-Type, X-API-Key	Allowed headers
cors_expose_headers	[]	Headers exposed to the browser

Example — allow only your frontend:

# config.yaml
api:
  cors_origins:
    - "https://app.example.com"
  cors_allow_credentials: true
  cors_allow_methods:
    - "GET"
    - "POST"
  cors_allow_headers:
    - "Authorization"
    - "X-API-Key"

# Or via environment variables
CIBERWEBSCAN_API_CORS_ORIGINS="https://app.example.com"
CIBERWEBSCAN_API_CORS_ALLOW_CREDENTIALS=true

Security notes:

• Default cors_origins: [] blocks all cross-origin requests (secure by default)
• ["*"] with credentials: true is rejected by browsers (CORS spec violation)
• Use specific production domains, not wildcards

API Documentation

Interactive API Docs

Once the API server is running, visit:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc
• OpenAPI JSON: http://localhost:8000/openapi.json

Example cURL Requests

Health Check:

curl http://localhost:8000/health

Analyze URL:

curl -X POST "http://localhost:8000/api/analyze" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "ssl": true,
    "fingerprint": true,
    "analyze_headers": true
  }'

Scrape URL:

curl -X POST "http://localhost:8000/api/scrape" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "selector": ".product"
  }'

Attack Test:

curl -X POST "http://localhost:8000/api/attack" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://testsite.example.com",
    "xss": true,
    "user_consent": true
  }'

Quick Scan:

curl -X POST "http://localhost:8000/api/quick/scan" \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "preset": "low"
  }'

List Quick Scan Presets:

curl "http://localhost:8000/api/quick/presets"

Best Practices

Security

• Never commit API keys to version control
• Store keys securely using environment variables or secrets manager
• Rotate keys regularly for accounts with multiple keys
• Use HTTPS only in production
• Use IP whitelisting if behind a firewall

Performance

• Use timeouts appropriately (30-60 seconds for most operations)
• Implement exponential backoff for retries
• Respect rate limits and implement queuing on your side
• Use batch endpoints for multiple URLs when available
• Cache results when appropriate

Error Handling

• Check HTTP status codes before parsing response
• Log all errors for debugging
• Implement retry logic with exponential backoff
• Handle timeouts gracefully
• Validate all responses before processing

Compliance

• Only test authorized targets - obtain written permission
• Follow responsible disclosure for vulnerabilities
• Comply with local laws regarding security testing
• Use appropriate delays between requests
• Respect robots.txt when configured

Support & Troubleshooting

Common Issues

401 Unauthorized: Verify your API key is correct and included in X-API-Key header

429 Too Many Requests: Rate limit exceeded. Implement exponential backoff

503 Service Unavailable: Download service disabled. Check configuration

Connection Refused: Ensure API server is running on the correct host/port

Getting Help

• Check API Documentation
• Review Configuration Guide
• See CLI Reference
• Report issues on GitHub