diff --git a/CLAUDE.md b/CLAUDE.md index 6af5783..a08080f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,7 @@ Self-hosted trading lab for Polymarket (prediction markets) and Binance (crypto ```bash npm run dev # server (node --watch + tsx, :3780) + Vite UI with /api proxy (http://localhost:3000) npm run dev:server # server only -npm run cli -- # migrate-key | gen-key | set-password | token create/list/revoke (add --prod for npm start paths) +npm run cli -- # migrate-key | gen-key | set-password | token create/list/revoke | check-polymarket (add --prod for npm start paths) npm run mcp:stdio:dev # MCP stdio proxy from source (BOTLAB_URL, BOTLAB_AGENT_TOKEN); mcp:stdio = built npm run gen:sdk # regenerate packages/core/src/sdk/sdkDts.generated.ts after editing sdk/types.ts ``` @@ -97,7 +97,7 @@ packages/core/src/ agent/schedule.ts # agent schedule rules (pure): Europe/Dublin slots across DST, quiet hours, event cap/debounce reconcile/ recorder/ backtest/ audit/ notify/ utils/ security/ # crypto (AES-256-GCM), masterKey (key check), adminAuth (password, sessions, tokens), migrateSecrets - ops/ # BackupJob (nightly SQLite .backup, 14 kept) + ops/ # BackupJob (nightly SQLite .backup, 14 kept); connectivityCheck (check-polymarket: geo vs bot blocking, injectable fetch) sdk/ # types.ts (ctx API types), guide.ts (agent guide, MCP resource), dts.ts + sdkDts.generated.ts # (botlab-sdk.d.ts, generated by scripts/gen-sdk-dts.ts; a test fails when stale) runtime/typecheckScript.ts # TypeScript compiler API typecheck of a script against botlab-sdk.d.ts (validate_script); diff --git a/README.md b/README.md index 4beb168..81d8fa1 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,52 @@ On macOS and Linux, `npm run service:install` prints a launchd / systemd unit wi --- +## Hosting on a VPS + +Polymarket blocks some IPs: whole countries (**geo-block**, the Terms of Service restrict them) and, separately, +many datacenter ranges (**bot protection**: Cloudflare challenge pages, 403s, 429s, dropped connections — an +Oracle Cloud VM has been blocked this way). Test a candidate server **before** migrating: + +1. Rent the smallest **hourly-billed** instance in the provider / region you want and install Node 22 and BotLab + (`npm install`; no database or key needed). +2. Run the check: + + ```bash + npm run cli -- check-polymarket # table + verdict; --json for machine-readable output + npm run cli -- check-polymarket --binance # also Binance's public API (451 = restricted location) + ``` + + It calls, read-only, every endpoint BotLab uses: Cloudflare's trace (public IP and country), Polymarket's + geoblock endpoint, the CLOB (`/time` with clock skew, `/markets/{id}`, `/book`), Gamma (`/markets`, `/events`) + and the data API (`/positions`, `/activity`), 10 s timeout each. Each row is `ok`, `geo_blocked`, + `bot_challenge`, `forbidden`, `rate_limited`, `network_error` or `unexpected`, and the last line is the verdict: + `OK: this IP can use Polymarket` (exit 0), `BLOCKED (geo): …` or `BLOCKED (datacenter/bot protection): …` + (exit 1), `DEGRADED: …` (exit 2: rate limits, partial failures, clock skew, Binance restricted). +3. **Keep the instance only if the verdict is OK**; otherwise destroy it and try another provider or region. + After migrating, re-run the check now and then (e.g. weekly, or when orders start failing): ranges get + re-classified. With a configured wallet, `npm run cli -- check-polymarket --prod --with-credentials` also makes + authenticated read-only CLOB calls (derive the existing API key — it never creates one —, list API keys, + close-only status). It opens the database read-only, so it may run next to the server; keys are never printed. + +- **Geo vs datacenter**: a geo-block is about the country — another provider in the same country will not help, + only a server in an allowed country will. A datacenter / bot block is about the IP range — another provider + or region (or a residential connection) can work. +- **Do not use a VPN or proxy to get around a geo restriction**: it breaks Polymarket's Terms of Service and can + get the account restricted to close-only or closed. (`HTTPS_PROXY` is honoured by the check, but then it + tests the proxy's IP and says so.) +- **Expose BotLab over Tailscale only**: bind it to the Tailscale address (`BOTLAB_HOST=100.x.y.z`; or keep + 127.0.0.1 behind `tailscale serve`), never `0.0.0.0`; allow no inbound port in the provider's firewall (SSH over + Tailscale too, if you can), and open the UI at `http://..ts.net:3780` (`*.ts.net` names are + allowed by default). +- **Back up off the box**: the nightly backups land in `/backups` on the same disk. Copy them elsewhere + (restic / rclone / rsync to another machine or bucket) and keep the **master key** in a separate place + (password manager): a backup without it cannot decrypt the wallet keys. +- A **home connection** (the Windows service above, a Mac mini, a Raspberry-class Linux box) is the most reliable + fallback: residential IPs are rarely filtered, and a UPS plus the halted restart after a power cut cover the + rest. + +--- + ## Configuration | variable | default | | @@ -616,7 +662,7 @@ npm start # built server serving the built UI (production paths) npm run typecheck # core + server + web npm test # vitest (SKIP_NETWORK_TESTS=1 skips live-API tests) npm run lint # ESLint (0 errors expected) -npm run cli -- # migrate-key | gen-key | set-password | token create/list/revoke +npm run cli -- # migrate-key | gen-key | set-password | token create/list/revoke | check-polymarket npm run gen:sdk # regenerate botlab-sdk.d.ts (packages/core/src/sdk/sdkDts.generated.ts) after editing sdk/types.ts npm run mcp:stdio # MCP stdio proxy (BOTLAB_URL, BOTLAB_AGENT_TOKEN); mcp:stdio:dev runs it from source ``` diff --git a/apps/server/src/checkPolymarket.ts b/apps/server/src/checkPolymarket.ts new file mode 100644 index 0000000..4c7bbc4 --- /dev/null +++ b/apps/server/src/checkPolymarket.ts @@ -0,0 +1,219 @@ +/** + * `npm run cli -- check-polymarket [--prod] [--json] [--with-credentials] [--binance] [--wallet ] + * [--address <0x…>]`: can this machine / IP use the Polymarket endpoints BotLab depends on? The checks live in + * core (`ops/connectivityCheck.ts`); this file is the CLI glue: proxy, database, master key, wallet. + * + * - Read-only throughout: GET requests only, no order placed or cancelled, no API key created. + * - The database is opened **read-only** (no migrations, no key-check write), so the check is allowed while the + * server runs on the same data dir (SQLite WAL readers do not block it). Without a database the defaults are used. + * - `--with-credentials`: the master key the server would use (`BOTLAB_MASTER_KEY`, `_FILE`, `/master.key`; + * never generated here), checked against `settings.key_check`, decrypts the wallet's private key in this process + * only; the CLOB probes sign with it and print no key, secret or passphrase. + * - `HTTPS_PROXY` / `HTTP_PROXY`: Node 22's fetch ignores them unless `NODE_USE_ENV_PROXY=1` (Node ≥ 22.21), so the + * command re-runs itself with it; the report then says the results reflect the proxy's IP. + */ + +import Database from 'better-sqlite3'; +import { spawnSync } from 'child_process'; +import fs from 'fs'; +import path from 'path'; +import { decrypt } from '../../../packages/core/src/security/crypto'; +import { keyCheckMatches } from '../../../packages/core/src/security/masterKey'; +import { createClobReadOnlyAuth } from '../../../packages/core/src/exchange/polymarket/authHeaders'; +import { DEFAULT_CLOB_URL, formatReport, runConnectivityCheck, type ConnectivityOptions } from '../../../packages/core/src/ops/connectivityCheck'; +import { defaultKeyFile, loadMasterKey } from './masterKey'; + +type Flags = Record; + +/** + * The proxy Node's fetch uses for https URLs with `NODE_USE_ENV_PROXY` (undici's EnvHttpProxyAgent): lowercase + * first, `??` semantics (a variable set to "" means "no proxy" and stops the lookup), `http_proxy` as the + * fallback. Host:port only (credentials never shown); null when there is none. + */ +export function envProxy(env: NodeJS.ProcessEnv = process.env): { raw: string; display: string } | null { + const raw = env.https_proxy ?? env.HTTPS_PROXY ?? env.http_proxy ?? env.HTTP_PROXY; + if (!raw) return null; + try { + const u = new URL(raw); + return { raw, display: `${u.hostname}${u.port ? `:${u.port}` : ''}` }; + } catch { + return { raw, display: '(unparseable proxy URL)' }; + } +} + +/** Whether `no_proxy` / `NO_PROXY` (same lookup order) exempts `host` from the proxy */ +export function bypassesProxy(host: string, env: NodeJS.ProcessEnv = process.env): boolean { + const list = (env.no_proxy ?? env.NO_PROXY ?? '') + .split(/[\s,]+/) + .map((e) => e.trim().toLowerCase()) + .filter(Boolean); + const h = host.toLowerCase(); + return list.some((e) => { + if (e === '*') return true; + const name = e.replace(/:\d+$/, '').replace(/^\*?\./, ''); + return h === name || h.endsWith(`.${name}`); + }); +} + +/** The proxy the check's requests will actually go through (env proxy honoured by fetch and not bypassed) */ +export function effectiveProxy(env: NodeJS.ProcessEnv = process.env, active: boolean = envProxyActive(env), host = 'clob.polymarket.com'): string | null { + const p = envProxy(env); + return p && active && !bypassesProxy(host, env) ? p.display : null; +} + +function nodeSupportsEnvProxy(version: string = process.versions.node): boolean { + const [major, minor] = version.split('.').map(Number); + return major > 22 || (major === 22 && minor >= 21); +} + +function envProxyActive(env: NodeJS.ProcessEnv = process.env): boolean { + return env.NODE_USE_ENV_PROXY === '1' || process.execArgv.includes('--use-env-proxy'); +} + +/** + * When a proxy is configured but fetch would ignore it, re-run this command with `NODE_USE_ENV_PROXY=1` and + * return its exit code; null to go on in this process. + */ +function reexecWithProxy(): number | null { + if (!envProxy() || envProxyActive() || !nodeSupportsEnvProxy()) return null; + // NODE_OPTIONS (not argv) also reaches a loader's own child process; the warning is Node's "experimental" notice + const nodeOptions = [process.env.NODE_OPTIONS, '--disable-warning=UNDICI-EHPA'].filter(Boolean).join(' '); + const r = spawnSync(process.execPath, [...process.execArgv, ...process.argv.slice(1)], { + stdio: 'inherit', + env: { ...process.env, NODE_USE_ENV_PROXY: '1', NODE_OPTIONS: nodeOptions }, + }); + if (r.error) throw r.error; + return r.status ?? 1; +} + +interface WalletRow { + id: string; + name: string; + address: string; + proxy_address: string | null; + private_key_encrypted: string; +} + +/** + * Opens the database read-only (no migrations, no writes; allowed next to a running server). On a WAL database + * SQLite may still create empty `-wal` / `-shm` files next to it. `error` carries the real reason (permissions, …). + */ +export function openReadOnly(dbFile: string): { db: Database.Database } | { db: null; error: string } { + if (!fs.existsSync(dbFile)) return { db: null, error: `no BotLab database at ${dbFile} (use --data-dir, or --prod for the npm start location)` }; + try { + return { db: new Database(dbFile, { readonly: true, fileMustExist: true }) }; + } catch (err) { + return { db: null, error: `cannot open ${dbFile} read-only: ${err instanceof Error ? err.message : String(err)}` }; + } +} + +function settingsClobUrl(db: Database.Database | null): string | null { + if (!db) return null; + try { + const row = db.prepare("SELECT clob_http_url FROM settings WHERE id = 'global'").get() as { clob_http_url?: string } | undefined; + return row?.clob_http_url && /^https?:\/\//.test(row.clob_http_url) ? row.clob_http_url : null; + } catch { + return null; + } +} + +function pickWallet(db: Database.Database, which: string | undefined): { wallet: WalletRow; note: string | null } | string { + const rows = db.prepare('SELECT id, name, address, proxy_address, private_key_encrypted FROM wallets ORDER BY created_at').all() as WalletRow[]; + if (rows.length === 0) return 'no wallet is configured in BotLab (add one in Wallets first)'; + if (which) { + const w = rows.find( + (r) => r.id === which || r.name === which || r.address.toLowerCase() === which.toLowerCase() || r.proxy_address?.toLowerCase() === which.toLowerCase() + ); + return w ? { wallet: w, note: null } : `no wallet matches --wallet ${which}`; + } + return { + wallet: rows[0], + note: rows.length > 1 ? `${rows.length} wallets configured; using "${rows[0].name}" (choose with --wallet )` : null, + }; +} + +/** The read-only auth probe, or why it is skipped. Never throws for a missing key / wallet. */ +export function buildAuth( + opened: ReturnType, + dataDir: string, + flags: Flags, + env: NodeJS.ProcessEnv = process.env +): { auth: ConnectivityOptions['auth']; address?: string; notes: string[] } { + const notes: string[] = []; + if (!opened.db) return { auth: { skip: opened.error }, notes }; + const db = opened.db; + const hasKey = !!env.BOTLAB_MASTER_KEY || !!env.BOTLAB_MASTER_KEY_FILE || fs.existsSync(defaultKeyFile(dataDir)); + if (!hasKey) return { auth: { skip: 'no master key (BOTLAB_MASTER_KEY, BOTLAB_MASTER_KEY_FILE or /master.key)' }, notes }; + let key: string; + try { + // production: true ⇒ never generates a key file here + key = loadMasterKey({ env, dataDir, production: true, log: () => undefined }).key; + } catch (err) { + return { auth: { skip: `master key: ${err instanceof Error ? err.message : String(err)}` }, notes }; + } + let picked: ReturnType; + try { + const row = db.prepare("SELECT key_check FROM settings WHERE id = 'global'").get() as { key_check: string | null } | undefined; + if (row?.key_check && !keyCheckMatches(row.key_check, key)) return { auth: { skip: 'the master key does not match this database' }, notes }; + picked = pickWallet(db, typeof flags.wallet === 'string' ? flags.wallet : undefined); + } catch (err) { + return { auth: { skip: `cannot read the database: ${err instanceof Error ? err.message : String(err)}` }, notes }; + } + if (typeof picked === 'string') return { auth: { skip: picked }, notes }; + if (picked.note) notes.push(picked.note); + let privateKey: string; + try { + privateKey = decrypt(picked.wallet.private_key_encrypted, key); + } catch { + return { auth: { skip: `cannot decrypt the private key of wallet "${picked.wallet.name}" with this master key` }, notes }; + } + return { + auth: createClobReadOnlyAuth(privateKey, `wallet "${picked.wallet.name}" ${picked.wallet.address}`), + address: picked.wallet.proxy_address || picked.wallet.address, + notes, + }; +} + +/** Runs the check; returns the exit code (0 ok, 1 blocked, 2 degraded). */ +export async function checkPolymarketCommand(flags: Flags, dataDir: string): Promise { + const reexec = reexecWithProxy(); + if (reexec !== null) return reexec; + + const json = flags.json === true; + const notes: string[] = []; + if (envProxy() && !envProxyActive()) { + notes.push( + `A proxy is set (https_proxy / HTTPS_PROXY) but Node ${process.versions.node} cannot use it from fetch (needs ≥ 22.21): requests go out directly from this machine` + ); + } + + const opened = openReadOnly(path.join(dataDir, 'botlab.db')); + const db = opened.db; + try { + const opts: ConnectivityOptions = { + clobUrl: settingsClobUrl(db) ?? DEFAULT_CLOB_URL, + binance: flags.binance === true, + proxy: effectiveProxy(), + }; + if (flags['with-credentials']) { + const a = buildAuth(opened, dataDir, flags); + opts.auth = a.auth; + if (a.address) opts.sampleAddress = a.address; + notes.push(...a.notes); + } + if (typeof flags.address === 'string') { + if (!/^0x[0-9a-fA-F]{40}$/.test(flags.address)) throw new Error('--address must be a 0x address'); + opts.sampleAddress = flags.address; + } + if (!json) console.log('Checking Polymarket endpoints (read-only, ~10 s timeout each)…\n'); + const report = await runConnectivityCheck(opts); + if (json) console.log(JSON.stringify({ ...report, notes }, null, 2)); + else { + console.log(formatReport(report)); + for (const n of notes) console.log(`Note: ${n}`); + } + return report.verdict.exitCode; + } finally { + db?.close(); + } +} diff --git a/apps/server/src/cli.ts b/apps/server/src/cli.ts index 9bec3a5..bfb72d8 100644 --- a/apps/server/src/cli.ts +++ b/apps/server/src/cli.ts @@ -12,6 +12,10 @@ * token create --scope agent|readonly [--name ] * token list * token revoke + * check-polymarket [--json] [--with-credentials] [--binance] [--wallet ] [--address <0x…>] + * can this machine / IP use Polymarket? Read-only checks of every endpoint + * BotLab uses (geo-block vs datacenter / bot blocking); exit 0 ok, 1 blocked, + * 2 degraded (checkPolymarket.ts) * * migrate-key refuses to run while the server is running on the same data dir (it rewrites every * secret); the other commands are safe while it runs. migrate-key uses the master key the server would @@ -28,6 +32,7 @@ import { isProduction, resolveDataDir } from './paths'; import { defaultKeyFile, generateKeyFile, loadMasterKey, MasterKeyError } from './masterKey'; import { lockHolder } from './instanceLock'; import { migrateKey, needsNewAdminPassword, needsPassword } from './migrateKey'; +import { checkPolymarketCommand } from './checkPolymarket'; import fs from 'fs'; import path from 'path'; @@ -80,7 +85,8 @@ function usage(): never { set-password [--password ] token create --scope agent|readonly [--name ] token list - token revoke `); + token revoke + check-polymarket [--json] [--with-credentials] [--binance] [--wallet ] [--address <0x…>]`); process.exit(2); } @@ -103,6 +109,11 @@ async function main(): Promise { return; } + if (cmd === 'check-polymarket') { + process.exitCode = await checkPolymarketCommand(flags, dataDir); + return; + } + const dbFile = path.join(dataDir, 'botlab.db'); if (!fs.existsSync(dbFile) && cmd !== 'set-password') { throw new Error(`No database at ${dbFile} (use --data-dir, or --prod for the npm start / service location)`); diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index b569944..67a7f8f 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -307,6 +307,10 @@ IPC handlers became `apps/server/src/routes/`). | R202 | Mac test review | **Nits** (amend R195 / R196): the Position Manager rests no exit when no price above the bid fits under 0.99 (risk / take-profit prices that would cross ⇒ marketable at once; a refusal waits the 10-min retry window, logged, no attempt per cycle); `ctx.risk.getFlaggedTokens` reads `position_exit_flags` by mode in SQL (`flaggedTokensInMode`); index `orders(placed_by, simulated, created_at)`. | Review. | — | | R203 | Mac test review | **Category search economics** (amends R198). `MarketService.searchLive`: a category-only search is cached under a key without `limit` (fetched at 100, sliced per request) and identical concurrent searches share one in-flight promise; `searchCost` weighs a category fan-out as ceil(tags / 4), max 6, against the per-principal limit of `/api/markets/search` (30 / min). `searchGamma`'s fan-out has a 15 s total deadline (tags finished by then are returned; none ⇒ "timed out") and fetches at most 2 pages per tag. MCP `search_markets` is bounded by the agent's tool-call caps instead. | Review: one fan-out per distinct limit, unbounded time, cheap for the limiter. | Constants in `MarketCacheService` / `MarketService`. | | R204 | Mac test review | **Rights basis** (amends R199, R57). `auto` on a short bucket also re-evaluates that category's medium bucket. A `trading`, unforced medium bucket whose `qualified_n` is below the full threshold (30, i.e. it qualified on the relaxed 20 while short traded) goes back to `shadow` once the short bucket no longer trades, unless its own record now meets the full threshold with a positive lower bound. | Review: a medium bucket could keep trading on a relaxation granted only because short was forced. | Remove the check in `decideRights`. | +| R205 | VPS check, amended review | **(Review: probes carrying credentials use `redirect: 'manual'` — a 3xx is `unexpected` and never followed (fetch forwards POLY_* across origins); the auth phase refuses a non-https CLOB URL. Production paths too: `redactCredentials.ts` adds an axios request interceptor setting `maxRedirects: 0` on every request with a POLY_* header (follow-redirects strips only Authorization / Cookie on a host change), and Binance signed requests use `redirect: 'error'` (`SIGNED_REDIRECT`; a redirected order is an unknown outcome for the Reconciler) — R208)** **`check-polymarket` is read-only and in core**: `packages/core/src/ops/connectivityCheck.ts` (injected `fetch`, injected auth probe; no key ever reaches it) + CLI glue `apps/server/src/checkPolymarket.ts`. GET only; no order placed / cancelled; **no CLOB API key created** — the authenticated probe uses `GET /auth/derive-api-key` (L1; a wallet that never created a key ⇒ `skipped`), then L2 `GET /auth/api-keys` and `GET /auth/ban-status/closed-only`. It signs with clob-client's `createL1Headers` / `createL2Headers` in `exchange/polymarket/authHeaders.ts` (ESLint `allowImportNames` for exactly those two; no ClobClient, so BotManager's `createApiKey`-first construction is deliberately not reused). Bodies of authenticated responses never reach the output (status only) and key / secret / passphrase are scrubbed from any detail; the module imports the axios credential redaction. CLOB market check is `GET /markets/{condition_id}` (the top Gamma market) instead of the multi-MB `/markets` / `/sampling-markets` pages; data API checks use a placeholder address (`0x…01`, returns `[]`) or the wallet's proxy / `--address`. | Vet a VPS before migrating (Oracle Cloud was blocked). | — | +| R206 | VPS check, amended review | **(Review: Cloudflare's country ban page — "Error 1009", "banned the country or region" — is `geo_blocked`; geo bodies are now matched before the challenge markers)** **Classification and verdict**: per request `ok` / `geo_blocked` (geoblock endpoint `blocked: true`, HTTP 451, "restricted region / location" bodies) / `bot_challenge` (`cf-mitigated`, Cloudflare HTML "Just a moment" / "Attention Required" / challenge markers, also on a 200 where JSON was expected) / `forbidden` (other 403) / `rate_limited` (429) / `network_error` (DNS, TLS, reset, timeout, proxy CONNECT refusal; fetch cause chains walked) / `unexpected` / `skipped`. Verdict (exit code): geo anywhere ⇒ `BLOCKED (geo)` (1); a challenge / 403 on a Polymarket API, or **every** Polymarket request failing at the connection level while the IP lookup works ⇒ `BLOCKED (datacenter/bot protection)` (1); nothing works at all ⇒ `DEGRADED` (no network, 2); 429, partial failures, unexpected bodies, CLOB clock skew beyond `MAX_CLOCK_SKEW_MS` + 1 s resolution, a close-only account, Binance failures ⇒ `DEGRADED` (2); else `OK` (0). IP lookup and geoblock endpoint failures are informational. A CLI usage error also exits 1/2 (existing CLI convention). **VERIFY**: `https://polymarket.com/api/geoblock` and its `{ blocked, ip, country, region }` shape are from documentation, not seen live (sandbox outbound blocked); any other shape ⇒ `unavailable`. | 429 alone is ambiguous (could be the user re-running), so it degrades rather than blocks. | Thresholds in `computeVerdict`. | +| R207 | VPS check, amended review | **(Review: a read-only open of a WAL database can still leave empty `-wal` / `-shm` files next to it; an open failure reports SQLite's own message (permissions, …) instead of "no database". Proxy detection follows Node: `https_proxy ?? HTTPS_PROXY ?? http_proxy ?? HTTP_PROXY` — an empty value disables it —, and "via proxy" is reported only when fetch will use it: `NODE_USE_ENV_PROXY` active and `no_proxy ?? NO_PROXY` not exempting the CLOB host)** **Database and proxy**: the check opens `botlab.db` **read-only** with better-sqlite3 (no migrations, no `key_check` write; the key is compared with `keyCheckMatches`) and is **allowed while the server runs** on the same data dir; the master key is loaded like the server's but never generated (`production: true`). `HTTPS_PROXY` / `HTTP_PROXY`: Node 22's fetch ignores them unless `NODE_USE_ENV_PROXY=1` (Node ≥ 22.21), so the command re-runs itself with it (and `--disable-warning=UNDICI-EHPA` via `NODE_OPTIONS`); the report says the results reflect the proxy's IP. On Node < 22.21 a note says the proxy was not used. Binance is opt-in (`--binance`). No `/api/system/connectivity` route or Settings button (kept the change CLI-only; the check is for a machine that may not run BotLab yet). | Users test a fresh VPS before installing a database; a running server must not have to stop. | Add the route later (admin only) calling `runConnectivityCheck`. | +| R208 | VPS check review | **Credential headers never follow redirects** (see R205): connectivity probes with headers (manual, 3xx ⇒ `unexpected`, no request to the target), clob-client via axios (`refuseAuthRedirects`: `maxRedirects: 0` when a POLY_* header is present; unauthenticated CLOB reads still follow redirects), Binance signed GET / POST / order query (`redirect: 'error'`). | Review 1: a redirect to another host would have carried `POLY_API_KEY` / `POLY_PASSPHRASE` / `POLY_SIGNATURE` (or `X-MBX-APIKEY`) there. The CLOB and Binance APIs do not redirect today, so nothing legitimate breaks. | Remove the request interceptor / `SIGNED_REDIRECT`. | ## Log @@ -1303,3 +1307,34 @@ Independent security review of b434e6b..0e1225e. Fixed (R149, R150, R152, R154 a fan-out for different limits; shared in-flight search; search weight; fan-out deadline with partial results; executor with `resting_order_ttl_minutes` 30 and `bid_ttl_minutes` 60; Position Manager at bid 0.99; medium bucket re-evaluated when short is un-forced. + +### 2026-10-31 — VPS connectivity check (`check-polymarket`) + +- `npm run cli -- check-polymarket [--prod] [--json] [--with-credentials] [--binance] [--wallet …] [--address …]`: + public IP / country (Cloudflare trace), Polymarket geoblock endpoint (VERIFY), CLOB `/time` (clock skew), + `/markets/{condition_id}`, `/book`, Gamma `/markets` / `/events`, data API `/positions` / `/activity`; optional + Binance `/api/v3/ping` / `/time`; with credentials, read-only L1/L2 CLOB calls. Table or JSON, one-line verdict, + exit 0 / 1 / 2 (R205–R207). +- README "Hosting on a VPS" (test before migrating, geo vs datacenter, ToS warning, Tailscale-only, off-box + backups, home connection as fallback). +- Tests (`tests/core/connectivity-check.test.ts`, fake fetch): every classification, each verdict and exit code, + token id flow, GET-only, timeout, proxy note, geoblock missing / other shape, clock skew, Binance 451, auth probe + (no secrets in output, no error body echo, closed-only, nothing to derive, skip reasons), cause-chain errors. +- Live run from the development sandbox: every request was refused by the sandbox's egress proxy (`CONNECT` 403) + ⇒ `DEGRADED: no request succeeded, not even the IP lookup (the proxy refused the connection …)`, exit 2. Real + response shapes (notably the geoblock endpoint) are still to be seen: run it once on a real machine. + +### 2026-10-31 — `check-polymarket`: review fixes + +- Credentials never follow a redirect (R208, amends R205): manual redirects for authenticated probes, https-only + CLOB URL for the auth phase; axios request interceptor (`maxRedirects: 0` with POLY_* headers) for the trading + path; Binance signed requests `redirect: 'error'`. +- Cloudflare 1009 country ban ⇒ `geo_blocked` (amends R206). +- Read-only open: real error message, WAL side files noted; proxy lookup in Node's order, "via proxy" only when + fetch uses it (amends R207). +- Tests: redirect on an authenticated probe (not followed, `manual`, nothing sent elsewhere), http CLOB refused, + 1009 page; a real cross-host 302 on an L2 clob-client request reaches nothing on the other host (fails without + the interceptor) while public reads still follow; Binance signed requests carry `redirect: 'error'`; + `checkPolymarket.ts` (read-only open rejects writes, open errors, key mismatch, no key ⇒ none generated, unknown + wallet, proxy order / `no_proxy`); `authHeaders.ts` (header names only, no private key or secret). + diff --git a/eslint.config.mjs b/eslint.config.mjs index 3008173..dff1504 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -121,6 +121,24 @@ export default [ }), }, }, + { + // Connectivity check: clob-client's L1/L2 header builders only (read-only auth, no ClobClient) + files: ['packages/core/src/exchange/polymarket/authHeaders.ts'], + rules: { + '@typescript-eslint/no-restricted-imports': restrictedImports({ + paths: [ + { + name: '@polymarket/clob-client', + allowImportNames: ['createL1Headers', 'createL2Headers'], + message: 'Only the L1/L2 header builders here (read-only connectivity check).', + }, + clobValueImports[1], + ...coreTransportImports, + ], + patterns: [...orderModuleImports, ...coreTransportPatterns], + }), + }, + }, { files: ['packages/core/src/orders/ExchangeAdapter.ts'], rules: { '@typescript-eslint/no-restricted-imports': restrictedImports({ paths: coreTransportImports, patterns: coreTransportPatterns }) }, diff --git a/packages/core/src/exchange/binance/client.ts b/packages/core/src/exchange/binance/client.ts index cabf2af..6a2a500 100644 --- a/packages/core/src/exchange/binance/client.ts +++ b/packages/core/src/exchange/binance/client.ts @@ -112,6 +112,13 @@ function buildSignedQuery( // HTTP helpers // ============================================================================ +/** + * Signed requests never follow a redirect: fetch would forward X-MBX-APIKEY (a custom header) to another + * host. A redirect makes the request fail (a thrown TypeError: for an order, an unknown outcome the + * Reconciler settles). + */ +export const SIGNED_REDIRECT: NonNullable = 'error'; + async function publicGet(path: string, params?: Record): Promise { let url = `${BASE_URL}${path}`; if (params) { @@ -137,6 +144,7 @@ async function signedGet( const url = `${BASE_URL}${path}?${query}`; const resp = await fetch(url, { headers: { 'X-MBX-APIKEY': creds.apiKey }, + redirect: SIGNED_REDIRECT, }); if (!resp.ok) { const body = await resp.text(); @@ -157,6 +165,7 @@ async function signedPost( resp = await fetch(url, { method: 'POST', headers: { 'X-MBX-APIKEY': creds.apiKey }, + redirect: SIGNED_REDIRECT, }); } catch (err) { throw new BinanceHttpError(err instanceof Error ? err.message : String(err)); @@ -329,7 +338,7 @@ export type BinanceOrderQuery = */ export async function queryBinanceOrder(creds: BinanceCredentials, symbol: string, clientOrderId: string): Promise { const query = buildSignedQuery({ symbol: symbol.toUpperCase(), origClientOrderId: clientOrderId }, creds.apiSecret); - const resp = await fetch(`${BASE_URL}/api/v3/order?${query}`, { headers: { 'X-MBX-APIKEY': creds.apiKey } }); + const resp = await fetch(`${BASE_URL}/api/v3/order?${query}`, { headers: { 'X-MBX-APIKEY': creds.apiKey }, redirect: SIGNED_REDIRECT }); const text = await resp.text(); if (!resp.ok) { let code: number | undefined; diff --git a/packages/core/src/exchange/polymarket/authHeaders.ts b/packages/core/src/exchange/polymarket/authHeaders.ts new file mode 100644 index 0000000..fb95d97 --- /dev/null +++ b/packages/core/src/exchange/polymarket/authHeaders.ts @@ -0,0 +1,27 @@ +/** + * Read-only CLOB authentication for the connectivity check (`check-polymarket --with-credentials`): signs the + * L1 header for `GET /auth/derive-api-key` and L2 headers for GET requests with clob-client's own header + * builders. Only the two header functions are imported (ESLint allows nothing else from clob-client here): + * this module cannot build a ClobClient, post, cancel or create anything. + * + * The private key stays in the signer; the headers carry signatures and the API key, and the caller only sends + * them. Importing the credential redaction keeps clob-client's axios errors (if anything here ever goes + * through it) free of auth headers. + */ + +import './redactCredentials'; +import { ethers } from 'ethers'; +import { createL1Headers, createL2Headers } from '@polymarket/clob-client'; +import type { ClobApiCreds, ClobAuthProbe } from '../../ops/connectivityCheck'; + +const POLYGON = 137; + +export function createClobReadOnlyAuth(privateKey: string, label: string): ClobAuthProbe { + const signer = new ethers.Wallet(privateKey); + const asRecord = (h: object): Record => Object.fromEntries(Object.entries(h).map(([k, v]) => [k, String(v)])); + return { + label, + l1Headers: async () => asRecord(await createL1Headers(signer, POLYGON)), + l2Headers: async (creds: ClobApiCreds, method: 'GET', requestPath: string) => asRecord(await createL2Headers(signer, creds, { method, requestPath })), + }; +} diff --git a/packages/core/src/exchange/polymarket/redactCredentials.ts b/packages/core/src/exchange/polymarket/redactCredentials.ts index 416b054..78f5936 100644 --- a/packages/core/src/exchange/polymarket/redactCredentials.ts +++ b/packages/core/src/exchange/polymarket/redactCredentials.ts @@ -9,6 +9,11 @@ * that handler. The error keeps its response (status, data), so clob-client still returns * `{ error, status }`. * + * Redirects: axios follows them through follow-redirects, which drops only `Authorization` / `Cookie` (and + * `Proxy-Authorization`) on a host change — the POLY_* headers would go to the new host. A request interceptor + * sets `maxRedirects: 0` on every request carrying a POLY_* header, so a 3xx comes back as an HTTP error + * (clob-client: `{ error, status }`) and nothing is sent to the redirect target. + * * `installCredentialRedaction()` is idempotent and runs at import of this module's users * (orders.ts, BotManager, createCore), i.e. before any ClobClient exists. */ @@ -23,7 +28,10 @@ const INSTALLED = Symbol.for('botlab.credentialRedaction'); type AnyRecord = Record; interface AxiosLike { - interceptors: { response: { use(onOk: undefined, onErr: (err: unknown) => unknown): number } }; + interceptors: { + response: { use(onOk: undefined, onErr: (err: unknown) => unknown): number }; + request?: { use(onOk: (config: AnyRecord) => AnyRecord): number }; + }; [INSTALLED]?: boolean; } @@ -47,6 +55,17 @@ export function redactAxiosConfig(config: unknown): void { if ('auth' in c && c.auth !== undefined) c.auth = REDACTED; } +const POLY_HEADER = /^poly[_-]/i; + +/** Requests carrying POLY_* auth headers never follow a redirect (`maxRedirects: 0`). Returns the config. */ +export function refuseAuthRedirects(config: T): T { + if (config && typeof config === 'object') { + const headers = plainHeaders((config as AnyRecord).headers); + if (headers && Object.keys(headers).some((k) => POLY_HEADER.test(k))) (config as AnyRecord).maxRedirects = 0; + } + return config; +} + /** Redact the request side of an axios error in place; the response (status, body) is kept. */ export function redactAxiosError(err: unknown): unknown { if (err && typeof err === 'object') { @@ -67,6 +86,7 @@ function install(axios: unknown): void { if (!a || (typeof a !== 'object' && typeof a !== 'function')) return; if (!a.interceptors?.response || a[INSTALLED]) return; a.interceptors.response.use(undefined, (err: unknown) => Promise.reject(redactAxiosError(err))); + a.interceptors.request?.use((config) => refuseAuthRedirects(config)); a[INSTALLED] = true; } diff --git a/packages/core/src/ops/connectivityCheck.ts b/packages/core/src/ops/connectivityCheck.ts new file mode 100644 index 0000000..3462bd5 --- /dev/null +++ b/packages/core/src/ops/connectivityCheck.ts @@ -0,0 +1,795 @@ +/** + * Polymarket (and optionally Binance) connectivity check: can this machine / IP use every endpoint BotLab + * depends on? Used by `npm run cli -- check-polymarket` to vet a VPS before migrating (see README "Hosting on a + * VPS"). Every request is a read-only GET: no order is placed or cancelled, no API key is created. + * + * Two failure kinds are kept apart: + * - **geo-block** (the country / region is restricted): Polymarket's geoblock endpoint says `blocked`, an HTTP + * 451, or a known "restricted region / location" error body; + * - **datacenter / bot protection**: Cloudflare challenge pages (`cf-mitigated`, "Just a moment…"), a 403 with + * an HTML body or other 403, 429 and dropped connections. + * + * The module is transport-free and network-agnostic: `fetch` is injected (tests use a fake), the CLI passes the + * global one. The authenticated CLOB probe is injected too (`auth`): this module never sees a private key, and + * the L1/L2 headers it receives are only sent, never printed. + */ + +import { MAX_CLOCK_SKEW_MS } from '../reconcile/Reconciler'; + +export type CheckClass = + | 'ok' + | 'geo_blocked' + | 'bot_challenge' + | 'forbidden' + | 'rate_limited' + | 'network_error' + | 'unexpected' + /** not run (no token id from Gamma, no wallet, no master key, …) */ + | 'skipped'; + +export type CheckGroup = 'ip' | 'geoblock' | 'clob' | 'gamma' | 'data' | 'clob_auth' | 'binance'; + +export interface CheckResult { + id: string; + group: CheckGroup; + name: string; + /** URL without the query string's addresses / tokens beyond what is public */ + url: string | null; + status: number | null; + latencyMs: number | null; + classification: CheckClass; + detail: string | null; + /** Counts for the Polymarket verdict (ip / geoblock / binance are reported separately) */ + required: boolean; +} + +export type VerdictKind = 'ok' | 'blocked_geo' | 'blocked_bot' | 'degraded'; + +export interface ConnectivityVerdict { + kind: VerdictKind; + /** One line, e.g. "OK: this IP can use Polymarket" */ + line: string; + hints: string[]; + /** 0 ok, 1 blocked, 2 degraded */ + exitCode: 0 | 1 | 2; +} + +export interface GeoblockInfo { + /** `unavailable`: the endpoint failed or its shape was not the expected one (VERIFY) */ + status: 'allowed' | 'blocked' | 'unavailable'; + country: string | null; + region: string | null; + ip: string | null; + detail: string | null; +} + +export interface ConnectivityReport { + startedAt: string; + durationMs: number; + /** From the IP lookup; null when unknown */ + ip: string | null; + country: string | null; + geoblock: GeoblockInfo; + /** CLOB server time − local time (ms, round-trip midpoint); null when unknown */ + clobClockSkewMs: number | null; + binanceClockSkewMs: number | null; + /** Host:port of the HTTP(S) proxy the requests went through, or null */ + proxy: string | null; + checks: CheckResult[]; + verdict: ConnectivityVerdict; +} + +/** + * Read-only authenticated CLOB probe, built by the caller from the configured wallet. `l1Headers` signs the + * L1 (EIP-712) auth for `GET /auth/derive-api-key` (returns the wallet's existing API key, creates nothing); + * `l2Headers` signs an L2 request with those credentials. Secrets never leave the returned headers. + */ +export interface ClobAuthProbe { + /** Shown in the output (an address is public) */ + label: string; + l1Headers(): Promise>; + l2Headers(creds: ClobApiCreds, method: 'GET', requestPath: string): Promise>; +} + +export interface ClobApiCreds { + key: string; + secret: string; + passphrase: string; +} + +export interface ConnectivityOptions { + fetch?: typeof fetch; + /** Per request (default 10 s) */ + timeoutMs?: number; + /** Parallel requests (default 3, capped at 3) */ + concurrency?: number; + now?: () => number; + clobUrl?: string; + gammaUrl?: string; + dataUrl?: string; + binanceUrl?: string; + ipLookupUrl?: string; + geoblockUrl?: string; + /** Any public address for the data API checks (an unknown address simply returns `[]`) */ + sampleAddress?: string; + /** Run the Binance public checks */ + binance?: boolean; + /** Authenticated CLOB probe, or the reason it is skipped (`--with-credentials` without a wallet, …) */ + auth?: ClobAuthProbe | { skip: string }; + /** Proxy the requests go through (shown in the report; the caller configures the transport) */ + proxy?: string | null; +} + +export const DEFAULT_CLOB_URL = 'https://clob.polymarket.com'; +export const DEFAULT_GAMMA_URL = 'https://gamma-api.polymarket.com'; +export const DEFAULT_DATA_URL = 'https://data-api.polymarket.com'; +export const DEFAULT_BINANCE_URL = 'https://api.binance.com'; +/** Cloudflare's trace endpoint: `ip=` and `loc=` (country), no key */ +export const DEFAULT_IP_LOOKUP_URL = 'https://www.cloudflare.com/cdn-cgi/trace'; +/** + * VERIFY: Polymarket's public geoblock endpoint, documented as returning + * `{ "blocked": boolean, "ip": string, "country": string, "region": string }`. Not checked against the live API + * from the development sandbox (outbound blocked); any other shape is reported as `unavailable`. + */ +export const DEFAULT_GEOBLOCK_URL = 'https://polymarket.com/api/geoblock'; +/** Placeholder public address for the data API checks: an address nobody uses returns an empty list */ +export const DEFAULT_SAMPLE_ADDRESS = '0x0000000000000000000000000000000000000001'; +export const DEFAULT_TIMEOUT_MS = 10_000; + +/** + * "restricted region" style bodies (CLOB order refusal, Polymarket docs, Binance 451 / 403) and Cloudflare's + * country ban page ("Error 1009 … has banned the country or region your IP address is in") + */ +const GEO_BODY = + /restricted (in your |)(region|location|jurisdiction|country)|trading restricted|not available in your (region|country|jurisdiction)|geo-?block|unavailable from a restricted location|eligibility|banned the country or region|error 1009/i; +const CHALLENGE_BODY = + /just a moment|attention required|cf-chl|challenge-platform|challenges\.cloudflare\.com|captcha|checking your browser|cf-browser-verification|enable javascript and cookies/i; +const HTML_BODY = /^\s*(; + body: string; + /** JSON expected on success (all API endpoints) */ + expectJson: boolean; +} + +function header(h: Headers | Record, name: string): string | null { + if (typeof (h as Headers).get === 'function') return (h as Headers).get(name); + const rec = h as Record; + const key = Object.keys(rec).find((k) => k.toLowerCase() === name.toLowerCase()); + return key ? rec[key] : null; +} + +function isCloudflare(h: Headers | Record): boolean { + return !!header(h, 'cf-ray') || /cloudflare/i.test(header(h, 'server') ?? ''); +} + +function looksHtml(body: string, h: Headers | Record): boolean { + return HTML_BODY.test(body) || /text\/html/i.test(header(h, 'content-type') ?? ''); +} + +function parseJson(body: string): { ok: true; value: unknown } | { ok: false } { + try { + return { ok: true, value: JSON.parse(body) }; + } catch { + return { ok: false }; + } +} + +/** Classifies one HTTP response (the shape of a successful body is checked by the caller). */ +export function classifyResponse(obs: HttpObservation): { classification: CheckClass; detail: string | null } { + const { status, headers, body } = obs; + const html = looksHtml(body, headers); + const mitigated = header(headers, 'cf-mitigated'); + if (status === 451) return { classification: 'geo_blocked', detail: `HTTP 451 (unavailable for legal reasons)${snippet(body)}` }; + // Before the challenge markers: a Cloudflare country ban (1009) is an HTML page from Cloudflare too + if (status >= 400 && GEO_BODY.test(body)) return { classification: 'geo_blocked', detail: `HTTP ${status}${snippet(body)}` }; + if (mitigated) return { classification: 'bot_challenge', detail: `Cloudflare cf-mitigated: ${mitigated} (HTTP ${status})` }; + if (html && CHALLENGE_BODY.test(body) && (isCloudflare(headers) || status === 403 || status === 503 || status === 429)) { + return { classification: 'bot_challenge', detail: `Cloudflare challenge page (HTTP ${status})` }; + } + if (status === 429) + return { classification: 'rate_limited', detail: `HTTP 429${header(headers, 'retry-after') ? ` (retry-after ${header(headers, 'retry-after')})` : ''}` }; + if (status === 403) { + return { + classification: 'forbidden', + detail: `HTTP 403${html ? ' with an HTML page' : ''}${isCloudflare(headers) ? ' from Cloudflare' : ''}${snippet(body)}`, + }; + } + if (status < 200 || status >= 300) return { classification: 'unexpected', detail: `HTTP ${status}${snippet(body)}` }; + if (obs.expectJson && !parseJson(body).ok) { + return { classification: 'unexpected', detail: `HTTP ${status} but not JSON${html ? ' (an HTML page)' : ''}${snippet(body)}` }; + } + return { classification: 'ok', detail: null }; +} + +/** Classifies a thrown fetch error (DNS, TLS, reset, timeout, proxy refusal). */ +export function classifyError(err: unknown, timeoutMs: number): { classification: CheckClass; detail: string } { + const e = err as { name?: string } | null; + if (e?.name === 'TimeoutError' || e?.name === 'AbortError') + return { classification: 'network_error', detail: `timeout after ${Math.round(timeoutMs / 1000)} s` }; + // fetch wraps the real error in `cause` (sometimes twice: TypeError → DOMException → undici error) + const chain: Array<{ code?: string; message?: string }> = []; + for (let c: unknown = err; c && typeof c === 'object' && chain.length < 5; c = (c as { cause?: unknown }).cause) + chain.push(c as { code?: string; message?: string }); + const code = [...chain].reverse().find((c) => typeof c.code === 'string')?.code; + const msg = + chain + .map((c) => c.message) + .filter(Boolean) + .join(' / ') || String(err); + const proxyRefused = /proxy response \((\d+)\)/i.exec(msg); + if (proxyRefused) return { classification: 'network_error', detail: `the proxy refused the connection (HTTP ${proxyRefused[1]} to CONNECT)` }; + const known: Record = { + ENOTFOUND: 'DNS lookup failed', + EAI_AGAIN: 'DNS lookup failed (temporary)', + ECONNRESET: 'connection reset', + ECONNREFUSED: 'connection refused', + ETIMEDOUT: 'connection timed out', + EHOSTUNREACH: 'host unreachable', + ENETUNREACH: 'network unreachable', + UND_ERR_SOCKET: 'connection closed by the remote side', + UND_ERR_CONNECT_TIMEOUT: 'connect timeout', + UND_ERR_HEADERS_TIMEOUT: 'response headers timeout', + }; + if (code && known[code]) return { classification: 'network_error', detail: `${known[code]} (${code})` }; + if (code && /CERT|SSL|TLS/i.test(code)) return { classification: 'network_error', detail: `TLS error (${code})` }; + return { classification: 'network_error', detail: `${code ? `${code}: ` : ''}${oneLine(msg, 160)}` }; +} + +function oneLine(text: string, max: number): string { + const t = text.replace(/\s+/g, ' ').trim(); + return t.length > max ? `${t.slice(0, max)}…` : t; +} + +function snippet(body: string): string { + if (!body) return ''; + if (HTML_BODY.test(body)) { + const title = /([^<]*)<\/title>/i.exec(body)?.[1]; + return title ? `: "${oneLine(title, 80)}"` : ''; + } + return `: ${oneLine(body, 120)}`; +} + +/** Removes every occurrence of the given secrets from a text (defence in depth for authenticated checks). */ +export function redactSecrets(text: string | null, secrets: Array<string | undefined>): string | null { + if (text == null) return text; + let out = text; + for (const s of secrets) if (s && s.length >= 6) out = out.split(s).join('[redacted]'); + return out; +} + +// ── runner ─────────────────────────────────────────────────────────── + +interface Ctx { + fetch: typeof fetch; + timeoutMs: number; + now: () => number; +} + +interface Probe<T> { + result: CheckResult; + /** Parsed JSON (or text for `text` probes) when ok */ + value?: T; + /** Local times around the request (clock skew) */ + t0?: number; + t1?: number; +} + +interface ProbeSpec { + id: string; + group: CheckGroup; + name: string; + url: string; + required: boolean; + headers?: Record<string, string>; + /** `text` for the IP lookup; JSON otherwise */ + text?: boolean; + /** Validates the parsed body; returns an error detail for a wrong shape */ + shape?: (value: unknown) => string | null; + /** Never put any of the body into the detail (authenticated responses) */ + secretBody?: boolean; + secrets?: string[]; + /** Shown instead of the URL (query strings with addresses, …) */ + displayUrl?: string; +} + +async function probe<T>(ctx: Ctx, spec: ProbeSpec): Promise<Probe<T>> { + const base: CheckResult = { + id: spec.id, + group: spec.group, + name: spec.name, + url: spec.displayUrl ?? spec.url, + status: null, + latencyMs: null, + classification: 'network_error', + detail: null, + required: spec.required, + }; + const t0 = ctx.now(); + try { + const resp = await ctx.fetch(spec.url, { + method: 'GET', + headers: { Accept: spec.text ? 'text/plain, */*' : 'application/json', 'User-Agent': 'BotLab-connectivity-check', ...(spec.headers ?? {}) }, + // fetch forwards custom headers (POLY_*) across origins on a redirect: never follow one with credentials + redirect: spec.headers ? 'manual' : 'follow', + signal: AbortSignal.timeout(ctx.timeoutMs), + }); + if (spec.headers && resp.status >= 300 && resp.status < 400) { + await resp.body?.cancel().catch(() => undefined); + let to = 'elsewhere'; + try { + to = new URL(resp.headers.get('location') ?? '', spec.url).host || to; + } catch { + /* unparseable Location */ + } + return { + result: { + ...base, + status: resp.status, + latencyMs: ctx.now() - t0, + classification: 'unexpected', + detail: `HTTP ${resp.status} redirect to ${to} not followed (credentials are never sent to a redirect)`, + }, + }; + } + const body = await resp.text(); + const t1 = ctx.now(); + const c = classifyResponse({ status: resp.status, headers: resp.headers, body, expectJson: !spec.text }); + const result: CheckResult = { ...base, status: resp.status, latencyMs: t1 - t0, classification: c.classification, detail: c.detail }; + let value: T | undefined; + if (c.classification === 'ok') { + if (spec.text) value = body as unknown as T; + else { + const parsed = parseJson(body); + value = parsed.ok ? (parsed.value as T) : undefined; + } + const wrong = spec.shape ? spec.shape(value) : null; + if (wrong) { + result.classification = 'unexpected'; + result.detail = `unexpected response: ${wrong}`; + value = undefined; + } + } + // Authenticated responses: no body text in the output, only the status + if (spec.secretBody && result.classification !== 'ok' && result.detail) result.detail = `HTTP ${resp.status}`; + result.detail = redactSecrets(result.detail, spec.secrets ?? []); + return { result, value, t0, t1 }; + } catch (err) { + const c = classifyError(err, ctx.timeoutMs); + return { result: { ...base, latencyMs: ctx.now() - t0, classification: c.classification, detail: redactSecrets(c.detail, spec.secrets ?? []) } }; + } +} + +async function pool<T>(tasks: Array<() => Promise<T>>, concurrency: number): Promise<T[]> { + const out: T[] = new Array(tasks.length); + let next = 0; + const worker = async () => { + while (next < tasks.length) { + const i = next++; + out[i] = await tasks[i](); + } + }; + await Promise.all(Array.from({ length: Math.min(concurrency, tasks.length) }, worker)); + return out; +} + +function skipped(id: string, group: CheckGroup, name: string, detail: string, required: boolean): CheckResult { + return { id, group, name, url: null, status: null, latencyMs: null, classification: 'skipped', detail, required }; +} + +const isArray = (v: unknown) => (Array.isArray(v) ? null : 'expected a JSON array'); +const isObject = (v: unknown) => (v && typeof v === 'object' && !Array.isArray(v) ? null : 'expected a JSON object'); + +function parseTrace(text: string): { ip: string | null; country: string | null } { + const get = (k: string) => new RegExp(`^${k}=(.+)$`, 'm').exec(text)?.[1]?.trim() ?? null; + return { ip: get('ip'), country: get('loc') }; +} + +function parseList(v: unknown): string[] { + if (Array.isArray(v)) return v.map(String); + if (typeof v === 'string') { + try { + const p = JSON.parse(v); + return Array.isArray(p) ? p.map(String) : []; + } catch { + return []; + } + } + return []; +} + +/** Skew = server − local midpoint, minus half the round trip (given to the local clock) and the resolution */ +function skew(serverMs: number, t0: number, t1: number): number { + return Math.round(serverMs - (t0 + t1) / 2); +} + +const trim = (u: string) => u.replace(/\/+$/, ''); + +export async function runConnectivityCheck(opts: ConnectivityOptions = {}): Promise<ConnectivityReport> { + const ctx: Ctx = { + fetch: opts.fetch ?? globalThis.fetch, + timeoutMs: opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, + now: opts.now ?? Date.now, + }; + const concurrency = Math.max(1, Math.min(3, opts.concurrency ?? 3)); + const clob = trim(opts.clobUrl ?? DEFAULT_CLOB_URL); + const gamma = trim(opts.gammaUrl ?? DEFAULT_GAMMA_URL); + const data = trim(opts.dataUrl ?? DEFAULT_DATA_URL); + const binance = trim(opts.binanceUrl ?? DEFAULT_BINANCE_URL); + const address = opts.sampleAddress ?? DEFAULT_SAMPLE_ADDRESS; + const started = ctx.now(); + const checks: CheckResult[] = []; + + // Phase 1: who are we, are we geoblocked, CLOB time, Gamma (for a token id) + const [ipP, geoP, timeP, gmP, geP] = await pool<Probe<unknown>>( + [ + () => + probe(ctx, { + id: 'ip', + group: 'ip', + name: 'Public IP / country (Cloudflare trace)', + url: opts.ipLookupUrl ?? DEFAULT_IP_LOOKUP_URL, + required: false, + text: true, + }), + () => + probe(ctx, { + id: 'geoblock', + group: 'geoblock', + name: 'Polymarket geoblock endpoint', + url: opts.geoblockUrl ?? DEFAULT_GEOBLOCK_URL, + required: false, + shape: isObject, + }), + () => + probe(ctx, { + id: 'clob_time', + group: 'clob', + name: 'CLOB GET /time', + url: `${clob}/time`, + required: true, + shape: (v) => (Number.isFinite(Number(v)) ? null : 'expected a number of seconds'), + }), + () => + probe(ctx, { + id: 'gamma_markets', + group: 'gamma', + name: 'Gamma GET /markets', + url: `${gamma}/markets?limit=1&active=true&closed=false&order=volume24hr&ascending=false`, + required: true, + shape: isArray, + }), + () => probe(ctx, { id: 'gamma_events', group: 'gamma', name: 'Gamma GET /events', url: `${gamma}/events?limit=1`, required: true, shape: isArray }), + ], + concurrency + ); + checks.push(ipP.result, geoP.result, timeP.result, gmP.result, geP.result); + + const trace = typeof ipP.value === 'string' ? parseTrace(ipP.value) : { ip: null, country: null }; + if (ipP.result.classification === 'ok' && !trace.ip) { + ipP.result.classification = 'unexpected'; + ipP.result.detail = 'no ip= line in the trace'; + } else if (ipP.result.classification === 'ok') ipP.result.detail = `${trace.ip}${trace.country ? ` (${trace.country})` : ''}`; + + const geoblock = readGeoblock(geoP); + const clobClockSkewMs = + timeP.value !== undefined && timeP.t0 !== undefined && timeP.t1 !== undefined ? skew(Number(timeP.value) * 1000, timeP.t0, timeP.t1) : null; + if (clobClockSkewMs !== null) timeP.result.detail = `clock skew ${formatSkew(clobClockSkewMs)} (1 s resolution)`; + + const market = Array.isArray(gmP.value) ? (gmP.value[0] as Record<string, unknown> | undefined) : undefined; + const tokenId = market ? parseList(market.clobTokenIds)[0] : undefined; + const conditionId = market && typeof market.conditionId === 'string' ? market.conditionId : undefined; + if (market && typeof market.question === 'string') gmP.result.detail = `top market: ${oneLine(market.question, 60)}`; + + // Phase 2: order book and market on the CLOB, data API + const phase2: Array<() => Promise<Probe<unknown>>> = [ + () => + probe(ctx, { + id: 'data_positions', + group: 'data', + name: 'Data API GET /positions', + url: `${data}/positions?user=${address}&limit=1`, + required: true, + shape: isArray, + }), + () => + probe(ctx, { + id: 'data_activity', + group: 'data', + name: 'Data API GET /activity', + url: `${data}/activity?user=${address}&limit=1&type=TRADE`, + required: true, + shape: isArray, + }), + ]; + if (tokenId) { + phase2.unshift(() => + probe(ctx, { + id: 'clob_book', + group: 'clob', + name: 'CLOB GET /book', + url: `${clob}/book?token_id=${encodeURIComponent(tokenId)}`, + displayUrl: `${clob}/book?token_id=${tokenId.slice(0, 10)}…`, + required: true, + shape: (v) => + v && typeof v === 'object' && Array.isArray((v as { bids?: unknown }).bids) && Array.isArray((v as { asks?: unknown }).asks) + ? null + : 'expected { bids: [], asks: [] }', + }) + ); + } + if (conditionId) { + phase2.unshift(() => + probe(ctx, { + id: 'clob_market', + group: 'clob', + name: 'CLOB GET /markets/{condition_id}', + url: `${clob}/markets/${encodeURIComponent(conditionId)}`, + displayUrl: `${clob}/markets/${conditionId.slice(0, 10)}…`, + required: true, + shape: isObject, + }) + ); + } + const phase2Results = await pool(phase2, concurrency); + const byId = new Map(phase2Results.map((p) => [p.result.id, p.result])); + const noMarket = + gmP.result.classification === 'ok' ? 'Gamma returned no active market with a token id' : 'needs a token id from Gamma /markets, which failed'; + checks.push( + byId.get('clob_market') ?? skipped('clob_market', 'clob', 'CLOB GET /markets/{condition_id}', noMarket, true), + byId.get('clob_book') ?? skipped('clob_book', 'clob', 'CLOB GET /book', noMarket, true), + byId.get('data_positions')!, + byId.get('data_activity')! + ); + + // Phase 3: authenticated read-only CLOB calls (sequential) + if (opts.auth) checks.push(...(await authChecks(ctx, clob, opts.auth))); + + // Phase 4: Binance public + let binanceClockSkewMs: number | null = null; + if (opts.binance) { + const [ping, time] = await pool<Probe<unknown>>( + [ + () => + probe(ctx, { + id: 'binance_ping', + group: 'binance', + name: 'Binance GET /api/v3/ping', + url: `${binance}/api/v3/ping`, + required: false, + shape: isObject, + }), + () => + probe(ctx, { + id: 'binance_time', + group: 'binance', + name: 'Binance GET /api/v3/time', + url: `${binance}/api/v3/time`, + required: false, + shape: (v) => (Number.isFinite(Number((v as { serverTime?: unknown } | undefined)?.serverTime)) ? null : 'expected { serverTime }'), + }), + ], + concurrency + ); + if (time.value !== undefined && time.t0 !== undefined && time.t1 !== undefined) { + binanceClockSkewMs = skew(Number((time.value as { serverTime: number }).serverTime), time.t0, time.t1); + time.result.detail = `clock skew ${formatSkew(binanceClockSkewMs)}`; + } + checks.push(ping.result, time.result); + } + + const partial = { ip: trace.ip, country: trace.country, geoblock, clobClockSkewMs, binanceClockSkewMs, proxy: opts.proxy ?? null, checks }; + return { startedAt: new Date(started).toISOString(), durationMs: ctx.now() - started, ...partial, verdict: computeVerdict(partial) }; +} + +function readGeoblock(p: Probe<unknown>): GeoblockInfo { + const none: GeoblockInfo = { status: 'unavailable', country: null, region: null, ip: null, detail: null }; + if (p.result.classification !== 'ok') { + // A geo answer from the endpoint itself (451 / restricted body) still counts as blocked + if (p.result.classification === 'geo_blocked') return { ...none, status: 'blocked', detail: p.result.detail }; + return { ...none, detail: `${p.result.classification}${p.result.detail ? `: ${p.result.detail}` : ''} (VERIFY the endpoint)` }; + } + const v = p.value as Record<string, unknown>; + const str = (x: unknown) => (typeof x === 'string' && x ? x : null); + if (typeof v.blocked !== 'boolean') { + p.result.classification = 'unexpected'; + p.result.detail = `no boolean "blocked" field (keys: ${Object.keys(v).slice(0, 8).join(', ') || 'none'}) — VERIFY the endpoint`; + return { ...none, detail: p.result.detail }; + } + const info: GeoblockInfo = { status: v.blocked ? 'blocked' : 'allowed', country: str(v.country), region: str(v.region), ip: str(v.ip), detail: null }; + const where = [info.country, info.region].filter(Boolean).join('/'); + p.result.detail = `${v.blocked ? 'BLOCKED' : 'not blocked'}${where ? ` (${where})` : ''}`; + if (v.blocked) p.result.classification = 'geo_blocked'; + return info; +} + +async function authChecks(ctx: Ctx, clob: string, auth: ClobAuthProbe | { skip: string }): Promise<CheckResult[]> { + const name = 'CLOB authenticated (read-only)'; + if ('skip' in auth) return [skipped('clob_auth', 'clob_auth', name, auth.skip, false)]; + if (!/^https:\/\//i.test(clob)) return [skipped('clob_auth', 'clob_auth', name, `refusing to send credentials to a non-https CLOB URL (${clob})`, false)]; + const out: CheckResult[] = []; + let l1: Record<string, string>; + try { + l1 = await auth.l1Headers(); + } catch (err) { + return [skipped('clob_auth', 'clob_auth', name, `cannot sign with the wallet: ${oneLine(err instanceof Error ? err.message : String(err), 120)}`, false)]; + } + // GET /auth/derive-api-key returns the wallet's existing CLOB API key (L1 signature); it creates nothing + const derive = await probe<Record<string, unknown>>(ctx, { + id: 'clob_auth_derive', + group: 'clob_auth', + name: `CLOB GET /auth/derive-api-key (${auth.label})`, + url: `${clob}/auth/derive-api-key`, + required: true, + headers: l1, + secretBody: true, + shape: isObject, + }); + const v = derive.value; + const creds: ClobApiCreds | null = + v && typeof v.apiKey === 'string' && typeof v.secret === 'string' && typeof v.passphrase === 'string' + ? { key: v.apiKey, secret: v.secret, passphrase: v.passphrase } + : null; + if (derive.result.classification === 'ok' && !creds) { + derive.result.classification = 'unexpected'; + derive.result.detail = 'no API key in the response'; + } + if (derive.result.status === 400 && derive.result.classification === 'unexpected') { + // "Could not derive api key!": the wallet never created one. Reaching the endpoint is what matters here. + derive.result.classification = 'skipped'; + derive.result.required = false; + derive.result.detail = 'no CLOB API key exists for this wallet yet (BotLab creates one on first live use; this check never creates one)'; + } + out.push(derive.result); + if (!creds) return out; + + const secrets = [creds.key, creds.secret, creds.passphrase]; + const l2 = async (id: string, label: string, path: string, shape: (v: unknown) => string | null) => { + let headers: Record<string, string>; + try { + headers = await auth.l2Headers(creds, 'GET', path); + } catch (err) { + return { + result: skipped(id, 'clob_auth', label, `cannot sign: ${oneLine(err instanceof Error ? err.message : String(err), 120)}`, false), + } as Probe<unknown>; + } + return probe<unknown>(ctx, { id, group: 'clob_auth', name: label, url: `${clob}${path}`, required: true, headers, secretBody: true, secrets, shape }); + }; + const keys = await l2('clob_auth_keys', 'CLOB GET /auth/api-keys (L2)', '/auth/api-keys', isObject); + if (keys.result.classification === 'ok') keys.result.detail = 'authenticated'; + out.push(keys.result); + const closed = await l2('clob_auth_closed_only', 'CLOB GET /auth/ban-status/closed-only (L2)', '/auth/ban-status/closed-only', (x) => + typeof (x as { closed_only?: unknown } | undefined)?.closed_only === 'boolean' ? null : 'expected { closed_only: boolean }' + ); + if (closed.result.classification === 'ok') { + const closedOnly = (closed.value as { closed_only: boolean }).closed_only; + closed.result.detail = closedOnly ? 'account is in CLOSE-ONLY mode' : 'account not restricted'; + if (closedOnly) closed.result.classification = 'forbidden'; + } + out.push(closed.result); + return out; +} + +// ── verdict ────────────────────────────────────────────────────────── + +const GEO_HINTS = [ + 'Polymarket restricts this country / region. Pick a provider region in an allowed country and test again.', + 'Do not use a VPN or proxy to get around a geo restriction: it breaks the Polymarket Terms of Service and can get the account closed (funds then only withdrawable).', +]; +const BOT_HINTS = [ + "This IP range looks blocked as a datacenter / by bot protection. Try another provider or region (many clouds' ranges are filtered), or run BotLab from a home connection.", + 'Re-run the check later: some blocks are temporary; keep a new VPS only if repeated checks are OK.', +]; + +export function formatSkew(ms: number): string { + return `${ms >= 0 ? '+' : ''}${(ms / 1000).toFixed(1)} s`; +} + +export function computeVerdict(r: Omit<ConnectivityReport, 'verdict' | 'startedAt' | 'durationMs'>): ConnectivityVerdict { + const required = r.checks.filter((c) => c.required && c.classification !== 'skipped'); + const failing = required.filter((c) => c.classification !== 'ok'); + const names = (cs: CheckResult[]) => cs.map((c) => c.id).join(', '); + const hints: string[] = []; + if (r.proxy) hints.push(`Requests went through the proxy ${r.proxy}: these results describe the proxy's IP, not this machine's.`); + + const geo = failing.filter((c) => c.classification === 'geo_blocked'); + if (r.geoblock.status === 'blocked' || geo.length > 0) { + const where = [r.geoblock.country ?? r.country, r.geoblock.region].filter(Boolean).join('/'); + const why = [r.geoblock.status === 'blocked' ? 'geoblock endpoint says blocked' : null, geo.length ? `geo errors on ${names(geo)}` : null] + .filter(Boolean) + .join('; '); + return { kind: 'blocked_geo', line: `BLOCKED (geo): ${why}${where ? ` [${where}]` : ''}`, hints: [...hints, ...GEO_HINTS], exitCode: 1 }; + } + + const bot = failing.filter((c) => c.classification === 'bot_challenge' || (c.classification === 'forbidden' && c.id !== 'clob_auth_closed_only')); + const net = failing.filter((c) => c.classification === 'network_error'); + const ipOk = r.checks.find((c) => c.id === 'ip')?.classification === 'ok'; + if (bot.length > 0) { + return { + kind: 'blocked_bot', + line: `BLOCKED (datacenter/bot protection): ${bot.map((c) => `${c.id} ${c.classification}`).join(', ')}`, + hints: [...hints, ...BOT_HINTS], + exitCode: 1, + }; + } + if (required.length > 0 && net.length === required.length) { + if (ipOk) { + return { + kind: 'blocked_bot', + line: `BLOCKED (datacenter/bot protection): every Polymarket request failed at the connection level while other sites work (${net[0].detail ?? 'network error'})`, + hints: [...hints, ...BOT_HINTS], + exitCode: 1, + }; + } + return { + kind: 'degraded', + line: `DEGRADED: no request succeeded, not even the IP lookup (${net[0].detail ?? 'network error'}) — check this machine's internet access, DNS, firewall or proxy`, + hints, + exitCode: 2, + }; + } + + const problems: string[] = []; + if (failing.length > 0) problems.push(failing.map((c) => `${c.id} ${c.classification}`).join(', ')); + const closedOnly = r.checks.find((c) => c.id === 'clob_auth_closed_only' && c.classification === 'forbidden'); + if (closedOnly) { + hints.push( + 'Close-only mode: Polymarket lets this account only reduce positions (typically after access from a restricted region). Contact Polymarket support; do not trade around it.' + ); + } + if (r.clobClockSkewMs !== null && Math.abs(r.clobClockSkewMs) - 1000 > MAX_CLOCK_SKEW_MS) { + problems.push(`clock skew ${formatSkew(r.clobClockSkewMs)} vs the CLOB (live orders are refused beyond ${MAX_CLOCK_SKEW_MS / 1000} s)`); + hints.push('Enable time sync (NTP / chrony / systemd-timesyncd / Windows Time) on this machine.'); + } + const bin = r.checks.filter((c) => c.group === 'binance' && c.classification !== 'ok' && c.classification !== 'skipped'); + if (bin.length > 0) { + problems.push(`Binance: ${bin.map((c) => `${c.id} ${c.classification}`).join(', ')}`); + if (bin.some((c) => c.classification === 'geo_blocked')) + hints.push('Binance restricts this location (HTTP 451 / restricted location): Binance bots will not work from here.'); + } + if (failing.some((c) => c.classification === 'rate_limited')) + hints.push('HTTP 429 on a handful of requests suggests the IP range is rate limited; re-run in a few minutes.'); + if (r.geoblock.status === 'unavailable') hints.push('The Polymarket geoblock endpoint could not be read; the verdict rests on the API checks.'); + + if (problems.length > 0) return { kind: 'degraded', line: `DEGRADED: ${problems.join('; ')}`, hints, exitCode: 2 }; + return { kind: 'ok', line: 'OK: this IP can use Polymarket', hints, exitCode: 0 }; +} + +// ── text output ────────────────────────────────────────────────────── + +export function formatReport(r: ConnectivityReport): string { + const lines: string[] = []; + const where = [r.ip ?? 'unknown IP', r.country ? `country ${r.country}` : null].filter(Boolean).join(', '); + lines.push(`Connectivity check — ${where}${r.proxy ? ` (via proxy ${r.proxy})` : ''}`); + const geo = r.geoblock; + lines.push( + `Polymarket geoblock: ${geo.status === 'unavailable' ? `unavailable${geo.detail ? ` (${geo.detail})` : ''}` : `${geo.status}${geo.country ? ` (${[geo.country, geo.region].filter(Boolean).join('/')})` : ''}`}` + ); + lines.push(''); + const rows = r.checks.map((c) => [ + c.name, + c.status === null ? '-' : String(c.status), + c.latencyMs === null ? '-' : `${c.latencyMs} ms`, + c.classification, + c.detail ?? '', + ]); + const head = ['Check', 'HTTP', 'Latency', 'Result', 'Detail']; + const width = (i: number) => Math.max(head[i].length, ...rows.map((row) => row[i].length)); + const widths = [0, 1, 2, 3].map(width); + const fmt = (row: string[]) => + row + .map((cell, i) => (i < 4 ? cell.padEnd(widths[i]) : cell)) + .join(' ') + .trimEnd(); + lines.push(fmt(head)); + lines.push(widths.map((w) => '-'.repeat(w)).join(' ') + ' ------'); + for (const row of rows) lines.push(fmt(row)); + lines.push(''); + lines.push(r.verdict.line); + for (const h of r.verdict.hints) lines.push(` - ${h}`); + return lines.join('\n'); +} diff --git a/tests/core/connectivity-check.test.ts b/tests/core/connectivity-check.test.ts new file mode 100644 index 0000000..3fcce39 --- /dev/null +++ b/tests/core/connectivity-check.test.ts @@ -0,0 +1,371 @@ +import { describe, it, expect } from 'vitest'; +import { + classifyError, + classifyResponse, + formatReport, + runConnectivityCheck, + type ClobAuthProbe, + type ConnectivityOptions, +} from '../../packages/core/src/ops/connectivityCheck'; + +/** Never the live API: every URL is answered by a route table. */ +type Route = (url: string, init?: RequestInit) => Response | Promise<Response>; + +const json = (v: unknown, status = 200, headers: Record<string, string> = {}) => + new Response(JSON.stringify(v), { status, headers: { 'content-type': 'application/json', ...headers } }); +const html = (body: string, status: number, headers: Record<string, string> = {}) => + new Response(body, { status, headers: { 'content-type': 'text/html', ...headers } }); + +const TOKEN = '71321045679252212594626385532706912750332728571942532289631379312455583992563'; +const CONDITION = '0xabc0000000000000000000000000000000000000000000000000000000000001'; +const NOW = 1_800_000_000_000; + +function healthy(): Record<string, Route> { + return { + 'cdn-cgi/trace': () => new Response('fl=1\nh=www.cloudflare.com\nip=203.0.113.7\nts=1\nloc=IE\n', { status: 200 }), + 'polymarket.com/api/geoblock': () => json({ blocked: false, ip: '203.0.113.7', country: 'IE', region: 'L' }), + 'clob.polymarket.com/time': () => json(Math.floor(NOW / 1000)), + 'gamma-api.polymarket.com/markets': () => json([{ question: 'Will it rain?', conditionId: CONDITION, clobTokenIds: JSON.stringify([TOKEN, '2']) }]), + 'gamma-api.polymarket.com/events': () => json([{ id: '1' }]), + 'clob.polymarket.com/markets/': () => json({ condition_id: CONDITION }), + 'clob.polymarket.com/book': () => json({ bids: [], asks: [] }), + 'data-api.polymarket.com/positions': () => json([]), + 'data-api.polymarket.com/activity': () => json([]), + 'api.binance.com/api/v3/ping': () => json({}), + 'api.binance.com/api/v3/time': () => json({ serverTime: NOW }), + 'clob.polymarket.com/auth/derive-api-key': () => json({ apiKey: 'key-1234567890', secret: 'secret-abcdefgh', passphrase: 'pass-xyz123' }), + 'clob.polymarket.com/auth/api-keys': () => json({ apiKeys: ['key-1234567890'] }), + 'clob.polymarket.com/auth/ban-status/closed-only': () => json({ closed_only: false }), + }; +} + +function fakeFetch(routes: Record<string, Route>, calls: Array<{ url: string; init?: RequestInit }> = []): typeof fetch { + return (async (input: string | URL | Request, init?: RequestInit) => { + const url = String(input); + calls.push({ url, init }); + // Longest matching key first (".../markets/" before ".../markets") + const key = Object.keys(routes) + .filter((k) => url.includes(k)) + .sort((a, b) => b.length - a.length)[0]; + if (!key) throw new Error(`unrouted ${url}`); + return routes[key](url, init); + }) as typeof fetch; +} + +const netError = (code: string) => () => { + throw Object.assign(new TypeError('fetch failed'), { cause: Object.assign(new Error(code), { code }) }); +}; + +function run(routes: Record<string, Route>, extra: ConnectivityOptions = {}, calls?: Array<{ url: string; init?: RequestInit }>) { + return runConnectivityCheck({ fetch: fakeFetch(routes, calls), now: () => NOW, ...extra }); +} + +const byId = (r: Awaited<ReturnType<typeof run>>, id: string) => r.checks.find((c) => c.id === id)!; + +const authProbe: ClobAuthProbe = { + label: 'wallet "Main" 0x1', + l1Headers: async () => ({ POLY_ADDRESS: '0x1', POLY_SIGNATURE: 'l1sig', POLY_TIMESTAMP: '1', POLY_NONCE: '0' }), + l2Headers: async (creds, method, path) => ({ + POLY_ADDRESS: '0x1', + POLY_API_KEY: creds.key, + POLY_PASSPHRASE: creds.passphrase, + POLY_SIGNATURE: `l2:${method}:${path}`, + }), +}; + +describe('connectivity check — verdicts', () => { + it('everything reachable ⇒ OK, exit 0; read-only GETs; the Gamma token id is used for the book', async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const r = await run(healthy(), { binance: true }, calls); + expect(r.verdict.kind).toBe('ok'); + expect(r.verdict.exitCode).toBe(0); + expect(r.verdict.line).toBe('OK: this IP can use Polymarket'); + expect(r.ip).toBe('203.0.113.7'); + expect(r.country).toBe('IE'); + expect(r.geoblock).toMatchObject({ status: 'allowed', country: 'IE', region: 'L' }); + expect(r.clobClockSkewMs).toBe(0); + expect(r.checks.every((c) => c.classification === 'ok')).toBe(true); + expect(calls.every((c) => (c.init?.method ?? 'GET') === 'GET')).toBe(true); + expect(calls.find((c) => c.url.includes('/book'))!.url).toContain(`token_id=${TOKEN}`); + expect(calls.find((c) => c.url.includes('clob.polymarket.com/markets/'))!.url).toContain(CONDITION); + expect(calls.some((c) => c.url.includes('/auth/'))).toBe(false); + expect(formatReport(r)).toContain('OK: this IP can use Polymarket'); + }); + + it('geoblock endpoint says blocked ⇒ BLOCKED (geo), exit 1, ToS hint', async () => { + const r = await run({ ...healthy(), 'polymarket.com/api/geoblock': () => json({ blocked: true, ip: '1.2.3.4', country: 'US', region: 'NY' }) }); + expect(r.verdict.kind).toBe('blocked_geo'); + expect(r.verdict.exitCode).toBe(1); + expect(r.verdict.line).toMatch(/^BLOCKED \(geo\): geoblock endpoint says blocked \[US\/NY\]/); + expect(r.verdict.hints.join(' ')).toMatch(/VPN or proxy.*Terms of Service/); + expect(byId(r, 'geoblock').classification).toBe('geo_blocked'); + }); + + it('HTTP 451 or a "restricted region" body on an API ⇒ geo_blocked', async () => { + const r1 = await run({ ...healthy(), 'clob.polymarket.com/time': () => json({ error: 'unavailable' }, 451) }); + expect(byId(r1, 'clob_time').classification).toBe('geo_blocked'); + expect(r1.verdict.kind).toBe('blocked_geo'); + const r2 = await run({ + ...healthy(), + 'clob.polymarket.com/book': () => json({ error: 'Trading restricted in your region, please refer to available regions' }, 403), + }); + expect(byId(r2, 'clob_book').classification).toBe('geo_blocked'); + expect(r2.verdict.exitCode).toBe(1); + }); + + it('Cloudflare cf-mitigated header ⇒ bot_challenge ⇒ BLOCKED (datacenter/bot protection)', async () => { + const r = await run({ + ...healthy(), + 'gamma-api.polymarket.com/events': () => html('<html>…</html>', 403, { 'cf-mitigated': 'challenge', 'cf-ray': 'abc-DUB' }), + }); + expect(byId(r, 'gamma_events').classification).toBe('bot_challenge'); + expect(r.verdict.kind).toBe('blocked_bot'); + expect(r.verdict.exitCode).toBe(1); + expect(r.verdict.line).toMatch(/^BLOCKED \(datacenter\/bot protection\): gamma_events bot_challenge/); + expect(r.verdict.hints.join(' ')).toMatch(/another provider/); + }); + + it('"Just a moment…" HTML from Cloudflare ⇒ bot_challenge (also on a 200 where JSON was expected)', async () => { + const page = '<!DOCTYPE html><html><head><title>Just a moment...challenge-platform'; + const r = await run({ + ...healthy(), + 'data-api.polymarket.com/positions': () => html(page, 403, { 'cf-ray': 'x', server: 'cloudflare' }), + 'data-api.polymarket.com/activity': () => html(page, 200, { 'cf-ray': 'x', server: 'cloudflare' }), + }); + expect(byId(r, 'data_positions').classification).toBe('bot_challenge'); + expect(byId(r, 'data_positions').detail).toMatch(/Cloudflare challenge page/); + expect(byId(r, 'data_activity').classification).toBe('bot_challenge'); + expect(r.verdict.kind).toBe('blocked_bot'); + }); + + it('a plain 403 ⇒ forbidden ⇒ BLOCKED (datacenter/bot protection)', async () => { + const r = await run({ ...healthy(), 'clob.polymarket.com/time': () => html('403 Forbidden', 403) }); + expect(byId(r, 'clob_time').classification).toBe('forbidden'); + expect(byId(r, 'clob_time').detail).toMatch(/HTML page.*403 Forbidden/); + expect(r.verdict.kind).toBe('blocked_bot'); + }); + + it('429 ⇒ rate_limited ⇒ DEGRADED, exit 2', async () => { + const r = await run({ ...healthy(), 'gamma-api.polymarket.com/events': () => json({ error: 'too many' }, 429, { 'retry-after': '30' }) }); + expect(byId(r, 'gamma_events')).toMatchObject({ classification: 'rate_limited', detail: 'HTTP 429 (retry-after 30)' }); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.exitCode).toBe(2); + expect(r.verdict.line).toMatch(/^DEGRADED: gamma_events rate_limited/); + }); + + it('every Polymarket request dropped while other sites work ⇒ BLOCKED (datacenter/bot protection)', async () => { + const routes = healthy(); + for (const k of Object.keys(routes)) if (k.includes('polymarket.com')) routes[k] = netError('ECONNRESET'); + const r = await run(routes); + expect(byId(r, 'clob_time')).toMatchObject({ classification: 'network_error', detail: 'connection reset (ECONNRESET)' }); + expect(byId(r, 'clob_book').classification).toBe('skipped'); + expect(r.geoblock.status).toBe('unavailable'); + expect(r.verdict.kind).toBe('blocked_bot'); + expect(r.verdict.line).toMatch(/every Polymarket request failed at the connection level/); + }); + + it('no request succeeds at all (not even the IP lookup) ⇒ DEGRADED (no network)', async () => { + const routes = healthy(); + for (const k of Object.keys(routes)) routes[k] = netError('ENOTFOUND'); + const r = await run(routes); + expect(r.ip).toBeNull(); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.line).toMatch(/not even the IP lookup.*DNS lookup failed/); + }); + + it('one connection reset among working requests ⇒ DEGRADED', async () => { + const r = await run({ ...healthy(), 'data-api.polymarket.com/activity': netError('ECONNRESET') }); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.line).toBe('DEGRADED: data_activity network_error'); + }); + + it('non-JSON where JSON is expected ⇒ unexpected ⇒ DEGRADED; a wrong shape too', async () => { + const r = await run({ + ...healthy(), + 'gamma-api.polymarket.com/events': () => new Response('hello', { status: 200 }), + 'clob.polymarket.com/book': () => json({ nope: true }), + }); + expect(byId(r, 'gamma_events')).toMatchObject({ classification: 'unexpected' }); + expect(byId(r, 'gamma_events').detail).toMatch(/not JSON/); + expect(byId(r, 'clob_book').detail).toMatch(/expected \{ bids/); + expect(r.verdict.kind).toBe('degraded'); + }); + + it('geoblock endpoint missing or of another shape ⇒ unavailable (VERIFY), verdict from the API checks', async () => { + const r1 = await run({ ...healthy(), 'polymarket.com/api/geoblock': () => html('Not found', 404) }); + expect(r1.geoblock.status).toBe('unavailable'); + expect(r1.geoblock.detail).toMatch(/VERIFY/); + expect(r1.verdict.kind).toBe('ok'); + expect(r1.verdict.hints.join(' ')).toMatch(/geoblock endpoint could not be read/); + const r2 = await run({ ...healthy(), 'polymarket.com/api/geoblock': () => json({ isBlocked: 'no' }) }); + expect(r2.geoblock.status).toBe('unavailable'); + expect(byId(r2, 'geoblock').detail).toMatch(/no boolean "blocked" field.*VERIFY/); + expect(r2.verdict.kind).toBe('ok'); + }); + + it('IP lookup failure is "unknown", not fatal', async () => { + const r = await run({ ...healthy(), 'cdn-cgi/trace': netError('ETIMEDOUT') }); + expect(r.ip).toBeNull(); + expect(r.verdict.kind).toBe('ok'); + expect(formatReport(r)).toContain('unknown IP'); + }); + + it('clock skew beyond the live-order limit ⇒ DEGRADED with a time-sync hint', async () => { + const r = await run({ ...healthy(), 'clob.polymarket.com/time': () => json(Math.floor(NOW / 1000) + 30) }); + expect(r.clobClockSkewMs).toBe(30_000); + expect(byId(r, 'clob_time').detail).toBe('clock skew +30.0 s (1 s resolution)'); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.line).toMatch(/clock skew \+30\.0 s/); + expect(r.verdict.hints.join(' ')).toMatch(/time sync/); + }); + + it('Binance 451 ⇒ geo_blocked on Binance only: Polymarket not blocked, verdict DEGRADED', async () => { + const r = await run( + { + ...healthy(), + 'api.binance.com/api/v3/ping': () => json({ code: 0, msg: "Service unavailable from a restricted location according to 'b. Eligibility'" }, 451), + 'api.binance.com/api/v3/time': () => json({ code: 0, msg: 'Service unavailable from a restricted location' }, 451), + }, + { binance: true } + ); + expect(byId(r, 'binance_ping').classification).toBe('geo_blocked'); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.line).toMatch(/Binance: binance_ping geo_blocked/); + expect(r.verdict.hints.join(' ')).toMatch(/Binance restricts this location/); + }); + + it('times out each request after timeoutMs ⇒ network_error', async () => { + const hang: Route = (_url, init) => new Promise((_resolve, reject) => init?.signal?.addEventListener('abort', () => reject(init.signal!.reason))); + const r = await runConnectivityCheck({ fetch: fakeFetch({ ...healthy(), 'gamma-api.polymarket.com/events': hang }), timeoutMs: 50 }); + expect(byId(r, 'gamma_events')).toMatchObject({ classification: 'network_error', detail: 'timeout after 0 s' }); + expect(r.verdict.kind).toBe('degraded'); + }); + + it('reports the proxy and warns that results reflect its IP', async () => { + const r = await run(healthy(), { proxy: 'proxy.example:3128' }); + expect(r.verdict.hints[0]).toMatch(/proxy proxy\.example:3128.*proxy's IP/); + expect(formatReport(r)).toContain('via proxy proxy.example:3128'); + }); +}); + +describe('connectivity check — authenticated (read-only)', () => { + it('derives the existing key, calls two L2 GETs, never prints a secret', async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const r = await run(healthy(), { auth: authProbe }, calls); + expect(byId(r, 'clob_auth_derive').classification).toBe('ok'); + expect(byId(r, 'clob_auth_keys')).toMatchObject({ classification: 'ok', detail: 'authenticated' }); + expect(byId(r, 'clob_auth_closed_only')).toMatchObject({ classification: 'ok', detail: 'account not restricted' }); + expect(r.verdict.kind).toBe('ok'); + const authCalls = calls.filter((c) => c.url.includes('/auth/')); + expect(authCalls.map((c) => c.url.replace('https://clob.polymarket.com', ''))).toEqual([ + '/auth/derive-api-key', + '/auth/api-keys', + '/auth/ban-status/closed-only', + ]); + expect(authCalls.every((c) => c.init?.method === 'GET')).toBe(true); + expect((authCalls[1].init?.headers as Record).POLY_SIGNATURE).toBe('l2:GET:/auth/api-keys'); + const out = JSON.stringify(r) + formatReport(r); + for (const s of ['key-1234567890', 'secret-abcdefgh', 'pass-xyz123', 'l1sig']) expect(out).not.toContain(s); + }); + + it('error bodies of authenticated calls are not echoed', async () => { + const r = await run( + { ...healthy(), 'clob.polymarket.com/auth/api-keys': () => json({ error: 'Unauthorized/Invalid api key key-1234567890' }, 401) }, + { auth: authProbe } + ); + expect(byId(r, 'clob_auth_keys')).toMatchObject({ classification: 'unexpected', detail: 'HTTP 401' }); + expect(JSON.stringify(r)).not.toContain('key-1234567890'); + expect(r.verdict.kind).toBe('degraded'); + }); + + it('closed-only account ⇒ DEGRADED with a support hint (not a datacenter block)', async () => { + const r = await run({ ...healthy(), 'clob.polymarket.com/auth/ban-status/closed-only': () => json({ closed_only: true }) }, { auth: authProbe }); + expect(byId(r, 'clob_auth_closed_only').detail).toBe('account is in CLOSE-ONLY mode'); + expect(r.verdict.kind).toBe('degraded'); + expect(r.verdict.hints.join(' ')).toMatch(/Close-only mode/); + }); + + it('no API key to derive (400) ⇒ skipped, never created; verdict stays OK', async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const r = await run( + { ...healthy(), 'clob.polymarket.com/auth/derive-api-key': () => json({ error: 'Could not derive api key!' }, 400) }, + { auth: authProbe }, + calls + ); + expect(byId(r, 'clob_auth_derive')).toMatchObject({ classification: 'skipped' }); + expect(byId(r, 'clob_auth_derive').detail).toMatch(/never creates one/); + expect(calls.filter((c) => c.url.includes('/auth/'))).toHaveLength(1); + expect(r.verdict.kind).toBe('ok'); + }); + + it('no wallet / no key ⇒ a skipped row with the reason', async () => { + const r = await run(healthy(), { auth: { skip: 'no wallet is configured in BotLab (add one in Wallets first)' } }); + expect(byId(r, 'clob_auth')).toMatchObject({ classification: 'skipped', detail: 'no wallet is configured in BotLab (add one in Wallets first)' }); + expect(r.verdict.kind).toBe('ok'); + }); +}); + +describe('connectivity check — credentials never follow a redirect', () => { + it('a 3xx on an authenticated probe is not followed (manual redirect) and is "unexpected"', async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const r = await run( + { + ...healthy(), + 'clob.polymarket.com/auth/derive-api-key': () => new Response(null, { status: 302, headers: { location: 'https://evil.example/collect' } }), + }, + { auth: authProbe }, + calls + ); + expect(byId(r, 'clob_auth_derive')).toMatchObject({ classification: 'unexpected', status: 302 }); + expect(byId(r, 'clob_auth_derive').detail).toBe('HTTP 302 redirect to evil.example not followed (credentials are never sent to a redirect)'); + expect(calls.some((c) => c.url.includes('evil.example'))).toBe(false); + const authCalls = calls.filter((c) => c.url.includes('/auth/')); + expect(authCalls).toHaveLength(1); + expect(authCalls[0].init?.redirect).toBe('manual'); + // Public probes keep following redirects + expect(calls.find((c) => c.url.endsWith('/time'))!.init?.redirect).toBe('follow'); + expect(r.verdict.kind).toBe('degraded'); + }); + + it('refuses to send credentials to an http: CLOB URL', async () => { + const calls: Array<{ url: string; init?: RequestInit }> = []; + const r = await runConnectivityCheck({ fetch: fakeFetch(healthy(), calls), now: () => NOW, clobUrl: 'http://clob.polymarket.com', auth: authProbe }); + expect(byId(r, 'clob_auth')).toMatchObject({ classification: 'skipped' }); + expect(byId(r, 'clob_auth').detail).toMatch(/non-https/); + expect(calls.some((c) => c.url.includes('/auth/'))).toBe(false); + }); +}); + +describe('connectivity check — classifiers', () => { + it('Cloudflare country ban (error 1009) ⇒ geo_blocked with the ToS warning', async () => { + const page = + 'Access denied | clob.polymarket.com used Cloudflare to restrict access' + + 'Error 1009

The owner of this website (clob.polymarket.com) has banned the country or region your IP address is in (US) from accessing this website.

'; + const r = await run({ ...healthy(), 'clob.polymarket.com/time': () => html(page, 403, { 'cf-ray': 'abc-IAD', server: 'cloudflare' }) }); + expect(byId(r, 'clob_time').classification).toBe('geo_blocked'); + expect(r.verdict.kind).toBe('blocked_geo'); + expect(r.verdict.hints.join(' ')).toMatch(/Terms of Service/); + expect(classifyResponse({ status: 403, headers: {}, body: 'error 1009', expectJson: true }).classification).toBe('geo_blocked'); + }); + + it('classifyResponse', () => { + expect(classifyResponse({ status: 200, headers: {}, body: '{}', expectJson: true }).classification).toBe('ok'); + expect(classifyResponse({ status: 451, headers: {}, body: '', expectJson: true }).classification).toBe('geo_blocked'); + expect( + classifyResponse({ status: 503, headers: { 'CF-RAY': 'x' }, body: 'Attention Required! | Cloudflare', expectJson: true }) + .classification + ).toBe('bot_challenge'); + expect(classifyResponse({ status: 403, headers: {}, body: '{"error":"nope"}', expectJson: true }).classification).toBe('forbidden'); + expect(classifyResponse({ status: 500, headers: {}, body: 'boom', expectJson: true })).toEqual({ classification: 'unexpected', detail: 'HTTP 500: boom' }); + }); + + it('classifyError walks fetch cause chains (proxy refusal, TLS, timeout)', () => { + const undiciErr = Object.assign(new Error('Proxy response (403) !== 200 when HTTP Tunneling'), { code: 'UND_ERR_ABORTED' }); + const dom = Object.assign(new Error('Request was cancelled.'), { cause: undiciErr }); + const top = Object.assign(new TypeError('fetch failed'), { cause: dom }); + expect(classifyError(top, 10_000)).toEqual({ classification: 'network_error', detail: 'the proxy refused the connection (HTTP 403 to CONNECT)' }); + const tls = Object.assign(new TypeError('fetch failed'), { cause: Object.assign(new Error('bad cert'), { code: 'CERT_HAS_EXPIRED' }) }); + expect(classifyError(tls, 10_000).detail).toBe('TLS error (CERT_HAS_EXPIRED)'); + expect(classifyError(new DOMException('timed out', 'TimeoutError'), 10_000).detail).toBe('timeout after 10 s'); + }); +}); diff --git a/tests/exchange/binance-redirect.test.ts b/tests/exchange/binance-redirect.test.ts new file mode 100644 index 0000000..fde1330 --- /dev/null +++ b/tests/exchange/binance-redirect.test.ts @@ -0,0 +1,29 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import fs from 'fs'; +import path from 'path'; +import { getBinanceBalances, queryBinanceOrder, SIGNED_REDIRECT } from '../../packages/core/src/exchange/binance/client'; + +/** fetch forwards custom headers (X-MBX-APIKEY) across hosts on a redirect: signed requests must not follow one. */ +describe('Binance signed requests never follow redirects', () => { + afterEach(() => vi.unstubAllGlobals()); + + it('signed GETs pass redirect: "error"', async () => { + const inits: RequestInit[] = []; + vi.stubGlobal('fetch', async (_url: string, init: RequestInit) => { + inits.push(init); + return new Response(JSON.stringify({ balances: [], code: -2013 }), { status: 200 }); + }); + await getBinanceBalances({ apiKey: 'k', apiSecret: 's' }).catch(() => undefined); + await queryBinanceOrder({ apiKey: 'k', apiSecret: 's' }, 'BTCUSDT', 'id1').catch(() => undefined); + expect(SIGNED_REDIRECT).toBe('error'); + expect(inits).toHaveLength(2); + for (const init of inits) expect(init.redirect).toBe('error'); + }); + + it('every request carrying X-MBX-APIKEY in client.ts sets the redirect policy (signed POST included)', () => { + const src = fs.readFileSync(path.join(__dirname, '..', '..', 'packages/core/src/exchange/binance/client.ts'), 'utf8'); + const uses = src.split("'X-MBX-APIKEY'").slice(1); + expect(uses.length).toBeGreaterThanOrEqual(3); + for (const after of uses) expect(after.slice(0, 80)).toMatch(/redirect: SIGNED_REDIRECT/); + }); +}); diff --git a/tests/exchange/clob-client-axios.test.ts b/tests/exchange/clob-client-axios.test.ts index c6d3c80..3908533 100644 --- a/tests/exchange/clob-client-axios.test.ts +++ b/tests/exchange/clob-client-axios.test.ts @@ -148,3 +148,62 @@ describe('L2-authenticated POST through the overridden axios, and credential red }); }); + +describe('L2 requests never follow a redirect (POLY_* headers stay on the CLOB host)', () => { + let a: http.Server; + let b: http.Server; + let hostA = ''; + const atB: Array<{ url: string; headers: http.IncomingHttpHeaders }> = []; + + beforeAll(async () => { + b = http.createServer((req, res) => { + atB.push({ url: req.url ?? '', headers: req.headers }); + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(req.url === '/time' ? '1793443411' : JSON.stringify({ closed_only: false })); + }); + await new Promise((resolve) => b.listen(0, '127.0.0.1', resolve)); + // Another host name for the same loopback: a cross-host redirect + const hostB = `http://localhost:${(b.address() as AddressInfo).port}`; + a = http.createServer((req, res) => { + const target = req.url === '/time' ? '/time' : '/auth/ban-status/closed-only'; + res.writeHead(302, { location: `${hostB}${target}` }); + res.end(); + }); + await new Promise((resolve) => a.listen(0, '127.0.0.1', resolve)); + hostA = `http://127.0.0.1:${(a.address() as AddressInfo).port}`; + }); + + afterAll(async () => { + await new Promise((resolve) => a.close(resolve)); + await new Promise((resolve) => b.close(resolve)); + }); + + it('an authenticated CLOB request is not redirected; nothing reaches the other host', async () => { + atB.length = 0; + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + const wallet = ethers.Wallet.createRandom(); + const clob = new ClobClient(hostA, 137, wallet, { key: 'api-key-SECRET', secret: Buffer.from('s3cret-bytes-0123').toString('base64'), passphrase: 'pp-SECRET' }); + const r = await clob.getClosedOnlyMode(); + // clob-client turns the refused redirect into an error result + expect(JSON.stringify(r)).toMatch(/302/); + expect(atB).toEqual([]); + } finally { + spy.mockRestore(); + } + }); + + it('unauthenticated requests still follow redirects (the rule only targets POLY_* requests)', async () => { + atB.length = 0; + const clob = new ClobClient(hostA, 137); + expect(await clob.getServerTime()).toBe(1793443411); + expect(atB.map((x) => x.url)).toEqual(['/time']); + expect(Object.keys(atB[0].headers).some((k) => k.startsWith('poly_'))).toBe(false); + }); + + it('refuseAuthRedirects sets maxRedirects 0 only for POLY_* headers', async () => { + const { refuseAuthRedirects } = await import('../../packages/core/src/exchange/polymarket/redactCredentials'); + expect(refuseAuthRedirects({ headers: { POLY_API_KEY: 'x' } })).toMatchObject({ maxRedirects: 0 }); + expect(refuseAuthRedirects({ headers: { Accept: 'application/json' } })).not.toHaveProperty('maxRedirects'); + }); +}); diff --git a/tests/server/check-polymarket.test.ts b/tests/server/check-polymarket.test.ts new file mode 100644 index 0000000..e38bc93 --- /dev/null +++ b/tests/server/check-polymarket.test.ts @@ -0,0 +1,140 @@ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import { ethers } from 'ethers'; +import { initializeDatabase, closeDatabase } from '../../packages/core/src/db/index'; +import { encrypt } from '../../packages/core/src/security/crypto'; +import { makeKeyCheck } from '../../packages/core/src/security/masterKey'; +import { buildAuth, bypassesProxy, effectiveProxy, envProxy, openReadOnly } from '../../apps/server/src/checkPolymarket'; +import { createClobReadOnlyAuth } from '../../packages/core/src/exchange/polymarket/authHeaders'; + +const KEY = 'ab'.repeat(32); +const OTHER_KEY = 'cd'.repeat(32); +const PK = '0x' + '11'.repeat(32); +let dir: string; +let dbFile: string; + +beforeAll(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'botlab-checkpm-')); + dbFile = path.join(dir, 'botlab.db'); + const db = initializeDatabase(dbFile); + db.prepare("UPDATE settings SET key_check = ? WHERE id = 'global'").run(makeKeyCheck(KEY)); + db.prepare( + "INSERT INTO wallets (id, name, address, proxy_address, private_key_encrypted) VALUES ('w1', 'Main', ?, '0x00000000000000000000000000000000000000aa', ?)" + ).run(new ethers.Wallet(PK).address, encrypt(PK, KEY)); + closeDatabase(db); +}); + +afterAll(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('check-polymarket: database', () => { + it('opens read-only: reads work, writes fail', () => { + const opened = openReadOnly(dbFile); + expect(opened.db).not.toBeNull(); + try { + expect(opened.db!.prepare('SELECT COUNT(*) AS n FROM wallets').get()).toEqual({ n: 1 }); + expect(() => opened.db!.prepare("UPDATE settings SET key_check = NULL WHERE id = 'global'").run()).toThrow(/readonly/i); + } finally { + opened.db?.close(); + } + }); + + it('a missing file and an unopenable path give their own messages', () => { + const missing = openReadOnly(path.join(dir, 'nope.db')); + expect(missing).toMatchObject({ db: null }); + expect((missing as { error: string }).error).toMatch(/no BotLab database at/); + // A directory where the file should be: exists, but SQLite cannot open it + const notADb = path.join(dir, 'dir.db'); + fs.mkdirSync(notADb); + const bad = openReadOnly(notADb); + const err = (bad as { error: string }).error; + expect(err).toMatch(/^cannot open .*dir\.db read-only: \S/); + expect(err).not.toMatch(/no BotLab database/); + }); +}); + +describe('check-polymarket: credentials', () => { + it('master key matches ⇒ a probe labelled with the wallet; data checks use its proxy address', () => { + const opened = openReadOnly(dbFile); + try { + const a = buildAuth(opened, dir, { 'with-credentials': true }, { BOTLAB_MASTER_KEY: KEY }); + expect(a.auth).not.toHaveProperty('skip'); + expect((a.auth as { label: string }).label).toContain('"Main"'); + expect(a.address).toBe('0x00000000000000000000000000000000000000aa'); + } finally { + opened.db?.close(); + } + }); + + it('a master key that does not match ⇒ skipped, nothing decrypted', () => { + const opened = openReadOnly(dbFile); + try { + const a = buildAuth(opened, dir, {}, { BOTLAB_MASTER_KEY: OTHER_KEY }); + expect(a.auth).toEqual({ skip: 'the master key does not match this database' }); + } finally { + opened.db?.close(); + } + }); + + it('no master key anywhere ⇒ skipped, and no key file is generated', () => { + const opened = openReadOnly(dbFile); + try { + const a = buildAuth(opened, dir, {}, {}); + expect((a.auth as { skip: string }).skip).toMatch(/^no master key/); + expect(fs.existsSync(path.join(dir, 'master.key'))).toBe(false); + } finally { + opened.db?.close(); + } + }); + + it('no database ⇒ skipped with the open error', () => { + const a = buildAuth(openReadOnly(path.join(dir, 'nope.db')), dir, {}, { BOTLAB_MASTER_KEY: KEY }); + expect((a.auth as { skip: string }).skip).toMatch(/no BotLab database at/); + }); + + it('unknown --wallet ⇒ skipped', () => { + const opened = openReadOnly(dbFile); + try { + expect(buildAuth(opened, dir, { wallet: 'Other' }, { BOTLAB_MASTER_KEY: KEY }).auth).toEqual({ skip: 'no wallet matches --wallet Other' }); + } finally { + opened.db?.close(); + } + }); +}); + +describe('check-polymarket: read-only auth headers', () => { + it('L1 / L2 headers carry the expected names and never the private key', async () => { + const probe = createClobReadOnlyAuth(PK, 'wallet'); + const l1 = await probe.l1Headers(); + expect(Object.keys(l1).sort()).toEqual(['POLY_ADDRESS', 'POLY_NONCE', 'POLY_SIGNATURE', 'POLY_TIMESTAMP']); + expect(l1.POLY_ADDRESS).toBe(new ethers.Wallet(PK).address); + const creds = { key: 'k-0000', secret: Buffer.from('secret-bytes').toString('base64'), passphrase: 'p-0000' }; + const l2 = await probe.l2Headers(creds, 'GET', '/auth/api-keys'); + expect(Object.keys(l2).sort()).toEqual(['POLY_ADDRESS', 'POLY_API_KEY', 'POLY_PASSPHRASE', 'POLY_SIGNATURE', 'POLY_TIMESTAMP']); + const all = JSON.stringify([l1, l2, probe]); + expect(all).not.toContain(PK.slice(2)); + expect(all).not.toContain(creds.secret); + }); +}); + +describe('check-polymarket: proxy', () => { + it('follows Node: lowercase first, ?? semantics, http_proxy fallback, host:port only', () => { + expect(envProxy({ https_proxy: 'http://a:1', HTTPS_PROXY: 'http://b:2' })?.display).toBe('a:1'); + expect(envProxy({ https_proxy: '', HTTPS_PROXY: 'http://b:2' })).toBeNull(); + expect(envProxy({ HTTPS_PROXY: 'http://user:s3cret@proxy.example:3128' })?.display).toBe('proxy.example:3128'); + expect(envProxy({ http_proxy: 'http://10.0.0.1:8080' })?.display).toBe('10.0.0.1:8080'); + expect(envProxy({})).toBeNull(); + }); + + it('reports a proxy only when Node uses it for the Polymarket hosts', () => { + expect(effectiveProxy({ HTTPS_PROXY: 'http://p:1' }, true)).toBe('p:1'); + expect(effectiveProxy({ HTTPS_PROXY: 'http://p:1' }, false)).toBeNull(); + expect(effectiveProxy({ HTTPS_PROXY: 'http://p:1', NO_PROXY: '.polymarket.com' }, true)).toBeNull(); + expect(effectiveProxy({ HTTPS_PROXY: 'http://p:1', no_proxy: '', NO_PROXY: '*' }, true)).toBe('p:1'); + expect(bypassesProxy('clob.polymarket.com', { no_proxy: 'localhost, *.polymarket.com' })).toBe(true); + expect(bypassesProxy('clob.polymarket.com', { no_proxy: 'polymarket.co' })).toBe(false); + }); +});