Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Self-hosted trading lab for Polymarket (prediction markets) and Binance (crypto
```bash
npm run dev # server (node --watch + tsx, :3780) + Vite UI with /api proxy (http://localhost:3000)
npm run dev:server # server only
npm run cli -- <command> # migrate-key | gen-key | set-password | token create/list/revoke (add --prod for npm start paths)
npm run cli -- <command> # migrate-key | gen-key | set-password | token create/list/revoke | check-polymarket (add --prod for npm start paths)
npm run mcp:stdio:dev # MCP stdio proxy from source (BOTLAB_URL, BOTLAB_AGENT_TOKEN); mcp:stdio = built
npm run gen:sdk # regenerate packages/core/src/sdk/sdkDts.generated.ts after editing sdk/types.ts
```
Expand Down Expand Up @@ -97,7 +97,7 @@ packages/core/src/
agent/schedule.ts # agent schedule rules (pure): Europe/Dublin slots across DST, quiet hours, event cap/debounce
reconcile/ recorder/ backtest/ audit/ notify/ utils/
security/ # crypto (AES-256-GCM), masterKey (key check), adminAuth (password, sessions, tokens), migrateSecrets
ops/ # BackupJob (nightly SQLite .backup, 14 kept)
ops/ # BackupJob (nightly SQLite .backup, 14 kept); connectivityCheck (check-polymarket: geo vs bot blocking, injectable fetch)
sdk/ # types.ts (ctx API types), guide.ts (agent guide, MCP resource), dts.ts + sdkDts.generated.ts
# (botlab-sdk.d.ts, generated by scripts/gen-sdk-dts.ts; a test fails when stale)
runtime/typecheckScript.ts # TypeScript compiler API typecheck of a script against botlab-sdk.d.ts (validate_script);
Expand Down
48 changes: 47 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,52 @@ On macOS and Linux, `npm run service:install` prints a launchd / systemd unit wi

---

## Hosting on a VPS

Polymarket blocks some IPs: whole countries (**geo-block**, the Terms of Service restrict them) and, separately,
many datacenter ranges (**bot protection**: Cloudflare challenge pages, 403s, 429s, dropped connections — an
Oracle Cloud VM has been blocked this way). Test a candidate server **before** migrating:

1. Rent the smallest **hourly-billed** instance in the provider / region you want and install Node 22 and BotLab
(`npm install`; no database or key needed).
2. Run the check:

```bash
npm run cli -- check-polymarket # table + verdict; --json for machine-readable output
npm run cli -- check-polymarket --binance # also Binance's public API (451 = restricted location)
```

It calls, read-only, every endpoint BotLab uses: Cloudflare's trace (public IP and country), Polymarket's
geoblock endpoint, the CLOB (`/time` with clock skew, `/markets/{id}`, `/book`), Gamma (`/markets`, `/events`)
and the data API (`/positions`, `/activity`), 10 s timeout each. Each row is `ok`, `geo_blocked`,
`bot_challenge`, `forbidden`, `rate_limited`, `network_error` or `unexpected`, and the last line is the verdict:
`OK: this IP can use Polymarket` (exit 0), `BLOCKED (geo): …` or `BLOCKED (datacenter/bot protection): …`
(exit 1), `DEGRADED: …` (exit 2: rate limits, partial failures, clock skew, Binance restricted).
3. **Keep the instance only if the verdict is OK**; otherwise destroy it and try another provider or region.
After migrating, re-run the check now and then (e.g. weekly, or when orders start failing): ranges get
re-classified. With a configured wallet, `npm run cli -- check-polymarket --prod --with-credentials` also makes
authenticated read-only CLOB calls (derive the existing API key — it never creates one —, list API keys,
close-only status). It opens the database read-only, so it may run next to the server; keys are never printed.

- **Geo vs datacenter**: a geo-block is about the country — another provider in the same country will not help,
only a server in an allowed country will. A datacenter / bot block is about the IP range — another provider
or region (or a residential connection) can work.
- **Do not use a VPN or proxy to get around a geo restriction**: it breaks Polymarket's Terms of Service and can
get the account restricted to close-only or closed. (`HTTPS_PROXY` is honoured by the check, but then it
tests the proxy's IP and says so.)
- **Expose BotLab over Tailscale only**: bind it to the Tailscale address (`BOTLAB_HOST=100.x.y.z`; or keep
127.0.0.1 behind `tailscale serve`), never `0.0.0.0`; allow no inbound port in the provider's firewall (SSH over
Tailscale too, if you can), and open the UI at `http://<machine>.<tailnet>.ts.net:3780` (`*.ts.net` names are
allowed by default).
- **Back up off the box**: the nightly backups land in `<data dir>/backups` on the same disk. Copy them elsewhere
(restic / rclone / rsync to another machine or bucket) and keep the **master key** in a separate place
(password manager): a backup without it cannot decrypt the wallet keys.
- A **home connection** (the Windows service above, a Mac mini, a Raspberry-class Linux box) is the most reliable
fallback: residential IPs are rarely filtered, and a UPS plus the halted restart after a power cut cover the
rest.

---

## Configuration

| variable | default | |
Expand Down Expand Up @@ -616,7 +662,7 @@ npm start # built server serving the built UI (production paths)
npm run typecheck # core + server + web
npm test # vitest (SKIP_NETWORK_TESTS=1 skips live-API tests)
npm run lint # ESLint (0 errors expected)
npm run cli -- <cmd> # migrate-key | gen-key | set-password | token create/list/revoke
npm run cli -- <cmd> # migrate-key | gen-key | set-password | token create/list/revoke | check-polymarket
npm run gen:sdk # regenerate botlab-sdk.d.ts (packages/core/src/sdk/sdkDts.generated.ts) after editing sdk/types.ts
npm run mcp:stdio # MCP stdio proxy (BOTLAB_URL, BOTLAB_AGENT_TOKEN); mcp:stdio:dev runs it from source
```
Expand Down
219 changes: 219 additions & 0 deletions apps/server/src/checkPolymarket.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
/**
* `npm run cli -- check-polymarket [--prod] [--json] [--with-credentials] [--binance] [--wallet <id|name|address>]
* [--address <0x…>]`: can this machine / IP use the Polymarket endpoints BotLab depends on? The checks live in
* core (`ops/connectivityCheck.ts`); this file is the CLI glue: proxy, database, master key, wallet.
*
* - Read-only throughout: GET requests only, no order placed or cancelled, no API key created.
* - The database is opened **read-only** (no migrations, no key-check write), so the check is allowed while the
* server runs on the same data dir (SQLite WAL readers do not block it). Without a database the defaults are used.
* - `--with-credentials`: the master key the server would use (`BOTLAB_MASTER_KEY`, `_FILE`, `<data dir>/master.key`;
* never generated here), checked against `settings.key_check`, decrypts the wallet's private key in this process
* only; the CLOB probes sign with it and print no key, secret or passphrase.
* - `HTTPS_PROXY` / `HTTP_PROXY`: Node 22's fetch ignores them unless `NODE_USE_ENV_PROXY=1` (Node ≥ 22.21), so the
* command re-runs itself with it; the report then says the results reflect the proxy's IP.
*/

import Database from 'better-sqlite3';
import { spawnSync } from 'child_process';
import fs from 'fs';
import path from 'path';
import { decrypt } from '../../../packages/core/src/security/crypto';
import { keyCheckMatches } from '../../../packages/core/src/security/masterKey';
import { createClobReadOnlyAuth } from '../../../packages/core/src/exchange/polymarket/authHeaders';
import { DEFAULT_CLOB_URL, formatReport, runConnectivityCheck, type ConnectivityOptions } from '../../../packages/core/src/ops/connectivityCheck';
import { defaultKeyFile, loadMasterKey } from './masterKey';

type Flags = Record<string, string | true>;

/**
* The proxy Node's fetch uses for https URLs with `NODE_USE_ENV_PROXY` (undici's EnvHttpProxyAgent): lowercase
* first, `??` semantics (a variable set to "" means "no proxy" and stops the lookup), `http_proxy` as the
* fallback. Host:port only (credentials never shown); null when there is none.
*/
export function envProxy(env: NodeJS.ProcessEnv = process.env): { raw: string; display: string } | null {
const raw = env.https_proxy ?? env.HTTPS_PROXY ?? env.http_proxy ?? env.HTTP_PROXY;
if (!raw) return null;
try {
const u = new URL(raw);
return { raw, display: `${u.hostname}${u.port ? `:${u.port}` : ''}` };
} catch {
return { raw, display: '(unparseable proxy URL)' };
}
}

/** Whether `no_proxy` / `NO_PROXY` (same lookup order) exempts `host` from the proxy */
export function bypassesProxy(host: string, env: NodeJS.ProcessEnv = process.env): boolean {
const list = (env.no_proxy ?? env.NO_PROXY ?? '')
.split(/[\s,]+/)
.map((e) => e.trim().toLowerCase())
.filter(Boolean);
const h = host.toLowerCase();
return list.some((e) => {
if (e === '*') return true;
const name = e.replace(/:\d+$/, '').replace(/^\*?\./, '');
return h === name || h.endsWith(`.${name}`);
});
}

/** The proxy the check's requests will actually go through (env proxy honoured by fetch and not bypassed) */
export function effectiveProxy(env: NodeJS.ProcessEnv = process.env, active: boolean = envProxyActive(env), host = 'clob.polymarket.com'): string | null {
const p = envProxy(env);
return p && active && !bypassesProxy(host, env) ? p.display : null;
}

function nodeSupportsEnvProxy(version: string = process.versions.node): boolean {
const [major, minor] = version.split('.').map(Number);
return major > 22 || (major === 22 && minor >= 21);
}

function envProxyActive(env: NodeJS.ProcessEnv = process.env): boolean {
return env.NODE_USE_ENV_PROXY === '1' || process.execArgv.includes('--use-env-proxy');
}

/**
* When a proxy is configured but fetch would ignore it, re-run this command with `NODE_USE_ENV_PROXY=1` and
* return its exit code; null to go on in this process.
*/
function reexecWithProxy(): number | null {
if (!envProxy() || envProxyActive() || !nodeSupportsEnvProxy()) return null;
// NODE_OPTIONS (not argv) also reaches a loader's own child process; the warning is Node's "experimental" notice
const nodeOptions = [process.env.NODE_OPTIONS, '--disable-warning=UNDICI-EHPA'].filter(Boolean).join(' ');
const r = spawnSync(process.execPath, [...process.execArgv, ...process.argv.slice(1)], {
stdio: 'inherit',
env: { ...process.env, NODE_USE_ENV_PROXY: '1', NODE_OPTIONS: nodeOptions },
});
if (r.error) throw r.error;
return r.status ?? 1;
}

interface WalletRow {
id: string;
name: string;
address: string;
proxy_address: string | null;
private_key_encrypted: string;
}

/**
* Opens the database read-only (no migrations, no writes; allowed next to a running server). On a WAL database
* SQLite may still create empty `-wal` / `-shm` files next to it. `error` carries the real reason (permissions, …).
*/
export function openReadOnly(dbFile: string): { db: Database.Database } | { db: null; error: string } {
if (!fs.existsSync(dbFile)) return { db: null, error: `no BotLab database at ${dbFile} (use --data-dir, or --prod for the npm start location)` };
try {
return { db: new Database(dbFile, { readonly: true, fileMustExist: true }) };
} catch (err) {
return { db: null, error: `cannot open ${dbFile} read-only: ${err instanceof Error ? err.message : String(err)}` };
}
}

function settingsClobUrl(db: Database.Database | null): string | null {
if (!db) return null;
try {
const row = db.prepare("SELECT clob_http_url FROM settings WHERE id = 'global'").get() as { clob_http_url?: string } | undefined;
return row?.clob_http_url && /^https?:\/\//.test(row.clob_http_url) ? row.clob_http_url : null;
} catch {
return null;
}
}

function pickWallet(db: Database.Database, which: string | undefined): { wallet: WalletRow; note: string | null } | string {
const rows = db.prepare('SELECT id, name, address, proxy_address, private_key_encrypted FROM wallets ORDER BY created_at').all() as WalletRow[];
if (rows.length === 0) return 'no wallet is configured in BotLab (add one in Wallets first)';
if (which) {
const w = rows.find(
(r) => r.id === which || r.name === which || r.address.toLowerCase() === which.toLowerCase() || r.proxy_address?.toLowerCase() === which.toLowerCase()
);
return w ? { wallet: w, note: null } : `no wallet matches --wallet ${which}`;
}
return {
wallet: rows[0],
note: rows.length > 1 ? `${rows.length} wallets configured; using "${rows[0].name}" (choose with --wallet <id|name|address>)` : null,
};
}

/** The read-only auth probe, or why it is skipped. Never throws for a missing key / wallet. */
export function buildAuth(
opened: ReturnType<typeof openReadOnly>,
dataDir: string,
flags: Flags,
env: NodeJS.ProcessEnv = process.env
): { auth: ConnectivityOptions['auth']; address?: string; notes: string[] } {
const notes: string[] = [];
if (!opened.db) return { auth: { skip: opened.error }, notes };
const db = opened.db;
const hasKey = !!env.BOTLAB_MASTER_KEY || !!env.BOTLAB_MASTER_KEY_FILE || fs.existsSync(defaultKeyFile(dataDir));
if (!hasKey) return { auth: { skip: 'no master key (BOTLAB_MASTER_KEY, BOTLAB_MASTER_KEY_FILE or <data dir>/master.key)' }, notes };
let key: string;
try {
// production: true ⇒ never generates a key file here
key = loadMasterKey({ env, dataDir, production: true, log: () => undefined }).key;
} catch (err) {
return { auth: { skip: `master key: ${err instanceof Error ? err.message : String(err)}` }, notes };
}
let picked: ReturnType<typeof pickWallet>;
try {
const row = db.prepare("SELECT key_check FROM settings WHERE id = 'global'").get() as { key_check: string | null } | undefined;
if (row?.key_check && !keyCheckMatches(row.key_check, key)) return { auth: { skip: 'the master key does not match this database' }, notes };
picked = pickWallet(db, typeof flags.wallet === 'string' ? flags.wallet : undefined);
} catch (err) {
return { auth: { skip: `cannot read the database: ${err instanceof Error ? err.message : String(err)}` }, notes };
}
if (typeof picked === 'string') return { auth: { skip: picked }, notes };
if (picked.note) notes.push(picked.note);
let privateKey: string;
try {
privateKey = decrypt(picked.wallet.private_key_encrypted, key);
} catch {
return { auth: { skip: `cannot decrypt the private key of wallet "${picked.wallet.name}" with this master key` }, notes };
}
return {
auth: createClobReadOnlyAuth(privateKey, `wallet "${picked.wallet.name}" ${picked.wallet.address}`),
address: picked.wallet.proxy_address || picked.wallet.address,
notes,
};
}

/** Runs the check; returns the exit code (0 ok, 1 blocked, 2 degraded). */
export async function checkPolymarketCommand(flags: Flags, dataDir: string): Promise<number> {
const reexec = reexecWithProxy();
if (reexec !== null) return reexec;

const json = flags.json === true;
const notes: string[] = [];
if (envProxy() && !envProxyActive()) {
notes.push(
`A proxy is set (https_proxy / HTTPS_PROXY) but Node ${process.versions.node} cannot use it from fetch (needs ≥ 22.21): requests go out directly from this machine`
);
}

const opened = openReadOnly(path.join(dataDir, 'botlab.db'));
const db = opened.db;
try {
const opts: ConnectivityOptions = {
clobUrl: settingsClobUrl(db) ?? DEFAULT_CLOB_URL,
binance: flags.binance === true,
proxy: effectiveProxy(),
};
if (flags['with-credentials']) {
const a = buildAuth(opened, dataDir, flags);
opts.auth = a.auth;
if (a.address) opts.sampleAddress = a.address;
notes.push(...a.notes);
}
if (typeof flags.address === 'string') {
if (!/^0x[0-9a-fA-F]{40}$/.test(flags.address)) throw new Error('--address must be a 0x address');
opts.sampleAddress = flags.address;
}
if (!json) console.log('Checking Polymarket endpoints (read-only, ~10 s timeout each)…\n');
const report = await runConnectivityCheck(opts);
if (json) console.log(JSON.stringify({ ...report, notes }, null, 2));
else {
console.log(formatReport(report));
for (const n of notes) console.log(`Note: ${n}`);
}
return report.verdict.exitCode;
} finally {
db?.close();
}
}
13 changes: 12 additions & 1 deletion apps/server/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@
* token create --scope agent|readonly [--name <name>]
* token list
* token revoke <id>
* check-polymarket [--json] [--with-credentials] [--binance] [--wallet <id|name|address>] [--address <0x…>]
* can this machine / IP use Polymarket? Read-only checks of every endpoint
* BotLab uses (geo-block vs datacenter / bot blocking); exit 0 ok, 1 blocked,
* 2 degraded (checkPolymarket.ts)
*
* migrate-key refuses to run while the server is running on the same data dir (it rewrites every
* secret); the other commands are safe while it runs. migrate-key uses the master key the server would
Expand All @@ -28,6 +32,7 @@ import { isProduction, resolveDataDir } from './paths';
import { defaultKeyFile, generateKeyFile, loadMasterKey, MasterKeyError } from './masterKey';
import { lockHolder } from './instanceLock';
import { migrateKey, needsNewAdminPassword, needsPassword } from './migrateKey';
import { checkPolymarketCommand } from './checkPolymarket';
import fs from 'fs';
import path from 'path';

Expand Down Expand Up @@ -80,7 +85,8 @@ function usage(): never {
set-password [--password <pw>]
token create --scope agent|readonly [--name <name>]
token list
token revoke <id>`);
token revoke <id>
check-polymarket [--json] [--with-credentials] [--binance] [--wallet <id|name|address>] [--address <0x…>]`);
process.exit(2);
}

Expand All @@ -103,6 +109,11 @@ async function main(): Promise<void> {
return;
}

if (cmd === 'check-polymarket') {
process.exitCode = await checkPolymarketCommand(flags, dataDir);
return;
}

const dbFile = path.join(dataDir, 'botlab.db');
if (!fs.existsSync(dbFile) && cmd !== 'set-password') {
throw new Error(`No database at ${dbFile} (use --data-dir, or --prod for the npm start / service location)`);
Expand Down
Loading
Loading