diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml new file mode 100644 index 0000000..498f779 --- /dev/null +++ b/.github/workflows/validate.yml @@ -0,0 +1,51 @@ +name: Validate Smart Web Fetch + +on: + pull_request: + push: + branches: + - main + +jobs: + offline-regression: + name: Offline Regression (Ubuntu) + runs-on: ubuntu-latest + + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Run offline regression + shell: bash + run: | + set -euo pipefail + bash spec/tests/offline-regression.sh + + - name: Run Bash JSON smoke tests + shell: bash + run: | + set -euo pipefail + bash spec/tests/json-smoke.sh + + powershell-smoke: + name: PowerShell Smoke Tests (Windows) + runs-on: windows-latest + + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Run PowerShell JSON smoke tests + shell: pwsh + run: | + .\spec\tests\json-smoke.ps1 diff --git a/.gitignore b/.gitignore index 49d4fb8..909b939 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,11 @@ Desktop.ini *.temp *.log +# Python cache / bytecode +__pycache__/ +*.py[cod] +.pytest_cache/ + # Generated output files output.md article.md diff --git a/README.md b/README.md index 1a1a3be..afd5d81 100644 --- a/README.md +++ b/README.md @@ -28,56 +28,45 @@ ```bash chmod +x ./scripts/smart-web-fetch -chmod +x ./scripts/smart-web-fetch-core ``` ### 依赖 -> Bash core 采用严格模式（`set -euo pipefail`）执行；参数缺值等边界情况会直接报错退出。 - -| 运行入口 | 依赖项 | 类型 | 启动时行为 | -| --- | --- | --- | --- | -| `scripts/smart-web-fetch`（Bash） | `curl` | 必需 | 缺失时自动切换到 PowerShell 路径；若 `pwsh` 也不存在则报错退出 | -| `scripts/smart-web-fetch`（Bash） | `jq` | 可选 | verbose 模式下提示缺失，继续执行并回退内置 JSON 解析 | -| `scripts/smart-web-fetch`（Bash） | `perl` | 可选 | verbose 模式下提示缺失，继续执行并回退 awk 轻量 HTML 清洗 | -| `scripts/smart-web-fetch`（Bash） | `html2text` / `lynx` | 可选 | 两者都缺失时仅提示，继续执行并输出清洗后的 HTML | -| `scripts/smart-web-fetch.ps1`（PowerShell） | PowerShell 7+ | 必需 | 版本不足时立即报错并退出 | -| `scripts/smart-web-fetch.ps1`（PowerShell） | `Invoke-WebRequest` | 必需 | 不可用时立即报错并退出 | -| `scripts/smart-web-fetch.ps1`（PowerShell） | `jq` | 可选 | verbose 模式下提示缺失，继续执行并回退 `ConvertFrom-Json` | -| `scripts/smart-web-fetch.ps1`（PowerShell） | `perl` | 可选 | verbose 模式下提示缺失，继续执行并回退 PowerShell 正则清洗 | -| `scripts/smart-web-fetch.ps1`（PowerShell） | `html2text` / `lynx` | 可选 | 两者都缺失时仅提示，继续执行并输出清洗后的 HTML | +- 统一运行时基线为 `Python 3.11+` +- 默认只依赖 Python 标准库 +- `core/` 是内部实现包；对外包装器会执行技能根目录下确定的 `main.py` bootstrap，再由它引导 `core.cli:main` +- 不再依赖 `curl` / `pwsh` / `jq` / `perl` / `html2text` / `lynx` ### 常见安装命令 #### macOS ```bash -brew install jq html2text lynx perl +brew install python ``` #### Debian / Ubuntu ```bash -sudo apt-get update && sudo apt-get install -y curl jq html2text lynx perl +sudo apt-get update && sudo apt-get install -y python3 ``` #### Fedora / RHEL / CentOS Stream ```bash -sudo dnf install -y curl jq html2text lynx perl +sudo dnf install -y python3 ``` #### Arch Linux ```bash -sudo pacman -S --needed curl jq html2text lynx perl +sudo pacman -S --needed python ``` #### Windows ```powershell -winget install Microsoft.PowerShell jqlang.jq StrawberryPerl.StrawberryPerl lynx.portable -py -m pip install html2text +winget install Python.Python.3.11 ``` ## 🚀 快速开始 @@ -90,21 +79,40 @@ py -m pip install html2text | Windows CMD / 原生 PowerShell | `.\scripts\smart-web-fetch ` | | PowerShell 7（显式调用） | `pwsh -File .\scripts\smart-web-fetch.ps1 ` | -入口脚本自动检测运行时：有 `curl` 时走 Bash 路径，否则回退 PowerShell 7。Windows 下 `.cmd` 文件直接调用 PowerShell 包装器。 +### 按场景使用 + +直接查看正文： + +```bash +./scripts/smart-web-fetch https://example.com +``` -### 常用示例 +保存到文件： ```bash ./scripts/smart-web-fetch https://example.com -o article.md +``` + +返回结构化 JSON： + +```bash +./scripts/smart-web-fetch https://example.com --json +``` + +指定服务源或调试抓取过程： + +```bash ./scripts/smart-web-fetch https://example.com -s jina ./scripts/smart-web-fetch https://example.com -v ./scripts/smart-web-fetch https://example.com --no-clean ``` -Windows CMD / 原生 PowerShell： +Windows CMD / 原生 PowerShell 示例： ```cmd +.\scripts\smart-web-fetch https://example.com .\scripts\smart-web-fetch https://example.com -o article.md +.\scripts\smart-web-fetch https://example.com --json .\scripts\smart-web-fetch https://example.com -s jina .\scripts\smart-web-fetch https://example.com -v .\scripts\smart-web-fetch https://example.com --no-clean @@ -112,21 +120,34 @@ Windows CMD / 原生 PowerShell： ## ⚙️ 参数一览 -| 功能 | 参数 | -| --- | --- | -| 显示帮助 | `-h` / `--help` | -| 输出到文件 | `-o ` / `--output ` | -| 指定服务源 | `-s ` / `--service ` | -| 显示详细日志 | `-v` / `--verbose` | -| 跳过 HTML 清洗 | `--no-clean` | +| 参数 | 说明 | 备注 | +| --- | --- | --- | +| `-h`, `--help` | 显示帮助 | 仅打印帮助并退出 | +| `-o `, `--output ` | 写入输出文件 | 默认写正文；配合 `--json` 时写结构化结果 | +| `-s `, `--service ` | 强制指定服务源 | 可选值：`jina`、`markdown`、`defuddle` | +| `--json` | 返回结构化结果 | 适合脚本、自动化和 agent 消费 | +| `-v`, `--verbose` | 显示详细日志 | 日志输出到 stderr | +| `--no-clean` | 跳过 basic fallback 的 HTML 清洗 | 只影响 direct/basic fallback 路径 | + +仅支持 `-s` / `--service`；`-Service` 和其他未知参数会直接报错。 + +## 🧾 输出方式 + +默认情况下，命令会直接输出抓取到的正文内容，适合在终端中阅读，或通过重定向保存到文件。 -`-s` / `--service` 可选值：`jina`、`markdown`、`defuddle`。未知参数直接报错。 +如果需要把结果交给脚本、自动化流程或 agent 继续处理，可以加上 `--json`。此时命令会返回结构化结果，例如： + +```json +{"success":true,"url":"https://example.com","content":"...","source":"jina"} +``` + +在 `--json` 模式下，失败时也会返回 JSON，并附带错误信息。`source` 会标明实际命中的来源。`--json` 也可以与 `-o` / `--output` 组合使用，此时文件会写入当前模式下的最终输出。 ## 🔄 抓取策略 默认按以下顺序依次尝试，前一个失败才进入下一个；全部失败时以非零状态退出并报告最后一次失败原因。显式指定服务源（`-s` / `--service`）后只尝试该源，失败直接报错。 -| # | 服务源 | 方式 | 请求端点 | +| 优先级 | 服务源 | 方式 | 请求端点 | | :---: | --- | :---: | --- | | 1 | Jina Reader | GET | `r.jina.ai/` | | 2 | markdown.new | POST | `api.markdown.new/api/v1/convert` | @@ -135,22 +156,11 @@ Windows CMD / 原生 PowerShell： 仓库中的详细判定规则见 [`spec/fetch-contract.md`](spec/fetch-contract.md)。 -## 📁 仓库目录结构 - -```text -smart-web-fetch/ -├── skills/ -│ └── smart-web-fetch/ -│ ├── SKILL.md -│ ├── assets/ -│ └── scripts/ -├── spec/ -│ ├── fetch-contract.md -│ └── fixtures/ -├── README.md -├── README_EN.md -└── LICENSE -``` +补充说明： + +- 未带 scheme 的 URL 会自动补成 `https://` +- 仅允许 `http://` / `https://`；如 `ftp://` 会直接失败 +- `basic fallback` 遇到图片、PDF、压缩包等二进制响应会直接失败，不返回乱码文本 ## License diff --git a/README_EN.md b/README_EN.md index 0705aa0..e4c754d 100644 --- a/README_EN.md +++ b/README_EN.md @@ -26,56 +26,45 @@ If you cloned the full repository, run `cd skills/smart-web-fetch` first. ```bash chmod +x ./scripts/smart-web-fetch -chmod +x ./scripts/smart-web-fetch-core ``` ### Dependencies -> The Bash core runs in strict mode (`set -euo pipefail`); missing option values and similar edge cases exit explicitly. - -| Entry Point | Dependency | Type | Startup Behavior | -| --- | --- | --- | --- | -| `scripts/smart-web-fetch` (Bash) | `curl` | Required | Falls back to PowerShell when missing; exits with an error if `pwsh` is also unavailable | -| `scripts/smart-web-fetch` (Bash) | `jq` | Optional | Warns in verbose mode and falls back to built-in JSON parsing | -| `scripts/smart-web-fetch` (Bash) | `perl` | Optional | Warns in verbose mode and falls back to awk-based HTML cleanup | -| `scripts/smart-web-fetch` (Bash) | `html2text` / `lynx` | Optional | Warns when both are missing and returns cleaned HTML instead of plain text | -| `scripts/smart-web-fetch.ps1` (PowerShell) | PowerShell 7+ | Required | Exits immediately when the runtime version is insufficient | -| `scripts/smart-web-fetch.ps1` (PowerShell) | `Invoke-WebRequest` | Required | Exits immediately when unavailable | -| `scripts/smart-web-fetch.ps1` (PowerShell) | `jq` | Optional | Warns in verbose mode and falls back to `ConvertFrom-Json` | -| `scripts/smart-web-fetch.ps1` (PowerShell) | `perl` | Optional | Warns in verbose mode and falls back to PowerShell regex cleanup | -| `scripts/smart-web-fetch.ps1` (PowerShell) | `html2text` / `lynx` | Optional | Warns when both are missing and returns cleaned HTML instead of plain text | +- The only runtime baseline is `Python 3.11+` +- Only Python standard-library modules are required at runtime +- `core/` is the internal implementation package; the public wrappers execute the deterministic `main.py` bootstrap in the skill root, which then dispatches to `core.cli:main` +- `curl`, `pwsh`, `jq`, `perl`, `html2text`, and `lynx` are no longer runtime dependencies ### Common install commands #### macOS ```bash -brew install jq html2text lynx perl +brew install python ``` #### Debian / Ubuntu ```bash -sudo apt-get update && sudo apt-get install -y curl jq html2text lynx perl +sudo apt-get update && sudo apt-get install -y python3 ``` #### Fedora / RHEL / CentOS Stream ```bash -sudo dnf install -y curl jq html2text lynx perl +sudo dnf install -y python3 ``` #### Arch Linux ```bash -sudo pacman -S --needed curl jq html2text lynx perl +sudo pacman -S --needed python ``` #### Windows ```powershell -winget install Microsoft.PowerShell jqlang.jq StrawberryPerl.StrawberryPerl lynx.portable -py -m pip install html2text +winget install Python.Python.3.11 ``` ## 🚀 Quick Start @@ -88,21 +77,40 @@ All terminals use the same command name and argument interface. The examples bel | Windows CMD / native PowerShell | `.\scripts\smart-web-fetch ` | | PowerShell 7 (explicit invocation) | `pwsh -File .\scripts\smart-web-fetch.ps1 ` | -The entry script detects the runtime automatically: it prefers the Bash path when `curl` is available and otherwise falls back to PowerShell 7. On Windows, the `.cmd` wrapper calls the PowerShell entry point. +### Common usage patterns + +Read the page body directly: + +```bash +./scripts/smart-web-fetch https://example.com +``` -### Common examples +Save the result to a file: ```bash ./scripts/smart-web-fetch https://example.com -o article.md +``` + +Return structured JSON: + +```bash +./scripts/smart-web-fetch https://example.com --json +``` + +Force a provider or debug the fetch: + +```bash ./scripts/smart-web-fetch https://example.com -s jina ./scripts/smart-web-fetch https://example.com -v ./scripts/smart-web-fetch https://example.com --no-clean ``` -Windows CMD / native PowerShell: +Windows CMD / native PowerShell examples: ```cmd +.\scripts\smart-web-fetch https://example.com .\scripts\smart-web-fetch https://example.com -o article.md +.\scripts\smart-web-fetch https://example.com --json .\scripts\smart-web-fetch https://example.com -s jina .\scripts\smart-web-fetch https://example.com -v .\scripts\smart-web-fetch https://example.com --no-clean @@ -110,21 +118,34 @@ Windows CMD / native PowerShell: ## ⚙️ Arguments -| Function | Argument | -| --- | --- | -| Show help | `-h` / `--help` | -| Write output to file | `-o ` / `--output ` | -| Force a service | `-s ` / `--service ` | -| Enable verbose logs | `-v` / `--verbose` | -| Skip HTML cleanup | `--no-clean` | +| Argument | Purpose | Notes | +| --- | --- | --- | +| `-h`, `--help` | Show help | Prints help and exits | +| `-o `, `--output ` | Write to an output file | Writes body text by default; writes structured output when combined with `--json` | +| `-s `, `--service ` | Force a specific service | Valid values: `jina`, `markdown`, `defuddle` | +| `--json` | Return structured output | Useful for scripts, automation, and agents | +| `-v`, `--verbose` | Enable verbose logs | Logs go to stderr | +| `--no-clean` | Skip HTML cleanup in the basic fallback | Only affects the direct/basic fallback path | + +Only `-s` / `--service` are supported; `-Service` and other unknown arguments exit with an error. + +## 🧾 Output Modes + +By default, the command prints the fetched body content directly, which is convenient for terminal use or shell redirection. -Valid values for `-s` / `--service`: `jina`, `markdown`, `defuddle`. Unknown arguments exit with an error. +If the result should be consumed by scripts, automation, or agents, add `--json`. In that mode the CLI returns structured output, for example: + +```json +{"success":true,"url":"https://example.com","content":"...","source":"jina"} +``` + +In `--json` mode, failures also return JSON and include error details. `source` reflects the backend that actually produced the result. `--json` can also be combined with `-o` / `--output`, in which case the output file receives the final payload for the active mode. ## 🔄 Fetch Strategy By default, services are tried in order. The next provider is only attempted if the previous one fails. If all providers fail, the command exits with a non-zero status and reports the last failure reason. When a service is explicitly selected with `-s` / `--service`, only that service is attempted. -| # | Service | Method | Endpoint | +| Priority | Service | Method | Endpoint | | :---: | --- | :---: | --- | | 1 | Jina Reader | GET | `r.jina.ai/` | | 2 | markdown.new | POST | `api.markdown.new/api/v1/convert` | @@ -133,22 +154,11 @@ By default, services are tried in order. The next provider is only attempted if See [`spec/fetch-contract.md`](./spec/fetch-contract.md) in the repository for the detailed decision contract. -## 📁 Repository Layout - -```text -smart-web-fetch/ -├── skills/ -│ └── smart-web-fetch/ -│ ├── SKILL.md -│ ├── assets/ -│ └── scripts/ -├── spec/ -│ ├── fetch-contract.md -│ └── fixtures/ -├── README.md -├── README_EN.md -└── LICENSE -``` +Additional notes: + +- URLs without a scheme are normalized to `https://` +- Only `http://` and `https://` are allowed; schemes such as `ftp://` fail fast +- The basic fallback rejects binary responses such as images, PDFs, and archive payloads instead of returning garbled text ## License diff --git a/skills/smart-web-fetch/SKILL.md b/skills/smart-web-fetch/SKILL.md index b04d0a5..9ae9afa 100644 --- a/skills/smart-web-fetch/SKILL.md +++ b/skills/smart-web-fetch/SKILL.md @@ -1,12 +1,12 @@ --- name: smart-web-fetch description: Fetch web pages and article-like URLs as clean Markdown with automatic fallback across Jina Reader, markdown.new, defuddle.md, and a direct fetch path. Use when an agent needs to read a URL, extract main content, reduce HTML noise, save fetched output, or convert webpage content into token-efficient text. -compatibility: Requires network access. Bash usage requires curl; if curl is absent, the entry point automatically falls back to PowerShell 7 with Invoke-WebRequest. +compatibility: Requires network access and Python 3.11+. --- # Smart Web Fetch -Use the bundled scripts to retrieve a URL as clean Markdown or text. +Use the bundled scripts to retrieve a URL as clean Markdown/text or as structured JSON for scripts and agents. ## When To Use @@ -18,8 +18,6 @@ Use the bundled scripts to retrieve a URL as clean Markdown or text. This skill is intended to be distributed as a standalone `smart-web-fetch/` directory or zip. The commands below assume you are running from inside that extracted directory. -所有终端使用相同的参数接口： - ### Bash / Unix-like systems ```bash @@ -35,12 +33,13 @@ This skill is intended to be distributed as a standalone `smart-web-fetch/` dire ### PowerShell 7（显式调用） ```powershell -./scripts/smart-web-fetch.ps1 +pwsh -File .\scripts\smart-web-fetch.ps1 ``` ## Preferred options - Use `-s jina` when you want the most stable cleaned result. +- Use `--json` when another tool or agent should consume structured output. - Use `-o ` when the fetched content should be reused later. - Use `-v` when debugging a failed fetch. - Use `--no-clean` only if you want the basic fallback to keep rawer HTML. @@ -56,30 +55,20 @@ The tool tries services in this order unless one is explicitly forced: When a service is explicitly forced via `-s`/`--service`, the CLI only attempts that service. If it fails, the command exits with an error instead of continuing to other services. -The entry point detects `curl` first; if present it routes to the Bash core. Otherwise it falls back to PowerShell 7. When routing to PowerShell, POSIX-style flags (`--no-clean`, `--verbose`, etc.) are translated automatically. - The clean-skip flag only changes the basic fallback path. External service output is passed through unchanged. -The runtime rules file is `assets/fetch-rules.json`. The scripts load it when present and fall back to built-in defaults if it is missing or cannot be parsed. +Only `-s` / `--service` are supported for service selection; `-Service` is not supported. URLs without a scheme are normalized to `https://`, but non-HTTP(S) schemes fail fast. The basic fallback rejects binary responses instead of returning garbled text. -## Requirements +Default mode prints only the fetched body. `--json` prints a single JSON object with `success`, `url`, `content`, and `source`; failures also include `error` and still exit non-zero. `source` resolves to the actual winning backend: `jina`, `markdown`, `defuddle`, `basic`, or `none`. -- `curl` **or** PowerShell 7 with `Invoke-WebRequest` (at least one required) -- Prefer `jq` for JSON parsing (Bash path) -- Prefer `html2text` or `lynx` for fallback HTML conversion -- Prefer `perl` for stronger fallback HTML cleanup when available - -## Files +## Requirements -- `scripts/smart-web-fetch`: 统一入口（Bash / Git Bash）。检测运行时并路由到对应 core。 -- `scripts/smart-web-fetch.ps1`: 统一入口（PowerShell 7）。接受 POSIX 风格参数并转发给 core。 -- `scripts/smart-web-fetch.cmd`: 统一入口（Windows CMD / 原生 PowerShell）。调用 PS1 包装器。 -- `scripts/smart-web-fetch-core`: Bash 核心实现。不直接调用。 -- `scripts/smart-web-fetch-core.ps1`: PowerShell 核心实现。不直接调用。 -- `assets/fetch-rules.json`: Runtime thresholds and keyword rules. +- Python 3.11+ +- No third-party runtime dependency; the bundled `core/` package only relies on Python standard-library modules +- `core/` is internal only; the shipped wrappers execute the skill-root `main.py` bootstrap and keep the public command surface unchanged -## Packaging +## Notes -- GitHub Actions builds `smart-web-fetch.zip` as the official distributable artifact. -- The zip root is `smart-web-fetch/` and contains only `SKILL.md`, `assets/`, and `scripts/`. -- Repository-only files such as `spec/` are not included in the packaged skill. +- `-o ` writes the final output for the current mode, including JSON mode. +- `--no-clean` only affects the direct/basic fallback path. +- Unknown arguments or fetch failures exit non-zero. diff --git a/skills/smart-web-fetch/core/__init__.py b/skills/smart-web-fetch/core/__init__.py new file mode 100644 index 0000000..23b983d --- /dev/null +++ b/skills/smart-web-fetch/core/__init__.py @@ -0,0 +1,84 @@ +from .cli import Config, ERROR_PREFIX, INTERPRETER_ERROR, VALID_SERVICES, main, normalize_url, parse_args, show_help +from .errors import CLIError, FetchTransportError +from .extract import HTMLTextExtractor, clean_html, html_to_text, looks_like_html +from .output import emit_json_failure, inspect_cli_mode, log, output_error_message, render_payload, write_output +from .rules import DEFAULT_RULES, RULES_FILE, Rules, contains_keyword, load_rules, normalize_keyword_text +from .sources import ( + DEFUDDLE_URL, + JINA_READER_BASE, + MARKDOWN_NEW_URL, + FetchResult, + build_jina_url, + extract_markdown_field, + fetch_basic, + fetch_defuddle, + fetch_jina, + fetch_markdown_new, + is_likely_html_error_payload, + is_structured_error_response, + parse_json_object, + run_fetch, +) +from .transport import ( + TIMEOUT_SECONDS, + ResponseData, + decode_response_body, + get_media_type, + has_binary_body_signature, + is_binary_media_type, + is_binary_response, + is_text_media_type, + request_text, +) + +__all__ = [ + "CLIError", + "Config", + "DEFAULT_RULES", + "DEFUDDLE_URL", + "ERROR_PREFIX", + "FetchResult", + "FetchTransportError", + "HTMLTextExtractor", + "INTERPRETER_ERROR", + "JINA_READER_BASE", + "MARKDOWN_NEW_URL", + "RULES_FILE", + "Rules", + "ResponseData", + "TIMEOUT_SECONDS", + "VALID_SERVICES", + "build_jina_url", + "clean_html", + "contains_keyword", + "decode_response_body", + "emit_json_failure", + "extract_markdown_field", + "fetch_basic", + "fetch_defuddle", + "fetch_jina", + "fetch_markdown_new", + "get_media_type", + "has_binary_body_signature", + "html_to_text", + "inspect_cli_mode", + "is_binary_media_type", + "is_binary_response", + "is_likely_html_error_payload", + "is_structured_error_response", + "is_text_media_type", + "load_rules", + "log", + "looks_like_html", + "main", + "normalize_keyword_text", + "normalize_url", + "output_error_message", + "parse_args", + "parse_json_object", + "render_payload", + "request_text", + "run_fetch", + "show_help", + "write_output", +] diff --git a/skills/smart-web-fetch/core/cli.py b/skills/smart-web-fetch/core/cli.py new file mode 100644 index 0000000..d500c29 --- /dev/null +++ b/skills/smart-web-fetch/core/cli.py @@ -0,0 +1,256 @@ +from __future__ import annotations + +import re +import sys +from dataclasses import dataclass +from urllib import parse + +from .errors import CLIError +from .output import emit_json_failure, inspect_cli_mode, output_error_message, render_payload, write_output +from .rules import load_rules +from .sources import run_fetch + + +VALID_SERVICES = {"jina", "markdown", "defuddle"} +ERROR_PREFIX = "smart-web-fetch: error: " +INTERPRETER_ERROR = ( + "smart-web-fetch: error: Python 3.11+ was not found. " + "Install Python 3.11 or newer and ensure a compatible interpreter is on PATH." +) + +_IPV4_RE = re.compile(r"^(?:25[0-5]|2[0-4]\d|1?\d?\d)(?:\.(?:25[0-5]|2[0-4]\d|1?\d?\d)){3}$") +_HOST_WITH_PORT_RE = re.compile(r"^(?P.+?)(?::(?P\d+))?$") +_HOST_LABEL_RE = re.compile(r"^(?!-)[A-Za-z0-9-]{1,63}(? str: + return """Smart Web Fetch - lightweight web-to-Markdown fetcher + +Usage: + smart-web-fetch [options] + +Options: + -h, --help Show help + -o, --output FILE Write output to file + -s, --service NAME Force service: jina|markdown|defuddle + --json Emit a structured JSON object + -v, --verbose Show verbose logs + --no-clean Skip HTML cleanup in the basic fallback + +Examples: + smart-web-fetch https://example.com + smart-web-fetch https://example.com -o output.md + smart-web-fetch https://example.com -s jina + smart-web-fetch https://example.com --json + smart-web-fetch https://example.com --no-clean + +Fallback order: + 1. Jina Reader + 2. markdown.new + 3. defuddle.md + 4. direct/basic fallback +""" + + +def normalize_url(url: str) -> str: + if not url: + raise CLIError("Please provide a URL") + candidate = url.strip() + if not candidate: + raise CLIError("Please provide a URL") + + explicit_scheme = "://" in candidate + if explicit_scheme: + parsed = _split_url_or_raise(candidate) + elif _is_host_like_input(candidate): + candidate = f"https://{candidate}" + parsed = _split_url_or_raise(candidate) + else: + parsed = _split_url_or_raise(candidate) + + scheme = parsed.scheme.lower() + if explicit_scheme or parsed.scheme: + if scheme not in {"http", "https"}: + raise CLIError(f"Unsupported URL scheme: {parsed.scheme}") + else: + candidate = f"https://{candidate}" + parsed = _split_url_or_raise(candidate) + scheme = parsed.scheme.lower() + + if scheme not in {"http", "https"}: + raise CLIError(f"Unsupported URL scheme: {parsed.scheme}") + if not parsed.netloc: + raise CLIError("Invalid URL: missing host") + return parse.urlunsplit((scheme, parsed.netloc, parsed.path, parsed.query, parsed.fragment)) + + +def _is_host_like_input(candidate: str) -> bool: + authority = candidate.split("/", 1)[0].split("?", 1)[0].split("#", 1)[0] + if not authority: + return False + if authority.startswith("["): + closing = authority.find("]") + if closing == -1: + return False + host = authority[: closing + 1] + rest = authority[closing + 1 :] + return _is_host_like_host(host) and (not rest or bool(re.fullmatch(r":\d+", rest))) + + if ":" not in authority: + return _is_host_like_host(authority) + + match = _HOST_WITH_PORT_RE.fullmatch(authority) + if not match or match.group("port") is None: + return False + host = match.group("host") or "" + return _is_host_like_host(host) + + +def _is_host_like_host(host: str) -> bool: + return ( + host == "localhost" + or "." in host + or bool(_IPV4_RE.fullmatch(host)) + or _is_bracketed_ipv6(host) + or bool(_HOST_LABEL_RE.fullmatch(host)) + ) + + +def _is_bracketed_ipv6(host: str) -> bool: + if not (host.startswith("[") and host.endswith("]")): + return False + literal = host[1:-1] + return bool(literal) and ":" in literal + + +def _split_url_or_raise(candidate: str) -> parse.SplitResult: + try: + return parse.urlsplit(candidate) + except ValueError as exc: + raise CLIError(f"Invalid URL: {exc}") from exc + + +def parse_args(argv: list[str]) -> Config: + config = Config() + index = 0 + while index < len(argv): + arg = argv[index] + if arg in ("-h", "--help"): + config.help_requested = True + break + if arg == "--json": + config.json_output = True + index += 1 + continue + if arg in ("-v", "--verbose"): + config.verbose = True + index += 1 + continue + if arg == "--no-clean": + config.no_clean = True + index += 1 + continue + if arg in ("-o", "--output"): + if index + 1 >= len(argv): + raise CLIError(f"Missing value for {arg}") + value = argv[index + 1] + if value.startswith("-"): + raise CLIError(f"Invalid value for {arg}: {value}") + config.output = value + index += 2 + continue + if arg.startswith("--output="): + value = arg.split("=", 1)[1] + if not value or value.startswith("-"): + raise CLIError("Invalid value for --output") + config.output = value + index += 1 + continue + if arg in ("-s", "--service"): + if index + 1 >= len(argv): + raise CLIError(f"Missing value for {arg}") + value = argv[index + 1] + if value.startswith("-"): + raise CLIError(f"Invalid value for {arg}: {value}") + if value not in VALID_SERVICES: + raise CLIError(f"Invalid service: {value}. Allowed values: jina|markdown|defuddle", source=value) + config.service = value + index += 2 + continue + if arg.startswith("--service="): + value = arg.split("=", 1)[1] + if value not in VALID_SERVICES: + raise CLIError(f"Invalid service: {value}. Allowed values: jina|markdown|defuddle", source=value or "none") + config.service = value + index += 1 + continue + if arg.startswith("-"): + raise CLIError(f"Unknown option: {arg}", source=config.service) + if config.url is not None: + raise CLIError("Only one URL argument is allowed", source=config.service) + config.url = arg + index += 1 + + if not config.help_requested and config.url is None: + raise CLIError("Please provide a URL", source=config.service) + return config + + +def main(argv: list[str] | None = None) -> int: + args = list(sys.argv[1:] if argv is None else argv) + json_output, requested_output = inspect_cli_mode(args) + + try: + config = parse_args(args) + except CLIError as exc: + if json_output: + return emit_json_failure("", exc.source, str(exc), requested_output) + print(f"{ERROR_PREFIX}{exc}", file=sys.stderr) + return 1 + + if config.help_requested: + sys.stdout.write(show_help()) + return 0 + + try: + normalized_url = normalize_url(config.url or "") + except CLIError as exc: + if config.json_output: + return emit_json_failure(config.url or "", config.service or "none", str(exc), config.output) + print(f"{ERROR_PREFIX}{exc}", file=sys.stderr) + return 1 + + config.url = normalized_url + rules = load_rules(config.verbose) + + try: + result = run_fetch(config, rules) + payload = result.content + if config.json_output: + payload = render_payload(True, normalized_url, result.content, result.source) + try: + write_output(payload, config.output) + except Exception as exc: + message = output_error_message(config.output, exc) + if config.json_output: + return emit_json_failure(normalized_url, result.source, message) + print(f"{ERROR_PREFIX}{message}", file=sys.stderr) + return 1 + return 0 + except CLIError as exc: + if config.json_output: + failure_source = config.service or "none" + return emit_json_failure(normalized_url, failure_source, str(exc), config.output) + print(f"{ERROR_PREFIX}{exc}", file=sys.stderr) + return 1 diff --git a/skills/smart-web-fetch/core/errors.py b/skills/smart-web-fetch/core/errors.py new file mode 100644 index 0000000..bf9895a --- /dev/null +++ b/skills/smart-web-fetch/core/errors.py @@ -0,0 +1,8 @@ +class CLIError(Exception): + def __init__(self, message: str, source: str | None = None) -> None: + super().__init__(message) + self.source = source or "none" + + +class FetchTransportError(Exception): + pass diff --git a/skills/smart-web-fetch/core/extract.py b/skills/smart-web-fetch/core/extract.py new file mode 100644 index 0000000..75ffbae --- /dev/null +++ b/skills/smart-web-fetch/core/extract.py @@ -0,0 +1,106 @@ +from __future__ import annotations + +import re +from html.parser import HTMLParser + + +class HTMLTextExtractor(HTMLParser): + BLOCK_TAGS = { + "article", + "aside", + "blockquote", + "br", + "div", + "footer", + "h1", + "h2", + "h3", + "h4", + "h5", + "h6", + "header", + "hr", + "li", + "main", + "nav", + "ol", + "p", + "pre", + "section", + "table", + "tr", + "ul", + } + IGNORE_TAGS = {"script", "style", "noscript"} + + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self._chunks: list[str] = [] + self._ignore_depth = 0 + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + tag = tag.lower() + if tag in self.IGNORE_TAGS: + self._ignore_depth += 1 + return + if self._ignore_depth: + return + if tag in self.BLOCK_TAGS: + self._chunks.append("\n") + + def handle_endtag(self, tag: str) -> None: + tag = tag.lower() + if tag in self.IGNORE_TAGS: + if self._ignore_depth: + self._ignore_depth -= 1 + return + if self._ignore_depth: + return + if tag in self.BLOCK_TAGS: + self._chunks.append("\n") + + def handle_data(self, data: str) -> None: + if self._ignore_depth or not data: + return + self._chunks.append(data) + + def get_text(self) -> str: + text = "".join(self._chunks) + text = re.sub(r"\r\n?", "\n", text) + text = re.sub(r"[ \t\f\v]+", " ", text) + text = re.sub(r"\n{3,}", "\n\n", text) + lines = [line.strip() for line in text.splitlines()] + return "\n".join(line for line in lines if line).strip() + + +def clean_html(html: str) -> str: + patterns = [ + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)]*>.*?", + r"(?is)", + ] + cleaned = html + for pattern in patterns: + cleaned = re.sub(pattern, "", cleaned) + cleaned = re.sub(r'\s(?:class|id|style|on\w+)=(".*?"|\'.*?\')', "", cleaned, flags=re.IGNORECASE | re.DOTALL) + return cleaned + + +def html_to_text(html: str) -> str: + parser = HTMLTextExtractor() + parser.feed(html) + parser.close() + return parser.get_text() + + +def looks_like_html(text: str, content_type: str) -> bool: + lowered_type = (content_type or "").lower() + if "text/html" in lowered_type or "application/xhtml+xml" in lowered_type: + return True + lowered_text = text.lower() + return " None: + if verbose: + print(f"[{level}] {message}", file=sys.stderr) + + +def render_payload(success: bool, url: str, content: str, source: str, error_message: str | None = None) -> str: + payload = { + "success": success, + "url": url, + "content": content, + "source": source, + } + if error_message: + payload["error"] = error_message + return json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + + +def inspect_cli_mode(argv: list[str]) -> tuple[bool, str | None]: + json_output = False + output: str | None = None + index = 0 + + while index < len(argv): + arg = argv[index] + if arg == "--json": + json_output = True + elif arg in ("-o", "--output"): + if index + 1 < len(argv): + value = argv[index + 1] + if value and not value.startswith("-"): + output = value + index += 1 + elif arg.startswith("--output="): + value = arg.split("=", 1)[1] + if value and not value.startswith("-"): + output = value + index += 1 + + return json_output, output + + +def write_output(payload: str, destination: str | None) -> None: + if not destination: + sys.stdout.write(payload) + if not payload.endswith("\n"): + sys.stdout.write("\n") + return + target = Path(destination) + if target.parent and not target.parent.exists(): + target.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_path = tempfile.mkstemp(prefix=f".{target.name}.tmp.", dir=str(target.parent or Path.cwd())) + try: + with os.fdopen(fd, "w", encoding="utf-8", newline="") as handle: + handle.write(payload) + if not payload.endswith("\n"): + handle.write("\n") + os.replace(tmp_path, target) + except Exception: + try: + os.unlink(tmp_path) + except OSError: + pass + raise + + +def output_error_message(destination: str | None, exc: Exception) -> str: + if destination: + return f"Failed to write output to {destination}: {exc}" + return f"Failed to write output: {exc}" + + +def emit_json_failure(url: str, source: str, error_message: str, destination: str | None = None) -> int: + payload = render_payload(False, url, "", source, error_message) + + if destination: + try: + write_output(payload, destination) + return 1 + except Exception as exc: + payload = render_payload( + False, + url, + "", + source, + f"{error_message}; {output_error_message(destination, exc)}", + ) + + write_output(payload, None) + return 1 diff --git a/skills/smart-web-fetch/core/rules.py b/skills/smart-web-fetch/core/rules.py new file mode 100644 index 0000000..8ccad66 --- /dev/null +++ b/skills/smart-web-fetch/core/rules.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +import json +import re +from dataclasses import dataclass +from pathlib import Path + +from .output import log + + +DEFAULT_RULES = { + "thresholds": { + "jina": 100, + "markdown_new": 40, + "defuddle": 40, + "basic": 40, + }, + "structured_error_keywords": [ + "error", + "fail", + "invalid", + "unauthorized", + "forbidden", + "denied", + "blocked", + "not found", + "rate limit", + "too many requests", + ], + "html_error_keywords": [ + "access denied", + "forbidden", + "captcha", + "cloudflare", + "just a moment", + "unauthorized", + "bad gateway", + "gateway timeout", + "service unavailable", + ], +} + +PACKAGE_DIR = Path(__file__).resolve().parent +SKILL_DIR = PACKAGE_DIR.parent +RULES_FILE = SKILL_DIR / "assets" / "fetch-rules.json" + + +@dataclass +class Rules: + jina_min_length: int + markdown_min_length: int + defuddle_min_length: int + basic_min_length: int + structured_error_keywords: list[str] + html_error_keywords: list[str] + + +def load_rules(verbose: bool) -> Rules: + data = json.loads(json.dumps(DEFAULT_RULES)) + if RULES_FILE.is_file(): + try: + loaded = json.loads(RULES_FILE.read_text(encoding="utf-8")) + thresholds = loaded.get("thresholds", {}) + for key in ("jina", "markdown_new", "defuddle", "basic"): + value = thresholds.get(key) + if isinstance(value, int) and value > 0: + data["thresholds"][key] = value + structured = loaded.get("structured_error_keywords") + if isinstance(structured, list) and structured: + data["structured_error_keywords"] = [str(item) for item in structured if str(item).strip()] + html_keywords = loaded.get("html_error_keywords") + if isinstance(html_keywords, list) and html_keywords: + data["html_error_keywords"] = [str(item) for item in html_keywords if str(item).strip()] + log(verbose, "INFO", f"Loaded thresholds and keywords from rules file: {RULES_FILE}") + except (OSError, ValueError, TypeError, json.JSONDecodeError): + log(verbose, "WARN", f"Failed to parse rules file, using built-in defaults: {RULES_FILE}") + return Rules( + jina_min_length=data["thresholds"]["jina"], + markdown_min_length=data["thresholds"]["markdown_new"], + defuddle_min_length=data["thresholds"]["defuddle"], + basic_min_length=data["thresholds"]["basic"], + structured_error_keywords=list(data["structured_error_keywords"]), + html_error_keywords=list(data["html_error_keywords"]), + ) + + +def normalize_keyword_text(text: str | None) -> str: + if not text: + return "" + normalized = text.lower().replace("_", " ").replace("-", " ") + return re.sub(r"\s+", " ", normalized).strip() + + +def contains_keyword(text: str | None, keywords: list[str]) -> bool: + normalized_text = normalize_keyword_text(text) + if not normalized_text: + return False + return any(normalize_keyword_text(keyword) in normalized_text for keyword in keywords if normalize_keyword_text(keyword)) diff --git a/skills/smart-web-fetch/core/sources.py b/skills/smart-web-fetch/core/sources.py new file mode 100644 index 0000000..c367d0f --- /dev/null +++ b/skills/smart-web-fetch/core/sources.py @@ -0,0 +1,228 @@ +from __future__ import annotations + +import json +import os +from dataclasses import dataclass +from typing import TYPE_CHECKING, Callable +from urllib import parse + +from .errors import CLIError, FetchTransportError +from .extract import clean_html, html_to_text, looks_like_html +from .output import log +from .rules import Rules, contains_keyword +from .transport import is_binary_response, request_text + +if TYPE_CHECKING: + from .cli import Config + + +JINA_READER_BASE = os.environ.get("SMART_WEB_FETCH_JINA_READER_BASE", "https://r.jina.ai") +MARKDOWN_NEW_URL = os.environ.get("SMART_WEB_FETCH_MARKDOWN_NEW_URL", "https://api.markdown.new/api/v1/convert") +DEFUDDLE_URL = os.environ.get("SMART_WEB_FETCH_DEFUDDLE_URL", "https://defuddle.md/api/convert") + + +@dataclass +class FetchResult: + content: str + source: str + + +def parse_json_object(text: str) -> dict[str, object] | None: + try: + value = json.loads(text) + except json.JSONDecodeError: + return None + if isinstance(value, dict): + return value + return None + + +def is_structured_error_response(text: str, rules: Rules) -> bool: + payload = parse_json_object(text) + if payload is None: + return False + error_value = payload.get("error") + if error_value is True: + return True + if isinstance(error_value, str) and contains_keyword(error_value, rules.structured_error_keywords): + return True + message_value = payload.get("message") + if isinstance(message_value, str) and contains_keyword(message_value, rules.structured_error_keywords): + return True + return False + + +def is_likely_html_error_payload(text: str, content_type: str, rules: Rules) -> bool: + lowered_type = (content_type or "").lower() + if "text/html" not in lowered_type and "application/xhtml+xml" not in lowered_type: + return False + lowered_text = text.lower() + if " str: + parsed = parse.urlsplit(target_url) + scheme = (parsed.scheme or "http").lower() + suffix = target_url.split("://", 1)[1] if "://" in target_url else target_url + return f"{JINA_READER_BASE.rstrip('/')}/{scheme}://{suffix}" + + +def extract_markdown_field(text: str) -> tuple[str | None, bool]: + payload = parse_json_object(text) + if payload is None: + return None, False + for key in ("markdown", "content", "data"): + if key not in payload or payload[key] is None: + continue + value = payload[key] + if isinstance(value, str): + return value, True + if isinstance(value, (dict, list)): + return json.dumps(value, ensure_ascii=False, separators=(",", ":")), True + return str(value), True + return None, True + + +def ensure_min_length(source: str, label: str, text: str, minimum: int) -> str: + if len(text) < minimum: + raise CLIError(f"{label} returned empty or too-short content", source=source) + return text + + +def extract_service_content(source: str, label: str, response_text: str, minimum: int) -> str: + markdown, is_json_object = extract_markdown_field(response_text) + if is_json_object: + if markdown is None or markdown == "": + raise CLIError(f"{label} returned JSON without usable markdown/content/data", source=source) + return ensure_min_length(source, label, markdown, minimum) + return ensure_min_length(source, label, response_text, minimum) + + +def fetch_jina(url: str, rules: Rules, verbose: bool) -> FetchResult: + log(verbose, "INFO", "Trying Jina Reader") + try: + response = request_text( + build_jina_url(url), + headers={"User-Agent": "SmartWebFetch/1.0"}, + ) + except FetchTransportError as exc: + raise CLIError(f"Jina Reader request failed: {exc}", source="jina") from exc + if not 200 <= response.status_code < 300: + raise CLIError(f"Jina Reader returned HTTP {response.status_code}", source="jina") + if is_structured_error_response(response.text, rules): + raise CLIError("Jina Reader returned structured error payload", source="jina") + return FetchResult( + content=ensure_min_length("jina", "Jina Reader", response.text, rules.jina_min_length), + source="jina", + ) + + +def fetch_markdown_new(url: str, rules: Rules, verbose: bool) -> FetchResult: + log(verbose, "INFO", "Trying markdown.new") + body = json.dumps({"url": url}, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + try: + response = request_text( + MARKDOWN_NEW_URL, + method="POST", + headers={ + "Content-Type": "application/json", + "User-Agent": "SmartWebFetch/1.0", + }, + body=body, + ) + except FetchTransportError as exc: + raise CLIError(f"markdown.new request failed: {exc}", source="markdown") from exc + if not 200 <= response.status_code < 300: + raise CLIError(f"markdown.new returned HTTP {response.status_code}", source="markdown") + if is_likely_html_error_payload(response.text, response.content_type, rules): + raise CLIError( + f"markdown.new returned HTML error page (content-type: {response.content_type or 'unknown'})", + source="markdown", + ) + if is_structured_error_response(response.text, rules): + raise CLIError("markdown.new returned structured error payload", source="markdown") + content = extract_service_content("markdown", "markdown.new", response.text, rules.markdown_min_length) + return FetchResult(content=content, source="markdown") + + +def fetch_defuddle(url: str, rules: Rules, verbose: bool) -> FetchResult: + log(verbose, "INFO", "Trying defuddle.md") + body = json.dumps({"url": url}, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + try: + response = request_text( + DEFUDDLE_URL, + method="POST", + headers={ + "Content-Type": "application/json", + "User-Agent": "SmartWebFetch/1.0", + }, + body=body, + ) + except FetchTransportError as exc: + raise CLIError(f"defuddle.md request failed: {exc}", source="defuddle") from exc + if not 200 <= response.status_code < 300: + raise CLIError(f"defuddle.md returned HTTP {response.status_code}", source="defuddle") + if is_likely_html_error_payload(response.text, response.content_type, rules): + raise CLIError( + f"defuddle.md returned HTML error page (content-type: {response.content_type or 'unknown'})", + source="defuddle", + ) + if is_structured_error_response(response.text, rules): + raise CLIError("defuddle.md returned structured error payload", source="defuddle") + content = extract_service_content("defuddle", "defuddle.md", response.text, rules.defuddle_min_length) + return FetchResult(content=content, source="defuddle") + + +def fetch_basic(url: str, rules: Rules, verbose: bool, no_clean: bool) -> FetchResult: + log(verbose, "INFO", "Trying basic fallback") + try: + response = request_text( + url, + headers={ + "User-Agent": "Mozilla/5.0", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + }, + ) + except FetchTransportError as exc: + raise CLIError(f"Basic fallback request failed: {exc}", source="basic") from exc + if not 200 <= response.status_code < 300: + raise CLIError(f"Basic fallback returned HTTP {response.status_code}", source="basic") + if is_binary_response(response.content_type, response.raw_body): + raise CLIError("Basic fallback returned non-text/binary content", source="basic") + if len(response.text) < rules.basic_min_length: + raise CLIError("Basic fallback returned empty or too-short content before cleanup", source="basic") + processed = response.text + if not no_clean and looks_like_html(response.text, response.content_type): + cleaned_html = clean_html(response.text) + if len(cleaned_html) < rules.basic_min_length: + raise CLIError("Basic fallback returned empty or too-short content after HTML cleanup", source="basic") + processed = html_to_text(cleaned_html) + if len(processed) < rules.basic_min_length: + raise CLIError("Basic fallback returned empty or too-short content after cleanup", source="basic") + return FetchResult(content=processed, source="basic") + + +def run_fetch(config: Config, rules: Rules) -> FetchResult: + if config.url is None: + raise CLIError("Please provide a URL") + log(config.verbose, "INFO", f"Fetching {config.url}") + fetchers: dict[str, Callable[[], FetchResult]] = { + "jina": lambda: fetch_jina(config.url, rules, config.verbose), + "markdown": lambda: fetch_markdown_new(config.url, rules, config.verbose), + "defuddle": lambda: fetch_defuddle(config.url, rules, config.verbose), + "basic": lambda: fetch_basic(config.url, rules, config.verbose, config.no_clean), + } + if config.service: + return fetchers[config.service]() + last_error: CLIError | None = None + for name in ("jina", "markdown", "defuddle", "basic"): + try: + return fetchers[name]() + except CLIError as exc: + last_error = exc + log(config.verbose, "WARN", str(exc)) + if last_error is not None: + raise CLIError(f"All fetch methods failed. Last error: {last_error}", source="none") from last_error + raise CLIError("All fetch methods failed", source="none") diff --git a/skills/smart-web-fetch/core/transport.py b/skills/smart-web-fetch/core/transport.py new file mode 100644 index 0000000..e5effee --- /dev/null +++ b/skills/smart-web-fetch/core/transport.py @@ -0,0 +1,124 @@ +from __future__ import annotations + +from dataclasses import dataclass +from urllib import error, request + +from .errors import FetchTransportError + + +TIMEOUT_SECONDS = 30 + + +@dataclass +class ResponseData: + status_code: int + content_type: str + raw_body: bytes + text: str + + +def decode_response_body(body: bytes, headers) -> str: + charset = None + try: + charset = headers.get_content_charset() + except AttributeError: + charset = None + + encodings: list[str] = [] + if charset: + encodings.append(charset) + if body.startswith((b"\xff\xfe", b"\xfe\xff")): + encodings.append("utf-16") + encodings.extend(["utf-8", "latin-1"]) + + for encoding in encodings: + try: + return body.decode(encoding) + except (LookupError, UnicodeDecodeError): + continue + return body.decode("utf-8", errors="replace") + + +def get_media_type(content_type: str) -> str: + return (content_type or "").split(";", 1)[0].strip().lower() + + +def is_text_media_type(content_type: str) -> bool: + media_type = get_media_type(content_type) + if not media_type: + return False + if media_type.startswith("text/"): + return True + return media_type in { + "application/json", + "application/xml", + "application/xhtml+xml", + "application/javascript", + "application/x-javascript", + "application/x-www-form-urlencoded", + } or media_type.endswith(("+json", "+xml")) + + +def is_binary_media_type(content_type: str) -> bool: + media_type = get_media_type(content_type) + if not media_type: + return False + if media_type.startswith(("image/", "audio/", "video/")): + return True + return media_type in { + "application/octet-stream", + "application/pdf", + "application/zip", + "application/x-zip-compressed", + "application/gzip", + "application/x-gzip", + "application/x-tar", + "application/x-bzip2", + "application/x-7z-compressed", + "application/vnd.rar", + "application/x-rar-compressed", + } + + +def has_binary_body_signature(body: bytes) -> bool: + if not body: + return False + if b"\x00" in body: + return True + sample = body[:1024] + try: + sample.decode("utf-8") + return False + except UnicodeDecodeError: + pass + non_printable = sum(byte < 32 and byte not in (9, 10, 13, 12, 8) for byte in sample) + return non_printable / max(len(sample), 1) >= 0.30 + + +def is_binary_response(content_type: str, body: bytes) -> bool: + if is_binary_media_type(content_type): + return True + if is_text_media_type(content_type): + return False + return has_binary_body_signature(body) + + +def request_text(url: str, method: str = "GET", headers: dict[str, str] | None = None, body: bytes | None = None) -> ResponseData: + req = request.Request(url=url, data=body, headers=headers or {}, method=method) + try: + with request.urlopen(req, timeout=TIMEOUT_SECONDS) as response: + status_code = getattr(response, "status", response.getcode()) + content_type = response.headers.get("Content-Type", "") + raw_body = response.read() + text = decode_response_body(raw_body, response.headers) + return ResponseData(status_code=status_code, content_type=content_type, raw_body=raw_body, text=text) + except error.HTTPError as exc: + content_type = exc.headers.get("Content-Type", "") if exc.headers else "" + raw_body = exc.read() + text = decode_response_body(raw_body, exc.headers or {}) + return ResponseData(status_code=exc.code, content_type=content_type, raw_body=raw_body, text=text) + except error.URLError as exc: + reason = exc.reason if getattr(exc, "reason", None) else exc + raise FetchTransportError(str(reason)) from exc + except OSError as exc: + raise FetchTransportError(str(exc)) from exc diff --git a/skills/smart-web-fetch/main.py b/skills/smart-web-fetch/main.py new file mode 100644 index 0000000..3a9989b --- /dev/null +++ b/skills/smart-web-fetch/main.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import sys +from pathlib import Path + + +INTERPRETER_ERROR = ( + "smart-web-fetch: error: Python 3.11+ was not found. " + "Install Python 3.11 or newer and ensure a compatible interpreter is on PATH." +) + +SKILL_DIR = Path(__file__).resolve().parent +SKILL_DIR_STR = str(SKILL_DIR) + +try: + sys.path.remove(SKILL_DIR_STR) +except ValueError: + pass +sys.path.insert(0, SKILL_DIR_STR) + +if sys.version_info < (3, 11): + print(INTERPRETER_ERROR, file=sys.stderr) + raise SystemExit(1) + +from core.cli import main as cli_main + + +if __name__ == "__main__": + raise SystemExit(cli_main()) diff --git a/skills/smart-web-fetch/scripts/smart-web-fetch b/skills/smart-web-fetch/scripts/smart-web-fetch index d53037c..284d317 100755 --- a/skills/smart-web-fetch/scripts/smart-web-fetch +++ b/skills/smart-web-fetch/scripts/smart-web-fetch @@ -1,50 +1,26 @@ #!/usr/bin/env bash -# smart-web-fetch — unified entry point -# Detects available runtime and routes to the correct core implementation. -# Accepts POSIX-style parameters; translates to PowerShell names when needed. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -# --------------------------------------------------------------------------- -# Route to Bash core when curl is available (preferred path) -# --------------------------------------------------------------------------- -if command -v curl > /dev/null 2>&1; then - exec "$SCRIPT_DIR/smart-web-fetch-core" "$@" -fi - -# --------------------------------------------------------------------------- -# Fall back to PowerShell 7 when curl is absent -# --------------------------------------------------------------------------- -if ! command -v pwsh > /dev/null 2>&1; then - echo "smart-web-fetch: error: neither curl nor pwsh (PowerShell 7) was found." >&2 - echo " Install curl for Bash usage, or install PowerShell 7 for Windows usage." >&2 - exit 1 +SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +VERSION_CHECK='import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' +ERROR_MESSAGE="smart-web-fetch: error: Python 3.11+ was not found. Install Python 3.11 or newer and ensure a compatible interpreter is on PATH." + +find_python() { + local candidate + for candidate in python3 python; do + if command -v "$candidate" >/dev/null 2>&1 && "$candidate" -c "$VERSION_CHECK" >/dev/null 2>&1; then + printf '%s\n' "$candidate" + return 0 + fi + done + return 1 +} + +if PYTHON_BIN="$(find_python)"; then + exec "$PYTHON_BIN" "$SKILL_DIR/main.py" "$@" fi -# Translate POSIX-style arguments to PowerShell parameter names. -# Build an array to avoid any quoting / special-character issues. -ps_args=() -skip_next=0 - -for arg in "$@"; do - if [[ "$skip_next" -eq 1 ]]; then - ps_args+=("$arg") - skip_next=0 - continue - fi - - case "$arg" in - --no-clean) ps_args+=("-NoClean") ;; - --verbose) ps_args+=("-VerboseMode") ;; - --output) ps_args+=("-Output"); skip_next=1 ;; - --service) ps_args+=("-Service"); skip_next=1 ;; - --help) ps_args+=("-Help") ;; - # Short flags and positional arguments pass through unchanged. - # The PS1 core accepts: -o, -s, -v, -h via [Alias] attributes. - *) ps_args+=("$arg") ;; - esac -done - -exec pwsh -File "$SCRIPT_DIR/smart-web-fetch-core.ps1" "${ps_args[@]}" +echo "$ERROR_MESSAGE" >&2 +exit 1 diff --git a/skills/smart-web-fetch/scripts/smart-web-fetch-core b/skills/smart-web-fetch/scripts/smart-web-fetch-core deleted file mode 100755 index 82161b2..0000000 --- a/skills/smart-web-fetch/scripts/smart-web-fetch-core +++ /dev/null @@ -1,1337 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' - -TIMEOUT=30 -JINA_MIN_LENGTH=100 -MARKDOWN_NEW_MIN_LENGTH=40 -DEFUDDLE_MIN_LENGTH=40 -BASIC_MIN_LENGTH=40 - -STRUCTURED_ERROR_KEYWORDS=( - "error" - "fail" - "invalid" - "unauthorized" - "forbidden" - "denied" - "blocked" - "not found" - "rate limit" - "too many requests" -) - -HTML_ERROR_KEYWORDS=( - "access denied" - "forbidden" - "captcha" - "cloudflare" - "just a moment" - "unauthorized" - "bad gateway" - "gateway timeout" - "service unavailable" -) - -JINA_READER_BASE="https://r.jina.ai" -MARKDOWN_NEW="https://api.markdown.new/api/v1/convert" -DEFUDDLE_MD="https://defuddle.md/api/convert" - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" -RULES_FILE="$SKILL_DIR/assets/fetch-rules.json" -TMP_FILES=() - - -create_tmp_file() { - local tmp - tmp=$(mktemp) - TMP_FILES+=("$tmp") - printf '%s\n' "$tmp" -} - -remove_tmp_file() { - local target="$1" - local remaining=() - rm -f -- "$target" - - for f in "${TMP_FILES[@]}"; do - if [[ "$f" != "$target" ]]; then - remaining+=("$f") - fi - done - - TMP_FILES=("${remaining[@]}") -} - -cleanup_tmp_files() { - local f - for f in "${TMP_FILES[@]}"; do - rm -f -- "$f" - done - TMP_FILES=() -} - -handle_signal_cleanup_and_exit() { - cleanup_tmp_files - case "$1" in - INT) exit 130 ;; - TERM) exit 143 ;; - *) exit 1 ;; - esac -} - -load_rules_from_file() { - if [[ ! -f "$RULES_FILE" ]]; then - return 0 - fi - - if command -v jq >/dev/null 2>&1; then - local value - value=$(jq -r '.thresholds.jina // empty' "$RULES_FILE" 2>/dev/null || true) - [[ "$value" =~ ^[0-9]+$ ]] && JINA_MIN_LENGTH="$value" - - value=$(jq -r '.thresholds.markdown_new // empty' "$RULES_FILE" 2>/dev/null || true) - [[ "$value" =~ ^[0-9]+$ ]] && MARKDOWN_NEW_MIN_LENGTH="$value" - - value=$(jq -r '.thresholds.defuddle // empty' "$RULES_FILE" 2>/dev/null || true) - [[ "$value" =~ ^[0-9]+$ ]] && DEFUDDLE_MIN_LENGTH="$value" - - value=$(jq -r '.thresholds.basic // empty' "$RULES_FILE" 2>/dev/null || true) - [[ "$value" =~ ^[0-9]+$ ]] && BASIC_MIN_LENGTH="$value" - - local -a structured_keywords=() - local -a html_keywords=() - - mapfile -t structured_keywords < <(jq -r '.structured_error_keywords[]? // empty' "$RULES_FILE" 2>/dev/null || true) - if [[ ${#structured_keywords[@]} -gt 0 ]]; then - STRUCTURED_ERROR_KEYWORDS=("${structured_keywords[@]}") - fi - - mapfile -t html_keywords < <(jq -r '.html_error_keywords[]? // empty' "$RULES_FILE" 2>/dev/null || true) - if [[ ${#html_keywords[@]} -gt 0 ]]; then - HTML_ERROR_KEYWORDS=("${html_keywords[@]}") - fi - - log_info "Loaded thresholds and keywords from rules file: $RULES_FILE" - return 0 - fi - - if load_rules_from_file_without_jq; then - log_info "Loaded thresholds and keywords from rules file without jq: $RULES_FILE" - return 0 - fi - - log_warn "Failed to parse rules file without jq; using built-in defaults: $RULES_FILE" -} - -normalize_keyword_text() { - local text="$1" - printf '%s' "$text" \ - | tr '[:upper:]' '[:lower:]' \ - | tr '_-' ' ' \ - | tr -s '[:space:]' ' ' \ - | sed 's/^ //; s/ $//' -} - -contains_error_keyword() { - local text="$1" - local normalized_text - local keyword - local normalized_keyword - - normalized_text=$(normalize_keyword_text "$text") - [[ -z "$normalized_text" ]] && return 1 - - for keyword in "${STRUCTURED_ERROR_KEYWORDS[@]}"; do - normalized_keyword=$(normalize_keyword_text "$keyword") - [[ -z "$normalized_keyword" ]] && continue - if [[ "$normalized_text" == *"$normalized_keyword"* ]]; then - return 0 - fi - done - - return 1 -} - -trap cleanup_tmp_files EXIT -trap 'handle_signal_cleanup_and_exit INT' INT -trap 'handle_signal_cleanup_and_exit TERM' TERM - -show_help() { - cat <<'EOF' -Smart Web Fetch - lightweight web-to-Markdown fetcher - -Usage: - smart-web-fetch [options] - -Options: - -h, --help Show help - -o, --output FILE Write output to file - -s, --service NAME Force service: jina|markdown|defuddle - -v, --verbose Show verbose logs - --no-clean Skip HTML cleanup in the basic curl fallback - -Examples: - smart-web-fetch https://example.com - smart-web-fetch https://example.com -o output.md - smart-web-fetch https://example.com -s jina - smart-web-fetch https://example.com --no-clean - -Fallback order: - 1. Jina Reader - 2. markdown.new - 3. defuddle.md - 4. direct curl fallback -EOF -} - -log_info() { - [[ "$VERBOSE" == "1" ]] && echo -e "${BLUE}[INFO]${NC} $1" >&2 -} - -log_success() { - [[ "$VERBOSE" == "1" ]] && echo -e "${GREEN}[SUCCESS]${NC} $1" >&2 -} - -log_warn() { - [[ "$VERBOSE" == "1" ]] && echo -e "${YELLOW}[WARN]${NC} $1" >&2 -} - -log_error() { - echo -e "${RED}[ERROR]${NC} $1" >&2 -} - -set_last_fetch_error() { - LAST_FETCH_ERROR="$1" -} - -log_dep_status() { - local level="$1" - local message="$2" - - case "$level" in - required-ok|optional-ok) - log_info "[Dependency] $message" - ;; - optional-missing) - log_warn "[Dependency] $message" - ;; - required-missing) - log_error "[Dependency] $message" - ;; - esac -} - -check_runtime_dependencies() { - local missing_required=0 - - if command -v curl >/dev/null 2>&1; then - log_dep_status "required-ok" "curl detected" - else - log_dep_status "required-missing" "Missing required dependency: curl" - missing_required=1 - fi - - if command -v jq >/dev/null 2>&1; then - log_dep_status "optional-ok" "Optional dependency detected: jq (JSON markdown extraction)" - else - log_dep_status "optional-missing" "Optional dependency not found: jq (fallback to raw/PowerShell JSON parsing)" - fi - - if command -v perl >/dev/null 2>&1; then - log_dep_status "optional-ok" "Optional dependency detected: perl (enhanced HTML cleanup)" - else - log_dep_status "optional-missing" "Optional dependency not found: perl (fallback to basic sed cleanup)" - fi - - if command -v html2text >/dev/null 2>&1 || command -v lynx >/dev/null 2>&1; then - log_dep_status "optional-ok" "Optional dependency detected: html2text/lynx (HTML-to-text fallback)" - else - log_dep_status "optional-missing" "Optional dependency not found: html2text or lynx (fallback returns cleaned HTML)" - fi - - if [[ "$missing_required" -ne 0 ]]; then - log_error "Dependency check failed. Please install required dependencies and retry." - exit 1 - fi -} - -exit_with_help_error() { - log_error "$1" - show_help - exit 1 -} - -validate_option_value() { - local option_name="$1" - local option_value="$2" - - if [[ -z "$option_value" ]]; then - exit_with_help_error "Missing value for ${option_name}" - fi - - if [[ "$option_value" == -* ]]; then - exit_with_help_error "Invalid value for ${option_name}: ${option_value}" - fi -} - -validate_service_value() { - local service_value="$1" - - case "$service_value" in - jina|markdown|defuddle) - return 0 - ;; - *) - exit_with_help_error "Invalid service: ${service_value}. Allowed values: jina|markdown|defuddle" - ;; - esac -} - -build_jina_url() { - local target_url="$1" - local scheme="http" - local url_without_scheme="$target_url" - - if [[ "$target_url" =~ ^([Hh][Tt][Tt][Pp][Ss]?)://(.*)$ ]]; then - scheme="${BASH_REMATCH[1]}" - url_without_scheme="${BASH_REMATCH[2]}" - fi - - scheme=$(printf '%s' "$scheme" | tr '[:upper:]' '[:lower:]') - - printf '%s/%s://%s\n' "$JINA_READER_BASE" "$scheme" "$url_without_scheme" -} - -is_structured_error_response() { - local response="$1" - local compact_response - local lowered_response - local error_value="" - local message_value="" - - if command -v jq >/dev/null 2>&1; then - if printf '%s' "$response" | jq -e 'type == "object"' >/dev/null 2>&1; then - if [[ "$(printf '%s' "$response" | jq -r 'if (.error | type) == "boolean" then .error else empty end' 2>/dev/null || true)" == "true" ]]; then - return 0 - fi - - error_value=$(printf '%s' "$response" | jq -r 'if (.error | type) == "string" then .error else empty end' 2>/dev/null || true) - message_value=$(printf '%s' "$response" | jq -r 'if (.message | type) == "string" then .message else empty end' 2>/dev/null || true) - - if contains_error_keyword "$error_value" || contains_error_keyword "$message_value"; then - return 0 - fi - - return 1 - fi - fi - - # Fallback without jq: only treat JSON-looking payloads with explicit - # "error"/"message" keys and common failure semantics as structured errors. - compact_response=$(printf '%s' "$response" | tr -d '[:space:]') - if [[ "$compact_response" != "{"* ]] && [[ "$compact_response" != "["* ]]; then - return 1 - fi - - lowered_response=$(printf '%s' "$compact_response" | tr '[:upper:]' '[:lower:]') - - if [[ "$lowered_response" =~ \"error\":true ]]; then - return 0 - fi - - error_value=$(printf '%s' "$response" | sed -n 's/.*"error"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -n 1) - message_value=$(printf '%s' "$response" | sed -n 's/.*"message"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -n 1) - - if contains_error_keyword "$error_value" || contains_error_keyword "$message_value"; then - return 0 - fi - - return 1 -} - -extract_http_status_code() { - local headers_file="$1" - awk 'toupper($1) ~ /^HTTP\// { code=$2 } END { print code }' "$headers_file" -} - -extract_content_type() { - local headers_file="$1" - awk -F': *' ' - BEGIN { IGNORECASE=1 } - tolower($1) == "content-type" { ct=$2 } - END { - sub(/\r$/, "", ct) - print tolower(ct) - } - ' "$headers_file" -} - -is_likely_html_error_payload() { - local response="$1" - local content_type="$2" - local lowered_response="" - local normalized_response="" - local keyword - local normalized_keyword - - if [[ "$content_type" != *"text/html"* ]] && [[ "$content_type" != *"application/xhtml+xml"* ]]; then - return 1 - fi - - lowered_response=$(printf '%s' "$response" | tr '[:upper:]' '[:lower:]') - if [[ "$lowered_response" != *"/dev/null 2>&1; then - perl -0pe 's{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{}{}gsi; s{\s(?:class|id|style|on\w+)=("[^"]*"|'"'"'[^'"'"']*'"'"')}{}gsi;' - else - # 无 perl 时做保守清洗： - # 1) 先去除跨行噪音块，同时保留同一行中的非标签正文 - # 2) 再删除同一行内残留的噪音标签片段 - # 3) 继续执行属性清洗（class/id/style/on*） - awk ' - BEGIN { - IGNORECASE = 1 - in_comment = 0 - in_block = "" - } - - function trim_closed_block(line, close_pat, pos) { - pos = match(line, close_pat) - if (pos) { - return substr(line, pos + RLENGTH) - } - return "" - } - - function remove_tag_pairs(line, tag, open_pat, close_pat, pos, prefix, rest, close_pos) { - open_pat = "<" tag "([[:space:]][^>]*)?>" - close_pat = "" - - while ((pos = match(line, open_pat))) { - prefix = substr(line, 1, pos - 1) - rest = substr(line, pos + RLENGTH) - close_pos = match(rest, close_pat) - - if (close_pos) { - line = prefix substr(rest, close_pos + RLENGTH) - } else { - in_block = tag - line = prefix - break - } - } - - return line - } - - { - line = $0 - - if (in_comment) { - line = trim_closed_block(line, "-->") - if (line == "") { - next - } - in_comment = 0 - } - - if (in_block != "") { - line = trim_closed_block(line, "") - if (line == "") { - next - } - in_block = "" - } - - while (match(line, "")) { - line = prefix substr(rest, RSTART + RLENGTH) - } else { - line = prefix - in_comment = 1 - break - } - } - - line = remove_tag_pairs(line, "script") - line = remove_tag_pairs(line, "style") - line = remove_tag_pairs(line, "noscript") - line = remove_tag_pairs(line, "nav") - line = remove_tag_pairs(line, "header") - line = remove_tag_pairs(line, "footer") - line = remove_tag_pairs(line, "aside") - - gsub(/<\/?(script|style|noscript|nav|header|footer|aside)([[:space:]][^>]*)?>/, "", line) - gsub(//, "", line) - gsub(/[[:space:]]+(class|id|style|on[[:alnum:]_]+)=("[^"]*"|'\''[^'\'']*'\'')/, "", line) - - if (line != "" || (!in_comment && in_block == "")) { - print line - } - } - ' - fi -} - -# Build {"url":"..."} safely for POST bodies. -# Do not directly concatenate JSON like "{\"url\":\"$url\"}" because URLs can contain -# quotes, backslashes, or control characters that would break JSON syntax or change payload meaning. -build_url_json_payload() { - local url="$1" - local escaped_url="" - - if command -v jq >/dev/null 2>&1; then - jq -n --arg url "$url" '{url:$url}' - return 0 - fi - - escaped_url=$(json_escape_minimal "$url") - printf '{"url":"%s"}' "$escaped_url" -} - -json_escape_minimal() { - local input="$1" - local output="" - local char="" - local byte="" - local hex="" - local i=0 - - for (( i=0; i<${#input}; i++ )); do - char="${input:i:1}" - case "$char" in - \\) output+='\\\\' ;; - '"') output+='\\"' ;; - $'\b') output+='\\b' ;; - $'\f') output+='\\f' ;; - $'\n') output+='\\n' ;; - $'\r') output+='\\r' ;; - $'\t') output+='\\t' ;; - *) - byte=$(printf '%s' "$char" | od -An -tu1 | tr -d '[:space:]') - if [[ -n "$byte" ]] && (( byte < 32 )); then - printf -v hex '%02X' "$byte" - output+="\\u00$hex" - else - output+="$char" - fi - ;; - esac - done - - printf '%s' "$output" -} - -extract_markdown_from_json_without_jq() { - local response="$1" - local value="" - - value=$(extract_json_string_field_without_jq "$response" "markdown" || true) - if [[ -n "$value" ]]; then - printf '%s' "$value" - return 0 - fi - - value=$(extract_json_string_field_without_jq "$response" "content" || true) - if [[ -n "$value" ]]; then - printf '%s' "$value" - return 0 - fi - - value=$(extract_json_string_field_without_jq "$response" "data" || true) - if [[ -n "$value" ]]; then - printf '%s' "$value" - return 0 - fi - - return 1 -} - -extract_json_field_segment_without_jq() { - local input="$1" - local target_key="$2" - local opening_char="$3" - local closing_char="$4" - local length=${#input} - local pos=0 - local char="" - local after_key_pos="" - local value_start="" - local cursor=0 - local depth=0 - local in_string=0 - local escaped=0 - - while (( pos < length )); do - char="${input:pos:1}" - if [[ "$char" != '"' ]]; then - ((pos++)) - continue - fi - - if ! parse_json_string_without_jq "$input" "$pos"; then - return 1 - fi - - after_key_pos="$JSON_PARSE_NEXT_INDEX" - if [[ "$JSON_PARSE_VALUE" == "$target_key" ]]; then - after_key_pos=$(skip_json_whitespace_without_jq "$input" "$after_key_pos") - if [[ "${input:after_key_pos:1}" != ":" ]]; then - pos="$after_key_pos" - continue - fi - - value_start=$(skip_json_whitespace_without_jq "$input" "$((after_key_pos + 1))") - if [[ "${input:value_start:1}" != "$opening_char" ]]; then - return 1 - fi - - cursor="$value_start" - depth=0 - in_string=0 - escaped=0 - - while (( cursor < length )); do - char="${input:cursor:1}" - - if (( in_string )); then - if (( escaped )); then - escaped=0 - elif [[ "$char" == '\' ]]; then - escaped=1 - elif [[ "$char" == '"' ]]; then - in_string=0 - fi - else - if [[ "$char" == '"' ]]; then - in_string=1 - elif [[ "$char" == "$opening_char" ]]; then - ((depth++)) - elif [[ "$char" == "$closing_char" ]]; then - ((depth--)) - if (( depth == 0 )); then - printf '%s' "${input:value_start:cursor-value_start+1}" - return 0 - fi - fi - fi - - ((cursor++)) - done - - return 1 - fi - - pos="$after_key_pos" - done - - return 1 -} - -extract_json_integer_field_without_jq() { - local input="$1" - local target_key="$2" - local length=${#input} - local pos=0 - local char="" - local after_key_pos="" - local value_start="" - local value_end="" - local remainder="" - local integer_value="" - - while (( pos < length )); do - char="${input:pos:1}" - if [[ "$char" != '"' ]]; then - ((pos++)) - continue - fi - - if ! parse_json_string_without_jq "$input" "$pos"; then - return 1 - fi - - after_key_pos="$JSON_PARSE_NEXT_INDEX" - if [[ "$JSON_PARSE_VALUE" == "$target_key" ]]; then - after_key_pos=$(skip_json_whitespace_without_jq "$input" "$after_key_pos") - if [[ "${input:after_key_pos:1}" != ":" ]]; then - pos="$after_key_pos" - continue - fi - - value_start=$(skip_json_whitespace_without_jq "$input" "$((after_key_pos + 1))") - remainder="${input:value_start}" - if [[ "$remainder" =~ ^(-?[0-9]+) ]]; then - integer_value="${BASH_REMATCH[1]}" - if [[ "$integer_value" =~ ^[0-9]+$ ]]; then - printf '%s' "$integer_value" - return 0 - fi - return 1 - fi - - return 1 - fi - - pos="$after_key_pos" - done - - return 1 -} - -extract_json_string_array_without_jq() { - local input="$1" - local target_key="$2" - local array_segment="" - local length=0 - local pos=1 - local char="" - local comma_pos="" - local value="" - - array_segment=$(extract_json_field_segment_without_jq "$input" "$target_key" "[" "]") || return 1 - length=${#array_segment} - - while (( pos < length - 1 )); do - pos=$(skip_json_whitespace_without_jq "$array_segment" "$pos") - if (( pos >= length - 1 )); then - break - fi - - char="${array_segment:pos:1}" - if [[ "$char" != '"' ]]; then - return 1 - fi - - if ! parse_json_string_without_jq "$array_segment" "$pos"; then - return 1 - fi - - value="$JSON_PARSE_VALUE" - printf '%s\n' "$value" - pos="$JSON_PARSE_NEXT_INDEX" - pos=$(skip_json_whitespace_without_jq "$array_segment" "$pos") - if (( pos >= length - 1 )); then - break - fi - - if [[ "${array_segment:pos:1}" != "," ]]; then - return 1 - fi - - ((pos++)) - done - - return 0 -} - -load_rules_from_file_without_jq() { - local rules_text="" - local thresholds_segment="" - local jina_threshold="" - local markdown_threshold="" - local defuddle_threshold="" - local basic_threshold="" - local -a structured_keywords=() - local -a html_keywords=() - - rules_text=$(<"$RULES_FILE") || return 1 - thresholds_segment=$(extract_json_field_segment_without_jq "$rules_text" "thresholds" "{" "}") || return 1 - - jina_threshold=$(extract_json_integer_field_without_jq "$thresholds_segment" "jina") || return 1 - markdown_threshold=$(extract_json_integer_field_without_jq "$thresholds_segment" "markdown_new") || return 1 - defuddle_threshold=$(extract_json_integer_field_without_jq "$thresholds_segment" "defuddle") || return 1 - basic_threshold=$(extract_json_integer_field_without_jq "$thresholds_segment" "basic") || return 1 - - mapfile -t structured_keywords < <(extract_json_string_array_without_jq "$rules_text" "structured_error_keywords") || return 1 - mapfile -t html_keywords < <(extract_json_string_array_without_jq "$rules_text" "html_error_keywords") || return 1 - - if [[ ${#structured_keywords[@]} -eq 0 ]] || [[ ${#html_keywords[@]} -eq 0 ]]; then - return 1 - fi - - JINA_MIN_LENGTH="$jina_threshold" - MARKDOWN_NEW_MIN_LENGTH="$markdown_threshold" - DEFUDDLE_MIN_LENGTH="$defuddle_threshold" - BASIC_MIN_LENGTH="$basic_threshold" - STRUCTURED_ERROR_KEYWORDS=("${structured_keywords[@]}") - HTML_ERROR_KEYWORDS=("${html_keywords[@]}") -} - -skip_json_whitespace_without_jq() { - local input="$1" - local pos="$2" - local length=${#input} - local char="" - - while (( pos < length )); do - char="${input:pos:1}" - case "$char" in - $' ' | $'\t' | $'\r' | $'\n') ((pos++)) ;; - *) break ;; - esac - done - - printf '%s' "$pos" -} - -parse_json_string_without_jq() { - local input="$1" - local start="$2" - local length=${#input} - local pos=$((start + 1)) - local output="" - local char="" - local esc="" - local hex="" - - JSON_PARSE_VALUE="" - JSON_PARSE_NEXT_INDEX="" - - while (( pos < length )); do - char="${input:pos:1}" - if [[ "$char" == '"' ]]; then - JSON_PARSE_VALUE="$output" - JSON_PARSE_NEXT_INDEX=$((pos + 1)) - return 0 - fi - - if [[ "$char" == '\' ]]; then - ((pos++)) - if (( pos >= length )); then - return 1 - fi - esc="${input:pos:1}" - case "$esc" in - '"'|'\'|'/') output+="$esc" ;; - b) output+=$'\b' ;; - f) output+=$'\f' ;; - n) output+=$'\n' ;; - r) output+=$'\r' ;; - t) output+=$'\t' ;; - u) - if (( pos + 4 >= length )); then - return 1 - fi - hex="${input:pos+1:4}" - if [[ ! "$hex" =~ ^[0-9A-Fa-f]{4}$ ]]; then - return 1 - fi - output+=$(printf '%b' "\\u$hex") - ((pos+=4)) - ;; - *) - return 1 - ;; - esac - else - output+="$char" - fi - ((pos++)) - done - - return 1 -} - -extract_json_string_field_without_jq() { - local input="$1" - local target_key="$2" - local length=${#input} - local pos=0 - local char="" - local after_key_pos="" - local value_start="" - - while (( pos < length )); do - char="${input:pos:1}" - if [[ "$char" != '"' ]]; then - ((pos++)) - continue - fi - - if ! parse_json_string_without_jq "$input" "$pos"; then - return 1 - fi - after_key_pos="$JSON_PARSE_NEXT_INDEX" - if [[ "$JSON_PARSE_VALUE" == "$target_key" ]]; then - after_key_pos=$(skip_json_whitespace_without_jq "$input" "$after_key_pos") - if [[ "${input:after_key_pos:1}" == ":" ]]; then - value_start=$(skip_json_whitespace_without_jq "$input" "$((after_key_pos + 1))") - if [[ "${input:value_start:1}" == '"' ]] && parse_json_string_without_jq "$input" "$value_start"; then - printf '%s' "$JSON_PARSE_VALUE" - return 0 - fi - fi - fi - - pos="$after_key_pos" - done - - return 1 -} - -fetch_jina() { - local url="$1" - local response - local http_code - local headers_file="" - local body_file="" - - log_info "Trying Jina Reader" - set_last_fetch_error "" - - local jina_url - jina_url=$(build_jina_url "$url") - - # 判定策略：仅将 HTTP 非 2xx、结构化错误字段、空/过短响应视为失败，不再按全文关键词匹配。 - headers_file=$(create_tmp_file) - body_file=$(create_tmp_file) - if ! curl -sSL --max-time "$TIMEOUT" \ - -H "User-Agent: SmartWebFetch/1.0" \ - -D "$headers_file" \ - -o "$body_file" \ - "$jina_url"; then - set_last_fetch_error "Jina Reader request failed" - log_warn "Jina Reader request failed" - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - return 1 - fi - http_code=$(extract_http_status_code "$headers_file") - response=$(<"$body_file") - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - - if [[ ! "$http_code" =~ ^2[0-9][0-9]$ ]]; then - set_last_fetch_error "Jina Reader returned HTTP ${http_code:-unknown}" - log_warn "Jina Reader returned HTTP ${http_code:-unknown}" - return 1 - fi - - if is_structured_error_response "$response"; then - set_last_fetch_error "Jina Reader returned structured error payload" - log_warn "Jina Reader returned structured error payload" - return 1 - fi - - if [[ -z "$response" ]] || [[ ${#response} -lt "$JINA_MIN_LENGTH" ]]; then - set_last_fetch_error "Jina Reader returned empty or too-short content" - log_warn "Jina Reader returned empty or too-short content" - return 1 - fi - - log_success "Jina Reader succeeded" - printf '%s\n' "$response" -} - -fetch_markdown_new() { - local url="$1" - local response - local http_code - local content_type="" - local markdown="" - local payload="" - local headers_file="" - local body_file="" - - log_info "Trying markdown.new" - set_last_fetch_error "" - payload=$(build_url_json_payload "$url") - headers_file=$(create_tmp_file) - body_file=$(create_tmp_file) - - # 判定策略：将 HTTP 非 2xx、结构化错误字段、空/过短响应、以及可识别的 HTML 错误页视为失败。 - if ! curl -sSL --max-time "$TIMEOUT" \ - -X POST \ - -H "Content-Type: application/json" \ - -H "User-Agent: SmartWebFetch/1.0" \ - -d "$payload" \ - -D "$headers_file" \ - -o "$body_file" \ - "$MARKDOWN_NEW"; then - set_last_fetch_error "markdown.new request failed" - log_warn "markdown.new request failed" - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - return 1 - fi - - http_code=$(extract_http_status_code "$headers_file") - content_type=$(extract_content_type "$headers_file") - response=$(<"$body_file") - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - - if [[ ! "$http_code" =~ ^2[0-9][0-9]$ ]]; then - set_last_fetch_error "markdown.new returned HTTP ${http_code:-unknown}" - log_warn "markdown.new returned HTTP ${http_code:-unknown}" - return 1 - fi - - if is_likely_html_error_payload "$response" "$content_type"; then - set_last_fetch_error "markdown.new returned HTML error page (content-type: ${content_type:-unknown})" - log_warn "markdown.new returned HTML error page (content-type: ${content_type:-unknown})" - return 1 - fi - - if is_structured_error_response "$response"; then - set_last_fetch_error "markdown.new returned structured error payload" - log_warn "markdown.new returned structured error payload" - return 1 - fi - - if [[ -z "$response" ]] || [[ ${#response} -lt "$MARKDOWN_NEW_MIN_LENGTH" ]]; then - set_last_fetch_error "markdown.new returned empty or too-short content" - log_warn "markdown.new returned empty or too-short content" - return 1 - fi - - local compact_response="" - local is_json_object="0" - - compact_response=$(printf '%s' "$response" | tr -d '[:space:]') - if [[ "$compact_response" =~ ^\{ ]]; then - is_json_object="1" - fi - - if command -v jq >/dev/null 2>&1; then - markdown=$(printf '%s' "$response" | jq -r '.markdown // .content // .data // empty' 2>/dev/null || true) - elif [[ "$is_json_object" == "1" ]]; then - markdown=$(extract_markdown_from_json_without_jq "$response" || true) - fi - - if [[ "$is_json_object" == "1" ]] && ([[ -z "$markdown" ]] || [[ "$markdown" == "null" ]]); then - set_last_fetch_error "markdown.new returned JSON without usable markdown/content/data" - log_warn "markdown.new returned JSON without usable markdown/content/data" - return 1 - fi - - log_success "markdown.new succeeded" - - if [[ -n "$markdown" ]] && [[ "$markdown" != "null" ]]; then - printf '%s\n' "$markdown" - else - printf '%s\n' "$response" - fi -} - -fetch_defuddle() { - local url="$1" - local response - local payload="" - local http_code - local content_type="" - local markdown="" - local headers_file="" - local body_file="" - - log_info "Trying defuddle.md" - set_last_fetch_error "" - payload=$(build_url_json_payload "$url") - headers_file=$(create_tmp_file) - body_file=$(create_tmp_file) - - # 判定策略：将 HTTP 非 2xx、结构化错误字段、空/过短响应、以及可识别的 HTML 错误页视为失败。 - if ! curl -sSL --max-time "$TIMEOUT" \ - -X POST \ - -H "Content-Type: application/json" \ - -H "User-Agent: SmartWebFetch/1.0" \ - -d "$payload" \ - -D "$headers_file" \ - -o "$body_file" \ - "$DEFUDDLE_MD"; then - set_last_fetch_error "defuddle.md request failed" - log_warn "defuddle.md request failed" - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - return 1 - fi - - http_code=$(extract_http_status_code "$headers_file") - content_type=$(extract_content_type "$headers_file") - response=$(<"$body_file") - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - - if [[ ! "$http_code" =~ ^2[0-9][0-9]$ ]]; then - set_last_fetch_error "defuddle.md returned HTTP ${http_code:-unknown}" - log_warn "defuddle.md returned HTTP ${http_code:-unknown}" - return 1 - fi - - if is_likely_html_error_payload "$response" "$content_type"; then - set_last_fetch_error "defuddle.md returned HTML error page (content-type: ${content_type:-unknown})" - log_warn "defuddle.md returned HTML error page (content-type: ${content_type:-unknown})" - return 1 - fi - - if is_structured_error_response "$response"; then - set_last_fetch_error "defuddle.md returned structured error payload" - log_warn "defuddle.md returned structured error payload" - return 1 - fi - - if [[ -z "$response" ]] || [[ ${#response} -lt "$DEFUDDLE_MIN_LENGTH" ]]; then - set_last_fetch_error "defuddle.md returned empty or too-short content" - log_warn "defuddle.md returned empty or too-short content" - return 1 - fi - - local compact_response="" - local is_json_object="0" - - compact_response=$(printf '%s' "$response" | tr -d '[:space:]') - if [[ "$compact_response" =~ ^\{ ]]; then - is_json_object="1" - fi - - if command -v jq >/dev/null 2>&1; then - markdown=$(printf '%s' "$response" | jq -r '.markdown // .content // .data // empty' 2>/dev/null || true) - elif [[ "$is_json_object" == "1" ]]; then - markdown=$(extract_markdown_from_json_without_jq "$response" || true) - fi - - if [[ "$is_json_object" == "1" ]] && ([[ -z "$markdown" ]] || [[ "$markdown" == "null" ]]); then - set_last_fetch_error "defuddle.md returned JSON without usable markdown/content/data" - log_warn "defuddle.md returned JSON without usable markdown/content/data" - return 1 - fi - - log_success "defuddle.md succeeded" - - if [[ -n "$markdown" ]] && [[ "$markdown" != "null" ]]; then - printf '%s\n' "$markdown" - else - printf '%s\n' "$response" - fi -} - -fetch_basic() { - local url="$1" - local response - local processed_response - local http_code - local headers_file - local body_file - - log_info "Trying direct curl fallback" - set_last_fetch_error "" - headers_file=$(create_tmp_file) - body_file=$(create_tmp_file) - - if ! curl -sSL --max-time "$TIMEOUT" \ - -H "User-Agent: Mozilla/5.0" \ - -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \ - -D "$headers_file" \ - -o "$body_file" \ - "$url"; then - set_last_fetch_error "Direct curl fallback request failed" - log_warn "Direct curl fallback failed" - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - return 1 - fi - http_code=$(extract_http_status_code "$headers_file") - response=$(<"$body_file") - remove_tmp_file "$headers_file" - remove_tmp_file "$body_file" - - if [[ ! "$http_code" =~ ^2[0-9][0-9]$ ]]; then - set_last_fetch_error "Direct curl fallback returned HTTP ${http_code:-unknown}" - log_warn "Direct curl fallback returned HTTP ${http_code:-unknown}" - return 1 - fi - - if [[ -z "$response" ]] || [[ ${#response} -lt "$BASIC_MIN_LENGTH" ]]; then - set_last_fetch_error "Direct curl fallback returned empty or too-short content before cleanup" - log_warn "Direct curl fallback returned empty or too-short content" - return 1 - fi - - processed_response="$response" - - if [[ "$NO_CLEAN" != "1" ]]; then - processed_response=$(printf '%s' "$response" | clean_html) - fi - - if [[ -z "$processed_response" ]] || [[ ${#processed_response} -lt "$BASIC_MIN_LENGTH" ]]; then - set_last_fetch_error "Direct curl fallback returned empty or too-short content after cleanup" - log_warn "Direct curl fallback returned empty or too-short content after cleanup" - return 1 - fi - - if command -v html2text >/dev/null 2>&1; then - processed_response=$(printf '%s' "$processed_response" | html2text -utf8 2>/dev/null || printf '%s\n' "$processed_response") - elif command -v lynx >/dev/null 2>&1; then - processed_response=$(printf '%s' "$processed_response" | lynx -stdin -dump -nolist 2>/dev/null || printf '%s\n' "$processed_response") - fi - - if [[ -z "$processed_response" ]] || [[ ${#processed_response} -lt "$BASIC_MIN_LENGTH" ]]; then - set_last_fetch_error "Direct curl fallback returned empty or too-short content after conversion" - log_warn "Direct curl fallback returned empty or too-short content after conversion" - return 1 - fi - - printf '%s\n' "$processed_response" - log_success "Direct curl fallback succeeded" -} - -smart_fetch() { - local url="$1" - local specified_service="$2" - - if [[ -z "$url" ]]; then - log_error "Please provide a URL" - return 1 - fi - - if [[ ! "$url" =~ ^[Hh][Tt][Tt][Pp][Ss]?:// ]]; then - url="https://$url" - log_info "Added https:// prefix: $url" - fi - - log_info "Fetching $url" - - if [[ -n "$specified_service" ]]; then - case "$specified_service" in - jina) - fetch_jina "$url" && return 0 - ;; - markdown) - fetch_markdown_new "$url" && return 0 - ;; - defuddle) - fetch_defuddle "$url" && return 0 - ;; - *) - exit_with_help_error "Invalid service: ${specified_service}. Allowed values: jina|markdown|defuddle" - ;; - esac - - if [[ -n "$LAST_FETCH_ERROR" ]]; then - log_error "Forced service failed: ${specified_service}. ${LAST_FETCH_ERROR}" - else - log_error "Forced service failed: ${specified_service}" - fi - return 1 - fi - - fetch_jina "$url" && return 0 - fetch_markdown_new "$url" && return 0 - fetch_defuddle "$url" && return 0 - fetch_basic "$url" && return 0 - - if [[ -n "$LAST_FETCH_ERROR" ]]; then - log_error "All fetch methods failed. Last error: ${LAST_FETCH_ERROR}" - else - log_error "All fetch methods failed" - fi - return 1 -} - -URL="" -OUTPUT="" -SERVICE="" -VERBOSE="0" -NO_CLEAN="0" -LAST_FETCH_ERROR="" - -while [[ $# -gt 0 ]]; do - case "$1" in - -h|--help) - show_help - exit 0 - ;; - -o|--output) - validate_option_value "$1" "${2-}" - OUTPUT="${2-}" - shift 2 - ;; - -s|--service) - validate_option_value "$1" "${2-}" - validate_service_value "${2-}" - SERVICE="${2-}" - shift 2 - ;; - -v|--verbose) - VERBOSE="1" - shift - ;; - --no-clean) - NO_CLEAN="1" - shift - ;; - -*) - log_error "Unknown option: $1" - show_help - exit 1 - ;; - *) - if [[ -n "$URL" ]]; then - exit_with_help_error "Only one URL argument is allowed" - fi - URL="$1" - shift - ;; - esac -done - -if [[ -z "$URL" ]]; then - log_error "Please provide a URL" - show_help - exit 1 -fi - -check_runtime_dependencies -load_rules_from_file - -if [[ -n "$OUTPUT" ]]; then - OUTPUT_DIR=$(dirname "$OUTPUT") - OUTPUT_BASENAME=$(basename "$OUTPUT") - - if [[ ! -d "$OUTPUT_DIR" ]]; then - if ! mkdir -p "$OUTPUT_DIR"; then - log_error "Failed to create output directory: $OUTPUT_DIR" - exit 1 - fi - fi - - TMP_OUTPUT=$(mktemp "$OUTPUT_DIR/.${OUTPUT_BASENAME}.tmp.XXXXXX") - TMP_FILES+=("$TMP_OUTPUT") - - if smart_fetch "$URL" "$SERVICE" > "$TMP_OUTPUT"; then - mv "$TMP_OUTPUT" "$OUTPUT" - remove_tmp_file "$TMP_OUTPUT" - log_success "Saved output to: $OUTPUT" - else - remove_tmp_file "$TMP_OUTPUT" - exit 1 - fi -else - smart_fetch "$URL" "$SERVICE" -fi diff --git a/skills/smart-web-fetch/scripts/smart-web-fetch-core.ps1 b/skills/smart-web-fetch/scripts/smart-web-fetch-core.ps1 deleted file mode 100644 index 0c6ad89..0000000 --- a/skills/smart-web-fetch/scripts/smart-web-fetch-core.ps1 +++ /dev/null @@ -1,756 +0,0 @@ -param( - [Parameter(Position = 0)] - [string]$Url, - [Alias('o')] - [string]$Output, - [Alias('s')] - [ValidateSet('jina', 'markdown', 'defuddle')] - [string]$Service, - [Alias('v')] - [switch]$VerboseMode, - [switch]$NoClean, - [Alias('h')] - [switch]$Help, - [Parameter(ValueFromRemainingArguments = $true)] - [string[]]$ExtraArgs -) - -$ErrorActionPreference = 'Stop' -$TimeoutSec = 30 -$JinaReaderBase = 'https://r.jina.ai' -$MarkdownNew = 'https://api.markdown.new/api/v1/convert' -$DefuddleMd = 'https://defuddle.md/api/convert' -$JinaMinLength = 100 -$MarkdownNewMinLength = 40 -$DefuddleMinLength = 40 -$BasicMinLength = 40 -$script:StructuredErrorKeywords = @( - 'error', - 'fail', - 'invalid', - 'unauthorized', - 'forbidden', - 'denied', - 'blocked', - 'not found', - 'rate limit', - 'too many requests' -) -$script:HtmlErrorKeywords = @( - 'access denied', - 'forbidden', - 'captcha', - 'cloudflare', - 'just a moment', - 'unauthorized', - 'bad gateway', - 'gateway timeout', - 'service unavailable' -) -$SkillRoot = Split-Path -Parent $PSScriptRoot -$RulesFile = Join-Path -Path $SkillRoot -ChildPath 'assets/fetch-rules.json' -$script:LastFetchError = $null - -function Load-RulesFromFile { - if (-not (Test-Path -LiteralPath $RulesFile)) { - return - } - - try { - $rules = Get-Content -LiteralPath $RulesFile -Raw | ConvertFrom-Json -ErrorAction Stop - - if ($rules.thresholds.jina -as [int]) { $script:JinaMinLength = [int]$rules.thresholds.jina } - if ($rules.thresholds.markdown_new -as [int]) { $script:MarkdownNewMinLength = [int]$rules.thresholds.markdown_new } - if ($rules.thresholds.defuddle -as [int]) { $script:DefuddleMinLength = [int]$rules.thresholds.defuddle } - if ($rules.thresholds.basic -as [int]) { $script:BasicMinLength = [int]$rules.thresholds.basic } - if ($rules.structured_error_keywords -and $rules.structured_error_keywords.Count -gt 0) { - $script:StructuredErrorKeywords = @($rules.structured_error_keywords | ForEach-Object { [string]$_ }) - } - if ($rules.html_error_keywords -and $rules.html_error_keywords.Count -gt 0) { - $script:HtmlErrorKeywords = @($rules.html_error_keywords | ForEach-Object { [string]$_ }) - } - - Write-Info "Loaded thresholds and keywords from rules file: $RulesFile" - } catch { - Write-WarnLog "Failed to parse rules file, using built-in defaults: $RulesFile" - } -} - -function Show-Help { - @" -Smart Web Fetch - native PowerShell web-to-Markdown fetcher - -Usage: - smart-web-fetch.ps1 [options] - -Options: - -Help, -h, --help Show help - -Output Write output to file - -Service Force service: jina|markdown|defuddle - -VerboseMode Show verbose logs - -NoClean Skip HTML cleanup in the basic fallback - -Examples: - ./scripts/smart-web-fetch.ps1 https://example.com - ./scripts/smart-web-fetch.ps1 https://example.com -Output output.md - ./scripts/smart-web-fetch.ps1 https://example.com -Service jina - ./scripts/smart-web-fetch.ps1 https://example.com -NoClean -"@ -} - -function Write-Info([string]$Message) { - if ($VerboseMode) { - [Console]::Error.WriteLine("[INFO] $Message") - } -} - -function Write-Success([string]$Message) { - if ($VerboseMode) { - [Console]::Error.WriteLine("[SUCCESS] $Message") - } -} - -function Write-WarnLog([string]$Message) { - if ($VerboseMode) { - [Console]::Error.WriteLine("[WARN] $Message") - } -} - -function Write-ErrorLog([string]$Message) { - [Console]::Error.WriteLine("[ERROR] $Message") -} - -function Write-DependencyCheckError([string]$Message) { - Write-ErrorLog "[Dependency] $Message" -} - -function Test-RuntimeDependencies { - $failures = New-Object System.Collections.Generic.List[string] - - if (-not $PSVersionTable.PSVersion -or $PSVersionTable.PSVersion.Major -lt 7) { - $actual = if ($PSVersionTable.PSVersion) { $PSVersionTable.PSVersion.ToString() } else { 'Unknown' } - $failures.Add("Missing required runtime: PowerShell 7+ (current: $actual)") - } - - $invokeWebRequestCmd = Get-Command Invoke-WebRequest -ErrorAction SilentlyContinue - if (-not $invokeWebRequestCmd) { - $failures.Add('Missing required command: Invoke-WebRequest') - } - - $jq = Get-OptionalCommand 'jq' - if ($jq) { - Write-Info '[Dependency] Optional dependency detected: jq (JSON markdown extraction)' - } else { - Write-WarnLog '[Dependency] Optional dependency not found: jq (fallback to ConvertFrom-Json/native parsing)' - } - - $perl = Get-OptionalCommand 'perl' - if ($perl) { - Write-Info '[Dependency] Optional dependency detected: perl (enhanced HTML cleanup)' - } else { - Write-WarnLog '[Dependency] Optional dependency not found: perl (fallback to PowerShell regex cleanup)' - } - - $html2text = Get-OptionalCommand 'html2text' - $lynx = Get-OptionalCommand 'lynx' - if ($html2text -or $lynx) { - Write-Info '[Dependency] Optional dependency detected: html2text/lynx (HTML-to-text fallback)' - } else { - Write-WarnLog '[Dependency] Optional dependency not found: html2text or lynx (fallback returns cleaned HTML)' - } - - if ($failures.Count -gt 0) { - foreach ($failure in $failures) { - Write-DependencyCheckError $failure - } - - throw 'Dependency check failed. Please install required dependencies and retry.' - } -} - -function Set-LastFetchError([string]$Message) { - $script:LastFetchError = $Message -} - -function Get-RequestFailureSummary([System.Management.Automation.ErrorRecord]$ErrorRecord) { - if (-not $ErrorRecord) { - return 'Unknown request failure' - } - - $exception = $ErrorRecord.Exception - $parts = @() - - if ($null -ne $exception.Response) { - try { - $statusCode = [int]$exception.Response.StatusCode - if ($statusCode) { - $parts += "HTTP $statusCode" - } - } catch { - } - - try { - $reasonPhrase = $exception.Response.ReasonPhrase - if (-not [string]::IsNullOrWhiteSpace($reasonPhrase)) { - $parts += $reasonPhrase.Trim() - } - } catch { - } - } - - if (-not [string]::IsNullOrWhiteSpace($exception.Message)) { - $parts += $exception.Message.Trim() - } - - if ($exception.InnerException -and -not [string]::IsNullOrWhiteSpace($exception.InnerException.Message)) { - $parts += $exception.InnerException.Message.Trim() - } - - $summary = ($parts | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Select-Object -Unique) -join ' - ' - if ([string]::IsNullOrWhiteSpace($summary)) { - return 'Unknown request failure' - } - - return $summary -} - -function Ensure-Url([string]$Value) { - if ([string]::IsNullOrWhiteSpace($Value)) { - throw 'Please provide a URL' - } - - if ($Value -notmatch '^https?://') { - $Value = "https://$Value" - Write-Info "Added https:// prefix: $Value" - } - - return $Value -} - -function Get-JinaRequestUrl([string]$TargetUrl) { - $urlWithoutScheme = [regex]::Replace($TargetUrl, '^(?i)https?://', '') - $scheme = 'http' - - try { - $uri = [Uri]$TargetUrl - if ($uri.Scheme -ieq 'https') { - $scheme = 'https' - } - } catch { - } - - return "$JinaReaderBase/${scheme}://$urlWithoutScheme" -} - -function Invoke-Request([string]$RequestUrl, [string]$Method = 'GET', [hashtable]$Headers = $null, [string]$Body = $null) { - $params = @{ - Uri = $RequestUrl - Method = $Method - TimeoutSec = $TimeoutSec - UseBasicParsing = $true - } - - if ($Headers) { - $params.Headers = $Headers - } - - if ($null -ne $Body) { - $params.Body = $Body - } - - $response = Invoke-WebRequest @params - $contentType = $null - if ($response.Headers -and $response.Headers['Content-Type']) { - $contentType = [string]$response.Headers['Content-Type'] - } - - return [PSCustomObject]@{ - Content = [string]$response.Content - StatusCode = [int]$response.StatusCode - ContentType = $contentType - } -} - -function Normalize-KeywordText([string]$Text) { - if ($null -eq $Text) { - return '' - } - - $normalized = $Text.ToLowerInvariant() - $normalized = $normalized -replace '[_-]', ' ' - $normalized = $normalized -replace '\s+', ' ' - return $normalized.Trim() -} - -function Test-ContainsKeyword([string]$Text, [string[]]$Keywords) { - $normalizedText = Normalize-KeywordText $Text - if ([string]::IsNullOrWhiteSpace($normalizedText)) { - return $false - } - - foreach ($keyword in $Keywords) { - $normalizedKeyword = Normalize-KeywordText ([string]$keyword) - if ([string]::IsNullOrWhiteSpace($normalizedKeyword)) { - continue - } - - if ($normalizedText.Contains($normalizedKeyword)) { - return $true - } - } - - return $false -} - -function Test-LikelyHtmlErrorPayload([string]$Content, [string]$ContentType) { - if ([string]::IsNullOrWhiteSpace($ContentType)) { - return $false - } - - $lowerContentType = $ContentType.ToLowerInvariant() - if (-not ($lowerContentType.Contains('text/html') -or $lowerContentType.Contains('application/xhtml+xml'))) { - return $false - } - - $lowerContent = $Content.ToLowerInvariant() - if (-not ($lowerContent.Contains('$null | Out-String).TrimEnd() - if (-not [string]::IsNullOrWhiteSpace($markdown) -and $markdown -ne 'null') { - Write-Info 'Using jq for markdown.new JSON extraction' - return $markdown - } - } catch { - Write-WarnLog 'jq extraction failed; falling back to ConvertFrom-Json' - } - - return $null -} - -function Get-MarkdownFieldValue($ParsedJson) { - if ($null -eq $ParsedJson) { - return $null - } - - $markdown = $ParsedJson.markdown - if ([string]::IsNullOrWhiteSpace($markdown)) { $markdown = $ParsedJson.content } - if ([string]::IsNullOrWhiteSpace($markdown)) { $markdown = $ParsedJson.data } - - if ([string]::IsNullOrWhiteSpace($markdown)) { - return $null - } - - return [string]$markdown -} - -function Try-CleanHtmlWithPerl([string]$Html) { - $perl = Get-OptionalCommand 'perl' - if (-not $perl) { - return $null - } - - $perlScript = 's{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{]*>.*?}{}gsi; s{}{}gsi; s{\s(?:class|id|style|on\w+)=("[^"]*"|''[^'']*'')}{}gsi;' - - try { - $cleaned = ($Html | & $perl.Source -0pe $perlScript 2>$null | Out-String).TrimEnd() - Write-Info 'Using perl for HTML cleanup' - return $cleaned - } catch { - Write-WarnLog 'perl HTML cleanup failed; falling back to PowerShell regex cleanup' - return $null - } -} - -function Clean-Html([string]$Html) { - $perlCleaned = Try-CleanHtmlWithPerl $Html - if ($null -ne $perlCleaned) { - return $perlCleaned - } - - $cleaned = $Html - $patterns = @( - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)]*>.*?', - '(?is)' - ) - - foreach ($pattern in $patterns) { - $cleaned = [regex]::Replace($cleaned, $pattern, '') - } - - $cleaned = [regex]::Replace($cleaned, '\s(?:class|id|style|on\w+)=("[^"]*"|''[^'']*'')', '', 'IgnoreCase') - return $cleaned -} - -function Convert-HtmlFallback([string]$Html) { - $html2text = Get-Command html2text -ErrorAction SilentlyContinue - if ($html2text) { - try { - return ($Html | & $html2text.Source -utf8 2>$null | Out-String).TrimEnd() - } catch { - } - } - - $lynx = Get-Command lynx -ErrorAction SilentlyContinue - if ($lynx) { - try { - return ($Html | & $lynx.Source -stdin -dump -nolist 2>$null | Out-String).TrimEnd() - } catch { - } - } - - return $Html -} - -function Fetch-Jina([string]$TargetUrl) { - Write-Info 'Trying Jina Reader' - Set-LastFetchError $null - try { - $jinaRequestUrl = Get-JinaRequestUrl $TargetUrl - $request = Invoke-Request -RequestUrl $jinaRequestUrl -Headers @{ 'User-Agent' = 'SmartWebFetch/1.0' } - $response = $request.Content - if (Test-InvalidContent -Content $response -MinLength $JinaMinLength -ContentType $request.ContentType) { - Set-LastFetchError 'Jina Reader returned invalid or incomplete content' - Write-WarnLog $script:LastFetchError - return $null - } - - Write-Success 'Jina Reader succeeded' - return $response - } catch { - Set-LastFetchError "Jina Reader request failed: $(Get-RequestFailureSummary $_)" - Write-WarnLog $script:LastFetchError - return $null - } -} - -function Fetch-MarkdownNew([string]$TargetUrl) { - Write-Info 'Trying markdown.new' - Set-LastFetchError $null - try { - $body = @{ url = $TargetUrl } | ConvertTo-Json -Compress - $request = Invoke-Request -RequestUrl $MarkdownNew -Method 'POST' -Headers @{ - 'Content-Type' = 'application/json' - 'User-Agent' = 'SmartWebFetch/1.0' - } -Body $body - $response = $request.Content - - if (Test-InvalidContent -Content $response -MinLength $MarkdownNewMinLength -ContentType $request.ContentType -ExpectJsonResponse) { - Set-LastFetchError "markdown.new returned invalid content (content-type: $($request.ContentType))" - Write-WarnLog $script:LastFetchError - return $null - } - - $parsedJson = $null - $isJsonResponse = $false - try { - $parsedJson = $response | ConvertFrom-Json -ErrorAction Stop - $isJsonResponse = $true - } catch { - } - - $markdown = Try-ExtractMarkdownWithJq $response - if ([string]::IsNullOrWhiteSpace($markdown) -and $isJsonResponse) { - $markdown = Get-MarkdownFieldValue $parsedJson - } - - if ($isJsonResponse -and [string]::IsNullOrWhiteSpace($markdown)) { - Set-LastFetchError 'markdown.new returned JSON without usable markdown/content/data' - Write-WarnLog $script:LastFetchError - return $null - } - - Write-Success 'markdown.new succeeded' - if (-not [string]::IsNullOrWhiteSpace($markdown)) { - return [string]$markdown - } - - return $response - } catch { - Set-LastFetchError "markdown.new request failed: $(Get-RequestFailureSummary $_)" - Write-WarnLog $script:LastFetchError - return $null - } -} - -function Fetch-Defuddle([string]$TargetUrl) { - Write-Info 'Trying defuddle.md' - Set-LastFetchError $null - try { - $body = @{ url = $TargetUrl } | ConvertTo-Json -Compress - $request = Invoke-Request -RequestUrl $DefuddleMd -Method 'POST' -Headers @{ - 'Content-Type' = 'application/json' - 'User-Agent' = 'SmartWebFetch/1.0' - } -Body $body - $response = $request.Content - - if (Test-InvalidContent -Content $response -MinLength $DefuddleMinLength -ContentType $request.ContentType -ExpectJsonResponse) { - Set-LastFetchError "defuddle.md returned invalid content (content-type: $($request.ContentType))" - Write-WarnLog $script:LastFetchError - return $null - } - - $parsedJson = $null - $isJsonResponse = $false - try { - $parsedJson = $response | ConvertFrom-Json -ErrorAction Stop - $isJsonResponse = $true - } catch { - } - - $markdown = Try-ExtractMarkdownWithJq $response - if ([string]::IsNullOrWhiteSpace($markdown) -and $isJsonResponse) { - $markdown = Get-MarkdownFieldValue $parsedJson - } - - if ($isJsonResponse -and [string]::IsNullOrWhiteSpace($markdown)) { - Set-LastFetchError 'defuddle.md returned JSON without usable markdown/content/data' - Write-WarnLog $script:LastFetchError - return $null - } - - Write-Success 'defuddle.md succeeded' - if (-not [string]::IsNullOrWhiteSpace($markdown)) { - return [string]$markdown - } - - return $response - } catch { - Set-LastFetchError "defuddle.md request failed: $(Get-RequestFailureSummary $_)" - Write-WarnLog $script:LastFetchError - return $null - } -} - -function Fetch-Basic([string]$TargetUrl) { - Write-Info 'Trying basic fallback' - Set-LastFetchError $null - try { - $request = Invoke-Request -RequestUrl $TargetUrl -Headers @{ - 'User-Agent' = 'Mozilla/5.0' - 'Accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' - } - $response = $request.Content - - if ([string]::IsNullOrWhiteSpace($response) -or $response.Length -lt $BasicMinLength) { - Set-LastFetchError 'Basic fallback returned invalid or incomplete content' - Write-WarnLog $script:LastFetchError - return $null - } - - $processed = $response - if (-not $NoClean) { - $processed = Clean-Html $processed - } - - if ([string]::IsNullOrWhiteSpace($processed) -or $processed.Length -lt $BasicMinLength) { - Set-LastFetchError 'Basic fallback returned invalid or incomplete content after cleanup' - Write-WarnLog $script:LastFetchError - return $null - } - - $result = Convert-HtmlFallback $processed - - if ([string]::IsNullOrWhiteSpace($result) -or $result.Length -lt $BasicMinLength) { - Set-LastFetchError 'Basic fallback returned invalid or incomplete content after conversion' - Write-WarnLog $script:LastFetchError - return $null - } - - Write-Success 'Basic fallback succeeded' - return $result - } catch { - Set-LastFetchError "Basic fallback failed: $(Get-RequestFailureSummary $_)" - Write-WarnLog $script:LastFetchError - return $null - } -} - -function Smart-Fetch([string]$TargetUrl, [string]$ForcedService) { - $normalizedUrl = Ensure-Url $TargetUrl - Write-Info "Fetching $normalizedUrl" - - if ($ForcedService) { - switch ($ForcedService) { - 'jina' { - $result = Fetch-Jina $normalizedUrl - if ($result) { return $result } - } - 'markdown' { - $result = Fetch-MarkdownNew $normalizedUrl - if ($result) { return $result } - } - 'defuddle' { - $result = Fetch-Defuddle $normalizedUrl - if ($result) { return $result } - } - } - - if ($script:LastFetchError) { - throw "Forced service failed: $ForcedService. $script:LastFetchError" - } - - throw "Forced service failed: $ForcedService" - } - - foreach ($fetcher in @( - { param($u) Fetch-Jina $u }, - { param($u) Fetch-MarkdownNew $u }, - { param($u) Fetch-Defuddle $u }, - { param($u) Fetch-Basic $u } - )) { - $result = & $fetcher $normalizedUrl - if ($result) { - return $result - } - } - - if ($script:LastFetchError) { - throw "All fetch methods failed. Last error: $script:LastFetchError" - } - - throw 'All fetch methods failed' -} - -function Validate-ExtraArgs([string[]]$Args, [bool]$HelpRequested) { - if (-not $Args -or $Args.Count -eq 0) { - return - } - - $helpTokens = @('-h', '-help', '--help') - foreach ($arg in $Args) { - if ($helpTokens -contains $arg) { - continue - } - - if ($arg -match '^-') { - throw "Unknown option: $arg" - } - - if ($arg -match '^(?i)https?://|^[A-Za-z0-9][A-Za-z0-9.-]*\.[A-Za-z]{2,}([/:?#].*)?$') { - throw 'Only one URL argument is allowed' - } - - throw "Unexpected argument: $arg" - } - - if (-not $HelpRequested -and ($Args | Where-Object { $helpTokens -contains $_ })) { - throw 'Help flag must be used without extra positional arguments' - } -} - -# Compatibility: allow GNU-style --help to reach Show-Help in PowerShell. -if (-not $Help) { - $helpTokens = @('-h', '-help', '--help') - - if ($helpTokens -contains $Url) { - $Help = $true - $Url = $null - } elseif ($ExtraArgs) { - foreach ($arg in $ExtraArgs) { - if ($helpTokens -contains $arg) { - $Help = $true - break - } - } - } -} - -try { - Validate-ExtraArgs -Args $ExtraArgs -HelpRequested $Help - - if ($Help) { - Show-Help - exit 0 - } - - if ([string]::IsNullOrWhiteSpace($Url)) { - Show-Help - throw 'Please provide a URL' - } - - Test-RuntimeDependencies - Load-RulesFromFile - - $content = Smart-Fetch -TargetUrl $Url -ForcedService $Service - - if ($Output) { - $parent = Split-Path -Parent $Output - if (-not [string]::IsNullOrWhiteSpace($parent) -and -not (Test-Path -LiteralPath $parent)) { - New-Item -ItemType Directory -Path $parent -Force | Out-Null - } - - $content | Set-Content -Encoding utf8NoBOM $Output - Write-Success "Saved output to: $Output" - } else { - $content - } -} catch { - Write-ErrorLog $_.Exception.Message - exit 1 -} diff --git a/skills/smart-web-fetch/scripts/smart-web-fetch.cmd b/skills/smart-web-fetch/scripts/smart-web-fetch.cmd index bd3b18b..5f64aad 100644 --- a/skills/smart-web-fetch/scripts/smart-web-fetch.cmd +++ b/skills/smart-web-fetch/scripts/smart-web-fetch.cmd @@ -1 +1,70 @@ -@pwsh -File "%~dp0smart-web-fetch.ps1" %* +@echo off +setlocal EnableExtensions EnableDelayedExpansion + +set "SCRIPT_DIR=%~dp0" +for %%I in ("%SCRIPT_DIR%..") do set "SKILL_DIR=%%~fI" +set "BOOTSTRAP_PATH=%SKILL_DIR%\main.py" +set "ERROR_MESSAGE=smart-web-fetch: error: Python 3.11+ was not found. Install Python 3.11 or newer and ensure a compatible interpreter is on PATH." +set "PY_SELECTOR=" +set /a BEST_MAJOR=-1 +set /a BEST_MINOR=-1 + +where py >nul 2>nul +if not errorlevel 1 ( + for /f "tokens=2 delims=:" %%I in ('py -0p 2^>nul ^| findstr /R /C:"-V:[0-9][0-9]*\.[0-9][0-9]*"') do call :consider_py "%%~I" +) +if defined PY_SELECTOR goto :run_py + +where python >nul 2>nul +if not errorlevel 1 ( + goto :run_python +) + +>&2 echo %ERROR_MESSAGE% +exit /b 1 + +:run_py +set "MODULE_SELECTOR=-%PY_SELECTOR%" +goto :run_module + +:run_python +set "MODULE_SELECTOR=" +goto :run_module + +:run_module +setlocal DisableDelayedExpansion +if defined MODULE_SELECTOR ( + py %MODULE_SELECTOR% "%BOOTSTRAP_PATH%" %* +) else ( + python "%BOOTSTRAP_PATH%" %* +) +set "EXITCODE=%errorlevel%" +endlocal & exit /b %EXITCODE% + +:consider_py +set "candidate=%~1" +for /f "tokens=1 delims= " %%A in ("%candidate%") do set "candidate=%%~A" +for /f "tokens=1,2 delims=.-[]" %%A in ("%candidate%") do ( + set "major=%%~A" + set "minor=%%~B" +) +if not defined major goto :eof +if not defined minor goto :eof + +2>nul set /a major_num=major +if errorlevel 1 goto :eof +2>nul set /a minor_num=minor +if errorlevel 1 goto :eof + +if !major_num! LSS 3 goto :eof +if !major_num! EQU 3 if !minor_num! LSS 11 goto :eof + +if !major_num! GTR !BEST_MAJOR! goto :set_best +if !major_num! EQU !BEST_MAJOR! if !minor_num! GTR !BEST_MINOR! goto :set_best +goto :eof + +:set_best +set /a BEST_MAJOR=!major_num! +set /a BEST_MINOR=!minor_num! +set "PY_SELECTOR=!major_num!.!minor_num!" +goto :eof diff --git a/skills/smart-web-fetch/scripts/smart-web-fetch.ps1 b/skills/smart-web-fetch/scripts/smart-web-fetch.ps1 index e63b776..3c99106 100644 --- a/skills/smart-web-fetch/scripts/smart-web-fetch.ps1 +++ b/skills/smart-web-fetch/scripts/smart-web-fetch.ps1 @@ -1,115 +1,75 @@ -#Requires -Version 7 -# smart-web-fetch.ps1 — unified PowerShell entry point -# Accepts both POSIX-style (--no-clean, --verbose) and native PowerShell names, -# then forwards to smart-web-fetch-core.ps1 via splatting. +$ErrorActionPreference = 'Stop' -param( - [Parameter(Position = 0)] - [string]$Url, +$SkillDir = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +$BootstrapPath = Join-Path $SkillDir 'main.py' +$ScriptArgs = $args +$VersionCheck = 'import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' +$ErrorMessage = "smart-web-fetch: error: Python 3.11+ was not found. Install Python 3.11 or newer and ensure a compatible interpreter is on PATH." - [Alias('o')] - [string]$Output, - - [Alias('s')] - [string]$Service, +function Get-PyLauncherSelector { + try { + $installed = & py -0p 2>$null | Out-String + if ($LASTEXITCODE -ne 0) { + return $null + } + } catch { + return $null + } - [Alias('v')] - [switch]$VerboseMode, + $bestMajor = -1 + $bestMinor = -1 + $bestSelector = $null - [Alias('h')] - [switch]$Help, + foreach ($match in [regex]::Matches($installed, '-V:(?\d+)\.(?\d+)')) { + $major = [int]$match.Groups['major'].Value + $minor = [int]$match.Groups['minor'].Value + if ($major -lt 3 -or ($major -eq 3 -and $minor -lt 11)) { + continue + } + if ($major -gt $bestMajor -or ($major -eq $bestMajor -and $minor -gt $bestMinor)) { + $bestMajor = $major + $bestMinor = $minor + $bestSelector = "$major.$minor" + } + } - # Accept POSIX-style --no-clean in addition to -NoClean - [switch]$NoClean, + return $bestSelector +} - # Catch any remaining arguments so we can handle POSIX-style long options - [Parameter(ValueFromRemainingArguments)] - [string[]]$ExtraArgs -) +function Test-PythonCommand { + param( + [Parameter(Mandatory = $true)] + [string]$Command, + [string[]]$PrefixArgs = @() + ) -# Handle POSIX-style long options that PowerShell leaves in $ExtraArgs. -# Supported: -# --no-clean -# --verbose -# --help -# --output / --output= -# --service / --service= -if ($ExtraArgs) { - for ($i = 0; $i -lt $ExtraArgs.Count; $i++) { - $a = $ExtraArgs[$i] + try { + & $Command @PrefixArgs -c $VersionCheck *> $null + return ($LASTEXITCODE -eq 0) + } catch { + return $false + } +} - if ($a -notmatch '^-') { - if ([string]::IsNullOrWhiteSpace($Url)) { - $Url = $a - continue - } +function Invoke-Bootstrap { + param( + [Parameter(Mandatory = $true)] + [string]$Command, + [string[]]$PrefixArgs = @() + ) - Write-Error "smart-web-fetch: unexpected extra argument: $a" - exit 1 - } + & $Command @PrefixArgs $BootstrapPath @ScriptArgs +} - switch -Regex ($a) { - '^--no-clean$' { - $NoClean = $true - continue - } - '^--verbose$' { - $VerboseMode = $true - continue - } - '^--help$|^-h$' { - $Help = $true - continue - } - '^--output=(.+)$' { - $Output = $Matches[1] - continue - } - '^--service=(.+)$' { - $Service = $Matches[1] - continue - } - '^--output$' { - if ($i + 1 -ge $ExtraArgs.Count) { - Write-Error 'smart-web-fetch: missing value for --output' - exit 1 - } - $i++ - if ($ExtraArgs[$i] -match '^-') { - Write-Error 'smart-web-fetch: missing value for --output' - exit 1 - } - $Output = $ExtraArgs[$i] - continue - } - '^--service$' { - if ($i + 1 -ge $ExtraArgs.Count) { - Write-Error 'smart-web-fetch: missing value for --service' - exit 1 - } - $i++ - if ($ExtraArgs[$i] -match '^-') { - Write-Error 'smart-web-fetch: missing value for --service' - exit 1 - } - $Service = $ExtraArgs[$i] - continue - } - default { - Write-Error "smart-web-fetch: unknown argument: $a" - exit 1 - } - } - } +if ($PySelector = Get-PyLauncherSelector) { + Invoke-Bootstrap -Command 'py' -PrefixArgs @("-$PySelector") + exit $LASTEXITCODE } -$splatArgs = @{} -if ($Url) { $splatArgs['Url'] = $Url } -if ($Output) { $splatArgs['Output'] = $Output } -if ($Service) { $splatArgs['Service'] = $Service } -if ($VerboseMode) { $splatArgs['VerboseMode'] = $true } -if ($Help) { $splatArgs['Help'] = $true } -if ($NoClean) { $splatArgs['NoClean'] = $true } +if (Test-PythonCommand -Command 'python') { + Invoke-Bootstrap -Command 'python' + exit $LASTEXITCODE +} -& "$PSScriptRoot/smart-web-fetch-core.ps1" @splatArgs -exit $LASTEXITCODE +[Console]::Error.WriteLine($ErrorMessage) +exit 1 diff --git a/spec/fetch-contract.md b/spec/fetch-contract.md index 70addd6..c945806 100644 --- a/spec/fetch-contract.md +++ b/spec/fetch-contract.md @@ -1,10 +1,10 @@ # Smart Web Fetch — 行为契约 -本文件是 `smart-web-fetch`（Bash）与 `smart-web-fetch.ps1`（PowerShell）在抓取判定上的统一规范。新增服务源或调整判定规则时，应先更新本文件，再同步修改两个脚本。 +本文件是 `smart-web-fetch` 的统一抓取判定规范。新增服务源或调整判定规则时，应先更新本文件，再同步修改 `core/` Python 包。包装器运行时发现逻辑与 smoke test 维护基线见 `spec/wrapper-runtime.md`；包装器会通过技能根目录下的 `main.py` bootstrap 引导内部 `core/` 包。 ## 1. 规则来源 -阈值与错误关键词集中管理于 `skills/smart-web-fetch/assets/fetch-rules.json`。两个脚本优先加载此文件；文件缺失或解析失败时回退到脚本内置默认值。 +阈值与错误关键词集中管理于 `skills/smart-web-fetch/assets/fetch-rules.json`。由技能根目录 `main.py` 启动的 Python core 优先加载此文件；文件缺失或解析失败时回退到内置默认值。 ## 2. 服务源与降级顺序 @@ -15,11 +15,16 @@ | 3 | `defuddle` | POST | `defuddle.md/api/convert` | | 4 | `basic fallback` | GET | 原始 URL，本地 HTML 清洗 | -显式指定服务源（`-s` / `-Service`）时，仅尝试该源；失败直接报错，不继续降级。 +显式指定服务源（`-s` / `--service`）时，仅尝试该源；失败直接报错，不继续降级。`-Service` 不受支持，应按未知参数报错。 ## 3. URL 归一化 -输入若不带 `http://` 或 `https://`（大小写不敏感），统一自动补 `https://`。 +输入 URL 按“解析 + 校验”执行： + +- 若不带 scheme，自动补 `https://` +- 仅允许 `http://` 与 `https://` +- 其他 scheme（如 `ftp://`）直接判定失败，不进入抓取链路 +- URL 缺少 host 时直接判定失败 ## 4. 成功判定 @@ -54,22 +59,30 @@ `Content-Type` 为 HTML，且内容含 `Basic Success" + "
" + f"

Title

{MARKDOWN_SUCCESS}

{MARKDOWN_SUCCESS}

" + "
" + ) + self._write(200, html, "text/html") + return + + if path == "/basic-short": + self._write(200, TOO_SHORT, "text/plain") + return + + if path == "/basic-binary": + self._write_bytes(200, BASIC_BINARY, "image/png") + return + + self._write(404, "not found", "text/plain") + + def do_POST(self): + path = urlparse(self.path).path + length = int(self.headers.get("Content-Length", "0")) + if length: + self.rfile.read(length) + + if path in {"/markdown-success", "/defuddle-success"}: + self._write(200, (FIXTURE_DIR / "markdown-success.json").read_text(encoding="utf-8"), "application/json") + return + + if path in {"/markdown-short", "/defuddle-short"}: + self._write(200, SHORT_EXTRACTED_JSON, "application/json") + return + + if path in {"/markdown-error", "/defuddle-error"}: + self._write(200, STRUCTURED_ERROR, "application/json") + return + + self._write(404, "not found", "text/plain") + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--port", type=int, required=True) + args = parser.parse_args() + + server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler) + try: + server.serve_forever() + finally: + server.server_close() + + +if __name__ == "__main__": + main() diff --git a/spec/tests/json-smoke.ps1 b/spec/tests/json-smoke.ps1 new file mode 100644 index 0000000..c78c795 --- /dev/null +++ b/spec/tests/json-smoke.ps1 @@ -0,0 +1,361 @@ +$ErrorActionPreference = 'Stop' + +$RootDir = Resolve-Path (Join-Path $PSScriptRoot '..\..') +$ScriptPath = Join-Path $RootDir 'skills\smart-web-fetch\scripts\smart-web-fetch.ps1' +$CmdPath = Join-Path $RootDir 'skills\smart-web-fetch\scripts\smart-web-fetch.cmd' +$ServerScript = Join-Path $RootDir 'spec\tests\json-smoke-server.py' +$Port = if ($env:SMART_WEB_FETCH_TEST_PORT) { [int]$env:SMART_WEB_FETCH_TEST_PORT } else { 18766 } +$ServerProcess = $null + +function Wait-ForServer([int]$TargetPort) { + for ($i = 0; $i -lt 30; $i++) { + try { + $client = [System.Net.Sockets.TcpClient]::new() + $async = $client.BeginConnect('127.0.0.1', $TargetPort, $null, $null) + if ($async.AsyncWaitHandle.WaitOne(200) -and $client.Connected) { + $client.Close() + return + } + $client.Close() + } catch { + } + + Start-Sleep -Milliseconds 200 + } + + throw "Smoke test server did not start on port $TargetPort" +} + +function Assert-JsonSuccess([string]$Path, [string]$ExpectedSource) { + $payload = Get-Content -LiteralPath $Path -Raw | ConvertFrom-Json + if (-not $payload.success) { throw "Expected success=true in $Path" } + if ($payload.source -ne $ExpectedSource) { throw "Expected source=$ExpectedSource in $Path" } + if ($payload.content -isnot [string]) { throw "Expected content to be a string in $Path" } + if ([string]::IsNullOrWhiteSpace([string]$payload.url)) { throw "Expected url in $Path" } +} + +function Assert-JsonUrlEquals([string]$Path, [string]$ExpectedUrl) { + $payload = Get-Content -LiteralPath $Path -Raw | ConvertFrom-Json + if ($payload.url -ne $ExpectedUrl) { throw "Expected url=$ExpectedUrl in $Path" } +} + +function Assert-JsonFailure([string]$Path, [string]$ExpectedSource) { + $payload = Get-Content -LiteralPath $Path -Raw | ConvertFrom-Json + if ($payload.success) { throw "Expected success=false in $Path" } + if ($payload.source -ne $ExpectedSource) { throw "Expected source=$ExpectedSource in $Path" } + if ($payload.content -ne '') { throw "Expected empty content in $Path" } + if ([string]::IsNullOrWhiteSpace([string]$payload.error)) { throw "Expected error message in $Path" } +} + +function Assert-JsonErrorContains([string]$Path, [string]$ExpectedFragment) { + $payload = Get-Content -LiteralPath $Path -Raw | ConvertFrom-Json + if (-not ([string]$payload.error).Contains($ExpectedFragment)) { + throw "Expected error containing '$ExpectedFragment' in $Path" + } +} + +function Assert-FileNotContains([string]$Path, [string]$UnexpectedFragment) { + $content = Get-Content -LiteralPath $Path -Raw + if ($content.Contains($UnexpectedFragment)) { + throw "Unexpected '$UnexpectedFragment' in $Path" + } +} + +function Assert-HelpOutput([string]$Text) { + if (-not $Text.Contains('Smart Web Fetch')) { + throw 'Expected wrapper help output' + } + if ($Text.Contains('FAKE_CORE_SHADOWED')) { + throw 'Wrapper resolved a shadowed core.py from the caller working directory' + } +} + +function Test-PyLauncherFallback([string]$ScriptPathToTest, [string]$CmdPathToTest) { + $shimDir = Join-Path $env:TEMP "smart-web-fetch-py-shim-$PID" + $realPython = (Get-Command python -CommandType Application | Select-Object -First 1).Source + $originalPath = $env:PATH + + New-Item -ItemType Directory -Force -Path $shimDir | Out-Null + + $pyShim = @" +@echo off +setlocal EnableDelayedExpansion +if "%~1"=="-0p" ( + echo -V:3.12 C:\fake\python312.exe + exit /b 0 +) +if "%~1"=="-3.12" ( + "$realPython" %2 %3 %4 %5 %6 %7 %8 %9 + exit /b !errorlevel! +) +exit /b 1 +"@ + + $pythonShim = @" +@echo off +exit /b 1 +"@ + + Set-Content -LiteralPath (Join-Path $shimDir 'py.cmd') -Value $pyShim -Encoding ascii + Set-Content -LiteralPath (Join-Path $shimDir 'python.cmd') -Value $pythonShim -Encoding ascii + + try { + $env:PATH = "$shimDir;$originalPath" + + & pwsh -File $ScriptPathToTest --help | Out-Null + if ($LASTEXITCODE -ne 0) { + throw 'Expected PowerShell wrapper to accept py-managed Python 3.12' + } + + & cmd.exe /c "`"$CmdPathToTest`" --help" | Out-Null + if ($LASTEXITCODE -ne 0) { + throw 'Expected CMD wrapper to accept py-managed Python 3.12' + } + } finally { + $env:PATH = $originalPath + Remove-Item -LiteralPath $shimDir -Recurse -Force -ErrorAction SilentlyContinue + } +} + +function Test-ShadowedCoreResolution([string]$ScriptPathToTest, [string]$CmdPathToTest) { + $shadowDir = Join-Path $env:TEMP "smart-web-fetch-shadow-$PID" + New-Item -ItemType Directory -Force -Path $shadowDir | Out-Null + Set-Content -LiteralPath (Join-Path $shadowDir 'core.py') -Value 'print("FAKE_CORE_SHADOWED")' -Encoding ascii + + try { + Push-Location $shadowDir + + $psHelp = (& pwsh -File $ScriptPathToTest --help 2>&1 | Out-String) + if ($LASTEXITCODE -ne 0) { + throw 'Expected PowerShell wrapper --help to succeed with shadowed core.py present' + } + Assert-HelpOutput -Text $psHelp + + $cmdHelp = (& cmd.exe /c "`"$CmdPathToTest`" --help" 2>&1 | Out-String) + if ($LASTEXITCODE -ne 0) { + throw 'Expected CMD wrapper --help to succeed with shadowed core.py present' + } + Assert-HelpOutput -Text $cmdHelp + } finally { + Pop-Location + Remove-Item -LiteralPath $shadowDir -Recurse -Force -ErrorAction SilentlyContinue + } +} + +try { + $ServerProcess = Start-Process -FilePath 'python' -ArgumentList @($ServerScript, '--port', "$Port") -PassThru -WindowStyle Hidden + Wait-ForServer -TargetPort $Port + + & pwsh -File $ScriptPath --help | Out-Null + if ($LASTEXITCODE -ne 0) { + throw 'Expected --help to succeed' + } + + & cmd.exe /c "`"$CmdPath`" --help" | Out-Null + if ($LASTEXITCODE -ne 0) { + throw 'Expected CMD --help to succeed' + } + + Test-PyLauncherFallback -ScriptPathToTest $ScriptPath -CmdPathToTest $CmdPath + Test-ShadowedCoreResolution -ScriptPathToTest $ScriptPath -CmdPathToTest $CmdPath + + $successPath = Join-Path $env:TEMP "smart-web-fetch-success-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com --json | Set-Content -Encoding utf8NoBOM $successPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected default --json success to succeed' + } + Assert-JsonSuccess -Path $successPath -ExpectedSource 'jina' + + $cmdSuccessPath = Join-Path $env:TEMP "smart-web-fetch-cmd-success-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & cmd.exe /c "`"$CmdPath`" example.com --json" | Set-Content -Encoding utf8NoBOM $cmdSuccessPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected CMD --json success to succeed' + } + Assert-JsonSuccess -Path $cmdSuccessPath -ExpectedSource 'jina' + + $forcedJinaPath = Join-Path $env:TEMP "smart-web-fetch-forced-jina-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com -s jina --json | Set-Content -Encoding utf8NoBOM $forcedJinaPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected forced jina --json to succeed' + } + Assert-JsonSuccess -Path $forcedJinaPath -ExpectedSource 'jina' + + $schemelessHostPortPath = Join-Path $env:TEMP "smart-web-fetch-schemeless-host-port-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath "localhost:$Port/demo" -s jina --json | Set-Content -Encoding utf8NoBOM $schemelessHostPortPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected schemeless host:port --json to succeed' + } + Assert-JsonSuccess -Path $schemelessHostPortPath -ExpectedSource 'jina' + Assert-JsonUrlEquals -Path $schemelessHostPortPath -ExpectedUrl "https://localhost:$Port/demo" + + $markdownPath = Join-Path $env:TEMP "smart-web-fetch-markdown-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-success" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com --json | Set-Content -Encoding utf8NoBOM $markdownPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected markdown fallback --json to succeed' + } + Assert-JsonSuccess -Path $markdownPath -ExpectedSource 'markdown' + + $defuddlePath = Join-Path $env:TEMP "smart-web-fetch-defuddle-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-success" + & pwsh -File $ScriptPath example.com --json | Set-Content -Encoding utf8NoBOM $defuddlePath + if ($LASTEXITCODE -ne 0) { + throw 'Expected defuddle fallback --json to succeed' + } + Assert-JsonSuccess -Path $defuddlePath -ExpectedSource 'defuddle' + + $basicPath = Join-Path $env:TEMP "smart-web-fetch-basic-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath "http://127.0.0.1:$Port/basic-success" --json | Set-Content -Encoding utf8NoBOM $basicPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected automatic basic fallback --json to succeed' + } + Assert-JsonSuccess -Path $basicPath -ExpectedSource 'basic' + + $failurePath = Join-Path $env:TEMP "smart-web-fetch-failure-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath "http://127.0.0.1:$Port/basic-short" --json | Set-Content -Encoding utf8NoBOM $failurePath + if ($LASTEXITCODE -eq 0) { + throw 'Expected --json failure to exit non-zero' + } + Assert-JsonFailure -Path $failurePath -ExpectedSource 'none' + + $cmdFailurePath = Join-Path $env:TEMP "smart-web-fetch-cmd-failure-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & cmd.exe /c "`"$CmdPath`" `"http://127.0.0.1:$Port/basic-short`" --json" | Set-Content -Encoding utf8NoBOM $cmdFailurePath + if ($LASTEXITCODE -eq 0) { + throw 'Expected CMD --json failure to exit non-zero' + } + Assert-JsonFailure -Path $cmdFailurePath -ExpectedSource 'none' + + $unsupportedSchemePath = Join-Path $env:TEMP "smart-web-fetch-unsupported-scheme-$PID.json" + & pwsh -File $ScriptPath 'ftp://example.com' --json | Set-Content -Encoding utf8NoBOM $unsupportedSchemePath + if ($LASTEXITCODE -eq 0) { + throw 'Expected unsupported-scheme --json failure to exit non-zero' + } + Assert-JsonFailure -Path $unsupportedSchemePath -ExpectedSource 'none' + Assert-JsonErrorContains -Path $unsupportedSchemePath -ExpectedFragment 'Unsupported URL scheme' + + $malformedUrlPath = Join-Path $env:TEMP "smart-web-fetch-malformed-url-$PID.json" + $malformedUrlCapturePath = Join-Path $env:TEMP "smart-web-fetch-malformed-url-capture-$PID.log" + & pwsh -File $ScriptPath '[::1]extra' --json 2>&1 | Tee-Object -FilePath $malformedUrlCapturePath | Set-Content -Encoding utf8NoBOM $malformedUrlPath + if ($LASTEXITCODE -eq 0) { + throw 'Expected malformed URL --json failure to exit non-zero' + } + Assert-JsonFailure -Path $malformedUrlPath -ExpectedSource 'none' + Assert-JsonErrorContains -Path $malformedUrlPath -ExpectedFragment 'Invalid URL:' + Assert-FileNotContains -Path $malformedUrlCapturePath -UnexpectedFragment 'Traceback' + + $cmdBangPath = Join-Path $env:TEMP "smart-web-fetch-cmd-bang-$PID.json" + & cmd.exe /c "`"$CmdPath`" `"ftp://example.com/!bang`" --json" | Set-Content -Encoding utf8NoBOM $cmdBangPath + if ($LASTEXITCODE -eq 0) { + throw 'Expected CMD bang-URL unsupported-scheme failure to exit non-zero' + } + Assert-JsonFailure -Path $cmdBangPath -ExpectedSource 'none' + Assert-JsonUrlEquals -Path $cmdBangPath -ExpectedUrl 'ftp://example.com/!bang' + Assert-JsonErrorContains -Path $cmdBangPath -ExpectedFragment 'Unsupported URL scheme' + + $invalidCharsetPath = Join-Path $env:TEMP "smart-web-fetch-invalid-charset-$PID.json" + $invalidCharsetCapturePath = Join-Path $env:TEMP "smart-web-fetch-invalid-charset-capture-$PID.log" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-invalid-charset" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com -s jina --json 2>&1 | Tee-Object -FilePath $invalidCharsetCapturePath | Set-Content -Encoding utf8NoBOM $invalidCharsetPath + if ($LASTEXITCODE -ne 0) { + throw 'Expected invalid-charset --json to succeed' + } + Assert-JsonSuccess -Path $invalidCharsetPath -ExpectedSource 'jina' + Assert-FileNotContains -Path $invalidCharsetCapturePath -UnexpectedFragment 'Traceback' + + $markdownShortPath = Join-Path $env:TEMP "smart-web-fetch-markdown-short-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-short" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com -s markdown --json | Set-Content -Encoding utf8NoBOM $markdownShortPath + if ($LASTEXITCODE -eq 0) { + throw 'Expected short extracted markdown --json failure to exit non-zero' + } + Assert-JsonFailure -Path $markdownShortPath -ExpectedSource 'markdown' + Assert-JsonErrorContains -Path $markdownShortPath -ExpectedFragment 'too-short' + + $defuddleShortPath = Join-Path $env:TEMP "smart-web-fetch-defuddle-short-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-short" + & pwsh -File $ScriptPath example.com -s defuddle --json | Set-Content -Encoding utf8NoBOM $defuddleShortPath + if ($LASTEXITCODE -eq 0) { + throw 'Expected short extracted defuddle --json failure to exit non-zero' + } + Assert-JsonFailure -Path $defuddleShortPath -ExpectedSource 'defuddle' + Assert-JsonErrorContains -Path $defuddleShortPath -ExpectedFragment 'too-short' + + $binaryFailurePath = Join-Path $env:TEMP "smart-web-fetch-binary-failure-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-error" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath "http://127.0.0.1:$Port/basic-binary" --json | Set-Content -Encoding utf8NoBOM $binaryFailurePath + if ($LASTEXITCODE -eq 0) { + throw 'Expected binary basic fallback --json failure to exit non-zero' + } + Assert-JsonFailure -Path $binaryFailurePath -ExpectedSource 'none' + Assert-JsonErrorContains -Path $binaryFailurePath -ExpectedFragment 'non-text/binary content' + + $parseFailureOutputPath = Join-Path $env:TEMP "smart-web-fetch-parse-failure-$PID.json" + & pwsh -File $ScriptPath --json --output $parseFailureOutputPath | Out-Null + if ($LASTEXITCODE -eq 0) { + throw 'Expected parse-time --json failure to exit non-zero' + } + Assert-JsonFailure -Path $parseFailureOutputPath -ExpectedSource 'none' + + $writeFailureCapturePath = Join-Path $env:TEMP "smart-web-fetch-write-failure-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com --json --output 'NUL\blocked\out.json' 2>&1 | Set-Content -Encoding utf8NoBOM $writeFailureCapturePath + if ($LASTEXITCODE -eq 0) { + throw 'Expected write-failure --json to exit non-zero' + } + Assert-JsonFailure -Path $writeFailureCapturePath -ExpectedSource 'jina' + + $outputPath = Join-Path $env:TEMP "smart-web-fetch-output-$PID.json" + $env:SMART_WEB_FETCH_JINA_READER_BASE = "http://127.0.0.1:$Port/jina-success" + $env:SMART_WEB_FETCH_MARKDOWN_NEW_URL = "http://127.0.0.1:$Port/markdown-error" + $env:SMART_WEB_FETCH_DEFUDDLE_URL = "http://127.0.0.1:$Port/defuddle-error" + & pwsh -File $ScriptPath example.com --json --output $outputPath | Out-Null + if ($LASTEXITCODE -ne 0) { + throw 'Expected --json with --output to succeed' + } + Assert-JsonSuccess -Path $outputPath -ExpectedSource 'jina' + + Write-Output '[PASS] PowerShell JSON smoke tests passed' +} finally { + Remove-Item Env:SMART_WEB_FETCH_JINA_READER_BASE -ErrorAction SilentlyContinue + Remove-Item Env:SMART_WEB_FETCH_MARKDOWN_NEW_URL -ErrorAction SilentlyContinue + Remove-Item Env:SMART_WEB_FETCH_DEFUDDLE_URL -ErrorAction SilentlyContinue + + if ($ServerProcess -and -not $ServerProcess.HasExited) { + Stop-Process -Id $ServerProcess.Id -Force + } +} diff --git a/spec/tests/json-smoke.sh b/spec/tests/json-smoke.sh new file mode 100644 index 0000000..ee1eee7 --- /dev/null +++ b/spec/tests/json-smoke.sh @@ -0,0 +1,317 @@ +#!/usr/bin/env bash + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SCRIPT_PATH="$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch" +SERVER_SCRIPT="$ROOT_DIR/spec/tests/json-smoke-server.py" +PORT="${SMART_WEB_FETCH_TEST_PORT:-18765}" +SERVER_PID="" +PYTHON_BIN="" +PYTHON_VERSION_CHECK='import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' + +fail() { + echo "[FAIL] $1" >&2 + exit 1 +} + +cleanup() { + if [[ -n "$SERVER_PID" ]]; then + kill "$SERVER_PID" >/dev/null 2>&1 || true + wait "$SERVER_PID" 2>/dev/null || true + fi +} + +wait_for_server() { + local attempt + for attempt in $(seq 1 30); do + if "$PYTHON_BIN" - "$PORT" <<'PY' +import socket +import sys + +sock = socket.socket() +sock.settimeout(0.2) +try: + sock.connect(("127.0.0.1", int(sys.argv[1]))) +except OSError: + sys.exit(1) +finally: + sock.close() +PY + then + return 0 + fi + sleep 0.2 + done + + fail "Smoke test server did not start on port $PORT" +} + +assert_json_success() { + local path="$1" + local expected_source="$2" + + "$PYTHON_BIN" - "$path" "$expected_source" <<'PY' +import json +import sys + +path = sys.argv[1] +expected_source = sys.argv[2] +with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + +assert payload["success"] is True, payload +assert isinstance(payload["content"], str), payload +assert payload["source"] == expected_source, payload +assert isinstance(payload["url"], str) and payload["url"], payload +PY +} + +assert_json_url_equals() { + local path="$1" + local expected_url="$2" + + "$PYTHON_BIN" - "$path" "$expected_url" <<'PY' +import json +import sys + +path = sys.argv[1] +expected_url = sys.argv[2] +with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + +assert payload["url"] == expected_url, payload +PY +} + +assert_json_failure() { + local path="$1" + local expected_source="$2" + + "$PYTHON_BIN" - "$path" "$expected_source" <<'PY' +import json +import sys + +path = sys.argv[1] +expected_source = sys.argv[2] +with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + +assert payload["success"] is False, payload +assert payload["source"] == expected_source, payload +assert payload["content"] == "", payload +assert isinstance(payload.get("error"), str) and payload["error"], payload +PY +} + +assert_json_error_contains() { + local path="$1" + local expected_fragment="$2" + + "$PYTHON_BIN" - "$path" "$expected_fragment" <<'PY' +import json +import sys + +path = sys.argv[1] +expected_fragment = sys.argv[2] +with open(path, encoding="utf-8") as fh: + payload = json.load(fh) + +assert expected_fragment in payload.get("error", ""), payload +PY +} + +assert_file_not_contains() { + local path="$1" + local unexpected_fragment="$2" + if grep -q -- "$unexpected_fragment" "$path"; then + fail "Unexpected '$unexpected_fragment' in $path" + fi +} + +assert_file_contains() { + local path="$1" + local expected_fragment="$2" + if ! grep -q -- "$expected_fragment" "$path"; then + fail "Expected '$expected_fragment' in $path" + fi +} + +assert_help_output() { + local path="$1" + assert_file_contains "$path" "Smart Web Fetch" + assert_file_not_contains "$path" "FAKE_CORE_SHADOWED" +} + +trap cleanup EXIT + +for candidate in python3 python; do + if command -v "$candidate" >/dev/null 2>&1 && "$candidate" -c "$PYTHON_VERSION_CHECK" >/dev/null 2>&1; then + PYTHON_BIN="$candidate" + break + fi +done + +[[ -n "$PYTHON_BIN" ]] || fail "Python 3.11+ is required for json-smoke.sh" + +"$PYTHON_BIN" "$SERVER_SCRIPT" --port "$PORT" & +SERVER_PID=$! +wait_for_server + +bash "$SCRIPT_PATH" --help >/dev/null || fail "--help should succeed" + +shadow_dir="$(mktemp -d)" +shadow_stdout="$(mktemp)" +shadow_stderr="$(mktemp)" +printf '%s\n' 'print("FAKE_CORE_SHADOWED")' > "$shadow_dir/core.py" +( + cd "$shadow_dir" + bash "$SCRIPT_PATH" --help >"$shadow_stdout" 2>"$shadow_stderr" +) || fail "shadowed core.py should not break Bash wrapper help" +assert_help_output "$shadow_stdout" +assert_file_not_contains "$shadow_stderr" "FAKE_CORE_SHADOWED" +rm -rf "$shadow_dir" + +success_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-success" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com --json >"$success_json" +assert_json_success "$success_json" "jina" + +forced_jina_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-success" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com -s jina --json >"$forced_jina_json" +assert_json_success "$forced_jina_json" "jina" + +schemeless_host_port_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-success" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" "localhost:$PORT/demo" -s jina --json >"$schemeless_host_port_json" +assert_json_success "$schemeless_host_port_json" "jina" +assert_json_url_equals "$schemeless_host_port_json" "https://localhost:$PORT/demo" + +markdown_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-success" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com --json >"$markdown_json" +assert_json_success "$markdown_json" "markdown" + +defuddle_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-success" \ + bash "$SCRIPT_PATH" example.com --json >"$defuddle_json" +assert_json_success "$defuddle_json" "defuddle" + +basic_json="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" "http://127.0.0.1:$PORT/basic-success" --json >"$basic_json" +assert_json_success "$basic_json" "basic" + +failure_json="$(mktemp)" +set +e +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" "http://127.0.0.1:$PORT/basic-short" --json >"$failure_json" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "--json failure should exit non-zero" +assert_json_failure "$failure_json" "none" + +unsupported_scheme_json="$(mktemp)" +set +e +bash "$SCRIPT_PATH" "ftp://example.com" --json >"$unsupported_scheme_json" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "unsupported scheme should exit non-zero" +assert_json_failure "$unsupported_scheme_json" "none" +assert_json_error_contains "$unsupported_scheme_json" "Unsupported URL scheme" + +invalid_charset_json="$(mktemp)" +invalid_charset_stderr="$(mktemp)" +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-invalid-charset" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com -s jina --json >"$invalid_charset_json" 2>"$invalid_charset_stderr" +assert_json_success "$invalid_charset_json" "jina" +assert_file_not_contains "$invalid_charset_stderr" "Traceback" + +malformed_url_json="$(mktemp)" +malformed_url_stderr="$(mktemp)" +set +e +bash "$SCRIPT_PATH" "[::1]extra" --json >"$malformed_url_json" 2>"$malformed_url_stderr" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "malformed URL should exit non-zero" +assert_json_failure "$malformed_url_json" "none" +assert_json_error_contains "$malformed_url_json" "Invalid URL:" +assert_file_not_contains "$malformed_url_stderr" "Traceback" + +markdown_short_json="$(mktemp)" +set +e +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-short" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com -s markdown --json >"$markdown_short_json" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "short extracted markdown should exit non-zero" +assert_json_failure "$markdown_short_json" "markdown" +assert_json_error_contains "$markdown_short_json" "too-short" + +defuddle_short_json="$(mktemp)" +set +e +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-short" \ + bash "$SCRIPT_PATH" example.com -s defuddle --json >"$defuddle_short_json" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "short extracted defuddle should exit non-zero" +assert_json_failure "$defuddle_short_json" "defuddle" +assert_json_error_contains "$defuddle_short_json" "too-short" + +binary_failure_json="$(mktemp)" +set +e +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-error" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" "http://127.0.0.1:$PORT/basic-binary" --json >"$binary_failure_json" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "binary basic fallback should exit non-zero" +assert_json_failure "$binary_failure_json" "none" +assert_json_error_contains "$binary_failure_json" "non-text/binary content" + +parse_failure_output="$(mktemp)" +set +e +bash "$SCRIPT_PATH" --json --output "$parse_failure_output" >/dev/null +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "parse-time --json failure should exit non-zero" +assert_json_failure "$parse_failure_output" "none" + +write_failure_json="$(mktemp)" +write_failure_stderr="$(mktemp)" +set +e +SMART_WEB_FETCH_JINA_READER_BASE="http://127.0.0.1:$PORT/jina-success" \ +SMART_WEB_FETCH_MARKDOWN_NEW_URL="http://127.0.0.1:$PORT/markdown-error" \ +SMART_WEB_FETCH_DEFUDDLE_URL="http://127.0.0.1:$PORT/defuddle-error" \ + bash "$SCRIPT_PATH" example.com --json --output . >"$write_failure_json" 2>"$write_failure_stderr" +status=$? +set -e +[[ "$status" -ne 0 ]] || fail "write-failure --json should exit non-zero" +assert_json_failure "$write_failure_json" "jina" +if grep -q "Traceback" "$write_failure_stderr"; then + fail "write failure should not emit a Python traceback" +fi + +echo "[PASS] Bash JSON smoke tests passed" diff --git a/spec/tests/offline-regression.sh b/spec/tests/offline-regression.sh index cf2a70b..dc15e15 100755 --- a/spec/tests/offline-regression.sh +++ b/spec/tests/offline-regression.sh @@ -3,9 +3,16 @@ set -euo pipefail ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SKILL_DIR="$ROOT_DIR/skills/smart-web-fetch" FIXTURE_DIR="$ROOT_DIR/spec/fixtures" RULES_FILE="$ROOT_DIR/skills/smart-web-fetch/assets/fetch-rules.json" CONTRACT_FILE="$ROOT_DIR/spec/fetch-contract.md" +PYTHON_CORE_DIR="$SKILL_DIR/core" +PYTHON_MAIN="$SKILL_DIR/main.py" +UNSUPPORTED_PYTHON_MAIN="$PYTHON_CORE_DIR/__main__.py" +PYTHON_BIN="" +PYTHON_VERSION_CHECK='import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' +LEGACY_CORE_PATTERN='smart_web_fetch_core' fail() { echo "[FAIL] $1" >&2 @@ -17,6 +24,11 @@ assert_file_exists() { [[ -f "$path" ]] || fail "Missing required file: $path" } +assert_file_not_exists() { + local path="$1" + [[ ! -e "$path" ]] || fail "Expected file to be removed: $path" +} + assert_file_contains() { local path="$1" local pattern="$2" @@ -25,6 +37,14 @@ assert_file_contains() { fi } +assert_file_not_contains() { + local path="$1" + local pattern="$2" + if grep -qi -- "$pattern" "$path"; then + fail "Unexpected pattern '$pattern' in $path" + fi +} + extract_json_string_field() { local path="$1" local field="$2" @@ -134,6 +154,24 @@ assert_json_boolean_true() { fi } +assert_json_boolean_false() { + local path="$1" + local field="$2" + if ! grep -Eq "\"$field\"[[:space:]]*:[[:space:]]*false" "$path"; then + fail "Expected $field=false in $path" + fi +} + +assert_json_string_equals() { + local path="$1" + local field="$2" + local expected="$3" + local actual + + actual=$(extract_json_string_field "$path" "$field") || fail "Expected string field $field in $path" + [[ "$actual" == "$expected" ]] || fail "Expected $field=$expected in $path, got $actual" +} + assert_positive_integer_threshold_field() { local path="$1" local field="$2" @@ -152,13 +190,58 @@ assert_bash_syntax() { fi } +find_python() { + local candidate + for candidate in python3 python; do + if command -v "$candidate" >/dev/null 2>&1 && "$candidate" -c "$PYTHON_VERSION_CHECK" >/dev/null 2>&1; then + PYTHON_BIN="$candidate" + return 0 + fi + done + fail "Python 3.11+ is required for offline regression" +} + main() { + find_python + assert_file_exists "$CONTRACT_FILE" assert_file_exists "$RULES_FILE" + assert_file_exists "$PYTHON_MAIN" + assert_file_not_exists "$UNSUPPORTED_PYTHON_MAIN" assert_file_exists "$FIXTURE_DIR/markdown-success.json" assert_file_exists "$FIXTURE_DIR/structured-error.json" assert_file_exists "$FIXTURE_DIR/html-error-page.html" assert_file_exists "$FIXTURE_DIR/too-short.txt" + assert_file_not_exists "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch-core" + assert_file_not_exists "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch-core.ps1" + assert_file_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch" "main.py" + assert_file_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.ps1" "main.py" + assert_file_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.cmd" "main.py" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch" "-m core" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.ps1" "-m core" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.cmd" "-m core" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch" "$LEGACY_CORE_PATTERN" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.ps1" "$LEGACY_CORE_PATTERN" + assert_file_not_contains "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch.cmd" "$LEGACY_CORE_PATTERN" + assert_file_not_contains "$ROOT_DIR/README.md" "python -m core" + assert_file_not_contains "$ROOT_DIR/README_EN.md" "python -m core" + assert_file_not_contains "$ROOT_DIR/spec/fetch-contract.md" "python -m core" + assert_file_not_contains "$ROOT_DIR/spec/wrapper-runtime.md" "python -m core" + assert_file_not_contains "$SKILL_DIR/SKILL.md" "python -m core" + + if rg -n --glob '!**/__pycache__/**' "$LEGACY_CORE_PATTERN" \ + "$ROOT_DIR/README.md" \ + "$ROOT_DIR/README_EN.md" \ + "$ROOT_DIR/spec/fetch-contract.md" \ + "$ROOT_DIR/spec/wrapper-runtime.md" \ + "$ROOT_DIR/spec/tests/json-smoke.sh" \ + "$ROOT_DIR/spec/tests/json-smoke.ps1" \ + "$ROOT_DIR/spec/tests/json-smoke-server.py" \ + "$SKILL_DIR/SKILL.md" \ + "$SKILL_DIR/scripts" \ + "$PYTHON_CORE_DIR" >/dev/null; then + fail "Found stale references to the legacy Python package name" + fi # HTML fixture should stay aligned with contract keywords. assert_file_contains "$FIXTURE_DIR/html-error-page.html" " "$cli_success_json" + printf '%s\n' '{"success":false,"url":"https://example.com","content":"","source":"none","error":"request failed"}' > "$cli_failure_json" + + assert_json_boolean_true "$cli_success_json" "success" + assert_json_string_equals "$cli_success_json" "source" "jina" + extract_json_string_field "$cli_success_json" "content" >/dev/null || fail "Expected content field in success JSON" + + assert_json_boolean_false "$cli_failure_json" "success" + assert_json_string_equals "$cli_failure_json" "source" "none" + extract_json_string_field "$cli_failure_json" "error" >/dev/null || fail "Expected error field in failure JSON" + rm -f "$cli_success_json" "$cli_failure_json" + # Basic script syntax checks. assert_bash_syntax "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch" - assert_bash_syntax "$ROOT_DIR/skills/smart-web-fetch/scripts/smart-web-fetch-core" + "$PYTHON_BIN" -m compileall -q "$SKILL_DIR" || fail "Python syntax check failed: $SKILL_DIR" + + "$PYTHON_BIN" - "$SKILL_DIR" <<'PY' +import pathlib +import sys + +skill_dir = pathlib.Path(sys.argv[1]) +sys.path.insert(0, str(skill_dir)) + +import core as module +import core.sources as sources + +rules = module.load_rules(False) +assert module.normalize_url("example.com") == "https://example.com" +assert module.normalize_url("myservice:8080/path") == "https://myservice:8080/path" +assert module.normalize_url("localhost:3000/foo") == "https://localhost:3000/foo" +assert module.normalize_url("example.com:8080/path") == "https://example.com:8080/path" +assert module.normalize_url("http://example.com") == "http://example.com" +assert module.normalize_url("HTTPS://Example.com/demo?q=1") == "https://Example.com/demo?q=1" +try: + module.normalize_url("ftp://example.com") +except module.CLIError as exc: + assert "Unsupported URL scheme" in str(exc) +else: + raise AssertionError("Expected ftp:// URL to fail validation") +try: + module.normalize_url("[::1]extra") +except module.CLIError as exc: + assert "Invalid URL:" in str(exc) +else: + raise AssertionError("Expected malformed IPv6 URL to fail validation") +assert rules.jina_min_length >= 100 +assert module.is_structured_error_response('{"error":true,"message":"forbidden"}', rules) is True +assert module.is_likely_html_error_payload( + "Access deniedforbidden", + "text/html; charset=utf-8", + rules, +) is True +markdown, is_json = module.extract_markdown_field('{"markdown":"hello world"}') +assert is_json is True +assert markdown == "hello world" +assert module.is_binary_response("image/png", b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR") is True +assert module.is_binary_response("application/octet-stream", b"abc") is True +assert module.is_binary_response("text/plain; charset=utf-8", "你好，world".encode("utf-8")) is False +assert module.is_binary_response("text/plain; charset=utf-16", b"\xff\xfeH\x00i\x00") is False +payload = module.render_payload(True, "https://example.com", "body", "jina") +assert payload == '{"success":true,"url":"https://example.com","content":"body","source":"jina"}' +decoded_utf16 = module.decode_response_body(b"\xff\xfeH\x00i\x00", {}) +assert decoded_utf16 == "Hi" +decoded_latin1 = module.decode_response_body("caf\xe9".encode("latin-1"), {}) +assert decoded_latin1 == "café" +class InvalidCharsetHeaders: + @staticmethod + def get_content_charset(): + return "not-a-real-charset" + + +decoded_invalid_charset = module.decode_response_body("caf\xe9".encode("latin-1"), InvalidCharsetHeaders()) +assert decoded_invalid_charset == "café" +short_json = '{"markdown":"tiny","meta":"' + ("x" * 200) + '"}' +sources.request_text = lambda *args, **kwargs: module.ResponseData( + status_code=200, + content_type="application/json", + raw_body=short_json.encode("utf-8"), + text=short_json, +) +try: + module.fetch_markdown_new("https://example.com", rules, False) +except module.CLIError as exc: + assert exc.source == "markdown" + assert "too-short" in str(exc) +else: + raise AssertionError("Expected markdown.new extracted content length validation to fail") +try: + module.fetch_defuddle("https://example.com", rules, False) +except module.CLIError as exc: + assert exc.source == "defuddle" + assert "too-short" in str(exc) +else: + raise AssertionError("Expected defuddle extracted content length validation to fail") +html = "

tiny

".format("x" * 200) +sources.request_text = lambda *args, **kwargs: module.ResponseData( + status_code=200, + content_type="text/html; charset=utf-8", + raw_body=html.encode("utf-8"), + text=html, +) +try: + module.fetch_basic( + "https://example.com", + module.Rules( + jina_min_length=100, + markdown_min_length=40, + defuddle_min_length=40, + basic_min_length=50, + structured_error_keywords=[], + html_error_keywords=[], + ), + False, + False, + ) +except module.CLIError as exc: + assert exc.source == "basic" + assert "after HTML cleanup" in str(exc) +else: + raise AssertionError("Expected cleaned HTML length validation to fail") +PY echo "[PASS] Offline regression checks passed" } diff --git a/spec/wrapper-runtime.md b/spec/wrapper-runtime.md new file mode 100644 index 0000000..f312442 --- /dev/null +++ b/spec/wrapper-runtime.md @@ -0,0 +1,220 @@ +# 包装器运行时说明 + +本文档只面向维护者，记录 `smart-web-fetch` 三个入口包装器的运行时发现逻辑、职责边界与测试基线。 + +## 1. 总体结构 + +- `skills/smart-web-fetch/main.py` 是唯一受支持的 Python bootstrap 文件。 +- `skills/smart-web-fetch/core/` 是内部实现包。 +- `smart-web-fetch`、`smart-web-fetch.ps1`、`smart-web-fetch.cmd` 都是薄启动器。 +- `scripts/` 目录只保留平台入口，不再承载业务实现。 +- 包装器的目标只有两件事： + - 找到可用的 Python 入口 + - 将原始 CLI 参数原样转发给确定文件入口 `main.py` + +## 2. 入口文件定位 + +| 入口 | 目标环境 | Python 启动目标 | +| --- | --- | --- | +| `scripts/smart-web-fetch` | Bash / Git Bash / WSL / 类 Unix Shell | 技能根目录 `main.py` | +| `scripts/smart-web-fetch.ps1` | PowerShell | 技能根目录 `main.py` | +| `scripts/smart-web-fetch.cmd` | Windows CMD / 原生 PowerShell | 技能根目录 `main.py` | + +禁止再引入额外的 shell core、PowerShell core 或重复实现的第二套主逻辑。 + +## 3. Bootstrap 发现方式 + +- 三个包装器都通过定位技能根目录 `skills/smart-web-fetch/`，直接执行该目录下的 `main.py`。 +- 包装器不得通过设置 `PYTHONPATH` 或切换当前工作目录来发现 `core/`，以免破坏用户相对路径语义（如 `--output relative/path.md`）。 +- `main.py` 负责在任意调用方 `cwd` 下优先解析同级 `core/` 包，并将控制权交给 `core.cli:main`。 + +## 4. 解释器发现顺序 + +### 4.1 Bash 包装器 + +文件：`skills/smart-web-fetch/scripts/smart-web-fetch` + +发现顺序： + +1. `python3` +2. `python` + +判定规则： + +- 每个候选命令都必须同时满足： + - `command -v ` 可执行 + - ` -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)'` 返回成功 +- 找到后使用： + - `exec "$PYTHON_BIN" "$SKILL_DIR/main.py" "$@"` + +失败行为： + +- 若两个候选都不满足，向 stderr 输出统一错误： + - `smart-web-fetch: error: Python 3.11+ was not found. Install Python 3.11 or newer and ensure a compatible interpreter is on PATH.` +- 退出码为 `1` + +### 4.2 PowerShell 包装器 + +文件：`skills/smart-web-fetch/scripts/smart-web-fetch.ps1` + +发现顺序： + +1. `py` 管理的任意 `>= 3.11` 解释器 +2. `python` + +判定规则： + +- 第一优先级是： + - 先执行 `py -0p` + - 解析其中所有 `-V:.` 项 + - 只保留版本 `>= 3.11` 的候选 + - 选择其中最高的 `major.minor` + - 命中后使用 `py -. \main.py @args` +- 若未命中任何 `>= 3.11` 的 `py` 解释器，再尝试 `python` + - 通过 `python -c ` 校验版本是否 `>= 3.11` + - 校验成功后执行 `python \main.py @args` + +失败行为： + +- 两条路径都不可用时，向 stderr 输出统一错误消息 +- 退出码为 `1` +- 若已成功启动 Python core，则包装器退出码必须回传 `$LASTEXITCODE` + +### 4.3 CMD 包装器 + +文件：`skills/smart-web-fetch/scripts/smart-web-fetch.cmd` + +发现顺序： + +1. `py` 管理的任意 `>= 3.11` 解释器 +2. `python` + +判定规则： + +- 若 `where py` 成功： + - 执行 `py -0p` + - 提取所有 `-V:.` 候选 + - 只保留版本 `>= 3.11` + - 选择其中最高的 `major.minor` + - 命中时执行 `py -. "%SKILL_DIR%\main.py" %*` +- 若未命中任何 `>= 3.11` 的 `py` 解释器，再尝试 `where python` + - 只检查命令是否存在 + - 当前 CMD 包装器不会在启动前自行校验 `python` 的版本 + - 版本下限由 `main.py` 在导入 `core.cli` 之前兜底 + +失败行为： + +- `py -3.11` 与 `python` 都不可用时，向 stderr 输出统一错误消息 +- 退出码为 `1` +- 成功启动 Python core 后，包装器退出码必须回传 `%errorlevel%` + +## 5. 包装器边界 + +包装器禁止承担以下职责： + +- 不重复实现 URL 归一化 +- 不重复实现服务降级顺序 +- 不重复实现规则文件加载 +- 不重复实现 JSON 成功/失败输出拼装 +- 不重复实现错误关键词判定 +- 不重复实现 HTML 清洗或正文提取 + +这些逻辑只能存在于 `core/` 包内。 + +## 6. 参数转发原则 + +- 包装器不解析业务参数，不重写参数，不添加默认参数。 +- Bash 包装器使用 `"$@"` 原样转发。 +- PowerShell 包装器使用脚本原始 `@args` 原样转发。 +- CMD 包装器使用 `%*` 原样转发。 +- `-h/--help`、`--json`、`-o/--output`、`-s/--service`、`-v/--verbose`、`--no-clean` 的语义全部由 `main.py` 引导后的 Python core 决定。 + +## 7. Python core 边界 + +`skills/smart-web-fetch/main.py` 负责： + +- Python 3.11+ 下限兜底校验 +- 保障从任意调用方 `cwd` 执行时优先导入同级 `core/` +- 将控制权移交给 `core.cli:main` + +`skills/smart-web-fetch/core/` 负责： + +- CLI 参数解析 +- 规则文件加载与默认值回退 +- 各服务源请求与降级 +- JSON / 文本输出 +- 文件写入 +- 非零退出码与错误信息 + +因此，包装器维护不应修改抓取契约；抓取行为变更应先更新 `spec/fetch-contract.md`。 + +## 8. 退出码要求 + +- 包装器自身找不到可用解释器时，必须返回 `1` +- Python core 正常完成时，包装器必须回传 `0` +- Python core 报错时，包装器必须回传 Python 进程的非零退出码 +- `--json` 模式失败时，仍然必须保留非零退出码，不得因为返回了 JSON 而吞掉失败状态 + +## 9. 与测试的对应关系 + +### 9.1 离线回归 + +文件：`spec/tests/offline-regression.sh` + +当前覆盖： + +- `spec/fetch-contract.md`、规则文件、`main.py`、Python core 包、fixture 是否存在 +- 已移除旧的 shell / PowerShell core 文件 +- Bash 包装器语法检查 +- Python core 包导入、语法检查与部分内置函数行为校验 +- 仓库内不应再残留旧的模块名启动 contract 引用 + +说明： + +- 该测试不直接验证 PowerShell / CMD 入口的运行时发现逻辑 +- 其目标是保证核心文件结构与基础契约未漂移 + +### 9.2 Bash JSON smoke + +文件：`spec/tests/json-smoke.sh` + +当前覆盖： + +- `scripts/smart-web-fetch --help` +- Bash 包装器能成功启动 `main.py` +- `--json` 成功/失败输出与退出码 +- 自动降级到 `markdown`、`defuddle`、`basic` +- 显式指定 `-s jina` +- 调用方工作目录存在假 `core.py` 时，仍应进入真实 CLI + +### 9.3 PowerShell / CMD JSON smoke + +文件：`spec/tests/json-smoke.ps1` + +当前覆盖： + +- `scripts/smart-web-fetch.ps1 --help` +- `scripts/smart-web-fetch.cmd --help` +- PowerShell 入口 `--json` 成功/失败输出与退出码 +- CMD 入口 `--json` 成功路径 +- `--output` 与 JSON 输出落盘 +- Windows `py` launcher 回退 +- 调用方工作目录存在假 `core.py` 时，PowerShell / CMD 入口仍应进入真实 CLI + +### 9.4 CI 维护基线 + +包装器调整后，至少要保证： + +- 离线回归仍通过 +- Bash JSON smoke 仍通过 +- PowerShell JSON smoke 仍通过 +- CMD 入口 smoke 仍通过 + +如果运行时发现顺序、错误消息或入口文件命名发生变化，应同步更新对应 smoke test 与本文件。 + +## 10. 维护约束 + +- 不要在 README 中展开解释器发现顺序或包装器内部职责 +- 不要把测试矩阵说明重新塞回 `README.md` / `README_EN.md` +- 包装器新增行为前，先判断该逻辑是否应该下沉到 Python core +- 若包装器实现与本文档不一致，应以代码为准修正文档，或在同一变更中一起修正两者