diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..ca521ba --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,33 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.10", "3.11", "3.12"] + steps: + - uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: pip + + - name: Install dependencies + run: pip install -r requirements-dev.txt + + - name: Lint (ruff) + run: ruff check app/ tests/ + + - name: Security scan (bandit) + run: bandit -r app/ + + - name: Test (pytest) + run: pytest -v diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..ea04cb6 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,78 @@ +# Changelog + +All notable changes to this project are documented here. Format loosely +follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Entries for +1.1.0–1.1.4 are reconstructed from git history rather than written at the +time of each release, so they're summaries rather than exhaustive. + +## [1.1.5] - 2026-09-22 + +### Added +- Request size limits on `pattern` (5,000 chars), `custom_patterns` (20,000 + chars) and `log_text` (200,000 chars). +- A wall-clock timeout (`MATCH_TIMEOUT_SECONDS`, default 3s, env-overridable) + around pattern matching and auto-generation, so a pathological pattern + can't hang a request indefinitely. +- A pytest test suite (`tests/test_grok_engine.py`, `tests/test_api.py`) + covering the matching engine and all four API endpoints - the project had + no tests before this. +- CI (`.github/workflows/ci.yml`): ruff, bandit and pytest run on every push + to `main` and every pull request, across Python 3.10-3.12. +- `requirements-dev.txt` and a `dev` extra in `pyproject.toml` (ruff, bandit, + pytest, httpx) for local linting and testing. +- README: Development, Request Limits & Timeouts, Production Notes and + License sections. + +### Fixed +- A pydantic validation error (e.g. a request exceeding the new size limits, + or a missing required field) used to return a list of error objects as + `detail`, which the frontend rendered as `[object Object]`. Validation + errors now return a single human-readable string, consistent with every + other error response. + +## [1.1.4] - 2026-09-22 + +### Fixed +- `find_partial_match` mis-tokenized patterns that mixed a `%{...}` Grok + field with a `(?P...)` / `(?...)` regex group, silently + breaking Partial Match Diagnostics for any such pattern. Replaced the + tokenizer with a paren-balancing version that also handles nested parens + inside a group's body. +- API error handlers now chain exceptions with `raise ... from e` instead of + discarding the original traceback. + +### Changed +- Pinned a `[tool.ruff]` lint configuration (previously unconfigured, so + `ruff check` meant whatever ruleset happened to be run by hand, and a + `# noqa: BLE001` comment in `grok_engine.py` referenced a rule nothing + enabled). +- `config.py` uses `Path.open()` instead of the builtin `open()`. + +## [1.1.3] - 2026-08-07 + +### Changed +- `pregenerate_pattern` uses diff-based sequence alignment across sample + lines instead of a positional zip, so it correctly handles optional or + variable-length segments between samples instead of misaligning on them. + +## [1.1.2] - 2026-08-07 + +### Changed +- Backend review pass: field-name sanitization regex fixes, improved IP + detection, lint cleanup, and split the inline CSS/JS out of `index.html` + into `app/static/`. +- Fixes to error message display, field matching, and suggested-pattern + replacement. + +## [1.1.1] - 2026-08-06 + +### Added +- Flexible vs. strict match mode toggle for the pattern checker. + +## [1.1.0] - 2026-08-05 + +### Added +- Initial tagged release: live Grok/regex matching, the pattern + auto-generator, a JSON view of results, ECS bracket/dot notation support, + pattern format-consistency normalization, and version/config loading from + `pyproject.toml`. diff --git a/README.md b/README.md index f831b0d..02f17d9 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,11 @@ This project was built to modernize and combine capabilities of Grok debugging t ``` . +├── .devcontainer/ +│ └── devcontainer.json # VS Code dev container config +├── .github/ +│ └── workflows/ +│ └── ci.yml # Lint (ruff), security scan (bandit) & tests (pytest) on push/PR ├── app/ │ ├── main.py # FastAPI application & API routes │ ├── config.py # App settings & version/feature metadata @@ -38,7 +43,12 @@ This project was built to modernize and combine capabilities of Grok debugging t │ └── static/ │ ├── css/style.css # UI styling │ └── js/app.js # Alpine.js application logic -├── requirements.txt # Python dependencies +├── tests/ +│ ├── test_grok_engine.py # Unit tests for the matching/generation engine +│ └── test_api.py # API-level tests (FastAPI TestClient) +├── pyproject.toml # Project metadata, ruff config, pytest config +├── requirements.txt # Runtime dependencies +├── requirements-dev.txt # + ruff, bandit, pytest, httpx └── README.md ``` @@ -72,3 +82,72 @@ uvicorn app.main:app --reload --port 8000 ``` 4. Open [http://localhost:8000](http://localhost:8000) in your browser. + +--- + +## Development + +Install the dev extras (adds ruff, bandit, pytest, httpx on top of the runtime dependencies): + +```bash +pip install -r requirements-dev.txt +``` + +**Run the tests:** + +```bash +pytest +``` + +**Lint:** + +```bash +ruff check app/ tests/ +``` + +The lint ruleset is pinned in `pyproject.toml` under `[tool.ruff.lint]` so `ruff check` gives the same result locally and in CI. + +**Security scan:** + +```bash +bandit -r app/ +``` + +All three run automatically on every push and pull request via [`.github/workflows/ci.yml`](.github/workflows/ci.yml), across Python 3.10–3.12. + +--- + +## Request Limits & Timeouts + +`/api/match` and `/api/generate` accept arbitrary user-supplied patterns and sample text, so a few limits are enforced server-side (see `app/main.py`): + +| Limit | Default | Purpose | +|---|---|---| +| Pattern length | 5,000 chars | Bounds pathological input | +| Custom pattern definitions length | 20,000 chars | Bounds pathological input | +| Log text length | 200,000 chars | Bounds pathological input | +| Match/generate timeout | 3s (`MATCH_TIMEOUT_SECONDS` env var) | Fails fast instead of hanging a request | + +The timeout bounds *response time*, not CPU usage — Python can't forcibly interrupt a regex match mid-flight, so it's a practical safeguard for trusted/internal use rather than a complete denial-of-service defense. See the comment above `MATCH_TIMEOUT_SECONDS` in `app/main.py` for the full reasoning, including why pygrok's use of the third-party `regex` package (installed automatically as one of pygrok's own dependencies) already mitigates some classic catastrophic-backtracking patterns on its own. + +--- + +## Production Notes + +The frontend is intentionally dependency-light for easy self-hosting, with two trade-offs worth knowing about before deploying it beyond a trusted local/internal setting: + +- **Tailwind via the CDN script** (`cdn.tailwindcss.com`) is Tailwind's "Play CDN" — convenient for zero-build local use, but Tailwind's own docs note it isn't intended for production (it compiles styles in the browser on every load). For a production build, compile Tailwind via its CLI or a bundler and link the resulting stylesheet instead. +- **Alpine.js is loaded from a floating version range** (`alpinejs@3.x.x`). Consider pinning it to an exact version (e.g. `alpinejs@3.17.1`) and adding a [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) hash, e.g.: + + ```bash + curl -s https://cdn.jsdelivr.net/npm/alpinejs@3.17.1/dist/cdn.min.js \ + | openssl dgst -sha384 -binary | openssl base64 -A + ``` + + and add `integrity="sha384-" crossorigin="anonymous"` to the `