UN-4008 [FEAT] Add the unstract CLI to run extractions and API deployments from the terminal

.claude/skills/bump-client-pins/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: bump-client-pins
+description: Bump the exact `unstract-client` / `llmwhisperer-client` pins, re-sync the vendored specs to match, and cut a CLI release. Use whenever a new version of either client is released, when `tests/test_specs.py` or `tests/test_contract.py` fails, when the CLI is missing a flag for an endpoint the API already has, or when someone asks to "bump the client", "update the pins", "refresh the specs", or "release the CLI". Reach for this even when the request sounds like a plain dependency bump — the pins, the vendored specs and `provenance.json` have to move together or the CLI derives flags the pinned client cannot carry.
+---
+
+# Bumping the client pins
+
+The CLI derives its flags and help text from two published clients and from
+vendored copies of the specs those clients were generated from. That makes a pin
+bump three coupled edits, not one: the pin, the spec, and the provenance record.
+Move one without the others and the tests say so — which is the point of them.
+
+## The pieces
+
+| Thing | Where |
+|---|---|
+| Exact pins | `pyproject.toml`, `[project].dependencies` |
+| Vendored specs | `src/unstract_cli/specs/{docstudio,llmwhisperer}.json` |
+| Provenance | `src/unstract_cli/specs/provenance.json` (client pin, upstream repo, commit, sha256) |
+| Coherence tests | `tests/test_specs.py`, `tests/test_contract.py`, `tests/derived_flags.json` |
+| Release | `.github/workflows/release.yml`, `workflow_dispatch` |
+
+`src/unstract_cli/specs/README.md` explains the vendoring rule in place; read it
+if any of the below is unclear.
+
+## The sequence
+
+1. **Move the pins** in `pyproject.toml` to the released versions you are
+   upgrading to. They are exact (`==`) on purpose: the CLI's published surface is
+   derived from these clients, so a client that moves reshapes the CLI.
+
+2. **Relock:** `uv lock` then `uv sync --extra dev --python 3.12`. CI installs
+   from `uv.lock`, not from a fresh resolve, so a lockfile left behind means the
+   gate tests a dependency set nobody ships.
+
+3. **Re-sync each vendored spec from the commit the pinned client was generated
+   from.** The chain is: the client repo's release tag → its `tools/gen_sdk.sh`,
+   which records the upstream service repo, path and revision the spec was copied
+   from → the spec file committed in that client at that tag. Copy that file here
+   byte-for-byte. Copying from anywhere else — upstream `main`, a newer service
+   commit — is what `tests/test_contract.py` guards: a spec parameter the pinned
+   client has no argument for cannot become a flag.
+
+4. **Update `provenance.json`** for each spec you moved. Set `client` to the
+   exact pin you wrote in `pyproject.toml` (`unstract-client==X.Y.Z`); the
+   tests compare the two, so a pin moved without its spec fails here. Check
+   `repo`, `path` and `commit` against what that client's `tools/gen_sdk.sh`
+   records, because an upstream that moved its spec file leaves `repo` and
+   `path` stale and the tests cannot see it: they check the pin, the `sha256`
+   and the entry names, nothing about where the file came from. The `sha256`
+   is of the file you just wrote (`sha256sum src/unstract_cli/specs/<file>`).
+   This record is what lets the next person tell a current copy from a stale
+   one.
+
+5. **Run the tests:** `uv run pytest -q`.
+
+   - `test_specs.py` fails if a vendored file stops matching its recorded
+     sha256, if its `client` no longer equals the pin in `pyproject.toml`, or if
+     a spec has no provenance entry. It is the cheap check that steps 1, 3 and
+     4 actually agree.
+   - `test_contract.py` fails if a spec parameter the pinned client cannot accept
+     would have become a flag, and separately if the derived flags stop matching
+     `tests/derived_flags.json`.
+
+6. **If `derived_flags.json` fails, read the difference before refreshing it.**
+   The failure names the flags that moved. A flag missing from the new set is a
+   flag the CLI has stopped offering; a narrowed choice or changed type is a value
+   the CLI used to take and now rejects. Once you have decided the change is
+   intended, refresh it deliberately:
+
+   ```bash
+   UNSTRACT_CLI_REFRESH_FLAG_SNAPSHOT=1 uv run pytest -q tests/test_contract.py
+   ```
+
+   and commit the snapshot in the same PR, so the diff shows what the CLI's
+   surface gained or lost.
+
+7. **Lint:** `uv run ruff check . && uv run ruff format --check .` — the release
+   run repeats exactly this, so a failure here is a failure there.
+
+## Versioning and release
+
+Choose the bump by what changed for CLI users: **minor** for new or changed
+flags and commands, **patch** for fixes that leave the surface identical.
+
+Do not bump `__version__` in `src/unstract_cli/__init__.py` in your PR. The
+in-repo value names the last released version; `release.yml` reads it, applies
+the bump chosen at dispatch and commits the result itself.
+
+Release by dispatching **Release Tag and Publish Package** on `main`:
+
+- `version_bump: none` publishes the version already in the repo — what the
+  first release of a version needs.
+- `pre_release: true` publishes `<next-version>rcN` and deliberately leaves the
+  committed version alone, counting N up from the rc tags already published for
+  that target. To promote to stable, dispatch again with it off **and the same
+  `version_bump`**: the workflow recomputes the target from that input every
+  time, so a different bump publishes a different version than the one the rc
+  tested.
+- It publishes to PyPI **before** it tags and releases, because publishing is the
+  only step that cannot be undone: a failure before it leaves nothing to
+  unpublish, and one after it is retried by hand against a live artifact.
+
+## Upstream
+
+If a client pin is missing an endpoint the service already offers, the fix is in
+that client, not here — see the `spec-upgrade` skill in `unstract-python-client`
+and `llm-whisperer-python-client`. Bump the pin here once it is released.

.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: ci
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  # Offline by design: no network, no credentials, sub-second. Live round trips
+  # are a manual pre-release step, not a per-PR gate.
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          version: "0.6.14"
+          enable-cache: true
+      # Synced from uv.lock rather than resolved fresh, so the gate tests the
+      # dependency set an install actually gets.
+      - run: uv sync --extra dev --python 3.12
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q
+      # `uv run` resolves from uv.lock, whose click sits in the middle of the
+      # range the pin allows. Both ends are what an install in the wild gets,
+      # and discovery is a published contract read out of Click's own objects,
+      # so both ends are run and their answers compared.
+      - run: uv pip install 'click~=8.1.0'
+      - run: .venv/bin/python -m pytest -q
+      - run: .venv/bin/python -m unstract_cli -o json --discover full > floor.json
+      - run: uv pip install -U 'click>=8.1,<9'
+      - run: .venv/bin/python -m pytest -q
+      - run: .venv/bin/python -m unstract_cli -o json --discover full > latest.json
+      - run: diff floor.json latest.json

.github/workflows/release.yml

@@ -0,0 +1,178 @@
+name: Release Tag and Publish Package
+
+on:
+  workflow_dispatch:
+    inputs:
+      version_bump:
+        description: "Version bump type. `none` publishes the version already in the repo, which is what the first release needs."
+        required: true
+        default: "patch"
+        type: choice
+        options:
+          - patch
+          - minor
+          - major
+          - none
+      pre_release:
+        description: "Publish a release candidate (`<version>rcN`) instead of the version itself. Dispatch again with this off to promote the same version to stable."
+        required: false
+        default: false
+        type: boolean
+      release_notes:
+        description: "Release notes (optional)"
+        required: false
+        type: string
+
+concurrency:
+  group: release
+  cancel-in-progress: false
+
+jobs:
+  release-and-publish:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      # Publishing is by PyPI Trusted Publisher, so there is no API token.
+      id-token: write
+    steps:
+      - name: Generate GitHub App Token
+        id: generate-token
+        uses: actions/create-github-app-token@v3
+        with:
+          client-id: ${{ vars.PUSH_TO_MAIN_APP_CLIENT_ID }}
+          private-key: ${{ secrets.PUSH_TO_MAIN_APP_PRIVATE_KEY }}
+          owner: Zipstack
+          repositories: |
+            unstract-cli
+
+      - uses: actions/checkout@v4
+        with:
+          token: ${{ steps.generate-token.outputs.token }}
+          fetch-depth: 0
+
+      - name: Configure Git
+        run: |
+          git config --global user.name "github-actions[bot]"
+          git config --global user.email "github-actions[bot]@users.noreply.github.com"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - uses: astral-sh/setup-uv@v6
+        with:
+          version: "0.6.14"
+          enable-cache: true
+
+      # The same install as ci.yml, so what the release run lints and tests is
+      # what the PR gate lints and tests.
+      - run: uv sync --extra dev --python 3.12
+
+      # Staged locally only: nothing is committed, tagged or released until the
+      # checks and the build have passed, so a failure leaves main untouched.
+      - name: Compute new version
+        id: version
+        run: |
+          VERSION_FILE=src/unstract_cli/__init__.py
+          CURRENT_VERSION=$(sed -nE 's/^__version__ = "(.*)"/\1/p' "$VERSION_FILE")
+          echo "Current version: $CURRENT_VERSION"
+
+          IFS='.' read -r MAJOR MINOR PATCH <<< "$CURRENT_VERSION"
+          case "${{ github.event.inputs.version_bump }}" in
+            major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
+            minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
+            patch) PATCH=$((PATCH + 1)) ;;
+          esac
+          NEXT_VERSION="$MAJOR.$MINOR.$PATCH"
+
+          # A pre-release is a candidate for NEXT_VERSION, not a version of its
+          # own, so it never moves the committed one: the file keeps naming the
+          # last stable release, and repeat dispatches count up from the rc tags
+          # already published for that target.
+          if [ "${{ github.event.inputs.pre_release }}" = "true" ]; then
+            HIGHEST_RC=$(git tag -l "v${NEXT_VERSION}rc*" \
+              | sed -nE "s/^v${NEXT_VERSION}rc([0-9]+)$/\1/p" | sort -n | tail -1)
+            NEW_VERSION="${NEXT_VERSION}rc$(( ${HIGHEST_RC:-0} + 1 ))"
+          else
+            NEW_VERSION="$NEXT_VERSION"
+          fi
+
+          echo "New version: $NEW_VERSION"
+          echo "version=$NEW_VERSION" >> "$GITHUB_OUTPUT"
+
+          sed -i "s/^__version__ = \".*\"/__version__ = \"$NEW_VERSION\"/" "$VERSION_FILE"
+
+          if git rev-parse -q --verify "refs/tags/v$NEW_VERSION" >/dev/null; then
+            echo "Tag v$NEW_VERSION already exists. Exiting..."
+            exit 1
+          fi
+
+      - name: Verify version update
+        run: |
+          BUILT_VERSION=$(uv run python -c "import unstract_cli; print(unstract_cli.__version__)")
+          echo "Package version: $BUILT_VERSION"
+          echo "Target version: ${{ steps.version.outputs.version }}"
+          if [ "$BUILT_VERSION" != "${{ steps.version.outputs.version }}" ]; then
+            echo "Version mismatch! Exiting..."
+            exit 1
+          fi
+
+      - name: Run linting
+        run: |
+          uv run ruff check .
+          uv run ruff format --check .
+
+      - name: Run tests
+        run: uv run pytest -q
+
+      - name: Build package
+        run: uv build
+
+      # Everything revertible runs first: a tag, a branch and a release can all
+      # be deleted, and publishing cannot. Failing here leaves nothing on PyPI
+      # to reconcile; failing the other way around leaves a released version
+      # that no tag names.
+      - name: Commit version bump and create release
+        env:
+          RELEASE_NOTES: ${{ github.event.inputs.release_notes }}
+          GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
+        run: |
+          NEW_VERSION="${{ steps.version.outputs.version }}"
+
+          # A pre-release leaves the committed version alone, and `none`
+          # publishes the version already in the file, so both reach here with
+          # nothing to commit.
+          if [ "${{ github.event.inputs.pre_release }}" = "true" ]; then
+            git checkout -- src/unstract_cli/__init__.py
+          elif ! git diff --quiet; then
+            git add src/unstract_cli/__init__.py
+            git commit -m "chore: bump version to $NEW_VERSION [skip ci]"
+            git push origin main
+          fi
+
+          git tag "v$NEW_VERSION"
+          git push origin "v$NEW_VERSION"
+
+          if [ -z "$RELEASE_NOTES" ]; then
+            gh release create "v$NEW_VERSION" \
+              --title "Release v$NEW_VERSION" \
+              --generate-notes \
+              ${{ github.event.inputs.pre_release == 'true' && '--prerelease' || '' }}
+          else
+            gh release create "v$NEW_VERSION" \
+              --title "Release v$NEW_VERSION" \
+              --notes "$RELEASE_NOTES" \
+              --generate-notes \
+              ${{ github.event.inputs.pre_release == 'true' && '--prerelease' || '' }}
+          fi
+
+          echo "Created release v$NEW_VERSION"
+
+      - name: Publish to PyPI
+        run: uv publish
+
+      - name: Success message
+        run: |
+          echo "Published ${{ steps.version.outputs.version }} to PyPI with uv publish using Trusted Publishers"
+          echo "Release: https://github.com/${{ github.repository }}/releases/tag/v${{ steps.version.outputs.version }}"
+          echo "PyPI: https://pypi.org/project/unstract-cli/${{ steps.version.outputs.version }}/"

.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+.coverage
+.coverage.*
+htmlcov/