Rust: rust-analyzer update automation #22590
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| --- | ||
| name: Update rust-analyzer | ||
| emoji: 🦀 | ||
| description: Update the rust-analyzer version used by the Rust extractor and prepare a pull request. | ||
| intent: Keep the Rust extractor on the latest compatible rust-analyzer version with a reviewable, validated pull request. | ||
| on: | ||
| workflow_dispatch: | ||
| roles: [admin, maintainer, write] | ||
| permissions: | ||
| contents: read | ||
| actions: read | ||
| pull-requests: read | ||
| copilot-requests: write | ||
| strict: true | ||
| checkout: | ||
| fetch-depth: 0 | ||
| concurrency: | ||
| group: update-rust-analyzer | ||
| cancel-in-progress: false | ||
| timeout-minutes: 180 | ||
| tools: | ||
| github: | ||
| mode: gh-proxy | ||
| toolsets: [repos, pull_requests, actions] | ||
| bash: ["*"] | ||
| edit: true | ||
| network: | ||
| allowed: | ||
| - defaults | ||
| - github | ||
| - github-actions | ||
| - rust | ||
| - bazel | ||
| - python | ||
| steps: | ||
| - name: Configure Git | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
|
|
||
| - name: Install cargo-edit | ||
| continue-on-error: true | ||
| run: cargo install cargo-edit@0.13.0 --locked | ||
|
|
||
| - name: Run rust-analyzer updater script | ||
| run: | | ||
| mkdir -p /tmp/gh-aw/agent | ||
| set +e | ||
| python3 rust/scripts/update_rust_analyzer.py \ | ||
| 2>&1 | tee /tmp/gh-aw/agent/rust-analyzer-update.log | ||
| status=${PIPESTATUS[0]} | ||
| set -e | ||
|
|
||
| printf '%s\n' "$status" > /tmp/gh-aw/agent/rust-analyzer-update-status.txt | ||
| if grep -Fxq "No new rust-analyzer version available." \ | ||
| /tmp/gh-aw/agent/rust-analyzer-update.log; then | ||
| printf '%s\n' "no-update" > /tmp/gh-aw/agent/rust-analyzer-update-result.txt | ||
| else | ||
| printf '%s\n' "update" > /tmp/gh-aw/agent/rust-analyzer-update-result.txt | ||
| fi | ||
| safe-outputs: | ||
| report-incomplete: {} | ||
| create-pull-request: | ||
| title-prefix: "Rust: " | ||
| branch-prefix: "automation/update-rust-analyzer/" | ||
| draft: true | ||
| max-patch-size: 10240 | ||
| max-patch-files: 1000 | ||
| allowed-files: | ||
| - "Cargo.lock" | ||
| - "shared/tree-sitter-extractor/Cargo.toml" | ||
| - "shared/yeast/Cargo.toml" | ||
| - "shared/yeast-macros/Cargo.toml" | ||
| - "shared/yeast-schema/Cargo.toml" | ||
| - "ruby/extractor/Cargo.toml" | ||
| - "unified/extractor/Cargo.toml" | ||
| - "unified/swift-syntax-rs/Cargo.toml" | ||
| - "MODULE.bazel" | ||
| - "MODULE.bazel.lock" | ||
| - "rust-toolchain.toml" | ||
| - "rust/**" | ||
| - "misc/bazel/3rdparty/**" | ||
| --- | ||
|
|
||
| # Update rust-analyzer | ||
|
|
||
| ## Task | ||
|
|
||
| The workflow has already run `rust/scripts/update_rust_analyzer.py`. Read: | ||
|
|
||
| - `/tmp/gh-aw/agent/rust-analyzer-update.log` for its complete output. | ||
| - `/tmp/gh-aw/agent/rust-analyzer-update-status.txt` for its exit status. | ||
| - `/tmp/gh-aw/agent/rust-analyzer-update-result.txt` for the deterministic result classification. | ||
|
|
||
| If the result is `no-update`, call `noop` with the reason | ||
| `No new rust-analyzer version available.` and stop immediately. Do not inspect | ||
| CI, modify files, or create a pull request. | ||
|
|
||
| Otherwise, continue the update from the existing working tree and commits: | ||
|
|
||
| 1. Read `rust/updating-rust-analyzer.md` and all applicable repository | ||
| instructions. | ||
| 2. Review the updater log, exit status, commits, and working tree. Do not rerun | ||
| the updater script. | ||
| 3. Complete as much of the documented update as possible. Fix extractor or | ||
| code-generation breakage, keep all `ra_ap_` dependency versions aligned, | ||
| regenerate required files, and add schema upgrade/downgrade scripts, tests, | ||
| and a change note when the schema changed. | ||
| 4. Run the relevant formatting, linting, code generation, build, and tests, | ||
| including `bazel run //rust:install`. Use `gh` to inspect relevant existing | ||
| CI configuration and prior failures while diagnosing problems. | ||
| 5. Review the complete diff and commits. Do not include secrets. Do not refer | ||
| to private repositories, internal issues, or internal pull requests in the | ||
| public pull request. | ||
| 6. Use `create_pull_request` exactly once to create a focused draft pull | ||
| request. Summarize the updater output, changes, and validation. If the | ||
| update cannot be finished, still create a partial draft pull request when | ||
| there are useful changes, and clearly list failures, missing work, and the | ||
| next commands for a maintainer. | ||
|
|
||
| The pull request is created after this agent execution, so its newly triggered | ||
| CI cannot be awaited in this run. Compensate with the strongest practical | ||
| local validation and state this limitation accurately in the pull request. | ||
| If the updater did not report `no-update` but no useful patch can be produced, | ||
| call `report_incomplete` with the updater failure and exact blocker. | ||
|
paldepind marked this conversation as resolved.
|
||
|
|
||
| ## Safe Outputs | ||
|
|
||
| - Use `create_pull_request` for the update or partial update. | ||
| - Use `noop` only for the exact no-update result. | ||
| - Use `report_incomplete` only when no useful pull request can be created. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| import json | ||
| import shlex | ||
| import subprocess | ||
| import sys | ||
| import tomllib | ||
| import urllib.request | ||
| from pathlib import Path | ||
|
|
||
|
|
||
| REPO_ROOT = Path(__file__).resolve().parents[2] | ||
| RUST_EXTRACTOR_MANIFEST = REPO_ROOT / "rust/extractor/Cargo.toml" | ||
|
|
||
| # The files that mention the fixed Rust toolchain version | ||
| TOOLCHAIN_RS = REPO_ROOT / "rust/extractor/src/toolchain.rs" | ||
| INTEGRATION_TEST_CONFTEST = REPO_ROOT / "rust/ql/integration-tests/conftest.py" | ||
| QL_TEST_SETUP = REPO_ROOT / "rust/ql/test/setup.sh" | ||
|
|
||
|
|
||
| def print_step(number: int, description: str) -> None: | ||
| print(f"\nStep {number}: {description}", flush=True) | ||
|
|
||
|
|
||
| def run(*command: str) -> None: | ||
| print(f"+ {shlex.join(command)}", flush=True) | ||
| subprocess.run(command, cwd=REPO_ROOT, check=True) | ||
|
|
||
|
|
||
| def run_codegen() -> None: | ||
| try: | ||
| run("bazel", "run", "//rust/codegen") | ||
| except subprocess.CalledProcessError as error: | ||
| print( | ||
| "\nCodegen failed. Carry out the instructions from step 4 in rust/updating-rust-analyzer.md manually.", | ||
| file=sys.stderr, | ||
| flush=True, | ||
| ) | ||
| raise SystemExit(error.returncode) from None | ||
|
|
||
|
|
||
| def get_rust_analyzer_version() -> str: | ||
|
Contributor
There was a problem hiding this comment. Please rename to |
||
| """Get the minimum version of all ra_ap dependencies from `Cargo.toml`.""" | ||
| dependencies = tomllib.loads(RUST_EXTRACTOR_MANIFEST.read_text())["dependencies"] | ||
| return min( | ||
| ( | ||
| version | ||
| for name, version in dependencies.items() | ||
| if name.startswith("ra_ap_") | ||
| ), | ||
| key=lambda value: tuple(map(int, value.split("."))), | ||
| ) | ||
|
|
||
|
|
||
| def fetch(url: str) -> bytes: | ||
| request = urllib.request.Request( | ||
| url, | ||
| headers={"User-Agent": "github/codeql rust-analyzer updater"}, | ||
| ) | ||
| with urllib.request.urlopen(request) as response: | ||
| return response.read() | ||
|
|
||
|
|
||
| def get_compatible_rust_toolchain(rust_analyzer_version: str) -> str: | ||
| """Get the latest Rust toolchain that precedes our version of rust-analyzer.""" | ||
| # Get the release date of the rust-analyzer version | ||
| crate_url = f"https://crates.io/api/v1/crates/ra_ap_syntax/{rust_analyzer_version}" | ||
| crate = json.loads(fetch(crate_url)) | ||
| rust_analyzer_release = crate["version"]["created_at"].split("T")[0] | ||
|
|
||
| # `manifests.txt` is a list of all toolchains. The one we're interested in looks like | ||
| # ``` | ||
| # static.rust-lang.org/dist/YYYY-MM-DD/channel-rust-stable.toml | ||
| # ``` | ||
| # where `YYYY-MM-DD` is the last that is earlier than the rust-analyzer release. | ||
| rust_manifest = next( | ||
| manifest | ||
| for manifest in reversed( | ||
| fetch("https://static.rust-lang.org/manifests.txt").decode().splitlines() | ||
| ) | ||
| if manifest.endswith("/channel-rust-stable.toml") | ||
| and manifest.split("/")[2] <= rust_analyzer_release | ||
| ) | ||
|
Comment on lines
+76
to
+83
Contributor
There was a problem hiding this comment. Slightly hard to read. |
||
| manifest = tomllib.loads(fetch(f"https://{rust_manifest}").decode()) | ||
| # The version looks like `version = "0.99.0 (797e8a9bc 2026-08-05)"` - we only want the first part. | ||
| return manifest["pkg"]["rust"]["version"].split()[0] | ||
|
|
||
|
|
||
| def update_fixed_rust_toolchain(version: str) -> None: | ||
|
Contributor
There was a problem hiding this comment. May I suggest |
||
| """Change the fixed toolchain in the places where it's hardcoded""" | ||
| toolchain_rs = TOOLCHAIN_RS.read_text() | ||
| prefix = 'const FIXED_RUST_TOOLCHAIN: &str = "' | ||
| old_version = toolchain_rs.split(prefix, 1)[1].split('"', 1)[0] | ||
| TOOLCHAIN_RS.write_text( | ||
| toolchain_rs.replace( | ||
| f'{prefix}{old_version}"', | ||
| f'{prefix}{version}"', | ||
| ) | ||
| ) | ||
|
|
||
| for test_setup in (INTEGRATION_TEST_CONFTEST, QL_TEST_SETUP): | ||
| contents = test_setup.read_text() | ||
| test_setup.write_text( | ||
| contents.replace( | ||
| f"rustup toolchain install {old_version} ", | ||
| f"rustup toolchain install {version} ", | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def align_rust_analyzer_versions() -> None: | ||
| """Align ra_ap dependencies and update the lockfile when versions differ. | ||
|
|
||
| rust-analyzer crates may be published gradually, so this selects the newest | ||
| version that is available for every ra_ap dependency. | ||
| """ | ||
| manifest = RUST_EXTRACTOR_MANIFEST.read_text() | ||
| dependencies = { | ||
| name: version | ||
| for name, version in tomllib.loads(manifest)["dependencies"].items() | ||
| if name.startswith("ra_ap_") | ||
| } | ||
| if len(set(dependencies.values())) == 1: | ||
| # All the `ra_ap_` dependencies agree | ||
| return | ||
|
|
||
| version = get_rust_analyzer_version() | ||
|
|
||
| for name, old_version in dependencies.items(): | ||
| manifest = manifest.replace( | ||
| f'{name} = "{old_version}"', | ||
| f'{name} = "{version}"', | ||
| ) | ||
| RUST_EXTRACTOR_MANIFEST.write_text(manifest) | ||
| run("cargo", "update") | ||
|
|
||
|
|
||
| def commit_all(title: str) -> None: | ||
| run("git", "add", "--all") | ||
| run("git", "commit", "--no-verify", "--allow-empty", "-m", title) | ||
|
|
||
|
|
||
| def main() -> None: | ||
| status = subprocess.run( | ||
| ["git", "status", "--porcelain"], | ||
| cwd=REPO_ROOT, | ||
| check=True, | ||
| capture_output=True, | ||
| text=True, | ||
| ).stdout | ||
| if status: | ||
| raise RuntimeError("the working tree must be clean") | ||
|
|
||
| print_step(1, "Update dependencies") | ||
| old_rust_analyzer_version = get_rust_analyzer_version() | ||
| run("cargo", "upgrade", "--incompatible", "--pinned") | ||
| new_rust_analyzer_version = get_rust_analyzer_version() | ||
| if new_rust_analyzer_version == old_rust_analyzer_version: | ||
| run("git", "restore", ".") | ||
| print("No new rust-analyzer version available.") | ||
| return | ||
|
|
||
| align_rust_analyzer_versions() | ||
| commit_all("Cargo: Upgrade dependencies") | ||
|
|
||
| print_step(2, "Update the fixed Rust toolchain used by the extractor") | ||
| rust_toolchain = get_compatible_rust_toolchain(new_rust_analyzer_version) | ||
| update_fixed_rust_toolchain(rust_toolchain) | ||
| commit_all("Rust: Update fixed toolchain") | ||
|
|
||
| print_step(3, "Regenerate vendored bazel files") | ||
| run("misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh") | ||
| commit_all("Bazel: Regenerate vendored cargo dependencies") | ||
|
|
||
| print_step(4, "Run codegen") | ||
| run_codegen() | ||
| commit_all("Rust: Run codegen") | ||
|
|
||
| print_step(5, "Try compiling") | ||
| run("bazel", "run", "//rust:install") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.