Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/1password-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,45 @@ The workflow uses 1Password GitHub Action to load secrets:

## Troubleshooting

### GitHub repository identity

The canonical repository is `apache/reqsign`. GitHub uses immutable OIDC
subjects for this repository, which include the organization and repository IDs:

- Main: `repo:apache@47359/reqsign@469069490:ref:refs/heads/main`
- Pull requests: `repo:apache@47359/reqsign@469069490:pull_request`

Check the live configuration before changing cloud trust policies:

```bash
gh api repos/apache/reqsign/actions/oidc/customization/sub
```

Repository redirects do not migrate cloud identity bindings. Keep these external
settings aligned with the repository identity:

- AWS: update the GitHub OIDC trust subjects on both
`REQSIGN_AWS_LIVE_ROLE_ARN` and `op://reqsign/aws-v4/web_identity_role_arn`.
Preserve the `sts.amazonaws.com` audience. The live-test role template is
[cloudformation.yml](aws-live-tests/cloudformation.yml); changing the template
alone does not update an existing role.
- Azure: update the CI application's federated identity credentials for both
subjects above, retaining the `api://AzureADTokenExchange` audience and GitHub
issuer. Azure DevOps service connections use a separate identity.
- Google Cloud: update service-account IAM members that bind the workload
identity pool's `attribute.repository` to `apache/reqsign`. Check provider
attribute conditions as well. This attribute uses the repository name, not
the OIDC subject with numeric IDs.
- Azure Pipelines: check the pipeline's external GitHub repository binding as
well as the clone URL in `azure-pipelines.yml`.
- crates.io: audit each crate's Trusted Publisher for `apache/reqsign`,
`release.yml`, and the `release` environment. See the
[release helpers](scripts/release_rust/README.md); public metadata verification
cannot establish the authenticated publisher configuration.

After updating cloud bindings, rerun the failed live-test jobs to verify actual
token exchange and access to the test resources.

### Tests are skipped
- Check if the PR is from a fork (integration tests don't run on forked PRs)
- Verify 1Password Connect is accessible
Expand Down
14 changes: 10 additions & 4 deletions .github/aws-live-tests/cloudformation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,15 @@ Parameters:
GitHubOrganization:
Type: String
Default: apache
GitHubOrganizationId:
Type: String
Default: "47359"
GitHubRepository:
Type: String
Default: opendal-reqsign
Default: reqsign
GitHubRepositoryId:
Type: String
Default: "469069490"
TestBucketName:
Type: String
Default: reqsign-aws-v4-test
Expand Down Expand Up @@ -319,10 +325,10 @@ Resources:
Condition:
StringEquals:
token.actions.githubusercontent.com:aud: sts.amazonaws.com
StringLike:
# GitHub immutable subjects include both names and numeric IDs.
token.actions.githubusercontent.com:sub:
- !Sub repo:${GitHubOrganization}/${GitHubRepository}:ref:refs/heads/main
- !Sub repo:${GitHubOrganization}/${GitHubRepository}:pull_request
- !Sub repo:${GitHubOrganization}@${GitHubOrganizationId}/${GitHubRepository}@${GitHubRepositoryId}:ref:refs/heads/main
- !Sub repo:${GitHubOrganization}@${GitHubOrganizationId}/${GitHubRepository}@${GitHubRepositoryId}:pull_request
Policies:
- PolicyName: reqsign-live-test-runtime
PolicyDocument:
Expand Down
6 changes: 5 additions & 1 deletion .github/scripts/release_rust/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Trusted Publishing cannot create the first version of a crate.
credentials.
- `apply` authenticates every existing crate before any write, publishes a
dependency-free `0.0.0` namespace reservation for each missing name,
configures the exact `apache/opendal-reqsign`, `release.yml`, `release`
configures the exact `apache/reqsign`, `release.yml`, `release`
Trusted Publisher, enables `trustpub_only`, and performs a final authenticated
audit.
- `verify` checks public crate metadata and `trustpub_only`. It cannot verify
Expand All @@ -49,6 +49,10 @@ including when discovery finds no missing names. It never changes an
established crate. Existing crates must be migrated independently before the
workflow can succeed.

Published crate metadata may still reference the former repository URL. The
audit accepts that historical metadata, but every Trusted Publisher must target
`apache/reqsign`. New placeholders and releases use the current repository URL.

Version `0.0.0` is an irreversible namespace reservation. It is not an ASF
software release and contains no implementation.

Expand Down
17 changes: 10 additions & 7 deletions .github/scripts/release_rust/bootstrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,20 +36,22 @@


REGISTRY_URL = "https://crates.io"
REPOSITORY = "https://github.com/apache/opendal-reqsign"
REPOSITORY = "https://github.com/apache/reqsign"
# Published versions retain their original manifest metadata after a repo rename.
LEGACY_REPOSITORY = "https://github.com/apache/opendal-reqsign"
PLACEHOLDER_VERSION = "0.0.0"
PLACEHOLDER_DESCRIPTION = (
"Namespace reservation for a crate planned by Apache OpenDAL reqsign."
)
PUBLISHER = {
"repository_owner": "apache",
"repository_name": "opendal-reqsign",
"repository_name": "reqsign",
"workflow_filename": "release.yml",
"environment": "release",
}
USER_AGENT = (
"apache-opendal-reqsign-release-bootstrap/1.0 "
"(https://github.com/apache/opendal-reqsign)"
"apache-reqsign-release-bootstrap/1.0 "
"(https://github.com/apache/reqsign)"
)


Expand Down Expand Up @@ -267,9 +269,10 @@ def validate_crate_metadata(
raise RuntimeError(
f"crate name mismatch for {planned.name}: got {metadata.get('id')!r}"
)
if _normalized_repository(metadata.get("repository")) != _normalized_repository(
REPOSITORY
):
if _normalized_repository(metadata.get("repository")) not in {
_normalized_repository(REPOSITORY),
_normalized_repository(LEGACY_REPOSITORY),
}:
raise RuntimeError(
f"{planned.name} already exists with an unexpected repository: "
f"{metadata.get('repository')!r}"
Expand Down
41 changes: 40 additions & 1 deletion .github/scripts/release_rust/test_bootstrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
from unittest import mock

from bootstrap import CratesIoClient
from bootstrap import LEGACY_REPOSITORY
from bootstrap import PLACEHOLDER_DESCRIPTION
from bootstrap import PLACEHOLDER_VERSION
from bootstrap import PUBLISHER
Expand Down Expand Up @@ -124,6 +125,8 @@ def test_publisher_matches_the_release_workflow(self):
publish_job = workflow.split("\n publish:\n", 1)[1]

self.assertEqual(PUBLISHER["workflow_filename"], "release.yml")
self.assertEqual(PUBLISHER["repository_owner"], "apache")
self.assertEqual(PUBLISHER["repository_name"], "reqsign")
self.assertEqual(PUBLISHER["environment"], "release")
self.assertIn(" environment: release\n", publish_job)
self.assertIn("github.event_name == 'push'", publish_job)
Expand Down Expand Up @@ -182,7 +185,7 @@ def test_public_read_retries_a_transient_registry_failure(self):
)
response = JsonResponse(
b'{"crate":{"id":"reqsign","repository":'
b'"https://github.com/apache/opendal-reqsign"}}'
b'"https://github.com/apache/reqsign"}}'
)

with (
Expand Down Expand Up @@ -253,6 +256,42 @@ def test_authenticated_preflight_rejects_an_unexpected_publisher(self):
self.assertEqual(client.created_configs, 0)
self.assertEqual(client.restricted, 0)

def test_authenticated_preflight_accepts_historical_repository_metadata(self):
planned = PlannedCrate("reqsign-core", "core")
krate = metadata(planned.name, trustpub_only=True)
krate["repository"] = LEGACY_REPOSITORY
client = FakeClient(planned.name, krate, [expected_config(planned.name)])

self.assertEqual(
preflight_authenticated([planned], set(), client), [planned.name]
)
self.assertEqual(verify_authenticated([planned], client), [planned.name])
self.assertEqual(client.created_configs, 0)

def test_authenticated_preflight_rejects_historical_publisher(self):
planned = PlannedCrate("reqsign-core", "core")
krate = metadata(planned.name, trustpub_only=True)
krate["repository"] = LEGACY_REPOSITORY
config = {**expected_config(planned.name), "repository_name": "opendal-reqsign"}
client = FakeClient(planned.name, krate, [config])

with self.assertRaisesRegex(RuntimeError, "unexpected Trusted Publisher"):
preflight_authenticated([planned], set(), client)

self.assertEqual(client.created_configs, 0)

def test_discovery_rejects_an_unrelated_repository(self):
planned = PlannedCrate("reqsign-core", "core")
krate = metadata(planned.name)
krate["repository"] = "https://github.com/other/reqsign"
client = FakeClient(planned.name, krate)

with (
mock.patch("bootstrap.planned_crates", return_value=[planned]),
self.assertRaisesRegex(RuntimeError, "unexpected repository"),
):
discover(Path(), client)

def test_authenticated_preflight_rejects_unmigrated_established_crate(self):
planned = PlannedCrate("reqsign-core", "core")
client = FakeClient(
Expand Down
4 changes: 1 addition & 3 deletions .github/scripts/release_rust/trusted_publishing.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,7 @@


DEFAULT_REGISTRY_URL = "https://crates.io"
USER_AGENT = (
"apache-opendal-reqsign-release/1.0 (https://github.com/apache/opendal-reqsign)"
)
USER_AGENT = "apache-reqsign-release/1.0 (https://github.com/apache/reqsign)"


def _response_error(operation: str, error: urllib.error.HTTPError) -> RuntimeError:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/aws_v4.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@ jobs:
id: check
run: |
if [[ "${{ github.event_name }}" == "push" || ( "${{ github.event_name }}" == "pull_request" && "${{ github.event.pull_request.head.repo.fork }}" == "false" && "${{ github.actor }}" != "dependabot[bot]" ) ]]; then
echo "has_secrets=true" >> $GITHUB_OUTPUT
echo "has_secrets=true" >> "$GITHUB_OUTPUT"
echo "::notice::Integration tests will be executed (base repository and non-dependabot PR)"
else
echo "has_secrets=false" >> $GITHUB_OUTPUT
echo "has_secrets=false" >> "$GITHUB_OUTPUT"
echo "::warning::Integration tests will be skipped (forked repository, dependabot PR, or no secrets available)"
fi

Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ resolver = "3"
[workspace.package]
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/apache/opendal-reqsign"
repository = "https://github.com/apache/reqsign"
rust-version = "1.86.0"

[workspace.dependencies]
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

[![Build Status]][actions] [![Latest Version]][crates.io] [![Crate Downloads]][crates.io]

[Build Status]: https://img.shields.io/github/actions/workflow/status/apache/opendal-reqsign/ci.yml?branch=main
[actions]: https://github.com/apache/opendal-reqsign/actions?query=branch%3Amain
[Build Status]: https://img.shields.io/github/actions/workflow/status/apache/reqsign/ci.yml?branch=main
[actions]: https://github.com/apache/reqsign/actions?query=branch%3Amain
[Latest Version]: https://img.shields.io/crates/v/reqsign.svg
[crates.io]: https://crates.io/crates/reqsign
[Crate Downloads]: https://img.shields.io/crates/d/reqsign.svg
Expand Down Expand Up @@ -183,7 +183,7 @@ Check out the [CONTRIBUTING.md](CONTRIBUTING.md) guide for more details on getti

## Getting help

Submit [issues](https://github.com/apache/opendal-reqsign/issues/new/choose) for bug report or asking questions in [discussion](https://github.com/apache/opendal-reqsign/discussions/new?category=q-a).
Submit [issues](https://github.com/apache/reqsign/issues/new/choose) for bug report or asking questions in [discussion](https://github.com/apache/reqsign/discussions/new?category=q-a).

## Acknowledge

Expand Down
2 changes: 1 addition & 1 deletion azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ steps:
esac

git fetch --no-tags --depth=1 \
https://github.com/apache/opendal-reqsign.git \
https://github.com/apache/reqsign.git \
"$GITHUB_REF"
fetched_sha=$(git rev-parse 'FETCH_HEAD^{commit}')
test "$fetched_sha" = "$GITHUB_SHA"
Expand Down
48 changes: 24 additions & 24 deletions release/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Use this skill when preparing or executing an Apache OpenDAL reqsign release.
- Do not push the formal `vX.Y.Z` tag before the Apache vote passes.
- The voted release candidate tag is `vX.Y.Z-rc.N`; it must be a signed tag.
- The formal `vX.Y.Z` tag must point to the exact same commit as the voted RC tag, even if `main` has advanced after the vote started.
- Every crate in the current publish plan must exist with the exact `apache/opendal-reqsign`, `release.yml`, `release` Trusted Publisher and `trustpub_only` enabled before creating the RC tag.
- Every crate in the current publish plan must exist with the exact `apache/reqsign`, `release.yml`, `release` Trusted Publisher and `trustpub_only` enabled before creating the RC tag.
- The repo release workflow publishes only formal `vX.Y.Z` tags. It uses short-lived GitHub OIDC credentials and never a long-lived crates.io token.
- Source release artifacts live under Apache dist:
- RC: `https://dist.apache.org/repos/dist/dev/opendal/reqsign-X.Y.Z/`
Expand Down Expand Up @@ -58,7 +58,7 @@ Use this skill when preparing or executing an Apache OpenDAL reqsign release.
## Bootstrap Rust Crates

Wait until the intended crate-name set is present on the
`apache/opendal-reqsign` `main` branch. The release manager chooses the exact
`apache/reqsign` `main` branch. The release manager chooses the exact
reservation time, normally about three days before the planned release. At that
time, check out the current `main` commit and run:

Expand All @@ -74,7 +74,7 @@ but it is a namespace reservation rather than an ASF software release.

The helper:

- Requires a clean checkout at the current `apache/opendal-reqsign` `main`.
- Requires a clean checkout at the current `apache/reqsign` `main`.
- Verifies that the `rust-bootstrap` environment has required reviewers.
- Dispatches the input-free `bootstrap_rust_crates.yml` workflow.
- Resolves the exact run, verifies its `headSha`, and waits for completion.
Expand All @@ -87,7 +87,7 @@ before any write. Established crates must already have exactly one Trusted
Publisher with:

- Repository owner: `apache`
- Repository name: `opendal-reqsign`
- Repository name: `reqsign`
- Workflow filename: `release.yml`
- Environment: `release`

Expand Down Expand Up @@ -129,22 +129,22 @@ The RC tag should not trigger the formal publish workflow.
Create the Apache source artifact from the RC tag.

```bash
rm -rf /tmp/opendal-reqsign-release-X.Y.Z
mkdir -p /tmp/opendal-reqsign-release-X.Y.Z/dist
rm -rf /tmp/reqsign-release-X.Y.Z
mkdir -p /tmp/reqsign-release-X.Y.Z/dist

git archive \
--format=tar.gz \
--prefix=apache-opendal-reqsign-X.Y.Z/ \
-o /tmp/opendal-reqsign-release-X.Y.Z/dist/apache-opendal-reqsign-X.Y.Z.tar.gz \
--prefix=apache-reqsign-X.Y.Z/ \
-o /tmp/reqsign-release-X.Y.Z/dist/apache-reqsign-X.Y.Z.tar.gz \
vX.Y.Z-rc.N

cd /tmp/opendal-reqsign-release-X.Y.Z/dist
gpg --armor --detach-sign apache-opendal-reqsign-X.Y.Z.tar.gz
shasum -a 512 apache-opendal-reqsign-X.Y.Z.tar.gz > apache-opendal-reqsign-X.Y.Z.tar.gz.sha512
cd /tmp/reqsign-release-X.Y.Z/dist
gpg --armor --detach-sign apache-reqsign-X.Y.Z.tar.gz
shasum -a 512 apache-reqsign-X.Y.Z.tar.gz > apache-reqsign-X.Y.Z.tar.gz.sha512

gpg --verify apache-opendal-reqsign-X.Y.Z.tar.gz.asc apache-opendal-reqsign-X.Y.Z.tar.gz
shasum -a 512 -c apache-opendal-reqsign-X.Y.Z.tar.gz.sha512
tar -tzf apache-opendal-reqsign-X.Y.Z.tar.gz | rg '(^|/)LICENSE$|(^|/)NOTICE$|(^|/)Cargo.toml$'
gpg --verify apache-reqsign-X.Y.Z.tar.gz.asc apache-reqsign-X.Y.Z.tar.gz
shasum -a 512 -c apache-reqsign-X.Y.Z.tar.gz.sha512
tar -tzf apache-reqsign-X.Y.Z.tar.gz | rg '(^|/)LICENSE$|(^|/)NOTICE$|(^|/)Cargo.toml$'
```

Confirm the signing key is present in Apache OpenDAL KEYS:
Expand All @@ -163,7 +163,7 @@ svn co --depth=empty https://dist.apache.org/repos/dist/dev/opendal /tmp/opendal

cd /tmp/opendal-dist-dev-reqsign-X.Y.Z
mkdir reqsign-X.Y.Z
cp /tmp/opendal-reqsign-release-X.Y.Z/dist/* reqsign-X.Y.Z/
cp /tmp/reqsign-release-X.Y.Z/dist/* reqsign-X.Y.Z/
svn add reqsign-X.Y.Z
svn status
svn commit --force-interactive -m "Prepare reqsign X.Y.Z release candidate"
Expand All @@ -173,16 +173,16 @@ Verify the remote copy:

```bash
svn ls https://dist.apache.org/repos/dist/dev/opendal/reqsign-X.Y.Z/
rm -rf /tmp/opendal-reqsign-verify-X.Y.Z
svn co https://dist.apache.org/repos/dist/dev/opendal/reqsign-X.Y.Z /tmp/opendal-reqsign-verify-X.Y.Z
cd /tmp/opendal-reqsign-verify-X.Y.Z
shasum -a 512 -c apache-opendal-reqsign-X.Y.Z.tar.gz.sha512
gpg --verify apache-opendal-reqsign-X.Y.Z.tar.gz.asc apache-opendal-reqsign-X.Y.Z.tar.gz
rm -rf /tmp/reqsign-verify-X.Y.Z
svn co https://dist.apache.org/repos/dist/dev/opendal/reqsign-X.Y.Z /tmp/reqsign-verify-X.Y.Z
cd /tmp/reqsign-verify-X.Y.Z
shasum -a 512 -c apache-reqsign-X.Y.Z.tar.gz.sha512
gpg --verify apache-reqsign-X.Y.Z.tar.gz.asc apache-reqsign-X.Y.Z.tar.gz
```

## Start Vote

Create a GitHub Discussion in `apache/opendal-reqsign` General.
Create a GitHub Discussion in `apache/reqsign` General.

Title:

Expand All @@ -207,7 +207,7 @@ https://downloads.apache.org/opendal/KEYS

Git tag for the release candidate:

https://github.com/apache/opendal-reqsign/releases/tag/vX.Y.Z-rc.N
https://github.com/apache/reqsign/releases/tag/vX.Y.Z-rc.N

The tag points to commit:

Expand Down Expand Up @@ -295,8 +295,8 @@ NAME
5. Monitor the GitHub Release workflow.

```bash
gh run list --repo apache/opendal-reqsign --workflow Release --limit 5
gh run view RUN_ID --repo apache/opendal-reqsign --json status,conclusion,url,jobs
gh run list --repo apache/reqsign --workflow Release --limit 5
gh run view RUN_ID --repo apache/reqsign --json status,conclusion,url,jobs
```

The workflow validates the complete workspace package set, then publishes
Expand Down
Loading
Loading