fix: serialize whole-record bucket metadata updates

[Vonng]

Contribution Licensing (no CLA, inbound=outbound, DCO required)

Every commit carries a DCO Signed-off-by trailer. The red-test commits intentionally precede their fixes so the reproduced failures remain auditable in history.

Description

Introduce one distributed namespace lock for each real bucket metadata record:

.minio.sys / buckets/<bucket>/metadata.lock

The PR serializes the authoritative disk read, mutation, .metadata.bin save, and local cache publication for:

• ordinary BucketMetadataSys.Update/Delete;
• local and peer CORS transitions;
• legacy/bulk peer metadata updates;
• normal/ForceCreate bucket creation and site-replication adoption;
• metadata import commits;
• legacy-config and target-config migrations.

Peer metadata reload fan-out and BucketMetaHook dispatch happen only after the lock is released. BucketMetadata.Save and the .metadata.bin object lock remain lock-free at this layer to avoid non-reentrant self-deadlock.

The PR also:

• prevents ForceCreate from replacing an existing bucket's policy, CORS, tags, SSE, timestamps, and other fields;
• prevents genuine new bucket creation from adopting orphan metadata from a deleted bucket incarnation;
• preserves object-lock => versioning-enabled invariants;
• rebases imports onto a fresh locked record and sends only imported types to site replication;
• bounds one-shot migration lock acquisition to five seconds, falling back to the already loaded configuration in memory only when the lock itself is unavailable;
• computes lifecycle deletion metadata from the locked current XML;
• updates the CORS LWW design record and corrects its old Audit site-replication source timestamps, tombstones, and per-site counters #77 references.

Refs #102.

Motivation and Context

.metadata.bin is one whole-record object. Previously, different config writers could read the same old record and save different one-field changes. The last save silently discarded the other field. CORS had a separate cors-config.lock, which serialized CORS with itself but not with policy, tagging, SSE, import, migration, or bucket creation.

The deterministic red tests demonstrate both policy overwriting CORS and tagging overwriting SSE on disk and in the resident cache. A separate red test demonstrates MakeBucket{ForceCreate:true} replacing Created, policy, and CORS with a fresh record.

How to test this PR?

Passed locally on macOS/arm64 with Go 1.27.0:

• deterministic policy+CORS and tagging+SSE barrier tests on ErasureSD and Erasure16;
• peer-bulk+local and lifecycle-delete+SSE conflict tests;
• ForceCreate preservation, ghost non-adoption, object-lock/versioning, peer adoption, import patch, and caller-cancellation tests;
• the focused metadata suite under -race;
• existing CORS, lifecycle, and site-adoption tests;
• go build ./...;
• go vet ./cmd;
• go test ./cmd -count=1 (final run: 128.758s).

Claude Code Opus 5 performed two adversarial implementation reviews. The first found one P0 and two P1 MakeBucket edge cases, all corrected with dedicated regressions. Final verdict: GO, no P0/P1; remaining P2 items are documented tradeoffs or deferred #102 follow-ups.

Compatibility impact

• No S3 wire, .metadata.bin schema, object format, or API response format changes.
• Lock order is <bucket>.lck -> metadata.lock -> .metadata.bin.
• Old nodes use cors-config.lock; new nodes use metadata.lock. The serialization guarantee is therefore complete only after every node in a cluster is upgraded. Avoid bucket-metadata writes during the rolling window.
• The on-disk format is unchanged, so rollback remains format-compatible.
• Genuine bucket creation deliberately starts with fresh metadata and will not adopt a readable orphan record from an older bucket incarnation.
• ForceCreate preserves valid existing metadata; corrupt/future-version metadata fails loudly rather than being silently overwritten.
• Legacy or target-config migration waits at most five seconds for metadata.lock; if the lock is unavailable, reads use the already loaded configuration in memory and retry persistence later. Real parse, I/O, KMS, and save errors still surface.
• Import replication now dispatches only fields actually imported, preventing unrelated metadata from receiving a fresh import timestamp.

Deliberately deferred under #102:

• higher-level site-replication lifecycle XML merge atomicity;
• DeleteBucket versus an already in-flight metadata writer;
• overlapping peer reloads leaving a local cache one save behind until refresh.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Optimization
• Breaking change

Checklist:

• All commits are signed off (git commit -s) per the DCO
• Unit tests added/updated
• make verifiers passes
• Relevant package tests and build pass
• Compatibility and rollback impact documented
• Internal documentation updated
• Public documentation update opened in pgsty/silo.pgsty.com