Prototype cross-platform LTS upgrade CI

[Amaury Chamayou (achamayou)]

Why

CCF 7.x is built on Azure Linux 3, while 8.x is planned to move to Azure Linux 4. The existing LTS compatibility test executes every node in one userspace, so it cannot exercise a rolling upgrade across those distribution boundaries.

This prototype validates the topology by upgrading CCF 7.0.14 on Azure Linux 3 to CCF 7.0.15 on Azure Linux 4. Manual dispatch exposes both versions as top-level parameters.

Approach

• Build each selected CCF tag in its own full CI build container on the matching Azure Linux version, using the existing VMSS pools.
• Export /opt/ccf install trees as artifacts; development RPM packaging remains unchanged.
• Download both artifacts in a job running directly on an Azure Linux 3 VMSS host.
• Provision Python 3.12 and the test environment on the host with the repository-pinned uv installer.
• Build separate runtime images that install only runtime shared-library packages and copy in the matching install tree.
• Let the existing Python LTS harness orchestrate CCF nodes through Docker host networking.
• Preserve mixed-version traffic, receipt checks, node retirement, the all-new service, and a final compatibility node.
• Run weekly or by manual dispatch, with bounded build and upgrade timeouts rather than adding this expensive path to every pull request. An exact prototype-branch push trigger is temporarily enabled while this draft is developed.

sequenceDiagram
    participant CI as GitHub Actions
    participant B3 as AL3 build container
    participant B4 as AL4 build container
    participant VM as AL3 VMSS host<br/>Python controller
    participant Old as AL3 runtime containers<br/>2 initial nodes
    participant New as AL4 runtime containers<br/>2 replacements + 1 final node

    par Build from_version
        CI->>B3: Checkout CCF 7.0.14 by default
        B3->>B3: Build and install /opt/ccf
        B3-->>CI: Upload AL3 install-tree artifact
    and Build to_version
        CI->>B4: Checkout CCF 7.0.15 by default
        B4->>B4: Build and install /opt/ccf
        B4-->>CI: Upload AL4 install-tree artifact
    end

    CI->>VM: Start upgrade job directly on VM
    VM->>CI: Download both install trees
    VM->>VM: Provision Python and test dependencies
    VM->>VM: Build separate AL3 and AL4 runtime images

    VM->>Old: Start and open 2-node CCF service
    VM->>Old: Submit transactions
    VM->>Old: Update constitution for target version

    loop Join 2 AL4 replacement nodes
        VM->>New: Start one AL4 node
        New->>Old: Join existing service
        VM->>New: Trust new node
    end

    Note over Old,New: Mixed service: 2 AL3 + 2 AL4 nodes
    VM->>Old: Exercise mixed-version traffic
    VM->>New: Verify node versions

    loop Retire 2 AL3 nodes
        VM->>Old: Retire and stop one AL3 node
    end

    Note over New: All-AL4 service: 2 nodes
    VM->>New: Refresh keys and update JS application
    VM->>New: Add and trust final AL4 compatibility node
    VM->>New: Renew certificates, submit transactions, verify receipts
    VM->>Old: Remove stopped containers
    VM->>New: Stop and remove containers
    VM-->>CI: Upload logs and compatibility report

Loading

Validation

The complete CCF 7.0.14 AL3 to CCF 7.0.15 AL4 workflow passed on the VMSS pools: run 34606789560.

Notes for review

The build containers contain the complete compilation toolchain. They do not become node images. The runtime images are built separately from clean Azure Linux bases, install only runtime libraries, and copy in the matching pre-built install tree.

The Python controller runs on the VM. Node containers use host networking and bind-mounted workspaces so the existing controller can inspect certificates, ledgers, snapshots, stdout, and stderr without a second remote-execution abstraction.

This remains a draft while the prototype design is reviewed.

Closes #8057