Add CPU placement policy, CPU topology and pool reporting by rucoder · Pull Request #155 · lf-edge/eve-api

rucoder · 2026-08-17T17:33:41Z

Adds the API for topology-aware CPU placement of edge applications: the
controller states what a workload needs, the device picks the host CPUs, and
reports what it can do and what it did.

The device side is lf-edge/eve#6335. It builds against this branch (c338d02)
through a replace directive until this merges, then takes it with
make bump-eve-api. Both sides were tested together: EVE's evetest CPU placement
tests with an Adam built on this API, and end to end against a controller and UI
built on it.

Two pieces: what the node has, and whether it applies the fields

Piece	Says	Here
Hardware inventory	what the node has: static, read at boot	CPU topology, cache domains, `KernelCPUIsolation`
`OptionalCapabilities`	this node applies the fields: per build flavor and hardware	`cpu_placement`: KVM on any architecture, with a CPU topology EVE could read

The controller gates the placement fields (28-32) on
optional_capabilities.cpu_placement; an EVE that predates them does not set
it, so no APICapability value is needed. pin_cpu predates them and may go
to any node. The inventory is never a gate: reporting is not permission (cache
domains are reported on every node, but that does not mean EVE can allocate
cache). It sizes requests instead, e.g. two-thread cores before
threads_per_core=2, isolated CPUs before ISOLATED. State sits beside the
two: what is free now (cpu_pools) and what each app got
(cpu_placement_status, cpu_placement_error, assigned_cpus).

Config: what a workload asks for (`config/vm.proto`)

Two independent choices, each meaning the same whatever the other says:

cpu_policy picks the pool the workload's CPUs come from: HOUSEKEEPING (the
default pool for workloads as well as EVE's processes, holding the CPUs EVE
reserves for its services), DEDICATED (CPUs of its own, no other workload on
them) or ISOLATED (CPUs of its own that the kernel isolates with isolcpus).
pin_cpu (field 20) is the pinning choice on any pool: each vCPU of a VM on a
host CPU of its own. With false EVE pins nothing, and the workload's threads
run on the pool's CPUs as the kernel or the workload places them; a real-time
container in the isolated pool sets its own affinity. A container has no vCPUs
to pin.

full_pcpus_only, threads_per_core, numa_policy and io_placement refine a
DEDICATED or ISOLATED request: whole physical cores, how many SMT threads of
each the workload uses, NUMA confinement, and where a VM's non-vCPU threads run.
The first three follow the Kubernetes CPU and Topology Managers.

A request the device cannot serve as sent, including a combination this EVE does
not implement, is refused, and ZInfoApp.cpu_placement_error says why; nothing
is reinterpreted or silently dropped. With cpu_policy unset, pin_cpu=true
means a dedicated workload, pinned on a VM, for a controller that does not send
cpu_policy; with both unset, the workload runs in the housekeeping pool.

Info: what the node has, and what it did

info/hardware.proto: each logical CPU reports its socket, physical core,
NUMA node, L2 and L3 cache domains, core class and base and maximum frequency;
CPUInfo.caches lists the cache domains with the CPUs sharing each. The
coordinates are optional: a value the device cannot read is unset, never
zero. CoreClass is reported for information; placement does not use it.
KernelCPUIsolation reports the CPUs the running kernel isolates (isolcpus)
and runs tickless (nohz_full). PCI devices report their local CPUs, whose
NUMA node is the device's.

info/info.proto:

ZInfoDevice.cpu_pools (62): each pool's CPUs, free CPUs, and thread and
whole-core counts, so a controller can tell whether a workload fits before
deploying it. A pool also carries warnings (CPUPoolWarning: a
CPUPoolIssue and its ErrorInfo) for mistakes in isolcpus.
ZInfoApp.assigned_cpus (21): the CPUs a dedicated or isolated workload
holds, as assignedAdapters reports its PCI devices.

ZInfoApp.cpu_placement_status (22): where a workload stands with its CPU
request, and ZInfoApp.cpu_placement_error (23): why it failed, set only
with FAILED. appErr keeps the text and retry_condition. Targeted
enums rather than a code in ErrorInfo, which is reported for many things
besides apps; a value keeps its meaning for good, and a retired one is
reserved.

`CPU_PLACEMENT_STATUS_`	meaning
`PLACED`	running on the CPUs the plan gives it
`WAITING`	not started yet; workloads stopping or in the same repack still hold its CPUs (WARNING)
`REPACK_ADVISED`	running, and part of a repack (WARNING)
`FAILED`	refused, not started; `cpu_placement_error` says why (ERROR)

`CPU_PLACEMENT_ERROR_`	meaning
`NEEDS_REPACK`	fits once the workloads in the repack restart together
`INSUFFICIENT`	no repack makes room
`POLICY_INVALID`	a field the request cannot carry, a value out of range, or a combination this EVE does not implement
`ODD_VCPU`	`threads_per_core=2` with an odd vCPU count
`TOPOLOGY_UNSUPPORTED`	unknown topology, no two-thread core, or a hypervisor without per-vCPU placement
`ISOLATION_UNAVAILABLE`	`ISOLATED` on a node that isolates no CPU

The controller's one automatic action is the repack: it restarts every app
with NEEDS_REPACK or in REPACK_ADVISED together. The device retries no
refusal; it stays until the config or the node changes, as for an
unavailable adapter.

The device-side code in this branch needs API that has not landed upstream yet: the VmConfig CPU placement fields, the CPU topology and capability reporting on device info, ZInfoDevice.cpu_pools, API_CAPABILITY_CPU_PLACEMENT_POLICY and ErrorInfo.error_code. Without them nothing here compiles, so this commit points pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155) through a replace directive, and vendors it. This commit exists only so the branch can be built, run and reviewed while the API is under discussion. It must be dropped and replaced by an ordinary `make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive pointing at a personal fork breaks dependency tracking and SBOM/licensing, and is never acceptable on master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

eriknordmark · 2026-10-12T15:34:50Z

  API_CAPABILITY_REPORT_TPM_EVENTLOG = 21; // Support for reporting TPM Event Log "as is" without parsing and selectively reporting events
  API_CAPABILITY_APP_INSTANCE_NET_INTERFACE_CHANGE = 22; // Support for adding and removing network interfaces without purge, but restart or the edge app
  API_CAPABILITY_DEFERRED_QUEUE_METRICS = 23; // Reports deviceMetric.deferred_queue and the per-URL retriable, rejected and delivered counts
+  API_CAPABILITY_CPU_PLACEMENT_POLICY = 24; // VmConfig CPU placement (cpu_policy incl. CPU_POLICY_ISOLATED, pin_cpu as the pinning choice, full_pcpus_only, threads_per_core, numa_policy, io_placement): each field is applied, or the workload is refused with a structured error_code. Advertised only where EVE implements CPU placement


A device which supports this will also always(?) include a CPUPoolUtilization cpu_pools with at least the housekeeping pool, correct?
Thus one could have the controllers check against that. But perhaps it is still helpful to have an explicit capability to signal this explicitly?

@eriknordmark yes, correct. current controller and UI do these checks. but since this feature will be evolving, does it make sense to add "sub capabilities" or we will introduce to many capabilities potentially.

@rucoder Can OptionalCapabilities be used for as you said 'sub capabilities' ?

@xyuria-zededa optional is really for things which might exist only in some images flavors (we use it for EVE-k).
@rucoder are we only adding this for amd64 images and not arm64? If so OptionalCapabilities makes a lot of sense.

With an OptionalCapabilties field there is no need for an API capability. Remove it.

The device-side code in this branch needs API that has not landed upstream yet: the VmConfig CPU placement fields, the CPU topology and capability reporting on device info, ZInfoDevice.cpu_pools, API_CAPABILITY_CPU_PLACEMENT_POLICY and ErrorInfo.error_code. Without them nothing here compiles, so this commit points pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155) through a replace directive, and vendors it. This commit exists only so the branch can be built, run and reviewed while the API is under discussion. It must be dropped and replaced by an ordinary `make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive pointing at a personal fork breaks dependency tracking and SBOM/licensing, and is never acceptable on master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

The device-side code in this branch needs API that has not landed upstream yet: the VmConfig CPU placement fields, the CPU topology and kernel isolation in the hardware inventory, ZInfoDevice.cpu_pools, ZInfoApp.cpu_placement_status and assigned_cpus, and the CPU placement capabilities. Without them nothing here compiles, so this commit points pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155) through a replace directive, and vendors it. This commit exists only so the branch can be built, run and reviewed while the API is under discussion. It must be dropped and replaced by an ordinary `make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive pointing at a personal fork breaks dependency tracking and SBOM/licensing, and is never acceptable on master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

rene · 2026-10-14T09:52:08Z

+enum CoreClass {
+        // EVE could not determine the class.
+        CORE_CLASS_UNSPECIFIED = 0;
+        CORE_CLASS_PERFORMANCE = 1; // e.g. Intel P-core, ARM "big"


Comment could follow the pattern and be above the definition....

@rene which style do we follow? above the definition or on the right?

xyuria-zededa

Looks good from the api perspective

The device-side code in this branch needs API that has not landed upstream yet: the VmConfig CPU placement fields, the CPU topology and kernel isolation in the hardware inventory, ZInfoDevice.cpu_pools, ZInfoApp.cpu_placement_status and assigned_cpus, and the CPU placement capabilities. Without them nothing here compiles, so this commit points pillar and evetest at the fork carrying the proposed API (lf-edge/eve-api#155) through a replace directive, and vendors it. This commit exists only so the branch can be built, run and reviewed while the API is under discussion. It must be dropped and replaced by an ordinary `make bump-eve-api` once the API lands in lf-edge/eve-api: a replace directive pointing at a personal fork breaks dependency tracking and SBOM/licensing, and is never acceptable on master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

rucoder · 2026-10-14T15:49:53Z

@eriknordmark could you re-review ? I think I addressed all the issues

eriknordmark · 2026-10-14T19:52:22Z

 | `hv_type_kubevirt` | Device runs the Kubevirt hypervisor | required before sending `EdgeNodeCluster`; the KVM flavor cannot join a cluster |
 | `hw_inventory_support` | Device can produce `HardwareInventory` | distinguishes "found no hardware" from "cannot report" when `ZInfoHardware.inventory` is empty |
 | `etcd_snapshot` | Device supports etcd snapshots | EVE-k cluster operations |
+| `cpu_placement` | Device applies `VmConfig.cpu_policy`, `.pin_cpu` as the pinning choice, `.full_pcpus_only`, `.threads_per_core`, `.numa_policy`, `.io_placement`, each served or the workload refused, and reports `ZInfoApp.cpu_placement_status`, `.assigned_cpus` and `ZInfoDevice.cpu_pools`. Set only on KVM with a readable CPU topology | required before sending any of those fields: Xen, EVE-k or a node whose topology EVE cannot read does not place CPUs; only an EVE at `CPU_PLACEMENT_POLICY` sets it |


Xen is gone so don't list it here. What about ARM images?

eriknordmark · 2026-10-14T19:53:23Z

 | `REPORT_TPM_EVENTLOG` = 21 | rpt | `ZAttestQuote.tpm_binary_event_log`, superseding the deprecated `ZAttestQuote.event_log` |
 | `APP_INSTANCE_NET_INTERFACE_CHANGE` = 22 | cfg | adding or removing an `AppInstanceConfig.interfaces` entry needs only `restart`, not `purge`, so the app keeps its volumes; the `AppInstanceConfig` comment still describes the older purge requirement **(?)** |
 | `DEFERRED_QUEUE_METRICS` = 23 | rpt | `deviceMetric.deferred_queue`, and `urlcloudMetric.retriableErrCount`, `.rejectedErrCount`, `.deliveredMsgCount` |
+| `CPU_PLACEMENT_POLICY` = 24 | cfg | `VmConfig.cpu_policy`, `.pin_cpu` as the pinning choice, `.full_pcpus_only`, `.threads_per_core`, `.numa_policy`, `.io_placement` are known, legacy `pin_cpu` follows the same rules, and outcomes are reported in `ZInfoApp.cpu_placement_status` and `.assigned_cpus`. Every build reports it; whether a node places CPUs is `OptionalCapabilities.cpu_placement` |


No need for this given cpu_placement optional below.

eriknordmark · 2026-10-14T19:57:00Z

+// stays until the config or the node changes, as for an unavailable
+// adapter. A value keeps its meaning for good; a retired one is reserved,
+// never reused.
+enum CPUPlacementStatus {


Odd to have a combination of status codes (non, placed, pending, repack_advised) and error enums.

Better to split in a Status enum (with a failed value) and an Error enum (set if failed).

eriknordmark · 2026-10-14T20:12:15Z

+  //  1. "ZInfoDevice.optional_capabilities.cpu_placement" is set: the node
+  //     places CPUs. Only an EVE at "API_CAPABILITY_CPU_PLACEMENT_POLICY" or
+  //     later sets it; an older one ignores these fields;
+  //  2. for CPU_POLICY_DEDICATED, CPU_POLICY_ISOLATED or pin_cpu=true, the


Condition 2 names pin_cpu=true, so together with condition 1 this reads as "a controller MUST NOT send pin_cpu=true unless optional_capabilities.cpu_placement is set". That would rule out pin_cpu on every EVE predating this API and on EVE-k, which never sets cpu_placement but honors pin_cpu today (kubevirt maps it to DedicatedCPUPlacement).

I assume the MUST NOT is meant only for fields 28-32. Could you say so explicitly, and state what a controller may send in pin_cpu to a node without cpu_placement?

Topology-aware CPU placement for VM and container workloads. The controller picks the pool a workload's CPUs come from and whether its vCPUs are pinned; the device reports the topology, isolation and pool state the controller needs to decide what it may ask for. Every field has one meaning whatever the others say, and a request the device does not serve is refused, never reinterpreted. A controller learns two things from two places: what the node has (the hardware inventory) and whether the node applies the fields (OptionalCapabilities.cpu_placement). The inventory is never a gate; it sizes requests. - config/vm.proto: VmConfig.cpu_policy picks the pool: HOUSEKEEPING (the default pool for workloads as well as EVE's processes, holding the CPUs EVE reserves for its services), DEDICATED (CPUs of the workload's own) or ISOLATED (CPUs the kernel isolates, isolcpus). pin_cpu is the pinning choice on any pool: each vCPU on a host CPU of its own. full_pcpus_only, threads_per_core, numa_policy and io_placement refine a placement on the workload's own CPUs. With cpu_policy unset, pin_cpu=true means a dedicated workload, pinned on a VM, as for a controller that does not send cpu_policy. cpus is deprecated. The section comment lists what a controller checks before sending the new fields, and what pin_cpu means on a node that does not place CPUs. - info/hardware.proto: each logical CPU reports its socket, core, NUMA node, L2 and L3 cache domains, core class and base and maximum frequency, and CPUInfo lists the cache domains with the CPUs sharing each. The coordinates are optional: a CPU without socket_id and core_id means the topology could not be read. KernelCPUIsolation reports the CPUs the running kernel isolates (isolcpus) and runs tickless (nohz_full). A PCI device reports its local CPUs, whose NUMA node is the device's. - info/info.proto: ZInfoDevice.cpu_pools reports each pool's CPUs, free CPUs and thread and whole-core counts, so a controller can tell whether a workload fits before deploying it and explain a refusal after. A pool also carries warnings about how the node defines it (CPUPoolWarning: a CPUPoolIssue and its ErrorInfo), so far a mistake in isolcpus: covering the CPUs reserved for EVE, or isolating only part of a core. ZInfoApp.cpu_placement_status (CPUPlacementStatus) says where a workload stands with its CPU request: placed, waiting for CPUs a repack frees, part of a repack, or failed; for a failure ZInfoApp.cpu_placement_error (CPUPlacementError) says why. appErr keeps the text and retry_condition. The enums are targeted rather than a code in ErrorInfo, which is reported for many things besides apps; a value keeps its meaning for good, and a retired one is reserved. ZInfoApp.assigned_cpus reports the CPUs a workload of the dedicated or isolated pool holds, as assignedAdapters reports its PCI devices: its vCPUs' host CPUs in vCPU order, the idle siblings it parks and the shared CPUs its IO threads run on. - Capabilities: OptionalCapabilities.cpu_placement says this node places CPUs (KVM on any architecture, a CPU topology EVE could read); a controller sends the new fields only where it is set. An EVE that predates them does not set it, so no APICapability value is needed. CAPABILITIES.md lists it. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

eriknordmark

LGTM

This EVE API update brings in the CPU placement fields, see lf-edge/eve-api#155: the CPU topology, cache domains and kernel CPU isolation in the hardware inventory, ZInfoDevice.cpu_pools, ZInfoApp.cpu_placement_status, cpu_placement_error and assigned_cpus, and OptionalCapabilities.cpu_placement. Adam stores device info re-encoded with protojson, which drops fields its eve-api does not know, so without this bump those fields are lost before evetest can read them. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

The controller re-serializes what a device sent using its own copy of the API, so fields newer than that copy never reach a test. 0.0.85 carries the eve-api with the CPU placement fields (lf-edge/eve-api#155, through lf-edge/adam#163), which the CPU placement tests assert on. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

lf-edge/eve-api#155 has merged: the CPU placement fields in VmConfig, the CPU topology, cache domains and kernel CPU isolation in the hardware inventory, the CPU pools on ZInfoDevice, the per-app placement status, error and assigned CPUs on ZInfoApp, and OptionalCapabilities.cpu_placement. Re-pin every subproject that imports eve-api/go with "make bump-eve-api", so the CPU placement code that follows builds against a real revision. That also picks up RuntimeStatus for runtime AppInstances, which landed since the previous pin. Nothing populates it yet, and the advertised ApiCapability is unchanged. pkg/pillar keeps the vendored pkg/kube/cnirpc that master carries: it is ahead of the version go.mod pins, and "go mod vendor" would revert it. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

rucoder mentioned this pull request Aug 17, 2026

go.mod: build against the CPU-placement eve-api lf-edge/adam#158

Closed

rucoder mentioned this pull request Aug 17, 2026

SMT/NUMA-aware CPU placement for pinned applications lf-edge/eve#6335

Open

rucoder requested a review from gkodali-zededa August 17, 2026 20:12

rucoder force-pushed the rucoder/cpu-placement-api branch from de1d6f3 to 5304c9d Compare October 7, 2026 18:25

rucoder changed the title ~~api: CPU placement policy, topology reporting and structured error codes~~ Add CPU placement policy, CPU topology and pool reporting Oct 7, 2026

rucoder force-pushed the rucoder/cpu-placement-api branch from 5304c9d to bcbb2e3 Compare October 7, 2026 18:37

rucoder marked this pull request as ready for review October 8, 2026 16:21

rucoder requested a review from eriknordmark as a code owner October 8, 2026 16:21

rucoder requested review from christoph-zededa, europaul and rene October 8, 2026 16:23

christoph-zededa reviewed Oct 8, 2026

View reviewed changes

Comment thread proto/info/hardware.proto Outdated

christoph-zededa reviewed Oct 8, 2026

View reviewed changes

Comment thread proto/info/common.proto Outdated

rucoder force-pushed the rucoder/cpu-placement-api branch from bcbb2e3 to 8dd8bac Compare October 8, 2026 17:03

rucoder requested a review from xyuria-zededa October 8, 2026 17:53

rucoder force-pushed the rucoder/cpu-placement-api branch 2 times, most recently from 5bf89ef to debcbb0 Compare October 9, 2026 22:20

rucoder marked this pull request as draft October 12, 2026 14:24

eriknordmark reviewed Oct 12, 2026

View reviewed changes

Comment thread proto/config/vm.proto

eriknordmark reviewed Oct 12, 2026

View reviewed changes

Comment thread proto/info/common.proto

rucoder force-pushed the rucoder/cpu-placement-api branch from debcbb0 to 224b97d Compare October 12, 2026 18:56

rucoder force-pushed the rucoder/cpu-placement-api branch from 224b97d to 8cdf5ac Compare October 14, 2026 08:37

rucoder requested review from christoph-zededa, eriknordmark and milan-zededa October 14, 2026 08:40

rucoder marked this pull request as ready for review October 14, 2026 08:40

rene reviewed Oct 14, 2026

View reviewed changes

xyuria-zededa approved these changes Oct 14, 2026

View reviewed changes

rucoder force-pushed the rucoder/cpu-placement-api branch from 8cdf5ac to 0f6f3c2 Compare October 14, 2026 10:37

rucoder requested a review from rene October 14, 2026 15:51

eriknordmark reviewed Oct 14, 2026

View reviewed changes

rucoder added 2 commits October 15, 2026 09:47

Generate golang and python code for CPU placement

c338d02

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

rucoder force-pushed the rucoder/cpu-placement-api branch from 0f6f3c2 to c338d02 Compare October 15, 2026 09:47

eriknordmark approved these changes Oct 15, 2026

View reviewed changes

eriknordmark merged commit e46e4c3 into lf-edge:main Oct 15, 2026
4 checks passed

rucoder mentioned this pull request Oct 15, 2026

eve-api bump: add CPU core pinning API lf-edge/adam#163

Merged

rucoder mentioned this pull request Oct 15, 2026

evetest: use adam 0.0.85 lf-edge/eve#6798

Merged

Conversation

rucoder commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Two pieces: what the node has, and whether it applies the fields

Config: what a workload asks for (config/vm.proto)

Info: what the node has, and what it did

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rucoder Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xyuria-zededa left a comment

Choose a reason for hiding this comment

Uh oh!

rucoder commented Oct 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eriknordmark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rucoder commented Aug 17, 2026 •

edited

Loading

Config: what a workload asks for (`config/vm.proto`)

rucoder Oct 12, 2026 •

edited

Loading