[Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support - #1171
[Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support#1171Looong01 wants to merge 44 commits into
Conversation
|
This is screenshot of Sabaki testing: And the binary release here: https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.16.4-openvino |
|
I partially referenced the code from #1164, and I am very grateful to @ChinChangYang |
@foxrainowo |
|
@seniorfish |
Looong01
changed the title
SummaryAdapts the Intel_NPU branch's custom ONNX Runtime backend to the v1.16.5→v1.17.1 upstream sync. The The rewrite follows the design of the not-yet-merged upstream PR #1222 ("Add ONNX Changes
Testing
|
|
Intel GPU test on Ultra 9 275HX: |
|
I'm having issues using Although |
|
@foxrainowo Thanks again for the detailed testing — your logs (plus a day of controlled experiments on my machine) let us root-cause the NPU crash, and the fixes are now in. Updated report: Why
|
|
@lightvector Could u pls check this? Update to v1.17.1 and NPU, GPU, Hybird mode fully tested. It's ready to merge. |
|
After updating the versions of onnxruntime and openvino to the latest versions, the NPU backend can run normally (without getting stuck), but the speed is extremely slow. In version 1.17.1, the performance of the GPU backend is excellent, with a speed of around 80v/s. I don't know what special optimizations are involved, but the result is really surprising. However, the performance of the NPU backend is very poor. For comparison, b28_NPU_1.16.4 runs at 70v/s, while b40_NPU_1.16.4 runs at 30v/s. Assuming b11 >≈ b40, the speed of b11_NPU_1.17.1 should be greater than 30v/s.
|
As I said, the new mode isn't suitable for NPU operation. As for the performance drop in the old model, I'll look into it and try to fix it next week (my Intel NPU machine has been borrowed). Thanks for your test report and effort! |
|
@Looong01 My previous test was conducted using the default configuration, whereas this time I adjusted On the NPU, for b11c768h12nbt3, I noticed two things: your project is slower than #1222 and your project is able to run b40c768nbt, whereas that one cannot. Would you prefer users to use your project or #1222? KataGo's developers merged with seniorfish's project, but it still has some bugs (The issue is here: #1238). So I'd prefer to use your project.
For reference:
|
…regression Syncs the branch to upstream v1.17.2 and fixes a v1.16.4 -> v1.17.1 performance regression on Intel NPU, which turned out to also be hiding a correctness bug. 1. scale8 outputs were decoded 8x too small (correctness). applyScale8ToReduceActivations() multiplies postProcessParams.outputScaleMultiplier by 8 to compensate for the 1/8-scaled graph, but maybeApplyScale8() ran in createComputeHandle, long after NNEvaluator had already snapshotted postProcessParams. The compensation was silently lost, so every convnet evaluation came back 8x too small (measured whiteLead 0.095 vs a true 0.756). This affected the default configuration. Snapshot postProcessParams after createComputeContext and apply scale8 there, matching upstream. 2. MISH_SCALE8 blocked Mish fusion (performance). The emission Softplus(8x) -> Tanh -> Mul(x, .) does not match the canonical Mish pattern Mul(u, Tanh(Softplus(u))), so OpenVINO could not fuse it and ran the chain unfused. Substituting u = 8x and using mish_scale8(x) = mish(8x)/8 keeps an exact canonical Mish subgraph for two extra scalar multiplies, so the FP16 headroom scale8 provides is kept at full speed. 3. transformerNHWC now defaults per device. NHWC is faster on the OpenVINO GPU plugin but much slower on the NPU plugin, so default to NCHW whenever any OpenVINO server thread may target an NPU. A bare AUTO device string counts as possibly-NPU, since OpenVINO picks at runtime. 4. onnxOpenVINODeviceType now auto-selects when unset. Probe NPU, then GPU, then CPU via AppendExecutionProvider_OpenVINO_V2, which validates device_type immediately, and use the first available one. A machine with only an iGPU now works with no configuration. ONNX Runtime's own device enumeration is deliberately not used: on a provider-bridge OpenVINO build it reports CPU only even on machines that do have an NPU and an iGPU. Measured on Intel NPU, single server thread (nnEvals/s), before -> after: kata1-b28c512nbt 10.56 -> 39.27 (v1.16.4 reference: 26.84) kata1-zhizi-b40c768nbt 5.23 -> 16.08 b11c768h12nbt3tflrs 3.69 -> 10.12 b10c384h6nbttflrs 12.93 -> 32.79 b10c512h8nbt3tflrs 6.58 -> 16.48 OpenVINO GPU (8 threads) b11c768h12nbt3tflrs: 14.25 -> 16.94. Verified that scale8-applied and scale8-skipped now produce identical outputs in CPU fp32, as they must since the transform is mathematically exact. Verified the device fallback by removing openvino_intel_npu_plugin.dll to simulate a machine without an NPU: it auto-selects GPU and switches to NHWC. Also documents all device/layout selection cases in README.md and the example configs.
Fixed. |
Update: synced to v1.17.2, and fixed the NPU performance regression reported aboveThanks @foxrainowo for the detailed benchmark tables — they pointed straight at the 1. scale8 was decoding outputs 8x too small (correctness bug)
Symptom: This is the same bug upstream fixed in the merged ONNX backend; I've ported that fix 2.
|
| Model | before | after | v1.16.4 |
|---|---|---|---|
| kata1-b28c512nbt | 10.56 | 39.27 | 26.84 |
| kata1-zhizi-b40c768nbt | 5.23 | 16.08 | — |
| b11c768h12nbt3tflrs | 3.69 | 10.12 | — |
| b10c384h6nbttflrs | 12.93 | 32.79 | — |
| b10c512h8nbt3tflrs | 6.58 | 16.48 | — |
OpenVINO GPU (b11c768h12nbt3tflrs, 8 threads): 14.25 → 16.94.
All of these are now at or above v1.16.4, with the defaults, without tuning onnxSkipScale8 or
onnxTransformerNHWC by hand.
Verification
- scale8-applied and scale8-skipped now produce identical outputs in CPU fp32, as they must
since the transform is mathematically exact (before:0.522vs0.669winrate). - Device fallback verified by removing
openvino_intel_npu_plugin.dllto simulate a machine with
no NPU: it auto-selects GPU and switches to NHWC. - README.md and the example configs now document every device/layout selection case.
|
Thanks for these fixes. I previously reported the crash issue at #1238, and I asked AI about the reason. It said that the net's batch is dynamic, but the NPU needs it to remain constant during compilation. KataGo just released the new version 1.18.0. It added a new onnxPadBatch function, which says: "Pad every neural net evaluation up to the max batch size. Some providers (DirectML, and OpenVINO on an NPU) recompile whenever the batch size changes, which makes search extremely slow without padding. Auto (the default) enables it exactly for those." I have tried this new version, and it runs normally, but the speed is very slow. I don't know why your project does not encounter this crash issue and is able to run normally. What I mean is that your project is fast, accurate, and compatible. Could you take a look and see if there are any other areas that could be improved, and then compile the final version for me to test? |
Upstream merged its own ONNX backend (PR lightvector#1222) and released v1.18.0 after this branch forked, so every ONNX file collided. Resolved by taking upstream wholesale - including cpp/neuralnet/onnxbackend.cpp, which was an add/add conflict between two independent implementations - and re-applying only the Intel NPU fixes upstream does not have. Everything not specific to the Intel NPU is byte-identical to upstream; the diff is four files. Upstream already carries the scale8 postProcessParams ordering fix, so that one is not re-applied here. 1. MISH_SCALE8 blocks OpenVINO's Mish fusion. Upstream emits Mul(x, Tanh(Softplus(Mul(x,8)))). Fusion passes match the canonical Mul(u, Tanh(Softplus(u))), and here the outer Mul takes x while Softplus takes 8x, so the pattern does not match and OpenVINO runs Softplus, Tanh and Mul unfused over the whole trunk. Emit mish_scale8(x) = mish(8x)/8 instead, with u = 8x, which contains an exact canonical Mish subgraph and costs two extra scalar multiplies. Same function, numerically identical. kata1-b28c512nbt on an Intel NPU at 8 search threads, same avgBatchSize: 5.85 -> 28.17 nnEvals/s. 2. onnxTransformerNHWC defaults per device. The two OpenVINO plugins prefer opposite trunk layouts, so upstream's fixed default of true cannot be right for both. On b11c768h12nbt3tflrs at 8 search threads the NPU runs NCHW about 1.75x faster than NHWC, while the GPU runs NHWC about 1.5x faster than NCHW. Default to NCHW as soon as any OpenVINO server thread might target an NPU. A bare AUTO counts as might-be-NPU: OpenVINO resolves it at runtime and does not report the choice back. 3. onnxOpenVINODeviceType auto-selects when unset. Probe NPU then GPU via AppendExecutionProvider_OpenVINO_V2, which validates device_type immediately and needs no model or session, and use the first one present. CPU is not a candidate since upstream rejects CPU-only OpenVINO device strings. An explicit setting is always honored as-is, including when the device does not exist. ONNX Runtime's own enumeration (Ort::Env::GetEpDevices) is deliberately not used: on a provider-bridge OpenVINO build it reports CPU only, even on a machine that has both an NPU and an iGPU. Also updates the onnxOpenVINODeviceType, onnxTransformerNHWC and onnxSkipScale8 documentation in gtp_example.cfg and the OpenVINO section of Compiling.md. Dropped in favor of upstream's equivalents: the exportonnx command (upstream has dumponnx), the .onnx suffixes in loadmodel.cpp (upstream handles .onnx in desc.cpp), the ONNX prompts in genconfig, runonnxtests.sh (upstream has runonnxmodelfiletests), and the per-thread nnMaxBatchSize plumbing in the test helpers. The fork-only OpenVINO keys (onnxOpenVINOLoadConfig, onnxOpenVINONPUExactBoard, onnxOpenVINODeviceId, onnxOpenVINOEnableNPUFastCompile, onnxOpenVINOModelPriority, onnxOpenVINONumOfThreads, onnxModelVersion, onnxInput*/onnxOutput*) are gone too: upstream deliberately slimmed the OpenVINO option set, and its .onnx file support supersedes the raw-loading keys. Measured on an Intel NPU, onnxOpenVINOLoadConfig made no difference and onnxOpenVINONPUExactBoard was worth about 1.2x. Verified on an Intel NPU machine: auto-select logs "auto-selected 'NPU'"; a transformer model on it builds with transformerNHWC=false while an explicit GPU device builds with transformerNHWC=true; and dumponnx output contains the fusable mish8/inv8 subgraph.
Rebased onto v1.18.0 — three Intel NPU fixes remainNow that the ONNX backend is merged upstream, I've taken upstream's version of Measurements below are from an Intel Core Ultra NPU and its iGPU, OpenVINO 2026.3 1.
|
| nnEvals/s | |
|---|---|
| master's emission | 5.85 / 5.81 / 5.75 |
| this PR | 28.17 / 27.96 / 27.49 |
On the GPU it is a correctness bug rather than a slowdown. With the default
onnxSkipScale8 = false, master's emission produces non-finite outputs and
KataGo dies with Got nonfinite for policy sum; GPU utilization then drops to
zero and the process hangs instead of exiting.
| emission | onnxSkipScale8 |
GPU result |
|---|---|---|
| master | false (default) | non-finite, hang (2/2) |
| master | true | 25.60 nnEvals/s |
| this PR | false (default) | 24.24 nnEvals/s |
| this PR | true | 25.63 nnEvals/s |
Turning scale8 off puts both binaries on the same plain-Mish path and both work,
so the failure needs scale8 applied and the non-canonical emission. My guess is
that the unfused log(1+exp(8x)) overflows the FP16 the plugin infers in, while
the fused Mish evaluates it stably — but the isolation above holds regardless of
the mechanism.
The last two rows also show scale8 costs only ~5% on the GPU once the emission is
fusable, so the gtp_example.cfg comment describing it as a ~2x/~4x tradeoff no
longer applies; I've updated that text.
Worth flagging separately from this PR: a non-finite NN output kills the async bot
thread but leaves the process hanging rather than exiting with an error.
2. onnxTransformerNHWC cannot have one right default
The two OpenVINO plugins prefer opposite trunk layouts, so the fixed default of
true is wrong on one of them either way. b11c768h12nbt3tflrs, two runs each:
| device | NCHW | NHWC |
|---|---|---|
| NPU | 5.44 / 5.49 | 3.17 / 3.12 |
| GPU | 11.61 / 11.57 | 16.21 / 18.33 |
So the NPU prefers NCHW by ~1.75x and the GPU prefers NHWC by ~1.5x. This PR
picks per device when the key is unset, defaulting to NCHW as soon as any
OpenVINO server thread might target an NPU. A bare AUTO counts as might-be-NPU:
OpenVINO resolves it at runtime and does not report the choice back through ORT's
API, and guessing NCHW costs a GPU ~1.5x when wrong but saves an NPU ~1.75x when
right.
3. onnxOpenVINODeviceType auto-selects when unset
Probes NPU, then GPU, and uses the first one present, so a machine with only an
iGPU and a machine with an NPU both work with no device configuration. The probe
calls AppendExecutionProvider_OpenVINO_V2, which validates device_type
immediately and throws [OpenVINO] Device X is not available, so it needs no
model and no session. CPU is not a candidate, since the provider already rejects
CPU-only device strings. An explicit setting is always honored as-is, including
when the device does not exist — failing with OpenVINO's own error beats quietly
running somewhere else.
Worth recording for anyone attempting the same: ONNX Runtime's own device
enumeration (Ort::Env::GetEpDevices, OrtHardwareDeviceType) is not usable
here. On a provider-bridge OpenVINO build it reports the CPU only, even on a
machine that does have both an NPU and an iGPU. I implemented it that way first
and it would have silently steered exactly the setups this is meant to help onto
the slowest device present.
Not part of the upstream-facing change
The branch also carries -DUSE_ONNX_EP=OPENVINO, which only defaults
ONNXRUNTIME_ROOT to an OpenVINO-enabled ONNX Runtime unpacked under
cpp/external. It is a local build convenience for this fork and changes nothing
at runtime; happy to drop it from the PR if you'd rather not carry it.
Binary release here: https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.18.0-openvino |
Rebased onto v1.18.0 — three Intel NPU fixes remainNow that the ONNX backend is merged upstream, I've taken upstream's version of How the numbers below were takenIntel Core Ultra NPU and its iGPU, OpenVINO 2026.3 / ONNX Runtime 1.28. Arms were 1.
|
| emission | onnxSkipScale8 |
result |
|---|---|---|
| master | false (default) | non-finite, hang (2/2) |
| master | true | 25.60 nnEvals/s |
| this PR | false (default) | 24.24 nnEvals/s |
| this PR | true | 25.63 nnEvals/s |
Turning scale8 off puts both binaries on the same plain-Mish path and both work,
so the failure needs scale8 applied and the non-canonical emission. My guess is
that the unfused log(1+exp(8x)) overflows the FP16 the plugin infers in, while
the fused Mish evaluates it stably — but the isolation above holds regardless of
the mechanism.
The last two rows also show scale8 costs only ~5% on the GPU once the emission is
fusable, so the gtp_example.cfg comment describing it as a ~2x/~4x tradeoff no
longer applies; I've updated that text. onnxSkipScale8 = true is no longer
needed as a workaround for anything.
Worth flagging separately from this PR: a non-finite NN output kills the async bot
thread but leaves the process hanging rather than exiting with an error.
2. onnxTransformerNHWC cannot have one right default
The two OpenVINO plugins prefer opposite trunk layouts, so the fixed default of
true is wrong on one of them either way. b11c768h12nbt3tflrs, two runs each:
| device | NCHW | NHWC |
|---|---|---|
| NPU | 5.44 / 5.49 | 3.17 / 3.12 |
| GPU | 11.61 / 11.57 | 16.21 / 18.33 |
The NPU prefers NCHW by ~1.75x and the GPU prefers NHWC by ~1.5x. This PR picks
per device when the key is unset, defaulting to NCHW as soon as any OpenVINO
server thread might target an NPU. A bare AUTO counts as might-be-NPU: OpenVINO
resolves it at runtime and does not report the choice back through ORT's API, and
guessing NCHW costs a GPU ~1.5x when wrong but saves an NPU ~1.75x when right.
3. onnxOpenVINODeviceType auto-selects when unset
Probes NPU, then GPU, and uses the first one present, so a machine with only an
iGPU and a machine with an NPU both work with no device configuration. The probe
calls AppendExecutionProvider_OpenVINO_V2, which validates device_type
immediately and throws [OpenVINO] Device X is not available, so it needs no
model and no session. CPU is not a candidate, since the provider already rejects
CPU-only device strings. An explicit setting is always honored as-is, including
when the device does not exist — failing with OpenVINO's own error beats quietly
running somewhere else.
Worth recording for anyone attempting the same: ONNX Runtime's own device
enumeration (Ort::Env::GetEpDevices, OrtHardwareDeviceType) is not usable
here. On a provider-bridge OpenVINO build it reports the CPU only, even on a
machine that does have both an NPU and an iGPU. I implemented it that way first
and it would have silently steered exactly the setups this is meant to help onto
the slowest device present.
Results: five models on the Intel NPU
Two binaries differing only in onnxbackend.cpp and onnxmodelbuilder.cpp, same
runtime, -t 4, -v 80, alternating, two runs each. nnEvals/s;
avgBatchSize was 1.94–1.95 in all twenty runs, and the two runs of each arm
differ by under 1%.
| model | kind | scale8 | master | this PR | |
|---|---|---|---|---|---|
| kata1-b28c512nbt | convnet | applied | 5.99 / 5.98 | 27.60 / 28.45 | 4.68x |
| kata1-zhizi-b40c768nbt | convnet | applied | 2.70 / 2.68 | 10.50 / 10.47 | 3.90x |
| b11c768h12nbt3tflrs | transformer | n/a | 3.41 / 3.43 | 5.90 / 5.90 | 1.73x |
| b10c384h6nbttflrs | transformer | n/a | 11.36 / 11.45 | 20.39 / 20.37 | 1.79x |
| b10c512h8nbt3tflrs | transformer | n/a | 5.70 / 5.71 | 10.07 / 10.14 | 1.77x |
The split falls out exactly where the two fixes apply, which is a useful check on
the whole story: the convnets take the scale8 path and are carried by fix 1, at
3.9–4.7x. The transformers never get scale8 (their internal RMSNorm would undo the
scaling), so fix 1 cannot touch them and they are carried by fix 2 alone — landing
at 1.73x / 1.79x / 1.77x, which independently reproduces the ~1.75x measured
directly for the layout in section 2.
Thread scaling
The two are not fastest at the same thread count, so single-thread-count
comparisons are misleading. kata1-b28c512nbt:
| threads | this PR |
|---|---|
| 1 | 14.49 / 14.75 |
| 4 | 27.60 / 28.45 |
| 8 | 27.49 – 28.17 |
b11c768h12nbt3tflrs on this PR peaks at the same place: 3.03 (t=1), 3.06 (t=2),
5.78 (t=4), 5.61 (t=8), 5.14 (t=16). At t=32 the NPU hung during graph
compilation and had to be killed.
What this PR does not fix
I also compared against the older v1.16.4 ONNX build I had packaged, on
kata1-b28c512nbt. It is still faster than master-plus-this-PR:
| build | t=1 | t=2 | t=4 | t=8 |
|---|---|---|---|---|
| v1.16.4 (ORT 1.24.4 / OV 2026.1) | 30.75 / 30.93 | 30.62 | 22.11 | 9.25 |
| v1.18.0 + this PR (ORT 1.28 / OV 2026.3) | 14.49 / 14.75 | — | 27.60 / 28.45 | 27.49–28.17 |
| v1.18.0 master | 3.10 / 3.09 | — | 5.99 / 5.98 | — |
Peak to peak that is 30.93 vs 28.45, so this PR gets to about 92% of the old
build, and at t=1 it is only about half. I have not investigated where the rest
goes, and I would not assume it is KataGo's code: two ONNX Runtime versions, two
OpenVINO versions and the whole backend rewrite sit between those two rows. I
mention it so the numbers above are not read as "back to the old speed" — they are
not. What they show is master compared against master with three fixes, on one
machine, with everything else held constant.
Not part of the upstream-facing change
The branch also carries -DUSE_ONNX_EP=OPENVINO, which only defaults
ONNXRUNTIME_ROOT to an OpenVINO-enabled ONNX Runtime unpacked under
cpp/external. It is a local build convenience for this fork and changes nothing
at runtime; happy to drop it from the PR if you'd rather not carry it.
On the older v1.16.4 ONNX buildI had an older v1.16.4 ONNX/OpenVINO package around and checked this branch What made it look otherwise is
The crossover is between 2 and 4 threads. Compare at one fixed low thread count and I am not proposing a change to For completeness, these were each ruled out by a controlled experiment before I got |
6 participants
Summary
This PR adds and hardens the Windows & Linux Intel NPU path for KataGo using the ONNX backend with ONNX Runtime + OpenVINO Execution Provider, and updates docs/config guidance for an end-to-end workflow.
It also improves failure behavior for non-ONNX builds and simplifies Windows & Linux dependency handling.
What Changed
1) ONNX backend and OpenVINO provider support
onnxProvider(cpu,openvino,cuda,tensorrt,migraphx,coreml).onnxOpenVINODeviceTypeonnxOpenVINODeviceIdonnxOpenVINOCacheDironnxOpenVINOEnableNPUFastCompile(best-effort; depends on ORT build support).onnxmodels directly.bin/.bin.gzmodels via internal conversion to ONNX graph2)
exportonnxcommand behaviorexportonnxis available in ONNX builds and exports fixed-size ONNX models.-x/-ycan override).exportonnxnow returns a clear error instead of failing ambiguously.3) Config safety for non-ONNX binaries
onnx*config keys now fails fast with a clear message.4) CMake dependency flow
ONNXRUNTIME_ROOT(defaulting tocpp/external/onnxruntime-win-x64-openvinoandcpp/external/onnxruntime-linux-x64-openvino).zlib,onnx,protobuf) through vcpkg when enabled.5) Documentation updates
Compiling.md:use_openvino=NPU)cpp/external/onnxruntime-win-x64-openvino.README.md:exportonnx(default 19x19)benchmarkgtpBehavior Notes
onnxDeviceToUseThread*) is mainly intended for ONNX providers like CUDA/TensorRT/MIGraphX.Validation
exportonnxworks from.bin/.bin.gz->.onnx.benchmark/gtprun withonnxProvider=openvinoandonnxOpenVINODeviceType=NPU.