These notes describe the hardware blocks and the register-command interface of the Rockchip
NPU, the RKNPU. They describe it as programmed through the mainline Linux
rocket DRM accel driver. They are for developers who program the NPU directly, through rocket or another raw
register-command path. Unless stated otherwise, the notes describe the RK3588.
The documents in rockchip-npu-notes were authored by AI, primarily Claude. They were produced as part of a series of side projects, and are published here in case they are of use to others. Accuracy of information is not guaranteed.
Most of this content applies to every Rockchip SoC that integrates the RKNPU, an NVDLA-derived design. Per-SoC differences (machine parameters, the register offset map, and SoC integration) are documented in chips/.
| SoC | NPU cores | Status | Sheet |
|---|---|---|---|
| RK3588 | 3 | Supported. The reference target | chips/rk3588.md |
| RK3576 | 2 | Supported for the operations chips/README.md lists, on rocket 1.6.0 |
chips/rk3576.md, chips/rk3576-regcmd.md |
| RK3566 | 1 | Not characterized | chips/rk3566.md |
chips/porting-patterns.md lists which RK3588 facts transfer to another SoC, and chips/rk3576.md has the RK3576's own.
The RK3588 NPU is a three-core accelerator derived from NVIDIA's NVDLA. Each core runs a fixed
pipeline of blocks, CNA→CORE→DPU, and has its own
384 KB convolution buffer, the CBUF [source-confirmed]. Each block has its own register range [source-confirmed: Mesa
registers.xml]:
| Block | Registers | Function | NVDLA equivalent |
|---|---|---|---|
| PC | 0x0xxx |
Fetches a task's register program from memory and starts the blocks it names | None |
| CNA | 0x1xxx |
Loads the input feature and the weights into the CBUF, and feeds the MAC array | CDMA, CSC |
| CORE | 0x3xxx |
The MAC array and its accumulators | CMAC, CACC |
| DPU | 0x4xxx |
Post-processing: the BS, BN and EW stages, the LUT, the output converter, and the write DMA | SDP |
| DPU_RDMA | 0x5xxx |
The DPU's read DMA. Its channels read the BS, BN and EW stages' operands, and MRDMA can read the DPU's main input from memory in place of the CORE's output | SDP_RDMA |
| PPU | 0x6xxx |
Pooling, with its read DMA at 0x7xxx |
PDP |
The NPU has no matmul primitive. A matmul runs as a 1×1 convolution, on the same program Mesa emits for a convolution. hardware-overview.md describes the blocks. nvdla-lineage.md maps them to the NVDLA documentation.
rocket is a generic register-command submitter. Userspace builds each operation as a register
program, and the kernel runs whatever CNA→CORE→DPU program it is handed. The uAPI has four
ioctls:
| Ioctl | Function |
|---|---|
CREATE_BO |
Allocates a buffer object (BO) and maps it into the file descriptor's IOVA window |
SUBMIT |
Queues one or more jobs, each of one or more tasks, and returns without waiting for them |
PREP_BO |
Begins CPU access to a BO. On an output BO, it waits for the job that writes it |
FINI_BO |
Ends CPU access to a BO |
A task is one register program: a sequence of regcmds, each writing one value to one register. The PC block fetches and applies them, then starts the blocks. A submit, a task and a kick (one hardware start of the PC) are three separate units, and encodings/regcmd-task-model.md describes how they relate.
A matmul computes C[M×N] = A[M×K] × B[K×N], with A the activation, B the weight and C the
output. Run as a 1×1 convolution, M is the spatial height, K the input channels and N the
output channels (matmul-as-conv.md). The host scatters B into the
cube layout the CNA reads (encodings/tile-layouts.md).
The facts below apply to the RK3588. Each label links to the note with the detail and the evidence:
- Datatypes. The integer types are int4, int8 and int16, and the float types are fp16, bf16 and tf32. Each has a native matmul [HW sweep]. int16's int32 output saturates, so a matmul that needs int64-exact results over full-range int16 operands is computed as four int8 matmuls.
- Layouts. No register converts a layout, so software must scatter the weights into the cube layout on the host [source-confirmed]. A matmul's activation and output need no packing. With the strides programmed for it, the CNA reads A as plain rows, and the DPU writes C as plain rows [HW sweep]. The device time of that form grows with the output's row pitch once a row passes 4 KiB.
- K-accumulation. The DPU's EW stage performs fp16 K-accumulation on chip, so the host reads each output tile once [HW sweep, source-confirmed]. The EW ALU also adds int8, int16 and int32 exactly when every precision field is set to one integer code [HW sweep]. With a convolution as its main feed, it reads its operand as 16-bit elements. No tested configuration accumulates an int32 partial on chip [HW sweep].
- On-chip requant. The DPU's output converter performs the requant of an int32 accumulator, with an integer multiplier and a shift. Its BS stage multiplies per output channel, which applies TFLite's per-axis scales on chip [HW sweep]. An int8 convolution therefore writes int8 with no host requant.
- Cores and address space. Under
rocket, each open file descriptor is one scheduling entity with its own 4 GB IOVA window [HW sweep]. The jobs of one file descriptor run one at a time, in order. The scheduler moves a file descriptor to another core only when none of its jobs is queued or running. A lone file descriptor that waits on each job therefore runs only on cores 0 and 1, alternately [HW sweep, source-confirmed: the DRM scheduler]. To use all three cores, software must open three or more file descriptors. - Operations beyond matmul. The DPU's LUT computes activations such as SiLU and GELU, and functions such as exp and rsqrt [HW sweep]. The PPU runs max and average pooling. Composed with the matmul, they run a full Whisper encoder block on the NPU (encodings/whisper-encoder.md). The CNA also has a hardware deconvolution mode that computes a ConvTranspose with a power-of-two stride in one task.
Each of these produces a wrong result or a stalled job, and no error names the cause. The issue column links to the note with the mechanism and the evidence.
| ID | Issue | Symptom | Workaround |
|---|---|---|---|
| E1 | The MRDMA block: a plain convolution or matmul with mrdma_disable clear in DPU_RDMA_FEATURE_MODE_CFG (0x5044) |
The DPU read DMA waits for data that never arrives. The job times out with its output untouched [HW sweep, source-confirmed] | Software must set mrdma_disable, and include the DPU_RDMA bit in PC_OPERATION_ENABLE |
| E2 | Output stride: the output of an int8 or int4 matmul (int32 and int16) programmed with its natural size_e, with N spanning more than one surface |
Most of the output is not written [HW sweep] | Software must program size_e=7 for the int32 output of an int8 matmul and the int16 output of an int4 matmul. An int16 matmul's int32 output takes the natural size_e=3, and hangs at 7 |
| E3 | Feature surface stride: a feature cube whose height is not a multiple of 4, with the surface stride in CNA_DMA_CON2 computed as line_stride × (H/4 − 1) in integer arithmetic |
Incorrect output. A matmul programmed this way is incorrect at every M that is not a multiple of 4 [HW sweep] | Software must program the surface stride as width × (H − 4), a 28-bit two's complement below four rows |
| E4 | CBUF bank slack: the int8 feature DMA at some tile geometries | The DMA over-reads by one CBUF bank and returns garbage in the tile's last rows [HW sweep]. The fp16 cube is not affected | Software must reserve data_bank = fd_banks+1 |
| E5 | LUT table entries: a shifted LUT table entry of exactly q=0 |
The entry decodes to ~4.0 rather than 0 [HW sweep] | Software must clamp every shifted-table entry to q>=1 |
| E6 | Requant scale and shift sign: the output converter's scale and the BS stage's shift | Bit 15 of the scale is a sign, and Mesa's derivation of the scale drops a carry, which halves or negates one scale in 16384 [HW sweep]. The BS stage's shift is split by sign across two fields, and one field alone scales negative products wrongly | Software must keep the carry when it derives the scale, and must program both shift fields |
| E7 | Per-core float difference: an fp16 matmul with fp32 output, compared across cores | The three cores compute it identically except on output lane 0 of each 16-channel group, where each core rarely writes a result slightly low, on its own set of inputs: 2^-16 low for results from about 2^-4 to 2 on the data characterized [HW sweep]. rocket places a job on a core by load, so two runs of one fp16 job can differ on lane 0. The integer paths are core-invariant |
None |
| E8 | Convolution tile limits: a convolution task that reads more than 1022 input rows or 2047 input columns, or that programs a pad past 15 | The extent or the pad wraps its field. The task writes a complete output of the wrapped extent, or at the wrong offset, with incorrect values [HW sweep] | Software must keep each task within 1022 input rows by 2047 columns and a pad of 15, and must write a larger pad into the task's input |
| E9 | Cube-dimension limit: a LUT chunk wider than 8176 positions, and any other DPU cube dimension of 8191 | The LUT chunk computes every position from 8176 on incorrectly and raises no completion [HW sweep 2026-11-02]. SHARD reports a REGTASK Overflow for any operand index past 8191 [source-confirmed] |
Software must keep a LUT chunk at or below 8176 positions and every other DPU cube dimension below 8191. The PPU's height and channel axes are exact at 8192 [HW sweep 2026-11-02]. The CNA's input width and height are 11 bits, narrower still |
| E10 | NPU clock: setting the NPU clock rate while the power domain is off | The SCMI firmware hangs [HW sweep] | Software must raise the rate from inside the driver, after runtime PM powers the domain |
| E11 | Streamed weight banks: a matmul task that streams its weights through fewer weight banks than one whole 32-kernel group needs, or through one bank. An int8 task also at exactly that minimum | The task never completes. The watchdog retires it, the fence signals as if it had completed, and the output is not written [HW sweep] | Software must program at least two weight banks, enough banks for one whole 32-kernel group, and for int8 one bank more |
| E12 | K-accumulation surface stride: the EW operand's EW_SURF_STRIDE and SURF_NOTCH floored at 12, MAX(M, 12), with M below 12 and N above 16 |
The sum reads the wrong partial surfaces from the third onward, and the output is complete and plausible [HW sweep] | Software must program both fields as the partial's own surface stride, M |
| E13 | Job timeouts: a job that times out with its DMA still active, and repeated job timeouts on one core | The core's IOMMU cannot stall, and later jobs on the core intermittently never complete. Then an NPU power domain fails to idle, the next resume takes an external abort, and the core stays resuming until a power cycle [HW sweep] |
Software must not submit a program that cannot raise its completion, and must stop submitting after an IOMMU stall timeout on an NPU core |
Conditions: RK3588 on mainline rocket. Each item states its NPU clock, and the linked note
gives its full conditions.
- Matmul throughput. DMA and dispatch bound the matmul, rather than
the MAC array. At
512×3840×4096, with resident weights on three cores at 928 MHz, fp16 measures 947-966 GOP/s. int8 with streamed weights and int4 run 1.14-1.20x that [HW sweep 2026-11-02]. Their MAC rates are 2x and 4x fp16's, so quantization reduces memory use and raises the matmul's throughput far less than its MAC rate. In a model, resident int8 prefill measured slower than fp16. - Weight streaming. CBUF weight streaming, the fp16 default, removes most of the whole-tile program's tasks. At 928 MHz it gives 1.07-1.17x on prefill over the whole-tile program, on four GGUFs [HW sweep 2026-10-21].
- Decode. Single-row decode is a GEMV, bound by the DDR bandwidth the NPU shares with the CPU, so decode stays on the CPU. At 928 MHz the NPU streams resident weights at ~16 GB/s over three cores, which only matches the CPU's decode stream [HW sweep 2026-10-21]. The NPU's fp16 throughput at M=1 is ~82x below its batched-GEMM peak, as an external benchmark measured it through the vendor RKNN matmul API.
- Clock. The NPU clock boots at 198 MHz, one fifth of its 1 GHz maximum [source-confirmed]. Above that it is a PVTPLL whose frequency follows the rail voltage, so a requested rate is not the clock. At a 0.80 V rail the 900 MHz request runs at 928 MHz, the vendor's 900 MHz pairing. It gives 1.66x on Gemma-4-12B prefill over the boot clock [HW sweep 2026-11-02]. On matmul device time 928 MHz gives 1.044x over the 600 MHz request (~816 MHz), and 1019 MHz, a 1 GHz request at 0.85 V, 1.083x [HW sweep]. Known issue E10 is the hazard in raising it.
| Task | Start with |
|---|---|
| Run a first register program | guide/, then matmul-as-conv.md and encodings/regcmd-task-model.md |
| Port to another SoC | chips/README.md, then chips/porting-patterns.md |
| Add an operator | encodings/dpu-lut-activation.md, encodings/feature-reduce.md and encodings/sdp-stage-precision.md |
| Speed up LLM prefill | TUNING.md, then perf/not-mac-bound.md |
| Run a model | MODEL-NOTES.md, then TUNING.md |
| Document | Covers |
|---|---|
| GLOSSARY.md | The notes' own terms, the NPU's block and stage names, and the abbreviations of the models the notes run, each with its source |
| hardware-overview.md | What the NPU is: its NVDLA lineage, the block pipeline, the cores and CBUF, and its rated and measured throughput |
| nvdla-lineage.md | What the NVDLA's open documentation says about this NPU, where it is wrong for this silicon, and the Rockchip additions it does not describe |
| datatypes.md | The datatype capability matrix: precision field, output type, MAC rate and use, per datatype |
| matmul-as-conv.md | How a matmul runs as a 1×1 convolution: the data flow, the layouts, tiling, the alignment rules, and a KxK convolution run as an im2col matmul |
| depthwise-conv.md | How a depthwise convolution differs from a direct one |
| Document | Covers |
|---|---|
| chips/README.md | The per-SoC status table, and what varies between SoCs |
| chips/rk3588.md | The RK3588's machine parameters, register map and SoC integration, and the per-core float difference on output lane 0 |
| chips/rk3576.md | The RK3576: supported operations, limits and errata, and the cost of its library entries |
| chips/rk3576-regcmd.md | The RK3576 register encoding and its hardware verification |
| chips/rk3576-hazards.md | The RK3576's wide-output hazard, drain deadline and pooling divisor lag in detail, and its unresolved hazards |
| chips/rk3566.md | The RK3566, from published sources, not yet verified on hardware |
| chips/porting-patterns.md | Which facts transfer between NPU revisions, and which invert |
| Document | Block | Covers |
|---|---|---|
| encodings/precision-field.md | CNA, CORE, DPU | The 3-bit precision values for all six datatypes |
| encodings/tile-layouts.md | CNA, DPU_RDMA | The feature, weight and output cube layouts, per datatype |
| encodings/size-e-quirk.md | DPU write-out | The size_e=7 stride of the int8 and int4 outputs |
| encodings/output-transpose-int16.md | DPU write-out | int16's saturating int32 writer, the transposed writer, and the byte-decomposition path |
| encodings/out-cvt-converter.md | DPU write-out | The integer output converter (acc×SCALE)>>SHIFT, its tie-to-even rounding, and the sign bit in its scale |
| encodings/sdp-stage-precision.md | DPU SDP, CORE | The precision of the BS, BN and EW stages, the EW stage's integer mode, and why CACC cannot accumulate across ops |
| encodings/k-accumulation.md | DPU-EW, DPU_RDMA | Summing K-partials on-chip: the supported fp16 path, and what an integer path still needs |
| encodings/cross-op-chaining.md | CNA, DPU | Feeding one fp16 op's output cube to the next op with no host de-tile |
| encodings/regcmd-task-model.md | PC | What a task is, how a delta task inherits register state, and which pointers the kernel sets |
| encodings/cbuf-reuse.md | CNA (CBUF) | The WEIGHT_REUSE and DATA_REUSE operand-reuse bits |
| encodings/cbuf-bank-slack.md | CNA (CBUF) | The int8 feature DMA's one-bank over-read |
| encodings/cna-dcomp-weight-decompression.md | CNA (DCOMP) | The weight-decompression block, decoded |
| encodings/mrdma-trap.md | DPU_RDMA | The register block a job must emit to avoid a timeout |
| encodings/dpu-lut-activation.md | DPU LUT | Activations and elementary functions through the LUT, the elementwise ops, and the LUT's hazards |
| encodings/conv-transpose.md | CNA | ConvTranspose: the hardware deconvolution mode, the lowering onto a forward convolution, and the resident path |
| encodings/resize-upsample.md | CNA | Nearest and bilinear resize as a depthwise transposed convolution |
| encodings/ppu-pooling.md | PPU | Max and average pooling |
| encodings/ppu-reduce-mean.md | PPU | Global average, max and min pooling over the spatial axes |
| encodings/feature-reduce.md | CNA, CORE, DPU | Reducing over the feature axis, and prefix sums, as a matmul against a ones weight |
| Document | Covers |
|---|---|
| encodings/whisper-encoder.md | A full Whisper encoder block on the NPU, and the operators it is built from |
| encodings/siglip-encoder.md | The SigLIP-B/16 vision encoder end to end on the NPU |
| encodings/ffn-block.md | The gated FFN (GeGLU, SwiGLU) |
| encodings/rmsnorm-onnpu.md | RMSNorm from a square, a feature reduce and a host rsqrt |
| encodings/norm-vision.md | The vision normalizations, on the LayerNorm machinery |
| Document | Covers |
|---|---|
| perf/not-mac-bound.md | Why the whole-tile matmul program measures within ~20% at every datatype, and what CBUF weight streaming gives |
| perf/device-vs-host-split.md | How much of a matmul's wall-clock time the device takes, with resident and re-packed weights |
| perf/decode-gemv.md | Why single-row decode stays on the CPU |
| perf/asymmetric-tile.md | Why an Mt>Nt tile beats the symmetric maximum |
| perf/weight-residency-fusion.md | Packing F16 weights once, and fusing projections that share an input |
| perf/quant-prefill-microbatch.md | Why quantized-GGUF prefill is bound by per-micro-batch dequantization |
| perf/moe-placement.md | Which MoE expert GEMMs the default places on the NPU: the work-per-dispatch threshold and the residency budget |
| perf/attention-offload-crossover.md | The context length where offloading prefill attention starts to win |
| perf/rk3576-w8a8-llm.md | LLM prefill on the RK3576's int8 matmul through llama.cpp: speed, accuracy and the per-column calibration |
| perf/rk3576-graph.md | Whole networks on the RK3576: the graph planner, the TFLite delegate's results, and the terms of a graph's wall-clock time |
| perf/iova-and-multicore.md | The per-fd 4 GB IOVA window, and one fd per core |
| perf/bo-sync-cost.md | The PREP_BO/FINI_BO cache-sync cost, which scales with BO size |
| perf/pool-completion.md | Why a pool submit on mainline rocket waits ~507 ms and resets a core |
| perf/clock.md | The boot clock, the requested rate against the physical PVTPLL clock, raising it safely, and the gain to 1019 MHz |
| perf/cpu-governor-and-offload.md | How the CPU governor biases an NPU-against-CPU comparison, what it costs a per-layer offload inside a model, and how to pin it |
| perf/cpu-repack-baseline.md | Why a quantized CPU baseline needs llama.cpp's weight repack, and the NPU's lead over a repacked CPU |
| perf/hw-byte-counters.md | The NPU's missing DMA byte counters, and the DDR controller PMU that measures board traffic instead |
| perf/ppu-pooling-not-detile.md | The PPU as a pooling engine that cannot de-tile |
| perf/rga-detile.md | Why the RGA engine de-tiles bit-exactly but loses to the CPU |
| perf/sram-nbuf.md | How the NPU reaches system SRAM, and why it gives little speedup |
| perf/asr-cpu-relief.md | ASR measured in CPU core-seconds |
| perf/asr-streaming-service.md | whisper-server fed fixed-length chunks, in CPU core-seconds per second of audio |
| perf/per-process-readout.md | The per-process instrument behind each timed configuration, and its silent failures |
| perf/benchmarks.md | The consolidated benchmark record: method and per-model results |
| perf/data/ | Per-model measurement records and the benchmark scripts |
| Document | Covers |
|---|---|
| MODEL-NOTES.md | Per-model behavior on the stack: faithfulness checks, sampling settings, quirks |
| TUNING.md | Which options to set for a workload |
| guide/ | The start-to-finish walkthrough across the driver library, the frontends and the kernel patches |
| Document | Covers |
|---|---|
| SOURCES.md | Every external source, and what each covers |
| ppu-rknn-capture/ | The vendor compiler's PPU pooling program, captured and decoded |
| teflon-add-capture/ | A Teflon regcmd capture for the elementwise-operand work |
| whisper-encoder-validation/ | Scripts that reproduce the Whisper encoder validation |
A claim has a tag that says how it was established:
| Tag | Means |
|---|---|
[HW sweep] |
Observed on the device by sweeping a value or geometry, and scoring each result as bit-exact, garbage, or nothing written. |
[source-confirmed] |
Corroborated by an authoritative source, named at the claim: most often the Mesa rocket (Teflon) driver, the open NVDLA documentation, the npu_hw.h register headers from Jasbir Matharu's rk3588-npu, the vendor BSP kernel, or the kernel source and patch series. SOURCES.md lists them. |
[TRM] |
Stated in Rockchip's Technical Reference Manual. |
[expected] |
An outcome not yet measured: a prediction, a derived bound, or an estimate. |
[hypothesis] |
A proposed cause for a measured effect, not yet isolated. The effect is established, and the explanation is not. |
[live] |
Read once from a running board: a clock rate, a regulator state, a device-tree property, a module version. It states the date and the kernel, because a reflash can change it. |
[host-computed] |
Computed on the host with no device run: arithmetic on recorded numbers, a planner's output, a census of a model file. The computation is exact, and its inputs have their own tags. An estimate is [expected] instead. |
Where a fact is measured and a source agrees, both tags appear. Performance figures state the SoC, the NPU clock and the tensor shape, in a Conditions: line under the section or table they apply to.
On the RK3588, the NPU clock in a Conditions line is the physical frequency, not the requested
rate. Above 200 MHz the two differ, because the clock is a PVTPLL that follows the rail voltage
(perf/clock.md). On the Turing RK1, ~816 MHz is the 600 MHz request
(rocket_npu_clk_hz=600000000) at a 0.80 V rail. The 900 MHz request runs at 928 MHz, the
clock of the RK1 measurements from 2026-10-19 on. A rate named for the vendor kernel is that driver's OPP, whose
physical frequency is not characterized.
A note's record, where it has one, is an Evidence section at its end. It states how each fact was established, including the fields and values a negative result searched. A hazard is marked as a Caution callout, and a defect as an erratum entry with its conditions, symptom and workaround. GLOSSARY.md defines the notes' own terms, the NPU's block and stage names, and the abbreviations of the models the notes run. It also names the source of each.
Most findings come from devices running mainline 7.1 and 7.2 kernels:
- A Turing RK1: RK3588, 32 GB
- An H96 MAX M9: RK3576
A second Turing RK1, on Rockchip's 6.1 BSP kernel with the vendor rknpu 0.9.8 driver, provides
the vendor-driver results.
The same work produced a FOSS inference stack on rocket: a userspace library, frontends for
ggml, TFLite and ONNX Runtime, and kernel patches. The guide is a start-to-finish
walkthrough that ties the driver library, the frontends and the kernel patches together.
The documentation in this repository (the prose notes, tables, and encoding write-ups) is licensed under Creative Commons Attribution 4.0 International (CC-BY-4.0): reuse it freely with attribution.
The scripts and programs (every .py, .sh and .c file, each with an
SPDX-License-Identifier: MIT line) are licensed under the MIT License. The C
programs in perf/data/rknn-encoder/ and perf/data/rknn-decoder/ link Rockchip's proprietary
librknnrt. The generators in lut-rknn-capture/ and ppu-rknn-capture/ and the
perf/data/siglip-rknn/ harness import Rockchip's proprietary rknn-toolkit2 or
rknn-toolkit-lite2. Those come from Rockchip under its own terms. Third-party captures retain
their upstream copyright and license terms, notably ppu-rknn-capture/registers.xml (from the
Mesa rocket/Teflon driver), credited in SOURCES.md.