Skip to content
chili-chipsPublic

About

Opensource PCIE backplane and RTL, with full end2end sim, and our tiny, soft, bare metal RISCV (eduBOS5) pretending to be a PC

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

NLnet:Funded Stars GitLab:Mirror License:BSD-3/CC-BY-SA KiBot:Pass eduBOS5:Pass Lint Verible Sim:Pass WaveCrux:YES JanusHDL:WIP openXC7:Pass PCIE:Pass

openPCIE Root Complex project logo

openPCIE is an open-source PCIE Root Complex (RC) for the Artix-7 FPGA, driven by a soft RISC-V SoC. It wraps the Xilinx Series-7 PCIE and GTP hard macros in open RTL, adds a bare-metal driver with a demo app, full HW/SW co-simulation, and an open backplane PCB to plug it all into. Gen2 link up on hardware; builds with Vivado or the fully open openXC7 toolchain.

Open PCIE peripherals for Artix-7 exist, but they are always controlled by a proprietary RC: a PC, or a Raspberry Pi. openPCIE lets makers drive PCIE peripherals from their own soft RISC-V SoC. It answers the call for open critical IP in "Recommendations and Roadmap for European Sovereignty in open source HW, SW and RISC-V Technologies (2021)". Looking for an open soft PCIE EndPoint (EP)? See openCologne-PCIE.

Where What
1.pcb openBackplane PCB (KiCad)
2.rtl open RC RTL: RC-direct, RC-switched, eduBOS5 CPU
3.sw bare-metal PCIE driver and demo app
4.build CSR, firmware and bitstream builds (openXC7)
5.sim HW/SW co-simulation with the pcieVHost VIP
2.amd-rtl-with-Vivado-build AMD/Vivado-IP reference designs

Long-term goal: a full open PCIE stack, phasing out the Xilinx hard macros, mixed-signal SerDes and PLLs included. This first phase builds on the Series-7 PCIE hard macro, the GTPs and their low-jitter PLL.

References


HW Architecture

Our openPCIE Backplane, SQRL Acorn CLE-215+ FPGA modules and PCIE adapters form the test platform.

openPCIE hardware ecosystem

openBackplane PCB

openPCIE backplane intro video (click to watch) openPCIE backplane block diagram

Details: 1.pcb. KiCad library: 1.pcb/KiCad-library. Plain-text versions of the PDF and slide documents: 0.doc/1.pcb.

FPGA Carrier

The SQRL Acorn CLE-215+, a former crypto-accelerator, is two boards:

  • M.2 FPGA module (Acorn CLE-215+): the Xilinx Artix-7 XC7A200T in an M.2 M-key form factor.
M.2 FPGA module, top view M.2 FPGA module, bottom view
M.2 FPGA Module (Top View) M.2 FPGA Module (Bottom View)
  • PCIE adapter board (Acorn Baseboard Mini): adapts M.2 to a PCIE x4 edge connector, for a regular motherboard slot.
PCIE adapter board, top view PCIE adapter board, bottom view
PCIE Adapter Board (Top View) PCIE Adapter Board (Bottom View)
Fully assembled Acorn CLE-215+
The fully assembled Acorn CLE-215+ development board, ready for use in a PCIE slot.

The Acorn CLE-215+ is a NiteFury with 1 GB of DDR3 instead of 512 MB, so the NiteFury schematic is an accurate reference. Its Xilinx Artix-7 XC7A200T-FBG484 hosts the RC, and on the other card the EP:

Specification Value
Family Xilinx Artix-7
Speed Grade -3
Logic Cells (LUT4-Equivalent)¹ 215,360
LUT6 134,600
Flip-Flops 269,200
Block RAM 13 Mbit
DSP Slices 740
GTP Transceivers 4 (up to 6.6 Gbit/s)
DDR3 SDRAM (Board) 1 GB, 16-bit
QSPI Flash (Board) 32 MB
User LEDs 4
General Purpose IOs 4
LVDS Pairs 4

¹ A Xilinx metric: the physical 6-input LUTs expressed as equivalent 4-input LUTs. The other counts are exact for the XC7A200T.

References:


RTL Architecture

A chain of blocks joined by SystemVerilog interfaces: the RISC-V SoC hands TLPs over AXI-Stream to the PCIE stack, which converts them for the PCIE_2_1 hard block and drives the GTP transceiver over PIPE.

RTL blocks of the opensource Root Complex and the SystemVerilog interfaces between them

Details: RTL README. Per design: common RTL, EP, RC-direct, RC-switched. AMD/Vivado-IP references: 2.amd-rtl-with-Vivado-build (EP, RC-direct, RC-switched).

References:


SW Architecture

The driver runs bare-metal on the soft RISC-V SoC and does every enumeration step itself, the job an OS does on a PC.

Firmware layers: application, PCIE driver, HAL, register map and startup, talking to the CSR block over the CPU memory bus, then through pcie_top to a direct endpoint or a switch with up to 4 endpoints
Layer Does
Application Memory Write of a payload to the EP, Memory Read to check it
PCIE driver enumeration: discovers the device, sizes the BARs by read-back, assigns aligned addresses, checks every Completion Status, sets the Command Register
HAL register reads and writes at fixed addresses

The HAL and the CSR RTL are auto-generated by peakrdl from one SystemRDL file, 4.build/csr_build/csr.rdl: the register RTL (csr.sv, csr_pkg.sv, bridged to the CPU bus by soc_csr.sv) and the header csr.h. The firmware reaches every register through CSR_REG32(): a load or store on eduBOS5 or PicoRV32, a VProc bus cycle when compiled for the host in co-sim.

Details: SW README.


Simulation Architecture

Sim Test Bench

One test bench, three interchangeable CPUs: the RTL core that goes into the bitstream (eduBOS5 by default, or PicoRV32), or the VProc virtual processor, which runs the firmware much faster.

openPCIE Root Complex test bench structure

openPCIE_BFM is a submodule (5.sim/vip/openPCIE_BFM): clone with git clone --recurse-submodules, or run git submodule update --init in a clone.

openPCIE_BFM logo

A first for open source: to our knowledge, no open-source, rule-based PCIE protocol checker exists at any layer. Our Artix-7 RC model drives the EP and checks every TLP and DLLP against PCIe Base 2.1 rules, failing the run on any violation (the gap).

Follow the co-simulation at TLP level in WaveCrux, with the open-source PCIE TLP decoder plugin and a ready-made session. Here the firmware sizes BAR0 by read-back and places it at 0x80000000 (more):

WaveCrux TLP view: firmware sizing BAR0

One level down, Ferrite's PCIe PIPE and Data Link Layer decoders show link training, flow-control credits, and every TLP with its sequence number and Ack (more). With all three layers in one view, a TLP spends about 0.4 µs each way inside the hard macro (more).

References


Implementation Workflow

Build flow: CSR generation, firmware, bitstream through Vivado or openXC7, co-simulation, lint and formal equivalence
Step Do Details
1. Tools install an FPGA toolchain (AMD Vivado or openXC7), a bare-metal RISC-V GCC and Python; source set-env.sh checks them 4.build
2. Co-sim run the HW/SW co-simulation: init, register access, link, RC-EP data exchange 5.sim
3. SW build compile the bare-metal driver and test app 4.build/sw_build
4. HW build synthesize, place, route, make the .bit (RC or EP) openXC7, Vivado
5. Deploy program RC and EP boards, press reset, check link LED and payload below

Bringup, Debug, Bench Testing and Demo

FPGA Board Setup

The SQRL board needs a custom JTAG cable, external power, and an open desktop chassis.

1. Custom JTAG Cable

The board's JTAG connector does not fit the Xilinx Platform Cable's 14-pin plug.

FPGA JTAG connection JTAG connector pinout on the board
Custom JTAG Cable connecting the Xilinx Programmer to the board JTAG Connector Pinout on the Board

The board has a Molex Pico-Lock 1.50 mm pitch male header; DuPont cables do not fit. Buy Molex 0369200603 on Digi-Key (female on both ends), cut it in half, and splice one end onto the Xilinx programmer's wires per this diagram:

Acorn JTAG wiring diagram
JTAG Connection Guide: Physical Pinout and Wiring Diagram.

2. External 12V Power Supply

Slot power is not enough to program or run the board. Feed 12 V through a 6-pin PCIE connector from an ATX supply.

External 12 V power connection
External 12V power connection.

3. Final Assembly

Complete FPGA system mounted in a PCIE slot
The complete FPGA system mounted in a PCIE slot.

4. JTAG Connectivity Test

Vivado Hardware Manager must detect the programmer and the xc7a200t_0 FPGA:

Successful device detection in Vivado Hardware Manager
Successful device detection in Vivado Hardware Manager.

Test Jig

  • PCB: openPCIE Backplane (1.pcb), with slots for RC, EP and the switch, on external 12 V.
  • FPGA boards: two SQRL Acorn CLE-215+, one RC, one EP.
  • JTAG: two Xilinx Platform Cable USB, each with the custom adapter.
Validation environment for the direct connection
The complete hardware validation environment for the Direct connection test.

The "Dual-PC" JTAG Approach

PC 1 drives the RC's JTAG, PC 2 the EP's. No cable swapping, no flashing per iteration, and two Vivado Hardware Manager ILAs triggering at once show a transaction from both sides of the link.

Test Procedure

  1. Insert both FPGA cards into the openPCIE Backplane and connect 12 V.
  2. Program the RC from PC 1 and the EP from PC 2 (Vivado Hardware Manager).
  3. Press and release the backplane's reset button: the RC enumerates and links with the EP. Each press re-initializes the link, no re-programming needed.

Goal: the RC enumerates, writes a payload to the EP, the EP stores it in its BRAM, and the RC reads it back (Memory Write and Memory Read).

Bench Validation Steps

  1. LEDs: the RC's 4 user LEDs show link status; the EP's show the received data.
  2. ILA: Vivado's Integrated Logic Analyzer on both PCs shows the data sent and received, its timing, and the transaction details.

Bench Validation Results

Verified on hardware, x1 Gen2: link up and payload transfer, RC to EP.

  • RC board: LED A3 lit = PCIE link up.
  • EP board: the LEDs show the payload. The RC sent decimal 6, binary 0110, and the LEDs match.
LEDs showing link up LEDs showing the data payload
RC-LED-Link-Up EP-LED-Data-Payload

Primary References

Secondary References

PCIE Protocol Analyzers


Project Status

All six parts of the plan are done.

PART 1. Mini PCIE Backplane PCB

Consumer RCs are soldered onto the motherboard, and FPGA boards are built as EPs: they expect clock, reset and sideband signals from the slot. Only professional systems (VPX, VITA 46.4) put RC and EP on add-on cards. No suitable ready-made backplane existed, so we built one: a mini backplane that gives the PCIE context of a motherboard, minus the soldered-down RC. Envox.eu developed the PCB.

  • ✅ Requirements document.
  • ✅ Components, schematic and PCB layout.
  • ✅ Review and iteration for 5 GT/s (PCIE Gen2), high-speed traces simulated with openEMS.
  • ✅ Prototype manufactured, brought up, Signal Integrity checked with AMD's on-chip IBERT.
  • ✅ Second batch with all improvements; design files released with full documentation.

PART 2. Project setup and preparatory activities

  • ✅ FPGA boards and PCIE accessories procured.
  • ✅ Prototype system brought up with proprietary RTL IP, SW driver, TestApp and Vivado.

PART 3. Initial HW/SW implementation

  • ✅ Open RTL that mimics the proprietary PCIE RC.
  • ✅ Open bare-metal driver for the RC, on the RISC-V SoC, no Linux (SW Architecture).
  • ✅ RISC-V SoC with the PCIE RC as its main peripheral.

PART 4. HW/SW co-simulation using full PCIE EP model

Planned as a simple EP BFM; grown, after NLnet's questions, into a full SoC co-simulation with wyvernSemi onboard. Their VProc models the RISC-V CPU and its SW, and their PCIE model provides an EP with a configurable configuration space, paired with our RC RTL.

  • ✅ PCIE model extended for more configurable EP capabilities.
  • ✅ Testbench built; testcases run and debugged.
  • ✅ EP model, TB and sim environment documented for reuse in other projects.

PART 5. Integration, testing and iterative design refinements

  • ✅ Proprietary elements of the PART 2 prototype replaced one by one with ours (except Vivado and TestApp), tested along the way.

PART 6. Prepare Demo and port it to openXC7

  • ✅ Open PIO TestApp and demo.
  • ✅ Design built with openXC7; issues reported and worked with the developers.

ScalePNR was planned as a fallback for nextpnr-xilinx's timing shortcomings; it was not needed. The nextpnr-xilinx bitstream brings the link up at Gen2 on hardware, as Vivado does. That took six fixes: one latent RTL bug and five workarounds for openXC7 defects, documented with the filed issues in 4.build/hw_build.openXC7.

The design now builds with openXC7 1.0.0, what "ScalePNR" evolved into: Yosys 0.69 with integrated Slang, nextpnr-himbaechel and fpga-as. Slang makes the sv2v prep step unnecessary; sv2v stays as a cross-check option. We assemble with fasm2frames and xc7frames2bit, as fpga-as writes one GTP setting wrong (#12). The 1.0.0 bitstreams are not tested on hardware yet; the Gen2 result is from the earlier flow.


Beyond the plan

  • x4 and Gen2. The RC builds as x4 Gen1 (default), x4 Gen2 or x1 Gen2, with the user clock scaled to the link (62.5, 125 or 250 MHz). Vivado meets timing on all three, and the x4 co-simulation passes. On hardware, the x4 link trained before the user-clock fix; x4 TLP traffic is not re-tested yet (2.rtl).
  • eduBOS5, our own RISC-V core. It replaces PicoRV32 as the default CPU, with twice its IPC: the firmware finishes in 390.2 µs instead of 667.2 µs. Its 3-stage form is the only core here that closes x4 Gen2's 250 MHz. Checked against the ISA with riscv-formal, and its multicycle paths proven formally (eduBOS5).
  • RC-switched in simulation. A switch model behind the link (two endpoints, two empty ports) runs the RC-switched firmware end to end, enumeration through memory traffic (5.sim).
  • A PCIE TLP decoder in WaveCrux. Our experimental TLP decoder is now Ferrite's official pcie-tlp decoder in wavecrux-decoders: every TLP of the co-simulation, named and paired, in the waveform viewer (5.sim).

Academia Outreach

openPCIE is the basis of several Master's theses at the Faculty of Electrical Engineering, University of Sarajevo (ETF Sarajevo), among them those of Anes Vrce and Aladin Brdar, shown here presenting his work on its hardware, software and simulation.

Aladin Brdar presenting his work on the openPCIE platform at ETF Sarajevo

Acknowledgements

We are grateful to NLnet Foundation for their sponsorship of this development activity.

NLnet logo NGI Zero, Next Gen Internet

This project was funded through the NGI0 Core Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101092990.

The wyvernSemi's wisdom and contribution made a great deal of difference -- Thank you, we are honored to have you on the project.

wyvernSemi-Logo

The Envox, our next-door buddy, is responsible for the birth of our backplane, which we like to call BB (not to be mistaken for their gorgeous blue beauty BB3)

EEZ / Envox logo
openCologne-PCIE logo

KiCad turns electrical idea into a manufacturable board. openXC7, with Yosys at its core, turns RTL into a loadable FPGA bitfile. PeakRDL turns our SystemRDL file into CSR RTL+headers+RegMap docs. WaveCrux greatly facilitates waveform decoding and viewing. But, that's not everything that made this project possible -- Thank you all for these wonderful opensource tools ✨.

KiCad logo openEMS logo
openXC7 logo Symbiotic EDA logo
PeakRDL logo YosysHQ logo
WaveCrux logo Verilator logo JanusHDL logo
MikePopoloski slang logo draw.io logo Chips Alliance SV Tools logo Verible logo openFPGALoader logo eduBOS5 logo PicoRV32 logo RISC-V logo EBMC logo ngscopeclient logo LiteX logo

Crowd Supply Campaign

Crowd Supply logo Open Source Hardware logo
Hackster logo Elecrow logo Kickstarter logo

Community Outreach

Issues filed upstream from this project, open and closed. Details of the open ones: openXC7 findings.

Tool / project Issue What State
openXC7 nextpnr nextpnr#21 GT attribute defaults differ from Xilinx's open
openXC7 nextpnr nextpnr#182 set_multicycle_path: -from drops the constraint; inferred registers cannot be selected open
openXC7 nextpnr nextpnr#183 BEL attribute: the reported name is rejected, an accepted one ignored open
fpga-assembler 282ef7c2 fpga-as writes RXCDR_CFG wrong on every GTP lane fixed upstream, not yet in the toolchain
Yosys yosys#6498 wrong IBUFDS_GTE2.CLKSWING_CFG default open
fpga-assembler lromor/fpga-assembler#66 compressed bitstreams (Multiple Frame Write), as Vivado makes open
Yosys, sv-elab yosys#6499, sv-elab#72 read_slang rejects the specify blocks of xilinx/cells_sim.v open
sv-elab sv-elab#282 no real parameters on primitives (MMCME2_ADV) open
EBMC hw-cbmc#2341 6.2 rejects a 1-bit enum (eduBOS5 proofs) open
JanusHDL #1 0.17.0: what simulates and what does not open
openXC7 nextpnr nextpnr#84, #88, #90, #92, #93, #94, #146 GT PLL settings; TXPI_SYNFREQ_PPM; unconstrained clock timed at 12 MHz; --placer sa; truncated FASM assembled; contradicting GT LOC; falling-edge FFs fixed on main, not yet released
openXC7 nextpnr PRs nextpnr#145, #149, #150, #152, #153, #154 set_multicycle_path: setup sign, Fmax, skew, tag check, XDC selectors, LUT-RAM tags merged on main, not yet released
openXC7 nextpnr-xilinx nextpnr-xilinx#100 IBUFDS_GTE2.O driving a GT and a BUFG: dead fabric clock closed (fixed on main)
EBMC hw-cbmc#2274 6.1: a typed parameter passed to a child parameter closed (fixed in 6.2)
WaveCrux wavecrux#22 decoder transactions drawn at the wrong time (time units) closed (fixed in 1.0.1)

Numbers without a link were filed from an account that is no longer available. JanusHDL has no public tracker (closed source, so it can read encrypted IP): its issues are in our repos, under the JanusHDL label.

Community Involvement

Want to help the open PCIE community grow? See Contributions.

Independent verification. openpcie-uvm by Sandeep-bhardwaj (MIT, Oct 2026) is a UVM environment for our riscv_pcie_soc on Vivado xsim: the AXI4-Stream TLP port under backpressure, the bridge registers, completion capture, reset in mid-traffic. It tests openPCIE at 6e48e11 (July 2026), above the hard block; PCIE protocol is out of its scope. Its three findings, and where they stand on main:

Finding On main
RTL-SIM-001: ram written by both $readmemh and an always_ff (VCS refuses it) fixed: a plain always
TX-PROTO-004: the TX state machine tests tready a cycle early; a stalling sink loses the TLP header TX state machine rewritten in 2d810aa to hold each beat until tvalid && tready; not re-run with its tests
REG-004: register writes ignore address bit 5 the registers are now generated from SystemRDL (soc_csr.sv); not re-checked

Questions are welcome ===> issues.


Future outlook

Next: openPCIE3, a 24-lane Gen3 backplane with Optical Copper Link (OCuLink), around the Diodes PI7C9X3G1224GP switch (PCIE 3.1, 24 lanes, 12 ports). It links one or two PCs, each over an x4 OCuLink cable, to four x4 EP connectors (two slots, two M.2).

Main use: a PC I/O extender for compute, storage or I/O cards, e.g. for open-source AI servers. The PC drives REFCLK and reset over the cable, so the board needs no local clock or reset generator. Design input is welcome in #2.


GenAI

Our repo may contain elements of work facilitated by the modern GenAI tools Claude, used under strict guidance and tight supervision of the blood-and-flesh developers, and solely as suggestions.

In the end, it is us, the human devs, who are fully responsible for any shortfalls you may uncover while tapping into repo resources.


End-of-Document

About

Opensource PCIE backplane and RTL, with full end2end sim, and our tiny, soft, bare metal RISCV (eduBOS5) pretending to be a PC

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages