FPGA PCIe P2P Communication: Implementing GPU Direct RDMA

Xingce Electronics' FPGA PCIe devices fully support GPU DIRECT DMA, enabling P2P (Peer-to-Peer) data transfer from FPGA cache to GPU memory bypassing the CPU. This significantly improves system performance, reduces transmission latency, and is suitable for latency-sensitive scenarios such as quantum computing and semiconductor testing.

This article focuses on P2P transmission between FPGA and GPU, and does not cover P2P between FPGAs or between GPUs. Stay tuned for more content.

1. Solution Overview

1.1 What is GPU Direct RDMA

GPU Direct RDMA allows FPGAs to bypass the CPU and system memory and directly exchange data with GPU video memory via the PCIe bus.

Traditional Path:  FPGA ──DMA──> CPU Memory ──cudaMemcpy──> GPU Memory  (2 copies)
GPU Direct: FPGA ──PCIe DMA──> GPU Memory                        (1 copy, zero CPU involvement)

1.2 Core Value

1.3 Application Scenarios

2. Validation Platforms

2.1 Hardware and Software Environment

Comprehensive validation has been completed on three major platforms, covering embedded, server, and workstation application scenarios:

Compatibility Highlight: The three platforms span aarch64 / x86_64 architectures, CUDA 9.1~12.6 versions, and Ubuntu 18.04~22.04 OS releases, representing industry-leading validation coverage (first in China).

The Jetson platform uses Xingce Electronics' JetKU board [JetKU, Jetson Orin + KU5P FPGA, Gen3x4]

The X86 platform uses Xingce Electronics' PCIe1004 board [PCIe1004, KU040 FPGA + CXP Acquisition, Gen3x8]

2.2 Driver Architecture

User Space:  libHelloFPGACore.so (TransferMode API)
               │
               ├─ CPU Mode:  open → read/write (Traditional Path)
               └─ GPU Direct: open → ioctl (Pin/Xfer/Unpin)
                                    │
Kernel Space:  HelloFPGA.ko (XDMA + GPU Direct Extension)
                                    │
Hardware:  Xilinx FPGA XDMA IP ←──PCIe DMA──> GPU BAR (Physical Address Direct Access)

Some customers have actually used the PCIe AXI Bridge IP, which is more compatible with P2P and is also a good alternative.

3. Test Results

Test Program: gpu_direct_api_test.cu, supporting Quick Test (~2 minutes) and 12-hour Long-term Stability Test modes.

3.1 Functional Validation: 15/15 Passed on All Three Platforms

12-hour continuous testing was completed on 2 platforms with no significant performance degradation and throughput fluctuation within 5%.

A total of 15 test cases were executed with 100% pass rate across all three platforms, fully validating functional reliability.

3.2 Performance Highlights

3.2.1 Peak Bandwidth

Note: For the RTX A4000 test environment, the speed is limited to Gen3x4 due to motherboard slot constraints.

The H100 platform achieves 93.5% of the theoretical PCIe Gen3 x8 bandwidth limit with GPU Direct, and the RTX A4000 reaches 89% of Gen3 x4 limit, representing industry-leading transmission efficiency.

3.2.2 Acceleration Ratio (vs Traditional FPGA→CPU→GPU Path)

The main benefit of GPU Direct over the traditional path comes from eliminating the cudaMemcpy intermediate copy, resulting in acceleration ratios ranging from 1.4x to 3.2x.

3.2.3 Multi-Size Performance (RTX A4000 Example)

GPU Direct outperforms the traditional path starting from 64KB, maintaining a stable 1.4x acceleration ratio for large block sizes. Data for Tegra and H100 platforms are available in the appendix.

3.2.4 Equivalent Frame Rate (1MB / Frame)

No performance loss with multi-buffer rotation - 4-frame rotation achieves identical performance to single-frame transfer, perfectly suited for image acquisition pipeline scenarios.

3.3 TransferMode: Zero-Code Modification Upgrade

Users only need to add 2 lines of configuration code to automatically switch existing HelloFPGA_DMA_MM_* calls to the GPU Direct path:

HelloFPGA_SetGPUBuffer(hDev, gpuAddr, size);                    // Added: Register GPU Buffer
HelloFPGA_SetTransferMode(hDev, HELLOFPGA_XFER_MODE_GPU_PINNED); // Added: Switch Mode
HelloFPGA_DMA_MM_ReadC2H(hDev, ch, buf, offset, len, &actual);   // Existing code unchanged!

TransferMode Acceleration Ratio (Three Platform Comparison)

The RTX A4000 achieves a remarkable 1.93x acceleration ratio for 4KB small blocks. The H100 shows only ~1.1x acceleration because CPU DMA already approaches peak performance with Gen3 x8 bandwidth.

3.4 12-Hour Long-Term Stability Test

Tegra Platform (4MB, 72 Samples)

H100 Platform (4MB, 72 Samples)

Continuous 12-hour operation with no performance degradation on both platforms, demonstrating production-grade stability of the GPU Direct solution.

3.5 Three Platform Comparison Overview

PCIe Link Difference Explanation: Tegra and RTX A4000 are both limited to Gen3 x4 by carrier board/motherboard slot constraints, resulting in similar peak bandwidth levels. The H100 server platform provides Gen3 x8 slots, delivering approximately twice the bandwidth of the other two platforms.

4. Integration Guide

4.1 Four-Step Integration of GPU Direct

// 1. Allocate GPU Buffer
cudaMalloc(&gpuBuf, size);

// 2. Pin Lock (Only once during initialization)
HelloFPGA_GPUDirect_Pin(hDev, gpuBuf, size, &handle);

// 3. DMA Transfer (Reuse handle indefinitely)
HelloFPGA_GPUDirect_ReadC2H(hDev, ch, handle, fpgaAddr, size, timeout);
HelloFPGA_GPUDirect_WriteH2C(hDev, ch, handle, fpgaAddr, size, timeout);

// 4. Release (On program exit)
HelloFPGA_GPUDirect_Unpin(hDev, handle);

4.2 Key Constraints

4.3 Automatic Platform Adaptation

The API automatically detects platform type (Tegra / Desktop) - no conditional compilation required in user code:

HelloFPGA_GetGPUStatus(&status);
// status.platform: 1 = Tegra (cudaHostAlloc), 2 = Desktop (cudaMalloc)
// status.supported: 1 = GPU Direct Available

5. Summary

Core Advantages of HelloFPGA GPU Direct RDMA Solution:

Reference:
https://docs.nvidia.com/cuda/gpudirect-rdma/index.html