1 Technical Principles

1.1 What is GPU Direct RDMA

GPU Direct RDMA (Remote Direct Memory Access) is a high-performance data transfer technology provided by NVIDIA that allows third-party PCIe devices (such as FPGAs, network cards, video capture cards) to bypass the CPU and system memory, and directly exchange data with GPU video memory through the PCIe bus.

Traditional data path (FPGA → GPU):

FPGA ──DMA──→ CPU Memory
         ↑
      PCIe Transfer (1st time)
                ──cudaMemcpy──→ GPU Video Memory
                                       ↑
                                  Memory Bus Copy (2nd time)

GPU Direct RDMA path (FPGA → GPU):

FPGA ──PCIe DMA──→ GPU Video Memory
         ↑
    Only one PCIe transfer, zero CPU copy

1.2 Core Advantages

1.3 Adapted Devices and Platforms

1.4 Workflow

1. GPU buffer allocation (cudaHostAlloc / cudaMalloc)
2. Pin operation: Map GPU virtual address to physical pages and lock in memory
3. DMA transfer: FPGA directly reads/writes GPU physical pages (via PCIe BAR)
4. Unpin operation: Release page lock

Key point: The Pin operation only needs to be executed once, and the same Handle can be reused infinitely for subsequent DMA transfers, avoiding the address translation overhead required for each transfer in the traditional method.

2 Test Environment

2.1 Hardware Platform

Xingce Electronics JetKU hardware, PCIe Gen3x4 between FPGA and Jetson NX, FPGA with 2GB cache, using Xilinx XDMA IP

2.2 Software Environment

2.3 Driver Architecture

User Space:  HelloFPGACore.so (TransferMode API)
               │
               ├─ CPU Mode: open(/dev/HelloFPGA0_c2h_*) → read/write
               │
               └─ GPU Direct Mode: open → ioctl(XDMA_IOC_GPU_PIN/XFER/UNPIN)
                        │
Kernel Space:  HelloFPGA.ko (XDMA + xdma_gpu_direct + xdma_gpu_tegra)
                        │
Hardware:      FPGA XDMA IP ←──PCIe──→ GPU BAR (Physical Address Direct Access)

3 Test Methods

3.1 Test Tools

Test program: gpu_direct_api_test.cu

Supports two running modes:

• Quick functional test: Verify API correctness + performance comparison (about 2 minutes)
• Long-term steady-state stress test: Continuous operation for 12 hours, recording data every 10 minutes (--long parameter)

3.2 Test Comparison Scheme

3.3 Test Items

3.4 Key Parameters

• Transfer block size: 4MB (long test) / 4KB~8MB (quick test)
• FPGA address step: 64MB (traverse 0~2GB, 32 test points / round)
• Sampling frequency: every 10 minutes (long test)
• Iterations per sampling point: average of 5 times
• Power consumption collection: INA3221 sensor (VDD_IN channel)

4 Test Results

4.1 Multi-size Performance Comparison (Quick Test)

Latency comparison (μs, lower is better), table format: Read / Write

Bandwidth comparison (GB/s, higher is better), table format: Read / Write

GPU Direct peak bandwidth reaches 2.84 GB/s, approaching the theoretical limit of PCIe Gen3 x4

4.2 TransferMode Compatibility Mode Performance

Zero modification to user code (still calling HelloFPGA_DMA_MM_*), switch mode only through 2 lines of configuration; Unit: μs, Format: Read / Write

4.3 Multi-buffer vs Single-buffer Comparison

Simulate actual image acquisition scenario: 4 GPU buffers rotation vs single buffer repeated read/write (100 iterations, 1MB/frame)

Conclusion: The overhead of multi-buffer address lookup and matching is negligible and does not affect performance.

4.4 12-hour Long-term Steady-state Test

Test duration: 12.00 hours; Number of sampling points: 72 times (every 10 minutes); Transfer block size: 4MB; Address traversal: 0 ~ 2GB (completed 2 full rounds of traversal)

Performance Statistics

Speedup Ratio (Baseline: Traditional Path)

GPU Direct Performance Stability

Performance Trend by Time Period (No Performance Degradation)

Power Consumption Statistics

Address Consistency Conclusion: There is no significant difference in GPU Direct transfer latency across the full 2GB address range of FPGA (0MB ~ 1984MB, 64MB step) (standard deviation < 15μs), and the performance of each area of FPGA DDR is balanced.

5 Conclusions

5.1 Performance Conclusions

• Compared with the traditional FPGA→CPU→GPU full path, GPU Direct accelerates Read by 1.97 times and Write by 2.49 times; the core benefit comes from eliminating the cudaMemcpy intermediate copy;
• It still improves by about 1.5 times compared with the CPU-only DMA scheme, and the zero-copy architecture has a natural advantage;
• The measured peak bandwidth is 2.84GB/s, close to the theoretical throughput upper limit of PCIe Gen3 x4 hardware.

5.2 Stability Conclusions

• No performance degradation throughout the 12-hour long test, and the latency difference between the first and last 4 hours is less than 0.3%;
• No memory leaks, and the Pin/Unpin resource release logic is normal;
• Power consumption is stable without overheating and frequency reduction, and there are no performance hotspots in all DDR addresses of FPGA.

5.3 Compatibility Conclusions

• Fully backward compatible with the original upper-layer business code, no modification required;
• Extremely low access cost, only two new lines of configuration code are needed to switch to GPU direct transfer mode;
• Multi-frame image buffer rotation has almost no performance loss, adapting to machine vision stream processing scenarios.

5.4 Recommended Scenarios

5.5 Recommendations and Limitations

• Optimal transfer block ≥64KB: The overhead of small packet DMA establishment accounts for a high proportion, and the gain of GPU Direct is limited;
• Pin operation is only executed once during initialization; it is forbidden to repeatedly Pin/Unpin in the loop;
• GPU cache address and transfer length must be 4K-byte aligned;
• GPU Direct device ioctl operation requires root privileges to run the program.