Hardware and Software Design of FPGA-based PCIe Gen3 interface for APEnet+ network interconnect system

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Ammendola, R.^1 ; Biagioni, A.^2 ; Frezza, O.^2 ; Lo Cicero, F.^2 ; Lonardo, A.^2 ; Martinelli, M.^2 ; Paolucci, P.S.^2 ; Pastorelli, E.^2 ; Rossetti, D.^3 ; Simula, F.^2 ; Tosoratto, L.^2 ; Vicini, P.^2

INFN, Sezione di Roma Tor Vergata, Italy^1

INFN, Sezione di Roma, Italy^2

NVIDIA Corp, Santa Clara

CA, United States^3

关键词: Hardware and software designs;    High performance networks;    Interconnect systems;    Interconnection architecture;    Interconnection standards;    Internal resources;    Low-level library;    Multi standard;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/9/092017/pdf
DOI  :  10.1088/1742-6596/664/9/092017

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

In the attempt to develop an interconnection architecture optimized for hybrid HPC systems dedicated to scientific computing, we designed APEnet+, a point-to-point, low-latency and high-performance network controller supporting 6 fully bidirectional off-board links over a 3D torus topology. The first release of APEnet+ (named V4) was a board based on a 40 nm Altera FPGA, integrating 6 channels at 34 Gbps of raw bandwidth per direction and a PCIe Gen2 x8 host interface. It has been the first-of-its-kind device to implement an RDMA protocol to directly read/write data from/to Fermi and Kepler NVIDIA GPUs using NVIDIA peer-to-peer and GPUDirect RDMA protocols, obtaining real zero-copy GPU-to-GPU transfers over the network. The latest generation of APEnet+ systems (now named V5) implements a PCIe Gen3 x8 host interface on a 28 nm Altera Stratix V FPGA, with multi-standard fast transceivers (up to 14.4 Gbps) and an increased amount of configurable internal resources and hardware IP cores to support main interconnection standard protocols. Herein we present the APEnet+ V5 architecture, the status of its hardware and its system software design. Both its Linux Device Driver and the low-level libraries have been redeveloped to support the PCIe Gen3 protocol, introducing optimizations and solutions based on hardware/software co-design.