嵌入式异构智能计算系统的PCIe总线传输带宽优化

doi:10.11772/j.issn.1001-9081.2024091299

《计算机应用》唯一官方网站 ›› 2025, Vol. 45 ›› Issue (9): 2913-2918.DOI: 10.11772/j.issn.1001-9081.2024091299

• 先进计算 • 上一篇

嵌入式异构智能计算系统的PCIe总线传输带宽优化

喻绪邦¹, 吴济文²(), 夏宏¹, 莫昊¹, 赵二虎²

^1.华北电力大学控制与计算机工程学院，北京 102206
^2.中国科学院计算技术研究所，北京 100190

收稿日期:2024-09-12 修回日期:2024-12-23 接受日期:2025-01-07 发布日期:2025-03-19 出版日期:2025-09-10
通讯作者: 吴济文
作者简介:喻绪邦（1998—），男，江西南昌人，硕士，主要研究方向：计算机系统架构、嵌入式系统
夏宏（1965—），男，北京人，副教授，博士，CCF会员，主要研究方向：计算机系统架构、嵌入式系统
莫昊（2000—），男，湖南长沙人，硕士研究生，主要研究方向：嵌入式系统、深度学习
赵二虎（1985—），男，河北邢台人，高级工程师，博士，CCF会员，主要研究方向：嵌入式智能计算系统。
基金资助:
中央高校基本科研业务费专项资金资助项目(2023JC007);中国科学院技术支撑人才项目;北航杭州创新研究院钱江实验室开放基金资助项目(2020-Y8-A-023)

PCIe bus transmission bandwidth optimization in embedded heterogeneous intelligent computing system

Xubang YU¹, Jiwen WU²(), Hong XIA¹, Hao MO¹, Erhu ZHAO²

^1.School of Control and Computer Engineering，North China Electric Power University，Beijing 102206，China
^2.Institute of Computing Technology，Chinese Academy of Sciences，Beijing 100190，China

Received:2024-09-12 Revised:2024-12-23 Accepted:2025-01-07 Online:2025-03-19 Published:2025-09-10
Contact: Jiwen WU
About author:YU Xubang， born in 1998， M. S. His research interests include computer system architecture， embedded system.
XIA Hong， born in 1965， Ph. D.， associate professor. His research interests include computer system architecture， embedded system.
MO Hao， born in 2000， M. S. candidate. His research interests include embedded system， deep learning.
ZHAO Erhu， born in 1985， Ph. D.， senior engineer. His research interests include embedded intelligent computing system.
Supported by:
Fundamental Research Funds for the Central Universities(2023JC007);Open Fund of Hangzhou Innovation Institute of Beihang University Qianjiang Laboratory(2020-Y8-A-023);Technical Support Talent Program of Chinese Academy of Sciences

摘要/Abstract

摘要：

近年来随着人工智能（AI）技术的发展，深度学习算法和专用AI处理器芯片已日益广泛地应用于边缘端和物端数据信号处理系统，如何在赋能系统高智能计算能力的同时，实现异构处理器之间高带宽、低延时的数据传输，已成为亟须解决的核心技术之一。因此，设计一种集成寒武纪MLU220芯片、国产飞腾FT2000/4型CPU和Xilinx XC7K325T现场可编程门阵列（FPGA）的嵌入式异构智能计算系统，系统异构处理器之间采用PCIe（Peripheral Component Interconnect express）总线实现高速互联与数据传输。此外，提出一种Linux下的PCIe总线Scatter-Gather DMA（Direct Memory Access）传输优化技术，通过基于双缓冲的预取技术和基于工作队列的中断处理，有效提高CPU与FPGA异构处理器之间的PCIe总线数据传输带宽。系统图像传输测试结果表明，CPU与FPGA异构处理器之间的PCIe2.0 X4总线传输10张2 048×1 024灰度图像数据时，所提系统在DMA双通道上的读写速率分别达到了1 610 MB/s和1 655 MB/s，为PCIe2.0 X4总线理论带宽值的81%和83%，验证了所设计系统的实用性和先进性。

关键词: PCIe总线, 异构计算系统, Scatter-Gather DMA, DMA多通道, 图像传输

Abstract:

In recent years， with development of Artificial Intelligence （AI） technology， deep learning algorithms and specialized AI processor chips are applied to edge and device data signal processing systems more and more widely. A key technical challenge is how to achieve high-bandwidth and low-latency data transmission between heterogeneous processors while enabling the system with high-level intelligent computing capabilities. Therefore， an embedded heterogeneous intelligent computing system was designed， which integrated Cambricon MLU220 chip， domestic Feiteng FT2000/4 CPU， and Xilinx XC7K325T Field Programmable Gate Array （FPGA）. High-speed interconnection and data transmission between the system’s heterogeneous processors were realized through the PCIe （Peripheral Component Interconnect express） bus. In addition， a PCIe bus Scatter-Gather DMA （Direct Memory Access） transmission optimization technique under Linux was proposed， which improved the PCIe bus data transmission bandwidth between the CPU and FPGA heterogeneous processors effectively through a prefetch technique based on double buffering and an interrupt handling based on work queues. Test results of system’s image transmission show that when 10 grayscale images of 2 048×1 024 size are transferred between the CPU and FPGA heterogeneous processors via a PCIe2.0 X4 bus， the proposed system achieves read/write speeds of 1 610 MB/s and 1 655 MB/s in dual-channel DMA mode， respectively， achieving 81% and 83% of the theoretical PCIe2.0 X4 bus bandwidth， respectively. These results verify practicality and advancement of the designed system.

Key words: Peripheral Component Interconnect express (PCIe) bus, heterogeneous computing system, Scatter-Gather DMA (Direct Memory Access), DMA multi-channel, image transmission

中图分类号:

TP368.1

喻绪邦, 吴济文, 夏宏, 莫昊, 赵二虎. 嵌入式异构智能计算系统的PCIe总线传输带宽优化[J]. 计算机应用, 2025, 45(9): 2913-2918.

Xubang YU, Jiwen WU, Hong XIA, Hao MO, Erhu ZHAO. PCIe bus transmission bandwidth optimization in embedded heterogeneous intelligent computing system[J]. Journal of Computer Applications, 2025, 45(9): 2913-2918.

图/表 11

图1 本文系统的整体结构

Fig. 1 Overall structure of proposed system

图2 FPGA的结构

Fig. 2 Structure of FPGA

图3 PCIe总线的结构

Fig. 3 Structure of PCIe bus

图4 SG DMA传输流程

Fig. 4 Transfer flow of SG DMA

图5 基于描述符预取的SG DMA传输流程

Fig. 5 SG DMA transfer flow based on descriptor prefetching

图6 DMA多通道的实现原理

Fig. 6 Principle of DMA multi-channel implementation

图7 基于工作队列的中断处理

Fig. 7 Interrupt handling based on work queues

表1 FPGA的资源占用

Tab. 1 FPGA resource occupancy

FPGA资源	消耗资源数	资源占用率/%
Slice Register	28 645	9.1
Slice LUTs	25 477	5.2
Block RAM	22	2.9

表2 最佳缓冲区大小的实验结果

Tab. 2 Experiment results of optimal buffer size

缓冲区大小/KB	DMA读速率/（MB·s^-1）	DMA写速率/（MB·s^-1）
128	800	778
256	945	900
512	1 200	1 144
1 024	1 345	1 400
2 048	1 447	1 490
4 096	1 500	1 550
8 192	1 502	1 546
16 384	1 487	1 533

表3 单通道DMA读写速率实验结果

Tab. 3 Single-channel DMA read/write rate experiment results

数据传输大小/KB	DMA读速率/（MB·s^-1）	DMA写速率/（MB·s^-1）
1	108	120
4	158	183
16	247	278
64	395	446
256	600	652
1 024	844	896
4 096	1 197	1 288
16 384	1 440	1 520
65 536	1 510	1 560

表4 PCIe总线利用率的对比 (%)

Tab. 4 Comparison of PCIe bus utilization

方法	读操作	写操作
文献［9］方法	66	73
文献［18］方法	64	64
Xilinx官方方法	62	73
本文方法	81	83

参考文献 18

[1]	PACINI F， PACINI T， LAI G， et al. Design and evaluation of CPU-， GPU-， and FPGA-based deployment of a CNN for motor imagery classification in brain-computer interfaces ［J］. Electronics， 2024， 13（9）： No.1646.
[2]	LIU X， XU W， WANG Q， et al. Energy-efficient computing acceleration of unmanned aerial vehicles based on a CPU/FPGA/NPU heterogeneous system ［J］. IEEE Internet of Things Journal， 2024， 11（16）： 27126-27138.
[3]	LIN Y C， ZHANG B， PRASANNA V K. HitGNN： high-throughput GNN training framework on CPU+ Multi-FPGA heterogeneous platform ［J］. IEEE Transactions on Parallel and Distributed Systems， 2024， 35（5）： 707-719.
[4]	ROSENFELD V， BREß S， MARKL V. Query processing on heterogeneous CPU/GPU systems ［J］. ACM Computing Surveys， 2023， 55（1）： No.11.
[5]	GUO Z， HUANG T W， LIN Y. Accelerating static timing analysis using CPU-GPU heterogeneous parallelism ［J］. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems， 2023， 42（12）： 4973-4984.
[6]	TAN G， SHUI C， WANG Y， et al. Optimizing the LINPACK algorithm for large-scale PCIe-based CPU-GPU heterogeneous systems ［J］. IEEE Transactions on Parallel and Distributed Systems， 2021， 32（9）： 2367-2380.
[7]	BANDARA S， SANAULLAH A， TAHIR Z， et al. Performance evaluation of VirtIO device drivers for Host-FPGA PCIe communication ［C］// Proceedings of the 2024 IEEE International Parallel and Distributed Processing Symposium Workshops. Piscataway： IEEE， 2024： 169-176.
[8]	XU Z， LIU H， LIU Y. PCIe P2P communication for the high performance heterogeneous computing system ［C］// Proceedings of the 2021 IEEE International Conference on Artificial Intelligence and Computer Applications. Piscataway： IEEE， 2021： 284-288.
[9]	GU X， SHAO L， BAI N， et al. A co-optimization of software and hardware for PCIe-based small packet DMA transfer ［J］. IEEE Embedded Systems Letters， 2025， 17（1）： 6-9.
[10]	YANG Y， YAN C. PCIe device drivers in Linux： design and implementation of a high-speed PCI Express bridge ［C］// Proceedings of the 3rd International Conference on Big Data， Artificial Intelligence and Internet of Things Engineering. Piscataway： IEEE， 2022： 430-437.
[11]	WANG Y， WANG C， ZHAO X， et al. A PCIe-based hardware acceleration architecture of the communication protocol stack ［C］// Proceedings of the 7th International Conference on Computer and Communications. Piscataway： IEEE， 2021： 192-198.
[12]	LIU B. Research and implementation of XDMA high speed data transmission IP core based on PCI express and FPGA ［C］// Proceedings of the IEEE 1st International Conference on Civil Aviation Safety and Information Technology. Piscataway： IEEE， 2019： 408-411.
[13]	KASAI S， OSANA Y. A driver-based approach for DMA transfer between FPGA-GPU ［C］// Proceedings of the 10th International Symposium on Computing and Networking Workshops. Piscataway： IEEE， 2022： 103-108.
[14]	DU L， ZHANG Z， TONG J， et al. Research on XDMA high-speed data transmission architecture based on PCIe ［C］// Proceedings of the IEEE 6th International Conference on Computer and Communications. Piscataway： IEEE， 2020： 1783-1787.
[15]	JIA L， YU X， WEI R， et al. Design of PCIe-Gigabit ethernet high-speed data interaction system based on FPGA ［C］// Proceedings of the 2021 Asia-Pacific Conference on Communications Technology and Computer Science. Piscataway： IEEE， 2021： 138-142.
[16]	甄国涌，苏慧思，陈建军，等. 基于FPGA的PCIe接口的数据传输设计与实现［J］. 国外电子测量技术， 2021， 40（12）：72-76.
	ZHEN G Y， SU H S， CHEN J J， et al. Design and implementation of data transmission based on FPGA PCIe interface［J］. Foreign Electronic Measurement Technology， 2021， 40（12）：72-76.
[17]	ZHUO Q. High speed data transmission model for optical fiber bus［C］// Proceedings of the 3rd International Conference on Electronic Information Engineering and Computer. Piscataway： IEEE， 2023： 537-541.
[18]	ZHANG Z， LI Z. ARM and FPGA heterogeneous accelerated processing system based on HLS and PCIe ［C］// Proceedings of the 4th International Conference on Information and Computer Technologies. Piscataway： IEEE， 2021： 300-304.

嵌入式异构智能计算系统的PCIe总线传输带宽优化

PCIe bus transmission bandwidth optimization in embedded heterogeneous intelligent computing system

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 18

相关文章 7

编辑推荐

Metrics

[1]	敬超, 全育涛, 陈艳. 基于多层感知机-注意力模型的功耗预测算法[J]. 《计算机应用》唯一官方网站, 2025, 45(8): 2646-2655.
[2]	赵欢江文李学辉. 异构系统中的综合性启发式任务调度算法[J]. 计算机应用, 2010, 30(05): 1316-1320.
[3]	杨宇何小海吕瑞. 基于H.264标准的码率控制算法[J]. 计算机应用, 2009, 29(3): 675-677.
[4]	周大伟耿金玲郑继明. 一种基于人眼视觉特性的ROI渐进图像传输算法[J]. 计算机应用, 2007, 27(7): 1654-1656.
[5]	王秀丽宁正元 . 基于动态适应度的独立任务调度算法[J]. 计算机应用, 2006, 26(12): 3001-3003.
[6]	蔡灿辉，陈婧，丁润涛. 基于分裂系数的多描述编码[J]. 计算机应用, 2005, 25(05): 1066-1068.
[7]	亓中瑞，高振明. 正交频分复用系统中基于人眼视觉系统的自适应图像传输[J]. 计算机应用, 2005, 25(02): 335-337.