CenterTransFuser: radar point cloud and visual information fusion for 3D object detection

EURASIP Journal on Advances in Signal Processing

Research

Kai Zeng¹  Tao Shen¹  Yan Li¹ 

[1] School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China;

关键词: Cross-transformer;    Depth threshold filtering;    3D detection;    Cross-modal fusion;    Contextual interaction;   

DOI  :  10.1186/s13634-022-00944-6

 received in 2022-02-22, accepted in 2022-11-02,  发布年份 2022

来源: Springer

【 摘 要 】

Sensor fusion is an important component of the perception system in autonomous driving, and the fusion of radar point cloud information and camera visual information can improve the perception capability of autonomous vehicles. However, most of the existing studies ignore the extraction of local neighborhood information and only consider shallow fusion between the two modalities based on the extracted global information, which cannot perform a deep fusion of cross-modal contextual information interaction. Meanwhile, in data preprocessing, the noise in radar data is usually only filtered by the depth information derived from image feature prediction, and such methods affect the accuracy of radar branching to generate regions of interest and cannot effectively filter out irrelevant information of radar points. This paper proposes the CenterTransFuser model that makes full use of millimeter-wave radar point cloud information and visual information to enable cross-modal fusion of the two heterogeneous information. Specifically, a new interaction called cross-transformer is explored, which cooperatively exploits cross-modal cross-multiple attention and joint cross-multiple attention to mine radar and image complementary information. Meanwhile, an adaptive depth thresholding filtering method is designed to reduce the noise of radar modality-independent information projected onto the image. The CenterTransFuser model is evaluated on the challenging nuScenes dataset, and it achieves excellent performance. Particularly, the detection accuracy is significantly improved for pedestrians, motorcycles, and bicycles, showing the superiority and effectiveness of the proposed model.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 64

42004_2022_800_Article_IEq75.gif

Fig. 2

42004_2022_800_Article_IEq73.gif

Fig. 3

Fig. 1

42004_2022_800_Article_IEq69.gif

Fig. 2

Fig. 2

Fig. 1

Fig. 3

Fig. 3

Fig. 1

40798_2022_490_Article_IEq64.gif

40798_2022_490_Article_IEq63.gif

Fig. 3

Fig. 4

40798_2022_490_Article_IEq60.gif

Fig. 4

Fig. 5

40798_2022_490_Article_IEq57.gif

40798_2022_490_Article_IEq2.gif

Fig. 1

Fig. 4

40854_2022_439_Article_IEq93.gif

Fig. 3

Fig. 57

Fig. 1

Fig. 4

Fig. 1

Fig. 3

Fig. 7

Fig. 56

Fig. 4

Fig. 2

Fig. 5

Fig. 1

12888_2022_4500_Article_IEq2.gif

Fig. 1

Fig. 7

Fig. 1

Fig. 2

Fig. 3

Fig. 5

41116_2022_35_Article_IEq659.gif

41116_2022_35_Article_IEq624.gif

Fig. 45

41116_2022_35_Article_IEq598.gif

Fig. 44

Fig. 2

Fig. 42

Fig. 1

Fig. 41

41116_2022_35_Article_IEq566.gif

41116_2022_35_Article_IEq564.gif

41116_2022_35_Article_IEq522.gif

41116_2022_35_Article_IEq520.gif

41116_2022_35_Article_IEq492.gif

Fig. 25

41116_2022_35_Article_IEq489.gif

41116_2022_35_Article_IEq487.gif

Fig. 22

Fig. 21

Fig. 5

Fig. 4

41116_2022_35_Article_IEq478.gif

41116_2022_35_Article_IEq458.gif

41116_2022_35_Article_IEq456.gif

41116_2022_35_Article_IEq417.gif

41116_2022_35_Article_IEq413.gif

41116_2022_35_Article_IEq355.gif

41116_2022_35_Article_IEq345.gif

Fig. 1

41116_2022_35_Article_IEq212.gif

41116_2022_35_Article_IEq208.gif

Fig. 2

41116_2022_35_Article_IEq196.gif