To address the issues of insufficient precision and high missed detection rate in existing object detection models when dealing with small objects of traffic signs, an improved object detection model based on YOLOv8 algorithm was proposed. Firstly, inspired by Residual Network (ResNet) design concept, residual connection mechanism was introduced in Backbone to integrate multi-layer feature information more efficiently by the model, so as to enhance the recognition ability of small objects. Secondly, a reversal of the Path Aggregation Feature Pyramid Network (PAFPN) structure in Neck was proposed as the I-PAFPN (Inverse PAFPN) structure, which enabled the network to capture the key features of the objects. Thirdly, the original three-level detection was extended to four-level detection, which enabled the model to focus on and extract more detailed features of small objects, so as to improve the sensitivity of the model to small objects. Finally, Wise Intersection over Union (WIoU) loss function was introduced to weaken the influence of low-quality samples on the model, so as to improve the model accuracy. Experimental results on TT100K (Tsinghua-Tencent 100K) dataset after data augmentation show that the improved YOLOv8 model has the mAP50 and mAP50:95 increased by 17.1 and 12.5 percentage points, respectively, compared to the original YOLOv8 model, verifying the effectiveness and superiority of the improved YOLOv8 model in the detection of small objects of traffic signs.