In view of the low accuracy of object detection algorithms in small object detection from drone perspective, a new small object detection algorithm named SFM-YOLOv8 was proposed by improving the backbone network and attention mechanism of YOLOv8. Firstly, the SPace-to-Depth Convolution (SPDConv) suitable for low-resolution images and small object detection was integrated into the backbone network to retain discriminative feature information and improve the perception ability to small objects. Secondly, a multi-branch attention named MCA (Multiple Coordinate Attention) was introduced to enhance the spatial and channel information on the feature layer. Then, a convolution FE-C2f fusing FasterNet and Efficient Multi-scale Attention (EMA) was constructed to reduce the computational cost and lightweight the model. Besides, a Minimum Point Distance based Intersection over Union (MPDIoU) loss function was introduced to improve the accuracy of the algorithm. Finally, a small object detection layer was added to the network structure of YOLOv8n to retain more location information and detailed features of small objects. Experimental results show that compared with YOLOv8n, SFM-YOLOv8 achieves a 4.37 percentage point increase in mAP50 (mean Average Precision) with a 5.98% reduction in parameters on VisDrone-DET2019 dataset. Compared to the related mainstream models, SFM-YOLOv8 achieves higher accuracy and meets real-time detection requirements.