For real-time and effective detection of unsafe driving behaviors such as not wearing seatbelts and using mobile phones under various complex lighting conditions, a deep learning-based unsafe driving behavior detection method under complex lighting conditions was designed. In the method, with YOLOv8n model selected as the foundation, a series of targeted improvements were implemented to enhance the detection performance. Firstly, a P6 scale was added to enable the model to capture the diversity of unsafe driving behaviors under various lighting conditions more comprehensively. Secondly, Spatial Separable Adaptive Convolution (SSAC) module was used to replace the traditional convolution module in the backbone network, thereby achieving lightweight design while improving feature extraction accuracy. Thirdly, Channel Prior Convolutional Attention (CPCA) was introduced to enhance the network’s focus on important features effectively and improve feature expression capability. Finally, the Selective Attention Feature Fusion (SAFF) structure was used to replace the original YOLOv8n neck network, thereby further improving the mode’s comprehensive performance. Experimental results show that compared to the original model, the improved YOLOv8n model increases the overall mean Average Precision (mAP) by 2.17%; under normal lighting conditions, the improvement is 1.76%; in night scenes, the improvement is 1.75%; in backlit environments, the improvement is 2.42%. Meanwhile, the improved YOLOv8n reaches 118 Frames Per Second (FPS) in comparison with other models (such as YOLO11n, RT-DETR(Real-Time DEtection TRansformer)), balancing precision and speed, demonstrating distinct advantages.