Printed circuit board (PCB) is the core carrier of modern electronic systems. Its quality directly affects the normal operation of electronic systems. To address the problems of small defect targets, irregular defect shapes and easy loss of feature details in PCB defect detection, an improved YOLOv8-based algorithm, named RMSA-YOLO was proposed. Specifically, the original C2f module was replaced by C2f-DySnake to enhance the adaptive modeling ability for irregular defects. Additionally, the BRA attention mechanism was introduced to suppress background interference and strengthen fine-grained defect features. Finally, the original SPPF module was replaced by SPPF_LSKA, and a ReCalibrationFPN with a high-resolution P2 layer was constructed to realize recalibrated multi-scale feature fusion. Experiments were conducted on the PKU-Market-PCB public dataset. The results show that the precision, recall, mAP50 and mAP50-95 of RMSA-YOLO are 96.6%, 91.0%, 95.2% and 50.9%, respectively, which are 0.8%, 3.0%, 1.4% and 1.7% higher than those of YOLOv8n. The proposed algorithm achieves better overall performance than other mainstream models.