In order to ensure the safe operation of the railway, the detection of rail surface defects on railway tracks is particularly important. However, at present, there are problems such as high manpower investment and dependence on experience for detecting defects, so this paper proposes a rail surface defect detection method and engineering implementation scheme based on an improved YOLOv11m. Aiming at the shortcomings of the existing rail surface defect detection, such as large defect shape difference, few fine textures, easy confusion of defect categories, and difficulty in accurately judging the defect location in a variety of complex scenarios, a dataset containing four types of typical rail surface defects is constructed, and a series of corresponding experimental procedures are designed to improve the YOLOv11m model. The AFEM adaptive feature enhancement module and the A2 prototype semantic enhancement module are introduced into the neck of YOLOv11m to enhance the characterization ability and category discrimination ability of weak boundary regions, respectively. Meanwhile, a C#/WPF-based host computer software system is developed for image management, defect inspection, result visualization, and information interaction. The results showed that the proposed method mAP@0.5 and mAP@0.5:0.95 reached 92.1% and 68.1%, respectively, which was 1.9 and 4.6 percentage points higher than the baseline, and the number of parameters was only increased by about 0.09M, and the performance was better than that of YOLOv8m, YOLOv10m and RT-DETR-L. It can be seen that the proposed method can indeed improve the accuracy of track surface defect detection and has good engineering practicability.