Complex field environments often cause large scale variation, severe occlusion, illumination fluctuation, and strong background interference in cotton boll detection, resulting in high missed detection rates for small targets, unstable localization, and difficulty in balancing real-time performance with detection accuracy in existing models. To address these problems, a small cotton boll detection algorithm based on an improved YOLOv11n, named CBSOD-YOLO, is proposed. The SPDConv down-sampling module is used to replace the stride-2 convolution in the original network, reducing the loss of texture and edge information during down-sampling and enhancing the feature retention ability for small targets. An SE channel attention mechanism is introduced into the three-scale output branches to enhance cotton boll-related feature responses and suppress complex background noise. An adaptive spatial feature fusion module is introduced in the feature fusion stage to achieve pixel-level weighted fusion of multi-scale features and improve target localization ability in densely occluded scenes. Comparative experiments are conducted on a self-built cotton boll dataset. The results show that, compared with Faster R-CNN, SSD, YOLOv5n, YOLOv6n, YOLOv7-tiny, YOLOv8n, YOLOv9t, YOLOv10n, and YOLOv11n, CBSOD-YOLO achieves better comprehensive detection performance and localization stability in the small cotton boll detection task. Its precision, recall, mAP@0.5, and mAP@0.5:0.95 reach 0.920, 0.910, 0.951, and 0.739, respectively, and mAP@0.5:0.95 is 5.4 percentage points higher than that of the baseline YOLOv11n. The model contains 4.219 M parameters, requires 7.534 GFLOPs, and achieves an inference speed of 84.57 FPS. While maintaining good real-time performance, the proposed algorithm effectively improves the detection accuracy and robustness of small cotton boll targets in complex field environments, providing stable perceptual support for visual recognition, cotton boll localization, and intelligent harvesting operations of cotton-picking robots.