In recent years, the prevalence of rectal cancer has generally increased, and accurate judgment of lymph node metastasis around rectal cancer can often significantly improve the patient's cure probability. In this paper, the convolutional neural network model was used to determine the lymph node metastasis of rectal cancer intelligently by establishing a mathematical model. A CT image recognition and segmentation model based on U-Net was established based on CT images and tumor mask samples of patients with rectal cancer in the arterial phase, and the region of rectal tumor was separated by down-sampling and up-sampling. Then the tumor area has the common characters of the metastasis lymph nodes depth excavation, feature points by using SIFT algorithm automatically identify tumor area, as well as the characteristics of the traditional texture, area, circle skillfully voxel intensity, respectively, into the traditional convolution neural network model and the improved convolution VGG16 network model, neural network model training, prediction, and comparison. Finally, it is concluded that the accuracy of the traditional convolutional neural network model is 75.32%, while the accuracy of the improved VGG16 network model is 90.04%. Therefore, the VGG16 network model has a better prediction effect on lymph node metastasis of rectal cancer. Its advantages of simplicity, efficiency and universality enable the VGG16 network model to better predict lymph node metastasis.