In order to meet the demand of high-speed data transmitting and high-speed image processing, an image processing system was proposed via high speed PCIe interface. Based on NetFPGA SUME, Riffa architecture was used to obtain PCIe interconnection between CPU and FPGA, give full play to the efficiency, adaptability, extendibility, low latency of PCIe interface. The unified interfaces of data accessing and managing were implemented, and CNN handwritten character recognition was efficiently proposed by FPGA. Experimental results show that the transfer rate can reach 2.86 GB/s; the time of CNN handwritten character recognition can be 1.58 ms. This study can promote the capability of transmission and data processing of image processing system.