Aiming at the problem of large amount of prefix sum computation data in numerical computation and huge time-consuming, this paper proposes a segmented prefix sum parallel algorithm based on the Open Computing Language (OpenCL). First，the parallel analysis of segmented prefix sum algorithms is performed，and a two-level parallel segmented prefix sum algorithm is designed through the hierarchical decomposition and combination of processing tasks. Then the prefix sum parallel algorithm is mapped to the hardware platform of CPU+GPU and the hierarchical parallel processing of prefix sum is implemented by the OpenCL programming. Finally, according to the resource conditions of the Compute Unit (CU), the allocation of local memory is increased in CU. In addition, the bank conflict is reduced by improving the work-items access mode to increase the memory access speed. The experimental results show that compared with the performance of the serial algorithm based on AMD Opteron 2439 SE CPU, parallel algorithm based on OpenMP (Open Multi-Processing) and parallel algorithm based on Compute Unified Device Architecture (CUDA), the prefix sum parallel algorithm obtains 33.51 times, 6.26 times and 2.41 times speedup in the NVIDIA Tesla C2075 computing platform under the OpenCL architecture respectively. The validity and performance portability of the proposed parallel optimization method are verified.