In this work, a novel flower pollination algorithm (FPA)-based tracking method is proposed. The state of the candidate is considered to be a flower pollen which is optimized iteratively by FPA. The target is located by the state of the “best” flower pollen. The FPA-based tracking architecture is presented and the parameters’ sensitivity and adjustment of FPA in tracking system are studied experimentally. To verify the tracking ability of the proposed tracker, comparative studies of tracking accuracy and speed of the proposed tracker with particle filter, meanshift and PSO are presented. Comparative results show that the FPA-based tracker outperforms the other three trackers.