Aiming at the object recognition in complex background, a object recognition method based on feature vocabulary tree and energy minimization, which simultaneously detects object location and classifies object types, is proposed. In order to join the spatial relationships between features, this paper uses the vocabulary trees of single feature and concatenated pairwise feaure. Then an energy function is proposed to combine the matched type probability results of these two vocabulary trees. at last the visual object location and object type are known by minimizing the energy function of the sliding window in test image. Experimental results on standard dataset UIUC and Caltech 101 demonstrate the proposed method has much higher recognition accuracy than other methods.