Aiming at the problems of facial expression recognition with the scarce feature samples, a Bayesian network(BN) modeling facial expression recognition method based on small data sets is proposed. Firstly, the geometric feature and HOG feature of facial expression image are extracted. The feature fusion and normalization are used to form the set of action unit label samples. Secondly, the BN structure is constructed according to the expert experience. And the qualitative expert experience is transformed into the constraint set of the BN conditional probability, and the BN model parameter is derived in the light of the convex optimization. Finally, we use junction tree inference algorithm to recognize facial expressions. The experimental results show that the method can obtain more accurate facial expression recognition results than support vector machine, Adaboost or convolution neural network method under the condition of small data sets.