Seismic damage prediction studies are of great importance for disaster mitigation planning, pre-earthquake risk prediction, and post-earthquake emergency assessment. Traditional methods usually consider the impacts by a limited number of influencing factors based on empirical experience due to the constraint of limited detailed damage data and computing resources. In this paper, based on the random forest algorithm, a classification model that integrates multidimensional information including ground shaking, structural information, and site conditions is designed to predict structural earthquake damage. Based on the detailed damage information of 27 earthquakes assembled by the Earthquake Commission of New Zealand, 142,000 high-quality post-processed building damage records were used for training the prediction model, which included 16 influencing factors such as spectral acceleration, building shape, and number of stories. Four damage state are used as learning labels of the model. The results show that the random forest algorithm has the best performance among six classification models used in the comparison study. After fine tuning the learning curve parameters and cost-sensitive learning, the overall prediction accuracy of the optimal random forest model can reach 75.4% in the test set, and the recall rates for the four damage labels reach 88.2%, 55.0%, 60.7% and 70.8%, respectively, demonstrating the efficacy of the proposed approach based on the large detailed damage data sets. When the first 12 factors are used based on their contribution to model accuracy, the overall classification accuracy of the model can still reach 73.7%. It can be seen that the prediction model trained based on this research framework has good accuracy and applicability, and the framework has a high reference value for future research on the domestic earthquake damage.