To address the issues of inductive bias misalignment and feature space collapse existing in current masked autoencoder (MAE)-based sequential MAE-clustering (SMC) methods when processing Gramian angular summation field (GASF) ECG images, the fine-grained classification of arrhythmias is investigated, and an alignment-fusion sequential MAE-clustering (AF-SMC) method is proposed. A radial context alignment module (RCAM) was designed, in which pinwheel-shaped convolution and convolutional additive self-attention are utilized to explicitly reconstruct the geometric topological features of the GASF. Moreover, a shuffled multi-scale fusion (SMSF) module was developed to enrich contextual semantics through deterministic multi-scale pooling, and the network was iteratively optimized utilizing a joint loss function. Through experimental evaluations, an average classification accuracy of 98.32% is achieved on the MIT-BIH arrhythmia database by the AF-SMC method, which is demonstrated to outperform mainstream contrastive learning and generative self-supervised methods. Additionally, the average sensitivity is increased by 10.14% compared to the baseline SMC method. Furthermore, the sensitivities for fusion beats (class F) and supraventricular ectopic beats (class S) are significantly increased to 56.79% and 48.39%, respectively; thus, exceptional performance in long-tailed sample recognition is demonstrated. Consequently, the discriminative capability and robustness for complex arrhythmia morphologies are effectively improved by the proposed AF-SMC method, which is of great value for clinical auxiliary diagnosis.