Abstract: Arrhythmia heartbeat classification is crucial for predicting cardiovascular diseases. Currently, artificial intelligence-based heartbeat classification algorithms are relatively well developed. However, a comprehensive evaluation method to assess the performance of these algorithms has yet to be established. This study introduces a framework for evaluating heartbeat classification algorithms, aimed at assessing model performance. Using the MIT-BIH standard database, we trained and validated eleven different heartbeat algorithms, including traditional machine learning, neural networks, and ensemble learning models. Based on a literature review, we developed an evaluation method comprising 12 direct metrics to assess algorithm performance. The results show that ensemble learning models performed best overall, achieving an accuracy greater than 98%, with LightGBM standing out as the top performer. Its accuracy, AUC, precision, and specificity all exceeded 98.8%, with strong fitting and generalization capabilities and a relatively short training time (6.3 seconds). Furthermore, metrics such as AUC, Precision, Sensitivity, F1 Score, and Accuracy demonstrated strong linear relationships. The evaluation framework in this study objectively assesses arrhythmic heartbeat classification algorithms, enhancing diagnostic accuracy and advancing clinical applications.