Acta Informatica Pragensia 2023, 12(1), 54-70 | DOI: 10.18267/j.aip.1983232

Comparative Analysis of Performance Metrics for Machine Learning Classifiers with a Focus on Alzheimer’s Disease Data

Sivakani Rajayyan ORCID...1, Syed Masood Mohamed Mustafa ORCID...2
1 Department of Computer Science and Engineering, B.S. Abdur Rahman Crescent Institute of Science and Technology, India
2 Department of Computer Applications, B.S. Abdur Rahman Crescent Institute of Science and Technology, India

Alzheimer's disease is a brain memory loss disease. Usually, it will affect persons over 60 years of age. The literature has revealed that it is quite difficult to diagnose the disease, so researchers are trying to predict the disease in the early stage. This paper proposes a framework to classify Alzheimer's patients and to predict the best classification algorithm. The Bestfirst and CfssubsetEval methods are used for feature selection. A multi-class classification is done using machine learning algorithms, namely the naïve Bayes algorithm, the logistic algorithm, the SMO/SMV algorithm and the random forest algorithm. The classification accuracy of the algorithms is 67.68%, 84.58%, 87.42%, and 88.90% respectively. The validation applied is 10-fold cross-validation. Then, a confusion matrix is generated and class-wise performance is analysed to find the best algorithm. The ADNI database is used for the implementation process. To compare the performance of the proposed model, the OASIS dataset is applied to the model with the same algorithms and the accuracy of the algorithms is 98%, 99%, 99% and 100% respectively. Also, the time for the model construction is compared for both datasets. The proposed work is compared with existing studies to check the efficiency of the proposed model.

Keywords: Cognitive normal; Early mild cognitive impairment; Late mild cognitive impairment; Naïve Bayes algorithm; Logistic regression algorithm; SVM algorithm; Random forest algorithm.

Received: June 14, 2022; Revised: October 30, 2022; Accepted: November 2, 2022; Prepublished online: November 3, 2022; Published: April 19, 2023  Show citation

ACS AIP APA ASA Harvard Chicago Chicago Notes IEEE ISO690 MLA NLM Turabian Vancouver
Rajayyan, S., & Mustafa, S.M.M. (2023). Comparative Analysis of Performance Metrics for Machine Learning Classifiers with a Focus on Alzheimer’s Disease Data. Acta Informatica Pragensia, 12(1), 54-70. doi: 10.18267/j.aip.198
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. ADNI. (2017). Alzheimer's Disease Neuroimaging Initiative. https://adni.loni.usc.edu/data-samples/access-data/
    2. Bansal, D., Chhikara, R., Khanna, K., & Gupta, P. (2018). Comparative Analysis of Various Machine Learning Algorithms for Detecting Dementia. Procedia Computer Science, 132, 1497-1502. https://doi.org/10.1016/j.procs.2018.05.102 Go to original source...
    3. Bari Antor, M., Jamil, A. H. M. S., Mamtaz, M., Monirujjaman Khan, M., Aljahdali, S., Kaur, M., Singh, P., & Masud, M. (2021). A Comparative Analysis of Machine Learning Algorithms to Predict Alzheimer's Disease. Journal of Healthcare Engineering, 2021, e9917919. https://doi.org/10.1155/2021/9917919 Go to original source...
    4. Brown, J. B. (2018). Classifiers and their Metrics Quantified. Molecular Informatics, 37(1-2), 1700127. https://doi.org/10.1002/minf.201700127 Go to original source...
    5. Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1). https://doi.org/10.1186/s12864-019-6413-7 Go to original source...
    6. Chicco, D., Tötsch, N., & Jurman, G. (2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining, 14(1). https://doi.org/10.1186/s13040-021-00244-z Go to original source...
    7. Eitel, F., & Ritter, K. (2019). Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classification. arXiv:1909.08856. https://doi.org/10.48550/arXiv.1909.08856 Go to original source...
    8. Gao, F., Yoon, H., Xu, Y., Goradia, D., Luo, J., Wu, T., & Su, Y. (2020). AD-NET: Age-adjust neural network for improved MCI to AD conversion prediction. NeuroImage: Clinical, 27, 102290. https://doi.org/10.1016/j.nicl.2020.102290 Go to original source...
    9. Guo, M., Li, Y., Zheng, W., Huang, K., Zhou, L., Hu, X., Yao, Z., & Hu, B. (2020). A novel conversion prediction method of MCI to AD based on longitudinal dynamic morphological features using ADNI structural MRIs. Journal of Neurology, 267(10), 2983-2997. https://doi.org/10.1007/s00415-020-09890-5 Go to original source...
    10. Gupta, Y., Lama, R. K., & Kwon, G.-R. (2019). Prediction and Classification of Alzheimer's Disease Based on Combined Features From Apolipoprotein-E Genotype, Cerebrospinal Fluid, MR, and FDG-PET Imaging Biomarkers. Frontiers in Computational Neuroscience, 13. https://doi.org/10.3389/fncom.2019.00072 Go to original source...
    11. Han, R., Chen, C. L. P., & Liu, Z. (2020). A Novel Convolutional Variation of Broad Learning System for Alzheimer's Disease Diagnosis by Using MRI Images. IEEE Access, 8, 214646-214657. https://doi.org/10.1109/access.2020.3040340 Go to original source...
    12. Hicks, S. A., Strümke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., & Parasa, S. (2022). On evaluation metrics for medical applications of artificial intelligence. Scientific Reports, 12(1), 5979. https://doi.org/10.1038/s41598-022-09954-8 Go to original source...
    13. Leandrou, S., Petroudi, S., Kyriacou, P. A., Reyes-Aldasoro, C. C., & Pattichis, C. S. (2018). Quantitative MRI Brain Studies in Mild Cognitive Impairment and Alzheimer's Disease: A Methodological Review. IEEE Reviews in Biomedical Engineering, 11, 97-111. https://doi.org/10.1109/rbme.2018.2796598 Go to original source...
    14. Li, F., Tran, L., Thung, K.-H., Ji, S., Shen, D., & Li, J. (2015). A Robust Deep Model for Improved Classification of AD/MCI Patients. IEEE Journal of Biomedical and Health Informatics, 19(5), 1610-1616. https://doi.org/10.1109/jbhi.2015.2429556 Go to original source...
    15. Li, Q., Wu, X., Xu, L., Chen, K., & Yao, L. (2018). Classification of Alzheimer's Disease, Mild Cognitive Impairment, and Cognitively Unimpaired Individuals Using Multi-feature Kernel Discriminant Dictionary Learning. Frontiers in Computational Neuroscience, 11. https://doi.org/10.3389/fncom.2017.00117 Go to original source...
    16. Li, Y., Fang, Y., Zhang, H., & Hu, B. (2019). Self-Weighting Grading Biomarker Based on Graph-Guided Information Propagation for the Prediction of Mild Cognitive Impairment Conversion. IEEE Access, 7, 116632-116642. https://doi.org/10.1109/access.2019.2936415 Go to original source...
    17. Mahyoub, M., Randles, M., Baker, T., & Yang, P. (2018). Comparison Analysis of Machine Learning Algorithms to Rank Alzheimer's Disease Risk Factors by Importance. In 2018 11th International Conference on Developments in eSystems Engineering (DeSE), (pp. 1-11). IEEE. https://doi.org/10.1109/DeSE.2018.00008 Go to original source...
    18. Maqsood, M., Nazir, F., Khan, U., Aadil, F., Jamal, H., Mehmood, I., & Song, O. (2019). Transfer Learning Assisted Classification and Detection of Alzheimer's Disease Stages Using 3D MRI Scans. Sensors, 19(11), 2645. https://doi.org/10.3390/s19112645 Go to original source...
    19. Mammone, N., Ieracitano, C., Adeli, H., Bramanti, A., & Morabito, F. C. (2018). Permutation Jaccard Distance-Based Hierarchical Clustering to Estimate EEG Network Density Modifications in MCI Subjects. IEEE Transactions on Neural Networks and Learning Systems, 29(10), 5122-5135. https://doi.org/10.1109/tnnls.2018.2791644 Go to original source...
    20. Martinez-Murcia, F. J., Ortiz, A., Gorriz, J.-M., Ramirez, J., & Castillo-Barnes, D. (2020). Studying the Manifold Structure of Alzheimer's Disease: A Deep Learning Approach Using Convolutional Autoencoders. IEEE Journal of Biomedical and Health Informatics, 24(1), 17-26. https://doi.org/10.1109/jbhi.2019.2914970 Go to original source...
    21. MCI Screen. (2021). Medical Care Corporation. https://www.mccare.com/education/alzprogression.html.
    22. Minhas, S., Khanum, A., Riaz, F., Khan, S. A., & Alvi, A. (2018). Predicting Progression From Mild Cognitive Impairment to Alzheimer's Disease Using Autoregressive Modelling of Longitudinal and Multimodal Biomarkers. IEEE Journal of Biomedical and Health Informatics, 22(3), 818-825. https://doi.org/10.1109/jbhi.2017.2703918 Go to original source...
    23. Minhas, S., Khanum, A., Riaz, F., Alvi, A., & Khan, S. A. (2017). A Nonparametric Approach for Mild Cognitive Impairment to AD Conversion Prediction: Results on Longitudinal Data. IEEE Journal of Biomedical and Health Informatics, 21(5), 1403-1410. https://doi.org/10.1109/jbhi.2016.2608998 Go to original source...
    24. Moser, R., Pedrycz, W., & Succi, G. (2008). A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In Proceedings of the 30th international conference on Software engineering, (pp. 181-190). ACM. https://doi.org/10.1145/1368088.1368114 Go to original source...
    25. National Institute on Aging. (2021, July 08). NIH National Institute on Aging. https://www.nia.nih.gov/health/alzheimers-disease-fact-sheet#stages
    26. Nozadi, S. H., & Kadoury, S. (2018). Classification of Alzheimer's and MCI Patients from Semantically Parcelled PET Images: A Comparison between AV45 and FDG-PET. International Journal of Biomedical Imaging, 2018, Article ID 1247430. https://doi.org/10.1155/2018/1247430 Go to original source...
    27. OASIS. (2021). The Alzheimer's Association: Brains project OASIS. http://www.oasis-brains.org/#data
    28. Oh, K., Chung, Y.-C., Kim, K. W., Kim, W.-S., & Oh, I.-S. (2019). Classification and Visualization of Alzheimer's Disease using Volumetric Convolutional Neural Network and Transfer Learning. Scientific Reports, 9(1). https://doi.org/10.1038/s41598-019-54548-6 Go to original source...
    29. Segovia, F., Górriz, J. M., Ramírez, J., Salas-Gonzalez, D., Álvarez, I., López, M., & Chaves, R. (2012). A comparative study of feature extraction methods for the diagnosis of Alzheimer's disease using the ADNI database. Neurocomputing, 75(1), 64-71. https://doi.org/10.1016/j.neucom.2011.03.050 Go to original source...
    30. Sivakani, R., & Ansari, G.A. (2020). Machine Learning Framework for Implementing Alzheimer's Disease. In 2020 International Conference on Communication and Signal Processing (ICCSP), (pp. 588-592). IEEE. https://doi.org/10.1109/ICCSP48568.2020.9182220 Go to original source...
    31. Syed, A. H., Khan, T., Hassan, A., Alromema, N. A., Binsawad, M., & Alsayed, A. O. (2020). An Ensemble-Learning Based Application to Predict the Earlier Stages of Alzheimer's Disease (AD). IEEE Access, 8, 222126-222143. https://doi.org/10.1109/access.2020.3043715 Go to original source...
    32. Tong, T., Gao, Q., Guerrero, R., Ledig, C., Chen, L., Rueckert, D., & Initiative, A. D. N. (2017). A Novel Grading Biomarker for the Prediction of Conversion From Mild Cognitive Impairment to Alzheimer's Disease. IEEE Transactions on Biomedical Engineering, 64(1), 155-165. https://doi.org/10.1109/tbme.2016.2549363 Go to original source...
    33. Tuan, T. A., Pham, T. B., Kim, J. Y., & Tavares, J. M. R. S. (2022). Alzheimer's diagnosis using deep learning in segmenting and classifying 3D brain MR images. International Journal of Neuroscience, 132(7), 689-698. https://doi.org/10.1080/00207454.2020.1835900 Go to original source...
    34. Weakley, A., Williams, J. A., Schmitter-Edgecombe, M., & Cook, D. J. (2015). Neuropsychological test selection for cognitive impairment classification: A machine learning approach. Journal of Clinical and Experimental Neuropsychology, 37(9), 899-916. https://doi.org/10.1080/13803395.2015.1067290 Go to original source...
    35. You, Z., Zeng, R., Lan, X., Ren, H., You, Z., Shi, X., Zhao, S., Guo, Y., Jiang, X., & Hu, X. (2020). Alzheimer's Disease Classification With a Cascade Neural Network. Frontiers in Public Health, 8. https://doi.org/10.3389/fpubh.2020.584387 Go to original source...
    36. Zhang, L., Lim, C. Y., Maiti, T., Li, Y., Choi, J., Bozoki, A., & Zhu, D. C. (2018). Analysis of conversion of Alzheimer's disease using a multi-state Markov model. Statistical Methods in Medical Research, 28(9), 2801-2819. https://doi.org/10.1177/0962280218786525 Go to original source...

    This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content