Acta Informatica Pragensia 2023, 12(1), 87-103 | DOI: 10.18267/j.aip.2072649

Deep Learning Techniques for Quantification of Tumour Necrosis in Post-neoadjuvant Chemotherapy Osteosarcoma Resection Specimens for Effective Treatment Planning

T. S. Saleena ORCID...1, P. Muhamed Ilyas1, V. M. Kutty Sajna2, A. K. M. Bahalul Haque ORCID...3
1 Sullamussalam Science College, Areekode, India
2 MVR Cancer Centre and Research Institute, Calicut, India
3 School of Engineering Science, LUT University, Finland

Osteosarcoma is a high-grade malignant bone tumour for which neoadjuvant chemotherapy is a vital component of the treatment plan. Chemotherapy brings about the death of tumour tissues, and the rate of their death is an essential factor in deciding on further treatment. The necrosis quantification is now done manually by visualizing tissue sections through the microscope. This is a crude method that can cause significant inter-observer bias. The suggested system is an AI-based therapeutic decision-making tool that can automatically calculate the quantity of such dead tissue present in a tissue specimen. We employ U-Net++ and DeepLabv3+, pre-trained deep learning algorithms for the segmentation purpose. ResNet50 and ResNet101 are used as encoder parts of U-Net++ and DeepLabv3+, respectively. Also, we synthesize a dataset of 555 patches from 37 images captured and manually annotated by experienced pathologists. Dice loss and Intersection over Union (IoU) are used as the performance metrics. The training and testing IoU of U-Net++ are 91.78% and 82.64%, and its loss is 4.4% and 17.77%, respectively. The IoU and loss of DeepLabv3+ are 91.09%, 81.50%, 4.77%, and 17.8%, respectively. The results show that both models perform almost similarly. With the help of this tool, necrosis segmentation can be done more accurately while requiring less work and time. The percentage of segmented regions can be used as the decision-making factor in the further treatment plans.

Keywords: Necrosis; U-Net++; DeepLabv3+; ResNet; Osteosarcoma; Tissue volume calculation.

Received: September 28, 2022; Revised: January 22, 2023; Accepted: January 30, 2023; Prepublished online: March 6, 2023; Published: April 19, 2023  Show citation

ACS AIP APA ASA Harvard Chicago Chicago Notes IEEE ISO690 MLA NLM Turabian Vancouver
Saleena, T.S., Muhamed Ilyas, P., Kutty Sajna, V.M., & Bahalul Haque, A.K.M. (2023). Deep Learning Techniques for Quantification of Tumour Necrosis in Post-neoadjuvant Chemotherapy Osteosarcoma Resection Specimens for Effective Treatment Planning. Acta Informatica Pragensia, 12(1), 87-103. doi: 10.18267/j.aip.207
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. Alawad, A. A. M. (2014). Evaluation of Clinical and Pathological Response after Two Cycles of Neoadjuvant Chemotherapy on Sudanese Patients with Locally Advanced Breast Cancer. Ethiopian Journal of Health Sciences, 24(1), 15. https://doi.org/10.4314/ejhs.v24i1.2 Go to original source...
    2. Badrinarayanan, V., & Cipolla, R. (2017). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481-2495. https://doi.org/10.1109/tpami.2016.2644615 Go to original source...
    3. Boulch, A. (2017). Sharesnet: reducing residual network parameter number by sharing weights. arXiv preprint arXiv:1702.08782. https://doi.org/10.48550/arXiv.1702.08782 Go to original source...
    4. Burcombe, R., Makris, A., Pittam, M., Lowe, J., Emmott, J., & Wong, W. C. (2002). Evaluation of good clinical response to neoadjuvant chemotherapy in primary breast cancer using [18F]-fluorodeoxyglucose positron emission tomography. European Journal of Cancer, 38(3), 375-379. https://doi.org/10.1016/s0959-8049(01)00379-3 Go to original source...
    5. Bychkov, D., Linder, N., Turkki, R., Nordling, S., Kovanen, P. E., Verrill, C., Walliander, M., Lundin, M., Haglund, C., & Lundin, J. (2018). Deep learning based tissue analysis predicts outcome in colorectal cancer. Scientific Reports, 8(1). https://doi.org/10.1038/s41598-018-21758-3 Go to original source...
    6. Chan, J. K. (2014). The Wonderful Colors of the Hematoxylin-Eosin Stain in Diagnostic Surgical Pathology. International Journal of Surgical Pathology, 22(1), 12-32. https://doi.org/10.1177/1066896913517939 Go to original source...
    7. Chandhok, S. (2021) U-Net: Training Image Segmentation Models in PyTorch. https://pyimagesearch.com/2021/11/08/u-net-training-image-segmentation-models-in-pytorch/
    8. Caparica, R., Lambertini, M., Pondé, N., Fumagalli, D., De Azambuja, E., & Piccart, M. (2019). Post-neoadjuvant treatment and the management of residual disease in breast cancer: state of the art and perspectives. Therapeutic Advances in Medical Oncology, 11, 175883591982771. https://doi.org/10.1177/1758835919827714 Go to original source...
    9. Chen, L. C., Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587. https://doi.org/10.48550/arXiv.1706.05587 Go to original source...
    10. Chen, L., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018a). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In Computer Vision - ECCV 2018, (pp. 833-851). Springer. https://doi.org/10.1007/978-3-030-01234-2_49 Go to original source...
    11. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018b). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834-848. https://doi.org/10.1109/tpami.2017.2699184 Go to original source...
    12. Davis, A., Bell, R., & Goodwin, P. J. (1994). Prognostic factors in osteosarcoma: A critical review. Journal of Clinical Oncology, 12(2), 423-431. https://doi.org/10.1200/jco.1994.12.2.423 Go to original source...
    13. Deng, J., Dong, W., Socher, R., Li, L., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/cvpr.2009.5206848 Go to original source...
    14. Elies, A., Rivière, S., Pouget, N., Becette, V., Dubot, C., Donnadieu, A., Rouzier, R., & Bonneau, C. (2018). The role of neoadjuvant chemotherapy in ovarian cancer. Expert Review of Anticancer Therapy, 18(6), 555-566. https://doi.org/10.1080/14737140.2018.1458614 Go to original source...
    15. ESMO. (2014). Bone sarcomas: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Annals of Oncology, 25, iii113-iii123. https://doi.org/10.1093/annonc/mdu256 Go to original source...
    16. Estienne, T., Vakalopoulou, M., Christodoulidis, S., Battistela, E., Lerousseau, M., Carré, A., Klausner, G., Sun, R., Robert, C., Mougiakakou, S., Paragios, N., & Deutsch, E. (2019). U-ReSNet: Ultimate Coupling of Registration and Segmentation with Deep Nets. In International conference on medical image computing and computer-assisted intervention, Lecture Notes in Computer Science, (pp. 310-319). Springer. https://doi.org/10.1007/978-3-030-32248-9_35 Go to original source...
    17. Facchin, C., Certain, A., Yoganathan, T., Delacroix, C., Garcia, A., Gaillard, F., Lenoir, O., Tharaux, P., Tavitian, B., & Balvay, D. (2022). FIBER-ML, an Open-Source Supervised Machine Learning Tool for Quantification of Fibrosis in Tissue Sections. American Journal of Pathology, 192(5), 783-793. https://doi.org/10.1016/j.ajpath.2022.01.013 Go to original source...
    18. Ferlay, J., Soerjomataram, I., Dikshit, R., Eser, S., Mathers, C., Rebelo, M. S., Parkin, D. M., Forman, D., & Bray, F. (2015). Cancer incidence and mortality worldwide: Sources, methods and major patterns in GLOBOCAN 2012. International Journal of Cancer, 136(5), E359-E386. https://doi.org/10.1002/ijc.29210 Go to original source...
    19. Gertych, A., Ing, N., Ma, Z., Fuchs, T. J., Salman, S., Mohanty, S. K., Bhele, S., Velásquez-Vacca, A., Amin, M. B., & Knudsen, K. E. (2015). Machine learning approaches to analyze histological images of tissues from radical prostatectomies. Computerized Medical Imaging and Graphics, 46, 197-208. https://doi.org/10.1016/j.compmedimag.2015.08.002 Go to original source...
    20. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/cvpr.2014.81 Go to original source...
    21. Guo, Y., Liu, Y., Georgiou, T., & Lew, M. S. (2018). A review of semantic segmentation using deep neural networks. International Journal of Multimedia Information Retrieval, 7(2), 87-93. https://doi.org/10.1007/s13735-017-0141-z Go to original source...
    22. He, K., Zhang, X., Ren, S., & Sun, J. (2016a). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). IEEE. https://doi.org/10.1109/CVPR.2016.90 Go to original source...
    23. He, K., Zhang, X., Ren, S., & Sun, J. (2016b). Identity mappings in deep residual networks. In European conference on computer vision (pp. 630-645). Springer, Cham. https://doi.org/10.1007/978-3-319-46493-0_38 Go to original source...
    24. Ikeda, T., Jinno, H., Matsui, A., Masamura, S., & Kitajima, M. (2002). The role of neoadjuvant chemotherapy for breast cancer treatment. Breast Cancer, 9(1), 8-14. https://doi.org/10.1007/bf02967540 Go to original source...
    25. Kann, B. H., Thompson, R., Thomas, C. R., Dicker, A., & Aneja, S. (2019). Artificial Intelligence in Oncology: Current Applications and Future Directions. Oncology, 30(2), 46-53. https://www.cancernetwork.com/view/artificial-intelligence-oncology-current-applications-and-future-directions
    26. Komura, D., & Ishikawa, S. (2017). Machine Learning Methods for Histopathological Image Analysis. Computational and Structural Biotechnology Journal, 16, 34-42. https://doi.org/10.1016/j.csbj.2018.01.001 Go to original source...
    27. Lee, C. Y., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015). Deeply-supervised nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, (pp. 562-570). PMLR. https://proceedings.mlr.press/v38/lee15a.html
    28. Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2018). Visualizing the loss landscape of neural nets. In Advances in neural information processing systems, vol. 31, NeurIPS Proceedings. https://proceedings.neurips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf
    29. Liu, D., & Yu, J. Z. (2009). Otsu Method and K-means. International Conference Hybrid Intelligent Systems. In 2009 Ninth International Conference on Hybrid Intelligent Systems. IEEE. https://doi.org/10.1109/his.2009.74 Go to original source...
    30. Luchini, C., Pea, A., & Scarpa, A. (2022). Artificial intelligence in oncology: current applications and future perspectives. British Journal of Cancer, 126(1), 4-9. https://doi.org/10.1038/s41416-021-01633-1 Go to original source...
    31. Maknuna, L., Kim, H. S., Lee, Y., Choi, Y., Kim, H. J., Yi, M., & Kang, H. (2022). Automated Structural Analysis and Quantitative Characterization of Scar Tissue Using Machine Learning. Diagnostics, 12(2), 534. https://doi.org/10.3390/diagnostics12020534 Go to original source...
    32. Picci, P., Sangiorgi, L., Rougraff, B. T., Neff, J. R., Casadei, R., & Campanacci, M. (1994). Relationship of chemotherapy-induced necrosis and surgical margins to local recurrence in osteosarcoma. Journal of Clinical Oncology, 12(12), 2699-2705. https://doi.org/10.1200/jco.1994.12.12.2699 Go to original source...
    33. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer. https://doi.org/10.1007/978-3-319-24574-4_28 Go to original source...
    34. Ruspini, E. H., Bezdek, J. C., & Keller, J. M. (2019). Fuzzy Clustering: A Historical Perspective. IEEE Computational Intelligence Magazine, 14(1), 45-55. https://doi.org/10.1109/mci.2018.2881643 Go to original source...
    35. Sabeerali, K. P., Saleena, T. S., Muhamed Ilyas, P., & Mohan, N. (2022). AI-Powered Semantic Segmentation and Fluid Volume Calculation of Lung CT Images in COVID-19 Patients. In Emergent Converging Technologies and Biomedical Systems (pp. 93-101). Springer. https://doi.org/10.1007/978-981-16-8774-7_9 Go to original source...
    36. Shukla, S., & Naganna, S. (2014). A review on K-means data clustering approach. International Journal of Information and Computation Technology, 4(17), 1847-1860.
    37. Strobel, H. A., Schultz, A., Moss, S. J., Eli, R., & Hoying, J. B. (2021). Quantifying Vascular Density in Tissue Engineered Constructs Using Machine Learning. Frontiers in Physiology, 12. https://doi.org/10.3389/fphys.2021.650714 Go to original source...
    38. Tiu, E. (2019). Metrics to Evaluate your Semantic Segmentation Model. https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2
    39. van Beers, F. (2018). Using intersection over union loss to improve binary image segmentation. University of Groningen. Go to original source...
    40. Wang, Y., Wang, G., Wei, L., Huang, L., Wang, J., Wang, S., Li, X., Shen, D., Bao, D., & Gao, J. (2011). Neoadjuvant chemotherapy for locally advanced cervical cancer reduces surgical risks and lymph-vascular space involvement. Chinese Journal of Cancer, 30(9), 645-654. https://doi.org/10.5732/cjc.011.10050 Go to original source...
    41. Yan, Z., Zhang, H., Jia, Y., Breuel, T., & Yu, Y. (2016). Combining the best of convolutional layers and recurrent layers: A hybrid network for semantic segmentation. arXiv preprint arXiv:1603.04871. https://doi.org/10.48550/arXiv.1603.04871 Go to original source...
    42. Zhang, Q., Cui, Z., Niu, X., Geng, S., & Qiao, Y. (2017). Image Segmentation with Pyramid Dilated Convolution Based on ResNet and U-Net. In ICONIP 2017: Neural Information Processing, (pp. 364-372). Springer. https://doi.org/10.1007/978-3-319-70096-0_38 Go to original source...
    43. Zhang, R., Du, L., Xiao, Q., & Liu, J. (2020). Comparison of Backbones for Semantic Segmentation Network. Journal of Physics: Conference Series, 1544(1), 012196. https://doi.org/10.1088/1742-6596/1544/1/012196 Go to original source...
    44. Zhou, Z., Siddiquee, M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. In DLMIA 2018, ML-CDS 2018: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, (pp. 3-11). Springer. https://doi.org/10.1007/978-3-030-00889-5_1 Go to original source...

    This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content