Citation

Lim, Yu Jie and Lim, Kian Ming and Chang, Roy Kwang Yang and Lee, Chin Poo (2024) KDViT: COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer. Automatika, 65 (3). pp. 1113-1126. ISSN 0005-1144

Text KDViT_ COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer.pdf - Published Version Restricted to Repository staff only Download (2MB)

Abstract

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods.

Item Type:	Article
Uncontrolled Keywords:	Image classification
Subjects:	T Technology > TA Engineering (General). Civil engineering (General) > TA1501-1820 Applied optics. Photonics
Divisions:	Faculty of Information Science and Technology (FIST)
Depositing User:	Ms Nurul Iqtiani Ahmad
Date Deposited:	30 May 2024 01:32
Last Modified:	30 May 2024 01:32
URII:	http://shdl.mmu.edu.my/id/eprint/12478

Downloads

Edit (login required)