Multi-view CNN with MLP for diagnosing tuberculosis patients using CT scans and clinically relevant metadata

Abstract

We propose a hybrid approach of multi-view convolutional neural networks with Multi-Layer Perceptron to generate an automatic medical CT report and evaluation of the severity stage of Tuberculosis patients, trained and evaluated on 335 chest 3D CT images and available metadata provided by Im-ageCLEF2019 organizers for the participants of tuberculosis computation track. Transfer learning and data augmentation techniques were applied to avoid over fitting and enhance performance of the model. Our multi-view CNN approach comprises the decomposition of the 3D CT image into 2D axial, coronal and sagittal slices and converting them to PNG format as preliminary to training. At the first stage, coronal and sagittal slices were used to train the CNN classifier using pre-trained AlexNet. In the second stage, MLPs were trained using features extracted during stage one alongside with the provided metadata. Our results ranked 6th and 4th ,with an AUC of 0.763 in predicting whether the severity stage is High or Low, and mean AUC of 0.707 in detecting whether left and right lungs are affected or not, detecting the absence or presence of calcifications, caverns, pleurisy and lung capacity decrease, respectively. © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).