Brain Tumor Radiogenomic Classification

WPI CS 539 ML Final Project

Felipe Mejias, Jose Raul Gamez Carias, Shreedhar Kodate, Khatera Alizada, Nicholas Josselyn

Project Goal: Determine methylation status of brain tumors using magnetic resonance imaging (MRI)

Introduction

Automated identification of brain tumors in MRI has many profound clinical applications in surgical treatment planning, image-guided interventions, monitoring tumor growth, and the generation of radiotherapy maps for treatment plans. However, manual identification of brain tumors on MRI is time-consuming, tedious, and subjective and at times it can require invasive follow-up surgeries [1]. Radiologists perform a manual, qualitative approach to identify tumors on MRI, which becomes impractical when dealing with a large number of patients. The automated solution will expedite the process and will decrease the cost of the procedure. The patients, who would otherwise have to wait for longer periods of time because of manual identification procedures by radiologists, could be diagnosed in a timely manner and start their treatment sooner, saving precious time that could extend their lives.This study aims to classify methylation status of brain tumors using magnetic resonance imaging (MRI).

Background

There is an existing work that ensembles multiple architectures trained on the BraTS data set, however, they do so in order to generate high quality segmentations of brain tumors. In our work, we intend to ensemble architectures, across various MRI scan types, for the classification of methylation status of the tumor. The promise of using ensemble techniques is shown as this implementation won the BraTS competition for segmentation in 2017 [2]. Additionally, work has been done using traditional machine learning to extract features and ensemble methods for classification [3]. However, we propose to remove the manual feature extraction step and employ deep learning models to learn the features to extract and ensemble them to build a reliable classifier. There also exists a work that uses a different MRI dataset with deep learning and ensemble methods for assessing MRI image quality. This work tunes a CNN by adapting and adjusting the VGG architecture to fit their data and ensembles models that were trained on one MRI scan type, but across different scan orientations (coronal, sagittal, and axial) [4].

Methodology

Description of the dataset

Link to data set: https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/data

The data consists of the MRI scans of 585 patients from the BraTS 2021 dataset that labels the MGMT status of the patients. The BraTS 2021 dataset is a comprehensive collection of brain tumor MRI scans of patients from various institutions that use different equipment and imaging protocols that represent heterogeneous image quality and diverse clinical practices from different institutions [1,5]. The data is essentially balanced with 47% being labeled ‘0’ (unmethylated) and 53% being labeled ‘1’ (methylated). As part of the BraTS 2021 data set, the participants are provided with binary class labels denoting their tumor methylation status. Each patient has 4 different scan types (T1w, T2w, FLAIR, and T1wCE) each of which captures unique information of the tumors.

image

3D FLAIR Brain MRI

3D FLAIR Brain MRI

3D Convolutional Neural Network

3D Convolutional Neural Networks have been found to be useful for image classification problems. Conv3D is mostly used with 3D image data such as Magnetic Resonance Imaging (MRI) data. These artificial neural networks are made up of convolutional, pooling and fully-connected layers. 3D convolutional neural networks are characterised by three main properties: local connectivity of the hidden units, parameter sharing and the use of pooling operations.

image

3D ResNet50

3D-ResNet50 model is a powerful deep neural network which has achieved amazing performance results in image classification. The architecture of ResNet50 has 4 stages with a sequence of (3, 4, 6, 3) residual blocks with 3 layers each as shown in the diagram. The addition of the identity connection does not introduce extra parameters. Therefore, the computation complexity for simple deep networks and deep residual networks is almost the same.

image

Ensemble Approach

To combine the information that each type of scan could provide to classify the tumors, we decided to use an ensemble approach. To do so, first high quality classifiers were trained individually on each type of MRI scans using the proposed network architectures (3D CNN and 3D ResNet50). After the training process of each model was completed, the model with the best performance on each type of scan was selected and the embeddings of the MRI scans that were used for each model to predict the methylated status were extracted (4 vector), concatenated and used as the input to train different supervised classification models like SVM’s, decision trees, random forest, neural networks and Naive Bayes to see if they were able to generate better predictions by combining the information of the 4 different types of MRI scans.

image

Solution Approach

Below is the diagram that puts together all the steps followed to train the individual models and generate the embedding models. First the MRI scans were extracted and preprocessed. The images of each patient were stacked together to create the volumes with the complete brain, then the values were normalized and the volumes resized to standardize their dimensions given that different patients presented different image dimensions and number of slices so the models could use them as their inputs. After all the volumes were processed, they were split by type and fed into the 3D CNN and 3D ResNet50 models to train them and assess their performance. When the training process was done, for each MRI type the model with the best performance on the validation set was selected and used to generate the embedding needed by the ensemble models to be trained. Finally, the ensemble approach explained before was followed and the performance of the resulting ensembled models were assessed as well and compared.

image

Proposed Experiments

Results

Results - Individual Scans

These were the results using a Simple 3D-CNN:

MRI Scan Type AVG Val Accuracy AVG Val Loss AVG Val AUC AVG Test Accuracy AVG Test Loss AVG Test AUC
T1w 0.54 ± 0.04 0.69 ± 0.002 0.54 ± 0.03 0.52 ± 0.03 0.70 ±.0.01 0.51 ± 0.02
T1wCE 0.54 ± 0.03 0.69 ± 0.01 0.57 ± 0.06 0.52 ± 0.01 0.69 ± 0.001 0.49 ± 0.11
T2w 0.53 ± 0.01 0.69 ± 0.001 0.52 ± 0.04 0.54 ± 0.01 0.69 ± 0.001 0.50 ± 0.04
FLAIR 0.54 ± 0.04 0.69 ± 0.004 0.56 ± 0.05 0.55 ± 0.05 0.69 ± 0.003 0.51 ± 0.02

These were the results using ResNet-50:

MRI Scan Type AVG Val Accuracy AVG Val Loss AVG Val AUC AVG Test Accuracy AVG Test Loss AVG Test AUC
T1w 0.53 ± 0.02 0.69 ± 0.003 0.54 ± 0.06 0.55 ± 0.03 0.69 ± 0.01 0.49 ± 0.12
T1wCE 0.53 ± 0.005 0.69 ± 0.005 0.56 ± 0.02 0.53 ± 0.0 0.69 ± 0.002 0.51 ±0.04
T2w 0.53 ± 0.03 0.69 ± 0.002 0.62 ± 0.02 0.52 ± 0.02 0.69 ± 0.001 0.49 ± 0.04
FLAIR 0.54 ± 0.04 0.68 ± 0.02 0.59 ± 0.06 0.54 ± 0.04 0.69 ± 0.004 0.55 ± 0.04

Results - Ensemble Approach

Taking the best performing 4 models on the validation set we ensembled them using the final 64 unit dense layer output to train a downstream classifier. After ensembling the 4 models as feature extractors, and finetuning on a training set and evaluating on an independent test set, a Naive Bayes classifier performs best along with k nearest neighbors and decision tree being more balanced in its predictions.

   
image image

Results - Ensemble - Standardized

After examining the sparsity of values learned in the final dense layers of each model independently, we create a smaller data set of each MRI type where the sizes across all 3 dimensions are roughly the same across all patients. Previously each dimension could range in values from extremes of 20 frames to 256+ frames and we were setting all these patients to one size leading to drastic upsampling in some cases and drastic downsampling in others. Once we selected a subset of MRI scans for each scan type, we resized all patient scans to the minimum of that subset of data and re-ran our ensemble models using this data to get the results below. We selected images with dimension 1 greater than 192, dimension 2 greater than 256, and dimension 3 greater than 60.

   
image image

Conclusion

References

[1] Baid, U. et al., “The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification”, arXiv:2107.02314 (2021)

[2] Kamnitsas, K. et al., “Ensembles of Multiple Models and Architectures for Robust Brain Tumor Segmentation”, International MICCAI brain lesion workshop (2017)

[3] Gupta, N. et al., “Glioma detection on brain MRIs using texture and morphological features with ensemble learning”, Biomedical Signal Processing and Control (2019)

[4] Sujit, S. et al., “Automated image quality evaluation of structural brain MRI using an ensemble of deep learning networks”, Journal of Magnetic Resonance Imaging (2019)

[5] RSNA-MICCAI Brain Tumor Radiogenomic Classification - Kaggle

[6] https://github.com/hasibzunair/3D-image-classification-tutorial