Context Aware Deep Learning for Brain Tumor Segmentation, Subtype Classification, and Survival Prediction Using Radiology Images

Gliomas are the most common primary brain malignancies with heterogeneous regions, variable aggressiveness, and prognosis. Incidence in the U.S. is ~23 per 100,000, with 5- and 10-year relative survival rates of 35.0% and 29.3%; median survival for glioblastoma is ~12–15 months. Accurate subtype diagnosis and grading guide treatment and prognosis. While WHO 2016 integrates phenotype and genotype (e.g., IDH mutation, 1p/19q codeletion), structural MRI remains central for identifying, localizing, and characterizing tumors. Multimodal MRI (T1, T1ce, T2, FLAIR) captures complementary tumor phenotypes, but segmentation and classification remain challenging due to similar intensity distributions across tissues (e.g., NC vs ED). Prior works typically treat segmentation, classification, and survival prediction independently. This study proposes an integrated deep learning framework spanning tumor segmentation, subtype classification, and overall survival prediction, leveraging shared representations across tasks to improve performance and clinical utility.

Traditional machine learning approaches (e.g., SVM, KNN, Random Forest) have been widely used for brain tumor analysis but rely on hand-crafted features, limiting adaptability. Deep learning alleviates hand-crafted feature dependence by learning task-optimal representations and has advanced computer vision, medical image segmentation, and speech recognition. Architectures such as ResNet, UNet, and variants (e.g., UNet-VAE) have been applied to brain tumor segmentation with strong results. For tumor classification, both structural MRI and pathology imaging have been explored, and for survival prediction, traditional machine learning with linear regression has been common. The paper builds upon context encoding networks to address class imbalance and global context capture, extending them to volumetric MRI for joint tasks (segmentation, classification, survival).

Overview: The framework processes four MRI modalities (T1, T1ce, T2, FLAIR). After preprocessing (co-registration, skull stripping, noise reduction, z-score intensity normalization within brain), a context-aware 3D CNN (CANet) performs tumor segmentation. The segmentation outputs (probability maps of subregions ET, WT, TC) feed a 3D CNN for tumor subtype classification. For survival prediction, high-dimensional features from the CANet encoder are combined with patient age, followed by LASSO feature selection and linear regression. Data: Segmentation and survival utilize BraTS 2019 (training 335 cases: 259 HGG, 76 LGG; validation 125; testing 166). Additional cases were incorporated to expand testing: +86 (BraTS 2020 and TCIA) for segmentation, totaling 252 test cases. Tumor classification uses CPM-RadPath 2019 (training 221, validation 35, testing 73) with +69 cases (from BraTS 2019) added to testing to total 142. Survival prediction uses BraTS 2019 (training 210, validation 29, testing 107) with +17 cases (BraTS 2020) to total 124 testing cases. Ground truths are available only for training; organizers perform blind validation/testing. Preprocessing and augmentation: Images are center-cropped to 160×192×128 due to GPU limits, ensuring tumor inclusion. Augmentation includes rotations (90°, 180°, 270°) and scaling (0.9–1.1). CANet architecture: A 3D encoder–decoder UNet-like backbone with an integrated context encoding module. The context module computes class-related scaling factors to capture global context and mitigate class imbalance via semantic loss regularization L_se. Total loss L = L_dice + L_se. The encoder extracts high-dimensional features; the context module refines features and produces L_se; the decoder reconstructs tumor subregion probability maps (ET, WT, TC). The best CANet model from segmentation is reused as a feature extractor for classification and survival. Tumor segmentation: CANet inputs the 4-channel (T1, T1ce, T2, FLAIR) volume of size 4×160×192×128. Training uses Adam, initial LR 1e-4 with decay to 0, 300 epochs, Leaky-ReLU activations, dropout 0.2. Tumor subtype classification: The CANet output probability maps (ET, WT, TC) are fed into a 3D CNN classifier with five convolution/pooling layers, two fully connected layers, and a softmax output over three classes: (A) lower grade astrocytoma, IDH-mutant; (O) oligodendroglioma, IDH-mutant, 1p/19q codeleted; (G) glioblastoma/diffuse astrocytic glioma, IDH-wildtype. Training uses similar hyperparameters with 2000 epochs. Testing is executed by the challenge organizer via Docker containers. Overall survival prediction: High-dimensional features from the CANet encoder (3D-CNN features) plus patient age are subjected to LASSO for feature selection, followed by linear regression to predict survival days. Training uses similar hyperparameters with 1000 epochs. Also implemented: a conventional baseline using radiomic/texture (e.g., GLCM) and intensity features with LASSO and linear regression for comparison. Evaluation metrics: Segmentation uses Dice similarity coefficient (DSC) and Hausdorff distance at 95th percentile (HD95). Classification uses accuracy and related challenge metrics (Dice/Kappa/Balance_acc/F1_micro). Survival uses accuracy, mean squared error (MSE), median error (medianSE), standard deviation of error (stdSE), and Spearman correlation.

Segmentation (BraTS): - Validation (125 cases): CANet achieved DSC ET 0.773, WT 0.905, TC 0.815; HD95 ET 3.220 mm, WT 4.916 mm, TC 6.809 mm, outperforming ResNet, UNet, and UNet-VAE in most metrics. - Testing (252 cases incl. BraTS 2019/2020 and TCIA): CANet achieved DSC ET 0.821, WT 0.895, TC 0.835; HD95 ET 3.319 mm, WT 4.897 mm, TC 6.712 mm. Compared to validation, ET and TC DSC improved by ~5% and ~2%, respectively; WT decreased by ~1%. HD95 values improved (lower) vs validation. - Relative improvements: CANet offered ~1–4% DSC gains and 0.2–2.0 mm HD95 reductions over baselines depending on subregion. Classification (CPM-RadPath): - Validation (35 cases): Dice 0.749, Kappa 0.715, Balanced accuracy 0.749, F1_micro 0.829. - Testing (142 cases with added data): Dice 0.639, Kappa 0.442, Balanced accuracy 0.639, F1_micro 0.657. The approach ranked 2nd in the challenge testing phase. Survival prediction (BraTS): - Validation (29 cases): Accuracy 0.586, MSE 79,146, medianSE 24,362, stdSE 113,801, SpearmanR 0.502. - Testing (124 cases): Accuracy 0.484, MSE 334,492. - Compared to a conventional machine learning baseline on validation: Proposed method achieved higher accuracy (0.586 vs 0.483) and lower MSE (79,146 vs 128,594), with substantially higher SpearmanR (0.502 vs 0.044). Additional analyses: - Gender/age effects on survival (training data): No statistically significant effect by ANOVA (p-values: gender 0.1636; age 0.101).

The integrated context-aware framework leverages shared representations across segmentation, classification, and survival prediction tasks. By incorporating a context encoding module and semantic loss, the segmentation model captures global class context and mitigates class imbalance, improving both overlap (DSC) and boundary (HD95) metrics versus established architectures. Using CANet outputs as inputs for classification and its encoder features for survival demonstrates the utility of shared features across tasks, yielding a top-2 ranking in CPM-RadPath classification and competitive survival prediction performance that surpasses a conventional radiomics baseline. These findings support the hypothesis that joint, context-aware deep representations derived from multimodal MRI can enhance multiple downstream brain tumor analysis tasks. The study also highlights practical challenges affecting performance and generalizability: imaging quality and preprocessing (e.g., intensity normalization), tumor heterogeneity, and data imbalance across tissues and outcome categories. The authors mitigate these through subregion-based analysis, data augmentation, and inclusion of additional external cases for testing, demonstrating robustness across datasets. Statistical analysis suggests no significant gender or age effects on OS within the limited sample, underscoring the need for larger cohorts to validate such associations.

This work presents an integrated, context-aware deep learning framework (CANet) for brain tumor segmentation, subtype classification, and overall survival prediction using multimodal MRI. Key contributions include: (1) a context encoding module with semantic loss for robust 3D tumor segmentation; (2) a CNN-based classifier that leverages CANet segmentation outputs for tumor subtype classification; and (3) a hybrid survival prediction pipeline combining CANet-derived deep features with LASSO and linear regression. The approach achieves state-of-the-art or competitive performance across tasks, including improved segmentation metrics over strong baselines, a top-2 ranking in CPM-RadPath classification, and superior survival prediction to a conventional machine learning baseline. Future work will integrate whole slide pathology images and molecular genetic features for tumor classification, aligning with updated WHO criteria, to further enhance predictive performance and clinical relevance.

- Data imbalance across tumor tissues (e.g., edema prevalence) and survival categories (narrow mid-term range) may bias learning and degrade classification performance. - Limited training sample sizes for classification (221 cases) and survival (210 cases) constrain deep learning effectiveness and generalizability. - Image quality variability and the necessity of robust preprocessing (e.g., intensity normalization) impact performance. - Ground truths are only available for training; validation/testing evaluations are blind and limited by challenge dataset characteristics. - Statistical analyses of age/gender effects are underpowered (n=106 for ANOVA), limiting conclusions. - GPU constraints necessitated cropping, which, while controlled, could omit peripheral context in some cases.