Deep learning's success in computer vision is hampered by its "black box" nature, making it difficult to trust its decisions. Explainable AI (XAI) aims to address this by providing human-understandable explanations. Attribution-based explanations, generating heatmaps highlighting image regions contributing to predictions, are common in computer vision. However, these are insufficient for fine-grained classification problems prevalent in expert domains (e.g., medical imaging, biology), where subtle details distinguish classes. In such scenarios, users need to understand not only "why" a specific class was predicted but also "why not" alternative classes. Existing attribution methods often highlight the entire object, failing to pinpoint the discriminative features separating similar classes. This paper proposes GALORE, a novel framework addressing these limitations by incorporating three types of explanations: attributive (highlighting pixels responsible for a prediction), deliberative (visualizing the model's uncertainties and ambiguous regions), and counterfactual (showing the minimal changes needed to elicit a different class prediction). GALORE uses a unified definition based on combining attribution maps with confidence scores, offering a computationally efficient approach applicable to various domains and interactive applications like machine teaching. The framework's core is the ability to reason about classification difficulty and ambiguity using confidence scores, extending these measures to image regions to identify ambiguous or discriminant areas. This self-awareness improves explanation accuracy.

Existing XAI methods for computer vision employ various approaches, including concept-based, example-based, image transformation-based, and language-based explanations. The focus here is on visual explanations, particularly saliency maps generated by attribution functions. Methods are categorized based on whether they are designed into the model (interpretable models) or applied post-hoc to pre-trained models. This paper concentrates on post-hoc methods. Attribution functions like Grad-CAM, SHAP, LIME, and RISE are widely used, often computing gradient variations of classifier predictions with respect to network layers. Counterfactual explanations usually involve image transformations (perturbations, synthesis, or feature replacement), but these often produce unrealistic images or are computationally expensive. XAI evaluation is challenging, often relying on human-in-the-loop experiments or proxy tasks (e.g., feature erasure/addition, localization). Self-aware systems, capable of measuring their limitations and predicting failures, are related to this work, particularly in terms of producing confidence scores for decision-making and ambiguity assessment.

GALORE unifies various explanations through a heatmap defined as a combination of attribution maps concerning various classifier predictions and a confidence score. The framework uses the following components: **A. Attributive Explanations:** These identify pixels responsible for a prediction. GALORE uses any attribution function (gradient-based or otherwise) to create a heatmap; self-aware attributive explanations incorporate confidence scores to sharpen the heatmap. **B. Deliberative Explanations:** These address the "why" question. GALORE identifies classes with the highest posterior probabilities, generates ambiguity maps for class pairs, and thresholds these to obtain segmentation masks representing "insecurities." Insecurities indicate regions causing classifier uncertainty. **C. Counterfactual Explanations:** These address the "why not" question by visualizing changes needed to switch to a counterfactual class. GALORE uses discriminant explanations to identify regions informative for the predicted but not the counterfactual class. A counterfactual explanation is composed of two discriminant explanations, one for the query image and one for a randomly selected image from the counterfactual class. **Multi-Class Extensions:** GALORE extends to multi-class scenarios for both deliberative and counterfactual explanations. Ambiguity maps consider multiple classes simultaneously, and counterfactual explanations incorporate multiple counterfactual classes. **Explanation Strength:** This is a quantitative measure of explanation clarity, defined as the average intensity of the attribution map within a segment. It's used to rank insecurities or counterfactual explanations. **Attribution Maps:** GALORE accommodates various attribution functions (gradient-based like vanilla gradient, Integrated Gradients, Grad-CAM; and non-gradient-based like Score-CAM, SHAP). **Confidence Scores:** GALORE supports different confidence scores (softmax score, certainty score, easiness score). **Network Implementation:** GALORE is implemented as a network that takes an image and generates explanations based on the chosen type (attributive, deliberative, or counterfactual).

The experiments, conducted on CUB200 (fine-grained bird classification) and ADE20K (scene classification) datasets, yielded several key findings: **1. Self-Awareness Improves Accuracy:** Incorporating confidence scores significantly improves the accuracy of all three explanation types. The "easiness score" consistently outperforms other confidence scores, except for counterfactual explanations with beginner users. **2. Attribution Function Impact is Limited:** While more sophisticated attribution functions (Integrated Gradients, Grad-CAM, Score-CAM, SHAP) generally outperform the basic gradient method, the differences are not substantial, particularly on the more challenging ADE20K dataset. **3. Network Architecture Matters:** ResNet-50 produces more accurate explanations than VGG16, especially on CUB200, suggesting ResNet's deliberations are more intuitive. **4. Multi-Class Explanations are Effective:** Multi-class explanations (deliberative with three classes, counterfactual with two classes) perform similarly to or better than binary explanations, even though this might seem counter-intuitive. **5. Explanation Strength Correlates with Quality:** The strength of an explanation (average attribution map intensity) is strongly positively correlated with its quality (precision). **6. Robustness to Data Shifts and Model Variance:** GALORE's explanations show robustness to image shifts (translations) and are insensitive to random weight initialization (passing sanity checks). **7. Superior Performance Compared to State-of-the-Art:** GALORE outperforms existing counterfactual explanation methods ([30], CounteRGAN) and a novel baseline for deliberative explanations, while achieving significantly higher computational efficiency (1000x-50x faster than [30]). **8. Human-Interpretable Explanations:** Human studies on Amazon Mechanical Turk confirm that GALORE's deliberative explanations are intuitive and accurately reflect human perception of ambiguity. Counterfactual explanations significantly enhance machine teaching effectiveness.

GALORE's unified approach significantly advances the field of XAI for deep learning models. The integration of deliberative explanations addresses a critical gap in current methods, providing insight into the model's reasoning process. The efficiency gains from the unified framework make it applicable to real-time interactive applications. The quantitative evaluation protocol offers a more replicable and objective assessment of explanation quality compared to solely relying on human studies. The findings emphasize the importance of self-awareness in enhancing explanation accuracy and the correlation between GALORE's explanations and human cognitive processes. This work strengthens the trustworthiness and interpretability of deep learning models, especially in critical domains like healthcare where understanding model decisions is paramount.

GALORE presents a unified framework for generating attributive, deliberative, and counterfactual explanations, addressing the need for diverse user requirements in XAI. It achieves this with increased computational efficiency and improved accuracy compared to existing methods. Future research can explore optimizing the threshold selection strategies, investigating the relationship between explanation strength and human perception, and extending the framework to other domains and model types. Further investigation into the interaction between different types of explanations and their effectiveness in various user contexts is also warranted.

While GALORE demonstrates improved accuracy and efficiency, some limitations exist. The evaluation relies on proxy tasks and human studies, which may not fully capture the nuances of human interpretation. The reliance on specific attribution functions and confidence scores might limit generalizability. Further research is needed to explore the optimal choice of these components across diverse datasets and tasks. The current implementation focuses on image classification; extending GALORE to other tasks (e.g., object detection, segmentation) would be valuable.