Abstract
Machine and deep learning algorithms have been proven their efficiency in the medical domain. This is particularly noticeable in medical imaging, where these algorithms have demonstrated exceptional capabilities, often surpassing human experts in detecting diseases, segmenting anatomical structures, and in aiding diagnosis. However, despite their high performance, the non-transparency of these models has raised concerns about trust and explainability, especially in critical domains, such as healthcare. Explainable AI (XAI) has emerged as a promising field that, aims to make the decision-making processes of AI systems more interpretable and understandable to humans. By offering clear, human-understandable justifications, XAI enhances clinicians' trust in AI-assisted decisions, and supports the safe integration of these technologies into clinical workflows. This paper focuses on perturbation-based and backpropagation-based explainability methods that are widely adopted for convolutional neural networks in medical imaging. We intend to provide deeper insight into the relevance of XAI methods in medical imaging. We propose a structured taxonomy that categorizes these methods into perturbation and backpropagation-based approaches. For each category, we highlight their respective strengths, limitations, and areas of applicability. Furthermore, we discuss the key challenges hindering the adoption of XAI and identify promising research directions to bridge the gap between high-performance AI models and their responsible deployment in real-world healthcare environments.