A PSO-Optimized stacked ensemble learning approach for accurate melanoma detection with deep transfer learning and explainability

Abstract Accurate melanoma detection is essential for early-stage dermatological diagnosis, as it significantly improves treatment outcomes and patient survival rates. This study proposes a robust stacked ensemble framework that combines transfer learning (TL) with particle swarm optimization (PSO) to enhance melanoma classification performance. Initially, four deep transfer learning models: VGG16, ResNet50, DenseNet121, and InceptionV3 are trained on the Melanoma Skin Cancer Dataset consisting of 10,605 dermoscopic images to generate probability scores for melanoma and benign classes. These probability outputs are used to construct feature vectors for the subsequent stacked ensemble stage. The generated feature vectors are then provided as inputs to multiple meta-classifiers, whose hyperparameters are optimized using the PSO algorithm. PSO explores the hyperparameter search space iteratively, improving candidate solutions based on validation accuracy while utilizing both personal best and global best evaluations. Through this process, a total of eleven ensemble transfer learning (ETL) models are developed. Experimental results demonstrate that ETL11, which integrates VGG16, ResNet50, DenseNet121, and InceptionV3 with Gradient Boosting (GB), achieves the highest accuracy of 92.1% on the held-out test dataset. However, statistical analysis and t-test evaluations reveal that ETL11 with Logistic Regression (LR) offers a more stable and reliable performance across multiple runs, achieving an accuracy of 91.6% and showing statistically significant improvements compared to several alternative meta-classifiers. On the test dataset, ETL11 with GB obtains Precision, Recall, and F1-Score values of 0.922, 0.921, and 0.921, respectively. In addition, Explainable AI (XAI) techniques, particularly SHAP (SHapley Additive exPlanations), are applied to interpret the contribution of transfer learning model probability scores to the final ensemble predictions, thereby improving transparency and clinical interpretability. The integration of TL-based feature extraction, PSO-driven hyperparameter optimization, statistical validation, and explainability results in a reliable and interpretable framework for melanoma detection. Finally, the proposed method is compared with existing approaches and demonstrates competitive or superior performance.