Deep learning-based models have made significant contributions to many fields in recent years but lack robust explainability in their decision making and interpretability in their inference processes. While deep learning is capable of processing information that traditional methods might struggle with, such as nonlinear data, the lack of results explainability can be detrimental to the credibility in such models. Hence, there is a need to enhance the level of explainability, which can come from multimodal analysis. In this study, we implement a Convolutional Neural Network (CNN) for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data to gain insights into how P-RF data can be utilized for target detection. The P-RF data first undergoes feature extraction via Short-Time Fourier Transform (STFT), Continuous Wavelet Transform (CWT), Wigner-Villle Distribution (WVD) and Constant-Q Gabor Transform (CQT). In previous experiments using the ESCAPE dataset, the multimodal design training was incentivized to utilize the P-RF data with I/Q histogram as the feature by purposefully restricting the available EO data. Prior experimentation with both Greedy Algorithms and Saliency Maps indicated that the fusion of P-RF and EO data still heavily focuses on the EO data, if possible, only relying on the P-RF data if it was necessary to detect the target. While P-RF has seen more use in vehicle detection for both autonomous driving and drone applications in recent years, its impact on sensor fusion based decision making is still under investigation. By expanding on the available P-RF data, this paper compares different features of P-RF data and their impact in the fusion using diverse counterfactual explanations (DiCE), as well as potentially increasing the reliability of the P-RF data for target detection.
Insights into Heterogenous Sensor Fusion and Passive RF Feature Extraction for xAI
2024-07-15
381887 byte
Conference paper
Electronic Resource
English
British Library Conference Proceedings | 1999
|Distributed Passive Sensor Trajectory Association and Fusion
Springer Verlag | 2022
|