Deep learning-based models have made significant contributions to many fields in recent years but lack robust explainability in their decision making and interpretability in their inference processes. While deep learning is capable of processing information that traditional methods might struggle with, such as nonlinear data, the lack of results explainability can be detrimental to the credibility in such models. Hence, there is a need to enhance the level of explainability, which can come from multimodal analysis. In this study, we implement a Convolutional Neural Network (CNN) for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data to gain insights into how P-RF data can be utilized for target detection. The P-RF data first undergoes feature extraction via Short-Time Fourier Transform (STFT), Continuous Wavelet Transform (CWT), Wigner-Villle Distribution (WVD) and Constant-Q Gabor Transform (CQT). In previous experiments using the ESCAPE dataset, the multimodal design training was incentivized to utilize the P-RF data with I/Q histogram as the feature by purposefully restricting the available EO data. Prior experimentation with both Greedy Algorithms and Saliency Maps indicated that the fusion of P-RF and EO data still heavily focuses on the EO data, if possible, only relying on the P-RF data if it was necessary to detect the target. While P-RF has seen more use in vehicle detection for both autonomous driving and drone applications in recent years, its impact on sensor fusion based decision making is still under investigation. By expanding on the available P-RF data, this paper compares different features of P-RF data and their impact in the fusion using diverse counterfactual explanations (DiCE), as well as potentially increasing the reliability of the P-RF data for target detection.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Insights into Heterogenous Sensor Fusion and Passive RF Feature Extraction for xAI


    Contributors:
    Vakil, Asad (author) / Ewing, Robert (author) / Blasch, Erik (author) / Li, Jia (author)


    Publication date :

    2024-07-15


    Size :

    381887 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Explainable Hybrid Decision Level Fusion for Heterogenous EO and Passive RF Fusion via xLFER

    Vakil, Asad / Blasch, Erik / Ewing, Robert et al. | IEEE | 2023


    Feature Level Sensor Fusion for Passive RF and EO Information Integration

    Vakil, Asad / Liu, Jenny / Zulch, Peter et al. | IEEE | 2020


    Feature-level sensor fusion

    Peli, T. Young, M. Knox, R. Ellis, K. K. | British Library Conference Proceedings | 1999


    FEATURE FUSION OF SENSOR DATA

    MOODIE DANIEL | European Patent Office | 2025

    Free access

    Distributed Passive Sensor Trajectory Association and Fusion

    Chang, Ruowang / Wang, Bolan / Xu, Shengli et al. | Springer Verlag | 2022