In machine learning, the ability to reliably determine potential pitfalls in the decision making process of an algorithm is essential. In previous research, the fusion of passive radio frequency (P-RF) histograms generated from in-phase quadrature component (I/Q) data and enhanced electro-optical (EO) data are fused together in order to implement classification and tracking of different vehicle targets using the AFRL's ESCAPE dataset. In previous research, the impact of the P-RF data was confirmed to essential for achieving a higher performance when fused with EO data. This research, however, did not provide an intuitive means by which inferences and explanations for a human expert could interpret. In this paper, saliency maps are implemented in order to visualize the impact of P-RF data in the fusion model and thereby confirm the role the different modalities play in the heterogeneous sensor fusion in an intuitive manner. These visualizations provide context for which pixels activate neurons in the final layer of the model. Overlaying the two modality inputs with respect to time, the method presented in this paper is able to provide explainability for the fusion model while also achieving an F1 score of 0.9. The research in this paper uses the distribution and frequency of the appearance of different types of visualizations. Combined this with context from the scenarios with respect to the timeline of events, it becomes possible to draw inferences for how the two modalities are utilized by the fusion model.
Visualizations of Fusion of Electro Optical (EO) and Passive Radio-Frequency (PRF) Data
16.08.2021
1272562 byte
Aufsatz (Konferenz)
Elektronische Ressource
Englisch
NTRS | 2005
|3.0202 Research and Development of an Integrated Electro-Optical and Radio Frequency Aperture
British Library Conference Proceedings | 2003
|