On intelligent vehicles, Deep Neural Networks (DNNs) may run on devices whose computational load varies over time. Within the context of variable network architectures, that can be used to constrain the inference latency for real-time deployment with varying system resources, we introduce LeAF (Legacy Augmentation for Flexible inference), a novel paradigm to augment a pre-trained DNN with trainable, shallow execution paths that can run in place of the legacy ones. While preserving the legacy DNN weights, LeAF allows changing the DNN architecture with minimal overhead to effectively adapt to different system performance targets. LeAF-ResNet-50 has less than 14% storage overhead over the legacy DNN; its accuracy varies from the legacy 76.1% to 70.15% (up to 5% better than Slimmable [1] with a latency that is 37% better than OFA [2] on an A100 GPU with batch size 256). Our analysis shows the importance of considering not only the target device, but also the batch size and the temporal dynamic of the DNN configuration to optimize the performances of variable architecture DNNs, LeAF in particular.
Augmenting Legacy Networks for Flexible Inference
2023-06-04
1325805 byte
Aufsatz (Konferenz)
Elektronische Ressource
Englisch