Compared with the feature-based distillation methods, logits distillation can liberalize the requirements of consistent feature dimension between teacher and student networks, while the performance is deemed inferior in face recognition. One major challenge is that the light-weight student network has difficulty fitting the target logits due to its low model capacity, which is attributed to the significant number of identities in face recognition. Therefore, we seek to probe the target logits to extract the primary knowledge related to face identity, and discard the others, to make the distillation more achievable for the student network. Specifically, there is a tail group with near-zero values in the prediction, containing minor knowledge for distillation. To provide a clear perspective of its impact, we first partition the logits into two groups, i.e., Primary Group and Secondary Group, according to the cumulative probability of the softened prediction. Then, we reorganize the Knowledge Distillation (KD) loss of grouped logits into three parts, i.e., Primary-KD, Secondary-KD, and Binary-KD. Primary-KD refers to distilling the primary knowledge from the teacher, Secondary-KD aims to refine minor knowledge but increases the difficulty of distillation, and Binary-KD ensures the consistency of knowledge distribution between teacher and student. We experimentally found that (1) Primary-KD and Binary-KD are indispensable for KD, and (2) Secondary-KD is the culprit restricting KD at the bottleneck. Therefore, we propose a Grouped Knowledge Distillation (GKD) that retains the Primary-KD and Binary-KD but omits Secondary-KD in the ultimate KD loss calculation. Extensive experimental results on popular face recognition benchmarks demonstrate the superiority of proposed GKD over state-of-the-art methods.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Grouped Knowledge Distillation for Deep Face Recognition


    Beteiligte:
    Zhao, Weisong (Autor:in) / Zhu, Xiangyu (Autor:in) / Guo, Kaiwen (Autor:in) / Zhang, Xiao-Yu (Autor:in) / Lei, Zhen (Autor:in)

    Erscheinungsdatum :

    2023


    Anmerkungen:

    9 pages, 2 figures, 7 tables, accepted by AAAI 2023


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Grouped vehicle

    TAKAHASHI KEISUKE / TATEISHI GO / OOTA MASATO | Europäisches Patentamt | 2024

    Freier Zugriff

    Grouped SLOR Deep Water Riser System and Installation Assessment

    Hatton, S. / Karunakaran, D. / Mair, J. et al. | British Library Conference Proceedings | 2007


    Grouped Road Vehicles

    N. E. Lobdell | NTIS | 1967


    PAIRED OR GROUPED DRONES

    KALE POORNA | Europäisches Patentamt | 2021

    Freier Zugriff

    Paired or grouped drones

    KALE POORNA | Europäisches Patentamt | 2023

    Freier Zugriff