Recent research on knowledge distillation has increasingly focused on logit distillation because of its simplicity, effectiveness, and versatility in model compression. In this paper, we introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods. Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions, creating a conflict between the standard distillation loss and the cross-entropy loss. This conflict can undermine the consistency of the student model's learning objectives. Previous attempts to use labels to empirically correct teacher predictions may undermine the class correlation. In contrast, our RLD employs labeling information to dynamically refine teacher logits. In this way, our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations, thus enhancing the value and efficiency of distilled knowledge. Experimental results on CIFAR-100 and ImageNet demonstrate its superiority over existing methods. The code is provided at \text{https://github.com/zju-SWJ/RLD}.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Knowledge Distillation with Refined Logits


    Beteiligte:
    Sun, Wujie (Autor:in) / Chen, Defang (Autor:in) / Lyu, Siwei (Autor:in) / Chen, Genlang (Autor:in) / Chen, Chun (Autor:in) / Wang, Can (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    11 pages, 7 figures


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    NormKD: Normalized Logits for Knowledge Distillation

    Chi, Zhihao / Zheng, Tu / Li, Hengjia et al. | ArXiv | 2023

    Freier Zugriff

    Adaptive Temperature Based on Logits Correlation in Knowledge Distillation

    Matsuyama, Kazuhiro / Anjum, Usman / Matsuyama, Satoko et al. | ArXiv | 2025

    Freier Zugriff

    Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

    Ma, Jing / Xiang, Xiang / Wang, Ke et al. | ArXiv | 2022

    Freier Zugriff

    Peak-Controlled Logits Poisoning Attack in Federated Distillation

    Tang, Yuhan / Zhang, Aoxu / Wu, Zhiyuan et al. | ArXiv | 2024

    Freier Zugriff

    BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation

    Li, Minchong / Zhou, Feng / Song, Xiaohui | ArXiv | 2024

    Freier Zugriff