EFFICIENT AI · KNOWLEDGE DISTILLATION · ACTIVE RESEARCH
CoGS-KD
Confidence-Gated Structural-Contrastive Knowledge Distillation for compressing transformer models while preserving useful teacher representations.
- Confidence-aware knowledge transfer
- Structural + contrastive supervision