DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

[jinmang2]

집현전 중급반 스터디

• 2022년 6월 26일 일요일 9시
• 장동건님 김제우님 김종은님 이기성님 발표
• 논문 링크: https://arxiv.org/abs/1910.01108

Abstract

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

[jinmang2]

발표자료: https://drive.google.com/file/d/1qTECNfIimcq-MSreySNGy3ziMmlXV--3/view?usp=sharing

[jinmang2]

• 발표링크: https://www.youtube.com/watch?v=Oy3Jv9P7i5w&list=PL2tRglRS_GqjCcBVM26gYNdoOnX12lLIh&index=12

[mycogno]

위의 발표 링크가 다른 발표와 연결되어서 다시 링크 올려드립니다.

• 발표링크: https://www.youtube.com/watch?v=zeCcK3vU5FM