X
Xiao Chen
Researcher at Huawei
Publications - 25
Citations - 1808
Xiao Chen is an academic researcher from Huawei. The author has contributed to research in topics: Language model & Natural language understanding. The author has an hindex of 8, co-authored 25 publications receiving 801 citations.
Papers
More filters
Proceedings ArticleDOI
TinyBERT: Distilling BERT for Natural Language Understanding
TL;DR: TinyBERT as discussed by the authors proposes a two-stage learning framework for TinyBERT, which performs transformer distillation at both the pre-training and task-specific learning stages to capture the general-domain as well as the task specific knowledge in BERT.
Posted Content
TinyBERT: Distilling BERT for Natural Language Understanding
TL;DR: A novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models is proposed and, by leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be effectively transferred to a small “student” TinyBERT.
Proceedings Article
DynaBERT: Dynamic BERT with Adaptive Width and Depth
TL;DR: A novel dynamic BERT model, which can run at adaptive width and depth, is proposed (abbreviated as DynaBERT), which has comparable performance as BERT (or RoBERTa), while at smaller widths and depths consistently outperforms existing BERT compression methods.
Proceedings ArticleDOI
TernaryBERT: Distillation-aware Ultra-low Bit BERT
TL;DR: This work proposes TernaryBERT, which ternarizes the weights in a fine-tuned BERT model, and leverages the knowledge distillation technique in the training process to reduce the accuracy degradation caused by the lower capacity of low bits.
Posted Content
NEZHA: Neural Contextualized Representation for Chinese Language Understanding.
Junqiu Wei,Xiaozhe Ren,Xiaoguang Li,Wenyong Huang,Yi Liao,Yasheng Wang,Jiashu Lin,Xin Jiang,Xiao Chen,Qun Liu +9 more
TL;DR: The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition, sentence matching, Chinese sentiment classification, and natural language inference (XNLI).