W
Weiguo Gao
Publications - 1
Weiguo Gao is an academic researcher. The author has contributed to research in topics: Linearization & Softmax function. The author has co-authored 1 publications.
Papers
More filters
Posted Content
SOFT: Softmax-free Transformer with Linear Complexity
Jiachen Lu,Jinghan Yao,Junge Zhang,Xiatian Zhu,Hang Xu,Weiguo Gao,Chunjing Xu,Tao Xiang,Li Zhang +8 more
TL;DR: Wang et al. as discussed by the authors proposed a softmax-free transformer (SOFT) to remove softmax in self-attention, which enables a full selfattention matrix to be approximated via a low-rank matrix decomposition.