Yuhuai Wu | Memorizing Transformers

[argosopentech]

SGConv - Structural Global Convolution Networks

In the past few years Transformer networks have been the dominant way to do natural language processing; they’re efficient to train in parallel and can understand the context a word is being used in well. Transformer models work by calculating an attention value between every pair of tokens in the input sequence which is O(n2); because Transformers are O(n2) it’s difficult to scale them up to large input sequences. Argos Translate currently uses Transformers and deals with this problem by splitt…