Think out of the box

Abstract

Sequence transduction models have long relied on recurrent or convolutional networks arranged as an encoder and a decoder. This work introduces the Transformer, an architecture built entirely on attention, with no recurrence or convolution. It parallelizes far better and trains in a fraction of the time, while reaching strong translation quality on the WMT 2014 English–German and English–French benchmarks.

1   Introduction

Recurrent networks process a sequence one step at a time, which limits parallel computation over long inputs. Attention instead lets every position relate to every other position directly. We stack multi-head self-attention with simple position-wise feed-forward layers, and add positional encodings so the model still knows the order of tokens.