Translate scientific papers with absolute geometric fidelity. Without ever breaking an equation, a column, or a figure.

| 1 | 1.02 | 0.02 | 1.07 | 12.74 | 0.20 | 1.90 | 0.44 | 48.17 | 0.39 | 0.31 |
| 2 | 0.80 | 0.01 | 1.33 | 2.42  | 0.20 | 1.79 | 0.29 | −21.74 | 0.18 | 0.26 |
| 3 | 0.44 | 0.01 | 5.63 | 3.88  | 0.39 | 2.37 | 0.47 | 9.95  | 0.25 | 0.06 |
| 4 | 0.72 | 0.02 | 1.75 | 22.14 | 0.33 | 0.37 | 0.18 | 66.71 | 0.53 | 0.23 |
| 5 | 0.77 | 0.01 | 1.75 | 26.98 | 0.34 | 1.27 | 0.38 | 12.88 | 0.20 | 0.09 |
…  45 rows

{
  "type": "Table",
  "element_id": "48bf2b723bd45202892c544214d92c4c",
  "metadata": {
    "page_number": 11,
    "coordinates": {
      "points": [[50.0, 97.12], [50.0, 747.27],
                 [546.89, 747.27], [546.89, 97.12]],
      "system": "PDFPoints"
    },
    "text_as_html": "<table><tr><td>1</td><td>1.02</td>…"
  }
}

### 3.1 Encoder and Decoder Stacks

Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. We employ a residual connection [ 11] around each of the two sub-layers, followed by layer normalization [ 1]. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension d<sub>model</sub> = 512.

Decoder: The decoder is also composed of a stack of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization. We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.

### 3.2 Attention

An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum

# Attention Is All You Need
## Abstract
## 1 Introduction
## 2 Background
## 3 Model Architecture
### 3.1 Encoder and Decoder Stacks
### 3.2 Attention
#### 3.2.1 Scaled Dot-Product Attention
#### 3.2.2 Multi-Head Attention
### 3.3 Position-wise Feed-Forward Networks
## 4 Why Self-Attention
…

## Abstract

> Résumé

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

> Les modèles dominants de transduction de séquences reposent sur des réseaux neuronaux récurrents ou convolutifs complexes comprenant un encodeur et un décodeur. Les modèles les plus performants relient également l'encodeur et le décodeur par un mécanisme d'attention. Nous proposons une nouvelle architecture de réseau simple, le Transformer, fondée uniquement sur des mécanismes d'attention, se dispensant entièrement de récurrence et de convolutions. Des expériences sur deux tâches de traduction automatique montrent que ces modèles sont de qualité supérieure tout en étant plus parallélisables et en nécessitant significativement moins de temps d'entraînement. Notre modèle atteint 28,4 BLEU sur la tâche de traduction anglais-vers-allemand du WMT 2014, améliorant de plus de 2 BLEU les meilleurs résultats existants, y compris les ensembles. Sur la tâche de traduction anglais-vers-français du WMT 2014, notre modèle établit un nouveau score BLEU de pointe de 41,8 pour un modèle unique après un entraînement de 3,5 jours sur huit GPU, soit une faible fraction des coûts d'entraînement des meilleurs modèles de la littérature. Nous montrons que le Transformer se généralise bien à d'autres tâches en l'appliquant avec succès à l'analyse syntaxique de constituants en anglais, tant avec de grandes qu'avec de limitées données d'entraînement.