Papyros
← Attention Is All You Need

Idea lineage.

From backpropagation to the architecture that powers every large language model in use today.

Ancestors

  1. 1986

    Learning Representations by Back-propagating Errors

    Rumelhart, Hinton & Williams

    Backpropagation made deep networks trainable.

  2. 1997

    Long Short-Term Memory

    Hochreiter & Schmidhuber

    Gated memory cells gave RNNs a longer reach.

  3. 2014

    Neural Machine Translation by Jointly Learning to Align and Translate

    Bahdanau, Cho & Bengio

    First soft attention: the decoder could look back at any encoder state.

  4. 2015

    Attention with Intention

    Xu et al.

    Hard vs soft attention in image captioning — the concept crystallised.

2017

This paper

Attention Is All You Need

Vaswani et al.

Drop the recurrence entirely. Multi-head self-attention as the whole architecture.

Built on this

  1. 2018

    BERT: Pre-training of Deep Bidirectional Transformers

    Devlin et al.

    Masked language modelling: read the whole sentence, predict any part.

  2. 2020

    Language Models are Few-Shot Learners

    Brown et al.

    175 billion parameters. In-context learning without fine-tuning.

  3. 2023

    Llama: Open and Efficient Foundation Language Models

    Touvron et al.

    Open weights. The architecture converged; scale became accessible.