What is a Transformer?

The Transformer is a neural network architecture introduced in the 2017 paper 'Attention Is All You Need.' It powers GPT, Claude, Gemini, Llama, and virtually every leading language model — plus many vision and audio models.

Its key innovation is self-attention: when processing a word, the model weighs how relevant every other word in the input is. This lets it connect a pronoun to a noun ten paragraphs back, or relate distant parts of an image.

Transformers process entire sequences in parallel during training, which is why they scaled so well on modern GPUs — and why AI progress accelerated after 2017.