In previous posts we've already talked about embeddings—what they are, how they're used to represent words, phrases, and documents. But today we're going a step further and delving into a absolute classic in the world of generative artificial intelligence: the paper Attention Is All You Need. This article gave rise to the architecture known as Transformer, which underpins many of the models behind the AI boom (including large language models).Wikipedia)
We want to explain it in a way that informative, Accessible to everyone, without losing the technical aspects. So, if you have knowledge of embeddings and AI vocabulary, you might become familiar with how that big black box called Transformer works "on the inside.".
Why is this paper so relevant?
- The article was published in 2017 by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. (arXiv)
- He proposed an architecture that It dispenses with recurrent (RNNs) and convolutional (CNNs) networks for sequence-to-sequence (seq2seq) tasks, and instead uses only care mechanisms. (papers.neurips.cc)
- For this reason, it is considered a watershed moment: it allowed for more scalable and parallelizable models, and laid the foundation for models such as BERT, GPT-3/GPT-4, etc.Wikipedia)
What is “attention” and why does it matter?
To fully understand the paper, it's helpful to remember what an embedding is (we've already seen this) and how sequential language models used to work:
- Before Transformers, RNNs (or LSTMs/GRUs) were used to process text as sequences: word after word, maintaining an internal "state" that accumulated information.
- These models have two important limitations:
- Dependence on previous stepsThe information that appeared at the beginning may be lost or "diluted" by the end of the sequence.
- Difficult to parallelBecause it has to process word by word, simultaneous calculations cannot be fully utilized in modern hardware.
- Then the idea arises of attentionInstead of relying solely on the previous state, a model can "look" directly all input positions and decide "which parts interest me most" to produce the output.
The paper defines the scaled-dot-product attention as:

where (Q) (query), (K) (key) and (V) (value) are vector matrices. (Wikipedia)
Simply put: for each query, the weight of all the keys is calculated, and this is used to combine the values. In natural language: "This word (or token), what other words should you pay attention to in order to make your decision?".
Transformer Architecture: Explained in a friendly way
Here we get into the technical part, but we'll explain it by analogy to make it understandable:
Encoder and decoder
The proposed model has two main blocks: one encoder that processes the input sequence, and a decoder that generates the output sequence (for example, translation from one language to another).Medium)
Each one is composed of several identical “layers” (in the original paper, 6 encoder layers and 6 decoder layers were used).Medium)
Self-attention
In the encoder block, each token "looks" at all the tokens in the sequence (including itself) and decides how much attention to give to each one. This allows capturing long dependencies (for example, the first word with the last) without processing word by word.
Analogy: Imagine that when reading a sentence, to understand a word I not only look at the previous one, but also that the first word gives me context.
Multi-head attention
But it's not enough to look at a "single perspective". The model has several "heads" of attention, each learning to look at different relationships (one can see grammatical relationships, another semantic ones, another broader context relationships). Then all those heads are combined.Wikipedia)
Simple analogy: as if several readers looked at the sentence from different angles at the same time, then came together to decide on the common interpretation.
Positional encoding
Since we eliminated sequential processing (RNN), the model needs to know the order in which the tokens appear. For this, a “position encoding” using sine and cosine functions is added to the original embedding.Wikipedia)
Analogy: When reading a sentence, I not only know what word it is, but "this one is in position 5, this one is in position 1"... and that information also matters.
Simplified flow
- We tokenize the phrase → we convert it into IDs → we see its embeddings.
- In addition to embedding, we add position encoding.
- We go through several layers of the encoder: each layer does self-attention + feed-forward + normalization + residuals.
- The decoder takes the output from the encoder + its own attention mechanism (looks at the encoder, and what it has already generated) → generates the output sequence token by token.
- Finally, the probability of each exit token is obtained, the most probable one is selected, and the process is repeated until the end.
What was the major contribution compared to what came before?
- The model is more parallelIt doesn't need to process step by step like an RNN. This allows for faster training and better use of modern hardware.Wikipedia)
- It improves the ability to capture long dependencies: one word can directly address another that is far away in the sequence.
- Simplify the architecture: by eliminating recurrence/convolution complications and focusing on attention + feed-forward.
- It was the basis for the great generative models that we know today.
What was it tested for and what results did it achieve?
In the original paper, they tested machine translation (English→German, English→French) and obtained better results than previous models, with lower training costs.arXiv)
For example, in WMT-14 English→French they obtained a BLEU of 41.8. (arXiv)
This showed that architecture was not only theoretical, but practical and competitive.
An explanation with analogies for “everyone”
Imagine you're in a meeting room (your sequence of words). Before, with RNN, the meeting was like "I speak, then someone else, then someone else..." in a line: everyone has to wait their turn, and what the first person said might already be forgotten by the end.
With the Transformer, it's as if all the participants they saw at the same time to everyone else—a “circle of attention”—and each person decides to listen to any of the others, and also “look” at several things at the same time (multi-headed). And at the same time they have a map of “who is in which seat” (positional encoding) so as not to lose order.
This way the entire “conversation” is understood at a glance, without depending on going one by one.
What are the implications for embeddings and your embedding work?
Since we've talked about embeddings in previous posts, it's worth noting that Transformers use them very effectively.
- Transformer-based architectures generate contextual embeddingsThe embedding of a word depends on the other words in the sentence. This differs from static embedding ("apple" always has the same vector) and is a huge improvement.
- Thanks to attention, these contextual embeddings can capture complex relationships: which words are related to each other, what meaning a word carries in that context, etc.
Therefore, by understanding the paper “Attention Is All You Need”, we better understand why we now use models that generate such rich and varied embeddings, and why we no longer only work with Word2Vec or FastText as before.
Conclusion
The paper Attention Is All You Need It represents a milestone in AI. It introduced an architecture that changed the way language models are built, text is generated, relationships between words are understood, and sequences are handled with great efficiency.
For you, who already work with embeddings and text analysis, reading this paper gives you a solid foundation to understand why current technologies work the way they do, and how you could consider using or explaining them in your project.
In future posts we could see how this paper translates into specific models (such as GPT, BERT…), how contextual embedding works, and how you yourself can take advantage of it for comment analysis, clustering, etc.
[pdf-embedder url=»https://germanmallo.com/wp-content/uploads/2025/11/NIPS-2017-attention-is-all-you-need-Paper.pdf» title=»attention-is-all-you-need-Paper»]