Section 1.1: The Encoder-Decoder Framework

Prior to the advent of transformers, architectures like LSTMs dominated NLP tasks. These models feature a feedback loop in their connections, enabling information to flow from one step to another, making them particularly suited for sequential data such as text. As shown on the left side of Figure 1–2, an RNN processes input (which can be a word or character), passing it through the network to produce a vector known as the hidden state. Concurrently, the model feeds information back to itself through the feedback loop for use in subsequent steps. This process becomes clearer when "unrolling" the loop, depicted on the right side of Figure 1–2: the RNN transmits state information at each step to the next operation in the sequence. This mechanism allows RNNs to retain information from earlier steps, aiding in output predictions.

These architectures are still widely used in various fields, including NLP, speech processing, and time series analysis. For a comprehensive understanding of their capabilities, refer to Andrej Karpathy's insightful blog post, "The Unreasonable Effectiveness of Recurrent Neural Networks."

One crucial application of RNNs has been in developing machine translation systems, where the goal is to convert sequences of words from one language to another. Typically, this task is approached using an encoder-decoder or sequence-to-sequence framework, ideal for handling input and output sequences of varying lengths. The encoder’s role is to convert the input sequence into a numerical representation known as the last hidden state, which is then passed to the decoder to generate the output sequence.

Generally, the encoder and decoder can be any neural network architecture capable of modeling sequences. This is exemplified with a pair of RNNs in Figure 1–3, where the English sentence "Transformers are great!" is transformed into a hidden state vector, which is subsequently decoded to yield the German translation "Transformer sind grossartig!" The input words are processed sequentially through the encoder, while the output words are produced one at a time, from top to bottom.

Despite its straightforwardness, this architecture has a significant limitation: the encoder's final hidden state creates an information bottleneck. It must encapsulate the meaning of the entire input sequence, which poses a challenge when generating outputs, especially for longer sequences, as information from the beginning of the sequence may be lost during the compression process.

Fortunately, a solution exists to overcome this bottleneck by granting the decoder access to all of the encoder's hidden states. This mechanism, known as attention, is crucial in many contemporary neural network architectures.

To fully grasp the function of attention, we must delve into how it was initially developed for RNNs.

Section 1.2: Attention Mechanisms

The essence of attention lies in allowing the encoder to output a hidden state at each step instead of a single state for the entire input sequence. However, to manage the data input for the decoder, a prioritization mechanism is required. Attention facilitates this by enabling the decoder to assign varying weights, or "attention," to each encoder state at each decoding timestep. This process is illustrated in Figure 1–4, which demonstrates the role of attention in predicting the third token in the output sequence.

By concentrating on the most relevant input tokens at each timestep, attention-based models can learn meaningful alignments between words in the generated translation and those in the source sentence. Figure 1–5 visualizes the attention weights for an English-to-French translation model, with each pixel representing a weight. It highlights how the decoder successfully aligns the words "zone" and "Area," which are arranged differently in both languages.

While attention significantly improved translation accuracy, recurrent models for the encoder and decoder still had a major drawback: their computations are sequential and cannot be parallelized across the input sequence.

With the introduction of the transformer, a new modeling paradigm emerged: eliminating recurrence entirely and relying solely on a unique form of attention known as self-attention. This approach enables attention to function across all states within the same layer of the neural network.

As depicted in Figure 1–6, both the encoder and decoder incorporate their own self-attention mechanisms, with their outputs directed to feed-forward neural networks (FF NNs). This architecture allows for much faster training compared to recurrent models, leading to numerous recent breakthroughs in NLP.

In the original Transformer paper, the translation model was trained from scratch using a large corpus of sentence pairs in various languages. However, in practical NLP applications, access to extensive labeled text data for training can often be limited. This gap needed to be bridged to initiate the transformer revolution: transfer learning.