TRANSFORMERS

The transformers work with 3 mechanisms: encoder-decoder, attention mechanism and transfer learning.

ENCODER-DECODER FRAMEWORK

Before transformers, NLP mainly used structures like LSTMs, which were considered top-notch. These structures have a loop that allows information to move from one step to the next. This makes them great for handling data in a sequence, like text.

In an RNN, you give it input (like a word or character), it processes it, and gives you a hidden state vector. At the same time, it sends some information back to itself through a loop, which it then uses in the next step. The RNN shares its state information at each step with the next operation in the sequence. This helps the RNN remember information from previous steps and use it to predict the output. RNNs were crucial in the advancement of machine translation systems, particularly in the task of converting a sequence of words from one language to another. For such tasks, an encoder-decoder or sequence-to-sequence architecture is commonly employed. This architecture is well suited for scenarios where both the input and output consist of sequences of varying lengths.

The encoder’s role is to convert information from the input sequence into a numerical representation commonly referred to as the last hidden state. This state is then forwarded to the decoder, which produces the output sequence.

Typically, the encoder and decoder components can be any neural network architecture capable of modeling sequences. This is demonstrated with a pair of RNNs in the figure below. The input words are processed sequentially through the encoder, and the output words are generated one at a time, from top to bottom. A drawback of this architecture is the creation of an information bottleneck by the final hidden state of the encoder (C in the figure). This state must encapsulate the meaning of the entire input sequence since it’s the only information available to the decoder during output generation. This becomes particularly challenging for lengthy sequences, where information from the beginning might get lost when compressing everything into a single, fixed representation. Fortunately, there’s a solution to this bottleneck by enabling the decoder to access all of the encoder’s hidden states. This mechanism is known as attention, and it serves as a fundamental component in numerous contemporary neural network architectures.

ATTENTION MECHANISM

The central concept behind attention is that instead of generating a single hidden state for the input sequence, the encoder produces a hidden state at each step that the decoder can access. However, using all these states simultaneously would overwhelm the decoder, so a mechanism is necessary to prioritize which states to utilize. This is where attention becomes crucial: it enables the decoder to assign varying amounts of weight, or ‘attention,’ to each of the encoder states at each decoding timestep.

By concentrating on which input tokens are most relevant at each timestep, these models that employ attention are capable of acquiring intricate relationships between the words in a translated output and those in the source sentence

So, I explained why we use the attention mechanism basically, we will focus on types of attention in future blog posts.

In numerous real-world NLP applications, obtaining substantial labeled text data for training models is often impractical. This is where we need the transfer learning.

TRANSFER LEARNING

In terms of architecture, transfer learning involves dividing the model into a body and a head, where the head represents a task-specific network. During training, the weights of the body learn broad features from the source domain, and these weights are utilized to initialize a new model for the target task. Compared to traditional supervised learning, this approach typically yields high-quality models that can be trained much more efficiently across various downstream tasks, requiring significantly less labeled data.

Okay, let’s take a deep breath and ask the question: ‘What will we do with these 3 mechanisms?’

In 2018, two transformers were introduced, which integrated self-attention with transfer learning: GPT and BERT

GPT:

Uses only the decoder part of the Transformer architecture, following the same language modeling approach as ULMFiT. GPT was pretrained on the BookCorpus, a collection of 7,000 unpublished books spanning various genres, including Adventure, Fantasy, and Romance.

BERT:

Utilizes the encoder part of the Transformer architecture and employs a unique form of language modeling known as masked language modeling. The objective of masked language modeling is to predict randomly masked words in a text. For instance, given a sentence like ‘I looked at my [MASK] and saw that [MASK] was late,’ the model must predict the most likely candidates for the masked words denoted by [MASK]. BERT was pretrained on the BookCorpus and English Wikipedia.” (Check for more information.)

In summary, transformers have redefined natural language processing through three key components: the encoder-decoder framework, attention mechanism, and transfer learning. They address challenges like information bottlenecks and limited labeled data by enabling efficient training on diverse tasks.

The introduction of transformers like GPT and BERT in 2018 marked a pivotal moment, combining self-attention with transfer learning. GPT, focused on the decoder, and BERT, leveraging the encoder, demonstrate the adaptability of transformers in language modeling, pretrained on extensive datasets.

The transformative impact of transformers continues to revolutionize language processing, promising innovative solutions for various linguistic challenges. Future blog posts will explore specific attention mechanisms in more detail.