Encoder–Decoder

Encoder–Decoder is a neural network architecture that transforms an input into a meaningful internal representation (encoding), then uses that representation to generate an output (decoding).

It is widely used in Machine Translation, Speech Recognition (ASR), Text Summarization, Image Captioning, and Large Language Models (LLMs).

1. How does Encoder–Decoder work?

For example, in machine translation:

• Input: "I love artificial intelligence."

• Encoder: Converts the input tokens into contextual representations.

• Decoder: Generates the translated tokens.

• Output: "Tôi yêu trí tuệ nhân tạo."

The encoder does not necessarily produce a single vector. In modern architectures, it often produces a sequence of vectors, one for each input token.

2. What does each component do?

Encoder

The encoder converts raw input into a representation that a neural network can use.

For text, it typically performs the following operations:

1. Tokenize the input into tokens.

2. Convert tokens into embeddings.

3. Process embeddings using neural network layers.

4. Produce contextual hidden representations.

For example, the word "bank" can mean a financial institution or a riverbank. The encoder uses surrounding words to create a representation that captures the appropriate meaning.

Decoder

The decoder uses the encoded information to produce the target output.

In sequence-generation tasks, it often generates one token at a time:

Where:

The decoder predicts the next token based on the preceding output tokens and the available encoded information.

3. Three major Encoder–Decoder designs

A. RNN / LSTM / GRU Encoder–Decoder

Classic Seq2Seq

Processes sequences step by step, passing hidden states between recurrent units. Attention can help the decoder access different parts of the input.

Applications: early machine translation, sequence prediction, some speech models.

B. Transformer Encoder–Decoder

Uses self-attention and feed-forward layers rather than recurrent processing. The decoder also uses cross-attention to access encoder outputs.

Applications: original Transformer translation model, T5, BART, and many speech translation systems.

C. Encoder-only and Decoder-only models

These are related Transformer architectures, but they are not full encoder–decoder models.

• Encoder-only: BERT - representation learning and classification.

• Decoder-only: GPT - autoregressive text generation.

Applications: text understanding, chatbots, code generation, and general-purpose LLMs.

4. Encoder–Decoder vs. Decoder-only

This distinction is particularly important when studying modern AI models.

A decoder-only model can still perform translation, summarization, and ASR. It does not require a separate encoder because it processes the input and output within the same autoregressive Transformer.

5. Example: Encoder–Decoder in ASR

Since speech recognition converts audio into text, it is a natural application of this architecture.

An encoder–decoder ASR system can learn to map audio features directly to text tokens. Depending on the model, training may use attention-based sequence-to-sequence loss or another objective, such as a combination of CTC and attention.