What is a Transformer? Attention mechanisms and model architectures
Understand attention through pronoun relationships, distinguish encoders, decoders, and trained weights, and learn why architecture names do not establish capability.
Understand the concept
A Transformer is a neural-network architecture centered on attention, which combines information from different sequence positions. The original sequence-transformation model used an encoder and decoder; later models may use only one type. It describes a computational structure, not a particular trained model or a guarantee of shared capabilities or accuracy.
Attention relates positions through learned representations
In a sentence where Wang gives Li a book and “he” starts reading, interpreting the pronoun requires context. Attention learns relationships between positions, but its weights are not equivalent to human understanding and cannot guarantee resolution of genuine ambiguity.
The original Transformer also uses feed-forward networks and positional information. Attention is more than manually highlighting keywords. Training and architecture affect sensitivity, and an attention visualization alone does not establish a full causal explanation for an answer.
Encoders and decoders support different task forms
Encoders often represent inputs, decoders often generate outputs, and their combination supports tasks such as translation. These are architectural generalizations; training objectives also determine use. A family label does not establish chat or image-input support.
Autoregressive generation predicts tokens from context and prior generated content, without access to future answer tokens. Distinguish training masks from runtime generation; parallel training does not mean the whole answer is generated simultaneously.
Check architectures together with checkpoints
Architecture defines computation; checkpoints store trained weights. The same architecture can have different data and task versions with different behavior. Match configuration and tokenizer rather than assuming arbitrary weights are compatible with a similarly named structure.
Validate concrete input-output requirements, such as whether a model acknowledges an ambiguous pronoun, instead of testing whether it can name attention. This page reviews papers and teaching sources without benchmarking architectures or inferring rankings from names.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.