Transformer Architecture (September 2026)¶
Version context: transformer families, positional encodings, attention kernels, tokenizer behavior, context limits, and cache implementations vary by model release. This page explains stable architectural ideas; it does not make a permanent claim about a particular model's internals or maximum context.
The 2017 Transformer paper introduced a sequence-transduction architecture based on attention instead of recurrence or convolution. The original design uses an encoder-decoder structure and attention to connect them. Attention Is All You Need
The Core Computation¶
A transformer processes token representations. For an attention layer, learned projections derive queries (Q), keys (K), and values (V) from those representations:
For each query position, the attention weights select and combine value vectors from allowed key positions. The exact projection layout, number of heads, normalization order, activation, and numerical kernel are model-specific implementation choices.
Multi-head attention runs several attention projections in parallel, then combines them. It gives a layer multiple representation subspaces; it is not a literal list of independent human-readable "topics."
Three Common Architectural Families¶
| Family | Attention pattern | Typical output role | Important boundary |
|---|---|---|---|
| Encoder-only | tokens attend across the permitted input | contextual representations | cannot be assumed to generate an autoregressive answer |
| Decoder-only | each token is masked from future tokens | next-token generation | inference produces tokens sequentially |
| Encoder-decoder | decoder self-attention plus cross-attention to encoded input | conditional generation | source representation and generated sequence have different roles |
These are patterns, not a product ranking. The correct choice depends on training objective, deployed model contract, tool behavior, and evaluation results.
Order, Masks, and Position¶
Attention alone does not give a sequence position an inherent order. A transformer therefore adds or otherwise represents positional information. The original paper uses positional encodings; later model families use other position mechanisms. Treat the positional scheme as part of the resolved model configuration.
Masks define which positions may attend to which others:
- causal mask: prevents a decoder position from reading future generated tokens;
- padding or attention mask: prevents invalid or absent positions from contributing;
- cross-attention mask: constrains which source positions a decoder may consult.
A wrong mask can produce a system that appears to work in a demo while leaking future labels during evaluation or mixing records across a batch.
Training and Autoregressive Generation Differ¶
During training, implementations can often evaluate many sequence positions in parallel subject to the model's mask. During decoder-style generation, token t + 1 depends on the state after token t, so each new token advances the request state.
Inference implementations may retain prior key/value projections in a cache to avoid recomputing the whole prefix. The cache is an optimization with a strict isolation requirement: it must be tied to the request, model/configuration revision, and applicable tenant/data boundary. Do not infer cache behavior or capacity from a model name. Hugging Face KV cache guide
A Practical Model Contract¶
An application should record the transformer-facing configuration that changes behavior:
{
"model_key": "approved-generation-model",
"resolved_model_revision": "observed-at-runtime",
"tokenizer_revision": "recorded-at-deploy",
"context_policy_revision": "context@8",
"attention_or_position_config": "provider-or-local-reference",
"inference_cache_policy": "per-request-v2",
"output_contract": "answer-with-citations/v3"
}
This is more useful for reproducibility than a prose statement such as "uses a transformer." A model alias can hide a changed tokenizer, context behavior, or kernel implementation.
Performance Work Is an Evaluation Problem¶
Attention cost, memory use, batching, quantization, and cache strategy can affect latency and quality. Optimize only with a representative workload and a task-level acceptance suite.
Measure:
- request and generated-token latency separately;
- peak memory and concurrency under the intended context mix;
- task quality, citation coverage, and structured-output validity;
- error, timeout, and retry rates;
- cache hit or reuse signals only where the runtime exposes them;
- isolation and cancellation behavior under concurrent requests.
PyTorch's MultiheadAttention documentation is a useful implementation reference, but a framework primitive does not define the architecture of every deployed model. PyTorch MultiheadAttention
Gotchas¶
- "Parallel transformer" does not mean parallel answer generation. Decoder-style generation still advances one next-token step at a time. Fix: measure end-to-end latency on the deployed workflow, not only a training-style kernel.
- A mask is a correctness boundary. A reversed or missing mask can leak future or unrelated context. Fix: test masks with adversarial and cross-record cases.
- Token position is not a character offset. Tokenization, truncation, and positional treatment can change what the model actually receives. Fix: record tokenizer and context-policy revisions with the release.
- KV caches are not shared memory. A cache keyed too broadly can cross requests or policy boundaries. Fix: scope reuse to the exact approved configuration and identity boundary.
- Attention weights are not a proof of explanation. They are model computations, not a causal audit trail for a business decision. Fix: retain source citations, validators, and explicit decision receipts.