Sebastian Raschka illustrates modern LLM positional embeddings: from GPT-2 to RoPE and NoPE.
Automated summary · Based on collected summaries; not individually verified
Prepared by HotAI
What happened
Sebastian Raschka publishes a long-form article explaining how LLMs encode token position information, covering GPT-2's learned absolute positional embeddings, the original Transformer's sinusoidal encoding, and RoPE's mechanism of rotating queries and keys within attention, including a PyTorch implementation.
Background
What is a Transformer? Attention mechanisms and model architectures
A Transformer is a neural-network architecture centered on attention, which combines information from different sequence positions. The original sequence-transformation model used an encoder and decoder; later models may use only one type. It describes a computational structure, not a particular trained model or a guarantee of shared capabilities or accuracy.
What is a token? Its relationship to text length, usage, and context budgets
A token is a discrete unit used to process text, potentially a character, part of a word, or punctuation depending on the tokenizer. It is not a fixed number of Chinese characters or English words. Budgets and usage often use tokens, but accounting differs by model and interface. Use the target model’s counting method and inspect reported usage.
Reading notes
The event summary comes from collected excerpts. Background passages quote existing HotAI articles to explain terms and do not establish new facts about this event. Check the original sources for product results, conditions and subsequent updates.
Original source
Sources and collection timeline
| Source | Report title | Original publication time | Collected by HotAI |
|---|---|---|---|
| magazine.sebastianraschka.com | Sebastian Raschka illustrates modern LLM positional embeddings: from GPT-2 to RoPE and NoPE. | 2026-11-15 20:10 | 2026-11-15 21:12 |