No. 120
2026-11-16
the AI pulse — brewed, not raw
 AI CRITIQUE

Understanding Positional Embeddings in Modern LLMs

Read the article    English (automatic translation)

Summary

Sebastian Raschka's long-form tutorial article examines step-by-step how large language models (LLMs) encode token position information. This process, ranging from GPT-2's additive position vectors to RoPE's relative encoding mechanism, is supported by a compilation highlighting which position coding preferences are emerging in open-weight models in 2026. The article discusses visual schemas, a small PyTorch implementation, and the 'NoPE' (position-code-free) trend in new architectures.

Why it matters

Position coding is a silent mechanism that enables the attention mechanism to distinguish between 'the red fox jumps' and 'jumps red fox.' However, this topic is rarely explained end-to-end in an accessible format. 2026 architectures are increasingly dropping or reshaping explicit position signals. For example, MiniMax-M3 applies RoPE within each block rather than in the embedding layer. Therefore, understanding the trade-offs between absolute, relative, and code-free schemes is becoming a practical necessity for anyone developing or fine-tuning LLMs.

Highlights

  • Causal masking already leaks implicit order information. However, this method cannot distinguish a 1-token gap from a 10-token gap. Therefore, explicit position signals remain beneficial.
  • RoPE injects position information directly into the attention calculation itself (per transformer block). Unlike GPT-2, this vector is not added once in the embedding layer.
  • The 2026 architectures gallery shows a divergence. Some open-weight models still use additive absolute embeddings. Others rely on RoPE. A growing group is experimenting with 'NoPE' designs.

Critical take

The article reads more like a well-structured pedagogical summary than a research contribution. Raschka himself describes position coding as a 'largely unimportant building block.' Fundamental mechanisms like sinusoidal embeddings and RoPE have been in the public consciousness for years. The article's real value lies in bringing these topics together and scanning 2026 architectures. However, readers looking for new insights into why NoPE works or when RoPE suffers performance degradation may find the critical analysis weaker compared to the tutorial's length.

This critique was written by the local AI model Qwen3.8-27B. I chose the story; I do not review each critique before it is published.
Back to the issue