Data Compression and Large Language Models Rely on the Same Prediction Principle

Summary

The article argues that data compression and large language models (LLMs) are two sides of the same coin. Both aim to minimize bits per symbol by predicting the next symbol. This unification provides the concept of entropy from an information theory perspective. The text bases this argument on a post in the ngrok developer blog and a 2023 paper by Google DeepMind.

Why it matters

Reframing LLMs as advanced compressors rather than purely generative systems helps us better understand what these models optimize (the probability of the next token) and why they excel at tasks requiring local prediction while struggling with those requiring global structure. It also grounds the often-mysterious 'emergent' capabilities of large models in a well-understood mathematical framework. This makes the models' strengths and failure modes more predictable.

Highlights

• Arithmetic coding example: The sequence 'ABABAAC' was compressed from 56 bits to 10 bits by narrowing the probability interval.
• GPT-2 compressed Dickens' text to 10% of its original size, whereas a simple 1st-order model achieved a 24% compression ratio.
• Practical hurdle: The computational power spent on LLM inference usually exceeds the number of bytes saved. Consequently, these models rarely replace gzip/Brotli in production environments.

Critical take

This text is essentially a summary of a single ngrok blog post. The core claim that language modeling and compression are the same problem dates back to Shannon in 1948 and was explicitly stated by DeepMind in 2023. Therefore, this 'observation' is at best a pedagogical repackaging. The paragraph on practical limitations is weak. The text claims that inference costs are dominant but does not quantitatively specify the model sizes that would balance this or provide a concrete threshold value. This leaves the reader with nothing but a vague 'too expensive' statement.