DeepMind Just Changed How AI Sees The World

Summary

DeepMind's Gemma 4 model (12 billion parameters) gains multimodal perception capabilities by eliminating separate image and audio encoders. The model feeds raw image patches and 40-millisecond audio fragments directly into a single transformer, enabling it to learn vision, hearing, and reasoning simultaneously. The model runs locally on a laptop, and its architectural details have been shared with the public.

Why it matters

Removing specialized encoder networks and having a single transformer handle both perception and reasoning could reduce hundreds of millions of parameters from multimodal workflows. This may enable access to capable image/audio AI without requiring cloud-scale hardware. Publishing the architecture offers a roadmap for the open-source community to develop more efficient local models, potentially accelerating research in resource-constrained environments.

Highlights

• Directly feeding patches into the main transformer bypasses separate image/audio encoders, blurring the line between perception and reasoning
• The 12-billion-parameter model processes image, audio, and text while running on a standard laptop
• DeepMind provides a reusable template for other open-model developers by publishing the architecture

Critical take

The phrase '99% smaller' is a misleading comparison between a 12-billion-parameter local model and a 1.6-trillion-parameter frontier model. These two models serve fundamentally different use cases. Additionally, the 'secret formula' narrative exaggerates the situation, which is an incremental architectural improvement rather than a paradigm-shifting revolution. The video is a sponsored explainer by Two Minute Papers, featuring a Lambda ad. The 300 million download figure is presented without clarifying whether it covers all Gemma versions or only Gemma 4.