Three novel approaches for measuring AI performance

Summary

A Science article led by José Hernández-Orallo (Cambridge/UPV) and co-authored by researchers from Princeton, MIT, and Google DeepMind argues that the Turing test and leaderboard comparisons have lost their validity. Instead, the paper proposes three new evaluation axes: capability 'nutrition labels,' measurable impacts on human cognition, and systematic societal impact.

Why it matters

As AI systems become widespread and integrated into daily business operations, the industry's current framework of 'which model scores highest in benchmark X?' obscures real risks. These risks include cognitive dependency, uneven labor displacement, and security gaps that only emerge at scale. The article calls for a multi-dimensional certification process, similar to those in aviation or pharmaceuticals. This demand poses a direct challenge to a sector that currently self-regulates with limited public funding.

Highlights

• The 'nutrition label' concept: Explaining what AI can and cannot do, rather than scoring it on a single ranking scale. This approach borrows the logic of species niches from biology.
• The second axis shifts the subject: It measures whether people working with AI think better or become cognitively dependent. The authors warn that anecdotal examples of atrophy and psychosis cases will turn into causal evidence within a few years.
• The third axis demands simulated multi-agent environments to capture societal externalities (job loss, power concentration) before deployment. It notes that regulated testing will increase costs and narrow the number of viable model developers.

Critical take

The framework is conceptually appealing but operationally vague. The article does not provide concrete metrics, validation protocols, or governance architecture for the proposed 'labels.' In the cognitive axis, reliance on conversation logs held by companies makes independent verification structurally impossible. The closing paragraph, which demands 'ten times more resources' from EU institutions, reads more like a policy wishlist than a research contribution. While the flying/insect analogy is rhetorically elegant, it postpones the question of who will define the relevant capability dimensions in a geopolitically fragmented environment.