Ontologies are insufficient for catching errors in AI agents

Summary

This article tests whether carefully curated, LLM-generated, or data-derived ontologies can reliably catch errors in LLM agents that query knowledge graphs. The results show that carefully curated ontologies typically flag correct answers rather than incorrect ones and perform only marginally better than random labeling on real agent outputs.

Why it matters

The industry is rapidly wrapping LLM agents with ontology-based validation layers, assuming that a formal schema guarantees correctness. However, if the 'referee' itself is unreliable—contradicting its own data and producing false alarms at near-random rates—the entire validation pipeline is built on sand. This situation incurs real engineering costs and leads to a decline in agent performance.

Highlights

• The carefully curated UniProt ontology flags more correct answers compared to data-derived or LLM-generated alternatives.
• In real agent responses, flags from the carefully curated ontology are only marginally more accurate than random flags.
• A published, carefully curated schema can contradict its own underlying data, undermining its authority as a controller.

Critical take

The finding that carefully curated ontologies perform nearly randomly as error detectors is striking. However, the article's scope—specifically UniProt and a few public graphs—limits generalizability. More concerning is the possibility that this near-random result is partly explained by the tested agents already being quite accurate. In such cases, the controller's task becomes extremely difficult, as it must distinguish errors from noise; the authors fail to clarify this distinction precisely.