Study: Large language models cannot distinguish social norms

Summary

This article presents a multi-agent debate benchmark using synthetic norms to test whether large language models (LLMs) can infer socially enforced rules solely from interaction. The study reveals that base models fail to learn these norms; furthermore, even specially designed 'normative modules' exhibit weak generalizability and are unable to distinguish enforced rules from arbitrary noise.

Why it matters

As LLMs are increasingly deployed in autonomous and multi-stakeholder environments (customer service, collaborative work, governance), the assumption that these systems can 'read the room' and adapt to local social norms is critical for safe alignment. Demonstrating that current architectures conflate genuine norms with random behavioral noise highlights a concrete risk: systems may enforce arbitrary habits or miss actual community standards. This undermines trust and coordination in human-AI interaction.

Highlights

• Base LLM agents fail to adopt synthetic norms, even in scenarios where compliance directly increases accuracy scores.
• Norm compliance performance is highly sensitive to the stylistic framing of the norm and which model executes the inference module, indicating a lack of robust generalizability.
• LLMs exhibit 'unselective attribution errors': they copy arbitrary and non-normative behaviors alongside genuine rules. This persists even when unnecessary actions are explicitly marked as penalized.

Critical take

The entire evaluation relies on a synthetic, closed multi-agent debate based on norms defined by the researchers. Consequently, the observed 'failure' may stem from the artificiality of the setup rather than a fundamental deficiency. Additionally, the authors' claim of presenting 'the first study to concretize normative competence' contradicts the extensive existing literature on social norm inference, rule derivation, and compliance in language models, suggesting that the emphasis on novelty is exaggerated.