On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML

Summary

The article evaluates ChatGPT's ability to generate UML class diagrams and OCL constraints, revealing significant shortcomings compared to its code generation performance.

Why it matters

Understanding the limitations of large language models (LLMs) in software modeling is crucial as these tools become an integral part of development processes. The study highlights that current LLMs struggle with complex modeling concepts, consistency, and scalability; this can hinder their adoption in professional modeling environments.

Highlights

• ChatGPT can only handle small UML models and fails at fundamental concepts such as association classes or multiple inheritance.
• Generated diagrams exhibit high variability and syntactic errors; for example, both ends of a relationship incorrectly contain composition symbols.
• OCL constraint generation performs well; likely due to its similarity to SQL and extensive training data.

Critical take

The study focuses on a single ChatGPT version and specific prompts, potentially missing the full capabilities of newer LLMs or alternative interaction strategies, which limits the generalizability of its findings.