SummaryA team from MIT and Dana-Farber scanned approximately 4,000 clinical prediction studies using large language models (LLMs) and examined hundreds of linked code repositories. The study found that only one in eight studies shared code, and even when code was shared, it frequently lacked the documentation necessary for genuine reuse. Published in Nature Medicine, these findings form the basis for the development of TRIPOD-Code, a new reporting guideline for code sharing in medical research. |
Why it mattersClinical prediction models directly influence patient care decisions. The inability to audit how these models were developed undermines trust in medical AI. The study quantifies a transparency gap not addressed by existing guidelines (TRIPOD, TRIPOD+AI). Conducted at the scale of thousands of articles and hundreds of repositories, this review makes possible a task that would be impractical to perform manually. By establishing an evidence base for what 'good' code sharing looks like, it offers concrete expectations to journals and authors instead of vague incentives. |
Highlights- Only about 12.5% of studies reported code sharing; sharing rates varied widely across journals, ranging from 0% to over 70%.
- Even when code was publicly available, documentation, software requirements, and reuse conditions were frequently missing or inconsistent.
- LLMs enabled large-scale scanning of unstructured text across articles and repositories, while human reviewers verified a sample of 500 articles.
|
Critical takeReliance on LLMs to analyze fragmented code-sharing declarations in publications and repositories creates classification ambiguity, a limitation also noted by the authors. Agreement levels were weaker on documentation adequacy, the most critical dimension for reuse. This implies that the disparity rates highlighted in the title carry a significant margin of error. Additionally, the study was limited to open-access articles in PubMed Central. Therefore, the actual sharing rate in paywalled or non-indexed venues remains unknown. |
This critique was written by the local AI model
Qwen3.8-27B. I chose the story; I do not review each critique before it is published.