| Journal of Orthopaedic Surgery | 卷:34 |
| Automating methodological quality assessment in orthopedic systematic reviews using large language models | |
| Sport | |
| Yu-Jui Huang1  Kai-Cheng Chang2  Ying-Chen Kuo3  Cheng-Chen Tai4  Cheng-Pang Yang1  | |
| [1] Department of Physical Medicine and Rehabilitation, , Taipei, Taiwan | |
| [2] Medical Library, Department of Medical Education, , Taoyuan, Taiwan | |
| [3] Department of Orthopedic Surgery, , Taoyuan City, Taiwan | |
| [4] Department of Pharmacy, , Taoyuan City, Taiwan | |
| 关键词: sports; artificial intelligence; evidence-based; large language model; | |
| DOI : 10.1177/10225536261459518 | |
| received in 2026-01-26, accepted in 2026-05-30, 发布年份 2026 | |
| 来源: Sage Journals | |
PDF
|
|
【 摘 要 】
BackgroundSystematic reviews represent the foundation of evidence-based orthopedic practice, yet their methodological rigor relies heavily on accurate and consistent methodological quality assessment. This step remains time-consuming, labor-intensive, and prone to subjectivity. Recent advances in large language models (LLMs) suggest potential for automating parts of evidence synthesis.PurposeThis study examined whether LLMs can perform AMSTAR-1–based methodological quality assessment evaluations in orthopedic systematic reviews with accuracy comparable to human experts.MethodsTen sports medicine knee reviews were analyzed using three LLMs—GPT-4o, GPT-5, and GPT Consensus—and their binary responses were compared against expert AMSTAR-1 ratings from a published umbrella review (110 decisions). An external validation set of four reviews published between 2022 and 2025 was included to assess generalizability and safeguard against information leakage.ResultsAgreement with human reviewers reached 87% for GPT-4o, 89% for GPT-5, and 90% for GPT Consensus; all models achieved 84% agreement in the validation set. Concordance was strongest for structured, explicitly reported domains such as a priori design, literature search, and study characteristics, and lowest for judgment-based items including grey literature inclusion, publication bias, and conflict of interest.ConclusionsLLMs cannot yet replace human reviewers, they can serve as reliable adjunct tools to enhance efficiency, transparency, and reproducibility in systematic review workflows within orthopedic research.
【 授权许可】
CC BY-NC
© The Author(s) 2026
【 预 览 】
| Files | Size | Format | View |
|---|---|---|---|
| RO202609140010990ZK.pdf | 1872KB | ||
| Figure 12 | 60KB | Image |
【 图 表 】
Figure 12
【 参考文献 】
- [1]
- [2]
- [3]
- [4]
- [5]
- [6]
- [7]
- [8]
- [9]
- [10]
- [11]
- [12]
- [13]
- [14]
- [15]
- [16]
- [17]
- [18]
- [19]
PDF