期刊论文详细信息
Journal of Orthopaedic Surgery 卷:34
Automating methodological quality assessment in orthopedic systematic reviews using large language models
Sport
Yu-Jui Huang1  Kai-Cheng Chang2  Ying-Chen Kuo3  Cheng-Chen Tai4  Cheng-Pang Yang1 
[1] Department of Physical Medicine and Rehabilitation, , Taipei, Taiwan
[2] Medical Library, Department of Medical Education, , Taoyuan, Taiwan
[3] Department of Orthopedic Surgery, , Taoyuan City, Taiwan
[4] Department of Pharmacy, , Taoyuan City, Taiwan
关键词: sports;    artificial intelligence;    evidence-based;    large language model;   
DOI  :  10.1177/10225536261459518
 received in 2026-01-26, accepted in 2026-05-30,  发布年份 2026
来源: Sage Journals
PDF
【 摘 要 】

BackgroundSystematic reviews represent the foundation of evidence-based orthopedic practice, yet their methodological rigor relies heavily on accurate and consistent methodological quality assessment. This step remains time-consuming, labor-intensive, and prone to subjectivity. Recent advances in large language models (LLMs) suggest potential for automating parts of evidence synthesis.PurposeThis study examined whether LLMs can perform AMSTAR-1–based methodological quality assessment evaluations in orthopedic systematic reviews with accuracy comparable to human experts.MethodsTen sports medicine knee reviews were analyzed using three LLMs—GPT-4o, GPT-5, and GPT Consensus—and their binary responses were compared against expert AMSTAR-1 ratings from a published umbrella review (110 decisions). An external validation set of four reviews published between 2022 and 2025 was included to assess generalizability and safeguard against information leakage.ResultsAgreement with human reviewers reached 87% for GPT-4o, 89% for GPT-5, and 90% for GPT Consensus; all models achieved 84% agreement in the validation set. Concordance was strongest for structured, explicitly reported domains such as a priori design, literature search, and study characteristics, and lowest for judgment-based items including grey literature inclusion, publication bias, and conflict of interest.ConclusionsLLMs cannot yet replace human reviewers, they can serve as reliable adjunct tools to enhance efficiency, transparency, and reproducibility in systematic review workflows within orthopedic research.

【 授权许可】

CC BY-NC   
© The Author(s) 2026

【 预 览 】
附件列表
Files Size Format View
RO202609140010990ZK.pdf 1872KB PDF download
Figure 12 60KB Image download
【 图 表 】

Figure 12

【 参考文献 】
  • [1]
  • [2]
  • [3]
  • [4]
  • [5]
  • [6]
  • [7]
  • [8]
  • [9]
  • [10]
  • [11]
  • [12]
  • [13]
  • [14]
  • [15]
  • [16]
  • [17]
  • [18]
  • [19]
  文献评价指标  
  下载次数:0次 浏览次数:0次