Comparative analysis of large language models and clinicians in thyroid eye disease using structured questionnaires

Scientific Reports

卷:16

Article

Shiqi Hui¹  Lihua Luo²  Zhijia Hou¹  Dongmei Li^1,3 

[1] Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University, 1 Dong Jiao Min Lane, 100730, Beijing, China

[2] Institute of Digital Ophthalmology and Visual Science, Changsha Aier Eye Hospital, Hunan, China

[3] Department of Ophthalmology, Beijing Friendship Hospital, Capital Medical University, Beijing, China

关键词: Thyroid eye disease;    Large language model;    Artificial intelligence;    Diagnostic agreement;    Pilot survey;   

DOI  :  10.1038/s41598-026-48930-4

 received in 2025-11-30, accepted in 2026-04-10,  发布年份 2026

来源: Springer

【 摘 要 】

To preliminarily evaluate agreement between large language models (LLMs) and ophthalmic clinicians regarding thyroid eye disease (TED) clinical decision-making, we developed a structured questionnaire for TED-related scenarios. A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians’ approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro). Agreement between LLMs and clinicians was assessed using similarity metrics, including Jaccard index and cosine similarity. Inter-physician agreement was additionally calculated as a reference baseline. The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician’s experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (p < 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (p < 0.001). LLMs exhibit partial alignment with clinician response patterns in a structured, questionnaire-based setting. However, these findings reflect performance under simplified and decontextualized conditions, further studies incorporating real patient data and open-ended clinical tasks are needed.

【 授权许可】

CC BY   
© The Author(s) 2026

【 图 表 】

Fig. 1

9

Fig. 1

Fig. 3