Finetuning of GLIDE stable diffusion model for AI-based text-conditional image synthesis of dermoscopic images

Frontiers in Medicine

Medicine

Jakob Wollborn¹  Babak Saravi²  Elisabeth Roider³  Stefan Hartmann⁴  Andreas Vollmer⁴  Alexander Kübler⁴  Michael Vollmer⁵  Gernot Lang⁶  Christos C. Zouboulis⁷  Christian Stoll⁸  Veronika Shavlokhova⁸ 

[1] Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, United States;Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, United States;Department of Orthopedics and Trauma Surgery, Medical Centre-Albert-Ludwigs-University of Freiburg, Faculty of Medicine, Albert-Ludwigs-University of Freiburg, Freiburg, Germany;Department of Dermatology, University Hospital of Basel, Basel, Switzerland;Department of Oral and Maxillofacial Plastic Surgery, University Hospital of Würzburg, Würzburg, Germany;Department of Oral and Maxillofacial Surgery, Tuebingen University Hospital, Tuebingen, Germany;Department of Orthopedics and Trauma Surgery, Medical Centre-Albert-Ludwigs-University of Freiburg, Faculty of Medicine, Albert-Ludwigs-University of Freiburg, Freiburg, Germany;Departments of Dermatology, Venereology, Allergology and Immunology, Staedtisches Klinikum Dessau, Medical School Theodor Fontane and Faculty of Health Sciences Brandenburg, Dessau, Germany;Maxillofacial Surgery University Hospital Ruppin-Fehrbelliner Straße Neuruppin, Neuruppin, Germany;

关键词: GLIDE;    text-to-image;    stable diffusion;    dermoscopy;    cancer;    dermatology;   

DOI  :  10.3389/fmed.2023.1231436

 received in 2023-05-30, accepted in 2023-10-09,  发布年份 2023

来源: Frontiers

【 摘 要 】

BackgroundThe development of artificial intelligence (AI)-based algorithms and advances in medical domains rely on large datasets. A recent advancement in text-to-image generative AI is GLIDE (Guided Language to Image Diffusion for Generation and Editing). There are a number of representations available in the GLIDE model, but it has not been refined for medical applications.MethodsFor text-conditional image synthesis with classifier-free guidance, we have fine-tuned GLIDE using 10,015 dermoscopic images of seven diagnostic entities, including melanoma and melanocytic nevi. Photorealistic synthetic samples of each diagnostic entity were created by the algorithm. Following this, an experienced dermatologist reviewed 140 images (20 of each entity), with 10 samples originating from artificial intelligence and 10 from original images from the dataset. The dermatologist classified the provided images according to the seven diagnostic entities. Additionally, the dermatologist was asked to indicate whether or not a particular image was created by AI. Further, we trained a deep learning model to compare the diagnostic results of dermatologist versus machine for entity classification.ResultsThe results indicate that the generated images possess varying degrees of quality and realism, with melanocytic nevi and melanoma having higher similarity to real images than other classes. The integration of synthetic images improved the classification performance of the model, resulting in higher accuracy and precision. The AI assessment showed superior classification performance compared to dermatologist.ConclusionOverall, the results highlight the potential of synthetic images for training and improving AI models in dermatology to overcome data scarcity.

【 授权许可】

Unknown   
Copyright © 2023 Shavlokhova, Vollmer, Zouboulis, Vollmer, Wollborn, Lang, Kübler, Hartmann, Stoll, Roider and Saravi.