Introduction and Objective: Adolescents with type 1 diabetes (T1D) require contextually-aware, evidence-based self-management support. General-purpose large language models (LLMs) lack integration with systematically synthesized and appraised evidence or contextual tailoring. This study aimed to construct and evaluate T1DiaBot, a knowledge graph-grounded agent designed for adolescents with T1D.Methods: A self-developed “Evidence-to-Performance” (E2P) framework was followed: 1) Knowledge Graph Construction: The Diabetes Self-Management Knowledge Graph (DSM-KG) was constructed through synthesizing clinical guidelines and reviews, followed by contextual adaptation. 2) Agent Development: T1DiaBot was developed using LangChain, employing a Graph-RAG (Retrieval-Augmented Generation) approach where the DSM-KG serves as the authoritative grounding source. 3) Validation: Clinical validity was assessed via a two-round Delphi experts panel (N=10); usability and safety via a one-month beta-test (N=20 users). 4) Benchmarking: Performance was compared against four general-purpose LLMs (DeepSeek-R1, DeepSeek-V3, Kimi, Doubao) using 81 real-world queries, blindly rated by clinical experts (N=6) and a youth advisory panel (N=27 users).Results: The DSM-KG comprises six sub-graphs (e.g., non-familial setting management, insulin adjustment, parent-child communication, psychosocial support). T1DiaBot achieved high expert consensus (Kendall’s W=0.48-0.63), excellent usability (mean System Usability Scale score: 86.5/100), and no adverse events. It outperformed the four general LLMs in Content Accuracy and Adolescent Scenario Suitability (all P<0.001) and showed consistent superiority over one benchmark (Kimi) in all user experience dimensions (p>0.05).Conclusion: T1DiaBot is a clinically accurate and culturally tailored AI tool for supporting T1D self-management. The E2P framework provides a rigorous methodology for creating evidence-based AI agents in chronic disease care.
J. Yang: None. J. Zhang: None. J. Luo: None. W. Guo: None. H. Zhao: None. S. Yu: None. Z. Liao: None. J. Guo: None.
The Noncommunicable Chronic Diseases-National Science and Technology Major Project (Grants 2023ZD0508200 and 2023ZD0508204), the National Natural Science Foundation of China (Grant 72264037), and Sinocare Diabetes Foundation (Grant 2024SD05)
Source link
Leave a Reply