Agweyu LLM
2026
Generative AI-Enabled Clinical Decision Support System in Primary Care: A Pragmatic, Cluster-Randomized Trial
In 9,691 patients at primary care facilities in Kenya, an LLM decision support system in the medical record did not reduce treatment failure within 14 days compared with the electronic medical record alone.
Result
Efficacy: Expert-adjudicated composite of treatment-failure events within 14 days of enrolmentaOR 0.77 (95% CI 0.55-1.08) · P = 0.13
Below 1 favours an LLM decision support system in the medical record, above 1 favours the electronic medical record alone.
- Absolute difference
- 0.2 points
The absolute difference is derived from the event rates.
Also reported: No serious adverse events judged related to the intervention and no safety signal on independent review.
- Subtopic
- Standalone consultation
- Design
- Pragmatic cluster RCT, clinical officers randomised, 16 primary care facilities, Kenya
- Relevance
- Practice-defining: a trial the field needs to know.
- Source
- Agweyu A, et al. Nat Med. 2026;32(8):3032-3039.doi 10.1038/s41591-026-04503-6
Discussion & critique
The first large pragmatic randomised trial of an LLM in real-world, low-resource primary care, and a null result on the patient-level primary outcome, with any benefit 'probably modest' in the authors' words. Reassuring safety, but a 2% event rate leaves little power, follow-up was 14 days, and the randomised units were clinicians rather than patients. A caution against extrapolating vignette-based LLM gains (Goh GPT-4 (management reasoning)) to patient outcomes.