in 2015, the BMJ (formerly British medical journal) did a thorough audit of online symptom checkers. It found that, on average, the sites only listed the correct diagnosis first about a third of the time. Carl Shen, an ophthalmology resident at McMaster University, led a team of researchers in a small-scale follow-up that looked specifically at eye health and got equally worrying results: Only a quarter of the time, the correct diagnoses came first. forward.
The results, presented at the American Academy of Ophthalmology’s annual meeting this week, are early and preliminary, but Dr. Shen and his team are planning a larger follow-up study. In the meantime, WebMD has done an update to its algorithm.
Vignettes of unpleasantness
To assess the accuracy of WebMD, Shen and colleagues compiled 42 “clinical vignettes” for eye health based on the medical literature. A decidedly unpleasant vignette from someone suffering from acute angle-closure glaucoma, for example, describes a “44-year-old woman who attended[ing] to the emergency room… with severe pain around her right eye for four hours… She is also nauseous and has vomited once… The intraocular pressure is extremely elevated.’
These vignettes were then boiled down to barebones symptoms so that they could easily be entered into the symptom checker twice: once by a medical professional and once by a non-professional. That helped to consider both how non-specialists would interact with the tool and how different people might describe the same condition in different ways. Because the diagnoses of the clinical cases were already known, the outcome of the symptom checker could be compared with the known, correct diagnosis.
Of the 42 cases, the correct diagnosis appeared somewhere in the list 18 times – just under half. Only in 11 cases did the correct result appear at the top of the list. And because the main diagnosis was usually wrong, the recommended course of action was often wrong. Of the 18 vignettes based on a clinical emergency, the symptom checker correctly advised necessary emergency care in only seven of them.
These results are slightly worse than the 2015 findings BMJ research, which reported that the correct diagnosis appeared in the top 20 symptom control options in 58 percent of cases and appropriate emergency care recommendations in 80 percent of medical emergencies. But the ophthalmic study of Dr. Shen is also much smaller than the… BMJ‘s, which looked at 23 different symptom checkers and collected results from many different people entering the same symptoms. So the ophthalmic examination may be less reliable. Shen and his team are also much earlier in the research process than the BMJ research, and they are considering crowdsourcing as a way to capture how different people would describe the same condition differently.
It is also important to note that in the time since both studies were conducted, WebMD has updated its algorithm, in part due to the BMJ Research. Where the old symptom checker required users to start by clicking on a body part, the new version only requires a description of the symptoms. “After testing the new version of the WebMD Symptom Checker, our analysis of 77 different cases showed a 70 percent accuracy rate for the condition that ranks in the top 3,” said WebMD’s medical director Michael Smith.
Independent assessment of that success rate and an exploration of whether it looks the same in different areas of health could follow in future research. As it stands, the picture may be a little brighter than Shen and his colleagues suggest, but it’s clear that online symptom checkers shouldn’t be used alone to make medical decisions. They are “intended for informational purposes only,” Dr. Smith says, stressing that physicians “should engage their patients in conversation to help them understand how best to use the information.”
Helping doctors have that conversation from an informed starting point is exactly why Shen was interested in the question. With more patients arriving at the doctor’s office who have self-assessed their symptoms, he and his colleagues write in their conference presentation, “It’s important for ophthalmologists to be aware of the capabilities and limitations of available health information tools.”