How Do You Know Which Medical Information on Wikipedia to Trust?

James Heilman isn't an easy man to get a hold of; he kept offering us odd, off-hour windows of availability to do a phone interview. When we finally connected, he explained: He works the night shift as an emergency room physician in British Columbia. He also puts in time as a clinical assistant professor in emergency medicine.

Then there's the 60 -- count'em, 60 -- hours a week he toils away editing Wikipedia, the massive online encyclopedia written and edited by, well, anyone who wants to give it a whirl.

"My wife likes to joke that Wikipedia's my mistress; I prefer to call it our first child,” he says, chuckling with a tinge of self-awareness that he's gone a bit off the rails in his goal of making the copious health-related pages on the site more accurate.

So how does an emergency doc become an obsessive Wikipedian?

About 10 years ago, he says, during a quiet night shift, he got to browsing one of the medical articles.

"I thought, 'Oh, my God this is really bad,'" he says. "And then I saw the edit button, and I realized, oh, my God, I could fix the internet! I sort of got madly hooked. I’ve been trying to fix the Internet ever since."

Accuracy Questioned

Part of that quest was creating the WikiProject Medicine group, where 320 like-minded editors work on the site's health content. Heilman estimates about half are medical professionals.

Reworking Wikipedia health entries is not a trivial task. A 2014 study found about 25,000 pages of English-language health-related articles. That number is now up to 32,000, Heilman says. The health pages worldwide attracted almost 4.9 billion pageviews in 2013. A 2012 survey of several hundred medical students found 94 percent use the site for health information.

But despite its popularity, the reliability of Wikipedia's medical content has often been questioned. A 2011 review on the accuracy and thoroughness of the site's medical entries found mixed results. Other studies show that the site fell short in gastroenterology and hepatology, drug information,  statins (later improved), and issues of the ear, nose and throat. Other studies found the website provided better information in nephrology and depression and schizophrenia.

But just recently, in The Lancet: Global Health, researchers reported that Wikipedia entries on stillbirths were missing critical information.

And a 2014 study comparing peer-reviewed literature to the site's articles on 10 different widespread diseases and conditions found a significant number of assertions in the Wikipedia material that were unsupported by the evidence.

Heilman doesn't think much of that study; he co-wrote a rebuttal to it with other medical experts/Wikipedians. Still, he acknowledges "there's a lot of work to do" to bring health entries up to snuff. The reality of that is borne out by the small number of health articles -- 63 -- that have received the highest grade by Wikipedia editors. Those articles qualify as featured articles on the encyclopedia's front page. (The next-best level -- "good" -- contains a mere 200 articles.) The criteria for a "featured" article is comprehensiveness, neutrality and accuracy. That means the information has been verified against high-quality sources.

Calls to Action

An article last month on the academic publishing site "The Conversation" by Thomas Shafee, one of the researchers who wrote about Wikipedia's inadequate "stillbirth" entries, is headlined "Wikipedia is already the world’s ‘Dr Google’ -- it’s time for doctors and researchers to make it better."

"Health professionals have a duty to improve the accuracy of medical entries in Wikipedia, because it’s the first port of call for people all over the world seeking medical information," Shafee wrote. "The accuracy of the site is vital, because every medical entry... has the potential for immediate real-world health consequences."

Others agree. University of California, San Francisco psychiatrist Amin Azzam recently co-authored a report published in the journal Academic Medicine titled “Why Medical Schools Should Embrace Wikipedia.”

Improving Wikipedia

Efforts to improve the entire site, including the health section, are underway. Hundreds of students in both the U.S. and Canada, for example, are working as editors through the Wiki Education Foundation, a nonprofit organization based in San Francisco that acts as a bridge between Wikipedia and academia. The organization encourages university faculty to assign students Wikipedia entries as a part of coursework.

"Rather than write a biography of a woman scientist for their class that their instructor would read and throw away and never engage in again," says the foundation's LiAnna Davis, "these students instead have the opportunity to create content for Wikipedia where thousands of people can gain from their work."

Pharmacology students at the University of California, San Francisco, for example, are working to improve entries on popular drugs like the HIV antiviral drug Abacavir. Students in a voice disorders class at McGill University in Montreal are adding information about laryngitis. Students fill in missing information, improve gross inadequacies, and rewrite articles using fewer technical terms to appeal to the general public.

In 2016, 280 science classes at universities in the U.S. and Canada have added content to 3,650 articles and generated 300 new entries receiving more than 102 million views, says Davis.

Heilman points to many efforts to improve articles, and a collaboration with the group Translators Without Borders to rewrite Wikipedia medical content in as many languages as possible. That includes Odia, spoken by tens of millions of people in India that is not included in Google Translate. Heilman also cites as a success the Wikipedia entry for dengue fever, which was published in the peer-reviewed open-access journal Open Medicine.

Federal agencies are also tapping Wikipedia. The National Institutes of Health is creating an online biomedical dictionary for PubMed Health. which provides consumer-friendly research information. The NIH usually creates the definitions by drawing on material in its own collection, but sometimes It will use Wikipedia as a source. Both organizations benefit, as the NIH will flag Heilman if it comes across something inaccurate in the material.

Heilman fervently hopes more health and science academics and professionals will help expand the accuracy of the project.

"I spend a fair amount of time trying to convince my colleagues to come work on Wikipedia," he says. "As a physician I believe all people deserve access to quality health care information."

How to Find Good Health Info

So ... after all is said and done, how should the average person approach looking for health information on the Web?

Azzam, the UCSF psychiatrist who called for medical schools to embrace Wikipedia, would actually first send patients to government sites like the National Institutes of Health, Centers for Disease Control and National Library of Medicine.

"Many of those sites have information targeting 'lay' people," he wrote in an email.  "Additionally there are national advocacy organizations for specific disease and/or health conditions. For example, as a psychiatrist I will refer my patients and their families to the National Alliance on Mental Illness (NAMI)."

For those who do use Wikipedia, he recommends using the "Talk" tab near the top left of every page. That will take you to a rating of how thorough and accurate Wikipedia editors think the page is. He also looks at the references that each assertion of fact in Wikipedia entries are required to link to. (You can do this by clicking on the superscript numbers at the end of a sentence.)

"That gives me some idea of how much to trust the content," Azzam says.

He recommends consulting multiple sources to ferret out any outliers, a method that is "effectively a proxy measure for validity."

If you want to consult material that health professionals use themselves, he recommends UpToDate, PubMed, and peer-reviewed journals.

Health information is on a "continuum," he believes, with subscription-only top tier scholarly journals on one end and the user-created Wikipedia on the other.

"We believe [journal] content to be the scientific truth as we know it because content experts have peer-reviewed the manuscripts," he says. "But the challenge at that extreme is gaining access to that knowledge, especially with high-cost subscription charges."

That is why he's working with students to improve Wikipedia, "pulling that pole toward the other end of the spectrum."

"Since so many consumers of health information online are reading that content, I want to be a part of a movement to make it as high quality as possible!"

As for Heilman, the avid health editor, he also recommends checking multiple, high-quality sources when looking for specific health information, paying attention to what they agree on. But that doesn't mean he's going to give up on improving Wikipedia.

"As a physician, I can help one person at a time," he says. "But as a Wikipedian, I can make a difference in millions and millions of people’s lives."