The Value of Accurate and Reliable Medical Information
- Artificial intelligence (AI) based technology platform ChatGPT is “not reliable” when it comes to offering solid cancer-related treatment advice according to a new study by researchers at Boston’s Brigham and Women’s Hospital.
- ChatGPT is powered by models that try to predict answers to questions from analyzing large bodies of language and data. Sometimes these predictions are wrong.
- The study found that while some of ChatGPT’s treatment recommendations met National Comprehensive Cancer Network (NCCN) guidelines, one-third of its responses did not.
- Patients seeking accurate, reliable, and understandable medical information should first consult with their doctor who relies on proven evidence-based medicine.
- SurvivorNet information is vetted by experts and always based on evidence from research that has been reviewed by experts in the field. Here is how we do it: https://www.survivornet.com/articles/survivornet-gives-you-information-straight-from-the-experts/
Concerns artificial intelligence-based technology such as ChatGPT will change proven methods of gathering cancer information lost some momentum thanks to a new study that found while it can provide accurate information, it tends to lack adequate nuance or produces incomplete information that needs further explanation.
Researchers at Brigham and Women’s Hospital in Boston tested the AI-based platform to see how well it could provide information regarding breast and lung cancer. The results further reassure practitioners in the oncology field that evidence-based medicine is likely to remain the tried-and-true approach. At SurvivorNet, we always provide factual medical information that derives from evidence-based medicine practiced by our experts.
Read More
Researchers found that “one-third of treatments recommended by the chatbot were at least partially nonconcordant with NCCN guidelines; recommendations varied based on how the question was posed.” Disagreement among annotators highlighted the challenges of interpreting the chatbot’s responses due to clunky descriptive large language models (LLC) which mimic human language.
At the conclusion of the study, researchers urge doctors to tell patients that AI-based chatbots “are not a reliable source of treatment information.” AI is likely to mix “incorrect recommendations among correct ones” which can be difficult to detect even for experts.
If Artificial Intelligence Isn’t Reliable for Cancer Information, Where to Look?
If you’re facing a possible cancer diagnosis, your emotions tend to run high and amid your anxiety, you likely want to learn more about the disease. While the internet provides a host of information, it’s important to look for credible sources.
Your quest should begin with your doctor. Ask your oncologist questions because doing so not only helps ease your anxiety you also learn the next steps in your journey to recovery. In addition to talking to your doctor, you can further your understanding of your diagnosis and possible treatment by determining if the information you’ve found is attributable to evidence-based medicine.
Evidence-based medicine “integrates clinical experience and patient values with the best available research information,” as described by researchers who published Evidence Base Medicine – New Approaches and Challenges in the Journal of Academy of Medical Sciences of Bosnia and Herzegovina.
SurvivorNet has a wealth of resources available for patients looking for reliable information on a host of cancers and chronic diseases. Our experts help explain the nuances of cancers, how they behave and the impact possible treatments can have. However, to further evaluate the usefulness of ChatGPT, Survivornet tested its ability to provide expert, user-friendly cancer information to readers seeking information like you.
We asked ChatGPT to tell us the top 10 questions people ask regarding breast cancer. We also asked the AI platform to answer those 10 questions. We then compared the responses with one of our breast cancer experts Dr. Connie Lehman, a diagnostic radiologist who specializes in breast cancer at Mass General Cancer Center.
After reviewing the questions and answers, Dr. Lehman said the text was “accurate” and “reads quite well.”
“Overall, it’s higher quality than a lot of the content you can find on the internet,” Lehman told SurvivorNet.
Perhaps most notably, Dr. Lehman found it encouraging that ChatGPT urged prospective breast cancer patients to talk to their doctor about managing treatment side effects.
“This is a nice example of where ChatGPT could be really helpful in healthcare — taking a diverse group of data and translating it into accessible language for patients,” Dr. Lehman adds.
Dr. Lehman stressed that “proper medical review and oversight [of the text] would be necessary” from a human before it’s given to readers. Two good examples of the necessity for this are the chatbot’s answer to breast cancer survival rates and its referencing of immunosuppression in treatment side effects. While the answers were technically accurate, additional context should be added so readers can better understand the full scope of the answer.
Leading up to this experiment regarding breast cancer, SurvivorNet received factually inaccurate answers from ChatGPT about other questions outside of this experiment.
READ MORE on SurvivorNet’s ChatGPT Experiment.
The Impact of Comprehensive Cancer Centers
Major cancer centers across the nation house leading cancer experts which include surgical oncologists, medical oncologists, pathologists, and radiologists among other cancer-related specialists. At these specialized hospitals, these experts convene to form a tumor board where they discuss cancer patients’ individual cases in great detail. They all work in conjunction and optimize your treatment plan.
“There are some advantages to being treated in a university setting or a major cancer center,” Dr. Ken Miller tells SurvivorNet.
Dr. Miller is Director of Outpatient Oncology at the University of Maryland, Greenebaum Cancer Center, and a member of the Thoracic Oncology Group.
“Pretty much automatically there’s going to be a team approach, surgical oncology, medical, and radiation oncology, and all the support services, and also wonderful pathology and radiology and all the services you need,” Dr. Miller continued.
The National Cancer Institute provides a list of all cancer centers in the United States.
Learn more about SurvivorNet's rigorous medical review process.