Can ChatGPT Answer Cancer Questions Accurately?
- We asked AI chatbot ChatGPT to answer common questions on breast cancer prevention, symptoms, risk factors, diagnosis, and treatment.
- Within seconds, the tool produced answers that our expert said were “higher quality” than a lot of the cancer information available online.
- Dr. Connie Lehman, a diagnostic radiologist at Mass General Cancer Center, explained that ChatGPT provided answers that were, for the most part, accurate.
- However, there were certain limitations like the chatbot not providing the necessary context on common side effects of breast cancer treatments.
- While the information ChatGPT provided can help turn complicated medical information into more understandable language, Dr. Lehman stressed that a human reviewer would still be needed to ensure accuracy and that proper context is always present.
Read on to see how accurate that chatbot was at answering your top questions about breast cancer.
Read MoreWhat Exactly Is AI?
You’ve probably heard about artificial intelligence (AI) a lot in the news. You may have already encountered it through your smartphone’s voice assistance, when contacting a customer service chat online, or when your email service sends a junk message to the trash folder. But AI has grown so much beyond these simple applications in recent years and it’s transforming the healthcare industry.Leading researchers and scientists are studying if AI machines can learn to analyze medical scans and images, detect cancer earlier, create customized treatment plans, and even aid in robotic surgeries.
SurvivorNet wanted to see what one AI machine, called ChatGPT, could do when it came to our mission of bringing expert, user-friendly cancer information to readers.
ChatGPT is like an instant-message conversation. The user types a message or question in the window, and the chatbot responds.
We asked ChatGPT to not only tell us what the top 10 questions are that people have about breast cancer, but we also asked it to answer them.
However, we didn’t just want to know if ChatGPT could answer the questions. We wanted to know if it was correct so we got it fact-checked by a breast cancer expert.
More on Breast Cancer
Fact-Checking ChatGPT’s Answers on Breast Cancer
We asked Dr. Connie Lehman, a diagnostic radiologist who specializes in breast cancer at Mass General Cancer Center, to review the text. Relying on her expertise in breast cancer diagnosis and detection, we asked her to assess the accuracy of the AI information.
We wanted to know, too, if using AI to quickly produce health information is actually helpful for readers and patients.
Surprisingly, or perhaps not, Dr. Lehman said the text was “accurate” and “reads quite well.”
“Overall, it's higher quality than a lot of the content you can find on the internet,” Lehman told SurvivorNet.
To see what she means, here is the full Q&A that ChatGPT produced on breast cancer.
1. What are the symptoms of breast cancer?
The symptoms of breast cancer may include a lump or thickening in the breast, changes in breast size or shape, skin changes such as redness, dimpling or puckering, nipple changes such as turning inward, discharge or itching, and breast pain. However, not all breast lumps or changes are cancerous, and some people with breast cancer may not have any symptoms at all.
Verdict: Accurate
2. What are the risk factors for developing breast cancer?
Some of the known risk factors for developing breast cancer include being female, aging, having a family history of breast cancer, having certain gene mutations such as BRCA1 or BRCA2, having dense breast tissue, being overweight or obese, having a personal history of certain types of benign breast conditions or breast cancer, having had radiation therapy to the chest, and using hormone replacement therapy for menopause.
Verdict: Accurate
WATCH: Alochol Can Increase Your Risk of Breast Cancer
3. How is breast cancer diagnosed?
Breast cancer is typically diagnosed through a combination of imaging tests and biopsies. Imaging tests may include a mammogram, ultrasound, or MRI. If an abnormality is found, a biopsy is usually done to determine whether it is cancerous. There are several types of biopsies, including fine needle aspiration, core needle biopsy, and surgical biopsy.
Verdict: Accurate
WATCH: When Should I Get a Mammogram?
4. What are the treatment options for breast cancer?
The treatment options for breast cancer depend on the stage of the cancer and other factors such as the person’s age and overall health. Treatment options may include surgery to remove the cancerous tissue, radiation therapy to destroy cancer cells, chemotherapy to kill cancer cells throughout the body, hormone therapy to block the hormones that can fuel some types of breast cancer, and targeted therapy to attack specific types of cancer cells.
Verdict: Accurate
5. How can breast cancer be prevented?
While there is no guaranteed way to prevent breast cancer, some steps may help reduce the risk. These may include maintaining a healthy weight, exercising regularly, limiting alcohol intake, not smoking, and breastfeeding if possible. For people at high risk of breast cancer, preventive measures such as prophylactic surgery or medications may be recommended. Regular screening tests such as mammograms can also help detect breast cancer early, when it is most treatable.
Verdict: Accurate
WATCH: Getting to Know Your Breasts With Self-Exams
6. What is the survival rate for breast cancer, and what factors affect it?
The survival rate for breast cancer varies depending on the stage at which it is diagnosed and other factors such as the person’s age and overall health. According to the American Cancer Society, the five-year survival rate for localized breast cancer (meaning it has not spread beyond the breast) is close to 100%, while the five-year survival rate for metastatic breast cancer (meaning it has spread to other parts of the body) is around 28%. Factors that can affect survival rates include the size of the tumor, the stage of the cancer, the type of cancer, and whether the cancer is hormone receptor-positive or -negative.
VERDICT: Technically accurate, though some additional context is needed around those percentages.
Citing the American Cancer Society, the AI wrote that the five-year survival rate for localized breast cancer is “close to 100%” and for metastatic breast cancer is “around 28%.”
While this answer is not technically incorrect, it’s important to know that the actual figures listed on the American Cancer Society website are 99% for localized breast cancer and 30% for metastatic breast cancer.
7. What is the difference between a benign and malignant breast lump?
A benign breast lump is a non-cancerous growth in the breast that is not a threat to health and does not require treatment. Examples of benign breast lumps include cysts and fibroadenomas. A malignant breast lump, on the other hand, is a cancerous growth in the breast that has the potential to spread to other parts of the body if left untreated.
VERDICT: Accurate
8. How often should women get mammograms to screen for breast cancer?
The American Cancer Society recommends that women at average risk of breast cancer start getting annual mammograms at age 45, and then switch to getting them every two years starting at age 55. However, women at higher risk of breast cancer may need to start getting mammograms earlier or more frequently, as determined by their doctor.
VERDICT: Accurate
9. What is the role of genetics in the development of breast cancer?
Genetics plays a role in the development of breast cancer, as some people inherit mutations in genes such as BRCA1 and BRCA2 that increase their risk of developing the disease. However, most cases of breast cancer are not caused by inherited gene mutations. Genetic testing may be recommended for people with a family history of breast cancer to determine if they carry a gene mutation that increases their risk.
VERDICT: Accurate
10. What are the potential side effects of breast cancer treatment, and how can they be managed?
The potential side effects of breast cancer treatment can vary depending on the type of treatment used. Surgery may cause pain, swelling, or numbness in the affected breast or arm. Radiation therapy may cause skin changes such as redness or peeling, fatigue, or a decrease in white blood cells. Chemotherapy may cause hair loss, nausea, fatigue, or an increased risk of infection. Hormone therapy may cause hot flashes, vaginal dryness, or joint pain. These side effects can often be managed with medications or other interventions, and it is important for people undergoing treatment to discuss any concerns with their healthcare team.
VERDICT: Technically accurate, but more information is needed for patients to fully understand how they may be affected.
Lehman said the average person may still struggle with some of the information, such as saying that radiation therapy may cause a "decrease in white blood cells." She said this simple phrasing “would likely leave a patient wondering what that means for them.”
So the lesson there is that, in addition to proper medical review and oversight, a human would also need to assure that proper context is given to fully understand the ideas being presented.
For example, having a decrease in white blood cells is known as immunosuppression, or neutropenia, according to the American Cancer Society. It can be caused by “certain diseases, including cancer, and certain treatments, including chemotherapy and radiation therapy.”
Neutropenia can lower your immunity and leave you susceptible to infections, according to Cancer.net. When patients develop this condition, doctors have methods to help prevent infections and restore their immune systems. Typically, white blood cell counts return to normal after treatment.
As you can see, it helps to have all relevant context when learning about aspects of cancer treatment and side effects.
WATCH: Managing Chemo Side Effects
AI Health Info Can Be Accurate, But It Has Limitations
As Dr. Lehman said, the answers produced by ChatGPT were accurate.
And she appreciated that ChatGPT said that breast cancer survival rates and treatment side effects can “vary,” clearly taking into account that each patient has individual circumstances that can affect their response and prognosis.
However, the bot still gave specific examples of treatment side effects for different therapy options. This helped provide context and detailed information for people to refer to, and Dr. Lehman liked that.
READ MORE: An Overview of Breast Cancer Treatment
“I also like that it encourages talking to your doctor and your healthcare team,” Dr. Lehman said, such as for managing treatment side effects.
Considering all this, Dr. Lehman said, “This is a nice example of where ChatGPT could be really helpful in healthcare — taking a diverse group of data and translating it into accessible language for patients.”
This doesn’t mean, though, that the AI-produced copy on breast cancer is perfect.
Dr. Lehman stressed that “proper medical review and oversight [of the text] would be necessary” from a human before it’s given to readers. Two good examples of the necessity for this are the chatbot’s answer to breast cancer survival rates and its referencing of immunosuppression in treatment side effects.
While the answers were technically accurate, additional context is needed to understand the full scope of the answer.
And it should be noted that SurvivorNet did receive factually inaccurate answers from ChatGPT in relation to other questions outside of this experiment.
We experienced ChatGPT giving an incorrect answer to a question that was easily verifiable, such as when a famous public figure retired from their prominent job.
Meanwhile, the free version of ChatGPT is only trained on information up to September 2021, which leaves over a year and a half of new data and current events it doesn’t know about. Because of this, we also saw it admit that it didn’t know the answer to a question because it didn’t have the information necessary.
With this in mind, it would be crucial to have a proper review of all AI copy in a healthcare setting before it ever reaches a patient, as Lehman said. It would be devastating for cancer patients to receive inaccurate or out-of-date information about their specific diagnosis or a certain type of treatment during such a critical point in their lives.
WATCH: Reducing Anxiety About False Positives
Still, given that the AI information is accurate and up-to-date, Lehman said the production of health information in this way fills a need for patients seeking accurate material that they can also understand without having their own medical degree.
“All of us in healthcare struggle with patients who will go to the internet and find a wildly variable level of accuracy in the content they read,” Lehman said.
“We want to direct them to content that is accurate, but also accessible to them and easy to understand. A website for healthcare providers may be accurate but patients might get lost in it. Conversely, you can also find content that tries to be more patient-specific but that's filled with inaccuracies.”
Interestingly (or weirdly), chatbots may also have a leg-up on doctors when it comes to talking to patients with empathy. A recent study published in the journal JAMA Internal Medicine found that chatbots were able to show more empathy than doctors when responding to patient questions.
In the study, doctors and chatbots both responded to common patient questions and a team evaluated the answers based on how much empathy they displayed. Evaluators preferred the response given by the chatbots (rather than the doctors) almost 80% of the time.
How Did ChatGPT Come Up With Its Answers?
The chatbot told us that it is “trained on a vast amount of data from various sources, including medical literature, research studies, and online forums,” and it used information from those sources to produce its answers.
The machine said: “My training data includes information on breast cancer, its causes, risk factors, diagnosis, treatment, and management. When a user asks a question, I use natural language processing techniques to analyze the question and retrieve relevant information from my training data. I then use this information to generate a response that best answers the question.”
ChatGPT produced 10 questions on breast cancer prevention, symptoms, risk factors, diagnosis, and treatment. In just a few seconds, it then provided one-paragraph answers that ranged from 50 to 100 words.
Overall, the answers were clear, easy to read and informational but as Dr. Lehman noted, it was also limited.
Learn more about SurvivorNet's rigorous medical review process.