Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study
Abstract BackgroundThere is breast cancer–related medical information on social media, but there is no established method for objectively evaluating the quality of this information. Principles for Health-Related Information on Social Media (PRHISM) is a newly developed tool fo...
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-06-01
|
| Series: | JMIR Infodemiology |
| Online Access: | https://infodemiology.jmir.org/2025/1/e66416 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849422460488777728 |
|---|---|
| author | Hiroki Kusama Yoshimitsu Takahashi Shunichiro Orihara Kayo Adachi Yumiko Ishizuka Ryoko Semba Hidetaka Shima Yoshiya Horimoto Hiroshi Kaise Masataka Taguri Sho Inoue Takeo Nakayama Takashi Ishikawa |
| author_facet | Hiroki Kusama Yoshimitsu Takahashi Shunichiro Orihara Kayo Adachi Yumiko Ishizuka Ryoko Semba Hidetaka Shima Yoshiya Horimoto Hiroshi Kaise Masataka Taguri Sho Inoue Takeo Nakayama Takashi Ishikawa |
| author_sort | Hiroki Kusama |
| collection | DOAJ |
| description |
Abstract
BackgroundThere is breast cancer–related medical information on social media, but there is no established method for objectively evaluating the quality of this information. Principles for Health-Related Information on Social Media (PRHISM) is a newly developed tool for objectively assessing the quality of health-related information on social media; however, there have been no reports evaluating its reliability and validity.
ObjectiveThe purpose of this study was to statistically examine the reliability and validity of PRHISM using videos about breast cancer treatment on YouTube (Google).
MethodsIn total, 60 YouTube videos were selected on January 5, 2024, with the Japanese words for “breast cancer,” “treatment,” and “chemotherapy,” and assessed by 6 Japanese physicians with expertise in breast cancer. These evaluators independently evaluated the videos using PRHISM and an established tool for assessing the quality of health-related information, DISCERN, as well as through subjective assessments. We calculated interrater and intrarater agreement among evaluators with CIs, measuring agreement using weighted Cohen kappa.
ResultsThe interrater agreement for PRHISM overall quality was κ=0.52 (90% CI 0.49-0.55), indicating that the expected level of agreement, statistically defined by the lower limit of the 90% CI exceeding 0.53, was not achieved. However, PRHISM demonstrated higher agreement compared with DISCERN overall quality, which had a κ=0.45 (90% CI 0.41-0.48). In terms of validity, the intrarater agreement between PRHISM and subjective assessments by breast experts was κ=0.37 (95% CI 0.14-0.60), while DISCERN showed an agreement of κ=0.27 (95% CI 0.07-0.48), indicating fair agreement and no significant difference in validity.
ConclusionsPRHISM has demonstrated sufficient reliability and validity for evaluating the quality of health-related information on YouTube, making it a promising new metric. To further enhance objectivity, it is necessary to explore the use of artificial intelligence and other approaches. |
| format | Article |
| id | doaj-art-cdb624eb2d9d42a0ad9233375f0d097f |
| institution | Kabale University |
| issn | 2564-1891 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Infodemiology |
| spelling | doaj-art-cdb624eb2d9d42a0ad9233375f0d097f2025-08-20T03:31:06ZengJMIR PublicationsJMIR Infodemiology2564-18912025-06-015e66416e6641610.2196/66416Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation StudyHiroki Kusamahttp://orcid.org/0000-0002-2163-7989Yoshimitsu Takahashihttp://orcid.org/0000-0003-4073-9945Shunichiro Oriharahttp://orcid.org/0000-0003-0168-1250Kayo Adachihttp://orcid.org/0000-0002-1425-2271Yumiko Ishizukahttp://orcid.org/0000-0002-1588-3067Ryoko Sembahttp://orcid.org/0009-0007-8670-4357Hidetaka Shimahttp://orcid.org/0000-0002-4913-1648Yoshiya Horimotohttp://orcid.org/0000-0001-8935-0768Hiroshi Kaisehttp://orcid.org/0000-0002-6638-783XMasataka Tagurihttp://orcid.org/0000-0001-8902-0056Sho Inouehttp://orcid.org/0009-0003-2434-9899Takeo Nakayamahttp://orcid.org/0000-0002-7918-6252Takashi Ishikawahttp://orcid.org/0000-0003-3450-4533 Abstract BackgroundThere is breast cancer–related medical information on social media, but there is no established method for objectively evaluating the quality of this information. Principles for Health-Related Information on Social Media (PRHISM) is a newly developed tool for objectively assessing the quality of health-related information on social media; however, there have been no reports evaluating its reliability and validity. ObjectiveThe purpose of this study was to statistically examine the reliability and validity of PRHISM using videos about breast cancer treatment on YouTube (Google). MethodsIn total, 60 YouTube videos were selected on January 5, 2024, with the Japanese words for “breast cancer,” “treatment,” and “chemotherapy,” and assessed by 6 Japanese physicians with expertise in breast cancer. These evaluators independently evaluated the videos using PRHISM and an established tool for assessing the quality of health-related information, DISCERN, as well as through subjective assessments. We calculated interrater and intrarater agreement among evaluators with CIs, measuring agreement using weighted Cohen kappa. ResultsThe interrater agreement for PRHISM overall quality was κ=0.52 (90% CI 0.49-0.55), indicating that the expected level of agreement, statistically defined by the lower limit of the 90% CI exceeding 0.53, was not achieved. However, PRHISM demonstrated higher agreement compared with DISCERN overall quality, which had a κ=0.45 (90% CI 0.41-0.48). In terms of validity, the intrarater agreement between PRHISM and subjective assessments by breast experts was κ=0.37 (95% CI 0.14-0.60), while DISCERN showed an agreement of κ=0.27 (95% CI 0.07-0.48), indicating fair agreement and no significant difference in validity. ConclusionsPRHISM has demonstrated sufficient reliability and validity for evaluating the quality of health-related information on YouTube, making it a promising new metric. To further enhance objectivity, it is necessary to explore the use of artificial intelligence and other approaches.https://infodemiology.jmir.org/2025/1/e66416 |
| spellingShingle | Hiroki Kusama Yoshimitsu Takahashi Shunichiro Orihara Kayo Adachi Yumiko Ishizuka Ryoko Semba Hidetaka Shima Yoshiya Horimoto Hiroshi Kaise Masataka Taguri Sho Inoue Takeo Nakayama Takashi Ishikawa Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study JMIR Infodemiology |
| title | Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study |
| title_full | Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study |
| title_fullStr | Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study |
| title_full_unstemmed | Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study |
| title_short | Assessing the Reliability and Validity of Principles for Health-Related Information on Social Media (PRHISM) for Evaluating Breast Cancer Treatment Videos on YouTube: Instrument Validation Study |
| title_sort | assessing the reliability and validity of principles for health related information on social media prhism for evaluating breast cancer treatment videos on youtube instrument validation study |
| url | https://infodemiology.jmir.org/2025/1/e66416 |
| work_keys_str_mv | AT hirokikusama assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT yoshimitsutakahashi assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT shunichiroorihara assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT kayoadachi assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT yumikoishizuka assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT ryokosemba assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT hidetakashima assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT yoshiyahorimoto assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT hiroshikaise assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT masatakataguri assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT shoinoue assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT takeonakayama assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy AT takashiishikawa assessingthereliabilityandvalidityofprinciplesforhealthrelatedinformationonsocialmediaprhismforevaluatingbreastcancertreatmentvideosonyoutubeinstrumentvalidationstudy |