Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study
Abstract BackgroundAs the importance of person-generated health data (PGHD) in health care and research has increased, efforts have been made to standardize survey-based PGHD to improve its usability and interoperability. Standardization efforts such as the Patient-Reported Ou...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-06-01
|
| Series: | JMIR Medical Informatics |
| Online Access: | https://medinform.jmir.org/2025/1/e71687 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850100723867451392 |
|---|---|
| author | Sunghoon Kang Hyewon Park Ricky Taira Hyeoneui Kim |
| author_facet | Sunghoon Kang Hyewon Park Ricky Taira Hyeoneui Kim |
| author_sort | Sunghoon Kang |
| collection | DOAJ |
| description |
Abstract
BackgroundAs the importance of person-generated health data (PGHD) in health care and research has increased, efforts have been made to standardize survey-based PGHD to improve its usability and interoperability. Standardization efforts such as the Patient-Reported Outcomes Measurement Information System (PROMIS) and the National Institutes of Health (NIH) Common Data Elements (CDE) repository provide effective tools for managing and unifying health survey questions. However, previous methods using ontology-mediated annotation are not only labor-intensive and difficult to scale but also challenging for identifying semantic redundancies in survey questions, especially across multiple languages.
ObjectiveThe goal of this work was to compute the semantic similarity among publicly available health survey questions to facilitate the standardization of survey-based PGHD.
MethodsWe compiled various health survey questions authored in both English and Korean from the NIH CDE repository, PROMIS, Korean public health agencies, and academic publications. Questions were drawn from various health lifelog domains. A randomized question pairing scheme was used to generate a semantic text similarity dataset consisting of 1758 question pairs. The similarity scores between each question pair were assigned by 2 human experts. The tagged dataset was then used to build 4 classifiers featuring bag-of-words, sentence-bidirectional encoder representations from transformers (SBERT) with bidirectional encoder representations from transformers (BERT)–based embeddings, SBERT with language-agnostic BERT sentence embedding (LaBSE), and GPT-4o. The algorithms were evaluated using traditional contingency statistics.
ResultsAmong the 3 algorithms, SBERT-LaBSE demonstrated the highest performance in assessing the question similarity across both languages, achieving area under the receiver operating characteristic and precision-recall curves of >0.99. Additionally, SBERT-LaBSE proved effective in identifying cross-lingual semantic similarities. The SBERT-LaBSE algorithm excelled at aligning semantically equivalent sentences across both languages but encountered challenges in capturing subtle nuances and maintaining computational efficiency. Future research should focus on testing with larger multilingual datasets and on calibrating and normalizing scores across the health lifelog domains to improve consistency.
ConclusionsThis study introduces the SBERT-LaBSE algorithm for calculating the semantic similarity across 2 languages, showing that it outperforms BERT-based models, the GPT-4o model, and the bag-of-words approach, highlighting its potential in improving the semantic interoperability of survey-based PGHD across language barriers. |
| format | Article |
| id | doaj-art-b3942cd968a040aab5f584b21b6df7fa |
| institution | DOAJ |
| issn | 2291-9694 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Informatics |
| spelling | doaj-art-b3942cd968a040aab5f584b21b6df7fa2025-08-20T02:40:14ZengJMIR PublicationsJMIR Medical Informatics2291-96942025-06-0113e71687e7168710.2196/71687Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development StudySunghoon Kanghttp://orcid.org/0009-0003-8133-0895Hyewon Parkhttp://orcid.org/0009-0004-8034-7377Ricky Tairahttp://orcid.org/0000-0001-9754-8993Hyeoneui Kimhttp://orcid.org/0000-0002-5931-7286 Abstract BackgroundAs the importance of person-generated health data (PGHD) in health care and research has increased, efforts have been made to standardize survey-based PGHD to improve its usability and interoperability. Standardization efforts such as the Patient-Reported Outcomes Measurement Information System (PROMIS) and the National Institutes of Health (NIH) Common Data Elements (CDE) repository provide effective tools for managing and unifying health survey questions. However, previous methods using ontology-mediated annotation are not only labor-intensive and difficult to scale but also challenging for identifying semantic redundancies in survey questions, especially across multiple languages. ObjectiveThe goal of this work was to compute the semantic similarity among publicly available health survey questions to facilitate the standardization of survey-based PGHD. MethodsWe compiled various health survey questions authored in both English and Korean from the NIH CDE repository, PROMIS, Korean public health agencies, and academic publications. Questions were drawn from various health lifelog domains. A randomized question pairing scheme was used to generate a semantic text similarity dataset consisting of 1758 question pairs. The similarity scores between each question pair were assigned by 2 human experts. The tagged dataset was then used to build 4 classifiers featuring bag-of-words, sentence-bidirectional encoder representations from transformers (SBERT) with bidirectional encoder representations from transformers (BERT)–based embeddings, SBERT with language-agnostic BERT sentence embedding (LaBSE), and GPT-4o. The algorithms were evaluated using traditional contingency statistics. ResultsAmong the 3 algorithms, SBERT-LaBSE demonstrated the highest performance in assessing the question similarity across both languages, achieving area under the receiver operating characteristic and precision-recall curves of >0.99. Additionally, SBERT-LaBSE proved effective in identifying cross-lingual semantic similarities. The SBERT-LaBSE algorithm excelled at aligning semantically equivalent sentences across both languages but encountered challenges in capturing subtle nuances and maintaining computational efficiency. Future research should focus on testing with larger multilingual datasets and on calibrating and normalizing scores across the health lifelog domains to improve consistency. ConclusionsThis study introduces the SBERT-LaBSE algorithm for calculating the semantic similarity across 2 languages, showing that it outperforms BERT-based models, the GPT-4o model, and the bag-of-words approach, highlighting its potential in improving the semantic interoperability of survey-based PGHD across language barriers.https://medinform.jmir.org/2025/1/e71687 |
| spellingShingle | Sunghoon Kang Hyewon Park Ricky Taira Hyeoneui Kim Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study JMIR Medical Informatics |
| title | Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study |
| title_full | Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study |
| title_fullStr | Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study |
| title_full_unstemmed | Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study |
| title_short | Detecting Redundant Health Survey Questions by Using Language-Agnostic Bidirectional Encoder Representations From Transformers Sentence Embedding: Algorithm Development Study |
| title_sort | detecting redundant health survey questions by using language agnostic bidirectional encoder representations from transformers sentence embedding algorithm development study |
| url | https://medinform.jmir.org/2025/1/e71687 |
| work_keys_str_mv | AT sunghoonkang detectingredundanthealthsurveyquestionsbyusinglanguageagnosticbidirectionalencoderrepresentationsfromtransformerssentenceembeddingalgorithmdevelopmentstudy AT hyewonpark detectingredundanthealthsurveyquestionsbyusinglanguageagnosticbidirectionalencoderrepresentationsfromtransformerssentenceembeddingalgorithmdevelopmentstudy AT rickytaira detectingredundanthealthsurveyquestionsbyusinglanguageagnosticbidirectionalencoderrepresentationsfromtransformerssentenceembeddingalgorithmdevelopmentstudy AT hyeoneuikim detectingredundanthealthsurveyquestionsbyusinglanguageagnosticbidirectionalencoderrepresentationsfromtransformerssentenceembeddingalgorithmdevelopmentstudy |