Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study
Abstract The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparin...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-06358-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849335138898411520 |
|---|---|
| author | Juan Gu Jiali Liu Lijuan Zeng Yiqing Yu Yufei Qiu Yake Yue Mengjie Tong Fen Yang Xiaohong Zhang |
| author_facet | Juan Gu Jiali Liu Lijuan Zeng Yiqing Yu Yufei Qiu Yake Yue Mengjie Tong Fen Yang Xiaohong Zhang |
| author_sort | Juan Gu |
| collection | DOAJ |
| description | Abstract The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparing scoring consistency between ChatGPT and the validated questionnaires. This was a cross-sectional study between June and August 2024. We pre-trained ChatGPT-4 based on the validated University of California Los Angeles Loneliness Scale-6 (ULS-6) and Chinese Youth Version of the Online Social Support Scale (OSSS-CS), creating a structured interview questionnaire. Participants were invited to complete both the ChatGPT-created questionnaire and the validated questionnaires. We used Spearman correlation analysis, Intra-class correlation coefficients (ICC), and Bland–Altman plots to assess the agreement between the scores from ChatGPT and the validated questionnaires. In addition, we evaluated ceiling and floor effects. A total of 216 college students participated the survey. The results demonstrated a good consistency between the scores obtained from ChatGPT and the validated questionnaires, with ICC of 0.81 (95% CI 0.75–0.85, p < 0.001) for ULS-6 and 0.95 (95% CI 0.94–0.96, p < 0.001) for OSSS-CS. The Spearman correlation coefficients were 0.64 (p < 0.001) for ULS-6 and 0.89 (p < 0.001) for OSSS-CS, indicating a moderate correlation. No ceiling or floor effects were observed. The ChatGPT-created questionnaire demonstrated acceptable consistency with the validated questionnaires. Future studies can further explore the performance of ChatGPT in different populations and domains, as well as how to integrate it with validated questionnaires to enhance the accessibility of assessments. |
| format | Article |
| id | doaj-art-3e2271eaece1483dbe688c58b015d4e7 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-3e2271eaece1483dbe688c58b015d4e72025-08-20T03:45:23ZengNature PortfolioScientific Reports2045-23222025-07-0115111010.1038/s41598-025-06358-2Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional studyJuan Gu0Jiali Liu1Lijuan Zeng2Yiqing Yu3Yufei Qiu4Yake Yue5Mengjie Tong6Fen Yang7Xiaohong Zhang8College of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineCollege of Nursing, Hubei University of Chinese MedicineHubei Shizhen LaboratoryAbstract The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparing scoring consistency between ChatGPT and the validated questionnaires. This was a cross-sectional study between June and August 2024. We pre-trained ChatGPT-4 based on the validated University of California Los Angeles Loneliness Scale-6 (ULS-6) and Chinese Youth Version of the Online Social Support Scale (OSSS-CS), creating a structured interview questionnaire. Participants were invited to complete both the ChatGPT-created questionnaire and the validated questionnaires. We used Spearman correlation analysis, Intra-class correlation coefficients (ICC), and Bland–Altman plots to assess the agreement between the scores from ChatGPT and the validated questionnaires. In addition, we evaluated ceiling and floor effects. A total of 216 college students participated the survey. The results demonstrated a good consistency between the scores obtained from ChatGPT and the validated questionnaires, with ICC of 0.81 (95% CI 0.75–0.85, p < 0.001) for ULS-6 and 0.95 (95% CI 0.94–0.96, p < 0.001) for OSSS-CS. The Spearman correlation coefficients were 0.64 (p < 0.001) for ULS-6 and 0.89 (p < 0.001) for OSSS-CS, indicating a moderate correlation. No ceiling or floor effects were observed. The ChatGPT-created questionnaire demonstrated acceptable consistency with the validated questionnaires. Future studies can further explore the performance of ChatGPT in different populations and domains, as well as how to integrate it with validated questionnaires to enhance the accessibility of assessments.https://doi.org/10.1038/s41598-025-06358-2ChatGPTLarge language modelsArtificial intelligenceLonelinessOnline social supportQuestionnaires |
| spellingShingle | Juan Gu Jiali Liu Lijuan Zeng Yiqing Yu Yufei Qiu Yake Yue Mengjie Tong Fen Yang Xiaohong Zhang Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study Scientific Reports ChatGPT Large language models Artificial intelligence Loneliness Online social support Questionnaires |
| title | Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study |
| title_full | Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study |
| title_fullStr | Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study |
| title_full_unstemmed | Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study |
| title_short | Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study |
| title_sort | comparing chatgpt and validated questionnaires in assessing loneliness and online social support among college students a cross sectional study |
| topic | ChatGPT Large language models Artificial intelligence Loneliness Online social support Questionnaires |
| url | https://doi.org/10.1038/s41598-025-06358-2 |
| work_keys_str_mv | AT juangu comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT jialiliu comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT lijuanzeng comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT yiqingyu comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT yufeiqiu comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT yakeyue comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT mengjietong comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT fenyang comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy AT xiaohongzhang comparingchatgptandvalidatedquestionnairesinassessinglonelinessandonlinesocialsupportamongcollegestudentsacrosssectionalstudy |