Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model
Multi-item scales are widely used in social research. The psychometric characteristics of a scale and the successful use of a scale in research depend in part on item wording. This article demonstrates a method for using natural language processing (NLP) tools to assist with the item development pr...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
European Survey Research Association
2024-08-01
|
| Series: | Survey Research Methods |
| Subjects: | |
| Online Access: | https://ojs.ub.uni-konstanz.de/srm/article/view/8240 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823861395162136576 |
|---|---|
| author | Jeffrey Stanton Angela Ramnarine-Rieks Yisi Sang |
| author_facet | Jeffrey Stanton Angela Ramnarine-Rieks Yisi Sang |
| author_sort | Jeffrey Stanton |
| collection | DOAJ |
| description |
Multi-item scales are widely used in social research. The psychometric characteristics of a scale and the successful use of a scale in research depend in part on item wording. This article demonstrates a method for using natural language processing (NLP) tools to assist with the item development process, by showing that numeric embedding representations of items are useful in predicting the characteristics of a scale. NLP comprises a set of algorithmic techniques for analysing words, phrases, and larger units of written language. We used NLP tools to create and analyse semantic summaries of the item texts for n=386 previously published multi-item scales. Results showed that semantic representations of items connect to scale characteristics such as Cronbach's alpha internal consistency.
|
| format | Article |
| id | doaj-art-16ef5a19b8114ea7926e7a7492b33e4b |
| institution | Kabale University |
| issn | 1864-3361 |
| language | English |
| publishDate | 2024-08-01 |
| publisher | European Survey Research Association |
| record_format | Article |
| series | Survey Research Methods |
| spelling | doaj-art-16ef5a19b8114ea7926e7a7492b33e4b2025-02-09T14:16:10ZengEuropean Survey Research AssociationSurvey Research Methods1864-33612024-08-01182Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network ModelJeffrey Stanton0Angela Ramnarine-Rieks1Yisi Sang2Syracuse UniversitySyracuse UniversitySyracuse University Multi-item scales are widely used in social research. The psychometric characteristics of a scale and the successful use of a scale in research depend in part on item wording. This article demonstrates a method for using natural language processing (NLP) tools to assist with the item development process, by showing that numeric embedding representations of items are useful in predicting the characteristics of a scale. NLP comprises a set of algorithmic techniques for analysing words, phrases, and larger units of written language. We used NLP tools to create and analyse semantic summaries of the item texts for n=386 previously published multi-item scales. Results showed that semantic representations of items connect to scale characteristics such as Cronbach's alpha internal consistency. https://ojs.ub.uni-konstanz.de/srm/article/view/8240Cronbach’s alphaAnswer behavior, Emotion prediction, Microphone, Natural Language Processing, Open-ended questions, Smartphone, Voice recordingsNeural networkrating scale |
| spellingShingle | Jeffrey Stanton Angela Ramnarine-Rieks Yisi Sang Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model Survey Research Methods Cronbach’s alpha Answer behavior, Emotion prediction, Microphone, Natural Language Processing, Open-ended questions, Smartphone, Voice recordings Neural network rating scale |
| title | Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model |
| title_full | Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model |
| title_fullStr | Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model |
| title_full_unstemmed | Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model |
| title_short | Evaluating Item Content and Scale Characteristics Using a Pretrained Neural Network Model |
| title_sort | evaluating item content and scale characteristics using a pretrained neural network model |
| topic | Cronbach’s alpha Answer behavior, Emotion prediction, Microphone, Natural Language Processing, Open-ended questions, Smartphone, Voice recordings Neural network rating scale |
| url | https://ojs.ub.uni-konstanz.de/srm/article/view/8240 |
| work_keys_str_mv | AT jeffreystanton evaluatingitemcontentandscalecharacteristicsusingapretrainedneuralnetworkmodel AT angelaramnarinerieks evaluatingitemcontentandscalecharacteristicsusingapretrainedneuralnetworkmodel AT yisisang evaluatingitemcontentandscalecharacteristicsusingapretrainedneuralnetworkmodel |