Using text embeddings for deductive qualitative research at scale in physics education
We propose a technique for performing deductive qualitative data analysis at scale on text-based data. Using a natural language processing technique known as text embeddings, we create vector-based representations of texts in a high-dimensional meaning space within which it is possible to quantify d...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2024-12-01
|
| Series: | Physical Review Physics Education Research |
| Online Access: | http://doi.org/10.1103/PhysRevPhysEducRes.20.020151 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850131511880187904 |
|---|---|
| author | Tor Ole B. Odden Halvor Tyseng Jonas Timmann Mjaaland Markus Fleten Kreutzer Anders Malthe-Sørenssen |
| author_facet | Tor Ole B. Odden Halvor Tyseng Jonas Timmann Mjaaland Markus Fleten Kreutzer Anders Malthe-Sørenssen |
| author_sort | Tor Ole B. Odden |
| collection | DOAJ |
| description | We propose a technique for performing deductive qualitative data analysis at scale on text-based data. Using a natural language processing technique known as text embeddings, we create vector-based representations of texts in a high-dimensional meaning space within which it is possible to quantify differences in meaning as vector distances. To apply the technique, we build off prior work that used topic modeling via latent Dirichlet allocation to thematically analyze 18 years of the Physics Education Research Conference Proceedings literature. We first extend this analysis through 2023. Next, we create embeddings of all texts and, using representative articles from the 10 topics found by the LDA analysis, define centroids in the meaning space. We calculate the distances between every article and centroid and use the inverted, scaled distances between these centroids and articles to create an alternate topic model. We benchmark this model against the LDA model results and show that this embeddings model recovers most of the trends from that analysis. Finally, to illustrate the versatility of the method, we define eight new topic centroids derived from a review of the physics education research literature by Docktor and Mestre and reanalyze the literature using these researcher-defined topics. Based on these analyses, we critically discuss the features, uses, and limitations of this method and argue that it holds promise for flexible deductive qualitative analysis of a wide variety of text-based data that avoids many of the drawbacks inherent to prior NLP methods. |
| format | Article |
| id | doaj-art-bf6b859eccd24cc29f3b440b00d9dc36 |
| institution | OA Journals |
| issn | 2469-9896 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review Physics Education Research |
| spelling | doaj-art-bf6b859eccd24cc29f3b440b00d9dc362025-08-20T02:32:25ZengAmerican Physical SocietyPhysical Review Physics Education Research2469-98962024-12-0120202015110.1103/PhysRevPhysEducRes.20.020151Using text embeddings for deductive qualitative research at scale in physics educationTor Ole B. OddenHalvor TysengJonas Timmann MjaalandMarkus Fleten KreutzerAnders Malthe-SørenssenWe propose a technique for performing deductive qualitative data analysis at scale on text-based data. Using a natural language processing technique known as text embeddings, we create vector-based representations of texts in a high-dimensional meaning space within which it is possible to quantify differences in meaning as vector distances. To apply the technique, we build off prior work that used topic modeling via latent Dirichlet allocation to thematically analyze 18 years of the Physics Education Research Conference Proceedings literature. We first extend this analysis through 2023. Next, we create embeddings of all texts and, using representative articles from the 10 topics found by the LDA analysis, define centroids in the meaning space. We calculate the distances between every article and centroid and use the inverted, scaled distances between these centroids and articles to create an alternate topic model. We benchmark this model against the LDA model results and show that this embeddings model recovers most of the trends from that analysis. Finally, to illustrate the versatility of the method, we define eight new topic centroids derived from a review of the physics education research literature by Docktor and Mestre and reanalyze the literature using these researcher-defined topics. Based on these analyses, we critically discuss the features, uses, and limitations of this method and argue that it holds promise for flexible deductive qualitative analysis of a wide variety of text-based data that avoids many of the drawbacks inherent to prior NLP methods.http://doi.org/10.1103/PhysRevPhysEducRes.20.020151 |
| spellingShingle | Tor Ole B. Odden Halvor Tyseng Jonas Timmann Mjaaland Markus Fleten Kreutzer Anders Malthe-Sørenssen Using text embeddings for deductive qualitative research at scale in physics education Physical Review Physics Education Research |
| title | Using text embeddings for deductive qualitative research at scale in physics education |
| title_full | Using text embeddings for deductive qualitative research at scale in physics education |
| title_fullStr | Using text embeddings for deductive qualitative research at scale in physics education |
| title_full_unstemmed | Using text embeddings for deductive qualitative research at scale in physics education |
| title_short | Using text embeddings for deductive qualitative research at scale in physics education |
| title_sort | using text embeddings for deductive qualitative research at scale in physics education |
| url | http://doi.org/10.1103/PhysRevPhysEducRes.20.020151 |
| work_keys_str_mv | AT torolebodden usingtextembeddingsfordeductivequalitativeresearchatscaleinphysicseducation AT halvortyseng usingtextembeddingsfordeductivequalitativeresearchatscaleinphysicseducation AT jonastimmannmjaaland usingtextembeddingsfordeductivequalitativeresearchatscaleinphysicseducation AT markusfletenkreutzer usingtextembeddingsfordeductivequalitativeresearchatscaleinphysicseducation AT andersmalthesørenssen usingtextembeddingsfordeductivequalitativeresearchatscaleinphysicseducation |