Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024)
Forster and Neugebauer's (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments "are not suited for studying hiring behavior" (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with "nearly identi...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Society for Sociological Science
2025-01-01
|
| Series: | Sociological Science |
| Subjects: | |
| Online Access: | https://sociologicalscience.com/articles-v12-5-97/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832590501159632896 |
|---|---|
| author | Justin T. Pickett |
| author_facet | Justin T. Pickett |
| author_sort | Justin T. Pickett |
| collection | DOAJ |
| description | Forster and Neugebauer's (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments "are not suited for studying hiring behavior" (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with "nearly identical" designs is false (P. 891). The two experiments included: (1) different factor levels (for three factors), (2) different unvalidated applicant names (to manipulate ethnicity), (3) different applicant photos, (4) different fixed factors (e.g., applicant stories about moving), and (5) different experimental settings (e.g., testing, instrumentation, and conditions of anonymity). In the current article, I discuss each of these major design differences and explain why it invalidates Forster and Neugebauer's (2024) comparison of their FE and FS findings. I conclude by emphasizing that social scientists are better served by asking why FE and FS findings sometimes differ than by assuming that any difference in findings across the experimental designs invalidates FS. |
| format | Article |
| id | doaj-art-e373ab80f3504174994122fbeb1711cd |
| institution | Kabale University |
| issn | 2330-6696 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Society for Sociological Science |
| record_format | Article |
| series | Sociological Science |
| spelling | doaj-art-e373ab80f3504174994122fbeb1711cd2025-01-23T16:59:36ZengSociety for Sociological ScienceSociological Science2330-66962025-01-0112510.15195/v12.a5Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024)Justin T. Pickett0University at Albany, SUNYForster and Neugebauer's (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments "are not suited for studying hiring behavior" (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with "nearly identical" designs is false (P. 891). The two experiments included: (1) different factor levels (for three factors), (2) different unvalidated applicant names (to manipulate ethnicity), (3) different applicant photos, (4) different fixed factors (e.g., applicant stories about moving), and (5) different experimental settings (e.g., testing, instrumentation, and conditions of anonymity). In the current article, I discuss each of these major design differences and explain why it invalidates Forster and Neugebauer's (2024) comparison of their FE and FS findings. I conclude by emphasizing that social scientists are better served by asking why FE and FS findings sometimes differ than by assuming that any difference in findings across the experimental designs invalidates FS.https://sociologicalscience.com/articles-v12-5-97/factorial survey experimentfield experimenthiringsurvey methodologyrace/ethnicitybias |
| spellingShingle | Justin T. Pickett Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) Sociological Science factorial survey experiment field experiment hiring survey methodology race/ethnicity bias |
| title | Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) |
| title_full | Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) |
| title_fullStr | Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) |
| title_full_unstemmed | Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) |
| title_short | Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024) |
| title_sort | invalidating factorial survey experiments using invalid comparisons is bad practice learning from forster and neugebauer 2024 |
| topic | factorial survey experiment field experiment hiring survey methodology race/ethnicity bias |
| url | https://sociologicalscience.com/articles-v12-5-97/ |
| work_keys_str_mv | AT justintpickett invalidatingfactorialsurveyexperimentsusinginvalidcomparisonsisbadpracticelearningfromforsterandneugebauer2024 |