Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024)

Forster and Neugebauer's (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments "are not suited for studying hiring behavior" (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with "nearly identi...

Full description

Saved in:
Bibliographic Details
Main Author: Justin T. Pickett
Format: Article
Language:English
Published: Society for Sociological Science 2025-01-01
Series:Sociological Science
Subjects:
Online Access:https://sociologicalscience.com/articles-v12-5-97/
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Forster and Neugebauer's (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments "are not suited for studying hiring behavior" (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with "nearly identical" designs is false (P. 891). The two experiments included: (1) different factor levels (for three factors), (2) different unvalidated applicant names (to manipulate ethnicity), (3) different applicant photos, (4) different fixed factors (e.g., applicant stories about moving), and (5) different experimental settings (e.g., testing, instrumentation, and conditions of anonymity). In the current article, I discuss each of these major design differences and explain why it invalidates Forster and Neugebauer's (2024) comparison of their FE and FS findings. I conclude by emphasizing that social scientists are better served by asking why FE and FS findings sometimes differ than by assuming that any difference in findings across the experimental designs invalidates FS.
ISSN:2330-6696