Steven Denney (Leiden University)
Last updated: 27 April 2026
Conjoint experiments rest on an assumption that is rarely tested: that respondents reason about profile attributes in the way the construct of interest specifies. Open-text questions, familiar as basic manipulation checks, can do much more. I argue they should be taken seriously as evidence about construct validity and experimental design, and I develop a framework that combines structural topic modeling (STM) for unsupervised discovery with large language model (LLM) classification for theory-driven confirmation. I apply the framework across three conjoint experiments. In a conjoint on immigrant naturalization in South Korea and Taiwan, a social integration cue shifts respondent reasoning from economic and practical criteria toward civic engagement; STM and LLM classification converge on this finding. A second conjoint on immigrant admission shows the framework transfers to designs without within-attribute randomization and refines constructs an earlier analysis grouped too coarsely. A third conjoint on North Korean migrant entrepreneurship benchmarks LLM performance against expert human-coded ground truth and confirms that iterative codebook refinement raises LLM-human agreement. Token-level log-probabilities measure model uncertainty, a calibration approach uncommon in survey text-as-data work; they predict when models disagree and reveal that all models are overconfident relative to human coding.
- denney_2026_what-were-they-thinking.pdf (manuscript with supplementary information)
- denney_2026_what-were-they-thinking_slides.pdf (presentation slides)
Denney, Steven. 2026. "What Were They Thinking? Using Open-Text Responses to Validate Constructs in Survey Experiments." Working Paper. https://github.com/scdenney/what-were-they-thinking
Steven Denney, Assistant Professor, Leiden University s.c.denney@hum.leidenuniv.nl