by Mohammad Hossein Jarrahi
Jarrahi, M. H. (2025). Interviewing AI: Using qualitative methods to explore and capture machines’ characteristics and behaviors. Big Data & Society, 12(3), 20539517251381697. (Original work published 2025)
AI systems are now woven into the everyday lives of millions, yet their behavior often surprises even their own creators. Traditional methods of researching technology can certainly tell part of the story, especially how different groups of users make sense of these tools, but they may miss how AI systems actually act and behave in unpredictable ways. I argue that we need to develop new methods of data collection and analysis. This paper therefore grew out of that gap. I asked a simple question: what if I studied AI the way social scientists study people, by interviewing it?
I introduce "interviewing AI," a qualitative toolkit for making sense of machine behavior. It begins with open exploration to learn a system's quirks. It then moves to structured probing that varies prompts, stages scenarios, pushes at boundaries, and poses counterfactual "what if" questions to surface hidden patterns, breakdowns, and traces of reasoning. To widen the lens, I offer two designs: Temporal Interaction Analysis tracks the same system over time to capture shifts after updates or sustained use, and Comparative Synchronic Analysis compares multiple systems on the same tasks to reveal differences that matter in practice.
I pair these methods with qualitative analysis techniques such as thematic coding and critical discourse analysis to identify patterns and uncover embedded biases. Throughout, I argue for transparency and reflexivity, and I caution against reading too much human intent into machine outputs (i.e., anthropomorphization). I also make a case for treating prompts and AI responses as a single unit of analysis, as humans and AI co-produce the exchange.
Readers can expect a practical, research-ready holistic qualitative framework that complements other research approaches in human-centered studies of AI, helping designers, governors, and scholars document where systems help, where they fail, and why, and turn messy interactions into rigorous insights.
