There has long been a data tug of war in market research and social sciences, and it centers on the benefits and difficulties between structured and unstructured data. Unstructured data (interviews, observations, text, pictures, etc.) provide a depth of rich information that is fodder for qualitative analysis and generating hypotheses, and more nuanced understandings. But there is a subjective nature to this type of analysis that is its Achilles’ heel. It’s mostly peoples’ opinions, and it’s also difficult to turn into numbers. Indeed, in the past the difficulty in turning such data into numbers prevented too strong a variable or did too much violence to the nuances of the data for advocates to even lean into quantitative analysis. In the social sciences, those practicing such qualitative arts are often looked down upon by quantitative scientists.
Decision-makers want “science.” That means going beyond opinion in favor of structure and precision. It’s why the market research field is largely survey-driven with some qualitative approaches that add depth and richness. There have been attempts to quantify the unstructured or qualitative, like social listening. Good data for background, but noisy and removes researchers’ ability to control for the kinds of information they want to uncover.
Meanwhile, structured data has its own limits. The problems with rating scales are legion. There are substantial (if ignored) difficulties with response tendencies and mind-numbing rating grids. Beyond that, and more importantly, we strip away the nuance of human experiences and fit people into the self-report cages of Likert rating scales and rating boxes. We do it for efficiency, but we pay for it in lost human understanding.
The advent of LLMs offers something new and better than even the Natural Language Processing (NLP) solutions that just preceded them. We’re now able to demand the rigors of quantitative research on numbers derived from rich qualitative sources. The real AI revolution in market research is leveraging the “mess” of richer data (Narrative Intelligence surveys, text, images, video) with new, proper tools to dimensionalize the signal out of all the noise quickly and efficiently with AI.
When the Tech Finally Met the Moment
New tech seems to take over rapidly, but the ideas behind it can be there before the capability. My dissertation and psychological background emphasized both rigorous advanced modeling and evaluation of open-ended responses. Simply put, I asked people their goals and how they might achieve them. Two sentences each: I want to X. By Y. Ten to 20 per person. I then had a group of raters evaluate each of these on a variety of dimensions. We could identify a person’s personality and predict their stress levels six months later via those rating dimensions based on their qualitative data. The problem was that this process is difficult, time-consuming work. Over the years, my team and I evaluated different tools to see if they could do the evaluations as well as humans. We’d compare our results to our original data. We were particularly hopeful when NLP was first the rage, and the answer was a disappointing no.
A conversation with a tech executive inspired me to try again right after LLMs really broke out onto the scene. This time, he said, it’s different. He was right. We compared the ratings from LLM model rater “agents” to the human coders. We not only found the agents to be in general agreement, but we agreed that the AI raters were superior to the humans. (Humans can make mistakes after all). That sparked the origin of our Narrative Intelligence surveys.
Christopher Frank, CEO here at PSB, asked: what if you could go beyond the confines of a single survey? He connected the dots and saw that the dimensions we created in our primary research unstructured data (Narrative Intelligence) could be applied to the huge scale of unstructured data in the wild or that companies have in their storehouse. Could we use the same approach for what consumers say, what brand marketers say, what news says about products and services online? The answer is yes. We have applied the idea of dimensionalizing Narrative Intelligence surveys to the larger unstructured data world. As a result, we can enhance our primary research efforts by looking at data in the wild and using the same structural framework via the creation of dimensions, a step above traditional ways of dealing with this type of data.
Expanded Data Worlds
This shift matters because it expands the capabilities of the insights professional from the standard quant, to new quant at qual depth methods, to data beyond. Here’s what that looks like:
- Standard Quant: The traditional world of rating scales and “cages.”
- Quant at Qual Depth: This is Narrative Intelligence. We get the depth of a qualitative interview but use our “army of coders” (proprietary AI) to turn it into quantitative dimensions. We get the “why” without losing the “how many.”
- Unstructured “Big Data,” (which we break into The Wild and The Well): This is a huge enhancement. It lets us actually tame the reams of data we generate as a digital society. Once we can dimensionalize at scale, we can enhance our understanding of context in the wider world. It maps the landscape at a general level and complements more precise approaches. Neither are a replacement for the other, but together they contribute to a full perspective.
Dimensionalization as the New Standard
There’s significantly more to this than just “coding” text. By deploying a virtual army of raters, we make our surveys inherently better. We are unburdened by the need to know every possible variable at the start of a project. Because AI can help uncover dimensions we might have missed, we have the freedom to be curious. If a new hypothesis emerges even after an analysis is complete, we simply go back and re-score the data against those new dimensions. It turns a static dataset into a living one. It allows us to access different kinds and scales of data, using a flexible, common framework.
But it’s also not a magic button. Turning unstructured primary research or unstructured wild data into numbers for analysis requires more than just a subscription to an LLM. It needs experience and methodological rigor. You don’t want to just dump data into a prompt and expect a valid result; you have to guard against drift and hallucination and employ best practices across the board. Among other things, this requires a bit of technical know-how from managing APIs to optimizing batch sizes and using multiple rater agents from different models to find a consensus. We’ve found that using descriptive words and stepped responses to create dimensions is far more effective than asking a model for a numerical evaluation. Most importantly, the researcher must define, refine, evaluate, and validate dimensions to ensure usefulness in making better business outcomes.
The Bottom Line
The AI revolution in research isn’t about replacing the researcher or building some all-powerful automated “insight engine.” It is about opening new horizons in how we can gather and analyze data. We are finally ending the tug-of-war between richness and precision. We can meet the consumer where they are—in all their messy, unstructured glory—and still bring the science back with us.
Let’s talk about how you can best position AI for your insights practice. Get in touch with me—robkphd@psbinsights.com