For anyone who’s done international research, you’ve likely observed the differences in response styles across countries—standout examples are respondents in China tending to say yes to more things, Germans are generally less ‘exuberant’ in their responses than markets such as India or Brazil, Japanese more often selecting ‘middle box’ responses while Americans are often a mix of all of these. When analyzing results and making cross-market comparisons, could the data lead to incorrect or imprecise cross-country comparisons? How do we know that the notable differences by market are ‘real’ differences rather than reflections of each culture’s response tendencies?
The reality of cultural response “tendencies”
Respondents from each country and culture have certain response tendencies, regardless of the question content, as noted above in the examples with China, Germany, Brazil, Japan and the US. The proportion of individuals who favor a specific response style differs in meaningful ways from country to country and person to person. Here’s an example from an actual study where the differences in favorability across the US, the UK, and India are dramatically different.
Using Multi-Dimensional Item Response Theory allows us to adjust for cross-cultural response style differences in our survey data.
The beauty of using IRT to help deal with response style differences
Item Response Theory (IRT) is a statistical framework that examines patterns of responses for multiple items to analyze the relationship between individuals’ responses (exhibited by brand attitudes and behaviors) and their underlying characteristic or “traits” (e.g., actual attitudes as well as culture and response styles). IRT provides a method for empirically determining the degree of influence that underlying traits have on responses, disentangling the nuisance response tendencies. IRT modeling provides the benefits of:
- Estimating individual scores on underlying content traits of interest for each respondent.
- Correcting for differences in response style tendencies, such as extreme, acquiescent, and noncommittal responses.
- Avoiding inconclusive findings by using adjusted scores for content of interest to aid in further analysis.
Applications for IRT in marketing research
PSB recently conducted several studies where IRT enabled us to achieve more comparable results across countries.
For a recent international segmentation project for a major technology company, we used IRT to help ensure that we did not get segments based on an individual countries (such as China or Japan). The initial results without IRT merely grouped and segmented the data by country. Using IRT to estimate the true scores on the substantive content, we achieved segments that proved far more effective in grouping respondents in a way useful to our client.
For a multi-country, multi-year, brand measurement project, our client requested the addition of another country, India. During the initial wave, the top 2-box scores in India were greater than ninety percent (>90%)—which leaves hardly any room to grow or improve. But, when we applied IRT to adjust for response style differences, India’s responses are revealed as more comparable to those of the United States and the United Kingdom. Thus, we were able to achieve actionable insights using the “adjusted” data.
Clear results in international research
Incorporating IRT into multi-country research is an ideal way to stop having to take the differences across markets with the proverbial grain of salt. This extra step to correct for response tendencies can help to ensure truly differentiated data and meaningful comparisons across countries.