I remember when I was in an Experimental Methods course for my undergraduate psychology major, where my professor spent hours discussing the reliability and validity in experimental research. At the time, I did not really understand why my professor was so “obsessed” with validity and reliability issues. Those days feel like decades ago, but the words of my professor have become more and more relevant as I get involved in researching and designing experiments. Unfortunately, the issues of reliability and validity are often neglected in market research. This blog will discuss the importance of reliability and validity in research in general, dive deeper into the relation of these issues in implicit association testing (IAT) and explore how market researchers should handle them.
What validity and reliability mean, and why bother?
Reliability refers to the consistency of a measure. A reliable measure includes test-retest reliability, meaning the scores should be the same if tested on the same group of people at different times. A reliable measure should also have internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. All the items on a measure should reflect the same underlying construct, so the scores on these items should be correlated with each other. Finally, a reliable measure should have inter-rater consistency, meaning that different individuals assessing the same stimuli should score similarly (Drost, 2011).
Researchers should also be concerned with validity, the ability of a study to measure what it intends to measure. There are several types of validity, but market researchers should particularly pay attention to content validity, construct validity and predictive validity. Content validity is the extent a measure covers the questions to match the research objectives. A valid measure should also have construct validity, the degree wherein an assessment corresponds to other variables, as predicted by some rationale or theory. Despite being important and central in academic research, construct validity is often not addressed in market research. Predictive validity, on the other hand, seems to be more relatable to market researchers, as it assesses how well a measurement can predict future actions or behaviors (Drost, 2011).
Validity and reliability are not always aligned. Reliability is needed, but not enough to establish validity. It is possible to get high reliability, but low validity (for example, when the wrong questions are asked repeatedly). It is also possible to have a valid, but not reliable measure, such as when results show large variation. Therefore, it is crucial to make sure your research is both reliable and valid simultaneously. If the measurement is not valid, it is meaningless to a study because the results cannot be used to answer the research question. Similarly, if results from a study are not reliable, market researchers should not use them for any decision-making processes.
Implicit Association Test
IAT is a popular measure in social psychology to measure the relative strength of association between pairs of concepts (Greenwald, McGhee, & Schwartz, 1998). The theory behind this form of testing is that making a response should be easier when closely related items share the same response key. IAT is also among one of the fastest growing approaches in market research for its objectivity and cost effectiveness in capturing consumers’ immediate, gut instinct, and subconscious responses to brands, new product concepts, and other marketing products (Calvert, 2015). IAT was developed in response to reports of low validity of explicit (self-report) measures, as most people are unwilling to report their true personal thoughts or feelings towards a stimulus. However, despite IAT’s popularity both in academia and in market research, its reliability and validity still raise some concerns that are worth discussing.
In psychology, a measure is considered reliable if it has a test-retest reliability of at least 0.7, although it is preferred to be over 0.8. Studies have found that racial bias IAT studies have a test-retest reliability score of only 0.44, while the IAT overall is just around 0.5. The second major concern with IAT is its validity. Validity is best established by showing that results from the test can accurately predict behaviors in real life. However, from 2009 to 2015, four separate meta-analyses came out all suggesting that the IAT is a weak predictor of discriminating behavior (Goldhill, 2017).
While these numbers might seem alarming, it is important to note that these statistics are mostly for the IAT studies that tried to understand racial implicit bias. As mentioned, validity and reliability are often not addressed in market research; hence, the literature focusing on the previously mentioned IAT concerns in market research is scarce.
Perhaps, it would not be fair to use the somewhat alarming statistics from the race IAT studies, a rather big and complex issue in our society, to infer that IAT should not be used in market research. In fact, studies among different contexts (other than racial bias) have shown that IAT is a better predictor of subsequent behavior than explicit responses. These studies include topics such as consumer choice, risk-taking behavior, and stress response (Calvert, 2015).
Some Recommendations
Regardless of some concerns with IAT’s validity and reliability, IAT can still be a powerful tool for insights into consumers’ implicit attitudes, if designed and studied properly. Below, we will include items that market researchers should be aware of and take into consideration when thinking about IAT for their research.
IAT only measures the relative strength of association. For example, it examines the relative favorableness toward two concepts; thus, results can only tell us whether one prefers A over B, not whether he/she dislikes B or is neutral toward B. It is important that researchers are aware of this difference, so that if the research objective is to study attitude toward a single object, perhaps IAT would not be their ideal method. Different approaches have been suggested to go around this limitation of IAT, but they still require more work until they can be applied widely (Brunel, Tietje, & Greenwald, 2004).
The use of reaction time makes IAT vulnerable. IAT uses reaction time to measure the strength of association. While this measurement is convenient, reaction time as a measure of association strength makes the test vulnerable in assessment of its validity and reliability (Rezaei, 2011). This is because even “a tenth of a second can have a sequential effect on a person’s score” (Blanton & Jaccard, 2008). Market researchers should keep this fact in mind when analyzing their IAT results to avoid jumping to conclusions that the test is not reliable.
Familiarity with IAT can help improve the reliability of the test. As suggested by Rezaei (2011), perhaps it would be beneficial for market researchers to include trials where participants practice the test before the actual study to improve ireliability measures.
Cautions on stimulus selection. In selecting stimuli for IAT, it is important that they are reasonably familiar and unambiguously fall into one of two categories (Brunel, Tietje, & Greenwald, 2004). Additionally, researchers should also be cautious with the length of the words/expressions they include in IAT (Neuromarketing Science & Business Association). Again, because the IAT uses reaction time as its measure for association, it is critical to use words of similar lengths, preferably single words, to ensure the validity of the test against individual differences in reading and comprehension time.
Do results from your IAT study correlate with other explicit measures? Although IAT has been shown as a better indicator of behaviors than explicit measures, it might be helpful to still include explicit measures in your study together with the implicit component. Comparing results from explicit measures with results from IAT can achieve two objectives. Firstly, it can be used to test the validity and reliability of IAT. In theory, explicit measures and IAT should provide results that are, while distinct, still correlated on some levels because they are essentially measuring the same construct. Secondly, the divergence of results from the implicit and explicit measures has the potential to complement each other in predicting consumers’ behavior (Maison et al., 2004).
While this list can go on and on, this blog includes perhaps the most important components of IAT to address. IAT can be a great tool for market researchers to understand their consumers at a deeper, implicit level, in addition to explicit measures. IAT provides the second layer to a full picture of your consumers’ thoughts, beliefs, and behaviors.
Citations:
Blanton, H., & Jaccard, J. (2008). Unconscious racism: A concept in pursuit of a measure. Annual Review of Sociology, 34, 277–297.
Brunel, F. F., Tietje, B. C., & Greenwald, A. G. (2004). Is the implicit association test a valid and valuable measure of implicit consumer social cognition?. Journal of consumer Psychology, 14(4), 385-404.
Calvert, G. (2015, September 30). Everything you need to know about Implicit Reaction Time (IRTs). Retrieved from http://gemmacalvert.com/everything-you-need-to-know-about-implicit-reaction-time/
Drost, E. A. (2011). Validity and reliability in social science research. Education Research and perspectives, 38(1), 105.
Goldhill, O. (2017, December 3). The world is relying on a flawed psychological test to fight racism. Retrieved from https://qz.com/1144504/the-world-is-relying-on-a-flawed-psychological-test-to-fight-racism/
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: the implicit association test. Journal of Personality and Social Psychology, 74(6), 1464.
Lane, K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007). Understanding and using the implicit association test: IV. Implicit measures of attitudes, 59-102.
Maison, D., Greenwald, A. G., & Bruin, R. (2004). Predictive validity of the Implicit Association Test in studies of brands, consumer attitudes, and behavior. Journal of Consumer Psychology,14, 405–415.
Neuromarketing Science & Business Association. Implicit measures: what is it? How to use it?. Retrieved from https://www.nmsba.com/buying-neuromarketing/neuromarketing-techniques/implicit-measures-what-is-it-how-to-use-it
Rezaei, A. R. (2011). Validity and reliability of the IAT: Measuring gender and ethnic stereotypes. Computers in human behavior, 27(5), 1937-1941.
