I remember when I was in an Experimental Methods course for my undergraduate psychology major, where my professor spent hours discussing the reliability and validity in experimental research. At the time, I did not really understand why my professor was so “obsessed” with validity and reliability issues. Those days feel like decades ago, but the words of my professor have become more and more relevant as I get involved in researching and designing experiments. Unfortunately, the issues of reliability and validity are often neglected in market research. This blog will discuss the importance of reliability and validity in research in general, dive deeper into the relation of these issues in implicit association testing (IAT) and explore how market researchers should handle them.
What validity and
reliability mean, and why bother?
Reliability refers to the consistency of a measure. A reliable measure includes test-retest reliability, meaning the scores should be the same if tested on the same group of people at different times. A reliable measure should also have internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. All the items on a measure should reflect the same underlying construct, so the scores on these items should be correlated with each other. Finally, a reliable measure should have inter-rater consistency, meaning that different individuals assessing the same stimuli should score similarly (Drost, 2011).
Researchers should also be concerned with
validity, the ability of a study to measure what it intends to measure. There
are several types of validity, but market researchers should particularly pay
attention to content validity, construct validity and predictive validity. Content
validity is the extent a measure covers the questions to match the research
objectives. A valid measure should also have construct validity, the degree wherein
an assessment corresponds to other variables, as predicted by some rationale or
theory. Despite being important and central in academic research, construct
validity is often not addressed in market research. Predictive validity, on the
other hand, seems to be more relatable to market researchers, as it assesses
how well a measurement can predict future actions or behaviors (Drost, 2011).
Validity and reliability are not always aligned. Reliability is needed, but not enough to establish validity. It is possible to get high reliability, but low validity (for example, when the wrong questions are asked repeatedly). It is also possible to have a valid, but not reliable measure, such as when results show large variation. Therefore, it is crucial to make sure your research is both reliable and valid simultaneously. If the measurement is not valid, it is meaningless to a study because the results cannot be used to answer the research question. Similarly, if results from a study are not reliable, market researchers should not use them for any decision-making processes.
Implicit Association Test
IAT is a popular measure in social
psychology to measure the relative strength of association between pairs of
concepts (Greenwald, McGhee, & Schwartz, 1998). The theory behind this form
of testing is that making a response should be easier when closely related
items share the same response key. IAT is also among one of the fastest growing
approaches in market research for its objectivity and cost effectiveness in
capturing consumers’ immediate, gut instinct, and subconscious responses to
brands, new product concepts, and other marketing products (Calvert, 2015). IAT
was developed in response to reports of low validity of explicit (self-report)
measures, as most people are unwilling to report their true personal thoughts
or feelings towards a stimulus. However, despite IAT’s popularity both in
academia and in market research, its reliability and validity still raise some
concerns that are worth discussing.
In psychology, a measure is considered
reliable if it has a test-retest reliability of at least 0.7, although it is
preferred to be over 0.8. Studies have found that racial bias IAT studies have
a test-retest reliability score of only 0.44, while the IAT overall is just
around 0.5. The second major concern with IAT is its validity. Validity is best
established by showing that results from the test can accurately predict
behaviors in real life. However, from 2009 to 2015, four separate meta-analyses
came out all suggesting that the IAT is a weak predictor of discriminating behavior
While these numbers might seem alarming, it
is important to note that these statistics are mostly for the IAT studies that
tried to understand racial implicit bias. As mentioned, validity and
reliability are often not addressed in market research; hence, the literature
focusing on the previously mentioned IAT concerns in market research is scarce.
Perhaps, it would not be fair to use the somewhat alarming statistics from the
race IAT studies, a rather big and complex issue in our society, to infer that
IAT should not be used in market research. In fact, studies among different
contexts (other than racial bias) have shown that IAT is a better predictor of
subsequent behavior than explicit responses. These studies include topics such
as consumer choice, risk-taking behavior, and stress response (Calvert, 2015).
Regardless of some concerns with IAT’s validity and reliability, IAT can still be a powerful tool for insights into consumers’ implicit attitudes, if designed and studied properly. Below, we will include items that market researchers should be aware of and take into consideration when thinking about IAT for their research.
- IAT only measures the relative
strength of association. For example, it examines the
relative favorableness toward two concepts; thus, results can only tell us
whether one prefers A over B, not whether he/she dislikes B or is neutral
toward B. It is important that researchers are aware of this difference, so
that if the research objective is to study attitude toward a single object,
perhaps IAT would not be their ideal method. Different approaches have been
suggested to go around this limitation of IAT, but they still require more work
until they can be applied widely (Brunel, Tietje, & Greenwald, 2004).
- The use of reaction time makes IAT
vulnerable. IAT uses reaction time to measure the strength of
association. While this measurement is convenient, reaction time as a measure
of association strength makes the test vulnerable in assessment of its validity
and reliability (Rezaei, 2011). This is because even “a tenth of a second can
have a sequential effect on a person’s score” (Blanton & Jaccard, 2008).
Market researchers should keep this fact in mind when analyzing their IAT
results to avoid jumping to conclusions that the test is not reliable.
- Familiarity with IAT can help improve
the reliability of the test. As suggested by Rezaei
(2011), perhaps it would be beneficial for market researchers to include trials
where participants practice the test before the actual study to improve
- Cautions on stimulus selection.
In selecting stimuli for IAT, it is important that they are reasonably familiar
and unambiguously fall into one of two categories (Brunel, Tietje, &
Greenwald, 2004). Additionally, researchers should also be cautious with the
length of the words/expressions they include in IAT (Neuromarketing Science
& Business Association). Again, because the IAT uses reaction time as its
measure for association, it is critical to use words of similar lengths,
preferably single words, to ensure the validity of the test against individual
differences in reading and comprehension time.
- Do results from your IAT study
correlate with other explicit measures? Although IAT has been
shown as a better indicator of behaviors than explicit measures, it might be
helpful to still include explicit measures in your study together with the
implicit component. Comparing results from explicit measures with results from
IAT can achieve two objectives. Firstly, it can be used to test the validity
and reliability of IAT. In theory,
explicit measures and IAT should provide results that are, while distinct,
still correlated on some levels because they are essentially measuring the same
construct. Secondly, the divergence of results from the implicit and explicit
measures has the potential to complement each other in predicting consumers’ behavior
(Maison et al., 2004).
While this list can go on and on, this
blog includes perhaps the most important components of IAT to address. IAT can
be a great tool for market researchers to understand their consumers at a
deeper, implicit level, in addition to explicit measures. IAT provides the
second layer to a full picture of your consumers’ thoughts, beliefs, and
H., & Jaccard, J. (2008). Unconscious racism: A concept in pursuit of a
measure. Annual Review of Sociology, 34,
Brunel, F. F., Tietje, B. C., & Greenwald,
A. G. (2004). Is the implicit association test a valid and valuable measure of
implicit consumer social cognition?. Journal of consumer Psychology, 14(4),
G. (2015, September 30). Everything you need to know about Implicit Reaction
Time (IRTs). Retrieved from http://gemmacalvert.com/everything-you-need-to-know-about-implicit-reaction-time/
E. A. (2011). Validity and reliability in social science research. Education Research and
perspectives, 38(1), 105.
O. (2017, December 3). The world is relying on a flawed psychological test to
fight racism. Retrieved from https://qz.com/1144504/the-world-is-relying-on-a-flawed-psychological-test-to-fight-racism/
A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuring individual
differences in implicit cognition: the implicit association test. Journal of Personality and Social
Psychology, 74(6), 1464.
K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007).
Understanding and using the implicit association test: IV. Implicit measures of attitudes, 59-102.
D., Greenwald, A. G., & Bruin, R. (2004). Predictive validity of the
Implicit Association Test in studies of brands, consumer attitudes, and
behavior. Journal of Consumer Psychology,
Science & Business Association. Implicit measures: what is it? How to use
it?. Retrieved from https://www.nmsba.com/buying-neuromarketing/neuromarketing-techniques/implicit-measures-what-is-it-how-to-use-it
A. R. (2011). Validity and reliability of the IAT: Measuring gender and ethnic
stereotypes. Computers in human
behavior, 27(5), 1937-1941.