Does the Leslie et al. (2020) meta-analysis prove that Coleman Hughes was wrong?
Dissecting the meta-analysis that Chris Anderson (Head of TED) relied upon to claim that the thesis of Coleman Hughes' TED talk (A Case for Color-Blindness) has been disproven by the science.
1. Context
Following the TED (“Ideas worth spreading”) talk by Coleman Hughes (A Case for Color-Blindness), Chris Anderson, the head of TED, received a request from a group of TED staffers called “Black@TED” requesting that the talk not be posted. Although Anderson ultimately did post the talk, he also sought feedback regarding whether the claims made by Hughes are consistent with the best available scientific evidence from Dr. Adam Grant, a popular science author and professor at the Wharton School of the University of Pennsylvania specializing in organizational psychology. Grant’s report was based heavily on his reading of a single meta-analysis of research on different “diversity ideologies” (the term used by the authors of the meta-analysis); he concluded that existing social science research does not support the claims made by Hughes in his TED talk. Anderson then echoed Grant’s conclusion, writing in a posting on X and a posting on the Free Press blog, specifically referencing the meta-analysis, that: “ As the researchers themselves write: “Multiculturalism is more consistently associated with improved intergroup relations than any identity-blind ideology.”“
The question I address below is whether the findings from the meta-analysis (that served as the foundation for Grant’s summary of the evidence and for Anderson’s claim that Hughes was wrong in his claims about color-blindness) really does provide evidence inconsistent with Hughes’ claims. My own reading of the paper is that it does NOT provide evidence that undermines Hughes’ claims, and that Grant, who is very well qualified to assess the paper, should have noted the rather glaring weaknesses of the paper that I note below.
The reference for the meta-analysis is:
Leslie, L, Bono, J.,Kim, Y., & Beaver, J. (2020). On Melting Pots and Salad Bowls: A Meta-Analysis of the Effects of Identity-Blind and Identity-Conscious Diversity Ideologies. Journal of Applied Psychology, 105, 453 - 471.
2. Summary
In short – I argue below that the meta-analysis suffers from several very significant shortcomings that should have been obvious to Dr. Grant. (i) The way in which the paper defines the elements of “high quality intergroup relations” is not based upon evidence of any kind, but instead reflects the progressive political philosophy of the authors, thereby baking into the design of the study the conclusions that the authors ultimately reached. (ii) The authors frequently draw causal conclusions from data which cannot be legitimately utilized as the foundation for a causal conclusion. (iii) In any case, the meta-analysis actually finds an association between color-blind beliefs and lower levels of prejudice and stereotyping. (iv) The claims by the authors of the meta-analysis (and by Grant and Anderson) notwithstanding, the study does not in any direct way assess factors contributing to “high quality intergroup relations”, and therefore it is unclear how the results of the meta-analysis actually bear on an assessment of the validity of the claims made in Hughes’ TED talk. Hughes, on the other hand, does provide evidence of a more direct kind supporting his claims regarding color blindness and intergroup relations.
3. What was the nature of the studies included in the meta-analysis?
The meta-analysis included studies that examined the association between one or more “diversity ideologies”, including “color-blindness” and “multiculturalism” (the two diversity ideologies relevant to assessing the validity of Hughes’ claims), and one or more “outcome measures” (scare quotes used here for reasons that will be made obvious in the next section) including prejudice, discrimination, stereotyping, and support for particular kinds of diversity policies. According to the paper’s authors, color-blindness “emphasizes minimizing the salience of [group] differences, specifically by ignoring them.” Thus, for example, if a professor grades student papers only after removing all student identity information from each paper, that would be classified as a color-blind approach to grading. According to the authors, multiculturalism, in contrast, “emphasizes acknowledging and valuing [group] differences.”
Most of the studies included in the meta-analysis were correlational studies that involved administering to participants measures of each individual’s belief in one or more diversity ideologies and measures of each individual’s placement on scales of prejudice, discrimination, stereotyping, and/or support for particular kinds of diversity policies. A minority (27%) of the studies were experimental in design, involving attempts to modify participant’s beliefs in diversity ideologies while assessing the effects of the change in beliefs on measure of prejudice, discrimination, stereotyping, and/or support for diversity policies.
4. The meta-analysis adopts a progressive political bias as part of its definition of “high quality intergroup relations”.
In the abstract of the paper, the authors refer to their focus on “4 indicators of high quality intergroup relations—reduced prejudice, discrimination, stereotyping, and increased diversity policy support. Thus, as part of their DEFINITION of “high quality intergroup relations”, the authors include support for what are generally considered to be politically progressive social policies, such as affirmative action, reparations, and granting citizenship automatically to the children of illegal aliens. It is apparently simply a part of the authors’ own political beliefs that support for what are generally considered to be politically progressive social policies is a component of or prerequisite for “high quality intergroup relations”. At a minimum, the authors should have cited actual evidence for this claim. They did not.
5. Correlation does not equal causation
Given that 73% of the studies included in the meta-analysis involved a correlational design, and given that the findings from correlation studies tell us little if anything about causation, the ability of the findings from the meta-analysis to draw strong causal conclusions is highly limited. The authors of the meta-analysis openly acknowledge the limitations of correlational studies, but still slip frequently into the use of causal language when describing results that are based heavily on the findings from correlational studies. Indeed, the simple labeling of discrimination, prejudice, stereotyping and support for diversity policies as “outcome measures” presupposes a particular direction of causation. Throughout the abstract, and in many places throughout the body of the paper, the authors refer to “the effects of” diversity ideologies, implying quite directly that ideologies have causal effects on the so-called outcome variables. Not only is it the case that such conclusions cannnot be conclusively drawn based upon correlational findings, but in at least one of the papers included in the analysis, the authors of the paper specifically argue for a reverse direction of causation (arguing that a so-called “outcome” measure is the real causal variable while adoption of a form of diversity ideology is the “effect”).
The meta-analysis does include the analysis of some experimental studies as well, and drawing causal conclusions is more firmly grounded when based upon evidence from such studies. Unfortunately, the authors do not provide separate analyses of the correlational vs experimental studies, so little can be said about how strongly the experimental findings support the overall conclusions of the paper. What the authors DO say, however, is that they lack sufficient evidence to determine “whether the effect of diversity ideologies on intergroup relations or the effect of intergroup relations on diversity ideologies is stronger”. Of course, the frequent use of causative language throughout the paper (references to the “effects” of diversity ideologies) is wholly inconsistent with the qualification quoted above. There should not have been ANY causal language used when describing the findings from the meta-analysis, and it is surprising that Dr. Grant did not comment on that problematic feature of the meta-analysis paper.
6. Did the meta-analysis actually find that color-blindness was not associated with what the authors define as elements of high quality inter-group relations?
No. What the meta-analysis DID find (a point noted by Hughes in his comments about Anderson’s Free Press comments) is that color blindness was significantly associated with lower levels of stereotyping and prejudice – findings entirely consistent with Hughes’ claims in his TED talk.
7. In that case, what about multiculturalism?
Like color-blindness, multiculturalism was associated with lower levels of stereotyping and prejudice, and was also negatively associated with discrimination. The magnitude of these associations was larger for multiculturalism than color-blindness. However, far and away the biggest difference between color-blindness and multiculturalism is that the former was not positively associated with support for progressive diversity-related social policies whereas there was a strong association between support for those policies and belief in multiculturalism.
The findings regarding social policy support are not surprising. Indeed, it is essentially a defining feature of belief in color blindness that one would support treating everyone equally, without regard to the individual’s race, whereas a defining feature of belief in multiculturalism (as operationalized in the meta-analysis paper) is that one supports taking race into account as part of governmental and business and educational policies.
As was noted above, however, the authors of the meta-analysis have simply defined support for those kinds of policies as an essential component of “high quality intergroup relations”, but provide zero evidence that intergroup relations actually improve when organizations (whether business or educational or governmental) adopt these kinds of social policies. One might argue that it would have been just as reasonable (and consistent with evidence mentioned below) to have replaced “support for diversity policies” with “support for equality policies” as one of the “outcome” measures in the meta-analysis – in which case, it seems very likely that color-blindness would have emerged as having the strongest relationship with “high quality intergroup relations”. The decision by the authors of the meta-analysis to make support for progressive political policies one of their so-called outcome variables reflected a political value judgment on their part, and was not based on a scintilla of evidence (certainly not any that they provide).
8. Well, then – what about the quality of “intergroup relations”– the variable that is ostensibly the focus of the meta-analysis?
The four “outcome” variable included in the meta-analysis serve, within the analysis, as proxies for the quality of intergroup relations. However, very few of the studies surveyed included any more direct kind of measure of intergroup relations, and the authors of the meta-analysis seem very comfortable completely ignoring the absence of that evidence (evidence that would seem to be required for them to make the arguments that they make in the discussion and conclusion sections of the paper).
Ironically, given Grant’s and Anderson’s claim that the findings from the meta-analysis disprove the thesis of Hughes’ talk, it was Hughes who presented evidence more directly bearing on the question of the relationship between diversity ideologies and the quality of intergroup relations. Hughes presented the findings from a large scale survey (just one of a number of surveys that have shown the same pattern) showing that during the past 10-15 years, as the ideology of color blindness has been increasingly and prominently criticized within progressive media circles and at elite universities, ratings by both racial minorities and whites of the quality of intergroup relations has DECLINED. In other words, coincident with the decline in belief in color blindness and increased adoption of multiculturalism within society, there has been a decline, rather than an increase, in average ratings of intergroup relations. Neither Grant, nor Anderson, nor the authors of the meta-analysis, address this seemingly dramatic contradiction between their claims and what has actually happened in our society during the past one-to-two decades.