Bad Study Claims Lawyers are Racist

Written in Black & White

Exploring Confirmation Bias in Racialized Perceptions of Writing Skills

This is a paper about bias which claims lawyers are racist. But they don't know what bias is:

CONFIRMATION BIAS: A mental shortcut – a bias – engaged by the brain that makes one actively seek information, interpretation and memory to only observe and absorb that which affirms established beliefs while missing data that contradicts established beliefs.

Some mental shortcuts, like some shortcuts while driving, are good ideas. They can be time savers. This is a stupid definition of what a "bias" is.

Some mental shortcuts work, and are valuable. Others don't. It's important to try to figure out which are which. And an unbiased person can make a mistake evaluating whether a particular mental shortcut works well.

Shortcuts don't make one do anything. They are options but can't control you.

The idea of a bias making someone do something makes more sense than a shortcut controlling a person, but is still mistaken. One way to see it's mistaken is to consider that sometimes a person recognizes he's biased and doesn't obey the bias.

The dictionary doesn't know what a bias is either:

a tendency to believe that some people, ideas, etc., are better than others that usually results in treating some people unfairly

Is the belief rational? This definition doesn't care.

I believe Objectivism is better than most rivals one might compare it with. But some anti-Objectivists believe I treat them unfairly in discussion by not conceding that Ayn Rand is a monster.

Recognizing something is superior isn't automatically bias. Some things are superior. Bias has to do with irrationality: e.g. believing something is superior for bad reasons you are unwilling to reconsider.

The partners were originally given 4 weeks to complete the editing and rating, but we had to extend deadline to 7 weeks in order to obtain more responses.

The study changed the rules midway in order to reach different conclusions than it would have if it followed the original plan.

we deliberately inserted 22 different errors

Maybe the response rate was worse than expected because people weren't thrilled about editing an essay containing 22 deliberate errors. I wonder how realistic the errors were, and why they didn't use a real research memo. Using an artificial memo adds an extra source of error: it could be poorly designed.

Name: Thomas Meyer

Seniority: 3rd Year Associate
Alma Mater: NYU Law School
Race/Ethnicity: African American

Half the participants saw the same headings except with "Caucasian" instead of "African American". I see a danger here that people would find it strange to be told the Race/Ethnicity of the author of what they are reading, and therefore act differently than in regular life.

One possibility is some people saw this was a transparent attempt at a racism study and gave a reply to manipulate the results according to their political preference. Others might decide not to participate when they see this. It's important the participants don't know it's a racism study, but this is a big clue.

An even bigger issue here is there was no control group which received memos with no Race/Ethnicity heading. Wouldn't a control group be a good idea?

There was no significant correlation between a partner’s race/ethnicity and the differentiated patterns of errors found between the two memos. There was also no significant correlation between a partner’s gender and the differentiated patterns of errors found between the two memos.

What about the partner's political party? His age? Or a million other things.

Why are race/ethnicity and gender the two things they looked at? It's plausible that one major US political party is more racist than the other one. And it's plausible that old people are more racist than young people.

In order to create a study where we could control for enough variables to truly see the impact of confirmation bias, we did not study the potential variances that can be caused due to the intersection of race/ethnicity, gender, generational differences and other such salient identities.

How does ignoring the age of participants make the study better controlled?

The exact same memo, averaged a 3.2/5.0 rating under our hypothetical “African American” Thomas Meyer and a 4.1/5.0 rating under hypothetical “Caucasian” Thomas Meyer.

This is their main point: they claim lawyers are racist.

We undertook this study with the hypothesis that unconscious confirmation bias in a supervising lawyer’s assessment of legal writing would result in a more negative rating if that writing was submitted by an African American lawyer in comparison to the same submission by a Caucasian lawyer.

What about conscious bias? They explicitly said the race. A participant could consciously notice.

When expecting to find fewer errors, we find fewer errors. When expecting to find more errors, we find more errors. That is unconscious confirmation bias. Our evaluators unconsciously found more of the errors in the “African American” Thomas Meyer’s memo, but the final rating process was a conscious and unbiased analysis based on the number of errors found.

This is a story which isn't contradicted by their study. Many other stories also aren't contradicted by the study. Why are they concluding this particular story? For example, the evaluators could have had conscious bias. Saying it's unconscious bias is just making up a story about what happened.

Other things could be going on. Maybe the writing style of the memo was culturally white. Then the people told it had it had a white author would just read it and nothing special happens. But the people told it had a black author might notice the clash between the white style and the black author claim. This could get them to wake up and pay more attention because there was something unexpected or surprising or interesting about the memo. They could then have found more errors simply because they were more awake while reading.

When partners say that they are evaluating assignments without bias

Wait, were they asked if they were racist? Wouldn't that give away what kind of study it was? Or was that only done afterwards? Why wasn't the procedure explained?

In any case, wouldn't you expect a lot of conscious racists to lie? So people claiming they aren't racist doesn't differentiate between conscious and unconscious racists very well.

Conclusion

The paper is too light on details, and has too many errors, to make a big deal out of. If racism is a big problem, it shouldn't be that hard to do a high quality study to show it. I would expect that already to have been done, given the intense interest in this topic.

So, people claiming racism is a big problem: where is that high quality study? Link me to it or tell me why it hasn't been done yet.

Addendum

At the end of the paper there is some extra stuff like brags about how the people doing the study are biased. They get paid to teach people to be less racist. Their study is marketing for their services. Is that sad or is it amusing? I don't know. I want to point out one more error:

EXAMPLE: In one law firm where we found that minority summer associates were consistently being evaluated more negatively than their majority counterparts, we created an interruption mechanism to infuse the subjective with objective. We worked with the firm to create an Assignment Committee, comprised of 3 partners through whom certain assignments were distributed to the summer associates and through whom the summer associates submitted work back to the partners who needed the work done. When the work was evaluated, the partners evaluating the work did not know which associate had completed the work. The assignments for this process were chosen judiciously, and there was a lot of work done to ensure buy-in from all partners. At the end of the summer, every associate had at least 2 assignments that had been graded blindly. The firm then examined how the blind evaluations compared with the rest of the associate’s evaluations and found that the blind evaluations were generally more positive for minorities and women and less positive for majority men.

It could be that people give better evaluations to their friends. And it could be that of the new employees, white men have the best social skills, due to different upbringings. So they make friends with the people doing the evaluating the best, and then get the best evaluations when it's not blind. But when there's blind evaluations, the social skills are irrelevant.

This is merely a story I made up. The point is it's possible. What happened in the example could involve racism by lawyers, or not. That the study authors only think about how their material is compatible with racism happening, but don't consider and discuss the non-racist explanations that account for it, shows their own selective attention, which they would call "bias".

[I wrote this post in one hour.]