I proposed a game on the Effective Altruism forum where people submit texts to me and I find three errors. I wrote this for that game. This is a duplicate of my EA post. I criticize Against the singularity hypothesis by David Thorstad.

Introduction

FYI, I disagree with the singularity hypothesis, but primarily due to epistemology, which isn't even discussed in this article.

Error One

As low-hanging fruit is plucked, good ideas become harder to find (Bloom et al. 2020; Kortum 1997; Gordon 2016). Research productivity, understood as the amount of research input needed to produce a fixed output, falls with each subsequent discovery.

By way of illustration, the number of FDA-approved drugs per billion dollars of inflation-adjusted research expenditure decreased from over forty drugs per billion in the 1950s to less than one drug per billion in the 2000s (Scannell et al. 2012). And in the twenty years from 1971 to 1991, inflation-adjusted agricultural research expenditures in developed nations rose by over sixty percent, yet growth in crop yields per acre dropped by fifteen percent (Alston et al. 2000). The problem was not that researchers became lazy, poorly educated or overpaid. It was rather that good ideas became harder to find.

There are many other reasons for drug research progress to slow down. The healthcare industry, as well as science in general (see e.g. the replication crisis), are really broken, and some of the problems are newer. Also maybe they're putting a bunch of work into updates to existing drugs instead of new drugs.

Similarly, decreasing crop yield growths (in other words, yields are still increasing but by lower percentages) could have many other causes. And also decreasing crop yields are a different thing than a decrease in the number of new agricultural ideas that researchers come up with – it's not even the right quantity to measure to make his point. It's a proxy for the actual thing his argument relies on, and he makes no attempt to consider how good or bad of a proxy it is, and I can easily think of some reasons it wouldn't be a very good proxy.

The comment about researchers not becoming lazy, poorly educated or overpaid is an unargued assertion.

So these are bad arguments which shouldn't convince us of the author's conclusion.

Error Two

Could the problem of improving artificial agents be an exception to the rule of diminishing research productivity? That is unlikely.

Asserting something is unlikely isn't an argument. His followup is to bring up Moore's law potentially ending, not to give an actual argument.

As with the drug and agricultural research, his points are bad because singularity claims are not based on extrapolating patterns from current data, but rather on conceptual reasoning. He didn't even claim his opponents were doing that in the section formulating their position, and my pre-existing understanding of their views is they use conceptual arguments not extrapolating from existing data/patterns (there is no existing data about AGI to extrapolate from, so they use speculative arguments, which is OK).

Error Three

one cause of diminishing research productivity is the difficulty of maintaining large knowledge stocks (Jones 2009), a problem at which artificial agents excel.

You can't just assume that AGIs will be anything like current software including "AI" software like AlphaGo. You have to consider what an AGI would be like before you can even know if it'd be especially good at this or not. If the goal with AGI is in some sense to make a machine with human-like thinking, then maybe it will end up with some of the weaknesses of humans too. You can't just assume it won't. You have to envision what an AGI would be like, or what many different things it might be like that would work (narrow it down to various categories and rule some things out) before you consider the traits it'd have.

Put another way, in MIRI's conception, wouldn't mind design space include both AGIs that are good or bad at this particular category of task?

Error Four

It is an unalterable mathematical fact that an algorithm can run no more quickly than its slowest component. If nine-tenths of the component processes can be sped up, but the remaining processes cannot, then the algorithm can only be made ten times faster. This creates the opportunity for bottlenecks unless every single process can be sped up at once.

This is wrong due to "at once" at the end. It'd be fine without that. You could speed up 9 out of 10 parts, then speed up the 10th part a minute later. You don't have to speed everything up at once. I know it's just two extra words but it doesn't make sense when you stop and think about it, so I think it's important. How did it seem to make sense to the author? What was he thinking? What process created this error? This is the kind of error that's good to post mortem. (It doesn't look like any sort of typo; I think it's actually based on some sort of thought process about the topic.)

Error Five

Section 3.2 doesn't even try to consider any specific type of research an AGI would be doing and claim that good ideas would get harder to find for that and thereby slow down singularity-relevant progress.

Similarly, section 3.3 doesn't try to propose a specific bottleneck and explain how it'd get in the way of the singularity. He does bring up one specific type of algorithm – search – but doesn't say why search speed would be a constraint on reaching the singularity. Whether exponential search speed progress is needed depends on specific models of how the hardware and/or software are improving and what they're doing.

There's also a general lack of acknowledgement of, or engagement with, counter-arguments that I can easily imagine pro-singularity people making (e.g. responding to the good ideas getting harder to find point by saying some stuff about mind design space containing plenty of minds that are powerful enough for a singularity with a discontinuity, even if progress slows down later as it approaches some fundamental limits). Similarly, maybe there is something super powerful in mind design space that doesn't rely on super fast search. Whether there is, or not, seems hard to analyze, but this paper doesn't even try. (The way I'd approach it myself is indirectly via epistemology first.)

Error Six

Section 2 mixes Formulating the singularity hypothesis (the section title) with other activities. This is confusing and biasing, because we don't get to read about what the singularity hypothesis is without the author's objections and dislikes mixed in. The section is also vague on some key points (mentioned in my screen recording) such as what an order of magnitude of intelligence is.

Examples:

Sustained exponential growth is a very strong growth assumption

Here he's mixing explaining the other side's view with setting it up to attack it (as requiring a super high evidential burden due to such strong claims). He's not talking from the other side's perspective, trying to present it how they would present it (positively); he's instead focusing on highlighting traits he dislikes.

A number of commentators have raised doubts about the cogency of the concept of general intelligence (Nunn 2012; Prinz 2012), or the likelihood of artificial systems acquiring meaningful levels of general intelligence (Dreyfus 2012; Lucas 1964; Plotnitsky 2012). I have some sympathy for these worries.[4]

This isn't formulating the singularity hypothesis. It's about ways of opposing it.

These are strong claims, and they should require a correspondingly strong argument to ground them. In Section 3, I give five reasons to be skeptical of the singularity hypothesis’ growth claims.

Again this doesn't fit the section it's in.

Padding

Section 3 opens with some restatements of material from section 2 which was also in the introduction some. And look at this repetitiveness (my bolds):

Near the bottom of page 7 begins section 3.2:

3.2 Good ideas become harder to find

Below that we read:

As low-hanging fruit is plucked, good ideas become harder to find

Page 8 near the top:

It was rather that good ideas became harder to find.

Later in that paragraph:

As good ideas became harder to find

Also, page 11:

as time goes on ideas for further improvement will become harder to find.

Page 17

As time goes on ideas for further improvement will become harder to find.

Amount Read

I read to the end of section 3.3 then briefly skimmed the rest.

Screen Recording

I recorded my screen and made verbal comments while writing this:

https://www.youtube.com/watch?v=T1Wu-086frA

---

Update: Thorstad replied and I wrote a followup post in response: Credentialed Intellectuals Support Misquoting

Link to the EA version of this post.

---

EA doesn’t have strong norms against misquoting or some other types of errors related to having high intellectual standards (which I claim are important to truth seeking). As I explained, misquoting is especially bad: “Misquoting puts words in someone else’s mouth without their consent. It takes away their choice of what words to say or not say, just like deadnaming takes away their choice of what name to use.”

Despite linking to lizka clarifying the lack of anti-misquoting norms, I got this feedback on my anti-misquoting article:

One of your post spent 22 minutes to say that people shouldn't misquote. It's a rather obvious conclusion that can be exposed in 3 minutes top. I think some people read that as a rant.

So let me try to explain that EA really doesn’t have strong anti-misquoting norms or strong norms for high intellectual standards and scholarship quality. What would such norms look like?

Suppose I posted a single misquote in Rationalization: AI to Zombies. Suppose it was one word added or omitted and it didn’t change the meaning much. Would people care? I doubt it. How many people would want to check other quotes in the book for errors? Few, maybe zero. How many would want to post mortem the cause of the error? Few, maybe zero. So there is no strong norm against misquotes. Am I wrong? Does anyone really think that finding a single misquote in a book this community likes would result in people making large updates to their views (even is the misquote is merely inaccurate, but doesn’t involve a large change in meaning)?

Similarly, I’m confident that there’s no strong norm against incorrect citations. E.g. suppose in RAZ I found one cite to a study with terrible methodology or glaring factual errors. Or suppose I found one cite to a study that says something different than what it’s cited for (e.g. it’s cited as saying 60% X but the study itself actually says 45% X). I don’t think anything significant would change based on pointing out that one cite error. RAZ’s reputation would not go down substantially. There’d be no major investigation into what process created this error and what other errors the same process would create. It probably wouldn’t even spark debates. It certainly wouldn’t result in a community letter to EY, signed by thousands of people with over a million total karma, asking for an explanation. The community simply tolerates such things. This is an example of intellectual standards I consider too low and believe are lowering EA’s effectiveness a large amount.

Even most of RAZ’s biggest fans don’t really expect the book to be correct. They only expect it to be mostly correct. If I find an error, and they agree it’s an error, they’ll still think it’s a great book. Their fandom is immune to correction via pointing out one error.

(Just deciding “RAZ sucks” due to one error would be wrong too. The right reaction is more complicated and nuanced. For some information on the topic, see my Resolving Conflicting Ideas, which links to other articles including We Can Always Act on Non-Criticized Ideas.)

What about two errors? I don’t think that would work either. What about three error? Four? Five? Nah. What exactly would work?

What about 500 errors? If they’re all basically indisputable, then I’ll be called picky and pedantic, and people will doubt that other books would stand up to a similar level of scrutiny either, and people will say that the major conclusions are still valid.

If the 500 errors include more substantive claims that challenge the book’s themes and concepts, then they’ll be more debatable than factual errors, misquotes, wrong cites, simple, localized logic errors, grammar errors, etc. So that won’t work either. People will disagree with my criticism. And then they won’t debate their disagreement persistently and productively until we reach a conclusion. Some people won’t say anything at all. Others will comment 1-5 times expressing their disagreement. Maybe a handful of people will discuss more, and maybe even change their minds, but the community in general won’t change their minds just because a few people did.

There are errors that people will agree are in fact errors, but will dismiss as unimportant. And there are errors which people will deny are errors. So what would actually change many people’s minds?

Becoming a high status, influential thought leader might work. But social climbing is a very different process than truth seeking.

If people liked me (or whoever the critic was) and liked some alternative I was offering, they’d be more willing to change their minds. Anyone who wanted to say “Yeah, Critical Fallibilism is great. RAZ is outdated and flawed.” would be receptive to the errors I pointed out. People with the right biases or agendas would like the criticisms because the criticisms help them with their goals. Other people would interpret the criticism as fighting against their goals, not helping – e.g. AI alignment researchers basing a lot of their work on premises from RAZ would tend to be hostile to the criticism instead of grateful for the opportunity to stop using incorrect premises and thereby wasting their careers.

I’m confident that I could look through RAZ and find an error. If I thought it’d actually be useful, I’d do that. I did recently find two errors in a different book favored by the LW and EA communities (and I wasn’t actually looking for errors, so I expect there are many others – actually there were some other errors I noticed but those were more debatable). The first error I found was a misquote. I consider it basically inexcusable. It’s from a blog post, so it would be copy/pasted not typed in, so why would there be any changes? That’s a clear-cut error which is really hard to deny is an error. I found a second related error which is worse but requires more skill and judgment to evaluate. The book has a bunch of statements summarizing some events and issues. The misquote is about that stuff. And, setting aside the misquote, the summary is wrong too. It gives an inaccurate portrayal of what happened. It’s biased. The misquote error is minor in some sense: it’s not particularly misleading. The misleading, biased summary of events is actually significantly wrong and misleading.

I can imagine writing two different posts about it. One tries to point out how the summary is misleading in a point-by-point way breaking it down into small, simple points that are hard to deny. This post would use quotes from the book, quotes from the source material, and point out specific discrepancies. I think people would find this dry and pedantic, and not care much.

In my other hypothetical post, I would emphasize how wrong and misleading what the book says is. I’d focus more on the error being important. I’d make less clear-cut claims so I’d be met with more denials.

So I don’t see what would actually work well.

That’s why I haven’t posted about the book’s problems previously and haven’t named the guilty book here. RAZ is not the book I found these errors in. I used a different example on purpose (and, on the whole, I like RAZ, so it’s easier for me avoid a conflict with people who like it). I don’t want to name the book without a good plan for how to make my complaints/criticisms productive, because attacking something that people like, without an achievable, productive purpose, will just pointlessly alienate people.

I quit the Effective Altruism forum due to a new rule requiring all new posts and comments be basically put in the public domain without copyright, so anyone could e.g. sell a book of my posts without my consent (they’d just have to give attribution). More info. I had a bunch of draft posts, so I’m posting some of them here with minimal editing. In general, I’m not going to submit them as link posts at EA myself. If you think they should be shared with EA as link posts, please do it yourself. I’m happy for other people to share links to my work at EA or on social media. Please share stuff in whatever ways you think are good to do.

---

Let me summarize events from my perspective.

I read a book EA likes and found a misquote in it (and other problems).

Someone misquoted me twice in EA forum discussion. They seemed to think that was OK, not a big deal, etc. And no one in the audience took my side or said anything negative about misquotes.

The person who misquoted me (as well as anyone reading) didn’t want to talk about it or debate the matter.

In an open questions thread, I asked about EA’s norms regarding misquotes.

In response, someone misquoted the EA norms to me, which is pretty ironic and silly.

Their claim about EA norms was basically that misquotes aren’t important.

When I pointed out that they had misquoted, they didn’t really seem to care or think that was bad. Again, there were no signs the audience thought misquoting was bad, either.

Lizka, who was the person being misquoted since she wrote the EA norms document, commented on the matter. Lizka’s comment communicated:

• She agrees with me that the norms were misquoted.
• But she didn’t really mind or care.
• EA has no strong norm against misquoting.
• The attitude to misquotes is basically like typos: mistakes and accidents happen and we should be tolerant and forgiving about that.

Again, no one wanted to talk with me about the matter or debate it.

I wrote an article explaining that misquoting is bad. I compared misquoting to deadnaming because the misquoted norm was actually about deadnaming, and I thought that read as a whole it’s actually a good norm, and the same norm should be used for misquoting.

The EA norm on deadnaming is basically: first, don’t do it, and second, if it’s a genuine accident, that’s alright, but don’t do it again.

Whereas EA’s current misquoting norm is more like: misquotes are technically errors, so that’s bad, but no one particularly cares.

Misquotes are actually like deadnaming. Deadnaming involves exercising control over someone else’s name without their consent, when their name should be within their control. Misquotes involve exercising control over someone else’s words/speech without their consent, when their words/speech should be within their control. Misquotes and deadnaming both violate the personal boundaries of a person and violate consent.

Misquotes are also bad for reasons of scholarship, accuracy and truth seeking. I believe the general attitude of not caring about “small” errors is a big mistake.

Misquotes are accepted at EA due to the combination of not recognizing how they violate consent and victimize someone (like deadnaming), and having a culture tolerant of “small” errors and imprecision.

So, I disagree, and I have two main reasons. And people are not persuaded and don’t want to debate or give any counter-arguments. Which gets into one of the other main topics I’ve posted about at EA, which is debating norms and methodology.

All this so far is … fine. Whatever. The weird part comes next.

The main feedback I’ve gotten regarding misquoting and deadnaming is not disagreement. No one has clearly admitted to disagreeing with me and e.g. claimed that misquoting is not like deadnaming.

Instead, I’ve been told that I’m getting downvoted because people agree with me too much: they think it’s so obvious and uncontroversial that it’s a waste of time to write about.

That is not what’s happening and it’s a very bizarre excuse. People are so eager to avoid a debate that they deny disagreeing with me, even when they could tell from the title that they do disagree with me. None of them has actually claimed that they do think misquoting is like deadnaming, and should be reacted to similarly.

Partly, people are anti-misquoting in some weaker way than I am, just like they are anti-typos but not very strongly. The nuance of “I am more against misquoting than you are, so we disagree” seems too complex for some people. They want to identify as anti-misquoting, so they don’t want to take the pro-misquoting side of a debate. The actual issue is how bad misquoting is (or we could be more specific and specify 20 ways misquoting might be bad, 15 of which I believe, and only 5 of which they believe, and then debate the other 10).

I wrote a second article trying to clarify to people that they disagree with me. I gave some guided thinking so they could see it for themselves. E.g. if I pointed out a misquote in the sequences, would you care? Would it matter much to you? Would you become concerned and start investigating other quotes in the book? I think we all know that if I found a single misquote in that book, it would result in no substantive changes. I think it should; you don’t; we disagree.

After being downvoted without explanation on the second article about misquoting, I wrote an article about downvotes being evidence, in which I considered what different interpretations of downvotes and different reactions. This prompted the most mixed voting I’d gotten yet and a response saying people were probably just downvoting me because they didn’t see the point of my anti-misquoting articles because they already agree with me. That guy refused to actually say he agrees with me himself, saying basically (only when pressed) that he’s unsure and neutral and not very interested in thinking or talking about it. If you think it’s a low priority unimportant issue, then you disagree with me, since I think it’s very important. Does he also think deadnaming is low priority and unimportant? If not, then he clearly disagrees with me.

It’s so weird for people who disagree with me to insist they agree with me. And Lizka already clarified that she disagrees with me, and made a statement about what the EA norms are, and y’all are still telling me that the community in general agrees with me!?

Guys, I’ve struck a nerve. I got downvotes because people didn’t like being challenged in this way, and I’m getting very bizarre excuses to avoid debate because this is a sensitive issue that people don’t want to think or speak clearly about. So it’s important for an additional reason: because people are biased and irrational about it.

My opinions on this matter predate EA (though the specific comparison to deadnaming is a new way of expressing an old point).

I suspect one reason the deadnaming comparison didn’t go over well is that most EAers don’t care much about deadnaming either (and don’t have nuanced, thoughtful opinions about it), although they aren’t going to admit that.

Most deadnaming and most misquoting is not an innocent accident. I think people know that with deadnaming, but deny it with misquoting. But explain to me: how did the wording change in a quote that you could have copy/pasted? That’s generally not an innocent accident. How did you leave out the start of the paragraph and take a quote out of context? That was not a random accident. How did you type in a quote from paper and then forget to triple check it for typos? That is negligence at best, not an accident.

Negligently deadnaming people is not OK. Don’t do it. Negligently misquoting is bad too for more reasons: violates consent and harms scholarship.

This is all related to more complex and more important issues, but if I can’t persuade anyone of this smaller initial point that should be easy, I don’t think trying to say more complex stuff is going to work. If people won’t debate a relatively small, isolated issue, they aren’t going to successfully debate a complex topic involving dozens of issues of similar or higher difficulty as well as many books. One purpose of talking about misquoting is it it’s test issue to see how people handle debate and criticism, plus it’s an example of one of the main broader themes I’d like to talk about which is about the value of raising intellectual standards. If you can’t win with the small test issue that shouldn’t be very hard, you’ve gotta figure out what is going on. And if the responses to the small test issue are really bizarre and involve things like persistently denying disagreeing while obviously disagreeing … you really gotta figure out what is going on instead of ignore that evidence. So I’ve written about it again (this post).

If you want to find details of this stuff on the EA forum and see exactly what people said to me, besides what is linked in my two articles about misquoting that I linked above, you can also go to my EA user profile and look through my post and comment history there.

Summary: I criticize David Thorstad's reply to me regarding his citation errors. I summarize his supporters' credentials as evidence about problems with academia, think tanks, and the world’s current thought leaders. There's a serious problem with scholarship standards, including misquoting, among credentialed intellectuals.

Introduction

In Criticizing "Against the singularity hypothesis", I criticized the content of David Thorstad's paper. In Checking Citations from David Thorstad, I criticized quoting and citation errors in the same paper. Thorstad tweeted replies about the quoting and citation issues but didn't reply to the criticisms of his ideas.

For context, Thorstad has a philosophy PhD from Harvard and works at an Oxford think tank (source). The main reason I’m writing this is because I think it reveals a lot about what kind of world and intellectual climate we live in.

The reason I wrote about Thorstad initially was because I let people at the Effective Altruism forum submit literature (that they liked a lot) to me for criticism. I checked three quotations and their citations from the paper partly because some people at the Effective Altruism forum denied that misquotes were a widespread or common problem.

Thorstad’s Tweets

Thorstad wrote four initial tweets in a row, plus one tweet replying to a supporter.

"Is Thorstad just one bad thinker" because ... I made two typos and said one word people don't like. Seriously? https://criticalfallibilism.com/checking-citations-from-david-thorstad/

This is a misquote. It suggests that I called Thorstad a bad thinker because of three criticisms which are mischaracterized (straw manned) as two typos and a disliked word.

Here’s what I actually wrote:

Is Thorstad just one bad thinker, while most intellectuals do better? Should we blame Thorstad personally? I don’t think so. Based on doing this kind of thing many times, I think Thorstad’s mistakes are pretty normal. There’s a widespread problem related to intellectual culture and norms. The attitudes of many people need to change, not the actions of a few.

Reading Thorstad, if you figured out that I was writing a question (which isn’t very clear), you’d think my answer to the question was yes. But it wasn’t. My answer was no. I said Thorstad is not “just one bad thinker”, but Thorstad is misleading his audience to believe that I claimed Thorstad is just one bad thinker.

Thorstad misquoted my article that pointed out his misquoting problem, and this misquote substantially changed the meaning of what I said. What I was actually saying is that Thorstad’s errors are representative and illustrative of what many other scholars are like. I was denying that the problem is Thorstad personally.

I also didn’t accuse Thorstad of making two typos and saying one disliked word. That’s an egregious mischaracterization. I accused him of making three errors related to quotes and citations. I didn’t say any of the errors were typos and I didn’t think any of the errors could be explained away as merely typos.

The disliked word comment is vague but I think Thorstad means that the paper noted something, but he said they “lamented” it. As the lead in to a quote, that misrepresented the meaning of the quote. The people Thorstad was talking about expressed themselves using a neutral word. Thorstad misrepresented their neutral writing as highly negative. Thorstad is welcome to form his own opinion, but not to speak for others and abuse quotations to mislead readers about what they said.

Back to quoting Thorstad tweets:

Correction, one mistake. The year on Good isn't wrong: https://www.sciencedirect.com/science/article/abs/pii/S0065245808604180

Here, Thorstad is reiterating his error about the year that Good’s paper was published. It was 1965, but Thorstad mistakenly cited it as 1966. (This is one of the errors he mischaracterized as a typo earlier. But if he’d merely made a typo, he wouldn’t still be defending 1966 as the correct year.)

Instead of learning from my correction, Thorstad has refused to change his mind. He still thinks it’s 1966. Why? He found a modern source that says 1966 and cherrypicked that information. It looks like he really wanted to be right and get me back for accusing him of making errors, so instead of doing unbiased research to find the truth, he just found the first thing he could that backed him up and then he claimed he was right all along. (Or maybe, rather than it being bias, he just doesn’t know how to do effective research on an issue like this.) I don’t think he noticed the text “Copyright © 1965” on the page he linked, which is a hint that 1966 might be an error.

It’s contradictory to, at the same time, claim it’s not a big deal but also be defensive enough to incorrectly claim your error was actually true. If it doesn’t matter, why go find some false information to try to defend yourself with?

How can we settle this issue definitively? Previously I linked a scan of an old reprint showing the date 1965. I thought that was convincing but apparently Thorstad didn’t. I didn’t expect this point to actually be disputed or I would have given more evidence. It’s pretty easy to do a better job of looking it up now that there’s a dispute.

The Internet Archive scanned the original document. It says 1965 on the title page.

It’s one thing to make a factual mistake. It’s much worse to refuse a correction and reiterate the mistake. Even if the original mistake wasn’t a big deal, Thorstad’s reaction to it matters.

Wait a moment ... my "typo" was correcting a grammar error someone else made and not writing [sic] like a jerk!?

What is “typo” a quote of? If it’s me, it’s a misquote. If it’s of himself in his earlier tweet where he wrote “typos”, that’s odd and unclear. It seems like he’s saying that I accused him of making a typo, and using quote marks, but I didn’t say that.

This writing is ambiguous. By “and not” does Thorstad mean “without” or “instead of”?

If Thorstad means “without”, then he’s wrong. You use [sic] when you don’t make edits to fix typos, not when you do.

If Thorstad means “instead of”, then he’s presenting a false alternative. He could have made the correction using square brackets or he could have given the exact quote without writing [sic]. Those were both reasonable alternatives. His choices weren’t just using [sic] or stealth editing a quote.

Also, writing [sic] is fine and doesn’t make one a jerk. [sic] is a scholarly tool used as part of literal, accurate quoting. Having a negative attitude towards using [sic] shows a bad attitude towards quotation literalness and accuracy.

Thorstad seems to be implying/confessing that he edited the quote intentionally. He’s denying it was a typo, or in other words denying it was an accident. I believe intentionally editing quotes (without using one of the few allowed exceptions or properly indicating the edit) is an ethics violation that is against the honor codes at many universities. I actually think the written rules about quotes and cites are often reasonable. The existence of those rules is then sometimes used to deny there’s a problem with intellectual culture. Surely whatever the rules say to do is what most people do, right? Sadly, no. Systemic reform is needed so that most intellectuals actually want to follow those rules and value them instead of considering them overly pedantic.

I didn’t think it was just an accidental typo. I thought it was due to some sort of bad attitude or other problem with Thorstad’s ideas. Thorstad has confirmed that I was right.

@curi42. Sad face.

I appreciate that Thorstad tagged me. Without a notification, I wouldn’t have seen these tweets.

I don’t appreciate the “Sad face” comment, which I read as unserious and mean. There are many similar comments – in terms of both style and content – in the replies to Thorstad.

As context for Thorstad’s fifth tweet, Dan Carey replied to Thorstad:

Odd for them to be so nit-picky about citations. This is from their criticism of your piece. Arn't all of these ideas addressed explicitly in Bloom et al (2020) which u cite in their quote of u right above?

I’ve left out an uncropped phone screenshot showing mostly the Error One section of my Criticizing "Against the singularity hypothesis”. Thorstad wrote back agreeing with Carey:

I know, right? I cite my sources like a normal person

Calling me an abnormal person is a social insult. One of its purposes is to pressure me to increase my conformity. Instead of arguing that his way of using citations is good or criticizing my way, Thorstad instead calls his way normal. But calling something normal isn’t a rational argument that it’s good. And I already said Thorstad is normal in my article! He doesn’t seem to comprehend that I’m criticizing something I consider normal instead of trying to fit in and be normal myself.

My article made statements like “I think Thorstad’s mistakes are pretty normal”. My point was that there’s a widespread problem with intellectual culture, not a problem with Thorstad individually. I was saying that what’s currently normal is problematic and should be reformed. In that context, Thorstad asserted something is normal and assumed that makes it good, which is the logical fallacy called begging the question (which basically means assuming a conclusion about one of the points currently under debate, like you don’t understand that it’s being disputed).

Carey’s and Thorstad’s claim is factually false. My points in that section are not “all” addressed “explicitly” in Bloom et al (hereafter just Bloom). I’ll give one example. I said “The healthcare industry, as well as science in general (see e.g. the replication crisis), are really broken”. I text searched the Bloom paper for “crisis” and “replic” but found no discussion of the replication crisis to address my argument. I also skimmed but didn’t see it covered.

Also, even if Bloom had “explicitly” addressed “all” my points – which they didn’t – there was no way to know that from Thorstad’s writing. Thorstad cited Bloom as one of three sources for the claim “As low-hanging fruit is plucked, good ideas become harder to find”. Thorstad provided no information about Bloom addressing specific counter-arguments to a claim (about healthcare research productivity) that Thorstad brought up in a later paragraph with a different citation. In general, you need to repeat a citation each time you use it for a different purpose or else give some kind of explanatory comments.

The Credentials of Thorstad’s Audience

Intellectuals commonly tweet with their real names and state their credentials and employer in their public profiles. I want to review their credentials to show what kind of people make or side with scholarship errors. I’m leaving out their names and refutations of what they said, but if any of them wants to debate me, I can elaborate.

First I’ll share credentials of people who wrote reply tweets. None of their tweets were significantly better than Thorstad’s tweets that I analyzed above. They generally seemed similar to Thorstad although some were worse.

There’s a PhD student at the philosophy department of the London School of Economics (which was founded by Karl Popper), a grantmaker in global priorities research and international policy at Longview who has a philosophy PhD from Rutgers, an assistant professor of philosophy at University of Sheffield who is a fellow-in-residence at a Harvard center, a Ph.D. student at Pardee RAND, an associate professor at University of Pennsylvania, an employee at OpenAI and a PhD student at Oxford.

Second, here are some credentials from people who clicked the like button on Thorstad’s tweets. There’s a Harvard medical doctor PhD student, a University of Minnesota philosophy professor, an executive research coordinator at Rethink Priorities, a philosophy professor who wrote a book, a professor of mathematical statistics, two AGI safety researchers, a philosopher working at Rutgers with a PhD from Ohio State, a philosopher of science at University of Groningen, and a philosopher with a PhD from Cambridge. There’s also a philosopher at University of Bristol, whose name I recognize because he wrote a Bayesian textbook that I’ve looked at. I was trying to find a book explaining Bayesian epistemology premises and reasoning in a way I could engage in debate with, but it wasn’t suitable.

(I gathered these credentials a few months ago. Some could have changed before publication.)

Thorstad Reinforced My Point

Thorstad and the people who replied to his tweets reinforced my point (and they seem unaware of this). My point was that many scholars are OK with misquoting, which is an example of how intellectual culture in general is broken and should be reformed. The people tweeting all seem to think misquoting isn’t a big deal, and that people (like me) who consider it a big deal should be shunned. So they are examples for my claim.

The typical response I get when I criticize misquoting in general is being ignored or being told that everyone already agrees with me that misquotes are bad so there’s no point in talking about it. But the typical responses I get when I criticize specific examples of misquoting are hatred, denials that misquotes matter, etc. I’ve written various things about this including Misquoting and Scholarship Norms at EA and EA Misquoting Discussion Summary.

Rather than say his mistakes were accidents, Thorstad reiterated them. He didn’t retract his inaccurate “lamented” paraphrase. He repeated his false claim that 1966 was the true publication year. He ignored the issue that he changed the type of quotation marks used within a quote. And he implied that he edited the Chalmers quote on purpose, not as an accidental typo, because he thinks that correctly using [sic] makes one a jerk. Thorstad also agreed with a commenter falsely claiming that Bloom had answered all my arguments about a particular issue already.

Who has bad attitudes to scholarship including low standards for quotation accuracy? A lot of the problem is PhD students and people who already have PhDs. These tweeters and the people who like the tweets are (not exclusively) a bunch of graduate students, professors and think tank employees. Many of them went to or work at good universities. Thorstad got his PhD at Harvard and works at Oxford.

In articles like Ignoring “Small” Errors, EA Should Raise Its Standards and “Small” Errors, Frauds and Violences, I explained my view that dismissing errors as “small” can result in large problems like incorrect conclusions. In general, you can’t actually judge which errors are large or important until after you find solutions. In retrospect, you can see how much work they were to solve and how much difference the solution makes. But before you understand the issue, you don’t know how big a deal it is.

Also, I don’t think all small or picky points matter. I say an error is a reason an idea will fail for a goal (that someone involved has). I just think some particular errors, like misquotes, matter to goals like having productive discussions. Lots of “little” errors in discussions add up to the current, widespread problem of most discussions becoming inconclusive messes. I think higher standards for some fairly objective issues like quotation and citation accuracy (as well as logic, reading comprehension, factual accuracy and using arguments instead of insults) is relatively easily achievable and would actually help a lot.

I’m aware that some people will respond to this article by thinking I’m even more unreasonably pedantic than they thought before. I wrote this anyway because I disagree and I think I’m commenting on some important, widespread social-intellectual problems that are making society, philosophy and science worse.

Conclusion

I wrote about how misquotes and citation errors are a widespread problem. The response – primarily from credentialed intellectuals – was basically to mock me for caring about details like those. In other words, people agreed with my position that many intellectuals, like them, think inaccurate quotes and citations are OK. Credentialed intellectuals aren’t even close to as rational as the general public views them as. This is one of the factors making it really difficult to have productive discussions about intellectual issues.

While it may be tempting for many academics (who aren’t Thorstad’s Twitter buddies) to place the blame on Thorstad, I believe it’s a widespread issue and Thorstad is merely a typical example. I’ve seen similar attitudes in many other places, and I’ve failed to find better attitudes anywhere. If you know of a group with better attitudes, please tell me.

Do you disagree? I’m open to organized, serious debate following explicit methods and aimed at reaching conclusions. See my debate policy.

In the last few months, Dennis Hackethal did an extensive review of my writing and wrote a large amount about me. One thing he did was look for misquotes. His attitude to misquotes has similarities to mine. It looks like he may have been inspired by me, but he doesn't credit me and he's pedantic in ways I disagree with.

If I misquoted, I'd want to know. But I'm happy to report he was unable to find a single error in my quotations.

I thought I was doing a good job with my quotations, even in informal contexts, but it's nice for an independent third party to spend unpaid hours validating this. And he's hostile towards me, so I'm not worried that he's holding back criticism to be nice. However, a potential source of error for the review is that he doesn't understand misquotes very well (see below); maybe a smarter reviewer would have found errors.

He claimed to have found errors for 15 quotes and called my results "terrible", but none are actually errors. For each quote, he presented specific information about what errors he's alleging. He didn't claim that I got a single word wrong. Instead, he was pedantic.

Most of the "... misquotes ..." are just that I often don't put an ellipsis at the start or end of a quote (see how weird it looks earlier in this sentence?). That's an intentional style choice which is widely used. Style guides have varied guidelines about this topic, but I don't know of any that agree with Hackethal's view that you always need starting and ending ellipses when quotes start or end mid-sentence. Those ellipses are commonly discouraged. On his website he recommends following style guides, seemingly unaware that they disagree with him.

To be thorough, here are more details covering all the other alleged errors. Sometimes emails or PDFs have mid-sentence line breaks which I fix. Also, I once clearly labelled a quote of song lyrics as abridged and linked the full lyrics, but he's calling that an error because I didn't individually indicate every abridgment within the quote. And once, when writing in plain text, I didn't include italics in a quote because that isn't supported by the plain text format. I was writing an email to people who I expected to be unfamiliar with internet norms and modern technology, and I wanted to keep it short and avoid confusing them, so I did it that way on purpose. Also, I quoted a dictionary as saying "the" even though the website for a different dictionary says "The" with a capital letter (Hackethal apparently confused the dictionaries). I copy/pasted my quote from the Mac Dictionary app, which has a lower case letter; I didn't change the capitalization.

So the review of my use of quotations didn't find any mistakes, only intentional style choices (mostly omitting ellipses at the start or end of quotes).

Update 2025-03-18: An error has been brought to my attention which I've corrected. I wrote "on the dictionary's website it says "The" with a capital letter", but I was wrong. I didn't carefully double check all of Hackethal's claims and repeated one of his errors. I didn't notice that he linked a web definition from the wrong dictionary: The Oxford Dictionary of Phrase and Fable. The webpage for the correct dictionary doesn't make the definition publicly available. The print version of New Oxford American Dictionary, like the Mac version and my quote, has lowercase "the". Thank you for the correction to my alert reader who says he wishes to remain anonymous out of fear of being targeted by Hackethal.

A critic of Karl Popper on Reddit wrote:

Here's an exercise for you: to the best of your ability, restate and respond to Putnam's criticism of Popper in 'The 'Corroboration' of Theories' or Lakatos's criticism of Popper in 'Falsification and the Methodology of Scientific Research Programmes'. Bonus points for also stating how they differ from each other. Shouldn't be too difficult since you're aware of these common criticisms of Popper!

So let's take a look at that Putnam text. It opens:

... Sir Karl's fundamental attitudes: 'There is no method peculiar to philosophy'. 'The growth of knowledge can be studied best by studying the growth of scientific knowledge.'

Philosophers should not be specialists. For myself, I am interested in science and in philosophy only because I want to learn something about the riddle of the world in which we live, and the riddle of man's knowledge of that world. And I believe that only a revival of interest in these riddles can save the sciences and philosophy from an obscurantist faith in the expert's special skill and in his personal knowledge and authority.

I checked the three unsourced quotes. They're from Popper's The Logic of Scientific Discovery, in the 1959 preface to the first English edition. The third quote has significant edits with no indication that Putnam made changes. Let's go through them in order.

The first quote omits Popper's italics.

The second quote omits italics, begins mid-sentence, and changes the first word to begin with a capital letter.

The third quote, the block quote, also starts mid-sentence and edits the first word to be capitalized.

So far these issues are somewhat minor. But it gets worse. The third and final sentence of Putnam's block quote is:

And I believe that only a revival of interest in these riddles can save the sciences and philosophy from an obscurantist faith in the expert's special skill and in his personal knowledge and authority.

Here's the sentence Popper actually wrote:

And I believe that only a revival of interest in these riddles can save the sciences and philosophy from narrow specialization and from an obscurantist faith in the expert's special skill, and in his personal knowledge and authority; a faith that so well fits our 'post-rationalist' and 'post-critical' age, proudly dedicated to the destruction of the tradition of rational philosophy, and of rational thought itself.

Putnam deleted the words "from narrow specialization and".

Putnam deleted the comma after "special skill".

Putnam replaced the semi-colon and everything after it with a period.

Putnam didn't communicate that he made edits.

Deleting words from the middle of a quote without using an ellipsis or square brackets is unacceptable.

Putnam's text is a reprint of a text that Popper already responded to, plus a brief response to Popper's response. In Putnam's followup he has a section "The Charge of Textual Misrepresentation" which responds to Popper's claim that Putnam had misrepresented Popper. It doesn't address these quotation issues, which seems careless to me: if you're going to address that kind of charge, you ought to double check the accuracy of your quotes!

This does not inspire confidence that Putnam's text is worth reading or contains good criticism of Critical Rationalism (or that the Lakatos text, recommended by the same Redditor, is good).

Lakatos, it should be noted, is a fan of Popper. Here's how Lakatos begins his contribution to The Philosophy of Karl Popper, edited by Paul A. Schilpp, volume 1, page 241:

Popper’s ideas represent the most important development in the philosophy of the twentieth century; an achievement in the tradition— and on the level—of Hume, Kant, or Whewell. Personally, my debt to him is immeasurable: more than anyone else, he changed my life.

So it doesn't make sense to cite Lakatos when you want to dismiss Popper as a bad thinker who should be ignored and whose fans shouldn't be debated.

And Putnam is a partial fan of Popper, not someone who thinks Popper should be dismissed or ignored. Here's how Putnam begins the text the Redditor recommended:

Sir Karl Popper is a philosopher whose work has influenced and stimulated that of virtually every student in the philosophy of science. In part this influence is explainable on the basis of the healthy-mindedness of some of Sir Karl’s fundamental attitudes

And Putnam wrote a text trying to engage with Popper, plus a followup. Putnam did not behave like some Redditors (who ironically cite Putnam) who think Popper should be treated as an unreasonable outcast and his claims and fans ignored without debate.