I wrote this in Feb 2009 and emailed it to Yudkowsky. He didn't reply.

---

Dear Eliezer Yudkowsky,

I am writing to criticize some of your statements regarding Karl Popper. I hope this will be of interest.

http://yudkowsky.net/rational/bayes

Previously, the most popular philosophy of science was probably Karl Popper's falsificationism - this is the old philosophy that the Bayesian revolution is currently dethroning. Karl Popper's idea that theories can be definitely falsified, but never definitely confirmed, is yet another special case of the Bayesian rules

That isn't Popper's idea because he doesn't believe in definite falsifications. Falsifications are themselves tentative conjectures which must be held open to criticism and reconsidering.

Popper also doesn't assert that confirmations are never definite, rather he denies there is confirmation at all. The reason is that any given confirming evidence for theory T is logically consistent with T being false.

More generally, Popper's philosophy is not about what we can do definitely. He does not address himself to the traditional philosophical problem of what we can and can't be certain of, or what is and isn't a justified, true belief. While he did comment on those issues, his epistemic philosophy is not an alternative answer to those questions. Rather, his positive contributions focus on a more fruitful issue: conjectural knowledge. How do people acquire conjectural knowledge? What is its nature? And so on.

BTW, conjectural knowledge does not mean the probabilistic knowledge that Bayesians are fond of. Probabilistic knowledge is just as much of an anathema to Popper as certain knowledge, because the same criticisms (for example that attempting justification leads to regress or circularity) apply equally well to each.

Your claim at the end of the quote that Popperian epistemology is a special case of Bayesian epistemology is especially striking. Popper considered the Bayesian approach and told us where he stands on it. On page 141 of Objective Knowledge he states, "I have combated [Bayesian epistemology] for thirty-three years."

To say that something which Popper combatted for over three decades is a more general version of his own work is an extraordinary claim. It should be accompanied with extraordinary substantiation, and some account of where Popper's arguments on the subject go wrong, but it is not.

Popper was a hardworking, academic person who read and thought about philosophy extensively, including ideas he disagreed with. He would often try to present the best possible version of an idea, as well as a history of the problem in question, before offering his criticism of it. I would ask that a similar approach be taken in criticizing Popper. Both as a matter of respect, and because it improves discussion.

In my understanding, there’s no one who speaks for Less Wrong (LW), as its representative, and is responsible for addressing questions and criticisms. LW, as a school of thought, has no agents, no representatives – or at least none who are open to discussion.

The people I’ve found interested in discussion on the website and slack have diverse views which disagree with LW on various points. None claim LW is true. They all admit it has some weaknesses, some unanswered criticisms. They have their own personal views which aren’t written down, and which they don’t claim to be correct anyway.

This is problematic. Suppose I wrote some criticisms of the sequences, or some Bayesian book. Who will answer me? Who will fix the mistakes I point out, or canonically address my criticisms with counter-arguments? No one. This makes it hard to learn LW’s ideas in addition to making it hard to improve them.

My school of thought (Fallible Ideas – FI) has representatives and claims to be correct as far as is known (like LW, it’s fallibilist, so of course we may discover flaws and improve it in the future). It claims to be the best current knowledge, which is currently non-refuted, and has refutations of its rivals. There are other schools of thought which say the same thing – they actually think they’re right and have people who will address challenges. But LW just has individuals who individually chat about whatever interests them without there being any organized school of thought to engage with. No one is responsible for defining an LW school of thought and dealing with intellectual challenges.

So how is progress to be made? Suppose LW, vaguely defined as it may be, is mistaken on some major points. E.g. Karl Popper refuted induction. How will LW find out about its mistake and change? FI has a forum where its representatives take responsibility for seeing challenges addressed, and have done so continuously for over 20 years (as some representatives stopped being available, others stepped up).

Which challenges are addressed? All of them. You can’t just ignore a challenge because it could be correct. If you misjudge something and then ignore it, you will stay wrong. Silence doesn’t facilitate error correction. For information on this methodology, which I call Paths Forward. BTW if you want to take this challenge seriously, you’ll need to click the link; I don’t repeat all of it. In general, having much knowledge is incompatible with saying all of it (even on one topic) upfront in forum posts without using references.

My criticism of LW as a whole is that it lacks Paths Forward (and lacks some alternative of its own to fulfill the same purpose). In that context, my criticisms regarding specific points don’t really matter (or aren’t yet ready to be discussed) because there’s no mechanism for them to be rationally resolved.

One thing FI has done, which is part of Paths Forward, is it has surveyed and addressed other schools of thought. LW hasn’t done this comparably – LW has no answer to Critical Rationalism (CR). People who chat at LW have individually made some non-canonical arguments on the matter that LW doesn’t take responsibility for (and which often involve conceding LW is wrong on some points). And they have told me that CR has critics – true. But which criticism(s) of CR does LW claim are correct and take responsibility for the correctness of? (Taking responsibility for something involves doing some major rethinking if it’s refuted – addressing criticism of it and fixing your beliefs if you can’t. Which criticisms of CR would LW be shocked to discover are mistaken, and then be eager to reevaluate the whole matter?) There is no answer to this, and there’s no way for it to be answered because LW has no representatives who can speak for it and who are participating in discussion and who consider it their responsibility to see that issues like this are addressed. CR is well known, relevant, and makes some clear LW-contradicting claims like that induction doesn’t work, so if LW had representatives surveying and responding to rival ideas, they would have addressed CR.

BTW I’m not asking for all this stuff to be perfectly organized. I’m just asking for it to exist at all so that progress can be made.

Anecdotally, I’ve found substantial opposition to discussing/considering methodology from LW people so far. I think that’s a mistake because we use methods when discussing or doing other activities. I’ve also found substantial resistance to the use of references (including to my own material) – but why should I rewrite a new version of something that’s already written? Text is text and should be treated the same whether it was written in the past or today, and whether it was written by someone else or by me (either way, I’m taking responsibility. I think that’s something people don’t understand and they’re used to people throwing references around both vaguely and irresponsibly – but they haven’t pointed out any instance where I made that mistake). Ideas should be judged by the idea, not by attributes of the source (reference or non-reference).

The Paths Forward methodology is also what I think individuals should personally do – it works the same for a school of thought or an individual. Figure out what you think is true and take responsibility for it. For parts that are already written down, endorse that and take responsibility for it. If you use something to speak for you, then if it’s mistaken you are mistaken – you need to treat that the same as your own writing being refuted. For stuff that isn’t written down adequately by anyone (in your opinion), it’s your responsibility to write it (either from scratch or using existing material plus your commentary/improvements). This writing needs to be put in public and exposed to criticism, and the criticism needs to actually get addressed (not silently ignored) so there are good Paths Forward. I hoped to find a person using this method, or interested in it, at LW; so far I haven’t. Nor have I found someone who suggested a superior method (or even any alternative method to address the same issues) or pointed out a reason Paths Forward doesn’t work.

Some people I talked with at LW seem to still be developing as intellectuals. For lots of issues, they just haven’t thought about it yet. That’s totally understandable. However I was hoping to find some developed thought which could point out any mistakes in FI or change its mind. I’m seeking primarily peer discussion. (If anyone wants to learn from me, btw, they are welcome to come to my forum. It can also be used to criticize FI.) Some people also indicated they thought it’d be too much effort to learn about and address rival ideas like CR. But if no one has done that (so there’s no answer to CR they can endorse), then how do they know CR is mistaken? If CR is correct, it’s worth the effort to study! If CR is incorrect, someone better write that down in public (so CR people can learn about their errors and reform; and so perhaps they could improve CR to no longer be mistaken or point out errors in the criticism of CR.)

One of the issues related to this dispute is I believe we can always proceed with non-refuted ideas (there is a long answer for how this works, but I don’t know how to give a short answer that I expect LW people to understand – especially in the context of the currently-unresolved methodology dispute about Paths Forward). In contrast, LW people typically seem to accept mistakes as just something to put up with, rather than something to try to always fix. So I disagree with ignoring some known mistakes, whereas LW people seem to take it for granted that they’re mistaken in known ways. Part of the point of Paths Forward is not to be mistaken in known ways.

Paths Forward is a methodology for organizing schools of thought, ideas, discussion, etc, to allow for unbounded error correction (as opposed to typical things people do like putting bounds on discussions, with discussion of the bounds themselves being out of bounds). I believe the lack of Paths Forward at LW is preventing the resolution of other issues like about the correctness of induction, the right approach to AGI, and the solution to the fundamental problem of epistemology (how new knowledge can be created).

I emailed this to some MIRI people and others related to Less Wrong.

---

I believe I know some important things you don't, such as that induction is impossible, and that your approach to AGI is incorrect due to epistemological issues which were explained decades ago by Karl Popper. How do you propose to resolve that, if at all?

I think methodology for how to handle disagreements comes prior to the content of the disagreements. I have writing about my proposed methodology, Paths Forward, and about how Less Wrong doesn't work because of the lack of Paths Forward:

http://curi.us/1898-paths-forward-short-summary

http://curi.us/2064-less-wrong-lacks-representatives-and-paths-forward

Can anyone tell me that I'm mistaken about any of this? Do you have a criticism of Paths Forward? Will any of you take responsibility for doing Paths Forward?

Have any of you written a serious answer to Karl Popper (the philosopher who refuted induction – http://fallibleideas.com/books#popper )? That's important to address, not ignore, since if he's correct then lots of your research approaches are mistakes.

In general, if someone knows a mistake you're making, what are the mechanisms for telling you and having someone take responsibility for addressing the matter well and addressing followup points? Or if someone has comments/questions/criticism, what are the mechanisms available for getting those addressed? Preferably this should be done in public with permalinks at a venue which supports nested quoting. And whatever your answer to this, is it written down in public somewhere?

Do you have public writing detailing your ideas which anyone is taking responsibility for the correctness of? People at Less Wrong often say "read the sequences" but none of them take responsibility for addressing issues with the sequences, including answering questions or publishing fixes if there are problems. Nor do they want to address existing writing (e.g. by David Deutsch – http://fallibleideas.com/books#deutsch ) which contains arguments refuting major aspects of the sequences.

Your forum ( https://agentfoundations.org ) says it's topic-limited to AGI math, so it's not appropriate for discussing criticism of the philosophical assumptions behind your approach (which, if correct, imply the AGI math you're doing is a mistake). And it states ( https://agentfoundations.org/how-to-contribute ):

It’s important for us to keep the forum focused, though; there are other good places to talk about subjects that are more indirectly related to MIRI’s research, and the moderators here may close down discussions on subjects that aren’t a good fit for this forum.

But you do not link those other good places. Can you tell me any Paths-Forward-compatible other places to use, particularly ones where discussion could reasonably result in MIRI changing?

If you disagree with Paths Forward, will you say why? And do you have some alternative approach written in public?

Also, more broadly, whether you will address these issues or not, do you know of anyone that will?

If the answers to these matters are basically "no", then if you're mistaken, won't you stay that way, despite some better ideas being known and people being willing to tell you?

The (Popperian) Fallible Ideas philosophy community ( http://fallibleideas.com ) is set up to facilitate Paths Forward (here is our forum which does this http://fallibleideas.com/discussion-info ), and has knowledge of epistemology which implies you're making big mistakes. We address all known criticisms of our positions (which is achievable without using too much resources like time and attention, as Paths Forward explains); do you?

---

Update (Dec 2019):

One person from MIRI responded the day I sent out the letter (Nov 9, 2017). He didn't answer anything I asked, but I decided to add the quotes for better completeness and record keeping. Below are Rob Bensinger's 3 emails, quoted, and my replies. After that he stopped responding.

Hi, Elliot. My short answer is that I think Popper is wrong; inductive reasoning works just as well as deductive in principle, though in practice we often have to rely on heuristic approximations of ideal inductive and deductive reasoning. The traditional problems with in-principle inductive reasoning (e.g., infinite hypothesis spaces) are well-addressed by Solomonoff's theory of algorithmic probability (http://world.std.com/~rjs/tributes/rathmannerhutter.pdf).

Have you written a serious and reasonably complete answer to Popper, or do you know of one that you will endorse, take responsibility for (if it's mistaken, you were mistaken), and address questions/criticisms/etc regarding?

And where is the Path Forward if you're mistaken?

I feel comfortable endorsing Solomonoff induction and Garrabrant induction (https://intelligence.org/2016/09/12/new-paper-logical-induction/) as philosophically unproblematic demonstrations that inductive reasoning works well in principle.

So you're disagreeing with Popper, but without addressing his arguments. If you're mistaken, and your mistakes have already been explained decades ago, you'll stay mistaken. No Paths Forward. Right?

I've read Popper before, and I believe the SEP when it says that he considered infinite hypothesis spaces a major underlying problem for induction (if not the core problem):

Popper gave two formulations of the problem of induction; the first is the establishment of the truth of a theory by empirical evidence; the second, slightly weaker, is the justification of a preference for one theory over another as better supported by empirical evidence. Both of these he declared insoluble, on the grounds, roughly put, that scientific theories have infinite scope and no finite evidence can ever adjudicate among them (LSD, 253–254; Grattan-Guiness 2004).

My claim is that Solomonoff induction addresses this problem handily, and that it more generally provides a good formal framework for understanding how and why inductive reasoning works well in practice.

I think we're getting off-topic. Do you agree or disagree with Paths Forward? Why? Do you have alternative written procedures for having issues like this addressed? Do you have e.g. a forum which would be a good place for me to reply to what you've said?

If I point out that you're mistaken about Popper's arguments and how they may be addressed, what happens next? BTW this would be much easier if there was a direct written answer to Popper by you, or by anyone else, that you were willing to take responsibility for. Why isn't there? That would also save effort for both of us – because responding to your unwritten views will require back-and-forth emails where I ask questions to find out what they are and get clarifications on what you're actually claiming. If your reasoning is that Popper is mistaken, so you don't want to bother properly answering him ... then your fallible criticism of Popper isn't itself being exposed to error correction very well.

Gyrodiot wrote at the Less Wrong Slack Philosophy chatroom:

I was waiting for an appropriate moment to discuss epistemology. I think I understood something about curi's reasoning about induction After reading a good chunk of the FI website. Basically, it starts from this:

He quotes from: http://fallibleideas.com/objective-truth

There is an objective truth. It's one truth that's the same for all people. This is the common sense view. It means there is one answer per question.

The definition of truth here is not the same as The Simple Truth as described in LW. Here, the important part is:

Relativism provides an argument that the context is important, but no argument that the truth can change if we keep the context constant.

If you fixate the context around a statement, then the statement ought to have an objective truth value

Yeah. (The Simple Truth essay link.)

In LW terms that's equivalent to "reality has states and you don't change the territory by thinking differently about the map"

Yeah.

From that, FI posits the existence of universal truths that aren't dependent on context, like the laws of physics.

More broadly, many ideas apply to many contexts (even without being universal). This is very important. DD calls this "reach" in BoI (how many contexts does an idea reach to?), I sometimes go with "generality" or "broader applicability".

The ability for the same knowledge to solve multiple problems is crucial to our ability to deal with the world, and for helping with objectivity, and for some other things. It's what enabled humans to even exist – biological evolution created knowledge to solve some problems related to survival and mating, and that knowledge had reach which lets us be intelligent, do philosophy, build skyscrapers, etc. Even animals like cats couldn't exist, like they do today, without reach – they have things like behavioral algorithms which work well in more than one situation, rather than having to specify different behavior for every single situation.

The problem with induction, with this view is that you're taking truths about some contexts to apply them to other contexts and derive truths about them, which is complete nonsense when you put it like that

Some truths do apply to multiple contexts. But some don't. You shouldn't just assume they do – you need to critically consider the matter (which isn't induction).

From a Bayesian perspective you're just computing probabilities, updating your map, you're not trying to attain perfect truth

Infinitely many patterns both do and don't apply to other contexts (such as patterns that worked in some past time range applying tomorrow). So you can't just generalize patterns to the future (or to other contexts more generally) and expect that to work, ala induction. You have to think about which patterns to pay attention to and care about, and which of those patterns will hold in what ranges of contexts, and why, and use critical arguments to improve your understanding of all this.

We do [live in our own map], which is why this mode of thought with absolute truth isn't practical at all

Can you give an example of some practical situation you don't understand how to address with FI thinking, and I'll tell you how or concede? And after we go through a few examples, perhaps you'll better understand how it works and agree with me.

So, if induction is out of the way, the other means to know truth may be by deduction, building on truth we know to create more. Except that leads to infinite regress, because you need a foundation

CR's view is induction is not replaced with more deduction. It's replaced with evolution – guesses and criticism.

So the best we can do is generate new ideas, and put them through empirical test, removing what is false as it gets contradicted

And we can use non-empirical criticism.

But contradicted by what? Universal truths! The thing is, universal truths are used as a tool to test what is true or false in any context since they don't depend on context

Not just contradicted by universal truths, but contradicted by any of our knowledge (lots of which has some significant but non-universal reach). If an idea contradicts some of our knowledge, it should say why that knowledge is mistaken – there's a challenge there. See also my "library of criticism" concept in Yes or No Philosophy (discussed below) which, in short, says that we build up a set of known criticisms that have some multi-context applicability, and then whenever we try to invent a new idea we should check it against this existing library of known criticisms. It needs to either not be contradicted by any of the criticisms or include a counter-argument.

But they are so general that you can't generate new idea from them easily

The LW view would completely disagree with that: laws of physics are statements like every other, they are solid because they map to observation and have predictive power

CR says to judge ideas by criticism. Failure to map to observation and lack of predictive power are types of criticism (absolutely not the only ones), which apply in some important range of contexts (not all contexts – some ideas are non-empirical).

Prediction is great and valuable but, despite being great, it's also overrated. See chapter 1 of The Fabric of Reality by David Deutsch and the discussion of the predictive oracle and instrumentalism.

http://www.daviddeutsch.org.uk/books/the-fabric-of-reality/excerpt/

Also you can use them to explain stuff (reductionism) and generate new ideas (bottom-up scientific research)

From FI:

When we consider a new idea, the main question should be: "Do you (or anyone else) see anything wrong with it? And do you (or anyone else) have a better idea?" If the answers are 'no'and 'no' then we can accept it as our best idea for now.

The problem is that by having a "pool of statements from which falsehoods are gradually removed" you also build a best candidate for truth. Which is not, at all, how the Bayesian view works.

FI suggests evolution is a reliable way to suggest new ideas. It ties well into the framework of "generate by increments and select by truth-value"

It also highlights how humans are universal knowledge machines, that anything (in particular, an AGI) created by a human would have knowledge than humans can attain too

Humans as universal knowledge creators is an idea of my colleague David Deutsch which is discussed in his book, The Beginning of Infinity (BoI).

http://beginningofinfinity.com

But that's not an operational definition : if an AGI creates knowledge much faster than any human, they won't ever catch up and the point is moot

Yes, AGI could be faster. But, given the universality argument, AGI's won't be more rational and won't be capable of modes of reasoning that humans can't do.

The value of faster is questionable. I think no humans currently maximally use their computational power. So adding more wouldn't necessarily help if people don't want to use it. And an AGI would be capable of all the same human flaws like irrationalities, anti-rational memes (see BoI), dumb emotions, being bored, being lazy, etc.

I think the primary cause of these flaws, in short, is authoritarian educational methods which try to teach the kid existing knowledge rather than facilitate error correction. I don't think an AGI would automatically be anything like a rational adult. It'd have to think about things and engage with existing knowledge traditions, and perhaps even educators. Thinking faster (but not better) won't save it from picking up lots of bad ideas just like new humans do.

That sums up the basics, I think The Paths Forwards thing is another matter... and it is very, very demanding

Yes, but I think it's basically what effective truth-seeking requires. I think most truth-seeking people do is not very effective, and the flaws can actually be pointed out as not meeting Paths Forward (PF) standards.

There's an objective truth about what it takes to make progress. And separate truths depending on how effectively you want to make progress. FI and PF talk about what it takes to make a lot of progress and be highly effective. You can fudge a lot of things and still, maybe, make some progress instead of going backwards.

If you just wanna make a few tiny contributions which are 80% likely to be false, maybe you don't need Paths Forward. And some progress gets made that way – a bunch of mediocre people do a bunch of small things, and the bulk of it is wrong, but they have some ability to detect errors so they end up figuring out which are the good ideas with enough accuracy to slowly inch forwards. But, meanwhile, I think a ton of progress comes from a few great (wo)men who have higher standards and better methods. (For more arguments about the importance of a few great men, I particularly recommend Objectivism. E.g. Roark discusses this in his courtroom speech at the end of The Fountainhead.)

Also, FYI, Paths Forward allows you to say you're not interested in something. It's just, if you don't put the work into knowing something, don't claim that you did. Also you should keep your interests themselves open to criticism and error correction. Don't be an AGI researcher who is "not interested in philosophy" and won't listen to arguments about why philosophy is relevant to your work. More generally, it's OK to cut off a discussion with a meta comment (e.g. "not interested" or "that is off topic" or "I think it'd be a better use of my time to do this other thing...") as long as the meta level is itself open to error correction and has Paths Forward.

Oh also, btw, the demandingness of Paths Forward lowers the resource requirements for doing it, in a way. If you're interested in what someone is saying, you can be lenient and put in a lot of effort. But if you think it's bad, then you can be more demanding – so things only continue if they meet the high standards of PF. This is win/win for you. Either you get rid of the idiots with minimal effort, or else they actually start meeting high standards of discussion (so they aren't idiots, and they're worth discussing with). And note that, crucially, things still turn out OK even if you misjudge who is an idiot or who is badly mistaken – b/c if you misjudge them all you do is invest less resources initially but you don't block finding out what they know. You still offer a Path Forward (specifically that they meet some high discussion standards) and if they're actually good and have a good point, then they can go ahead and say it with a permalink, in public, with all quotes being sourced and accurate, etc. (I particularly like asking for simple things which are easy to judge objectively like those, but there are other harder things you can reasonably ask for, which I think you picked up on in some ways your judgement of PF as demanding. Like you can ask people to address a reference that you take responsibility for.)

BTW I find that merely asking people to format email quoting correctly is enough barrier to entry to keep most idiots out of the FI forum. (Forum culture is important too.) I like this type of gating because, contrary to moderators making arbitrary/subjective/debatable judgements about things like discussion quality, it's a very objective issue. Anyone who cares to post can post correctly and say any ideas they want. And it lacks the unpredictability of moderation (it can be hard to guess what moderators won't like). This doesn't filter on ideas, just on being willing to put in a bit of effort for something that is productive and useful anyway – proper use of nested quoting improves discussions and is worth doing and is something all the regulars actively want to do. (And btw if someone really wants to discuss without dealing with formatting they can use e.g. my blog comments which are unmoderated and don't expect email quoting, so there are still other options.)

It is written very clearly, and also wants to make me scream inside

Why does it make you want to scream?

Is it related to moral judgement? I'm an Objectivist in addition to a Critical Rationalist. Ayn Rand wrote in The Virtue of Selfishness, ch8, How Does One Lead a Rational Life in an Irrational Society?, the first paragraph:

There's a lot of reasoning for this which goes beyond the one essay. At present, I'm just raising it as a possible area of disagreement.

There are also reasons about objective truth (which are part of both CR and Objectivism, rather than only Objectivism).

The issue isn't just moral judgement but also what Objectivism calls "sanction": I'm unwilling to say things like "It's ok if you don't do Paths Forward, you're only human, I forgive you." My refusal to actively do anti-judgement stuff, and approve of PF alternatives, is maybe more important than any negative judgements I've made, implied or stated.

It hits all the right notes motivation-wise, and a very high number of Rationality Virtues. Curiosity, check. Relinquishment, check. Lightness, check. Argument, triple-check.

Yudkowsky writes about rational virtues:

Haha, yeah, no wonder a triple check on that one :)

Simplicity, check. Perfectionism, check. Precision, check. Scholarship, check. Evenness, humility, precision, Void... nope nope nope PF is much harsher than needed when presented with negative evidence, treating them as irreparable flaws (that's for evenness)

They are not treated as irreparable – you can try to create a variant idea which has the flaw fixed. Sometimes you will succeed at this pretty easily, sometimes it’s hard but you manage it, and sometimes you decide to give up on fixing an idea and try another approach. You don’t know in advance how fixable ideas are (you can’t predict the future growth of knowledge) – you have to actually try to create a correct variant idea to see how doable that is.

Some mistakes are quite easy and fast to fix – and it’s good to actually fix those, not just assume they don’t matter much. You can’t reliably predict mistake fixability in advance of fixing it. Also the fixed idea is better and this sometimes helps leads to new progress, and you can’t predict in advance how helpful that will be. If you fix a bunch of “small” mistakes, you have a different idea now and a new problem situation. That’s better (to some unknown degree) for building on, and there’s basically no reason not to do this. The benefit of fixing mistakes in general, while unpredictable, seems to be roughly proportional to the effort (if it’s hard to fix, then it’s more important, so fixing it has more value). Typically, the small mistakes are a small effort to fix, so they’re still cost-effective to fix.

That fixing mistakes creates a better situation fits with Yudkowsky’s virtue of perfectionism.

(If you think you know how to fix a mistake but it’d be too resource expensive and unimportant, what you can do instead is change the problem. Say “You know what, we don’t need to solve that with infinite precision. Let’s just define the problem we’re solving as being to get this right within +/- 10%. Then the idea we already have is a correct solution with no additional effort. And solving this easier problem is good enough for our goal. If no one has any criticism of that, then we’ll proceed with it...")

Sometimes I talk about variant ideas as new ideas (so the original is refuted, but the new one is separate) rather than as modifying and rescuing a previous idea. This is a terminology and perspective issue – “modifying" and “creating" are actually basically the same thing with different emphasis. Regardless of terminology, substantively, some criticized flaws in ideas are repairable via either modifying or creating to get a variant idea with the same main points but without the flaw.

PF expects to have errors all other the place and act to correct them, but places a burden on everyone else that doesn't (that's for humility)

Is saying people should be rational burdensome and unhumble?

According to Yudkowsky's essay on rational virtues, the point of humility is to take concrete steps to deal with your own fallibility. That is the main point of PF!

PF shifts from True to False by sorting everything through contexts in a discrete way.

The binary (true or false) viewpoint is my main modification to Popper and Deutsch. They both have elements of it mixed in, but I make it comprehensive and emphasized. I consider this modification to improve Critical Rationalism (CR) according to CR's own framework. It's a reform within the tradition rather than a rival view. I think it fits the goals and intentions of CR, while fixing some problems.

I made educational material (6 hours of video, 75 pages of writing) explaining this stuff which I sell for $400. Info here:

https://yesornophilosophy.com

I also have many relevant, free blog posts gathered at:

http://curi.us/1595-rationally-resolving-conflicts-of-ideas

Gyrodiot, since I appreciated the thought you put into FI and PF, I'll make you an offer to facilitate further discussion:

If you'd like to come discuss Yes or No Philosophy at the FI forum, and you want to understand more about my thinking, I will give you a 90% discount code for Yes or No Philosophy. Email curi@curi.us if interested.

Incertitude is lack of knowledge, which is problematic (that's for precision)

The clarity/precision/certitude you need is dependent on the problem (or the context if you don’t bundle all of the context into the problem). What is your goal and what are the appropriate standards for achieving that goal? Good enough may be good enough, depending on what you’re doing.

Extra precision (or something else) is generally bad b/c it takes extra work for no benefit.

Frequently, things like lack of clarity are bad and ruin problem solving (cuz e.g. it’s ambiguous whether the solution means to take action X or action Y). But some limited lack of clarity, lower precision, hesitation, whatever, can be fine if it’s restricted to some bounded areas that don’t need to be better for solving this particular problem.

Also, about the precision virtue, Yudkowsky writes,

FI/PF has no issue with this. You can specify required precision (e.g. within plus or minus ten) in the problem. Or you can find you have multiple correct solutions, and then consider some more ambitious problems to help you differentiate between them. (See the decision chart stuff in Yes or No Philosophy.)

PF posits time and again that "if you're not achieving your goals, well first that's because you're not faillibilist". Which is... quite too meta-level a claim (that's for the Void)

Please don't put non-quotes in quote marks. The word "goal" isn't even in the main PF essay.

I'll offer you a kinda similar but different claim: there's no need to be stuck and not make progress in life. That's unnecessary, tragic, and avoidable. Knowing about fallibilism, PF, and some other already-known things is adequate that you don't have to be stuck. That doesn't mean you will achieve any particular goal in any particular timeframe. But what you can do is have a good life: keep learning things, making progress, achieving some goals, acting on non-refuted ideas. And there's no need to suffer.

For more on these topics, see the FI discussion of coercion and the BoI view on unbounded progress:

http://beginningofinfinity.com

(David Deutsch, author of BoI, is a Popperian and is a founder of Taking Children Seriously (TCS), a parenting/education philosophy created by applying Critical Rationalism and which is where the the ideas about coercion come from. I developed the specific method of creating a succession of meta problems to help formalize and clarify some TCS ideas.)

I don't see how PF violates the void virtue (aspects of which, btw, relate to Popper's comments on Who Should Rule? cuz part of what Yudkowsky is saying in that section is don't enshrine some criteria of rationality to rule. My perspective is, instead of enshrining a ruler or ruling idea, the most primary thing is error correction itself. Yudkowsky says something that sorta sounds like you need to care about the truth instead of your current conception of the truth – which happily does help keep it possible to correct errors in your current conception.)

(this last line is awkward. The rationalist view may consider that rationalists should win, but not winning isn't necessarily a failure of rationality)

That depends on what you mean by winning. I'm guessing I agree with it the way you mean it. I agree that all kinds of bad things can happen to you, and stuff can go wrong in your life, without it necessarily being your fault.

(this needs unpacking the definition of winning and I'm digging myself deeper I should stop)

Why should you stop?

---

Justin Mallone replied to Gyrodiot:

hey gyrodiot feel free to join Fallible Ideas list and post your thoughts on PF. also, could i have your permission to share your thoughts with Elliot? (I can delete what other ppl said). note that I imagine elliot would want to reply publicly so keep that in mind.

Gyrodiot replied:

@JUSTINCEO You can share my words (only mine) if you want, with this addition: I'm positive I didn't do justice to FI (particularly in the last part, which isn't clear at all). I'll be happy to read Elliot's comments on this and update in consequence, but I'm not sure I will take time to answer further.

I find we are motivated by the same "burning desire to know" (sounds very corny) and disagree strongly about method. I find, personally, the LW "school" more practically useful, strikes a good balance for me between rigor, ease of use, and ability to coordinate around.

Gyrodiot, I hope you'll reconsider and reply in blog comments, on FI, or on Less Wrong's forum. Also note: if Paths Forward is correct, then the LW way does not work well. Isn't that risk of error worth some serious attention? Plus isn't it fun to take some time to seriously understand a rival philosophy which you see some rational merit in, and see what you can learn from it (even if you end up disagreeing, you could still take away some parts)?

---

For those interested, here are more sources on the rationality virtues. I think they're interesting and mostly good:

https://wiki.lesswrong.com/wiki/Virtues_of_rationality

https://alexvermeer.com/the-twelve-virtues-of-rationality/

http://madmikesamerica.com/2011/05/the-twelve-virtues-of-rationality/

That last one says, of Evenness:

Maybe. Depends on how "cautious" would be clarified with more precision. This could be interpreted to mean something I agree with, but also there are a lot of ways to interpret it that I disagree with.

I also think Occam's Razor (mentioned in that last link, not explicitly in the Yudkowsky essay), while having some significant correctness to it, is overrated and is open to specifications of details that I disagree with.

And I disagree with the "burden of proof" idea (I cover this in Yes or No Philosophy) which Yudkowsky mentions in Evenness.

The biggest disagreement is empiricism. (See the criticism of that in BoI, and FoR ch1. You may have picked up on this disagreement already from the CR stuff.)

View this post, with discussion, on Less Wrong.

---

Consider an idea consisting of a group of strongly connected sub-ideas. If any sub-idea is an error (doesn’t work), then the whole idea is an error (doesn’t work). We can metaphorically model this as a metal chain made of links. How strong is a chain? How hard can you pull on it before it breaks? It’s as strong as its weakest link. If you measure the strength of every link in the chain, and try to combine them into an overall strength score for the chain, you will get a bad answer. The appropriate weight to give the non-weakest links, in your analysis of chain strength, is ~zero.

There are special cases. Maybe the links are all equally strong to high precision. But that’s unusual. Variance (statistical fluctuations) is usual. Perhaps there is a bell curve of link strengths. Having two links approximately tied for weakest is more realistic, though still uncommon.

(A group of linked ideas may not be a chain (linear) because of branching (tree structure). But that doesn’t matter to my point. Stress the non-linear system of chain links and something will break first.)

The weakest link of the chain is the bottleneck or constraint. The other links have excess capacity – more strength than they need to stay unbroken when the chain gets pulled on hard enough to break the weakest link.

Optimization of non-bottlenecks is ~wasted effort. In other words, if you pick random chain links, and then you reinforce them, it (probably) doesn’t make the chain stronger. Reinforcing non-weakest links is misallocating effort.

So how good is an idea made of sub-ideas? It’s as strong as its weakest link (sub-idea). Most ideas have excess capacity. So it’d be a mistake to measure how good each sub-idea is, including more points for excess capacity, and then combine all the scores into an overall goodness score.

Excess capacity is a general feature and requirement of stable systems. Either most components have excess capacity or the system is unstable. Why? Because of variance. If lots of components were within the margin of error (max expected or common variance) of breaking, stuff would break all over the place on a regular basis. You’d have chaos. Stable systems mostly include parts which remain stable despite variance. That means that in most circumstances, when they aren’t currently dealing with high levels of negative variances, then they have excess capacity.

This is why manufacturing plants should not be designed as a balanced series of workstations, all with equal production capacity. A balanced plant (code) lacks excess capacity on any workstations (chain links), which makes it unstable to variance.

Abstractly, bottlenecks and excess capacity are key issues whenever there are dependency links plus variance. (Source.)

Applied to Software

This is similar to how, when optimizing computer programs for speed, you should look for bottlenecks and focus on improving those. Find the really slow part and work on that. Don’t just speed up any random piece of code. Most of the code is plenty fast. Which means, if you want to assign an overall optimization score to the code, it’d be misleading to look at how well optimized every function is and then average them. What you should actually do is a lot more like scoring the bottleneck(s) and ignoring how optimized the other functions are.

Just as optimizing the non-bottlenecks with lots of excess capacity would be wasted effort, any optimization already present at a non-bottleneck shouldn’t be counted when evaluating how optimized the codebase is, because it doesn’t matter. (To a reasonable approximation. Yes, as the code changes, the bottlenecks could move. A function could suddenly be called a million times more often than before and need optimizing. If it was pre-optimized, that’d be a benefit. But most functions will never become bottlenecks, so pre-optimizing just in case has a low value.)

Suppose a piece of software consists of one function which calls many sub-functions which call sub-sub-functions. How many speed bottlenecks does it have? Approximately one, just like a chain has one weakest link. In this case we’re adding up time taken by different components. The vast majority of sub-functions will be too fast to matter much. One or a small number of sub-functions use most of the time. So it’s a small number of bottlenecks but not necessarily one. (Note: there are never zero bottlenecks: no matter how much you speed stuff up, there will be a slowest sub-function. However, once the overall speed is fast enough, you can stop optimizing.) Software systems don’t necessarily have to be this way, but they usually are, and more balanced systems don’t work well.

Applied to Ideas

I propose viewing ideas from the perspective of chains with weakest links or bottlenecks. Focus on a few key issues. Don’t try to optimize the rest. Don’t update your beliefs using evidence, increasing your confidence in some ideas, when the evidence deals with non-bottlenecks. In other words, don’t add more plausibility to an idea when you improve a sub-component that already had excess capacity. Don’t evaluate the quality of all the components of an idea and combine them into a weighted average which comes out higher when there’s more excess capacity for non-bottlenecks.

BTW, what is excess capacity for an idea? Ideas have purposes. They’re meant to accomplish some goal such as solving a problem. Excess capacity means the idea is more than adequate to accomplish its purpose. The idea is more powerful than necessary to do its job. This lets it deal with variance, and may help with using the idea for other jobs.

Besides the relevance to adding up the weight of the evidence or arguments, this perspective explains why thinking is tractable in general: we’re able to focus our attention on a few key issues instead of being overwhelmed by the ~infinite complexity of reality (because most sub-issues we deal with have excess capacity, so they require little attention or optimization).

Note: In some ways, I have different background knowledge and perspective than the typical poster here (and in some ways I’m similar). I expect large inferential distance. I don’t expect my intended meaning to be transparent to readers here. (More links about this: one, two.) I hope to get feedback about which ideas people here accept, reject or want more elaboration on.

Acknowledgments: The ideas about chains, bottlenecks, etc., were developed by Eliyahu Goldratt, who developed the Theory of Constraints. He was known especially for applying the methods of the hard sciences to the field of business management. Above, I’ve summarized some Goldratt ideas and begun relating them to Bayesian epistemology.

View discussion of this post at Less Wrong.

---

This post follows my Chains, Bottlenecks and Optimization. The goal is to give hypothetical examples of bottlenecks and non-bottlenecks (things with excess capacity), and to answer johnswentworth, who helpfully commented:

I really like what this post is trying to do. The idea is a valuable one. But this explanation could use some work - not just because inferential distances are large, but because the presentation itself is too abstract to clearly communicate the intended point. In particular, I'd strongly recommend walking through at least 2-3 concrete examples of bottlenecks in ideas.

I’ll give a variety of examples starting with simpler ones. If you want a different type, let me know.

Note: The term “bottleneck” has synonyms like “constraint” or “limiting factor”. I’ll often use “key factor”. This contrasts with a non-bottleneck, or secondary factor, which is something with excess capacity (above a margin of error), so improving it isn’t very useful. Doing better at a bottleneck makes a significant difference to doing better at your goal; doing better at a non-bottleneck doesn’t. My basic point is that we should focus our attention on key factors.

Oven

In The Goal by Eli Goldratt, the main example is a factory. One of the bottlenecks is the heat treat oven: the rate of baking parts in the oven was limiting the overall output of the factory.

A non-bottleneck example is quality assurance. It was possible to check parts for defects significantly faster than they came out of the oven. So hiring more QA people wouldn’t result in more finished products.

One of the main points of Goldratt’s book is that trying to have a balanced production line (no excess capacity at any workstation) is a bad idea.

Software

Focusing on key factors or bottlenecks is well known in software: To speed up a program, measure where most of the run time is being spent, then speed up that part(s). Don’t just optimize any function. Most functions have excess capacity (they are more than fast enough already to get a satisfactory result, and their impact is orders of magnitude less than the bottleneck’s impact).

Chair

I weigh 150lbs and buy an office chair that can hold someone up to 300lbs. It has excess capacity. Would a chair that can hold 400lbs be 33% better (regarding this factor)? Nope, that wouldn’t be useful to me. Everything else being equal, a sturdier chair is better, but I should focus my attention elsewhere: other factors are going to matter orders of magnitude more than having more excess capacity on sturdiness.

I have a budget. Price is a key factor. If a buy a cheaper chair, I can buy more Fortnite skins. So when I’m chair shopping, I focus on variations in price, but I don’t pay attention to variations in weight capacity. (Every chair in the store holds my weight plus a margin of error, so it’s a non-issue.)

Another non-bottleneck is smoothness. I want a chair that doesn’t poke me. Every chair in the store is far more than smooth enough. If I measured the bumps, I’d fine one chair has 50 micrometers bumps, another has 100 micrometer bumps, and so on, but it’d take 4000 micrometer bumps to poke me uncomfortably. I shouldn’t assign a higher score to the chair with smaller bumps when both have plenty small enough bumps. And there’s so much excess capacity here that I don’t need to and shouldn’t even do those measurements – that’d be wasteful.

Ideas

These examples involve ideas. E.g. “I’ll buy the Aeron chair” is an idea about how to proceed in a life situation. It has excess capacity on chair smoothness and sturdiness. It unfortunately fails horribly on the price bottleneck.

Factories are designed according to ideas. Someone’s design plan (or someone’s ideas about how to modify the factory) created that bottleneck at the oven.

Computer code corresponds to ideas that programmers have about what steps should be used to accomplish tasks. A programmer’s idea about how to design a program can have a speed bottleneck for one sub-idea and excess speed capacity for many other sub ideas.

“I should go to Stanford” is an idea with excess capacity on distance because it’s more than far enough away from my parents. It also does great on the prestige key factor.

Another type of idea is a skill. E.g. I have ideas about how to play chess. They have excess capacity for the goal of beating a 1000 rated player – they are more than good enough to do the job. For the goal of getting a higher rating, my endgame knowledge is a bottleneck, but my opening knowledge has excess capacity. The positions I get out of the opening are more than good enough to move up in the chess world, but I lose too many drawn endgames.

When constructing a birdhouse, I have excess capacity for reading and understanding a guide, but a bottleneck for patience to go slowly and carefully enough given my poor skill at making wood come out the right shape. The wood has excess capacity for strength, but not for weight because I want to hang the birdhouse from a thin branch.

Evolution

We’re debating selfish gene, group selection or Lamarckism as the primary driver of biological evolution. The key factors involve causal explanations.

Lamarckism lacks specifics about the mechanism for transmitting change to the next generation (it’s also experimentally questionable). Sure you can hypothetically imagine a system which saves information about bodily system usage during a lifetime and then puts information into eggs or sperm. But that system hasn’t been found in reality, studied, observed under a microscope, etc. Genes have been, e.g. we’ve studied the shape, chemical composition and copying mechanisms of DNA.

A key issue with group selection was what happens with traits which help the group but harm the individual. What are the causal mechanisms by which those traits would end up in the next generation at higher rather than lower rates (lower due to the harm to the holders of the trait)? No good answer is known for the general case.

These theories all have excess capacity at being able to tell a high level story to account for the animals we observe. Their ability to do that could survive infinitely many variations of the animals to be explained (e.g. if giraffes were 1.1 inches taller on average, or 1.11, or 1.111…). They could also still tell their stories successfully given an infinity of additional constraints, e.g. that the story doesn’t use the number 888111, or the constraint it doesn’t use 888112, or a constraint on 888113, etc.

It’d be an error to pick some evidence, e.g. observations of spiders, and then try to estimate how well each theory fits the evidence, and assign them differing scores. Each theory, if it was assumed to be right about the key issues, would be able to explain spiders fine. (Our view of how well an idea deals with a non-bottleneck factor is often a proxy for our judgment of a key factor – I don’t like Lamarckism’s explanation of the origin of spiders because I don’t think acquired traits are inherited in genes.)

College Rankings

College rankings are discussed in The Order of Things, an article by Malcom Gladwell about why it’s hard to usefully combine many factors into a single overall ranking score.

Many dimensions, like class size, graduation rate or prestige, come in different units with no conversions (and some dimensions are hard to measure at all). It’s not like converting inches to meters, it’s like trying to convert inches to minutes (or converting both inches and minutes to something else, e.g. grams).

The key factors for colleges vary by person/context. I want a college which is at least 1000 miles away from my parents, but you strongly prefer a local college so you can save money by not moving out. And neither of those factors can be taken into account by one-size-fits-all college rankings published nationally, even if they wanted to include them, because college seekers live in different places.

Joe has excess capacity on graduation rate. He doesn’t mind going to a school where 80% of people graduate over a school where 90% of people graduate. He’s a great student and is confident that he can graduate regardless. His parents have PhDs and he’s had exposure to professors, to what type of skills are needed to graduate, etc., so he’s in a good position to make this judgment.

Steve will be the first person in his family to go to college. He struggled in high school, both with the academics and with communicating in English with his teachers. For Steve, a college with a 99% graduation rate looks way less risky – that’s a key factor.

Key factors are situational. Kate wants a prestige degree, but Sue wants any degree at all just to satisfy her parents. Sue also wants somewhere she can live on campus with her dog.

Kate and Sue have excess capacity in different areas. Kate is so good at basketball that she can get a full scholarship anywhere, so she doesn’t care about tuition price. Sue is way less bothered by dirt and bad smells than most people, so she has excess capacity on attending a dirty, smelly college.

Some factors are about the same for everyone. They all want a college with plenty of air available to breathe. Fortunately, every single college has excess capacity on air. Even if you came and took some air away, or the college had a bad air day (where, due to the motion of gas atoms and statistical fluctuations, there were an unusually low number of air molecules on campus that day), there’d still be plenty of air left. This example is a reminder of the importance of focusing on only a few factors out of infinite factors that could be evaluated.

Physics

In science, we want our empirical theories to match our observations but not match a ton of other, logically possible observations. A law like E=hf (the energy of a photon is Plank’s constant times the photon’s frequency) is valuable in large part because of how much it excludes. It’s pretty specific. We don’t want excess capacity for the set of physical events and states allowed by the law; we prefer a minimal and highly accurate set. So that’s a key factor where we want as much as we can get (more of it translates to more success at our goal).

E=hf has excess capacity on shortness. It could be a longer formula and we’d still accept it.

E=hf has excess capacity on experimental data. We could have less data and still accept it. The data is also much more precise than necessary to accept E=hf. And we have excess documented counter examples to E=hf^7, E=hf^8, E=hf^9, and to infinitely many other rival theories.

E=hf has excess capacity on ease of use. It could be more of a hassle to do the calculation and we’d still accept it.

E=hf has excess capacity of rhetorical value. It could be less persuasive in speeches and we’d still accept it. This would remain true even if it’s rhetorical value was ~zero. We don’t judge science that way (at least that’s the aspiration).

Peter tries to debate me. No, E=Gd, he claims. What’s Gd I ask? God’s decision. But that’s not even a multiplication between G and d! This reminds me that E=hf does great on the “actually math” criterion, which normally isn’t a key factor in my discussions or thinking, but it becomes a key factor when I’m talking with Peter. Related to this, I have a bunch of excess capacity that Peter doesn’t: I could be really tired and distracted but I’d still remember the importance of math in scientific laws.

As long as Peter disagrees re using math, many other issues that I’d normally talk about are irrelevant. I shouldn’t try to debate with Peter how significant figures and error bars for measurements work. That wouldn’t address his no-math perspective; it’d be the wrong focus in the situation. It’d be a mistake for me to say that my approach has a really great, nuanced approach to measurement precision, so Peter should increase his confidence that I’m right. If I said that, he should actually become more doubtful about me because I’d be showing inflexible thinking that’s bad at understanding what’s relevant to other contexts that I’m not used to.

Minimum Wage Debate

We’re debating minimum wage. We agree that low skill workers shouldn’t get screwed over. I say minimum wage laws screw over workers by reducing the supply of jobs. You say minimum wage laws prevent workers from being screwed over by outlawing exploitative jobs.

The key factor for my claim is economics (specifically the logic and math of supply and demand in simple hypothetical scenarios). When I convince you about that, you change your mind. I should focus on optimizing for that issue. During the debate, I have excess capacity on many dimensions, such as theism, astrology or racism. I’m not even close to causing you to think my position is based on God, the stars, or race. I don’t need to worry about that. When I’m considering what argument to use next, I don’t need to avoid arguments associated with Christianity; I can ignore that factor. Similarly, I don’t need to factor in the race of the economists I cite.

There are many factors which could be seen positively in some way. E.g. economics books with more pages and more footnotes are more impressive, in some sense. This is contextual: some people would be more impressed instead by a compact, very clear book.

But we actually have tons of excess capacity on page count and footnotes. You’re tolerant of a wide variety of books. I don’t need to worry about optimizing this factor. I can focus on other factors like choosing the book with the best clarity and relevance (key factors).

If you were picky about dozens of factors, our discussion would fail. Your tolerance lets me focus on optimizing only a few things, which makes productive discussion possible.

So I convince you that I’m right about minimum wage. But next year you come back with a new argument.

Don’t government regulations make it harder to start a business and to hire people? There’s lots of paperwork that discourages entrepreneurship. This artificially reduces the supply of jobs. It prevents the supply and demand of jobs from reaching the proper equilibrium (market clearing price). Therefore, workers are actually being underpaid because they’re competing for too few jobs, which drives wages down.

Now what? You’re right that my simplified market model didn’t fully correspond to reality. The bottleneck is no longer your ignorance of basic economics. You’ve actually read a bunch and now have excess capacity there: you know more than enough for me to bring up some economics concepts without confusing you. Also you’re very patient and highly motivated, so I don’t have to keep things really short. However, you’re sensitive to insults against less fortunate people, so I have to check for something potentially offensive when I do an editing pass. I only want to share arguments with you that have excess capacity for that – they are more than inoffensive enough.

What should I do? I could defend my model and tell you all kinds of merits it has. The model is useful in many ways. There are many different ways to argue for its correctness given its premises. But those aren’t bottlenecks. You aren’t denying that. That won’t change your mind because the problem you brought up focuses on a different issue.

I judge that the bottleneck is your understanding of what effect minimum wage has on a scenario where the supply of jobs is artificially suppressed. Yes that’s a real problem, but does minimum wage help fix it? I need to focus on that. When I’m considering candidate arguments to tell you, I should look at which one will best address that (and then check offensiveness in editing), while not worrying about factors with excess capacity like your patience and motivation. All the arguments I’m considering will work OK given the available patience and motivation (yes, I could make up a tangled argument that takes so much patience to get through that it turns patience into a bottleneck, but it doesn’t require conscious attention for me to avoid that). Improvements to those factors (like requiring one less unit of patience) are orders of magnitude less important than the key factors (like creating one more unit of understanding of the effects of minimum wage on a system with a government-constrained job supply).

Read this post, with replies, on Less Wrong.

---

Replies to comments on my Chains, Bottlenecks and Optimization:

abramdemski and Hypothesis Generation

Following the venerated method of multiple working hypotheses, then, we are well-advised to come up with as many hypotheses as we can to explain the data.

I think come up with as many hypotheses as we can is intended within the context of some background knowledge (some of which you and I don’t share). There are infinitely many hypotheses that we could come up with. We’d die of old age while brainstorming about just one issue that way. We must consider which hypotheses to consider. I think you have background knowledge filtering out most hypotheses.

Rather than consider as many ideas as we can, we have to focus our limited attention. I propose that this is a major epistemological problem meriting attention and discussion, and that thinking about bottlenecks and excess capacity can help with focusing.

If you’ve already thought through this issue, would you please link to or state your preferred focusing criteria or methodology?

I did check your link (in the quote above) to see if it answered my question. Instead I read:

Now we've got it: we see the need to enumerate every hypothesis we can in order to test even one hypothesis properly. […]

It's like... optimizing is always about evaluating more and more alternatives so that you can find better and better things.

Maybe we have a major disagreement here?

abramdemski and Disjunction

The way you are reasoning about systems of interconnected ideas is conjunctive: every individual thing needs to be true. But some things are disjunctive: some one thing needs to be true. […]

A conjunction of a number of statements is -- at most -- as strong as its weakest element, as you suggest. However, a disjunction of a number of statements is -- at worst-- as strong as its strongest element.

Yes, introducing optional parts to a system (they can fail, but it still succeeds overall) adds complexity to the analysis. I think we can, should and generally do limit their use.

(BTW, disjunction is conjunction with some inversions thrown in, not something fundamentally different.)

Consider a case where we need to combine 3 components to reach our goal and they all have to work. That’s:

A & B & C -> G

And we can calculate whether it works with multiplication: ABC.

What if there are two other ways to accomplish the same sub-goal that C accomplishes? Then we have:

A & B & (C | D | E ) -> G

Using a binary pass/fail model, what’s the result for G? It passes if A, B and at least one of {C, D, E} pass.

What about using a probability model? Problematically assuming independent probabilities, then G is:

AB(1 - (1-C)(1-D)(1-E)))

Or more conveniently:

AB!(!C!D!E)

Or a different way to conceptualize it:

AB(C + D(1 - C) + E(1 - C - D(1 - C)))

Or simplified in a different way:

ABC + ABD + ABE - ABCD - ABCE - ABDE + ABCDE

None of this analysis stops e.g. B from being the bottleneck. It does give some indication of greater complexity that comes from using disjunctions.

There are infinitely many hypotheses available to generate about how to accomplish the same sub-goal that C accomplishes. Should we or together all of them and infinitely increase complexity, or should we focus our attention on a few key areas? This gets into the same issue as the previous section about which hypotheses merit attention.

Donald Hobson and Disjunction

Disjunctive arguments are stronger than the strongest link.

On the other hand, [conjunctive] arguments are weaker than the weakest link.

I don’t think this is problematic for my claims regarding looking at bottlenecks and excess capacity to help us focus our attention where it’ll do the most good.

You can imagine a chain with backup links that can only replace a particular link. So e.g. link1 has 3 backups: if it fails, it’ll be instantly replaced with one of its backups, until they run out. Link2 doesn’t have any backups. Link3 has 8 backups. Backups are disjunctions.

Then we can consider the weakest link_and_backups group and focus our attention there. And we’ll often find it isn’t close: we’re very unevenly concerned about the different groups failing. This unevenness is important for designing systems in the first place (don’t try to design a balanced chain; those are bad) and for focusing our attention.

Structures can also be considerably more complicated than this expanded chain model, but I don’t see that that should change my conclusions.

Dagon and Feasibility

I think I've given away over 20 copies of _The Goal_ by Goldratt, and recommended it to coworkers hundreds of times.

…

The limit is on feasibility of mapping to most real-world situations, and complexity of calculation to determine how big a bottleneck in what conditions something is.

Optimizing software by finding bottlenecks is a counter example to this feasibility claim. We do that successfully, routinely.

Since you’re a Goldratt fan too, I’ll quote a little of what he said about whether the world is too complex to deal with using his methods. From The Choice:

"Inherent Simplicity. In a nutshell, it is at the foundation of all modern science as put by Newton: 'Natura valde simplex est et sibi consona.' And, in understandable language, it means, 'nature is exceedingly simple and harmonious with itself.'"

…

"What Newton tells us is that […] the system converges; common causes appear as we dive down. If we dive deep enough we'll find that there are very few elements at the base—the root causes—which through cause-and-effect connections are governing the whole system. The result of systematically applying the question "why" is not enormous complexity, but rather wonderful simplicity. Newton had the intuition and the conviction to make the leap of faith that convergence happens, not just for the section of nature he examined in depth, but for any section of nature. Reality is built in wonderful simplicity."

(I also posted this at Less Wrong.)

This post follows my Chains, Bottlenecks and Optimization (which has the followups Bottleneck Examples and Comment Replies for Chains, Bottlenecks and Optimization). This post expands on how to think about bottlenecks.

---

There are deeper concepts behind bottlenecks (aka constraints, limiting factors, or key factors).

First, different factors contribute different amounts to goal success. Second, there’s major variation in the amounts contributed.

E.g. I’m adding new features to my software. My goal is profit. Some new features will contribute way more to my profit than others There are lots of features my (potential) customers don’t care about. There are a few features that tons of customers would pay a bunch for.

A bottleneck is basically just a new feature that matters several orders of magnitude more than most others. So most features are approximately irrelevant if the bottleneck isn’t improved.

Put another way: improving the bottleneck translates fairly directly to more goal success, while improving non-bottlenecks translates poorly, e.g. only at 1/1000th effectiveness, or sometimes 0. (It’s possible, but I think uncommon, to have many factors that contribute similarly effectively to goal success. Designing stuff that way doesn’t work well. It’s the same issue as balanced production lines being bad, which Eli Goldratt explains in The Goal: A Process of Ongoing Improvement. It’s also similar to the Pareto Principle which says 80% of effects come from 20% of the causes – meaning most factors aren’t very important.)

What about the software not crashing, not corrupting saved data, and not phoning home with location tracking data? People want those things but I could have them and easily still make zero profit.

A good, typical model for viewing goal pursuit is:

1. There are many factors that would help, and just one or a few of them are the most important to focus on. This is because most factors have a significantly smaller impact. This is focusing on the key positives.
2. There are also many dealbreaker factors that cause failure if screwed up. This is avoiding major negatives.

People care about (1) conditional on (2) not being broken. Avoid anything awful, then optimize in the right places.

When buying a cat, I might try to optimize cuteness and cheapness, while also making sure the cat has 4 paws, a tail, no rabies, is tame, and isn’t too old. I want to do well on a couple key factors and also a bunch of easy factors need to be non-broken. It’s generally not that hard to brainstorm dozens of dealbreakers, many of which are quite easy to avoid in your current situation, even to the point of sounding silly to mention it at all.

(Dealbreakers are also contextual. If there were no cats available meeting all my criteria, I might lower my standards.)

The type (2) factors don’t require much attention. If a factor did need attention, it’d switch categories. (2) is just for failure conditions which are pretty easy to handle. This means most of our attention is available to focus on a few key issues.

I think this model is more effective than e.g. something like “consider all the factors; find out it’s way too complicated; try to approximate what you’d do if you had enough attention for all the factors”.

The model I’m proposing can be thought of as a method of organized, effective approximation from a more complex “take everything fully into account” approach. It tells us how to approximate. Thought of another way, I’m saying don’t distribute your significant figures equally.

You might think “Why not just weight all the factors relevant to my goal, then distribute my attention and significant figures according to the weightings?” The difficulty with that is how to weight things. Having a cat that doesn’t attack me and give me rabies is really important. If I’m just weighting factors normally, I’ll give that high weight because I want to reject any cat purchase which fails at that issue.

So if you just start assigning weights straightforwardly, you’ll give the type (2) factors high weights, e.g. 50,000 each, and if they all pass then the type (1) factors will function as tiebreakers worth e.g. 1-50 points each (minor detail: you can scale the weights so they add up to 1, but it’s easier to do that after you have all the factors with weights assigned – I don’t know what fraction of 1 a big factor should be until I know how many big factors there are). But the high value type (1) factors are actually the best place to put a bunch of significant figures. We don’t need a bunch of precision to address our cat having a tail, 4 legs, and no rabies. So attention allocation shouldn’t correspond to weighting.

In general, when we pursue a goal, there are many important but easy factors, and a few important but hard factors. For goals which are achievable but not easy, it has to be this way. If there were dozens of hard factors, that basically means we’re not ready to do it (though with a huge budget and a big team, sometimes it can be done – that lets you have specialists each working on just one hard factor each, plus some additional people figuring out how to coordinate and combine the results). But the standard progression is: if a project has 10 hard factors, that’s too many for me to focus my attention on at once, so I need to work on some easier sub-projects first – e.g. learning about some of those issues in isolation or doing smaller projects that help build up to the bigger one.

Another way to view the difference is that an increase in the key factors increases our success at the goal. E.g. adding the right new feature will increase profit. Or getting a cuter cat will increase enjoyment. Loosely, the more the better (there’s sometimes an upper limit, at which point it stops helping or is even actively harmful, or it keeps helping but now some other factors matter more). But for type (2) factors, the attitude isn’t anything like “more is better”; it’s just “don’t screw it up”.

In this analysis, I’ve basically assumed that type (1) factors and goal success come in matters of degree (can have more or less of them), but type (2) factors have a binary, pass/fail evaluation. The analysis needs extending for how to deal with binary goals, binary type 1 factors, and matters of degree for type (2). Those issues will come up and we need some way to think about them. I’ll leave that extension for a future post.

That extension is part of a broader issue of how binary and degree issues come up in life, how to think about them, how to convert from one to the other (and when that’s possible or not), when one type is preferable to the other, and so on. They’re both important tools to know how to think about and work with.

Factory Example

Now let’s go through an extended example to clarify how some of these issues work.

In my factory, I’m combining foos and bars to make foobars, which I sell. I have more bars than foos. So foos are the bottleneck. Getting even more bars won’t result in producing additional foobars. I already have an excess capacity of bars.

I also have excess capacity for assembly and QA. My current work area and team could produce many more foobars without hiring new people, getting more space, or getting new tools. And they could already check more foobars for defects.

And I have excess capacity in the market: I could sell more foobars if only I could produce them.

I also have excess capacity on foobar quality. I could redesign them to be nicer, but they’re good enough. Customers are satisfied. They do the job.

And I have excess capacity on price. Cheaper would be nicer, sure, but there isn’t much competition and my customers are people with a good reason to get a foobar. They get benefits from the foobar which are well above the price I’m charging.

Excess capacity means non-bottleneck.

Supply of foos is the bottleneck and the other issues are non-bottlenecks.

Using bars is limited by the availability of foos. That’s a traditional, standard bottleneck.

I call niceness a non-bottleneck because, as with foos, there is excess capacity. It won’t make much difference to achieving more of my goal (profit via foobar sales).

Key factor and secondary factor may be better terminology. It has some advantages, mostly because 1) foo supply isn’t blocking niceness from mattering in the way it’s blocking more supply of bars from mattering 2) niceness would help a little (a few orders of magnitude less than getting more foos, but not zero), which contrasts with bars – getting more bars wouldn’t help at all (in current circumstances).

Bottlenecks can be changed. E.g. I find a new supplier who can deliver far more foos than I need. Foos are no longer a bottleneck. Now what’s the bottleneck?What limits my profit? Maybe I’ll start running out of bars now. Maybe I won’t have enough customers and I’ll need better marketing. Maybe I’ll need to hire more workers. Maybe price will become the crucial issue: if I could lower the price, it’d get me a million new customers. Maybe price is key to breaking into the hobbyist market whereas price isn’t so important for the business market I currently serve.

To break into the hobbyist market, I might need to expand production capacity and lower the price and do a new marketing campaign. There could be several key factors. Doing three things at once is realistic (though not ideal), but we can’t split our focus too much. It’d be nice to find a way to improve things more incrementally. Maybe I’ll figure out how to produce foobars more cheaply first while leaving my price the same, and I’ll get some immediate benefit from higher profit margins. Then once I have the price low enough I’ll try to start selling some to hobbyists as a test (sell in some small stores instead of the big chains, or I could try online sales), and only if that works will I try to ramp up production and hobbyist marketing together.

I can also view the new project (selling to hobbyists, via expanding production, producing more cheaply, and a new marketing campaign) as a whole and then look at what the bottleneck(s) and excess capacity are. They might be quite unequal between the different parts of the new project.

(This is just a toy example. I didn’t worry about new distribution for hobbyists nor about designing a different version of the product for them which better meets the needs of a different market, nor did I worry about market segmentation and how to maintain my higher prices for business customers (a separate production version is one way to do that, using different regions is another, e.g. I could do my hobbyist sales in a different country than my existing business sales.))

Category (1) above (key positives) is the bottlenecks, the things that are valuable to pay attention to and optimize. Category (2) above (avoiding major negatives) is the non-bottlenecks, the things with excess capacity, which I can view as either “good enough” or “failure”. Relevant non-bottlenecks are important. I can’t just ignore them. They need to work. They’re in a position to potentially cause failure. But I’m not very worried about getting them to work and I don’t need to optimize them.

(I also posted this on Less Wrong.)

Below I quote from Process of On Going Improvement forum, letter 6. Eli Goldratt shares a letter he received. I added a few notes to help people follow acronyms.

My question is: Does anyone know of any applications of Less Wrong philosophy to a situation like this? How can LW ideas about rationality explain or fix this sort of problem? The scenario is that someone tried to use rational thinking to make business improvements, was highly successful (which was measured and isn't in dispute), but nevertheless has met so much ongoing resistance that he's at the point of giving up.

I am no expert in TOC but I believe my recent experiences have impact as to what you are writing about.

TOC = Theory Of Constraints. Summary.

About 2 years ago I started on my TOC adventure. Read everything I could get a hold off etc. Tried to get the company interested, etc. In fact, I finally got them interested enough that we had multiple locations participate in the satellite sessions and had enough for three facilitators (myself included). However, I could never get the company to spend for training at AGI. So, in the old air cav fashion, I felt it was up to me to make it happen.

Last year we had real problems with cost, service, high inventory, etc. My plant, I am the plant manager, was being analyzed for a possible shutdown or sell off. We were asking for 17 machines at about $300,000 each due to "lack of capacity" and we were being supplemented by outside producers.

Again, I am not a TOC expert. Basically my exposure has been reading and researching and building computer simulations to understand. But I put on TOC classes for all of my associates (200). I spent 8 hours with each of these associates in multiple classes. We talked about the goal, TIOE, we played the dice game (push, KanBan, DBR) with poker chips, paper clips, and different variations of multiple sided dice and talked about its impact, etc.

The Goal (summary) is a book by Eli Goldratt that has sold over 6 million copies.

TIOE = Throughput, Inventory and Operating Expense. These are the measurements Goldratt recommends.

The dice game is explained in The Goal. It's also now taught by e.g. MIT (section 3-2).

DBR = drum buffer rope. It's about coordinating activities around the bottleneck/constraint.

Last summer we started development on DBR and a new distribution strategy based on what I have read and researched on TOC. I used Bill Dettmer‘s book to develop trees and the clouds. I check our plan against some presentations last November in Memphis when we attended the TOC symposium there.

We had many in the company who doubted but we stuck our necks out and started at the beginning of this year. And we knew we would not be perfect.

YTD results:

YTD = Year to date

Achieved Company President‘s Award for Safety (First Plant to do so) and the planning was based on things I had read about TOC and techniques on establishing teamwork.
Service is up from high 80 to low 90 percentile to averaging above 98.5%
Costs are under budget for the first time in some years
Total Inventory has decreased over 30% and is still dropping
No Longer being supplemented by outside companies for our production
No longer need additional machines to supply demand
We do need additional business to fill our machines
Plant is no longer being considered for close, in fact production from other
facilities are being transferred in.

The chief concern when we told the big wigs we were going to this, was that the cost of freight would go up because our transfer batch sizes would get small. I told them correct but we would stop shipping product back and forth between distribution centers and repacking of product would be almost non-existant. YTD: Our total freight dollars spent is 10% less than the previous year but they look at $/ lb of freight which has gone up. I know this is wrong, they state they know it is wrong, but it still gets measured and used for evaluations.

Anyway, as we shipped more often but smaller quantities our distribution centers complained that we were costing them too much. I have tried for 9 months to get them to quantify this to me. "If I increase batch size of the transfer how many people will it reduce or how much overtime will it reduce" or any other real incremental cost will it get rid off? The general response is, it is hard to quantify but we know it is there. Maybe their intuition is correct, but maybe it is not.

So finally, I am at my end. The DCs continue to insist that we are driving their costs up with small transfer batch sizes. They have complained greatly to my boss and my bosses boss. I am growing weary of the continual fight, which has cost me and my family so much time and effort. I have chosen togive up. I have grown tired of the comments, "Well it was said in a meeting that the concept did not deliver what we expected." Then I show them the numbers and ask, "What else was expected." The reply, "That is what I heard at the meeting."

DCs = distribution centers.

Maybe I made a mistake trying to bring TOC to my plant myself. I would have loved to hire a consultant who really knew what they were doing, but any mention of that brought long talks about cost, etc. I hate to give up but my frustration level has impacted my family, which is something I cannot let happen.

In the end, I have decided this week to give them their large transfer batch sizes while I begin to look for somewhere else to go.

I did not mean for this to be a bitch session. But I can not believe the sheer level of frustration on trying to achieve buy in, even when:
1) Prior to going to our concept we had meetings with our leadership where I presented the UDES from the previous year, and all agreed,

UDES = UnDesirable EffectS. He's saying that before starting he discussed what problems the company was facing with leadership and got unanimous agreement.

2) Showed our potential solution, not all agreed but they were willing to try it
3) Now showing the best numbers the plant has ever turned out.
I just cannot understand the skepticism.

What insight can LW bring to this problem of negative response to rational improvement?

This topic is for discussing anything related to Less Wrong.

I have read Harry Potter and the Methods of Rationality and Inadequate Equilibria by Eliezer Yudkowsky. I'm over half way through reading his Rationality: From AI to Zombies. I also plan to read The Codex by Scott Alexander.

I've recently dual posted several articles on Less Wrong and curi blog. In this thread I'll post copies of comments I write on Less Wrong, and you can too.

I posted this on Less Wrong too.

---

Is there any conjunction fallacy research which addresses the alternative hypothesis that the observed results are mainly due to social dynamics?

Most people spend most of their time thinking in terms of gaining or losing social status, not in terms of reason. They care more about their place in social status hierarchies than about logic. They have strategies for dealing with communication that have more to do with getting along with people than with getting questions technically right. They look for the social meaning in communications. E.g. people normally try to give – and expect to receive – useful, relevant, reasonable info that is safe to make socially normal assumptions about.

Suppose you knew Linda in college. A decade later, you run into another college friend, John, who still knows Linda. You ask what she’s up to. John says Linda is a bank teller, doesn’t give additional info, and changes the subject. You take this to mean that there isn’t more positive info. You and John both see activism positively and know that activism was one of the main ways Linda stood out. This conversation suggests to you that she stopped doing activism. Omitting info isn’t neutral in real world conversations. People mentally model the people they speak with and consider why the person said and omitted things.

In Bayesian terms, you got two pieces of info from John’s statement. Roughly: 1) Linda is a bank teller. 2) John thinks that Linda being a bank teller is key info to provide and chose not to provide other info. That second piece of info can affect people’s answers in psychology research.

So, is there any research which rules out social dynamics explanations for conjunction fallacy experimental results?

I also posted this on Less Wrong.

---

This post follows my question Can Social Dynamics Explain Conjunction Fallacy Experimental Results? The results of the question were that no one provided any research contradicting the social dynamics hypothesis.

There is research on social dynamics. Asch’s conformity experiments indicate that wanting to fit in with a group is a very powerful factor that affects how people answer simple, factual questions like “Which of these lines is longer?” People will knowingly give wrong answers for social reasons. (Unknowingly giving wrong answers, e.g. carelessly, is easier.)

Conformity and other social dynamics can explain the conjunction fallacy experimental data. This post will focus on conformity, the dynamic studied in the Asch experiments.

This post assumes you’re already familiar with the basics of both the Asch and Conjunction Fallacy research. You can use the links if you need reminders.

First I’ll talk about whether conformity applies in the Conjunction Fallacy research setting, then I’ll talk about how conformity could cause the observed results.

Conformity in Groups

The Asch Conformity Experiments have people publicly share answers in a group setting. This was designed to elicit conformist behavior. Should we also expect conformist behavior in a different setting like the Conjunction Fallacy experiments setting? I suspect the different setting is a major reason people don’t connect the Asch and Conjunction Fallacy results.

I haven’t seen specific details of the Conjunction Fallacy research settings (in the text I read, details weren’t given) but I think basically people were given questionnaires to fill out, or something close enough to that. The setting is a bit like taking a test at school or submitting homework to a teacher. Roughly: Someone (who is not a trusted friend) will look over and judge your answers in some manner. In some cases, people were interviewed afterwards about their answers and asked to explain themselves.

Is there an incentive to conformity in this kind of situation? Yes. Even if there was no peer-to-peer interaction (not a safe assumption IMO), it’s possible to annoy the authorities. (Even if there were no real danger, how would people know that? They’d still have a reasonable concern.)

What could you do to elicit a negative reaction from the researchers? You could take the meta position that your answers won’t impact your life and choose the first option on every question. Effort expended on figuring out good answers to stuff should relate to its impact on your life, right? This approach would save time but the researchers might throw out your data, refuse to pay you, ask to speak with you, tell your professors about your (alleged) misbehavior (even if you didn’t violate any written rule or explicit request), or similar. You are supposed to abide by unwritten, unstated social rules when answering conjunction fallacy related questions. I think this is plenty to trigger conformity behaviors. It’s (like most of life) a situation where most people will try to get along with others and act in a way that is acceptable to others.

Most people don’t even need conformity behavior triggers. Their conformity is so automatic and habitual that it’s just how they deal with life. They are the blue pill normies, who aren’t very literal minded, and try to interpret everything in terms of its consequences for social status hierarchies. They don’t think like scientists.

What about the red pill autists who can read things literally, isolate scenarios from cultural context, think like a scientist or rationalist, and so on? Most of them try to imitate normies most of the time to avoid trouble. They try to fit in because they’ve been punished repeatedly for nonconformity.

(Note: Most people are some sort of hybrid. There’s a spectrum, not two distinct groups.)

When attending school people learn not to take questions (like those posed by the conjunction fallacy research) hyper literally. That’s punished. Test and homework questions are routinely ambiguous or flawed. What happens if you notice and complain? Generally you confuse and annoy your teacher. You can get away with noticing a few times, but if you complain about many questions on everything you’re just going to be hated and punished. (If people doubt this, we could analyze some public test questions and I can point out ambiguities and flaws.)

If you’re the kind of person who would start doing math when you aren’t in math class, you’ve gotten negative reactions in the past for your nonconformity. Normal people broadly dislike and avoid math. Saying “Hmm, I think we could use math to get a better answer to this.” is a discouraged attitude in our culture.

The Conjunction Fallacy research doesn’t say “We’re trying to test your math skills. Please do your best to use math correctly.” Even if it did, people routinely give misleading information about how literal/logical/mathematical they want things. You can get in trouble for using too much math, too advanced math, too complicated math, etc., even after being asked to use math. You can very easily get in trouble for being too literal after being asked to be literal, precise and rigorous.

So people see the questions and know that they generally aren’t supposed to sweat the details when answering questions, and they know that trying to apply math to stuff is weird, and most of them would need a large incentive to attempt math anyway, and the more rationalist types often don’t want to ruin the psychology study by overthinking it and acting weird.

I conclude that standard social behavior would apply in the Conjunction Fallacy research setting, including conformity behaviors like giving careless, non-mathematical answers, especially when stakes are low.

How Does Conformity Cause Bad Math?

Given that people are doing conformity behavior when answering Conjunction Fallacy research questions, what results should we expect?

People will avoid math, avoid being hyper literal, avoid being pedantic, not look very closely at the question wording, make normal contextual assumptions, and broadly give the same sorta answers they would if their buddy asked them a similar question in a group setting. Most people avoid developing those skills (literalism, math, ambiguity detection, consciously controlling the context that statements are evaluated in, etc.) in the first place, and people with those skills commonly suppress them, at least in social situations if not throughout life.

People will, as usual, broadly avoid the kinds of behaviors that annoy parents, teachers or childhood peers. They won’t try to be exact or worry about details like making probability math add up correctly. They’ll try to guess what people want from them and what other people will like, so they can fit in. They’ll try to take things in a “reasonable” (socially normal) way which uses a bunch of standard background assumptions and cultural defaults. That can mean e.g. viewing “Linda is a bank teller” as information a person chose to tell you, not as more like an out-of-context factoid chosen randomly by a computer, as I proposed previously.

Conformity routinely requires making a bunch of socially normal assumptions about how to read things, how to interpret instructions, how to take questions, etc. This includes test questions and similar, and most people (past early childhood) have past experiences with this. So many people won’t take conjunction fallacy questions literally.

People like the college students used in the research have taken dozens of ambiguous tests and had to figure out how to deal with it. Either they make socially normal assumptions (contrary to literalism and logic) without realizing they’re doing anything, or they noticed a bunch of errors and ambiguities but figured out a way to cope with tests anyway (or a mix like only noticing a few of the problems).

Conclusions

Conformity isn’t a straight error or bias. It’s strategic. It has upsides. There are incentives to do it and continue doing it (as well as major costs to transitioning to a different strategy).

If this analysis is correct, then the takeaway from the Conjunction Fallacy shouldn’t be along the lines of “People are bad at thinking.” It should instead be more like “People operate in an environment with complex and counter-intuitive incentives, including social dynamics.”

Social status hierarchies and the related social behaviors and social rules are one of the most important features of the world we live in. We should be looking to understand them better and apply our social knowledge more widely. It’s causally connected to many things, especially when there are interactions between people like interpretations of communications and requests from others, as is present in Conjunction Fallacy research.

Continuing the theme that the “Conjunction Fallacy” experimental results can be explained by social dynamics, let’s look at another social dynamic: the Law of Least Effort (LoLE).

(Previously: Can Social Dynamics Explain Conjunction Fallacy Experimental Results? and Asch Conformity Could Explain the Conjunction Fallacy.)

The Law of Least Effort says:

the person who appears to put the least amount of effort out, while getting the largest amount of effort returned to him by others, comes across as the most socially powerful.

In other words, it’s higher status to be chased than to chase others. In terms of status, you want others to come to you, rather than going to them. Be less reactive than others.

Visible effort is a dominant issue even when it’s easy to infer effort behind the scenes. Women don’t lose status for having publicly visible hair and makeup which we can infer took two hours to do. You’re not socially permitted to call them out on that pseudo-hidden effort. Similarly, people often want to do learning and practice privately, and then appear good at stuff in front of their friends. Even if you can infer that someone practiced a bunch in private, it’s often socially difficult to point that out. Hidden effort is even more effective when people can’t guess that it happened or when it happened in the past (particularly childhood).

To consider whether LoLE contributes to the Conjunction Fallacy experimental results, we’ll consider three issues:

1. Is LoLE actually part of the social dynamics of our culture?
2. If so, would LoLE be active in most people while in the setting of Conjunction Fallacy research?
3. If so, how would LoLE affect people’s behavior and answers?

Is LoLE Correct Today?

LoLE comes from a community where many thousands of people have put a large effort into testing out and debating ideas. It was developed to explain and understand real world observations (mostly made by men in dating settings across many cultures), and it’s stood up to criticism so far in a competitive environment where many other ideas were proposed and the majority of proposals were rejected.

AFAIK LoLE hasn’t been tested in a controlled, blinded scientific setting. I think academia has ignored it without explanation so far, perhaps because it’s associated with groups/subcultures that are currently being deplatformed and cancelled.

Like many other social dynamics, LoLE is complicated. There are exceptions, e.g. a scientist or CEO may be seen positively for working hard. You’re sometimes socially allowed to put effort into things you’re “passionate” about or otherwise believed to want to work hard on. But the broad presumption in our society is that people dislike most effort and avoid it when they can. Putting in effort generally shows weakness – failure to avoid it.

And like other social dynamics, while the prevalence is high, not everyone prioritizes social status all the time. Also, people often make mistakes and act in low social status ways.

Although social dynamics are subtle and nuanced, they aren’t arbitrary or random. It’s possible to observe them, understand them, organize that understanding into general patterns, and critically debate it.

Is there a rival theory to LoLE? What else would explain the same observations in a different way and reject LoLE? I don’t know of something like that. I guess the main alternative is a blue pill perspective which heavily downplays the existence or importance of social hierarchies (or makes evidence-ignoring claims about them in order to virtue signal) – but that doesn’t make much sense in a society that’s well aware of the existence and prevalence of social climbing, popularity contests, cliques, ingroups and outgroups, etc.

Would LoLE Be Active For Conjunction Fallacy Research?

People form habits related to high status behaviors. For many, lots of social behavior and thinking is intuitive and automatic before high school.

People don’t turn off social status considerations without a significant reason or trigger. The Conjunction Fallacy experiments don’t provide participants with adequate motivation to change or pause their very ingrained social-status-related habits.

Even with a major reason and trigger, like Coronavirus, we can observe that most people still mostly stick to their socially normal habits. If people won’t context switch for a pandemic, we shouldn’t expect it for basically answering some survey questions.

It takes a huge effort and established culture to get scientists to be less social while doing science. And even with that, my considered opinion is that over 50% of working scientists don’t really get and use the scientific, rationalist mindset. That’s one of the major contributors to the replication crisis.

How Would LoLE Affect Answers?

Math and statistics are seen as high effort. They’re the kinds of things people habitually avoid due to LoLE as well as other social reasons (e.g. they’re nerdy). So people often intuitively avoid that sort of thinking even if they could do it.

Even many mathematicians or statisticians learn to turn that mindset off when they aren’t working because it causes them social problems.

LoLE encourages people to try to look casual, chill, low effort, even a little careless – the opposite of tryhard. The experimental results of Conjunction Fallacy research fit these themes. Rather than revealing a bias regarding how people are bad at logic, the results may simply reveal that social behavior isn’t very logical. Behaving socially is a different thing than being biased. It’s not just an error. It’s a prioritization of social hierarchy issues over objective reality issues. People do this on purpose and I don’t think we’ll be able to understand or address the issues without recognizing the incentives and purposefulness involved.

---

View this post on Less Wrong.

These are some summary notes on social dynamics.

• conformity
  • try to fit in
  • pandering
  • pleasing people
  • avoiding conflict
  • do whatever the group thinks is high status
    • follow trends
  • you need to already have friends. people are impressed by people who other people already like (pre-selection).
  • have standard interests like the right TV shows, music, movies and sports. talk about those. don’t say weird stuff.
    • you’re allowed to have interests people think they should have. lots of people think they should be more into art galleries and operas. you can talk about that stuff to people who don’t actually like it but pretend they want to. they’ll be impressed you actually do that stuff which seems a bit inaccessible but valuable to them.
• law of least effort
  • being chased, not chasing
  • people come to you
  • opposite of tryhard
  • less reactive
    • don’t get defensive or threatened (important for confidence too)
    • hold frame without showing visible effort
    • but also don’t let people get away with attacking you
  • when you attack people, it should seem like the conflict isn’t your fault, was just a minor aside, no big deal to you, preferably you weren’t even trying to attack them
  • people do what you say
  • you don’t have to do what other people say
  • you generally aren’t supposed to care that much about stuff. instead, be kinda chill about life
    • if you get ahead while appearing this way, it looks like success comes naturally to you. that impresses people. (it should not look like you got lucky)
• confidence
  • hide weakness
  • pretend to be strong
  • know what you’re doing, having a strong frame, have goals
  • be able to lead
  • best to already be leader of your social group, or at least high up like second in command
• value
  • DHVs (demonstrations of higher value, e.g. mentioning high value things in passing while telling a story)
  • money, popularity, fame, social media followers, loyal friends, skills, knowledge, SMV (sexual market value, e.g. looks)
  • abundance mentality
  • well spoken, know other languages, can play an instrument or sing, cultured, can cook, etc.
  • virtues like being moral and enlightened are important. these are group specific. some groups value environmentalism, being woke, anti-racist signaling, inclusive attitudes towards various out groups and low status people (poor people, immigrants, disabled, homeless, drug addicts), etc. other groups value e.g. patriotism, toughness, guns, Christianity and limited promiscuity.
• trend setting
  • this is hard and uncommon but possible somehow
  • mostly only available for very high status people (~top status in a small group can work; it doesn’t have to be overall societal status)
• non-verbal communications
  • clothes send social signals
  • voice tones
  • eye contact
  • body language
  • posture
  • leaning in or having people lean to you
• congruence
  • do not ever get caught faking social stuff; that looks really bad
• compliance
  • getting compliance from other people, while expending low effort to get it, it socially great.
    • it can especially impress the person you get compliance from, even more than the audience
• plausible deniability
  • there are often things (communications, actions) that a group understands but won’t admit that they understand the meaning of
  • there are ways to insult someone but, if called on it, deny you were attacking them, and most people will accept your denial
  • there are subtle, tricky rules about what is considered a covert attack that you’re allowed to deny (or e.g. a covert way to ask someone on a date, which you’re allowed to deny was actually asking them out if they say no) and what is an overt attack so denials would just make you look ridiculous.
  • social combat heavily uses deniable attacks. deniability is also great for risky requests
  • you’re broadly allowed to lie, even if most people know you’re lying, as long as it isn’t too obvious or blatant, so it’s considered deniable
  • basically social dynamics have their own rules of evidence about what is publicly, socially known or established. and these rules do not match logic or common analytical skill. so what people know and what is socially proven are different. sometimes it goes the other way too (something is considered socially proven even though people don’t know whether or not it’s true).
    • many social climbing techniques use the mismatch between what is socially known to the group and what is actually known to individuals. it lets you communicate stuff so that people understand you but, as far as the social group is concerned, you never said it.

Overall, high status comes from appearing to fit in effortlessly, while wanting to not being pushed into it, and not having social problems, weaknesses or conflicts. You can also gain status from having something valuable, e.g. money, looks, fame, followers or access to someone famous. Besides extreme cases, you still need to do pretty well at social skill even when you have value. Value is an advantage but if you act low status that can matter more than the value. If you have a billion dollars or you’re a movie star, you can get away with a ton and people will still chase you, but if you just have a million dollars or you’re really hot, then you can’t get away with so much.

Desired attitude: You have your own life going on, which you’re happy with. You’re doing your thing. Other people can join you, or not. It isn’t that big a deal for you either way. You don’t need them. You have value to offer, not to suck up to them, but because your life has abundance and has room for more people. You already have some people and aren’t a loaner. You only would consider doing stuff with this new person because they showed value X – you are picky but saw something good about them, but you wouldn’t be interested in just anyone. (Elicit some value from people and mention it so it seems like you’re looking for people with value to offer. You can do this for show, or you can do it for real if you have abundance. Lots of high status stuff is acting like what people think a person with a great life would do, whether you have that or not. Fake it until you make it!)

People socially attack each other. In this sparring, people gain and lose social status. Insults and direct attacks are less common because they’re too tryhard/reactive/chasing. It’s better to barely notice people you don’t like, be a bit dismissive and condescending (without being rude until after they’re overtly rude first, and even then if you can handle it politely while making them look bad that’s often better).

If you sit by a wall and lean back, you look more locked in and stable, so it appears that people are coming to you. Then you speak just slightly too softly to get people to lean in to hear you better, and now it looks like they care what you say and they’re chasing you.

---

These notes are incomplete. The responses I’d most value are some brainstorming about other social dynamics or pointing out data points (observed social behaviors) which aren’t explained by the above. Alternatively if anyone knows of a better starting point which already covers most of the above, please share it.

---

View on Less Wrong.

Discussions should end gracefully. There should be some clarity about why they’re ending and why that’s rationally acceptable. If someone wants to continue the discussion, they should have the opportunity to see why this ending is reasonable. (Or, failing that, they should have some evidence available to enable them to argue their case that it was unreasonable. They shouldn’t be left with little or no data about what happened.)

If you discuss poorly, it’s important that you can learn from that and do better next time. If you want it, you should have access to some critical feedback, some explanation of the other person’s perspective, or something to enable progress. If you’re running into problems interacting with people, but no one will tell you what the problem is, that’s bad for rational progress.

The more easily discussions can end, the more easily they can start. We don’t want discussing to be a big burden or commitment (it’s OK if a few discussions require high effort, when people have reason to voluntarily choose that, but that shouldn’t be the default).

Discussions can end by mutual agreement. If no one objects to the ending, that’s fine. That’s one way to end discussions. Pretty much everyone agrees on this. The difficulty is that sometimes people don’t agree. Someone wants to stop discussing but someone else wants to pursue the matter further.

I don’t think agreeing to disagree, by itself, is a good reason to end a discussion by mutual agreement. Disagreements can be resolved! There should be an extra factor like agreeing that it’s reasonable to prioritize other stuff (which we may leave implied without explicitly mentioning). There are many problems to work on, and time/energy/etc are scarce resources, so it’s fine to drop a disagreement (for now, indefinitely) if people think they can get more value working on other stuff.

Ending discussions when a single person wants to, for any reason, with no explanation, is problematic. For example, people will end discussions when they (the ideas they are biased in favor of) start losing the argument.

But we don’t want to trap people in discussions. Sometimes one guy has a lot of energy to discuss something forever but you want to stop. There are lots of legitimate reasons to end a discussion.

You can try to explain your reasons for ending a discussion, but no matter what you say, the other guy might disagree. This is a real concern because you don’t want to discuss extra with the most unreasonable people who give low quality counterarguments to all your reasons for stopping discussion. Meanwhile the most reasonable people tend voluntarily to let you out of discussions, so you discuss with them the least!?

There has to be a way to unilaterally end discussions. You end it without agreement. But it should have some protections against abuse. We don’t want it to be acceptable to arbitrarily end any discussion at any time for no reason or awful reasons. This is for your own sake, too, not just for the benefit of others. If I want to end a discussion and the other guy disagrees, I ought to consider that maybe I’m biased. I might be evading the issue, avoiding being corrected, etc. I shouldn’t just fully trust my own judgment. I should want some discussion policies that make it harder for me to be and stay irrational, biased, unreasonable, etc.

Note: Of course anyone can stop talking at any time for no reason. That’s a matter of freedom. No one should be held hostage. The issue is the consequences for your reputation. What is considered reasonable or rational? Some discussion ending behavior ought to be seen negatively by the community. Some good ways of ending discussions out to be encouraged, widespread and normal. Some bad ways of ending discussions should be disincentivized.

Note: Even if a discussion actively continues, you could participate in it on less than a daily basis. Discussions don’t have to be fast. Some norms are needed for what is stalling a discussion out (e.g. one reply per decade would be a way to pretend you didn’t end the discussion when basically you did). In my experience, people are usually pretty reasonable about discussion pacing, with a few notable exceptions. (The main problem I see is people who discuss a bunch initially and then never come back to it as soon as they go do something else or as soon as they sleep once.)

So we want a way to end a discussion, even if other people disagree with ending the discussion, but which has some protection against abuse (bad faith), error, bias, irrationality, etc. It should ideally provide some transparency: some evidence about why the discussion is ending that can be judged neutrally, positively or negatively by the audience. And it should provide some sort of feedback or learning opportunity for the guy who didn’t want to stop here.

So here’s the first draft of a policy: When you want to end a discussion, you are expected to write one final message which explains why you’re ending the discussion. You’re also expected to read one more message from the other guy, so he has one chance to point out that you’re making a mistake and he could potentially tell you why you should change your mind about ending the discussion.

What’s good about this policy? It helps limit abuse. It’s harder to end a discussion for a bad reason if you have to explain yourself. The other guy gets some info about what happened. The other guy has a chance at a rebuttal so you could potentially be corrected. And it’s not very time consuming. It puts a small, strict limit on how much more discussion happens after you decide you’d like to wrap it up.

This is a pretty minimal policy. I think it could be a good default expectation that LW could use for any discussion where people have each written 3+ messages (or maybe 5+ to reduce the burden? The number could be tuned if this was tried out for a while). That way it won’t add an extra burden to really small discussions. Tiny discussions would be discouraged by any overhead at all. Tiny discussions are also lower investment so ending them is a smaller deal. People haven’t formed a reasonable expectation of reaching a conclusion, getting their questions answered, or anything else. They’re just sharing some thoughts on an ad hoc, no-obligation basis. That’s fine. But for more substantive discussions, I think adding a little bit of an ending cost is reasonable.

The minimal policy has some downsides. If we had a policy that takes more effort, we could fix some problems and get some benefits. So I think for discussions that go longer (e.g. 10+ messages each) or when people mutually agree to make it a substantive discussion, then a longer but better approach could be used for unilaterally ending a discussion.

What are problems with the single parting message? It could be unclear. It could ignore a key issue. It could misrepresent the other guy’s positions or misrepresent what happened in the discussion. It could be poorly thought through and show major signs of bias.

What are the problems with a single rebuttal to the parting message that won’t be replied to? If it asks any clarifying questions, they won’t be answered. Any great points could be ignored without explanation.

So as a next step up, we could have a two-part discussion ending. Instead of one more back and forth (parting message + rebuttal), we could have two more back and forths. Initial parting message, rebuttal and questions, final parting message, and then final rebuttal.

BTW, the rebuttals are semi-optional. You can just decide to agree with the guy’s parting message if you want (maybe it makes sense to you once he explains his position). Or instead of a rebuttal or agreement, your other option is to write your own parting message. But you shouldn’t disagree with their parting message and then silently end the discussion with zero explanation of what’s going on.

Note: Parting messages don’t have to be very long. A few clearly written sentences can cover the key points (e.g. your opinion of the discussion, your final comments on some open issues, your reasons for ending). A bit longer is needed if you write fluff. And generally you should write a bit more if the discussion was longer.

With a two back-and-forth discussion ending, it’s still possible to dodge questions, avoid key issues, etc. It can take quite a few iterations to get some stuff cleared up, and that’s if people are being reasonable. Unreasonable people can sabotage discussions indefinitely.

So what about a three back-and-forth discussion ending? Or four or five? Nothing will be perfect or give a guarantee.

Let’s consider other approaches. What about a 5% ending? However many words you wrote in the discussion, you should write 5% of that number in the discussion ending phase. That seems kinda reasonable. That means for every 20 words of discussion you write, you’re potentially obligating yourself to one word later. This might need to be capped for very long discussions. This means your effort to end the discussion gracefully is proportional to the effort you put into the discussion.

This approach still suffers from being a fairly arbitrary cutoff. You just decide to end the discussion, say a few things that hopefully do a good job of exposing your reasoning to criticism and giving the other guy the chance to learn that he’s wrong, and say a few things to wrap up the open issues (like briefly answering a few key questions you hadn’t gotten to, so the discussion is left in a more complete form and your case is left adequately complete that someone could learn from you). I think that’s way better than nothing but still has significant potential to go wrong.

One useful technique is agreeing to escalate the commitment to the discussion. You can say “I will discuss X but only if you’ll agree to a 3 back and forth ending if you want to end the discussion when I don’t (which I’ll also agree to if I want to end it unilaterally).” It sometimes makes sense to not want to invest in a discussion then have it end abruptly in a way that’s unsatisfying and inconclusive from your perspective.

It makes sense to want a discussion to either be productive or else the other guy makes a clear claim – explained enough that you could learn from it – about what you’re doing wrong, so you have the opportunity to improve (or maybe to criticize his error and judge him, rather than being left with a lack of data). Someone could also explain why the discussion isn’t working in a no-fault way, e.g. you and he have some incompatible traits.

Saying “I’m busy” is broadly a bad excuse to end discussions. You were busy when you started, too, right? What changed? Sometimes people actually get significantly busier in an unforeseeable way in the middle of a discussion, but that shouldn’t be very common. Usually “I’m busy” is code for “I think your messages are low quality and inadequately valuable”. That claim isn’t very satisfying for the other guy without at least one example quote and some critical analysis of what is bad about the quote. Often people speak in general terms about low quality discussion without any quoted examples, which also tends to be unsatisfactory, because the person being criticized is like “Uhh, I don’t think I did that thing you’re accusing me of. I can’t learn from these vague claims. You aren’t showing me any examples. Maybe you misunderstood me or something.”

It can be reasonable to say “I thought I’d be interested in this topic but it turns out I’m not that interested.” You shouldn’t say this often but occasionally is OK. Shit happens. You can end a discussion due to your own mistake. When you do, you shouldn’t hide it. Let the other guy and the audience know that you aren’t blaming him. And maybe by sharing the problem you’ll be able to get some advice about how to do better next time. Or if you share the problem, maybe after a bunch of discussions you’ll be able to review why they ended and find some patterns and then realize you have a recurring problem you should work on.

Impasse Chains

Besides trying to end a discussion gracefully with a parting phase where a few things get explained, I have a different proposal: impasse chains.

An impasse is a statement about why the discussion isn’t working. We’re stuck because of this impasse. It’s explaining some problem in the discussion which is important enough to end the discussion (rather than being minor and ignorable). What if no one problem is ruinous but several are adding up to a major problem? Then the impasse is the conjunction of the smaller problems: group them together and explain why the group is an impasse.

Stating an impasse provides transparency and gives the other guy some opportunity to potentially address or learn from the discussion problem.

Impasses are meant to, hopefully, be solved. You should try to say what’s going wrong that, if it was changed to your satisfaction, you’d actually want to continue.

The other guy can then suggest a solution to the impasse or agree to stop. A solution can be a direct solution (fix the problem) or an indirect solution (a workaround or a better way to think about the issue, e.g. a reason the problem is misconceived and isn’t really a problem). You should also try to think about solutions to impasses yourself.

Sometimes the guy will recognize the impasse exists. Other times it’ll seem strange to him. He wasn’t seeing the discussion that way. So there’s some opportunity for clarification. Lots of times that someone wants to end a discussion, it follows some sort of fixable misunderstanding.

So far an impasse is just a way to think about a parting message, and you can hopefully see why continuing at least one more message past the initial impasse claim makes sense. So you may think this impasse approach just suggests having 2-5 messages (per person) to end discussions. And that’s decent – a lot of discussions do way worse – but I also have a different suggestion.

The suggestion is to chain impasses together.

So step 0, we discuss.

Step 1, I say an impasse and we try to solve it. This is an impasse regarding step 0.

Step 2, I decide the problem solving isn’t working in some way (otherwise I’d be happy to continue). So I state an impasse with the problem solving. This is an impasse regarding step 1.

Step 3, we try to solve the second impasse. Either this problem solving discussion satisfies me or I see something wrong with it. If it’s not working, I say an impasse with this discussion. This is the third impasse.

Each time an impasse is stated, the previous discussion is set aside and the impasse becomes the new topic of discussion. (Though a few closing comments on the previous discussion are fine and may be a good idea.) The impasse is either solved (and then you can return to the prior discussion) or leads to a new impasse. This can repeat indefinitely. You can have an impasse about the discussion of an impasse about the discussion of an impasse in the original discussion.

The impasses are chained together. Each one is linked to the previous one. This is different than multiple separate impasses with the original discussion. Here, we’re dealing with one impasse for the original discussion and then the other impasses in the chain are all at meta levels.

Note: If you see multiple impasses with the original discussion, often that means you tried to ignore one. Instead of bringing up the first one and trying to do problem solving, you let problems accumulate. It’s possible for multiple impasses to come up at the same time but it isn’t very common. In any case, you can deal with the impasses one at a time. Pick one to focus on. If it gets resolved, move on to the next one.

It doesn’t make sense to ask someone to discuss X further when he sees an impasse with discussion X (meaning a reason that discussion isn’t working). You’ll have to address that problem in some way or agree to stop. Discussing the problem itself is a different discussion than discussing X, so it should be possible to try it.

The more impasses chain, usually the clearer the situation gets. Each level tends to be simpler than the previous level. There are fewer issues in play. It becomes more obvious what to do. This helps but isn’t nearly enough to make impasse chains get addressed (either solve every impasse or agree to stop) in a reasonable amount of time.

Impasse chains often get repetitive. Problems reoccur. Suppose I think you keep saying non sequiturs. We can’t discuss the original topic because of that. So then we try to discuss that impasse. What happens? More non sequiturs (at least from my point of view)!

Some discussion problems won’t affect meta levels but some are more generic and will. You can try to say “OK given our disagreements about X, Y and Z, including methodology disagreements, what can we still do to continue problem solving which is neutral – which makes sense regardless of what’s correct about those open issues, and makes sense from both of our points of view?” Often what happens is you can’t think of anything. It’s hard. Oh well. Then you can mutually agree to end the discussion since neither of you knows a good way to continue.

When impasse chains get long, you tend to either have a lot of issues that are being set aside (given A, B, C, D, E, F, G, H are unresolved, what can we do?) or you have a lot of repetition (every meta level is the same problem, e.g. my belief that you’re replies are non sequiturs). Repetition is itself a reason to end the discussion. It’s just repeating and neither of us knows a way to fix it, so we can agree to stop.

This kind of ending is satisfying in a way. It gives transparency about why the discussion ended. It means (worst case scenario) I’ve gotten to make my case about what you’re doing wrong, and you’ve failed to answer it substantively, so now I’m reasonably content (not my favorite outcome, but what more would I hope to gain from continuing?).

So a policy can be e.g. to discuss until an impasse chain with the same problem 3 times in a row. Or to discuss until an impasse chain of at least length 5. Generally 5 is plenty. Reaching the 5th metal level can be quite fast and clarifying.

Impasses can be declared in bad faith. People can disagree about what is an impasse. Then what? Discussions have to proceed in ways that make sense to all parties. If someone thinks there is an impasse, then there is one, even if the impasse consists of his misconception. And yes bad faith is possible. What can be done about that? Transparency. Exposing it to daylight. Having systems like this that make bad faith more visible and easier to criticize. Having procedures that create more evidence of bad faith.

In general, by an impasse chain of length 5, if one person is being rational and the other isn’t, it gets really obvious who is who. This gives the rational guy reason to be satisfied with ending the discussion (he knows what happened and why, and he had some chances to try to solve it) and it gives evidence about both parties. If both people are fairly equal in skill or rationality, or both are pretty bad (even if unequally bad), then you can much more easily have muddy waters after an impasse chain of length 5. Oh well. Creating clearer impasse chains is a skill you can work on. You can learn from your attempts to create some clarity and what didn’t work and why, and try to do better next time. And you can try to learn from the other guy’s attempts to.

The impasse chain system is unnecessary for every discussion. It’s a bit heavyweight and high transaction cost to use all the time. But it’s pretty limited. If you agree to a 5 impasse chain, you’re always 5 messages away from getting out of the discussion. The only reason it’d take more is if the other guy said reasonable stuff. But if he says unreasonable stuff, you’re done in 5 messages, and some of those messages will often be just one paragraph or even one sentence long.

This approach is good when people want to claim they are open to debate and that their views have stood up to debate. That leads to disputes over what it means to be open to debate, etc. I propose being willing to enter into discussions terminated by a length 5 impasse chain (or mutual agreement) as a reasonable criterion for a (self-declared) serious intellectual to say he’s actually open to substantive debate about something and is actually addressing critics.

And the impasse chain approach can be requested when you want to have a discussion if and only if there will be a substantial ending to protect your effort investment. If you want to avoid a case of putting in a bunch of effort now and then the guy just leaves, you can ask for an impasse chain precommitment or 5 parting message precommitment or other way to help protect (and therefore enable) your energy investment into the discussion.

Concluding Thoughts

What’s the typical failure case look like? Joe is trying to have a rational discussion and Bob says “eh, your messages are lame; bye” and won’t answer questions or address arguments. Or, worse, Bob explains even less than that. If Bob would explain that much, at least people could see “OK Bob accused Joe of lame messages and gave zero arguments. Therefore Bob is lame.” Impasse chains or even just parting messages help enable problem solving as well as clarifying what happened in bad outcomes.

Often a discussion looks like this: Joe writes a blog post. Bob says some criticism. Joe sees many flaws in the criticism. Joe explains the flaws. Bob stops talking. This isn’t satisfying for Joe. He never got feedback on his post from post-misconception Bob. And Bob probably didn’t change his mind. And Bob didn’t even say what the outcome was. Or if Bob did change his mind about something, it’s often a partial change followed immediately by like a “you win; bye”. People routinely use conceding as a way to end discussions with no followups: no post mortem (learning about why the error happened and how to fix the underlying cause), no working out the consequences of the right ideas, etc. The correction doesn’t go anywhere.

That’s sad for Joe. He didn’t want to correct Bob just for fun. He wanted to correct Bob so it’d lead to something more directly beneficial to Joe. E.g. Joe’s correction could be criticized and that’d have value for Joe (he learns he was actually wrong in some way). Or Joe corrects Bob and then it leads to further discussion that’s valuable to Joe. If correcting people is pure charity – and you usually get ghosted without them admitting they were corrected – then people will even try to do it way less. There should be rewards like some sort of resolution to the issues and continuation. Discuss productively and keep going (and maybe Joe learns something later or, failing that, at least gets a good student who learns a bunch of things and may be able to suggest good ideas in the future), or say the impasse for why it’s not productive.

Often Bob thinks Joe is doing something wrong in the discussion but won’t explain it enough for Joe to have a reasonable opportunity to learn better. Note that cites are fine. If it’s already explained somewhere, link it. Just take responsibility for the material you refer people to: you’re using it as our proxy, to speak for you, so you ought to have similar willingness to address questions and criticisms as if you’d written it yourself (but if it’s popular stuff with lots of existing discussion, then you can address FAQs by referring the guy to the FAQ, can direct him to the forum for that community to get questions answered and only answer them yourself if the forum won’t answer, and you can link other blog posts, books, papers, etc. to address followup issues if those exist, etc.)

Impasse chains help address these problems and help make it harder to end discussions due to your own error and bias. And they provide opportunities to solve discussion problems instead of just giving up at the first problem, or in the alternative at least more transparency about the problems can be achieved.

See also

My prior article on Impasse Chains.

My articles on Paths Forward (about discussing in such a way that if you’re wrong and anyone knows it and is willing to tell you, you never block that off with no way for your error to be corrected), including the article Using Intellectual Processes to Combat Bias.

My debate policy.

---

View this post at Less Wrong.

Discussion with gigahurt started here. He wrote (quoting me):

Disagreements can be resolved!

I see your motivation for writing this up as fundamentally a good one. Ideally, every conversation would end in mutual understanding and closure, if not full agreement.

At the same time, people tend to resent attempts at control, particularly around speech. I think part of living in a free and open society is not attempting to control the way people interact too much.

I hypothesize the best we can do is try and emulate what we see as the ideal behavior and shrug it off when other people don't meet our standards. I try to spend my energy on being a better conversation partner (not to say I accomplish this), instead of trying to make other people better at conversation. If you do the same, and your theory of what people want from a conversation partner accurately models the world, you will have no shortage of people to have engaging discussions with and test your ideas. You will be granted the clarity and closure you seek.

By 'what people want' I don't mean being only super agreeable or flattering. I mean interacting with tact, brevity, respect, receptivity to feedback, attention and other qualities people value. You need to appeal to the other person's interest. Some qualities essential to discussion, like disagreeing, will make certain folks back off, even if you do it in the kindest way possible, but I don't think that's something that can be changed by policy or any other external action. I think it's something they need to solve on their own.

Then I asked if he wanted to try to resolve one of our disagreements by discussion and he said yes. I proposed a topic related to what he'd written: what people want from a discussion partner and what sort of discussion partners are in shortage. I think our models of that are significantly different.

---

Post with gigahurt discussion tree and YouTube video playlist:

http://curi.us/2368-gigahurt-discussion-videos

I thought about how to use LW comments better and get along with people better. I wrote notes about it:

• Write comments that would be appreciated by a new observer, who hasn’t read any of my previous stuff, hasn’t read the sequences, and only skimmed the post he’s commenting under.
• Only reply to comments I actually think are good. If I see any signs of low quality, hostility, or social aggression, don’t reply.
• Make it clear to people in my bio, and at the end of some posts, that I’m open to more discussion by request. I can change policies and be more responsive if asked.
• Avoid meta discussion. Lots of LW people don’t like it. I think meta discussion is very important and valuable, but I can write it at my own forums.
• I plan to have two clearly distinguished commenting modes. I think a middle ground between them causes trouble and I want to avoid that.
  • Mode one is anything goes, zero pressure to reply, drop anything on a total whim with no explanation. This mode will be my default and means I’ll be replying less than I was.
  • For these comments, I’ll try to make most of my comments standalone and interesting. That means only engaging with people who say something significant and worthwhile. Otherwise I can write a monologue “reply” (that doesn’t engage with specifics of what they said, just talks about the topic) or not reply.
    • A short answer like “Yes” is OK too because it won’t annoy any readers. People who don’t get value from it will know it’s contextual and won’t mind.
    • It’s important to be careful about comments which rely on context but don’t obviously do so. People can think those comments are meant to stand alone when they aren’t. So try to make comments really blatantly be minor followups or else offer standalone value.
  • Mode two is high effort discussion after some mutual agreement to try to use some sort of written rules. Examples of discussion policies that could be used:
    • Discussion to agreement or length 5 impasse chain. Agreement can be about the topic or agreeing to stop the discussion – any sort of agreement is fine.
    • Discussion until agreement or someone claims that they made an adequate case that the other person ought to be able to learn from and be corrected by. They believe their case would persuade a neutral, reasonable observer. Plus minimum two followups to address parting questions (like which text constitutes the adequate case, and isn’t the case inadequate due to not addressing questions X and Y that were already asked?) or potentially be persuaded to change their mind about ending there.
    • Discussion to agreement or to one stated impasse plus two followups to address final questions and have a brief window to potentially solve the impasse.
• The discussion mode I do not want is a medium effort discussion following unwritten rules (particularly social hierarchy climbing related rules). I prefer either small or large discussion. Either anyone can leave at any moment with no negative judgments or we set up a more organized, serious discussion project. I don’t want to half-ass adding transaction costs and commitments into discussions. Do that as a big enough deal to agree on and write down some discussion rules and policies, or don’t do it and stick to anarchy. Unwritten rules suck so either use written rules or no rules.
• I don’t trust people to be OK with no-commitment discussion, despite having recently been told by a several people that that’s how LW works. I think LW mostly works by medium commitment discussion where there are social pressures. I think people routinely are judged for not replying.
  • It’s hard to deal with because asking people if they want a serious discussion, in reply to their topical comment, gets negative responses. They don’t want to state what sort of discussion they want (zero commitment, medium commitment with unwritten rules, or more serious project with written rules). I take people’s general dislike of stating what rules they are playing by, or want me to play by, as a piece of evidence that it’s unwritten rules and medium commitment that are commonly desired. I don’t think people are usually really 100% fine with me ignoring them an unlimited number of times without explanation because, hey, no commitment and no request for anything different. I think they’ll see me as violating unwritten rules saying I should be somewhat responsive. (I personally wouldn’t mind explaining why I think someone’s comments are bad and why I don’t want to reply, but I think the LW forum generally does mind me doing that. If people want to know that they are welcome to ask me at my forum about particular cases from LW. But I don’t like being asked privately because I want my regular audience to be able to see my answers.)
  • Broadly if anyone has any problem with me, I hope they’ll state it or ask a direct question about it. I don’t expect it but I do prefer it. I know people often ask for such things and don’t mean it, but I have an 18 year track record of public discussion archives showing I actually mean it, and I run a discussion community where such things are normal. Posts like “Why did no one reply to this?” or “Why didn’t you reply to this?” are well within norms at my forums, and I do get asked meta questions like why I dealt with a discussion in a particular way.
  • I don’t plan to ask other people at LW direct questions about problems I have with them, or state the problems, unless they ask me to do that and I find the request convincing (e.g. I can’t find signs of dishonesty or incompetence). Even then, I might ask them to forum switch first because I think other people at LW could easily take that discussion out of context and mind it (the context being the convincing request).
• I would like some discussions where we try to make an organized, serious effort to seek the truth, resolve some disagreements, etc. But that is not the LW norm. The LW norm is mostly a mix of small and medium discussions. OK. My solution is to make big discussions available by request and otherwise do small discussions. This should be acceptable for both me and others.

---

View this post at Less Wrong.

This post contains highlights from my discussion at The Law of Least Effort Contributes to the Conjunction Fallacy on Less Wrong. The highlights are all about social dynamics.

---

I view LoLE [Law of Least Effort] as related to some other concepts such as reactivity and chasing. Chasing others (like seeking their attention) is low status, and reacting to others (more than they're reacting to you) is low status. Chasing and reacting are both types of effort. They don't strike me as privacy related. However, for LoLE only the appearance of effort counts (Chase's version), so to some approximation that means public effort, so you could connect it to privacy that way.

---

I think basically some effort isn't counted as effort. If you like doing it, it's not real work. Plus if it's hidden effort, it usually can't be entered into evidence in the court of public opinion, so it doesn't count. But my current understanding is that if 1) it counts as effort/work; and 2) you're socially allowed to bring it up then it lowers status. I see privacy as an important thing helping control (2) but effort itself, under those two conditions, as the thing seen as undesirable, bad, something you're presumed to try to avoid (so it's evidence of failure or lack of power, resources, helpers, etc), etc.

---

Maybe another important thing is how your work is.... oriented. I mean, are you doing X to impress someone specific (which would signal lower status), or are you doing X to impress people in general but each of them individually is unimportant? A woman doing her make-up, a man in the gym, a professor recording their lesson... is okay if they do it for the "world in general"; but if you learned they are actually doing all this work to impress one specific person, that would kinda devalue it. This is also related to optionality: is the professor required to make the video? is the make-up required for the woman's job?

…

You can also orient your work to a group, e.g. a subculture. As long as its a large enough group, this rounds to orienting to the world in general.

…

I think orienting to a single person can be OK if 1) it's reciprocated; and 2) they are high enough status. E.g. if I started making YouTube videos exclusively to impress Kanye West, that's bad if he ignores me, but looks good for me if he responds regularly (that'd put me as clearly lower status than him, but still high in society overall). Note that more realistically my videos would also oriented to Kanye fans, not just Kanye personally, and that's a large enough group for it to be OK.

---

Do the PUAs really have a good model of an average human, or just a good model of a drunk woman who came to a nightclub wanting to get laid?

PUAs have evidence of efficacy. The best is hidden camera footage. The best footage that I’m aware of, in terms of confidence the girls aren’t actors, is Mystery’s VH1 show and the Cajun on Keys to the VIP. I believe RSD doesn’t use actors either and they have a lot of footage. I know some others have been caught faking footage.

My trusted friend bootcamped with Mystery and provided me with eyewitness accounts similar to various video footage. My friend also learned and used PUA successfully, experienced it working for him in varied situations … and avoids talking about PUA in public. He also observed other high profile PUAs in action IRL.

Some PUAs do daygame and other venues, not just nightclubs/parties. They have found the same general social principles apply, but adjustments are needed like lower energy approaches. Mystery, who learned nightclub style PUA initially, taught daygame on at least one episode of his TV show and his students quickly had some success.

PUAs have also demonstrated they’re effective at dealing with males. They can approach mixed-gender sets and befriend or tool the males. They’ve also shown effectiveness at befriending females who aren’t their target. Also standard PUA training advice is to approach 100 people on the street and talk with them. Learning how to have smalltalk conversations with anyone helps people be better PUAs, and also people who get good at PUA become more successful at those street conversations than they used to be.

I think these PUA Field Reports are mostly real stories, not lies. Narrator bias/misunderstandings and minor exaggerations are common. I think they’re overall more reliable than posts on r/relationships or r/AmITheAsshole, which I think also do provide useful evidence about what the world is like.

There are also notable points of convergence, e.g. Feynman told a story ("You Just Ask Them?” in Surely You’re Joking) in which he got some PUA type advice and found it immediately effective (after his previous failures), both in a bar setting and later with a “nice” girl in another setting.

everyone lives in a bubble

I generally agree but I also think there are some major areas of overlap between different subcultures. I think some principles apply pretty broadly, e.g. LoLE applies in the business world, in academia, in high school popularity contests, and for macho posturing like in the Top Gun movie. My beliefs about this use lots of evidence from varied sources (you can observe people doing social dynamics ~everywhere) but also do use significant interpretation and analysis of that evidence. There are also patterns in the conclusions I’ve observed other people reach and how e.g. their conclusion re PUA correlates with my opinion on whether they are a high quality thinker (which I judged on other topics first). I know someone with different philosophical views could reach different conclusions from the same data set. My basic answer to that is that I study rationality, I write about my ideas, and I’m publicly open to debate. If anyone knows a better method for getting accurate beliefs please tell me. I would also be happy pay for useful critical feedback if I knew any good way to arrange it.

Business is a good source of separate evidence about social dynamics because there are a bunch of books and other materials about the social dynamics of negotiating raises, hiring interviews, promotions, office politics, leadership, managing others, being a boss, sales, marketing, advertising, changing organizations from the bottom-up (passing on ideas to your boss, boss’s boss and even the CEO), etc. I’ve read a fair amount of that stuff but it’s not my main field (which is epistemology/rationality).

There are also non-PUA/MGTOW/etc relationship books with major convergence with PUA, e.g. The Passion Paradox (which has apparently been renamed The Passion Trap). I understand that to be a mainstream book:

About the Author
Dr. Dean C. Delis is a clinical psychologist, Professor of Psychiatry at the University of California, San Diego, School of Medicine, and a staff psychologist at the San Diego V.A. Medical Center. He has more than 100 professional publications and has served on the editorial boards of several scientific journals. He is a diplomate of the American Board of Professional Psychology and American Board of Clinical Neuropsychology.

The main idea of the book is similar to LoLE. Quoting my notes from 2005 (I think this is before I was familiar with PUA): “The main idea of the passion paradox is that the person who wants the relationship less is in control and secure, and therefore cares about the relationship less, while the one who wants it more is more needy and insecure. And that being in these roles can make people act worse, thus reinforcing the problems.”. I was not convinced by this at the time and also wrote: “I think passion paradox dynamics could happen sometimes, but that they need not, and that trying to analyse all relationships that way will often be misleading.” Now I have a much more AWALT view.

The entire community is selecting for people who have some kinds of problems with social interaction

I agree the PUA community is self-selected to mostly be non-naturals, especially the instructors, though there are a few exceptions. In other words, they do tend to attract nerdy types who have to explicitly learn about social rules.

Sometimes I even wonder whether I overestimate how much the grass is greener on the other side.

My considered opinion is that it’s not, and that blue pillers are broadly unhappy (to be fair, so are red pillers). I don’t think being good at social dynamics (via study or “naturally” (aka via early childhood study)) makes people happy. I think doing social dynamics effectively clashes with rationality and being less rational has all sorts of downstream negative consequences. (Some social dynamics is OK to do, I’m not advocating zero, but I think it’s pretty limited.)

I don’t think high status correlates well with happiness. Both for ultra high status like celebs, which causes various problems, and also for high status that doesn’t get you so much public attention.

I think rationality correlates with happiness better. I would expect to be wrong about that if I was wrong about which self-identified rational people are not actually rational (I try to spot fakers and bad thinking).

I think the people with the best chance to be happy are content and secure with their social status. In other words, they aren’t actively trying to climb higher socially and they don’t have to put much effort into maintaining their current social status. The point is that they aren’t putting much effort into social dynamics and focus most of their energy on other stuff.

Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz) by Ben Pace:

Here's my overview of their positions.

• Zvi thinks that blackmail would incentivize a whole host of terrible actions, such as trying to trick people into norm violation, and people becoming intensely secretive even around their closest friends and family.
• Robin thinks that blackmail is a weird rule, where you cannot ask for money to keep a secret, but the other person is allowed to offer it (e.g. people can offer you money if you sign an NDAs). This makes no sense and Robin is looking for any clear reason why making one side of this deal should be illegal.

Thoughts Before Reading More of the Post

there are practical problems. e.g. blackmailers may make additional demands later, and it’s hard to hold them to the original deal b/c if you bring up the deal for enforcement then your secret comes out

and ppl actively trying to get blackmail material is a problem, as Zvi says

NDAs are done preemptively which is different. you ask someone to agree to keep X secret as a condition of sharing X with him.

blackmail is someone who already knows X in advance and is now negotiating for potential secrecy.

NDAs are commonly signed with no payment, or at least no direct payment for them (required with employment)

ppl sign NDAs b/c they want access to the info b/c they have an actual productive purpose for using that info themselves – often a business purpose like providing consulting advice about it or developing software related to it.

blackmailers commonly have no productive uses of the info.
their choices are roughly do nothing or hurt someone.

in cases where the blackmailer got the info legally and it isn’t e.g. copyrighted, and he isn’t lying (no fraud), it’s unclear where the initiation of force is if he shares it with the world. and if that’s not force, it’s hard to see how offering not to do it for money would be force.

it’s easy to see why this sort of thing is bad for society, at least in the sorts of scenarios typically associated with blackmail.

let’s consider an example. i see you in public with a woman. you kiss. i know you’re married. i don’t take any photos. i just witness it and then tell you that i’m going to (accurately and honestly) tell your wife unless you pay me $1000. if you pay once, i contractually agree to never use this again, even if I see you with that woman again. (i can ask for more money if i catch you with a different woman, though.)

i tried to keep it simple by having everything occur in public and avoiding photos/recordings. and i wouldn’t exaggerate to your wife, and you don’t think i would, so there isn’t an issue of fraud (tricking her, or you getting the impression that i’m threatening to say anything untrue or misleading to her).

damages ought to be big – much more than $1000 – if i tell your wife anyway or carelessly tell other people who spread the news. this can be a practical problem if i’m not wealthy enough to recover significant damages from. blackmailers are often hard to sue if they break contract. but in terms of principles, the contract excludes asking for additional payments or screwing you over anyway, and if i do that i’m now clearly guilty of initiating force.

why is this such a problem? because you’re doing something you don’t want your wife to know about. you’re lying to your wife and you want to get away with it. maybe we should blame you for the problematic situation, not me. if i keep my mouth shut, we still don’t get an OK situation. getting a good situation requires you changing your actions.

i think that’s a known idea. roughly: don’t do stuff that you could be blackmailed for.

in other words, if it’s worth $10,000 to you for no one to know you’re cheating on your wife, then don’t kiss other women in public, unless you don’t mind paying. (and if 50 people see you, and realize what’s going on, then you’re not going to be able to pay enough money to them all while staying under the limit of what the secret is worth to you; you’ll just have to accept you’re caught).

what’s hard about avoiding blackmailable actions?

there are lots of reasonable choices which society disapproves of. desire for privacy is often ok. this can happen even if you don’t lie to anyone. a simple example is you’re gay and don’t want people to know, though that generally involves some lying.

some things look bad but are ok if explained. but if the public is told, you won’t get a chance to immediately explain to everyone as soon as they get the info. e.g. maybe there is a photo of you with a young child. people might believe you’re a pedophile when actually you’re volunteering for the Big Brother charity. the photo is misleading because you were roleplaying an abduction scenario to help educate him. the photographer doesn’t even known, but instead of telling the public he goes to ask you for money … wait that’s not ok, he shouldn’t be asking for money to help cover up what he believes to be a crime. so let’s make the scenario milder, you’re just hugging him … well either that’s innocent or evidence of a crime so from either pov he shouldn’t be asking you for money (either he thinks it was fine so what is his threat exactly … to mislead the public? – or he thinks it suggests a crime so he shouldn’t cover it up).

There’s no positive duty to report crimes, but getting into a financial relationship to hide a crime (or what you think may be a crime) sounds rightly illegal. That’s like being hired by the criminal as a sort of accomplice. Note: It’s still bad if you’re paid with favors (services) instead of money.

OK so I want a scenario where it’s merely embarrassing, so being paid to help you hide it is ok. And the person being blackmailed hasn’t lied to anyone or done anything that’s actually bad. So I misjudge you as being a friend and tell you about my BDSM sex with my wife which I don’t want the public to know about. I didn’t think you were contractually bound not to tell; I just thought you were nice. I haven’t lied to the public about my sex life; I’ve just kept it private.

So you say you’ll post it on your blog unless I give you $100. You’d be doing this for the sole purpose of hurting me. But it seems to be within your rights to post it. And if it is, what’s wrong with offering a trade where you don’t do it in exchange for $100?

Normally trade is value for value. It’s supposed to be mutual benefit. Refraining from hurting me isn’t a service in the usual sense.

We could try to compare it to: I offer my neighbors $100 not to play music this weekend (it’s fine with headphones, I just don’t want my party disrupted). So now i’m paying for them not to do something. Why? Well they often play music with speakers. It’s something they gain value from. I want them to give it up so I compensate them.

We can imagine the reverse scenario in two ways. First, the neighbors come to me and say “I heard you have a party this weekend. We know our music often annoys you. We’ll be quiet all weekend for $100 if you want. Otherwise we’ll act normally.” Now compare this to scenario 2: The neighbors are normally quiet, hear about the party, and come say they will be louder than usual, just to disrupt your party, unless you pay $100 for quiet.

I think this is important. Paying someone to change their usual behavior and refrain from something is fine. The source of that behavior was to live their own life, not to hurt you. But if they are threatening to do a behavior for the purpose of hurting you, not to improve their own life, then something bad is going on. This is kinda hard for law to deal with though – it’s hard for courts to judge they aren’t playing loud music as a new hobby or whatever – though it becomes easier to judge if they approach you with the threat to play loud music unless you pay.

The purpose of political rights is to enable you to live your own life and have a good life. It’s not intended to give you any right to hurt others for no benefit to yourself (or to threaten to do that in order to demand money).

So I should reexamine first principles. Basically I have a right to control my body and, by extension, my property. I have freedom to live my life. But I don’t get to interfere with others. Interfering with other people’s body or property is initiation of force. In some cases, it’s hard to tell what counts as interference. Light and sound from my activities on my land often find their way onto my neighbor’s land and could annoy him (or could leak information that I don’t want him to have – and my first guess is I should need to take only reasonable precautions, and if he buys super high tech spying equipment then he’s doing something wrong despite only using it on the physical particles on his property). Tradition and law have to deal with stuff like sharing rivers, noise pollution, adding a tall building on my property that could block the view from your house, ugly stuff on my property being visible from your property, etc.

Part of how we deal with these things is by good faith and good will, not just by strictly defining what is force. It’s too hard to get along purely by saying what is force or not. That’s not enough to build up a good society. You need people who want to get along and mostly try to avoid conflict. Blackmail violates that.

Since blackmail is unnecessary to pursuing positive values in your own life, and undermines and attacks social harmony, I think it’s reasonable to make it illegal. This may change in the future when everything in public is getting recorded and posted online automatically and you’ll just need to be more careful in public or figure out some other way to deal with the situation, but for now it’s good that people generally don’t have to be too scared while in public (or while on their own property but inadvertently sending quite a bit of information (information from a physics perspective) off their property because they lack adequate technology to control that).

I do think we should be very careful about legitimizing laws that go beyond prohibiting force (including fraud and threat). The main danger of additional laws is they can restrict people’s freedom to have a good life, which this one doesn’t because I think the defining feature of blackmail is threatening to do something that isn’t part of pursuing your own values, it’s something you’d do just to hurt others. Laws against malice and tricky and dangerous in general but this looks OK to me. Trade should be for mutual benefit, not for “i found a way to harm you that doesn’t count as force; pay me not to do it”.

But fundamentally rights have a positive purpose to enable a good life and to resolve conflicts peacefully. The fundamental thing isn’t to ban certain actions like force. That’s a means to an end. So if someone finds a way to hurt people that isn’t banned (and IRL it is banned, but it isn’t clearly banned by the non-aggression principle), that looks to me like finding a hole in the rules while going against the purpose of the rules. Whereas if I thought the NAP (non-aggression principle, aka don’t initiate force) was primary, then I might think “technically doesn’t violate NAP, therefore good”. But I don’t see the NAP as a source of values or morality itself, directly.

I do think this merits more thought and analysis. I’m not fully satisfied yet. But I might be done anyway. This isn’t my main field and I have other things to do. Now I’ll read the debate highlights and see if I have any reason to change my mind already.

Thoughts After Reading the Rest of the Post

Hanson says legalizing blackmail could reduce affairs. So I guess the positive value of blackmail, in some cases, is trying to clean up society. I want to out people who cheat on their spouses to make a better world. That’s part of pursuing my values.

But if that’s really what I want, I can just do it. I wouldn’t be like “pay me to let you keep cheating on your wife”. I’d just get the info and share it. I’d try to get paid by the tabloids or by people who also value cleaning up the sexual morals of society.

So Hanson’s first claim, under “What's good about blackmail”, hasn’t changed my mind/analysis.

Zvi Mowshowitz: I believe that most of the time, most people are not, when they seek information, primarily looking to harm someone. They're primarily looking to benefit themselves, or benefit their friends or their allies in some way. Hurting someone else is mostly a side effect.

Robin Hanson: That's also true with blackmail. The main effect is the money, not the hurt. Their main motivation is to get the compensation.

This part stood out to me because it’s similar to some stuff I was talking about. I think seeking out and publishing info to pursue your own values is fine, but I take issue with doing it in order to threaten to harm someone. If you really want the info because you value it, try to get paid by other people who value it, rather than getting paid to hide the info that you value being public.

At the end they poll the audience re changing their mind. What about me? My initial opinion was blackmail sounds bad but I’m not that sure because it’s not clear how it initiates force. My updated opinion is that, due to my own analysis, I’m more confident that blackmail should be (stay) illegal. But if it came up in my life I’d want to do more analysis before acting (unless I didn’t have time to analyze, in which case I’d view blackmailing as bad and e.g. refrain from blackmailing someone even if I had no fear of getting in legal trouble for doing it).

---

View this post at Less Wrong.

Well-Kept Gardens Die By Pacifism by Eliezer Yudkowsky:

Good online communities die primarily by refusing to defend themselves.

Somewhere in the vastness of the Internet, it is happening even now. It was once a well-kept garden of intelligent discussion, where knowledgeable and interested folk came, attracted by the high quality of speech they saw ongoing. But into this garden comes a fool, and the level of discussion drops a little—or more than a little, if the fool is very prolific in their posting. (It is worse if the fool is just articulate enough that the former inhabitants of the garden feel obliged to respond, and correct misapprehensions—for then the fool dominates conversations.)

And what if you’re wrong about who the fools are?

What if you’re biased?

Where are the error correction mechanisms? Where are the objective tests to check that you aren’t screwing up?

Where’s the transparency? The rule of law, not rule of man? Where are the checks and balances?

If all you’ve got (as the moderator) is “I trust my judgment” then you’re just like everyone else, including the fool.

If you add some methods to try to have good judgment and try not to be biased … that’s not enough. Trying really hard, but ultimately trusting yourself and just going by what makes sense to you … is not a good enough defense against bias. Bias is powerful enough to beat that.

If fundamentally you’re just assuming your biases are pretty small and manageable, you are a fool. We are all alike in our infinite ignorance. We’re at the beginning of infinity. We’re so lost and confused in so many ways. We have big errors. Really big errors. Some are society-wide. And when we suppress everything that seems really quite wrong to us, what we’re doing is suppressing outliers. Yeah negative outliers are more common than positive outliers. It’s easier to be wrong than right. But how do you avoid suppressing positive outliers?

There are mechanisms you can put in place to make it harder to act on whim, bias, irrationality, etc.

E.g. you can explain why you take moderator actions. And you can field questions about your actions and reply to criticism. You could reply to all such things, plus followups. If you won’t do that, that’s a cutoff where you’re blocking error correction. And what have you done to prevent this cutoff from protecting the biases you may have?

Yes defense mechanisms are needed. But why can’t they be efficient and reusable ways to address the arguments of everyone, including fools? And a resilient forum where people think for themselves about what to focus attention on. If people want curation, fine, no problem, post curated selections somewhere – and leave the open forum also in existence. Post weekly favorites or whatever for the readers who don’t want to find the good stuff themselves. The curated view on the forum doesn’t have to be the only view. You can have an open public space, and a walled garden, both. Or dozens of competing walled gardens with different curators (thought only a few would probably have the vast majority of the popularity). But that’s dangerous. The curators may be biased. They may curate mostly by social status, for example. They may not know they do that. They may not understand some of their own systematic biases.

You have to fundamentally stop assuming you’re right or probably right and take seriously that you need error correction mechanisms to keep yourself honest. You can’t trust your own integrity. Don’t bet your life or your forum on your own integrity.

Scientists don’t bet science on their own integrity. Integrity helps. Try to have it. But science isn’t like “ok well do your best with the experiment and if you have integrity it should work out ok”. Instead experiments are designed to work out ok even if the experimenters can’t be trusted. The mechanisms don’t have unlimited resilience. Egregious scientific fraud has to get caught by outsiders. The scientific method makes that easier to catch. It’s harder to fake your experiments when there are procedures to follow, documentation to provide, etc. Most people are trying to cheat without full conscious intention though, so that’s easier to deal with. Having a little bit of integrity is really helpful compared to none. And anti-bias mechanisms with transparency and stuff do put a leash on the bad faith cheaters.

Double blind makes it harder to cheat even if you want to. You can be really biased, and your bias can control you, but if you follow the rules of double blind then it’s much harder for you to bias the results.

Control groups don’t care if you’re biased. That’s a system which is hard to cheat without extreme blatantness like skipping the control group entirely and using fabricated data. And it’s hard to do that because of transparency. And even if you get away with it, your results won’t replicate.

You’re expected to write about sources of error. If you don’t, people will write them for you and be more wary of your claims. If you do, people will consider how severe the issues are. If you write them but try to bias them, you’ll have a harder time convincing people who don’t share your biases. And when you leave stuff out, people can notice and then it’s harder for you to answer critics by claiming you totally knew about that and took it into account already.

Even when everyone shares biases, methods like “make a hypotheses. write it down. plan an experiment to test it. write down what results will agree with or disagree with the hypotheses. test it. compare the results to the predictions.” are capable of correcting everyone. That kind of method makes it way harder to fool yourself. If you skip steps like writing things down as you go along, then it’s much easier to fool yourself.

Forums need the same thing. Write down rules in advance. Make the rules totally predictable so people can know in advance what violates them or not. Don’t rely on potentially-biased judgment.

And when you must use judgment at a forum, be like a damn scientist, publish results, talk about sources of error, answer your critics, answer all the questions and doubts, etc. Take that seriously. Discuss the matter to conclusion. At least don’t be and stay wrong if anyone in your community knows you’re wrong and will explain it.

If you can’t resolve discussions, fix that and consider that maybe you’re a bit of a fool. If you don’t know how to get to conclusions with your critics, or manage those discussions without just ignoring arguments without answering, you need better systems and shouldn’t be so high and mighty to proclaim who is a fool.

Ideally, let the fool defend himself, too, Don’t just let others defend him. Topic-limit him during that discussion if you must so he can’t participate in other discussions until it’s resolved.

---

Also in the article, EY says academia is a walled garden that keeps the fools out, so that’s why ppl are naive and don’t realize they need censors. And I’m like: Yeah that is exactly what academia is and it’s fucking awful there. And academia’s walls are 99% based on social status.

What is your forum doing to prevent social status bias from deciding who the fools are? What explicit policies does it have that could actually work in case that you, the moderators, are social status biased?

EY’s answer is basically “if the mods suck, the forum is fucked”. Just find other, better people to rule. What an awful answer. Seriously that’s his position:

Any community that really needs to question its moderators, that really seriously has abusive moderators, is probably not worth saving.

No! You need good systems, not sinless, unbiased moderators (nor mediocre moderators who aren’t all that bad and you just put up with their errors). It’s like: dear God we’re not going to get unbiased politicians; we need a government system that works anyway. Forums are the same thing. Write out laws in advance. Make new laws with a process. Anyone who doesn’t violate a law in a crystal clear way gets away with it. etc. Otherwise the moderators will have some sort of biases – doesn’t everyone? – and they’re going to oppress the people with other biases who are such a source of intellectual diversity. Broaden your horizons instead of getting rid of all your dissidents and misfits and outliers. God damn you, “overcoming bias” you say?

We know a lot about how to deal with bias, abuse, unfairness, etc. from our legal system and from science. Then people don’t apply those lessons when they have power over a forum.

I have seen rationalist communities die because they trusted their moderators too little.

…

Here—you must trust yourselves.

You don’t overcome your biases by trusting yourself.

Don’t be a skeptic. Don’t never reach any conclusions or judgments. Don’t be scared to act in your life. But don’t trust your moderation to be unbiased. Have layers and layers of safety valves that provide different opportunities for you to be corrected if you’re wrong or biased. Never trust yourself such that you don’t think you need any mechanisms to keep you honest.

---

The scientific method makes it harder to be wrong and stay wrong. We need stuff like that for running forums too.

The scientific method does not consider it adequate for scientists to read about how to be good scientists, think about it, discuss it, do their best, and trust themselves. That would be ridiculous.

And the laws have lots of protection against the errors and biases of the individuals who enforce them. E.g. police, judge, jury and executioner are all different people. At forums they’re usually all the same people, which means you need even more safety mechanisms of other types. And we write out clear, objective laws in advance – no retroactive crimes. And there are appeals. And defendants have rights and privileges that the judges have to respect, and that really is enforced in various ways. And we try to limit the discretion of judges to be biased or play favorites by making the law really clear about what they should do. We don’t do that perfectly but we try and it helps. And when we make laws, they are (or at least should be) pretty generic and based on general principles instead of targeting specific individuals or small groups – we want our laws to be relevant for future generations and different societal situations, rather than overly specific to deal some current events.

stucchio and Mason

stucchio retweeted Mason writing:

"Everything can be free if we fire the people who stop you from stealing stuff" is apparently considered an NPR-worthy political innovation now, rather than the kind of brain fart an undergrad might mumble as they come to from major dental work https://twitter.com/_natalieescobar/status/1299018604327907328

There’s no substantial objective-world content here. Basically “I disagree with whatever is the actual thing behind my straw man characterization”. There’s no topical argument. It’s ~all social posturing. It’s making assertions about who is dumb and who should be associated with what group (and, by implication, with the social status of that group). NPR-worthy, brain fart, undergrad, mumble and being groggy from strong drugs are all social-meaning-charged things to bring up. The overall point is to attack the social status of NPR by associating it with low status stuff. Generally smart people like stuchhio (who remains on the small list of people whose tweets I read – I actually have a pretty high opinion of of him) approve of that tribalist social-political messaging enough to retweet it.

Yudkowsky

Eliezer Yudkowsky wrote on Less Wrong (no link because, contrary to what he says, someone did make the page inaccessible. I have documentation though.):

Post removed from main and discussion on grounds that I've never seen anything voted down that far before. Page will still be accessible to those who know the address.

The context is my 2011 LW post “The Conjunction Fallacy Does Not Exist”.

In RAZ, Yudkowsky repeatedly brings up subculture affiliations he has. He read lots of sci fi. He read 1984. He read Feynman. He also refers to “traditional rationality” which Feynman is a leader of. (Yudkowsky presents several of his ideas as improvements on traditional rationality. I think some of them are good points.) Feynman gets particular emphasis. I think he got some of his fans via this sort of subculture membership signaling and by referencing stuff they like.

I bring this up because Feynman has a book title "What Do You Care What Other People Think?": Further Adventures of a Curious Character. This is the sequel to the better known "Surely You're Joking, Mr. Feynman!": Adventures of a Curious Character.

Yudkowsky evidently does care what people think and has provided no indication that he’s aware that he’s contradicting one of his heroes, Feynman. He certainly doesn’t provide counter arguments to Feynman.

Downvotes are communications about what people think. Downvotes indicate dislike. They are not arguments. They aren’t reasons it’s bad. They’re just opinions. They’re like conclusions or assertions. Yudkowsky openly presents himself as taking action because of what people think. It’s also basically just openly saying “I use power to suppress unpopular ideas”. Yudkowsky also gave no argument himself, nor did he endorse/cite/link any argument he agreed with about the topic.

Yudkowsky is actually reasonably insightful about social hierarchies elsewhere, btw. But this quote shows that, in some major way, he doesn’t understand rationality and social dynamics.

Replies to my “Chains, Bottlenecks and Optimization”

https://www.lesswrong.com/posts/Ze6PqJK2jnwnhcpnb/chains-bottlenecks-and-optimization

Dagon

I think I've given away over 20 copies of _The Goal_ by Goldratt, and recommended it to coworkers hundreds of times.

Objective meaning: I took the specified actions.

Social meaning: I like Goldratt. I’m aligned with him and his tribe. I have known about him for a long time and might merit early adopter credit. Your post didn’t teach me anything. Also, I’m a leader who takes initiative to influence my more sheep-like coworkers. I’m also rich enough to give away 20+ books.

Thanks for the chance to recommend it again - it's much more approachable than _Theory of Constraints_, and is more entertaining, while still conveying enough about his worldview to let you decide if you want the further precision and examples in his other books.

Objective meaning: I recommend The Goal.

Social meaning: I’m an expert judge of which Goldratt books to recommend to people, in what order, for what reasons. Although I’m so clever that I find The Goal a bit shallow, I think it’s good for other people who need to be kept entertained and it has enough serious content for them to get an introduction from. Then they can consider if they are up to the challenge of becoming wise like me, via further study, or not.

This is actually ridiculous. The Goal is the best known Goldratt book, it’s his best seller, it’s meant to be read first, and this is well known. Dagon is pretending to be providing expert judgment, but isn’t providing insight. And The Goal has tons of depth and content, and Dagon is slandering the book by condescending to it in this way. By bringing up Theory of Constraints, Dagon is signaling he reads and values less popular, less entertaining, less approachable non-novel Goldratt books.

It's important to recognize the limits of the chain metaphor - there is variance/uncertainty in the strength of a link (or capacity of a production step), and variance/uncertainty in alternate support for ideas (or alternate production paths).

Objective meaning (up to the dash): Goldratt’s chain idea, which is a major part of your post, is limited.

Social meaning (up to the dash): I’ve surpassed Goldratt and can look down on his stuff as limited. You’re a naive Goldratt newbie who is accepting whatever he says instead of going beyond Goldratt. Also calling chains a “metaphor” instead of “model” is a subtle attack to lower status. Metaphors aren’t heavyweight rationality (while models are, and it actually is a model). Also Dagon is implying that I failed to recognize limits that I should have recognized.

Objective meaning continued: There’s some sort of attempt at an argument here but it doesn’t actually make sense. Saying there is variance in two places is not a limitation of the chain model.

Social meaning continued: saying a bunch of overly wordy stuff that looks technical is bluffing and pretending he’s arguing seriously. Most people won’t know the difference.

Most real-world situations are more of a mesh or a circuit than a linear chain, and the analysis of bottlenecks and risks is a fun multidimensional calculation of forces applies and propagated through multiple links.

Objective meaning: Chains are wrong in most real world situations because those situations are meshes or circuits [both terms undefined]. No details are given about how he knows what’s common in real world situations. And he’s contradicting Goldratt who actually did argue his case and know math. (I also know more than enough math so far and Dagon never continued with enough substance to potentially strain either of our math skills sets).

Social meaning: I have fun doing multidimensional calculations. I’m better than you. If you knew math so well that it’s a fun game to you, maybe you could keep up with me. But if you could do that, you wouldn’t have written the post you wrote.

It’s screwy how Dagon presents himself as a Goldratt superfan expert and then immediately attacks Goldratt’s ideas.

Note: Dagon stopped replying without explanation shortly after this, even though he’d said how super interested in Goldratt stuff he is.

Donald Hobson

I think that ideas can have a bottleneck effect, but that isn't the only effect. Some ideas have disjunctive justifications.

Objective meaning: bottlenecks come up sometimes but not always. [No arguments about how often they come up, how important they are, etc.]

Social meaning: You neglected disjunctions and didn’t see the whole picture. I often run into people who don’t know fancy concepts like “disjunction”.

Note: Disjunction just means “or” and isn’t something that Goldratt or I had failed to consider.

Hobson then follows up with some math, socially implying that the problem is I’m not technical enough and if only I knew some math I’d have reached different conclusions. He postures about how clever he is and brings up resistors and science as brags.

I responded, including with math, and then Hobson did not respond.

TAG

What does that even mean?

Objective meaning: I don’t understand what you wrote.

Social meaning: You’re not making sense.

He did give more info about what his question was after this. But he led with this, on purpose. The “even” is a social attack – that word isn’t there to help with any objective meaning. It’s there to socially communicate that I’m surprisingly incoherent. It’d be a subtle social attack even without the “even”. He didn’t respond when I answered his question.

abramdemski

There is another case which your argument neglects, which can make weakest-link reasoning highly inaccurate, and which is less of a special case than a tie in link-strength.

Objective meaning: The argument in the OP is incomplete.

Social meaning: You missed something huge, which is not a special case, so your reasoning is highly inaccurate.

The way you are reasoning about systems of interconnected ideas is conjunctive: every individual thing needs to be true.

Objective meaning: Chain links have an “and” relationship.

Social meaning: You lack a basic understanding of the stuff you just said, so I’ll have to start really basic to try to educate you.

But some things are disjunctive: some one thing needs to be true.

Objective meaning: “or” exists. [no statement yet about how this is relevant]

Social meaning: You’re wrong because you’re an ignorant novice.

(Of course there are even more exotic logical connectives, such as implication or XOR, which are also used in everyday reasoning. But for now it will do to consider only conjunction and disjunction.)

Objective meaning: Other logic operators exist [no statement yet about how this is relevant].

Social meaning: I know about this like XOR, but you’re a beginner who doesn’t. I’ll let you save face a little by calling it “exotic”, but actually, in the eyes of everyone knowledgeable here, I’m insulting you by suggesting that for you XOR is exotic.

Note: He’s wrong, I know what XOR is (let alone OR). So did Goldratt. XOR is actually easy for me, and I’ve used it a lot and done much more advanced things too. He assumed I didn’t in order to socially attack me. He didn’t have adequate evidence to reach the conclusion that he reached; but by reaching it and speaking condescendingly, he implied that there was adequate evidence to judge me as an ignorant fool.

Perhaps the excess accuracy in probability theory makes it more powerful than necessary to do its job? Perhaps this helps it deal with variance? Perhaps it helps the idea apply for other jobs than the one it was meant for?

Objective meaning: Bringing up possibilities he thinks are worth considering.

Social meaning: Flaming me with some rather thin plausible deniability.

I skipped the middle of his post btw, which had other bad stuff.

johnswentworth

I really like what this post is trying to do. The idea is a valuable one. But this explanation could use some work - not just because inferential distances are large, but because the presentation itself is too abstract to clearly communicate the intended point. In particular, I'd strongly recommend walking through at least 2-3 concrete examples of bottlenecks in ideas.

This is an apparently friendly reply but he was lying. I wrote examples but he wouldn’t speak again.

There are hints in this text that he actually dislikes me and is being condescending, and that the praise in the first two sentences is fake. You can see some condescension in the post, e.g. in how he sets himself up like a mentor telling me what to do (and note the unnecessary “strongly” before “recommend”. And how does he know the idea is valuable when it’s not clearly communicated? And his denial re inferential distance is actually both unreasonable and aggressive. The “too abstract” and “could use some work” are also social attacks, and the “at least 2-3” is a social attack (it means do a lot) with a confused objective meaning (if you’re saying do >= X, why specify X as a range? you only need one number.)

The objective world meaning is roughly that he’s helping with some presentation and communication issues and wants a discussion of the great ideas. But it turns out, as we see from his following behavior, that wasn’t true. (Probably. Maybe he didn’t follow up for some other reason like he died of COVID. Well not that because you can check his posting history and see he’s still posting in other topics. But maybe he has Alzheimer’s and he forgot, and he knows that’s a risk so he keeps notes about stuff he wants to follow up on, but he had an iCloud syncing error and the note got deleted without him realizing it. There are other stories that I don’t have enough information to rule out, but I do have broad societal information about them being uncommon, and there are patterns across the behavior of many people.)

MakoYass

I posted in comments on different Less Wrong thread:

curi:

Are you interested in extended discussion about this, with a goal of reaching some conclusions about CR/LW differences, or do you know anyone who is?

MakoYass:

I am evidently interested in discussing it, but I am probably not the best person for it.

Objective meaning: I am interested. My answer to your question is “yes”. I have agreed to try to have a discussion, if you want to. However, be warned that I’m not very good at this.

Social meaning: The answer to your question is “no”. I won’t discuss with you. However, I’m not OK with being declared uninterested in this topic. I love this topic. How dare you even question my interest when you have evidence (“evidently”) that I am interested, which consists of me having posted about it. I’d have been dumb to post about something I’m not interested in, and you were an asshole to suggest I might be dumb like that.

Actual result: I replied in a friendly, accessible way attempting to begin a conversation, but he did not respond.

Concluding Thoughts

Conversations don’t go well when a substantial portion of what people say has a hostile (or even just significantly different) social (double) meaning.

It’s much worse when the social meaning is the primary thing people are talking about, as in all the LW replies I got above. It’s hard to get discussions where the objective meanings are more emphasized than the social ones. And all the replies I quoted re my Chains and Bottlenecks post were top level replies to my impersonal article. I hadn’t said anything to personally offend any of those people, but they all responded with social nastiness. (Those were all the top level replies. There were no decent ones.) Also it was my first post back after 3 years, so this wasn’t carrying over from prior discussion (afaik – possibly some of them were around years ago and remembered me. I know some people do remember me but they mentioned it. Actually TAG said later, elsewhere, to someone else, that he knew about me from being on unspecified Critical Rationalist forums in the past).

Even if you’re aware of social meanings, there are important objective meanings which are quite hard to say without getting offensive social meaning. This comes up with talking about errors people make, especially ones that reveal significant weaknesses in their knowledge. Talking objectively about methodology errors and what to do about them can also be highly offensive socially. Also objective, argued judgments of how good things are can be socially offensive, even if correct (actually it’s often worse if it’s correct and high quality – the harder to plausibly argue back, the worse it can be for the guy who’s wrong).

The main point was to give examples of how the same sentence can be read with an objective and a social meaning. This is what most discussions on rationalist forums where explicit knowledge of social status hierarchies is common look like to me. It comes up a fair amount on my own forums too (less often than at LW, but it’s a pretty big problem IMO).

Note: The examples in this post are not representative of the full spectrum of social behaviors. One of the many things missing is needy/chasing/reactive behavior where people signal their own low social status (low relative to the person they’re trying to please). Also, I could go into more detail on any particular example people want to discuss (this post isn’t meant as giving all the info/analysis, it’s a hybrid between some summary and some detail).

---

Update: Adding (on same day as original) a few things I forgot to say.

Audiences pick up on some of the social meanings (which ones, and how they see them, varies by person). They see you answer and not answer things. They think some should be answered and some are ignorable. They take some things as social answers that aren’t intended to be. They sometimes ignore literal/objective meanings of things. They judge. It affects audience reactions. And the perception of audience reactions affects what the actual participants do and say (including when they stop talking without explanation).

The people quoted could do less social meaning. They’re all amplifying the social. There’s some design there; it’s not an accident. It’s not that hard to be less social. But even if you try, it’s very hard to avoid any problematic social meanings, especially when you consider that different audience members will read stuff differently, according to different background knowledge, different assumptions about context, different misreadings and skipped words, etc.

I posted this on Less Wrong 10 days ago. At the end, I summarize the answer they gave.

---

What counts as a hypothesis for Solomonoff induction? The general impression I’ve gotten in various places is “a hypothesis can be anything (that you could write down)”. But I don’t think that’s quite it. E.g. evidence can be written down but is treated separately. I think a hypothesis is more like a computer program that outputs predictions about what evidence will or will not be observed.

If X and Y are hypotheses, then is “X and Y” a hypothesis? “not X”? “X or Y?” If not, why not, and where can I read a clear explanation of the rules and exclusions for Solomonoff hypotheses?

If using logic operators with hypotheses does yield other hypotheses, then I’m curious about a potential problem. When hypotheses are related, we can consider what their probabilities should be in more than one way. The results should always be consistent.

For example, suppose you have no evidence yet. And suppose X and Y are independent. Then you can calculate the probability of P(X or Y) in terms of the probability of P(X) and P(Y). You can also calculate the probability of all three based on their length (that’s the Solomonoff prior). These should always match but I don’t think they do.

The non-normalized probability of X is 1/2^len(X).

So you get:

P(X or Y) = 1/2^len(X) + 1/2^len(Y) - 1/2^(len(X)+len(Y))

and we also know:

P(X or Y) = 1/2^len(X or Y)

since the left hand sides are the same, that means the right hand sides should be equal, by simple substitution:

1/2^len(X or Y) = 1/2^len(X) + 1/2^len(Y) - 1/2^(len(X)+len(Y))

Which has to hold for any X and Y.

We can select X and Y to be the same length and to minimize compression gains when they’re both present, so len(X or Y) should be approximately 2len(X). I’m assuming a basis, or choice of X and Y, such that “or” is very cheap relative to X and Y, hence I approximated it to zero. Then we have:

1/2^2len(X) = 1/2^len(X) + 1/2^len(X) - 1/2^2len(X)

which simplifies to:

1/2^2len(X) = 1/2^len(X)

Which is false (since len(X) isn’t 0). And using a different approximation of len(X or Y) like 1.5len(X), 2.5len(X) or even len(X) wouldn’t make the math work.

So Solomonoff induction is inconsistent. So I assume there’s something I don’t know. What? (My best guess so far, mentioned above, is limits on what is a hypothesis.)

Also here’s a quick intuitive explanation to help explain what’s going on with the math: P(X) is both shorter and less probable than P(X or Y). Think about what you’re doing when you craft a hypotheses. You can add bits (length) to a hypothesis to exclude stuff. In that case, more bits (more length) means lower prior probability, and that makes sense, because the hypothesis is compatible with fewer things from the set of all logically possible things. But you can also add bits (length) to a hypothesis to add alternatives. It could be this or that or a third thing. That makes hypotheses longer but more likely rather than less likely. Also, speaking more generally, the Solomonoff prior probabilities are assigned according to length with no regard for consistency amongst themselves, so its unsurprising that they’re inconsistent unless the hypotheses are limited in such a way that they have no significant relationships with each other that would have to be consistent, which sounds hard to achieve and I haven’t seen any rules specified for achieving that (note that there are other ways to find relationships between hypotheses besides the one I used above, e.g. looking for subsets).

---

Less Wrong's answer, in my understanding, is that in Solomonoff Induction a "hypothesis" must make positive predictions like "X will happen". Probabilistic positive predictions – assigning probabilities to different specific outcomes – can also work. Saying X or Y will happen is not a valid hypothesis, nor is saying X won't happen.

This is a very standard trick by so-called scholars. They take a regular English word (here "hypothesis") and define it as a technical term with a drastically different meaning. This isn't clearly explained anywhere and lots of people are misled. It's also done with e.g. "heritability".

Solomonoff Induction is just sequence prediction. Take a data sequence as input, then predict the next thing in the sequence via some algorithm. (And do it with all the algorithms and see which do better and are shorter.) It's aspiring to be the oracle in The Fabric of Reality but worse.

Eliezer Yudkowsky tweeted:

EY:

What on Earth is up with the people replying "billionaires don't have real money, just stocks they can't easily sell" to the anti-billionaire stuff? It's an insanely straw reply and there are much much better replies.

DI:

What would be a much better reply to give to someone who thinks for example that Elon Musk is hoarding $100bn in his bank account?

EY:

A better reply should address the core issue whether there is net social good from saying billionaires can't have or keep wealth: eg demotivating next Steves from creating Apple, no Gates vaccine funding, Musk not doing Tesla after selling Paypal.

Eliezer Yudkowsky (EY) frequently brings up names (e.g. Feynman or Jaynes) of smart people involved with science, rationality or sci-fi. He does this throughout RAZ. He communicates that he's read them, he's well-read, he's learned from them, he has intelligent commentary related to stuff they wrote, etc. He presents himself as someone who can report to you, his reader, about what those books and people are like. (He mostly brings up people he likes, but he also sometimes presents himself as knowledgeable about people he's unfriendly to, like Karl Popper and Ayn Rand, who he knows little about and misrepresents.)

EY is a liar who can't be trusted. In his tweets, he reveals that he brings up names while knowing basically nothing about them.

Steve Jobs and Steve Wozniak were not motivated by getting super rich. Their personalities are pretty well known. I guess EY never read any of the biographies and hasn't had conversations about them with knowledgeable people. Or maybe he doesn't connect what he reads to what he says. (I provide some brief, example evidence at the end of this post in which Jobs tells Ellison "You don’t need any more money." EY is really blatantly wrong.)

EY brings up Jobs and Wozniak ("Steves") to make his assertions sound concrete, empirical and connected to reality. Actually he's doing egregious armchair philosophizing and using counter examples as examples.

Someone who does this can't be trusted whenever they bring up other names either. It shows a policy of dishonesty: either carelessness and incompetence (while dishonestly presenting himself as a careful, effective thinker) or outright lying about his knowledge.

There are other problems with the tweets, too. For example, EY is calling people insane instead of arguing his case. And EY is straw manning the argument about billionaires having stocks not cash – while complaining about others straw manning. Billionaires have most of their wealth in capital goods, not consumption goods (that's the short, better version of the argument he mangled), and that's a more important issue than the incentives that EY brings up. EY also routinely presents himself as well-versed in economics but seems unable to connect concepts like accumulation of capital increasing the productivity of labor, or eating the seed corn, to this topic.

Some people think billionaires consume huge amounts of wealth – e.g. billions of dollars per year – in the form of luxuries or other consumption goods. Responding to a range of anti-billionaire viewpoints, including that one, by saying basically "They need all that money so they're incentivized to build companies." is horribly wrong. They don't consume anywhere near that much wealth per year. EY comes off as justifying them doing something they don't do that would actually merit concern if they somehow did it.

If Jeff Bezos were building a million statues of himself, that'd be spending billions of dollars on luxuries/consumption instead of production. That'd actually somewhat harm our society's capital accumulation and would merit some concern and consideration. But – crucial fact – the real world looks nothing like that. EY sounds like he's conceding that that's actually happening instead of correcting people about reality, and he's also claiming it's obviously fine because rich people love their statues, yachts and sushi so much that it's what inspires them to make companies. (It's debateable, and there are upsides, but it's not obviously fine.)

---

Steve Jobs is the authorized biography by Walter Isaacson. It says (context: Steve didn't want to do a hostile takeover of Apple) (my italics):

“You know, Larry [Ellison], I think I’ve found a way for me to get back into Apple and get control of it without you having to buy it,” Jobs said as they walked along the shore. Ellison recalled, “He explained his strategy, which was getting Apple to buy NeXT, then he would go on the board and be one step away from being CEO.” Ellison thought that Jobs was missing a key point. “But Steve, there’s one thing I don’t understand,” he said. “If we don’t buy the company, how can we make any money?” It was a reminder of how different their desires were. Jobs put his hand on Ellison’s left shoulder, pulled him so close that their noses almost touched, and said, “Larry, this is why it’s really important that I’m your friend. You don’t need any more money.”

Ellison recalled that his own answer was almost a whine: “Well, I may not need the money, but why should some fund manager at Fidelity get the money? Why should someone else get it? Why shouldn’t it be us?”

“I think if I went back to Apple, and I didn’t own any of Apple, and you didn’t own any of Apple, I’d have the moral high ground,” Jobs replied.

“Steve, that’s really expensive real estate, this moral high ground,” said Ellison. “Look, Steve, you’re my best friend, and Apple is your company. I’ll do whatever you want.”

(Note that Ellison, too, despite having a more money-desiring attitude, didn't actually prioritize money. He might be the richest man in the world today if he'd invested heavily in Steve Jobs' Apple, but he put friendship first.)

habryka wrote about why LW banned me. This is habryka’s full text plus my comments:

Today we have banned two users, curi and Periergo from LessWrong for two years each. The reasoning for both is bit entangled but are overall almost completely separate, so let me go individually:

The ban isn’t for two years. It’s from Sept 16 2020 through Dec 31 2022.

They didn’t bother to notify me. I found out in the following way:

First, I saw I was logged out. Then I tried to log back in and it said my password was wrong. Then I tried to reset my password. When submitting a new password, it then gave an error message saying I was banned and until what date. Then I messaged them on intercom and 6 hours later they gave me a link to the public announcement about my ban.

That’s a poor user experience.

Periergo is an account that is pretty easily traceable to a person that Curi has been in conflict with for a long time, and who seems to have signed up with the primary purpose of attacking curi. I don't think there is anything fundamentally wrong about signing up to LessWrong to warn other users of the potentially bad behavior of an existing user on some other part of the internet, but I do think it should be done transparently.

It also appears to be the case that he has done a bunch of things that go beyond merely warning others (like mailbombing curi, i.e. signing him up for tons of email spam that he didn't sign up for, and lots of sockpupetting on forums that curi frequents), and that seem better classified as harassment, and overall it seemed to me that this isn't the right place for Periergo.

Periergo is a sock puppet of Andy B. Andy harassed FI long term with many false identities, but left for months when I caught him, connected the identities, and blogged it. But he came back in August 2020 and has written over 100 comments since returning, and he made a fresh account on Less Wrong for the purpose of harassing me and disrupting my discussions there. He essentially got away with it. He stirred up trouble and now I’m banned. What does he care that his fresh sock puppet, with a name he’ll likely never use again anywhere, is banned? And he’ll be unbanned at the same time as me in case he wants to further torment me using the same account.

Curi has been a user on LessWrong for a long time, and has made many posts and comments. He also has the dubious honor of being by far the most downvoted account in all of LessWrong history at -675 karma.

I started at around -775 karma when I returned to Less Wrong recently and went up. I originally debated Popper, induction and cognitive biases at LW around 9 years ago and got lots of downvotes. I returned around 3 years ago when an LW moderator invited me back because he liked my Paths Forward article. That didn’t work out and I left again. I returned recently for my own reasons, instead of because someone incorrectly suggested that I was wanted, and it was going better. I knew some things to expect, and some things that wouldn’t work, and I'd just read LW's favorite literature, RAZ.

BTW, I don’t know how my karma is being calculated. My previous LW discussions were at the 1.0 version of the site where votes on posts counted for 10 karma, and votes on comments counted for 1 karma. When I went back the second time, a moderator boosted my karma enough to be positive so that I could write posts instead of just comments. LW 2.0 allows you to write posts while having negative karma and votes on posts and comments are worth the same amount, but your votes count for multiple karma if you have high karma and/or use the strong vote feature. I don’t know how old stuff got recalculated when they did the version 2.0 website.

Overall I have around negative 1 karma per comment, so that’s … not all that bad? Or apparently the lowest ever. If downvotes on the old posts still count 10x then hundreds of my negative karma is from just a few posts.

In general, I think outliers should be viewed as notable and potentially valuable, especially outliers that you can already see might actually be good (as habryka says about me below). Positive outliers are extremely valuable.

The biggest problem with his participation is that he has a history of dragging people into discussions that drag on for an incredibly long time, without seeming particularly productive, while also having a history of pretty aggressively attacking people who stop responding to him. On his blog, he and others maintain a long list of people who engaged with him and others in the Critical Rationalist community, but then stopped, in a way that is very hard to read as anything but a public attack. It's first sentence is "This is a list of ppl who had discussion contact with FI and then quit/evaded/lied/etc.", and in-particular the framing of "quit/evaded/lied" sure sets the framing for the rest of the post as a kind of "wall of shame".

I consider it strange to ban me for stuff I did in the distant past but was not banned for at the time.

I find it especially strange to ban me for 2 years over stuff that’s already 3 or 9 years old (the evaders guest post by Alan is a year old, and btw "evade" is standard Objectivist philosophy terminology). I already left the site for longer than the ban period. Why is a 5 year break the right amount instead of 3? habryka says below that he thinks I was doing better (from his point of view and regarding what the LW site wants) this time.

They could have asked me about that particular post before banning me, but didn’t. They also could have noted that it’s an old post that only came up because Andy linked it twice on LW with the goal of alienating people from me. They’re letting him get what he wanted even though they know he was posting in bad faith and breaking their written rules.

I, by contrast, am not accused of breaking any specific written rule that LW has, but I’ve been banned anyway with no warning.

Those three things in combination, a propensity for long unproductive discussions, a history of threats against people who engage with him, and being the historically most downvoted account in LessWrong history, make me overall think it's better for curi to find other places as potential discussion venues.

I didn’t threaten anyone. I’m guessing it was a careless wording. I think habryka should retract or clarify it. Above habryka used “attack[]” as a synonym for criticize. I don’t like that but it’s pretty standard language. But I don’t think using “threat[en]” as a synonym for criticize is reasonable.

“threaten” has meanings like “state one's intention to take hostile action against someone in retribution for something done or not done” and “express one's intention to harm or kill“ (New Oxford Dictionary). This is the thing in the post that I most strongly object to.

I do really want to make clear that this is not a personal judgement of curi. While I do find the "List of Fallible Ideas Evaders" post pretty tasteless, and don't like discussing things with him particularly much, he seems well-intentioned, and it's quite plausible that he could me an amazing contributor to other online forums and communities. Many of the things he is building over on his blog seem pretty cool to me, and I don't want others to update on this as being much evidence about whether it makes sense to have curi in their communities.

I do also think his most recent series of posts and comments is overall much less bad than the posts and comments he posted a few years ago (where most of his negative karma comes from), but they still don't strike me as great contributions to the LessWrong canon, are all low-karma, and I assign too high of a probability that old patterns will repeat themselves (and also that his presence will generally make people averse to be around, because of those past patterns). He has also explicitly written a post in which he updates his LW commenting policy towards something less demanding, and I do think that was the right move, but I don't think it's enough to tip the scales on this issue.

So I came back after 3 years, posted in a way they liked significantly better … I’m building cool things and plausibly amazing while also making major progress at compatibility with LW … but they’re banning me anyway, even though my old posts didn’t get me banned.

More broadly, LessWrong has seen a pretty significant growth of new users in the past few months, mostly driven by interest in Coronavirus discussion and the discussion we hosted on GPT3. I continue to think that "Well-Kept Gardens Die By Pacifism", and that it is essential for us to be very careful with handling that growth, and to generally err on the side of curating our userbase pretty heavily and maintaining high standards. This means making difficult moderation decision long before it is proven "beyond a reasonable doubt" that someone is not a net-positive contributor to the site.

In this case, I think it is definitely not proven beyond a reasonable doubt that curi is overall net-negative for the site, and banning him might well be a mistake, but I think the probabilities weigh heavily enough in favor of the net-negative, and the worst-case outcomes are bad-enough, that on-net I think this is the right choice.

I don’t see why they couldn’t wait for me to do something wrong to ban me, or give me any warning or guidance about what they wanted me to do differently. I doubt this would have happened this way if Andy hadn’t done targeted harassment.

At least they wrote about their reasons. I appreciate that they’re more transparent than most forums.

In another message, habryka clarified his comment about others not updating their views of me based on this ban:

The key thing I wanted to communicate is that it seems quite plausible to me that these patterns are the result of curi interfacing specifically with the LessWrong culture in unhealthy ways. I can imagine him interfacing with other cultures with much less bad results.

I also said "I don't want others to think this is much evidence", not "this is no evidence". Of course it is some evidence, but I think overall I would expect people to update a bit too much on this, and as I said, I wouldn't be very surprised to see curi participate well in other online communities.

I’m unclear on what aspect of LW culture that I’m a mismatch for. Or put another way: I may interface better with other cultures which have or lack what particular characteristics compared to LW?

---

Also, LW didn't explain how they decided on ban lengths. 2.3 year bans don't correspond to solving the problems raised. Andy or I could easily wait and then do the stuff LW doesn't want. They aren't asking us to do anything to improve or to provide any evidence that we've reformed in some way. Nor are they asking us to figure out how we can address their concerns and prevent bad outcomes. They're just asking us to wait and, I guess, counting on us not to hold grudges. Problems don't automatically go away due to time passing.

Overall, I think LW’s decision and reasoning are pretty bad but not super unreasonable compared to the general state of our culture. I wouldn’t expect better at most forums and I’ve seen much worse. Also, I’m not confident that the reasoning given fully and accurately represents the actual reasons. I'm not convinced that they will ban other people using the same reasoning like that they didn't break any particular rules but might be a net-negative for the site, especially considering that "the moderators of LW are the opposite of trigger-happy. Not counting spam, there is on average less than one account per year banned." (source from 2016, maybe they're more trigger-happy in 2020, I don't know).