SI murders people. a lot.

Date: 2020-09-09

This was first a post to FI, then posted here.

Looking for feedback; thinking of posting to LW. I’d probably change some bits so I didn’t sound as much like an outsider.

Posters on LW often bring up the idea of emulating a person to establish some bounds on complexity of SI programs.

The problem with this idea is that an ~infinite number of people are murdered before SI finds a good answer.

Here’s the common logic:

Peoples’ minds take lots of space to specify.
People can do knowledge-y stuff: solve problems, make predictions, etc.
SI wants to do some of the things that people do
Minds can be represented as classical programs
Therefore a capable mind is an upper limit on program size

I add:

Minds take lots of space, and the important parts of the mind grow in length as the mind grows in knowledge.

I note: I’ve had the goal-posts of the word ‘simple’ changed on me in curi’s code/english thread with:

When I say the ‘sense of simplicity of SI’, I use ‘simple program’ to mean the programs that SI gives the highest weight to in its predictions(these will by definition be the shortest programs that haven’t been ruled out by data).

I’ll use “SI-simple” to label that idea going forward

My further reasoning:

SI-simple minds are more common. if nothing else, babies are SI-simpler minds. Their brains might have more neurons (than adults), but those brains are also easier to specify b/c they’re more generic.
As we grow and learn our brains get more complex (and by ‘complex’ i mean both: not simple and not SI-simple).
SI doesn’t throw programs out too early, esp when they’re long, so if minds are emulated they’re run for enough time to start thinking.
SI has no problem killing the person in search of a better program (in fact, it’s necessary).

I predict the common response will be like:

sure, but that doesn’t matter because SI will find other programs first. we don’t expect we need programs this long, it’s just to find an upper limit.

I assume that LW people agree that killing a mind is murder (i.e. non-humans count), and that SI programs can be minds, and turning one off is killing it.

(If they disagreed with any of that I would be much more worried about their AGI pursuits. I assume ppl here agree with it too)

I think this is problematic b/c:

a) there’s a known lesser limit (murder) that has not been said “out loud” as far as I know

b) we don’t know how large the SI-simplest mind is, or how to reason about these as SI programs

c) if we continue to not be able to reason about it, but also put serious resources behind SI research, then it’s possible “we” (humanity) start generating and killing AGIs before realising it

d) if there is a better way to do minds and AGI than SI, then SI would find that method along the way, but also murder lots of new AGI people

e) if we keep not understanding AGI and SI keeps getting more funding: someone might actually try something of the complexity necessary to do this randomly. if they optimise with stuff like evolutionary algorithms (esp if there are some good developments in related areas), then it gets easier to end up with these sorts of programs

f) SI is basically guaranteed to do this if it’s used to try and find programs which are also AGIs.

Now, I don’t think it’s likely some horrible future like this will come to pass. It’s not serious in that sense.

However, I do think it’s serious for fans of SI. It’s a new, important upper limit that they haven’t mentioned before, and it rules out certain programs.

Crucially, we can’t say all the problems we care about will have SI-simpler solutions than AGIs. English (or w/e) words represented as code - generally - has an upper limit in the range of AGIs b/c maybe an AGI + english words are the SI-simplest way to write down some programs. We can’t know what the crossover point is for SI-simple programs to go from non-AGI to AGI… unless we have a theory of AGI. So, whatever understanding of AGI we get, we should not do it incrementally if that means making lots of incremental steps through a program-space “full” of AGIs.

By the above logic I think an important conclusion is that one of the two following ideas has to be true. Option 1: AGI researchers (as humans) are/will be mass-murdering AGIs one day - if humans work like SI does. Or, option 2: there’s some other way to learn stuff / create knowledge - that’s what the AGI researchers are doing - and it must be SI-simpler than an SI-AGI would be. If option 2 is correct, but it is physically possible to make meaningful progress towards AGI via SI, then we’re basically guaranteed to murder lots of AGIs if ppl pursue that avenue long enough (and with enough compute resources).

As far as the idea of a “moral proof” goes, I think this fits the bill. I don’t think AGI research requires the murder of AGIs.

You can leave a comment anonymously. No sign up or login is required. Use a junk email if not your own; email is only for notifications—though, FYI, I will be able to see it.