Context/background

(You don't need to read this section, you can just skip to the resolution criteria.)

I was reading Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?, and saw this section:

"Human Content is Unimportant Compared to the Right Architecture"

A topic that comes up over and over again over the course of the debate -- particularly later, though -- is how important the prior "content" of all prior human civilization might be.

That is, consider of all the explicit knowledge encoded in all the books humans have written. Consider also all the implicit knowledge encoded in human praxis and tradition: how to swing an axe to cut down a tree, how to run a large team of AI science researchers, how to navigate different desired levels of kitchen cleanliness among roommates, how to use an arc-welder, how to calm a crying baby, and so on forever. Consider also all the content encoded not even in anyone's brains, but in the economic and social relationships without which society does not function.

How important is this kind of "content"?

...

Yudkowsky thinks that with the right architecture you can just skip over a lot of human content.

...

In 2017 AlphaGoZero was released, which was able to learn Go at a superhuman level without learning from any human games at all. Yudkowsky then explained how this was evidence for his position:

If we round off Hanson's position to "content from humans is likely to matter a lot" and Yudkowsky's to "human content is crap," then I think that AlphaGoZero is some level of evidence in support of Yudkowsky's view. (Although Hanson responded by saying it was a very small piece of evidence, because his view always permitted narrow tools to make quick progress without content, and AGZ is certainly a narrow tool.)

On the other hand, is it the only piece of evidence reality gives us on this matter? Is it the most important?

One additional piece of data is that some subsequent developments of more complex game-playing AI have not been able to discard human data. Neither DeepMind's StarCraft II, nor OpenAI's Dota2 playing agents -- both post Go-playing AIs -- were able to train without being jumpstarted by human data. Starcraft II and Dota2 are far more like the world than Go -- they involve partial information, randomness, and much more complex ontologies. So this might be an iota of evidence for something like a Hansonian view.

But far more importantly, and even further in the same direction -- non-narrow tools like GPT-4 are generally trained by dumping a significant fraction of all written human content into them. Training them well currently relies in part on mildly druidical knowledge about the right percent of the different parts of human content to dump into them -- should you have 5% code or 15% code? Multilingual or not? More ArXiV or more Stack overflow? There is reasonable speculation that we will run out of sufficient high-quality human content to feed these systems. The recent PaLM-2 paper has 18 authors for the data section -- more than it has for the architecture section! (Although both have fewer than the infrastructure section gets, of course -- how to employ compute still remains big.) So content is hugely important for LLMs.

Given that GPT-4 and similar programs look to be by far the most generally intelligent AI entities in the real world rather than a game world yet made, it's hard for me to see this as anything other than some evidence that content in Hanson's sense might matter a lot. If LLMs matter more for future general intelligence than AlphaGoZero -- which is a genuinely uncertain "if" for me -- then Hanson probably gets some fractional number of Bayes points over Yudkowsky. If not, maybe the reverse?

I don't think the predictions are remotely clear enough for either person to claim reality as on their side.

If interpreted solely as a claim about the current state of the models, I don't disagree with this resolution.

But it seems to me that fundamentally, there is lots of space for algorithm improvements that could let an AI develop its own content from scratch, and that this will probably be developed once we exhaust all human content.

The main doubt I can see is, maybe it will always be more economical to re-use tons of human content, and rarely economical to develop it on one's own. I'm not sure this is the case though as certain self-supervised algorithms might benefit from observing things on their own.

Other doubts people may propose but which I don't really buy:

• Maybe it is never cost-effective to dump all content into a single form in a single system, and instead we will only use narrow specialized AI systems.

• Maybe it is fundamentally impossible for a single artificial intelligence to develop content in the style of human civilization, because human brains have some ?magic sauce? that cannot be replicated artificially.

I would be open to hearing other options.

Resolution criteria

Market resolves YES if someone develops an AI that learns information and skills on a similar scale and across similar topics and to similar usefulness as GPT-4 can, but through the AI itself experimenting, observing, deducing, and similar, rather than by mimicking humans.

Market resolves NO if we reach 2030 before any such AI has been revealed.