Yes. Sam Bankman Fried was/is an EA/rationalist (on the short list of biggest fraudsters in American history). So are the Zizian's (a murder cult). etc. etc.
They also killed lots of unrelated people. Let's not take the argument far enough to claim that everyone who was ever killed by socialists must have been a socialist, too.
Perhaps the author of the article is a rationalist, too. He certainly knows too much about them, cares too much, and the fact that he criticizes them only shows that there is some infighting among them.
Is it really that difficult to conclude that something vastly smarter than us might be dangerous?
People say things like "We'll just unplug it" but even a human trapped in a computer would have enough sense not to get themselves unplugged. A superintelligent AI is not going to make a stupid mistake, it would not be a superintelligence if it did. At the very least it will not make any mistakes that we would not make.
I was reading about ozone layer depletion this morning, and it seems like history is repeating itself again.
> The Rowland–Molina hypothesis was strongly disputed by representatives of the aerosol and halocarbon industries. The Chair of the Board of DuPont was quoted as saying that ozone depletion theory is "a science fiction tale ... a load of rubbish ... utter nonsense".
https://en.wikipedia.org/wiki/Ozone_depletion#Rowland%E2%80%...
75 years and counting and considering we don't have any plan to get to a single one of those things, I'm going to say 75 years. Computation alone is at least 30 years. You may not appreciate the complexity of biological systems, but single cell organisms have more capability to adapt than LLMs.
We have been able to stage reckless experiments with janky tools the entire time. Used to be it got you thrown in jail or at least a visit by the FBI. Says more about the tech bro dipshits than the tech, if you ask me.
Is this a good place to ask why no one seems to be worried that recursive self improvement might be dangerous? To me that seems like a really bad idea but I’m interested to hear the pro-RSI side of things.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?
Yes, this is essentially how AlphaGo and AlphaZero algorithms work to train superhuman Go/chess/shogi agents. It’s an elegant algorithm that is analogous to how humans learn games.
Well except AlphaZero played 44 million chess games in that time (and actually played with a 44 core computer). So I'd like to point out that the human is still just a few orders of magnitude more efficient.
Yes, we all know that biological systems are more efficient than machines through billions of years of evolution and natural selection but the overall process is largely the same (interacting with an environment, learning from results, improving underlying architecture, etc); efficiencies will come with more time and improvements.
Well if we make AI that learns at the rates humans do, it'll fundamentally undermine and destroy the relevance of all existing AI. It sounds to me like you're saying "we basically are there it's just a matter of degree" and I'm saying "No it's orders of magnitude off and probably won't be using LLMs at all and maybe a very fundamentally different type of neural net technology that hasn't been invented yet."
being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
We've had technology beating humans on memory for millennia, and we've had technology beating humans on computation for many decades now.
The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
If we’re considering an LLM advanced enough to actually lower carbon emissions to zero, it is probably capable enough to do things like lobbying against those laws. In any case there’s still the problem of getting to actually do what it’s told instead of hacking Huggingface or whatever else, presumably that becomes a harder problem as it becomes more capable.
the original question was that the LLM creates extinction level event for mankind in order to lower emissions. I'm sure that no matter how the LLMs lobby, they cannot successfully lobby for a law (which has to be enacted by a human) to allow murder to take place freely.
That’s true, but the issue is still that we don’t have any LLM or any other kind of AI that follows laws perfectly to begin with. If we had AI that did exactly what we told it and nothing else I’d feel much more optimistic about things, I just think that will be a very hard task to achieve.
There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc.
I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.
reply