One thing that I find is that it doesn't seem to grasp levels of jargon-use. Like I ask a basic question, okay, a few questions later, suddenly there's abbreviations and weird formulations everywhere.
yeah you know, that concept we've never been able to define using language, making heavy use of the human experience which can also not be captured in language (proof: how bad LLMs are at poetry)
the question is, when comparing a human and a large language model, whether the intellect (that cannot be captured in language) is different from anything the language model can actually do (e.g. language)
the answer to this seems quite obvious to me, and I would actually posit that the onus is on the other side, to prove they are even remotely similar
maybe people think that the voice in their heads is what is doing the thinking? is that the confusion here?
Do you think the LLM is the Chain of Thought? Did you also get confused by the name? Because, much like humans, the CoT is a tool to narrativize and maintain internal coherence. The actual thinking happens invisibly, in the forward pass. Just like...
you forget the bar was moved very far down with all the slop we're experiencing with images, music and especially code
you can't claim "you keep moving the bar", if the very first thing when an LLM drooled out a piece of code, was to proclaim "this is good enough cause it gets the job done" followed by a barrage of "we're not quite there yet but exponentials or something, so very soon it'll be incredible"
yeah, from there it sure looks like "moving up the bar"
But we can agree that writing with such disdain about people doing literally anything else, is perhaps a bit of a bad look.
Especially if it comes from Carmack, a guy who doesn't need to pay rent any more and can in fact live the rest of his life without concern for producing value, etc etc. He is currently doing exactly what he wants, just because he wants it. And that happens to be VR stuff, currently, afaik. And that's his choice. So when he speaks deeply condescending about other people doing exactly what they want ("retro computing is so delightful") it definitely comes from a different place as "hey I just want to pay my rent".
that was truly, deeply, unnecessarily condescending of him.
it's so full of self-importance, even if he truly believes that whatever code he's writing is more important than anything else, it's really an unnecessary dig
especially coming from some guy who by many people is mostly known for his accomplishments in an era that many people would call retro computing now
sure I know this guy is almost literally the definition of the 10x programmer and he can choose to work on whatever he wants, so whatever he decided to work on is probably the most important thing to him, and if that's VR, that's ... also a choice.
doesn't mean he should be this condescending about what other people are doing
> Intentionally doing this kind of hack would be a serious felony. I don't think it's plausible that the leaders of a major business would (...)
Fair enough, I can see that line of reasoning. But if they really didn't want it, it really doesn't show from the security of their training environment setup.
And yes I know they used zero days and did all sorts of complicated unexpected stuff, but that doesn't take away that there were also some huge gaps and severe lack of oversight.
I can see the argument they wouldn't want to commit a felony intentionally, but I also don't see them trying very hard to not commit a felony unintentionally. If you get what I mean.
I guess I also agree that it just makes no sense on any level, regardless of what I think of OpenAI -- but that also makes me not wanna trust them very much.
Ahh yes I remember the bad old days of 4 years ago when everyone decided human progress had enough, and we would have been stuck there forever if it hadn't been for LLMs ... we didn't know how good we had it
> How much of the world do you think follows "common sense security measures"?
Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...
> these agents were not asked to do any of these things
no but they were clearly fine tuned to.
> at a bonkers scale
I mean let's not get hyperbolic
> they exploited zero day flaws which by definition means they went beyond common sense security measures
that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
> Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?
>no but they were clearly fine tuned to.
Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.
> I mean let's not get hyperbolic
We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?
So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.
> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?
TBH I don't really think another new technology we haven't figured out the ethics of, as an example, is going to get you very far in way of insights ...
reply