> It must be impossible for AI to: make purchases you did not intend, send messages you did not intend, and access information you did not want to share.
Hrm. I'd say that all of these things are possible when using services of companies that are valued at billions/trillions in the stock market. So "impossible" might be too strict.
Not even then, necessarily. A great number of recognized personal names and surnames are proactively reserved and cannot be registered even if there are no 3LD registrations under it. I've confirmed that in the history of .name there has never been a third-level registrant under maxwell.name other than myself, and yet I was not able to register maxwell.name directly when I first got my .name 3LD in 2013.
Why would a human review it? That only drives up costs for the company. Easier to use algorithms for everything and maybe pay the occasional fine when something goes seriously wrong and legal proceedings result.
If a human did read it, it very likely was not someone who was a native English speaker making well below minimum wage of the area they are trying to contextualize.
I'm sorry, but a multi-kilogram (at a guess), five-figure-priced (at a conservative guess) device with kilometer-scale error (per their actual press release) is not a substitute for a GPS.
And you probably can't improve on many of those specs no matter how much "inventing" you do. There's just not that much information in magnetic fields, especially if you don't spend an infeasible amount of money on keeping maps up to date. You're probably jammable, too.
The same applies to the gravitational hacks, and the mapping issue applies even to visual navigation.
Those guesses, by the way, are because not only is AQNav's "data sheet" worthless fluff, but they seem to be allergic to letting anybody even see a picture of the product. Which makes me doubt that they actually have what a normal person would think of as a "product" to begin with.
Datasheets that look like that are a pretty strong indicator you're dealing with something that is giga export controlled, not as in "can only go to some countries, you have to declare it and theres a LOT of paperwork", more "attempting to, or even making plans about trying to leave the country with one is a pile of federal crimes"
Virtually anything navigational that cannot be easily or reliably jammed falls into this category far before meeting any of the accuracy/drift requirements.
The magnetic approach is very interesting. I can't imagine it would ever be reliable enough for a primary instrument since its susceptible to solar weather and certain regions with more geological activity.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF file
Is it? ELF's specified enough that I can't imagine that it's too hard to avoid checksum mismatches between traditionally-modified ELF binaries and ELF-SELF-ELF-modified ones.
Like, it's an ugly spec, but it's clear enough on ordering/layout that I think checksumming is probably tractable to get parity for.
> I gave it the hardest real task that fits on one machine: reverse-engineering a commercial app's license check...
Respectfully, tasks that allow for explicit straightforward true/false or done/not-done tests are not the "hardest real task[s]." In fact, those are the ones that see the most gains from AI-assisted coding.
Testable tasks are where the largest opportunity is.
Maybe so, but there were other elements that I've seen frontier models struggle with in the past, which was the perspective I had coming into this. It's the type of test I run frequently and this is the first small local model I've seen pull it off.
It had a very non-standard RSA key implementation that was obfuscated heavily. As well, it has an online license check at first run, and that part typically trips up most of the local models I've tried. I've been running this test for about a year now with different models, and it was the first I've seen not only figure out the RSA key implementation, but the first that didn't just give up once it saw the online license check. Even though it's only a first-time launch check.
That's why I call it one of the hardest, because in my experience, it has been. It's the first local model I've seen pull it off end-to-end. For some of the reverse engineering work that I've done with LLMs, none have been as consistent as this particular test at highlighting a model's failure in this domain.
I have access to Daybreak Blue and I'm approved for Anthropic's Cybersecurity program, so I might run the same test with both of those just to see, because it's been a while since I used a frontier model on this test. I imagine they'll make relatively light work of it, though, assuming it doesn't trip the relaxed guardrails.
It was surprisingly easy for me to be approved in their CVP programme as an independent security researcher, that said, abliterated Qwen3.8-27B model feels better and cheaper.
Not sure to be honest. I have a couple of CVEs and also linked a talk I had presented as it mentioned those as an example. Got approved in a few minutes.
I've included docs and tests as part of my vibe coding endevours. It doesn't matter if either is litterally correct, but they create guardrails for future context to prevent regresssions and blind avenues, etc.
It's fairly successful but hits the time constrains and reduces the "value" of getting a local model to develop software.
how do incorrect tests or docs help create correct guardrails?
if your tests and docs are possibly incorrect, and you're not writing the code.. how do you know if it even works? for extremely simple software you can just use it but for anything with access to disk or the network or with user options...
you sound psychotic. actually. so nevermind, LLM psychosis is extremely common on this website, that's def all that's happening here
Also a reverse engineering tasks that can be don with just static analysis is arguably not the hardest reverse engineering task.
For those small models I would say it's not about the capabilities but more about the context size it can actually use.
It is probably no coincidence that AI is exceedingly good at finding small counter examples. But for the Riemann hypothesis no such counter examples exist. And likely none exist.
An answer would be a proof that either it holds, or that it doesn't hold. You would test such an answer with an automated proof checker (probably lean).
I still think that Anthropic went the wrong way. It would have been much more entertaining to ask the model to find a non trivial zero not on the line and give it encouragement. To see what exactly it will come up with.
as he rightly says, fuck all benchmarks and metric as long as it can get <my task> done. who cares what it specifically good at or not and trying to create benchmarks as long as it solves <my problem>.
Hrm. I'd say that all of these things are possible when using services of companies that are valued at billions/trillions in the stock market. So "impossible" might be too strict.
reply