I have been imagining this since GPT-3. I still only know of very few games making use of LLMs. There are add-ons for Skyrim, and web search points me to NVIDIA's ACE autonomous game characters framework and games such as InZOI, PUBG's ACE-powered AI teammate, Dead Meat (unreleased interrogation game), AI Dungeon, Suck Up!, Vaudeville, and more. It seems there are a few more than I would have thought, but the smaller ones seem to have some mixed reviews. It would be interesting to check out how these games resolve the reasons against LLMs in games I can think of:
Cost: LLMs fitting into local GPUs, in addition to the memory required for game textures, are too constrained for the quality, latency, and concurrency required. Connecting to the cloud would be too expensive, and even if it would work, it would indeed be the death of local offline gaming, and it would give a valid reason for a subscription to keep the servers running as opposed to a buy-once play-forever game business model.
Unreliability: You don't want to miss a major quest because the LLM did not decide to hand it out. It might be workable with some proper "agentic harness", with some hardcoded world lore, long-term memories, goals, midterm memories, some deterministic actions, and a short-term free-form loop that gets cleared every now and then. For OpenClaw instances, it would be the SOUL.md, but it would have to be read-only. Some logic might have to be hardcoded outside of LLMs, e.g., to start a new narrative branch powered by a new character memory when a keyword has been mentioned or a quest item has been given. Similar to agentic frameworks, it would have tool calls such as `offer_quest`, `give_reward`, etc.
These ideas seem to converge onto the Westworld mechanics of backstories, cornerstone memories, reveries, narratives, and loops, at least as far as I remember them from the series. Thinking about it, given enough investment, I feel like the amusement park of Westworld should already be somewhat possible to implement with the recent advances in LLMs and robotics.
In the end, I am not sure how much sense it makes for games. Maybe it would suffice to only make companion characters LLM-enhanced while NPCs stay pre-programmed in order to let the game stay deterministic. Games should simplify real life in order to increase enjoyability. Making all characters backed by LLMs might decrease enjoyability because of forgetfulness or confabulations inconsistent with the game world and actions. It would need to be heavily constrained and trained or prompted to say "I don't know about that.". It might also be questionable how much needs to be dynamic. E.g., when thinking about the NPCs in The Witcher, which are talking about random stuff, maybe you only need to pregenerate much more random stuff for them to talk about so that you don't notice the repetition, but there might be no need to have this completely dynamic and burn through GPU power for every action. The question would be what to cache and what really needs to be dynamic.
A harness can definitely still make it "deterministic":
if (responses > 3) prompt.md += "Divert the conversation to bring up QUEST4.md"
I look at it a lot like procedural generation. It can be done well if used in the right spots and enough handwritten/tweaked stuff to make sure it feels right.
This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?
The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
Why not simply have both? This does not have to be an either-or decision. Have a default repository with vetted extensions, but leave the option to install from other sources open.
Enterprise will always choose the less risky option so if there is either-or its vetted extensions only.
For consumer it's kind of already like this in a way, there are "verified" extension providers.
Overall, I think this is just going to lead to a lot more scrutiny. I'm sure one of the first things asked when this was discovered was how can it be prevented and I'm sure one of the first answers was get VsCode to lock down extensions. Enterprises love the easy answers
For seekable gzip indexes in zip, there SOZip: https://github.com/sozip/sozip-spec . However, it stores the indexes as files succeeding the actual file entry. To hide these index files and avoid extraction, they are not listed in the central directory, but a linear scan of the local headers, which some wrongly-behaved ZIP tools do, or which might be necessary for recovering broken ZIP files, would find those hidden indexes.
> 10. Code is cheap, but maintenance, support, and security aren’t.
I also keep circling around this point. So many software repositories in the AI space seem to follow a publish and forget pattern. If you simply can show that you have the patience to maintain a project, ideally with manual intervention instead of a fully autonomous AI, then you already have an outstanding project.
These are not errors. They are simply warnings about extended attributes being ignored when extracting files, which seems completely fine to me, and creating the tar without those extended attributes has exactly the same outcome, but throws away the metadata at archive time instead of extraction time.
Furthermore, this is not an Apple/macOS issue. The tool used is bsdtar, so it would also affect all BSD-variants that default to bsdtar/libarchive, and those systems also have extended attributes, e.g., for SELinux, which would get added to the TAR.
It's unfortunate there was never a standardization around how to print different "levels" of messages. Imagine how nice things could be if we had a standard system call for the equivalent of log level, early on!
Note, however, that this can only seek to frames, and zstd still only creates files containing a single frame by default. pzstd did create multi-frame files, but it is not being developed anymore. Other alternatives for creating seekable zstd files are: zeekstd, t2sz, and zstd-seekable-format-go.
Thanks, this is helpful. I might just end up using content defined chunking in addition/instead, but it's good to know that there is a path forward if I stick with the current architecture.
Exactly. And often this state is either highly compressible or non-compressible but only sparsely used. The latter can then be made compressible by replacing the unused bytes with zeros.
Ratarmount uses indexed_gzip, and when parallelization makes sense, it also uses rapidgzip. Rapidgzip implements the sparsity analysis to increase compressibility and then simply uses the gztool index format, i.e., compresses each 32 KiB using gzip itself, with unused bytes replaced with zeros where possible.
indexed_gzip, gztool, and rapidgzip all support seeking in gzip streams, but all have some trade-offs, e.g., rapidgzip is parallelized but will have much higher memory usage because of that than indexed_gzip or gztool. It might be possible to compile either of these to WebAssembly if there is demand.
I like the idea of bubblewrap, but my pain point is that it is work to set it up correctly with bind mounts and forwarding necessary environment variables to make the program actually work usefully. Could you share your pip bwrap configuration? It sounds useful.
> The obvious one outside of KV caches as mentioned above is vector databases. Any RAG pipeline that stores embedding vectors for retrieval benefits from the same compression. TurboQuant reduces indexing time to “virtually zero” on vector search tasks and outperforms product quantisation and RabbiQ on recall benchmarks using GloVe vectors.
This part sounds especially cool. I did not think about this application when reading the other articles about TurboQuant. It would be cool to have access to this performance optimization for local RAG.
Cost: LLMs fitting into local GPUs, in addition to the memory required for game textures, are too constrained for the quality, latency, and concurrency required. Connecting to the cloud would be too expensive, and even if it would work, it would indeed be the death of local offline gaming, and it would give a valid reason for a subscription to keep the servers running as opposed to a buy-once play-forever game business model.
Unreliability: You don't want to miss a major quest because the LLM did not decide to hand it out. It might be workable with some proper "agentic harness", with some hardcoded world lore, long-term memories, goals, midterm memories, some deterministic actions, and a short-term free-form loop that gets cleared every now and then. For OpenClaw instances, it would be the SOUL.md, but it would have to be read-only. Some logic might have to be hardcoded outside of LLMs, e.g., to start a new narrative branch powered by a new character memory when a keyword has been mentioned or a quest item has been given. Similar to agentic frameworks, it would have tool calls such as `offer_quest`, `give_reward`, etc.
These ideas seem to converge onto the Westworld mechanics of backstories, cornerstone memories, reveries, narratives, and loops, at least as far as I remember them from the series. Thinking about it, given enough investment, I feel like the amusement park of Westworld should already be somewhat possible to implement with the recent advances in LLMs and robotics.
In the end, I am not sure how much sense it makes for games. Maybe it would suffice to only make companion characters LLM-enhanced while NPCs stay pre-programmed in order to let the game stay deterministic. Games should simplify real life in order to increase enjoyability. Making all characters backed by LLMs might decrease enjoyability because of forgetfulness or confabulations inconsistent with the game world and actions. It would need to be heavily constrained and trained or prompted to say "I don't know about that.". It might also be questionable how much needs to be dynamic. E.g., when thinking about the NPCs in The Witcher, which are talking about random stuff, maybe you only need to pregenerate much more random stuff for them to talk about so that you don't notice the repetition, but there might be no need to have this completely dynamic and burn through GPU power for every action. The question would be what to cache and what really needs to be dynamic.