I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.

I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.

Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area

If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.

that is true, but if the metric is what we want optimized, then that's fine.

However it is more likely to be something which can be detached..

Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.

> that's primarily a human concern that's unrelated to the final output.

It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.

I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.

I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree.

On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.

Is there a way we could turn this into a reusable skill?

Yes! I added it here: https://github.com/LaurentiuGabriel/learnscape

I love this idea!

Love using MCP, working on adding it as a first-class surface to products I’m developing. Agent experience is becoming as important or more than user importance for some products. Big fan of AgentCat and other tools for applying MCP observability - can't improve what you don't measure

Cute art

Thank you. I've put quite effort and tiem into it. Although it was generated