this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.
i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)
1. labs have lots of inference capacity
2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)
You can't exceed roofline performance on hardware. There is an performance cap you can hit. This recursive self improvement stuff lets you be closer to the pareto frontier, but the idea that it is leading to some exponential growth is a total pipe dream.
Can probably give access to tools like ast-grep to Claude. Will help it see all references. I still agree some dynamic references might still be left. Only way is to prompt well enough. Since I tested this on a Ruby on Rails codebase, I dealt with this.
