It's interesting the 6b model outperforms the 33b model. I wonder if it means the 33b model needs more training data? It was pretrained on ~1 million C programs, compared to DeepSeek-Coder, which was trained on 2 trillion tokens, which is a few orders of magnitude more data.
I'm also curious about how this compares to non-LLM solutions.
This has been the dynamics with LLMs for awhile. The majority of LLMs are massively undertrained. 7b models are the least "undertrained" mainstream models we have, hence why they have proliferated so much among the LLM fine-tuning community.
Yes, it's not easy to train a 33B model. An interesting point is, naive fine-tuning, which means if one followed the standard way to fine-tune the model. Training a larger model is tricky, not only the data amount matters, everything like data cleaning, learning rate, and decays will affect the final performance.
> on ~1 million C programs, compared to [...] 2 trillion tokens, which is a few orders of magnitude more data.
Is that comparable like that? This would assume that the average C program of the set is orders (plural) of magnitude less than 2m tokens in size, which could indeed be true but sounds like an optimistic assumption.
I'm also curious about how this compares to non-LLM solutions.