207 - Scale Paradox

It is a truly baffling thing to realize that even without funding and very little engineering time currently being dedicated, the relative benefits of our technology are still actually increasing. This is made possible in part by the companies who think of themselves as competitors or market leaders growing progressively worse by steadily upping the snake oil-to-product ratio in favor of snake oil.

In years previous we were ~100x more efficient in electrical usage terms while remaining far more effective and fundamentally capable than OpenAI's best, which increased to a ~447x advantage in our favor thanks to GPT-4, even if you use Nvidia's figures that pose the hypothetical case of "...if (Nvidia's) best GPUs in 2024 were to train GPT-4 today...", which tells you that the actual numbers from when it was originally trained would be considerably worse, and remain undisclosed because of that.

Better still, they and others in the industry have a track record of 10x-ing whatever scale they're currently operating at with the next version while delivering higher fidelity parroting and precisely zero reasoning or understanding. That means that when they take that next leap up in scale we'll be easily over 1000x more efficient in electrical usage terms, and still at least 10,000x more data efficient.

I'm sure they'd have improved our data efficiency ratio relative to their own by becoming much worse if they possibly could have, but once you move to "internet-scale data" then there isn't much else to add except fake ("synthetic") data. Thanks to the overflowing volume of LLMs recent hype has produced then there is already a quickly growing abundance of such fake data, which will predictably serve as data poisoning for all further LLMs. Since they also lack the ability to reliably detect AI-generated data, particularly when it is designed and tweaked to avoid detection, they have no viable path to mitigation where they get to keep greedily scraping without suffering the predictable consequences.

It remains mind-boggling even as I continue to think over it that our position can improve so quickly just by practically standing still. Is the rest of the industry too busy dreaming of sailing the Seven Seas of Snake Oil while flying the Hype flag?

They'll have nothing left to do but go down with that ship once we really begin moving. One of the only things keeping them afloat now is that humanity has largely lost the ability to distinguish fraud from fact, epitomized by the meme of an explosion of people claiming AI expertise. The irony is that the people often pointed to as being at the top of that AI expertise hierarchy that many more attempt to climb have either already lost all credibility, or never had any to begin with.

The relative competence of "experts", adopters, investors, and many more categories may be measured by such ratios far more accurately than h-index scores, years in the industry, or follower counts.