205 - Unsustainable Costs

I can sincerely and ecstatically thank Nvidia for shooting themselves and OpenAI in the foot several times over, while deep in the delusion that they had something to brag about. As I've mentioned before they've made a point to hide all data they possibly could relating to electricity, particularly deployed operational electricity usage and training costs. The figures are quite damning, even if they incompetently measured electricity in dollars rather than an objective measure like watts.

If it would cost $400m to train a model at OpenAI's current scale for GPT-4, and nearly $60m in electricity alone to keep it running for 3 years, presumably multiplied by many such instances, then even Nvidia's hardware improvements they eagerly gloat over can't save OpenAI and similar frauds from blowing absurd amounts of money. To Nvidia that is great news, as they're pitching themselves as a cost-savings supplier for the world's most criminally incompetent tech companies, digging for gold in the dead-end of LLMs.

Consider that OpenAI and others like them train many models for every one model that they release, according to their own engineers, and that if they kept up the habit of increasing scale by 10x per generation then even Nvidia's current figures would mean a $4bn expense on training per model alone, and nearly $600m in electricity to run one instance of that new model for 3 years.

Using the electricity cost from Phoenix, AZ, where one data center is located, $59.2m would buy you about 395 Gigawatt Hours of electricity ($0.15 per kWh), so a 10x scale GPT-5 would require nearly 3,950 gigawatt-hours of electricity to operate for a single instance. If they deploy multiple such instances then that quickly skyrockets even further. At this point, you're basically already talking about dedicated nuclear reactors to power trashbot technology.

Also note that facilities like that have to spend a lot of energy on cooling costs, which may not be included in these figures, as they're frequently built in the middle of the desert, and known to pump the local groundwater supply until it runs dry. That is just one of the most tangible and easily quantifiable harms these companies directly cause.

I recently encountered a research paper describing general intelligence as the learning efficiency of a system, by which measure models that use "internet-scale data" while understanding nothing of the contents would be at the very bottom of the barrel, and even further down at larger scales.

GPT-4 and similar systems are still far, far behind what a small team accomplished on volunteered spare time and pocket change, years earlier, and no amount of scaling can overcome the divide between competently designed technology, built from scratch, and the trashbot technology that they produce. Nvidia can sell shovels to OpenAI, but those shovels dig a deep grave for them both.