160 - Environmental Impact

One of the most infuriating things about AI today is that companies go to extreme lengths to guard all information related to the environmental impact of their AI systems. I haven't been able to update comparative figures on the electrical waste of LLMs and other AI systems since Nvidia's "Megatron" model in 2021.

Information on environmental impact should be public knowledge, and routinely audited, not something that can be treated as a trade secret. Even to calculate the electricity wasted on Megatron's training I had to dig into hardware spec documentation for the servers they used.

Many companies also hide behind the fraudulent illusion of being "green" by buying "carbon credits" to make sure that they technically run on renewable energy sources...by making sure that nobody else gets access to that renewable energy. This is the typical song and dance of bribery and patronage among corrupt bureaucratic systems.

A meeting with an investor from the venture capital arm of a particular hardware company reminded me recently that some people do indeed still care about sustainability, beyond the thin veneer of placating critics. That is also one area where we've always held an insurmountable advantage since our technology wasn't built on the brute-force computation of neural networks burning up GPUs.

When the basis of your technology isn't brute-force computation then memory bandwidth becomes far more important than the number of operations per second. Ordinary DDR5 RAM operates at voltages so low that over 200 Terabytes of it could run on the same electricity required for a single Nvidia cluster of 8 A100 GPUs.

Not only that, but the RAM has a life cycle several times longer than GPU hardware, and shy of ridiculous and wholly unnecessary overclocking, such RAM doesn't require much cooling, avoiding the need to extract waste heat via liquid cooling and the subsequent groundwater depletion that causes.

Try as companies might to hide the skeletons in their environmental closet, we already know that they're causing at least 100x the waste today compared to what competently designed technology can accomplish. Any groups with a genuine interest in the environment could fund the remaining work to deploy said technology, or they can pay 100-1,000 times more than that trying to clean up the environmental mess later.

No LLM has ever achieved "State of the Art" (SOTA) in any non-trivial sense, nor can scaling ever do so. What they can do is waste a ton of electricity and water, while also polluting the internet.