182 - Game vs. Reality

In revisiting N.N. Taleb's book The Black Swan: 2nd Edition, it struck me that the "Ludic Fallacy" applies very strongly to typical AI researchers, engineers, influencers, and investors, but that cybersecurity specialists and hackers frequently avoid this problem. The former is couched heavily in theory, bankrupt "benchmarks", and "normal distributions", while the latter must inherently operate in the real world to defend or break those real systems.

For those unfamiliar with the Ludic Fallacy, it is "the misuse of games to model real-life situations", such as "basing studies of chance on the narrow world of games and dice". This makes it a form of what Daniel Kahneman called "Substitution Bias", a subset of cognitive biases where one simple thing is used as a (weak or fraudulent) proxy for a more complex or difficult one.

Which weak or fraudulent proxy is selected is often guided heavily by theory, further compounding the problem and systematically bankrupting the results. "Benchmarks" serve as springboards for this cognitive bias, giving the illusion of being something firm, while remaining fully arbitrary and trivial, which allows those who lean into them to put even greater distance between themselves and reality.

"Normal Distributions" also tend to highlight only a marginal subset of information that holds any actual value, as it may describe the rule, but remains inherently unable to show the exceptions to that rule. As I noted in a previous recent post, what separates complete fools from wisdom is the ability to recognize exceptions to any given rule, iteratively learning them, and that information is simultaneously heuristically rare and the most critically important in terms of impact.

The problem is compounded by weak and shallow attempts to "steer" systems like LLMs or RL with RLHF. Such systems attempt to gloss over exceptions with vague and incompatible sources of corporately curated bias injected at a higher level, which offer no value at the level of exception recognition, often only producing canned responses or denials at a higher level.

In contrast, Cybersecurity specialists and hackers must operate in the real world to defend and break real systems, as it is their job to seek out the exceptions to the rule. While remaining aware of the same theories, benchmarks, and distributions, they aren't chained to them, recognizing the marginal-to-trivial value they actually offer. Substitution biases can't do the job of cybersecurity, even if dart-throwing-chimpanzees can call themselves "Gen AI" researchers, engineers, influencers, and investors.

The overwhelming majority of real-world value can't be reliably delivered by systems built for the trivially simple dynamics of a game environment, even using internet-scale data and massive amounts of compute. After all, that data is gathered under massively simplified game-like dynamics, and processed via the brute-force math of neural networks. No amount of overtly fraudulent hype and marketing can change the utter lack of actual value delivered.

The search continues for a hypothetical investor who doesn't eat the Ludic Fallacy for breakfast, lunch, and dinner. If the recent valuation of "Devin" at $2 billion tells us anything, it is that such an investor would be the exception to the rule of utter incompetence.