143 - Disinformation Networks

A rising concern in scientific circles is that disinformation networks could form, creating plausible-looking pages and papers that link to other plausible-looking pages and papers, and so on across large networks and loops, all of which are fraudulent, and most of which are AI-generated. Unfortunately, this is already a reality and has been building in severity for years.

The basic principle of bad actors in real-world conditions is that if they aren't punished they're almost guaranteed to continue engaging in any forms of fraud available to them, optimizing their methods and tactics. This has become not only popular but highly incentivized in recent years. Before the recent boom of "Generative AI," this process was present, and lucrative for bad actors, but also much slower.

One of the core tactics of bad actors relies on highly gullible groups of students and enthusiasts, as well as other lesser bad actors and opportunists, usually exploiting "Theory-Induced Blindness" as Kahneman termed it. In this tactic, a foundation is laid using theories and assumptions that have either already been debunked, or are debunked immediately thereafter. The foundation is architected to exploit cognitive biases, driving people to act on emotional desires in ways that they're often only marginally conscious of.

After this foundation is laid, a cult following develops around it, resisting all evidence debunking that foundation. These cults then go on to build on it, pushing out papers and studies one on top of another, giving the appearance of increasing plausibility, even if the original paper was just a corporate blog post that was never even put to peer review. The most obvious frauds in the AI industry today, with the second-highest startup valuation at present, exploited this method heavily. The fraudulent corporate blog posts their names were attached to, among 5-50 other "co-authors", were cited thousands of times, giving the illusion of prestige via vulnerabilities in the H-index.

When searching for related research papers recently I first began encountering the new circuits of 100% disinformation networks as they appeared in search results. These are now a real hazard thanks to AI-generated SEO, pages, and papers, and link after link never leads to an actual research paper.

Other popular mechanisms of abuse involve targeting "high prestige" journals, as well as commonly used benchmarks. Benchmarks are frequently abused via fallacies, correlations, data leakage, and various methodological flaws to support the debunked narratives of "emergence", "understanding", and "reasoning" in trivial systems like LLMs. The bad actors abusing these benchmarks then move on to target the most prestigious journals, and once one of them makes it through then an increasing volume of similar fraud is likely to follow.

Where the cash flows, bad actors will follow. Where those bad actors aren't punished, they'll eventually dominate the resource supply, even with zero viable technology.