167 - Vulnerable By Design

One of the biggest red flags that any financial institution can offer is to integrate LLMs into their systems. These systems are both vulnerable-by-design and not built for handling math. Since "Hallucination" is a feature, not a bug, and that feature is fundamentally impossible to remove, then every step poses an increasing liability under the best of circumstances.

Since the systems are vulnerable-by-design in cybersecurity terms, with cybercrime being the 3rd largest market in the world at over $8 trillion annually (by Statistica's relatively conservative estimates), and researchers rapidly discovering new and complimentary ways to break LLMs, "the best of circumstances" will virtually never be the case in real-world conditions.

The same problems hit a variety of other industries to varying degrees, but the financial domain offers a very clear-cut least-compatible pairing of LLM technology and the strengths/weaknesses as they pertain to the domain.

The two domains that actually benefit from LLMs are marketing and cybercrime, largely for the same reasons. Both domains only require "plausible-sounding, but wrong" content and the preparation of that content explicitly requires the absence of ethics. This combination of needs means that more advanced systems with alignment and ethics would blacklist most marketers today, alongside their cybercrime counterparts.

LLMs and RL are both narrow tools, so there is a narrow subset of cases where each may be the optimal tool for a given task, but using them outside of those tasks is just "cutting glass with a hammer".

Determining where your domain and the tasks within that domain have legitimate opportunities for various forms of AI and automation is a challenge, made considerably worse by the copious amounts of snake oil being peddled by every pop-up AI startup attempting to reskin other people's algorithms and scalp some profit without adding any value, or even by selling the wrong tools and services for the job.

On a final note, most "AI experts" today, even those who still hold a measure of credibility, have absolutely no clue what the current cutting-edge looks like. They know LLMs and RL (hammers), so they often tend to treat every problem like it can be solved to some degree by them (nails). I constantly hear people discussing problems that have been solved and demonstrated as though they were some future research goal, which said individuals had little or no idea how to reach.

You may be blind in the domain of AI, but the people leading you are most likely blind too.