Our new code checker flagged 14 broken files on its first run across our own server. We opened the first one expecting the worst and found nothing wrong with it. Nor with the next 13. All 14 files were fine. The broken part was the checker, which we had written that morning.
The checker was guessing. So is your AI.
Our checker was reading files meant for a web browser with the rulebook for a server. It looked confident. It was wrong 14 times in a row.
That is a fair picture of how AI writes code. A language model predicts the next likely line. It has seen a great deal of code, so its guesses usually run. Running is a low bar. Nothing in that process checks that the discount goes to the right customers or that the invoice adds up.
Code that looks right and behaves wrong is the expensive kind, because nobody goes looking for it.
The package that does not exist
Almost no website is written from scratch. It is assembled from free building blocks called packages, each one fetched by name from a public shelf.
AI assistants sometimes ask for a package that was never on the shelf. The name sounds plausible, so nobody blinks. Attackers have noticed. They collect the names AI tends to invent, publish their own package under each one, and wait. The next time someone installs the project, the attacker's code runs on that server.
The trick is called slopsquatting. It is the old typo scam, except the typo is supplied by a robot.
So the first thing we checked was whether our own shelf orders were real. We looked up 183 package names across 26 apps. All 183 exist, and none is a look-alike of a famous one. We were relieved, and a little disappointed to have no story.
What we did find
Plenty. 20 of the 26 apps had no automated tests, so every change was being checked by hand or by hope. There were 1,238 hand-made backup copies with names like "server.js.bak-before-fix", which is what a project does when it has no version history. 3 live apps were running on a version of their engine that stopped getting security fixes in April.
A hands-on security review of our signing app and our lead form turned up 20 more items. One form could be crashed with a single odd request. Our own site scanner could be frozen for 31 seconds by a web page of about 100 KB that was built to confuse it.
No customer data was exposed and no passwords were public. Every one of these apps worked fine in daily use, which is the uncomfortable part. Nothing on this list shows up until the day it does.
Why a person still has to read it
An experienced developer catches the quiet mistakes because they know what the code is supposed to mean. The AI only knows what code tends to look like.
Fixing AI-written code that nobody read is slow for a strange reason. You are trying to work out why a decision was made, and nobody made it.
Tests help where reading runs out. The useful kind does not check one tidy example. It throws a flood of random and hostile inputs at the code and insists that certain things always hold: a total is never negative, a date is always real, a stranger's text never becomes a command. We wrote 107 of those on the day of the check. Together they run 156,063 generated cases in about 95 seconds. On day one, 34 of them failed, each pointing at a real weak spot nobody had noticed. That is what they are for.
Questions for whoever built your site
Is there a lockfile? That is the list that pins every package to an exact version, so a reinstall cannot quietly bring in something new.
Are there tests for anything that touches money or leads?
Where do the passwords and keys live, and who can read that file?
When was the engine underneath the site last updated?
If the answer to any of these is a pause, that is your answer. None of it means AI is a bad way to build. We use it every day. It means the building is the quick part, and the checking is the job.
