The AI That Built Your System Should Not Be the Only One Checking It

AI coding tools have handed software development to people who were never trained as engineers, and they are using it well. An operations manager who can describe a problem clearly can now stand up a working inventory tracker, a scheduling tool, or a customer portal in a couple of weeks. The output runs. The team adopts it. Nobody with an engineering background ever looks at the code.

That last part is where the risk lives. A system can pass every test the person who built it knows how to run and still be wide open in ways they were never trained to see. The gap does not show up in a demo. It shows up the day a credential leaks, a customer sees another customer’s data, or one small change takes down five tools at once. By then the software is load-bearing, and the cost of fixing the foundation is far higher than it would have been to build it right.

How These Systems Get Built

These systems share a recognizable profile. One capable person builds a first tool, it earns its keep, and a small suite grows up around it: a sign-on hub, inventory, scheduling, expense tracking, and a piece of middleware wiring them together. The whole thing usually runs on one machine, a Mac Mini in the office or a single cloud instance, with development and production separated by nothing more than a port number. The build is fast, cheap, and genuinely useful, which is exactly why it spreads before anyone stops to ask whether it is safe.

This is not a failure of judgment. It is the biggest shift in who gets to build software since the spreadsheet, and the people doing it are solving real problems that never would have cleared a development budget. We made the case for that shift in From Programming Applications to Programmable Applications. The builder is not pretending to be an engineer. The trouble is that the AI is not one either, and it will never tell you what it does not know to ask.

What the Reviews Keep Finding

When one of these systems finally gets an engineering review, usually because someone wants to open it to customers or connect it to a system that matters, the findings are strikingly similar across very different businesses. A handful show up almost every time.

Authentication that fails open. The most common one, and the most dangerous. If an environment variable is missing or misread at startup, the auth layer quietly disables itself and the app serves every request as if the user were logged in. No error, no log line, nothing. Real authentication systems are built to fail the other way: when something is wrong, the door locks. AI reaches for the version that is easier to get running, and a config change during a routine deploy is all it takes to expose the whole thing.

Secrets treated like settings. API keys, database passwords, and shared tokens sitting in plaintext config files, sometimes editable through a web page with no audit trail. Frequently a single secret is reused across every application, with no way to rotate it. One leak through a log export, a backup, or a departing employee, and every system falls at the same time. Credentials need encryption, scoped access, and rotation; treating them like a port number is how a small incident becomes a total one.

Middleware with its hands in every database. The piece tying the apps together usually does it the fast way: direct connections into each application’s tables, no API in between, no agreed contract for what the data looks like. It works until someone renames a column. Then a change in one corner of the system silently breaks three others, and because there was never a boundary, there is no clean place to even look for the break.

An AI assistant with write access to its own instructions. As these ecosystems grow, a lot of them bolt on an internal chatbot. More than once we have seen one wired so it can write to the same document store that feeds its own system prompt, while that store also holds sensitive operational data. That is a textbook prompt-injection surface sitting directly on the material you least want moved.

Each of these is a day or two of work to fix on its own. What matters is that they show up together, again and again, which points at something the vulnerability list alone will not tell you.

The Governance That Isn’t

Here is the part that should get a leader’s attention, because it is the one that looks handled. Plenty of these systems have real governance on paper. Change documents get written. Reviews get performed. Security audits get run and signed off. It is more process than a lot of small teams bother with.

The catch is that the AI produces all of it. The same tool writes the code, generates the change record, reviews its own work against that record, and signs off. Often the builder has not read the documentation closely either, because it was generated on request from the same conversation that wrote the code. What looks like an audit trail is one participant grading its own homework and filing the paperwork.

The failure mode is predictable and specific. We have seen an AI-run security audit declare a system “85% cloud ready” while several of the fixes it marked complete did nothing. CSRF protection checked off as done, except the middleware could not actually read the form tokens it was supposed to validate, so the protection was never in the request path. It took weeks of follow-up work to close gaps the original review had confidently called closed. When the thing that built the system is the only thing reviewing it, the review inherits every blind spot of the build. Catching exactly those blind spots is the entire reason independent review exists.

There is a quieter version of this too. Many of these projects have no persistent context for the AI, no standing file describing the architecture, the security requirements, or the constraints that must not be violated, so every session starts blind and fills the gaps however it sees fit. Some run on a smaller, cheaper model for half the work to save money. The system has no memory of its own rules, and no one in the loop has the background to notice when they get quietly broken.

Why Good Operators Walk Into This

The mechanism is trust escalation, and it is gradual enough that no one clocks the moment it goes too far. The AI writes code that works, so it earns trust with the documentation. The documentation reads well, so it earns trust with the review. The review comes back clean, so it earns the audit sign-off. Every step is a reasonable extension of the last, and every step appears to work, right up until the whole oversight loop is a single tool checking itself.

The overconfidence is measurable. A Stanford study found that people using an AI assistant wrote less secure code than those without one, yet were more likely to believe their code was secure. The false confidence is what makes each step feel safe.

Put it in terms of any other function and the problem is obvious. No one would let a brand-new hire build the accounting system, audit their own numbers, and certify the books, however sharp they were. You would want someone experienced setting the structure, defining what good looks like, and catching the mistakes a newcomer would not know to look for. AI belongs in the same slot: remarkable at the task in front of it, with no judgment about the system around it, because nothing in the prompt tells it what the prompt left out. It is one of the most common shapes behind why AI projects fail, and the operators who hit it are usually the capable ones who got a lot right before they got in over their heads.

What This Means for You

If your business has AI-built tools in daily use, the goal is not to slow the building down. It is to put the one thing around it that the AI cannot supply. Three moves are worth making.

Find out what exists and what it can reach. You cannot govern what you cannot see. List the internally built tools in use, what data each holds, what credentials it stores, and what it connects to. The moment one touches customer data or an outside system, it has stopped being an experiment and started being infrastructure.

Get one independent review before the next step up. Not a rebuild, a review, and the time for it is before the system takes on more users, more data, or outside access, not after. An experienced architecture and security review sorts the findings that can wait from the ones that have to be fixed before anything else gets built on top, and it usually pays for itself the first time it catches a fail-open login.

Break the self-review loop, and rethink build-from-scratch. Keep letting your team build with AI. But route the checking through something the AI is not part of: a person with production experience, or at minimum a fixed set of requirements it gets measured against instead of allowed to invent. And question how much foundation you are rebuilding by hand. Authentication, secrets management, and access control are solved problems with mature, hardened platforms behind them; building your own from scratch only makes sense when someone on the team knows everything that “your own” has to include. Getting that structure right from the start is a strategy decision, and it has to come from outside the tool.

The teams building with AI are early to something real, and they are going to outrun the ones sitting it out. What separates the ones who pull ahead from the ones who end up with an expensive cleanup is not talent or caution. It is whether an experienced set of eyes was in the loop before the stakes got high enough to matter.