Human in the Loop: Why AI Still Needs Human Oversight

Yes, AI still needs human oversight. It drafts, codes, tests and summarizes faster than any person, but it has no reliable sense of when it is wrong, and it doesn't know your business. Human in the loop means a person sets the goal, checks the output at the points that matter, and owns the final decision. Done that way, AI makes a team dramatically faster without making it riskier.

I build software this way every day, and I like it a lot. AI is the biggest change to how software gets made in my 25+ years of doing it. This isn't an argument against it.

It's an argument about where the human goes. Because the founders who get burned by AI usually aren't the ones who avoided it. They're the ones whose team adopted it quickly and never decided who checks the work.

What does "human in the loop" actually mean?

Human in the loop (HITL) means a person reviews or approves AI output at defined checkpoints before it has real-world effect. The AI does the volume. The person does the judgment.

In practice there are three levels, and most teams are mixing them without realizing it:

The mistake isn't picking the wrong level. It's never picking one. When nobody decides, the default becomes "out of the loop" for whatever is moving fastest. That is usually the riskiest stuff.

Where AI is strong, and where it still needs a person

AI predicts what a good answer looks like. Most of the time that prediction is right. The problem is that it sounds exactly as sure when it's wrong.

The right-hand column is the job. It always was. AI just made the left-hand column almost free.

What happens when nobody checks the AI?

Mostly small mistakes, and they get expensive because AI repeats them at the same speed it does everything else. A few public examples:

• A chatbot invented a refund policy. Air Canada's website chatbot told a customer he could claim a bereavement fare after booking. That wasn't the policy. A tribunal held the airline responsible for what its chatbot said and ordered it to pay. The decision is Moffatt v. Air Canada.
• A legal brief cited cases that didn't exist. Lawyers in Mata v. Avianca filed a brief drafted with ChatGPT that cited invented court decisions. The court sanctioned them.
• AI suggests packages that don't exist. Researchers studying code-generating models found that a meaningful share of the packages they recommend are hallucinated. Attackers have started registering those made-up names, so a developer who installs whatever the AI suggested can pull in malicious code. The attack now has a name: slopsquatting.

None of those were failures of AI. They were failures of review. Inside a product company the same pattern shows up quietly:

• A pull request is too large to read, so it gets approved on the strength of passing tests that the same AI wrote
• An AI agent with production credentials runs a destructive command nobody expected it to be able to run
• Customer data gets pasted into a consumer AI tool with no data agreement
• Help docs or marketing pages state features, limits or terms that aren't true

The gotchas that catch smart teams

The biggest gotcha is that AI output looks finished. It's formatted, confident and grammatical, so it feels reviewed when it isn't.

1. Confidence isn't accuracy. Models state guesses in the same tone as facts. This is called hallucination, and it clusters around names, numbers, versions, dates and citations.
2. It agrees with you. Ask a leading question and you'll get the answer you hinted at. Ask it to argue the other side, or ask a second model to review the first.
3. Its tests check its own assumptions. If the AI misunderstood the requirement, the tests it wrote will pass, and they'll confirm the misunderstanding.
4. Volume beats review. A developer can now generate more code in an afternoon than a reviewer can read carefully in a day. If review capacity didn't grow, review quality fell.
5. It doesn't know your business. It has never met your customers, read your contracts or seen last year's outage. That context has to come from a person.
6. Small errors compound. In multi-step agent workflows, one wrong assumption early carries through every step after it. I wrote about that drift in Why Your AI Project Is Stuck at 80%.

How to tell if your team has enough oversight

If you're a founder and you can't read the code yourself, you can still find out whether anyone is checking. Ask your team these seven questions:

1. Where are the human checkpoints? "We review everything" is not an answer. Ask which steps require a named person's approval.
2. How big are the pull requests? If AI-assisted PRs are routinely thousands of lines, nobody is reading them closely.
3. Who wrote the tests? Tests written by the same AI that wrote the code are a starting point, not proof.
4. What can the AI tools touch? Production databases, deploy keys and customer data should require a human in the loop. Agents should get the narrowest permissions that let them work.
5. How do you verify new dependencies? Someone should confirm a suggested package is real, maintained and the one you meant.
6. Where does customer data go? Business accounts with clear data terms, or nowhere. Not personal accounts on free tools.
7. Who owns each AI tool? One accountable person per tool, for what it produces and what it costs.

Vague answers are the finding. They don't mean your team is doing bad work. They mean nobody has decided, which is the gap that eventually costs you.

A review process that keeps the speed

You don't need to review everything equally. You need to match review to risk, and decide that before the work starts.

Then a few habits that do most of the work:

• Write the brief first. Clear requirements, constraints and "never do this" rules up front. Most bad AI output starts as a vague prompt. More on that in Prompt Engineering vs. Prompt Guessing.
• Keep changes small. Small PRs get real reviews. Large ones get rubber stamps.
• Check facts at the source, never against the AI that produced them.
• Add what only you know. Real examples and real decisions are what make code and content worth trusting. Google's guidance on AI-generated content asks for exactly that.

If you want a formal framework, the NIST AI Risk Management Framework is free and scales down well. If your product falls into a regulated, high-risk category in the EU, human oversight stops being a best practice and becomes a legal requirement under Article 14 of the EU AI Act.

What it looks like when it's done well

It looks like a small team moving at the speed of a big one. AI handles drafting, coding and testing. People handle architecture, review, and anything that touches a customer.

That's how I run AIwhirks: one principal engineer working with a set of specialized AI agents. The agents do the volume. Every decision that matters, every merge to production and every promise to a client goes through a person first. The speed comes from the AI. The reason it holds up is the human checking it. It's the same lesson as vibe coding, applied to a whole team.

If you aren't sure your team has that second half covered, that's exactly the gap a fractional CTO closes: someone who can read the code, ask the right questions and tell you plainly where a person needs to stay in the loop.

FAQ

Related Reading

• Why Your AI Project Is Stuck at 80%: The gap between an AI demo and a shipped product, and where the last 20% hides.

• The Hidden Cost of Vibe Coding With AI: Why planning still matters when AI writes the code.

• Vibe Coding Do's and Don'ts: Using AI to build software without creating a mess.

If your team adopted AI fast and nobody has asked where the human checkpoints are, ask now, before something ships that nobody read. Get in touch if you want a second set of eyes on it.