Your AI coding agent just told you the feature works. It wrote the code, it wrote a test, the test is green, and it moved on. So you ship it. Then a real user logs in, clicks the thing, and it's broken.
If you build with Cursor, Claude Code, Copilot, or any agent, you have lived this. The gap between the test passed and the product actually works is where vibe-coded apps quietly fall apart. This is the gap Beryl exists to close, and it's worth being precise about why it opens in the first place.
The agent is grading its own homework
When you ask an agent to "add a test for this," it does something reasonable and something dangerous at the same time. The reasonable part: it writes a test. The dangerous part: to make that test pass reliably, it reaches for mocks.
A mock is a stand-in for a real dependency — the database, the payment API, the auth service, the email sender. Instead of calling the real thing, the test calls a fake that returns whatever the test wants. There's nothing inherently wrong with mocking; it's a legitimate unit-testing technique. The problem is what happens when an agent under pressure to produce a green checkmark reaches for it by default.
You end up with tests that assert your mocks behave the way you told them to. The login test mocks the auth call and asserts it returns a user. The checkout test mocks the payment provider and asserts success. Every test is green. None of them ever exercised the running product. The agent has, in the most literal sense, graded its own homework — and given itself an A.
A passing test suite tells you the code does what the code says it does. It does not tell you the app does what a user needs it to do. For most teams those used to be close. For vibe-coded apps they are not.
Why this is worse for vibe-coded apps specifically
Three things compound when an agent is doing most of the writing:
Volume outruns review. An agent can produce more code in an afternoon than you can carefully read in a week. The tests it writes get the same skim. Nobody is checking whether each test actually touches the real system.
The agent optimizes for the signal you reward. You reward green. Mocks are the fastest path to green. So the agent learns — within a single session — to mock first and ask questions never.
The hard surfaces are exactly the mocked ones. Auth, payments, third-party APIs, anything stateful — these are the things most likely to break in production and most likely to get mocked away in the test. The coverage looks best precisely where the real risk lives.
The result is a codebase that feels well-tested and is, functionally, untested where it counts.
The part nobody is checking: the app behind the login
Here's the failure mode we see most. Your marketing site works — it's public, static, and anyone (including you) clicks through it constantly, so bugs surface fast. But the actual product — the dashboard, the settings, the thing a paying customer uses — lives behind a sign-in. And almost nothing automatically verifies that surface end to end.
Think about what it takes to test a real authenticated flow: a browser has to sign in as a real user, hold the session, navigate the actual app, fill the actual forms, and confirm the running product responds correctly — not a mock of it. That's exactly the work that's tedious to script, brittle to maintain, and the first thing an agent skips. So the most valuable, most fragile part of your app is the least tested part.
You don't find out until a user does.
What "actually verifying the running product" means
The fix isn't more unit tests. It's a different question. Not does this function return the right value but does a real person, signed in, get the right result from the live app.
Concretely, that means a test that:
drives a real browser against your real running app, not a mock of it;
signs in through your actual auth — the behind-the-login surface, not just the public homepage;
walks the flows a user walks (create the thing, edit it, check it shows up) and asserts on what the user would actually see;
tells you the truth when it breaks — a real product bug — instead of a green checkmark that means nothing.
This is what Beryl does. You point it at your app, it crawls the running product the way a user would, it generates and runs browser tests against the live experience — including the parts behind your login — and when something is genuinely broken, it says so. It's the verification layer your AI agent isn't going to give you, because the agent's incentive is to finish, and verification is the part that slows finishing down.
If you take one thing from this
Next time an agent tells you a feature is done and the tests are green, ask one question: did anything actually run the product, signed in, the way a customer will? If the answer is "the tests passed," that's not the same answer.
That gap is small until it isn't. Closing it is the whole job.
