AI-Generated Code Testing Beyond Happy Paths

AI-generated code testing should start with the task contract, then attack the boundaries the generated happy path is likely to ignore. Test valid and invalid inputs, missing identity, wrong permissions, empty and maximum values, duplicate delivery, retries, concurrent requests, schema compatibility, partial failure, and rollback. Run deterministic checks on the exact final diff and keep a human responsible for product intent and release risk.

Do not begin by asking whether a detector thinks the code was written by AI. Authorship is not correctness. A detector cannot prove the implementation matches your requirement, respects a license, preserves authorization, or survives production timing.

Build The Matrix Before More Code

Turn the feature contract into failure dimensions. Not every row applies to every change, but every omitted row should be a conscious decision.

I would rather choose six relevant rows and test them well than paste this whole table into a prompt and accept shallow output for every line. The change's consequence decides the depth.

Start With The Requirement, Not The Diff

Generated code can be internally consistent and still implement the wrong behavior. Give the tester the original task contract before showing the implementation summary.

A useful contract names:

• observable inputs and outputs;
• behavior that must remain unchanged;
• authorization and data ownership;
• side effects and external calls;
• supported retries and duplicate behavior;
• rollout and compatibility constraints;
• checks that define acceptance;
• conditions that require a stop rather than a guess.

If the requirement says "owners can archive their draft," the tests should not stop at one owner success. They should ask what a different user, an editor role, an already archived item, a published item, and a repeated request do. Which cases are valid depends on the product rule. The point is to make that rule visible.

Do not let the agent infer policy from the code it generated. That turns implementation into its own specification.

Test The Contract Boundary

The first tests should prove the named behavior and its nearest exception. For a pure function, that may be a small input table. For a request handler, it includes the request and response shape. For an asynchronous job, it includes the event contract and durable side effect.

Use examples that distinguish competing implementations. A test that says a formatter returns a string is too weak if the contract cares about preserving Unicode or collapsing whitespace.

I ask three questions of every acceptance test:

1. Would the old defect fail this test?
2. Could a wrong implementation pass it for an irrelevant reason?
3. Did the test preserve the actual product rule after the fix?

The second question catches mocks that bypass the changed boundary, assertions that check only status codes, and tests that never observe the durable effect.

Push Input Past The Friendly Example

AI-generated implementations often mirror the examples in their prompt. That can leave the spaces around those examples almost untouched.

For each input, consider:

• absent versus present-but-empty;
• zero, one, normal, and maximum supported size;
• repeated separators or whitespace;
• Unicode normalization and non-ASCII characters where supported;
• duplicate identifiers;
• ordering differences;
• timestamps at day, timezone, and expiry boundaries;
• numeric precision and unit conversion;
• unexpected extra fields;
• old clients omitting a newly introduced field.

Do not invent a maximum merely to add a boundary test. Find it in the schema, platform policy, or product contract. If no limit exists, that may be the issue to resolve before implementation.

Property-based testing can help when invariants are clearer than examples, but it is not automatic truth. The generator and invariant still encode human assumptions.

Test Permissions From The Wrong Side

The happy path normally uses a valid account with access to its own object. That proves almost nothing about authorization.

Build principals deliberately:

Check the side effect as well as the response. A handler can return a denial after a write already occurred. Verify that unauthorized attempts leave no new row, file, message, or external call.

Error shape and timing can leak whether another user's resource exists. Whether that matters depends on the system's threat model, but the tester should ask.

Repeat Every Side Effect

Networks retry. Users double-click. Queues redeliver. Providers send the same event again. Test duplicates with the exact same identity, not merely two similar payloads.

For a side-effecting operation, define what repeated delivery means:

• one durable result and a stable success response;
• a clear conflict with no second effect;
• a safe update of the existing record;
• a deliberately repeated action.

Then send the fixture twice and inspect the database or external-call recorder. A response assertion alone may miss duplicate writes.

Move the failure point. What happens if the process crashes after the external provider accepts the request but before the local record commits? What if the local write succeeds and the acknowledgement is lost? The correct design depends on idempotency keys, transactions, reconciliation, and the provider contract. A generated retry loop without that model can multiply the damage.

Create A Real Race

Sequential tests do not prove concurrent behavior. If two requests can legitimately arrive together, run them together against a database or synchronization boundary that behaves like the real one.

Useful race targets include:

• claiming the last available slot;
• applying one coupon twice;
• creating a resource with a supposedly unique key;
• incrementing a balance or counter;
• refreshing an expired token;
• processing one queue event on two workers;
• editing a record from two versions.

Assertions should inspect both responses and final durable state. If the rule says exactly one reservation wins, verify one success, one defined loser, and one stored reservation.

I do not trust a sleep-based test that happens to pass once. Use barriers, transactions, worker coordination, or repeated stress where the stack allows it. Keep database constraints as the final protection when uniqueness or conservation matters. Application-level "check then insert" logic is vulnerable when both requests pass the check before either inserts.

Test Migrations As A Time Window

A migration is not just a final schema. During a rolling release, old application instances, new instances, background workers, and partially migrated data may coexist.

Test the sequence the release will actually use:

1. Start from a representative old schema and data set.
2. Apply the migration in a disposable environment.
3. Run compatibility checks required while old code may still execute.
4. Deploy or run the new code against the migrated state.
5. Verify backfill assumptions and constraints.
6. Rehearse rollback when it is safe, or document the forward-repair plan when it is not.

Generated migration code deserves the same review as application code. Look for table rewrites, long locks, non-null columns without safe population, lossy conversions, missing indexes, irreversible operations, and defaults that behave differently on existing rows.

Never infer production safety from a migration succeeding on an empty local database.

Inject Partial Failure

Happy-path tests usually let every dependency answer immediately. Real systems fail between steps.

For a multi-step action, draw the side effects in order:

validate -> write pending row -> call provider -> store provider id -> publish event

Now interrupt each arrow. Ask what durable state remains, whether retry is safe, what the user sees, and how an operator can repair the result.

A mock can produce a timeout, malformed response, rate-limit error, connection reset, or success followed by a local commit failure. The exact cases should come from the dependency contract, not from a random list of exceptions.

I want errors to retain enough context for diagnosis without logging secrets or personal data. A test can assert that an event identifier and safe error category are present while credentials and raw sensitive payloads are absent.

Test Rollback Or Forward Repair

"We can roll back" is not evidence. A code rollback may fail after a schema change or after new-format data has been written.

For reversible changes, rehearse the reversal in a disposable environment and confirm the old version can read the result. For irreversible changes, design a forward repair and identify the trigger, owner, and data needed to execute it.

Ask what happens to work already in flight. Queued jobs may carry an old payload. Browser clients may keep an old bundle. A provider may retry yesterday's webhook after today's schema deploy. Compatibility tests should reflect the overlap window rather than pretending release is instantaneous.

Review The Test Changes Too

An agent can make its own code look correct by weakening the proof. Inspect test diffs with the same skepticism as production files.

Red flags include:

• deleted assertions;
• broader mocks that bypass the changed code;
• skipped or focused-only tests committed accidentally;
• snapshots updated without reading the behavioral difference;
• expected errors replaced with generic success;
• timing loosened until a race disappears;
• fixtures changed so the original failure no longer occurs;
• coverage added for lines but not decisions.

Keep the original failing case visible. If the contract changes, review that as a product decision and explain why the old assertion is no longer correct.

OpenAI currently positions Codex review as an additional reviewer, not a replacement for human review. Its Codex upgrade guidance also emphasizes command output and test results as evidence. Use an agent to search for missing cases, then have a person judge whether the test actually represents the system.

A Worked Hypothetical Matrix

Consider a hypothetical endpoint that lets a project owner archive a draft. This is an invented fixture, not a claim about a real Vibe Built application.

The contract says:

An authenticated project owner can archive a draft project.
Archiving is idempotent.
Published projects cannot be archived through this endpoint.
Non-owners receive the existing not-found response.
Every successful first archive writes one audit event.

The initial matrix becomes:

Suppose the generated handler passes the first row and fails the duplicate row by writing another audit event. The right correction is not necessarily an in-memory flag. The reviewer should inspect transaction boundaries, uniqueness identity, audit semantics, and concurrent behavior before choosing a fix.

This small matrix creates more information than a large suite of happy-path variations. Each row protects a distinct boundary.

How I Choose The First Five Tests

When time is limited, I do not begin with whatever cases are easiest to generate. I start with the places where the implementation crosses trust or makes a durable promise.

First, I keep the acceptance case that would fail the old behavior. Without it, I cannot tell whether the patch solved the stated problem. Second, I test the wrong principal whenever identity or ownership exists. Third, I repeat the side effect. Fourth, I force the most plausible partial failure around a database or provider boundary. Fifth, I test the release-specific compatibility risk, such as old payloads against new code or new data against old code.

That order changes with the task. For a parser, hostile and boundary inputs move to the front. For a migration, representative old data and coexistence matter before an HTTP permission case. For a static component, keyboard behavior and rendering states may be the real boundaries. I want the matrix to follow consequence, not imitate a backend checklist where it does not belong.

I also keep one honest line between unit proof and system proof. A unit test can demonstrate a branch under controlled inputs. It cannot tell me that the route applies the function, the database constraint exists, or the deployed artifact includes the change. When I cannot run the integration boundary, I record the unit pass and the missing integration check separately.

My stopping rule is equally important. If I cannot state the expected result for a permission, retry, or rollback case, I stop generating tests and ask for the product or operational decision. An impressive suite full of guessed expectations makes the uncertainty harder to see.

When A Generated Test Is Actually Useful

I keep a generated test when it makes a failure observable, follows the repository's real test boundary, and would fail under a plausible regression. The test should help a future maintainer understand the contract without rereading the agent transcript.

I discard or rewrite it when it mirrors the implementation line by line, mocks away the behavior under review, asserts private structure with no product meaning, or passes only because the fixture cannot reach the branch. Generated tests often look thorough because they contain many cases. I read the arrangement and assertion before counting them.

A good challenge is to introduce a small, deliberate regression after the test exists. If removing the authorization check, uniqueness boundary, or normalization behavior does not make the relevant test fail, the proof is weaker than its title. Revert the mutation afterward and rerun the real suite. This is a focused sanity check, not a reason to turn every small project into a full mutation-testing program.

I would keep the test names behavioral too. "Rejects a non-owner without writing an audit event" carries more contract than "handles error case 3." When the expected behavior changes later, a reviewer can see which promise is being revised.

How To Check If Code Is AI-Generated

You usually cannot establish code authorship reliably from style alone. Repeated comments, generic names, unusual abstractions, or confident mistakes may raise a review question, but humans produce them too. Detector output is not provenance.

Use repository history, pull-request records, agent logs, task transcripts, and policy disclosures to understand how a change was produced. A verified commit signature can connect a commit to a signing identity or key under the hosting platform's verification rules. It does not establish whether a human or an AI generated the code. Even a complete production record would not prove the code is correct.

If a policy requires disclosure of generated contributions, enforce the disclosure in the contribution process. Do not make a probabilistic code detector the gate that decides whether a security flaw can merge.

There Is No General 30 Percent Rule

There is no universal software-engineering rule saying a particular percentage of generated code is safe, detectable, legally acceptable, or sufficiently tested. A percentage without a defined denominator and source is not a standard.

Measure outcomes that matter instead: accepted tasks, escaped defects, false-positive review findings, manual repair time, boundary coverage, rollback success, and unverified conditions. One generated line in an authorization policy can carry more risk than hundreds of lines of static presentation code.

Is It Legal To Code With AI?

Legality and policy depend on the code, source material, licenses, contracts, jurisdiction, employer rules, data handling, and tool terms. Copyright is only one part of that analysis; the US Copyright Office's current Copyright and Artificial Intelligence report addresses the human-authorship boundary for copyrightability rather than granting blanket clearance to use generated code. Testing cannot answer the wider legal question by itself, and this article is not legal advice.

Keep provenance and dependency records, avoid feeding restricted code or data into unapproved systems, review generated output for copied or incompatible material, and involve qualified counsel when the consequence warrants it. Do not treat the absence of a detector alert as legal clearance.

A Practical Release Gate

Before accepting AI-generated code, I would require this record:

Contract:
[observable behavior and invariants]

Changed boundaries:
[input, identity, permission, persistence, external systems]

Failure matrix:
[relevant cases and observed results]

Commands:
[exact commands, exit status, and summaries]

Diff review:
[material findings and resolutions]

Migration or rollback:
[rehearsal evidence or not applicable]

Release proof:
[artifact and scoped runtime check]

Unverified:
[known gaps]

This sits naturally inside a broader AI coding workflow. The AI code review guide covers the separate question of how to evaluate review findings without confusing them with proof.

Make The Code Fail On Purpose

The generated happy path is the beginning of testing. Push on the contract, identity, permissions, durable state, retries, concurrency, migrations, dependencies, and reversal. Preserve the exact final diff and command evidence. Keep uncertainty visible.

You are not trying to prove that an AI wrote good code. You are trying to prove that this specific change behaves acceptably inside this specific system, and to know where that proof stops.