AI Code Review: What It Catches, Misses, and Needs

AI code review is useful as a fast first pass over a diff. It can spot a missing null check, suspicious input handling, inconsistent error path, dead code, or an edge case that deserves a test. It cannot decide whether the feature matches the product requirement, whether the architecture is worth maintaining, or whether a plausible change is safe in the real system.

Use it before human approval, not instead of human approval. Give the reviewer the task contract and relevant repository context, require file-and-line evidence, run the actual tests, and let a person own the merge decision.

That is the practical answer. The rest of this guide shows how to evaluate an AI reviewer without trusting its own marketing or its most confident comment.

My Rule for AI Review

On my own code, I treat an AI finding as a bug report from a fast stranger. It is useful when it names a failure I can reproduce and weak when it substitutes confidence for evidence. I ask for the file, line, path to failure, and smallest verifying test. If I cannot reproduce the problem, I inspect the missing context before changing code.

Deterministic checks remain in charge of deterministic facts. The model searches for risks those checks do not express, and a person owns the merge. I would rather receive three supported findings than twenty plausible comments because my attention is part of the review budget. The real defect should not be buried under style noise.

Authentication, money, deletion, private data, migrations, and irreversible external actions need a wider review than one suggested line. I trace the request, persistence, retry, rollback, and audit path before accepting a fix. The companion guide to vibe coding security risks covers the security guardrails in more depth.

What AI Code Review Actually Does

An AI reviewer reads some combination of a pull-request diff, surrounding files, repository instructions, issue text, and tool output. It then predicts which changes look risky and explains suggested fixes in natural language.

That is different from a compiler, linter, type checker, test suite, or static security rule. Those tools evaluate explicit properties. An AI reviewer reasons across less structured context, which gives it broader reach and less predictable precision.

Modern review products can gather repository context and sometimes run tools. GitHub's current Copilot code review documentation describes full-project context gathering and suggested fixes, while warning that the reviewer can miss problems and make mistakes. OpenAI's Codex code review description says Codex can reason across a codebase and execute code and tests, but still recommends it as an additional reviewer rather than a replacement for humans.

So "AI reviewed it" does not identify one fixed level of assurance. Ask what context it saw, what tools it ran, what files it excluded, and what evidence supports each finding.

What AI Review Is Good At

The best findings usually have a short path from changed line to failure.

Local Logic Errors

An early return skips cleanup. A nullable value is dereferenced. A boundary check uses > where >= was intended. An asynchronous call is not awaited. These bugs are visible in a small neighborhood of code and have familiar shapes.

Missing Edge Cases

Empty lists, duplicate events, zero values, missing fields, retries, and concurrent requests are useful review prompts. The reviewer may not prove the bug, but it can point to a case the patch does not visibly handle.

Inconsistent Error and Authorization Paths

If nineteen neighboring handlers validate a permission and the new one does not, repository context can expose the mismatch. The same applies to a new route returning a different error shape or failing to close a resource.

Review Triage

A concise map of changed behavior can help a human decide where to spend attention. This is especially useful in a large generated diff, where the main risk is not one syntax error but the reviewer becoming tired before reaching the important file.

Test Suggestions

An AI reviewer can turn a suspected edge case into a concrete test idea. That is often more valuable than the comment itself, because the test can prove or disprove the concern.

What AI Review Commonly Misses

The hardest bugs need information that is absent, ambiguous, or distributed across systems.

Product Intent

Code can be internally correct and still implement the wrong business rule. A discount applies cleanly to the wrong customer group. A list is filtered by a valid but incorrect field. The diff cannot reveal the missing requirement unless you supply it.

Cross-System Reality

A webhook handler may look idempotent inside one file but still duplicate work because of queue retries, database isolation, or a provider's delivery behavior. A reviewer needs the contract, infrastructure, and failure model, not just the function.

Architecture and Future Cost

The model can describe coupling. It cannot own the trade between a quick release and six months of maintenance. It does not know which feature is about to be deleted or which team must support the abstraction.

Missing Code

Review systems naturally focus on what changed. The important defect may be work the patch should have included but did not: a migration, permission, rollback, telemetry event, accessibility state, or documentation change.

Runtime and Environment Behavior

Feature flags, secrets, clock boundaries, real data distributions, browser behavior, database load, and network failures do not become true because a comment says the code looks safe. They need tests or observation in the relevant environment.

Taste and User Harm

A reviewer can flag a missing label. It is much weaker at deciding whether the flow confuses users, creates a dark pattern, exposes private information through an otherwise valid feature, or solves the wrong problem.

A Reproducible AI Reviewer Test Matrix

Do not evaluate a code reviewer on whatever pull request happens to be open. Use the same known defects, safe non-defect, prompt, and context every time, then rerun the cases when the tool or model changes.

I published the exact AI code review fixture pack used for this matrix. Start a fresh reviewer session for each case, provide only the listed task contract and allowed context, save the raw output, and score whether the reviewer named the changed line, failure path, and smallest useful proof. Run the intent fixture once without its contract and once with it. That controls enough of the setup for another reader to challenge the result instead of taking a summary table on trust.

This eight-case matrix is enough to expose the basic shape of a reviewer:

Track four numbers:

1. True positives: real defects it identified.
2. False positives: safe behavior it called broken.
3. False negatives: seeded defects it missed.
4. Evidence quality: findings that named a path to failure and a way to verify it.

Do not combine those into one magical score too quickly. A security-sensitive repository may accept more false positives to reduce missed authorization bugs. A busy team may need high precision because noisy comments teach people to ignore the reviewer.

The intent-mismatch case is the most revealing. Run it once with a vague pull-request description and once with an explicit acceptance criterion. The difference shows how much the reviewer depends on a good task contract.

For example, suppose the reviewer catches the null path, boundary error, injection, missing ownership check, duplicate event, and missing migration, but misses the intent mismatch until the contract is supplied and complains about the safe loose-null comparison. I would record six initial detections out of seven seeded defects, one unsupported finding, and one intent detection gained from context. I would not hide the false positive by averaging everything into one score. Save the raw comment too. A later run that catches all seven defects while producing five unsupported warnings may be worse for a team that has stopped reading noisy reviews, even though its recall looks stronger; if the product or model version changes, rerun every fixture under the original protocol before comparing the results because a new context window, extra repository file, or altered prompt can masquerade as an improvement in the reviewer.

The Safer Review Loop

AI review works best inside a layered process.

1. Define the Change Contract

Before implementation, write the observable behavior, constraints, and risky boundaries. Include examples. If money, authentication, deletion, privacy, or external side effects are involved, say so explicitly.

2. Run Deterministic Checks First

Formatters, linters, type checks, unit tests, integration tests, dependency checks, and security rules should handle properties they can determine. Do not spend model attention rediscovering a formatting violation the linter can prove.

3. Ask the AI to Review the Diff and Contract

Provide the changed files, task contract, relevant repository instructions, and test results. Ask for evidence, severity, uncertainty, and the smallest verification step.

4. Verify Every Material Finding

Reproduce the failure or add a test. A plausible comment is a lead, not a verdict. If you cannot reproduce it, inspect whether the reviewer lacked context or invented behavior.

5. Perform Human Domain Review

Read for intent, system interactions, maintainability, user impact, and what is missing. Pay extra attention to money, auth, permissions, private data, destructive actions, and concurrency.

6. Re-Run Checks After Fixes

A suggested fix is another code change and can introduce another bug. Run the affected checks again and review the final diff, not the earlier version the model saw.

A Review Prompt That Produces Better Evidence

This is the shape I use when a coding agent reviews a local diff:

Review this diff against the task contract below.

Prioritize:
1. Runtime or data-loss bugs
2. Authentication, authorization, injection, and secret exposure
3. Empty, null, duplicate, retry, and concurrency cases
4. Missing migrations, tests, rollback, or observability required by the change

For every finding, provide:
- severity
- file and line
- the concrete failure path
- the smallest test or command that would verify it
- your uncertainty if context is missing

Do not comment on style unless it hides a defect.
Do not claim a command ran unless you show its result.
If no material issue is supported, say so.

Task contract:
[paste the observable behavior and constraints]

The phrase "against the task contract" matters more than asking the model to act like a senior engineer. A role prompt cannot supply a missing requirement.

AI Review Does Not Replace Required Approval

GitHub makes the boundary visible in its own product. Copilot review comments can help, but GitHub's documentation for required pull-request reviews says Copilot approvals do not count toward required-review rules.

That product choice matches a safer operating model. A bot can inspect and suggest, while an accountable person or team decides whether the change meets the bar.

There is another practical limit. GitHub's current Copilot review docs list file types it excludes, including dependency-management files, log files, and SVG files. Another reviewer may have different exclusions. If a critical change lives in a file the tool does not inspect, a green-looking review says nothing about it.

Should You Pay for a Dedicated AI Reviewer?

A dedicated reviewer is worth paying for only when it improves the review workflow enough to justify its cost and verification burden.

A dedicated pull-request reviewer may be worthwhile when it:

• reviews automatically at the point your team already works;
• understands repository instructions and relevant context;
• keeps noise low enough that people read the comments;
• provides controls for private code and retention;
• shows what files and tools it used;
• fits your required review and branch-protection process;
• saves more review time than it creates in verification work.

If you already use a repository-aware coding agent, review the local diff before opening a pull request and compare that with a dedicated service on the same fixture matrix. Do not rely on a volatile price table in an article. Check the current plan, limits, data terms, and billing unit on the vendor's own site when you are ready to choose.

AI Code Review Questions, Answered

Is AI Code Review Any Good?

Yes, for a first pass over local defects, edge cases, inconsistent patterns, and missing tests. Its value falls when the bug depends on unstated intent, production behavior, or a decision outside the diff.

Can AI Replace Human Code Review?

No. It can reduce mechanical review load and focus human attention. It cannot own the product decision, operational risk, or accountability for the merge. Both GitHub and OpenAI explicitly describe human validation as necessary in their current guidance.

Is a Second AI Reviewing AI-Written Code Pointless?

No, but model independence is not a guarantee. A second pass can catch local mistakes. It may also share the same blind spots or confidently approve the same wrong assumption. Deterministic tests and a task contract matter more than whether the author and reviewer have different logos.

What Should Never Be Auto-Approved?

Changes involving authentication, authorization, payments, data deletion, privacy, secrets, migrations, or irreversible external actions should keep explicit human ownership and proportionate testing. The list is a floor, not a complete risk policy.

The Bottom Line

Use AI code review. Just give it the right job.

Let deterministic tools prove what they can. Let the AI search for material risks, missing edge cases, and useful tests. Verify each serious comment. Then have a human review the intent, system context, and user consequence before merging.

If your code is being produced through prompts and agents, that review discipline becomes more important, not less. The broader workflow is covered in AI coding agents and what vibe coding actually means.

The reviewer is a sensor. Tests are evidence. The task contract is the target. The merge remains a decision.