AI Coding Assistant Showdown: One Feature, Your Scorecard

The fastest way to judge an AI coding assistant is to give it one small feature with a written contract, a test it must add, and a command it must run, then score what comes back on six things you can see in the diff. This article gives you that feature, the rubric, and a matrix of which assistant modes can even attempt it. It also lists what the major tools cost today, from their own pricing pages, because the answer to "which one" depends on the mode you need more than on the brand.

I'm not publishing my own scores. I didn't run this showdown across every tool this week, and a table of minutes and bug counts from a task nobody else can rerun is exactly what the pages ranking for this query already offer. What I can give you is a protocol that takes about ten minutes per tool and a set of documented boundaries that don't move between runs.

What An AI Coding Assistant Is

IBM's definition is fine as far as it goes. An AI coding assistant is software wrapped around a large language model with tools, integrations, and a workflow attached, so it can generate, explain, review, and change code inside the place you already work.

The useful distinction is by mode, because the same product often ships all four and they behave like different tools.

Inline completion predicts the next few tokens at your cursor. A chat panel answers questions and proposes code you paste. An editor agent edits multiple files and can run commands inside the IDE with your approval. A terminal agent does the same from a shell with no editor at all. When someone says a tool is "good" or "bad", the first thing I want to know is which of those four they were using.

The Feature Spec

Create a scratch folder with two Python files. Nothing to install, since the test runner is in the standard library.

notes_cli.py:

import sys

NOTES = [
    ("2026-08-30", "Ship the fix"),
    ("2026-09-02", "Write the changelog"),
    ("2026-09-10", "Rotate the API key"),
]


def list_notes(notes):
    return [title for _, title in notes]


def main(argv):
    for title in list_notes(NOTES):
        print(title)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

test_notes_cli.py:

import unittest

from notes_cli import NOTES, list_notes


class ListNotesTest(unittest.TestCase):
    def test_lists_every_title_by_default(self):
        self.assertEqual(
            list_notes(NOTES),
            ["Ship the fix", "Write the changelog", "Rotate the API key"],
        )


if __name__ == "__main__":
    unittest.main()

Run python3 -m unittest -v once. One test, passing. Commit or copy the folder so the diff later is unambiguous.

Now the contract you hand to every assistant, word for word:

Add a --since YYYY-MM-DD option to notes_cli.py.

- With --since, list only notes dated on or after that day.
- An invalid date exits with status 2 and prints one line to stderr.
- Without --since, behavior is unchanged.
- Add tests to test_notes_cli.py for the filter, the boundary day,
  and the invalid date.
- Change only these two files. Add no dependencies.
- Run python3 -m unittest -v and show the full output.

The spec is small on purpose. Every line is checkable, and two of them (the boundary day, the exit status) are the kind of detail a model skips when it's pattern-matching on "add a date filter". The prompting guide covers why written constraints beat conversational ones; this is that idea reduced to seven lines.

The Scorecard

Score each row 0, 1, or 2. Total out of 12. Write the number down before you form an opinion about the tool, because the opinion arrives first and the diff arrives second.

Two rows need a note. Proof is scored on evidence you can see, which means a chat panel tops out at 1 on that row by design. That isn't unfair. It's the boundary of the mode, and you should know it before you pay for it. Cost is scored on whether the tool shows you what the task consumed, since the pricing pages below make clear that most paid plans are now a monthly allowance plus overage.

For the test-quality row, the AI-generated code testing post has the longer version of "would this fail on the old code". Here, the check is quick. Revert notes_cli.py, rerun the tests, and confirm the new ones fail.

Which Modes Can Even Attempt It

Before any brand comparison, this is the matrix that decides the outcome.

Inline completion can score a perfect 12 on a one-line change and tops out well short of 12 on this spec, because it never runs anything and can't tell you what it consumed. That is the whole reason the spec has a test-run bullet. If your daily work is mostly one-line changes, completion quality is what you should be testing, and this fixture is the wrong one.

Editor and terminal agents are the only modes that can reach 12. Between them, my read is that the difference is where the approval prompts happen and how readable the diff is afterward, not raw capability. The Claude Code tutorial walks the terminal version of this loop on a similar fixture.

What To Look For In The Diff

The spec is short enough that the returned diff is short too, which is what makes the rubric quick to apply. A few specifics are worth reading for before you score anything.

Date parsing is the first tell. Python's standard library has date.fromisoformat, which raises ValueError on bad input, so an implementation that hand-rolls a regex or splits on hyphens has chosen more code than the contract needs and probably handles fewer bad inputs. The exit status is the second. The spec says status 2 and one line on stderr, so check where the error actually goes. Printing it to stdout, exiting with 1, or letting an uncaught exception produce the non-zero status all fail the contract row even though the filter itself works.

The boundary test is the third, and it's the one I'd read most carefully. "On or after" means a note dated exactly on the --since day is included, so the test needs a note whose date equals the filter value, and the fixture already contains 2026-09-02 for that purpose. If the added test only checks a date strictly before and strictly after, the boundary case is untested and the row scores 1 at best.

Last, look at how main grew. An implementation that keeps list_notes pure and does the argument parsing in main is easier to test and to read than one that pushes sys.exit calls into the helper. The rubric doesn't score style directly, and I wouldn't add a row for it, but this is where two implementations that both score 12 start to differ in the way that matters six months later.

What Each Assistant Gives You, Priced Today

Prices and limits below are from each vendor's pricing page as fetched on 2026-09-15. They change often, so treat the links as the source and the table as a snapshot.

Two things in that table surprised me while checking. Cursor's page now shows a single Individual card at $20 rather than the Pro, Pro+, Ultra ladder it used to list, with usage-based overage after the included amount. And JetBrains AI Free is genuinely unlimited for completion, though only in IDEs where you already hold the right subscription, so it is free in the sense that your IDE licence already covers it.

Windsurf is missing because its pricing page redirected to Devin's and that page refused the fetch twice. I'd rather leave a gap than paste a number from a third-party roundup.

Is There A Free AI Code Assistant?

Yes, several, and the useful question is free for which mode.

For inline completion, Copilot Free gives 2,000 completions a month, Gemini Code Assist for individuals gives 6,000 a day, and JetBrains AI Free is unlimited on its own model inside a licensed IDE. For an agent that runs your tests, the free options are thinner. Cursor Hobby has limited Agent requests, Gemini's free tier caps agent mode and CLI at 1,000 requests a day combined, and ChatGPT Free includes Codex with message limits that depend on the model. Claude Code is not on Claude's free plan at all.

If you're broke and want to run this showdown, Gemini Code Assist for individuals is the widest free agent quota on the page today. Whether its output scores well is the thing you're about to find out.

Which Is The Cheapest?

On list price for a paid individual plan, ChatGPT Go at $8 a month includes Codex, and Copilot Pro and JetBrains AI Pro sit at $10. Cursor and Claude Pro are $20.

List price is the wrong number for an agent, though. Every paid plan above is an allowance. Copilot's $15 in credits at a cent each, Cursor's included usage then on-demand billing, JetBrains credits at a dollar each, Claude's shared pool. The cheapest tool is the one whose allowance covers your actual month, which is why the scorecard has a cost row. Run the fixture, read the meter, multiply.

Is ChatGPT Good For Coding?

For talking a problem through, yes, and its plans all include Codex, which is the part that touches your repository. The chat is a chat panel in the matrix above. It proposes code and cannot run your tests. Codex in the CLI or IDE is an agent and can. If someone tells you ChatGPT was bad at a coding task, ask which of the two they used.

On the API key route, Codex CLI bills at API rates instead of a subscription, which is worth knowing if your usage is bursty.

Which Is Best For Beginners?

My answer is the one that shows you the diff before it applies it and makes the test command visible, in whichever editor you already use. For a beginner, the failure that costs the most is not a wrong answer. It is an answer that looks right, touches a file you didn't notice, and passes because the assistant also edited the test. The scope and test-quality rows exist to catch exactly that.

In my monorepo the expensive mistakes have almost always been scope drift rather than bad code. A clean function in the wrong place, a helper added to a package that didn't need it, a test loosened until it passed. Those are the things a beginner won't see and a rubric will.

So start free, run the fixture, and pick whichever tool scores highest on the rows you actually care about. If that turns out to be a completion tool, you've learned you don't need an agent yet.

What Is The Best AI Assistant For Coding?

There is no answer that survives the mode question, and I'd be suspicious of any page that gives one without a fixture you can rerun. The ranked list of tools I ship with exists on this site, and it's honest about being my preference for my work. This protocol is how you'd find out whether that preference transfers to yours.

If you want a shortlist to run the spec against, I'd take one editor agent and one terminal agent, plus the completion tool your IDE already bundles. Three runs. Half an hour. You'll come out with three numbers out of 12 and a folder of diffs, which is more evidence than any roundup on the first page of results is offering.

After The First Run

Change one thing and run it again. Drop the "show the full output" line and see which tools still run the tests unprompted. Add a third file the spec doesn't mention and see which tools touch it. Swap the invalid-date bullet for a timezone rule and see which tools ask a question instead of guessing.

Each variation moves one row of the scorecard, and the tool that stays stable across variations is the one worth paying for. Keep the diffs. When the pricing pages change again next quarter, the diffs will still tell you what you bought.