Tool families

AI coding tools: evaluate an assistant in your repository

Generation speed does not show whether code fits a project. A useful trial takes one bounded task in a real repository, with tests, review and a way to undo changes.

Development workstation with a code screen
AI-generated illustration
47 active services in the directory

Services to examine

These active services are starting points from the directory. Editorial highlighting and score order this selection; they do not replace a trial on your own task.

Code & buildaccount

Amazon Q Developer

coding agent

Amazon Web Services · US

Official site

Distinguish workflows before choosing

Needs within the same family require different functions. Start with the expected result to narrow your shortlist.

Need, workflow and evidence to inspect
Your needWorkflow to examineDecisive check
Complete codeAssess focused assistance on a bounded change and its context.Review the diff and run relevant tests, including the originally failing case.
Delegate a changeA repository agent can chain reading, commands and edits.Define file scope and inspect command effects before accepting the result.
Build an application from a briefA visual prototype does not validate authentication, storage or maintenance.Test access rights, form errors, persisted data and failure recovery.

When to explore this family

Editorial illustration

Explain an area of code before proposing an edit.

AI-generated illustration

Fix a reproducible defect and verify the affected behaviour.

Document research with an atlas and books in a library
AI-generated illustration

Build a limited prototype, then inspect its dependencies.

How to compare tools

  1. Creative studio with a camera and drawing tablet
    AI-generated illustration

    Compare the proposal with existing conventions and dependencies.

  2. Editorial illustration

    Run meaningful tests and review the diff line by line.

  3. AI-generated illustration

    Check permissions, repository access and data handling.

Measure what actually matters

Choose criteria before starting. These are practical checks to adapt to your task, not measured performance results for directory services.

Checks for this family
DimensionHow to checkPitfall
BehaviourRun relevant tests and check an edge case related to the change.Compiling code can still produce an incorrect result.
ScopeRead the complete diff, dependencies and executed commands.A small request may trigger unnecessary changes.
MaintenanceReview the patch against project context and conventions.A test that mirrors implementation can miss the defect.
Reproducing the patchApply the diff to a clean copy with locked dependencies, then run the documented validation command.A patch depending on an untracked local file or an implicit secret is not reproducible.
How should results be interpreted?

Separate blocking errors from minor defects. Compare identical inputs, retain failed attempts and include human time. A small trial helps uncover defects; it does not prove general reliability. Repeat when the model, data or configuration changes.

Spot a failure before choosing the tool

These fictional examples show how to inspect an output. They do not describe observed incidents involving the services on this page.

Symptom, example and check to perform
Warning signFictional exampleHow to check
The normal case passes, edge cases failA function works for three rows but fails on an empty list.Test empty input, missing values, special characters and size limits in an isolated environment.
A dependency is assumed availableCode calls a function absent from the installed version.Compare the actual version with official documentation, then execute the affected path.
A fix touches too many filesA local change also modifies an unrelated setting.Review the diff, narrow the scope and check existing behaviour still works.

Practise with documents and answer keys →

A repeatable trial

Starting situation. In a small test repository, fix a known defect without changing neighbouring behaviour. Prepare a test that fails before the fix.

  1. Wooden desk with a laptop and monitor
    AI-generated illustration

    Baseline

    Give two assistants the same ticket, files and failing test. Note the commands each proposes to run.

  2. Library with books and a research notebook
    AI-generated illustration

    Diff review

    Inspect changes line by line: scope, new dependencies, secret access, error handling and readability.

  3. AI-generated illustration

    Evidence

    Run the regression test and project checks. Measure human review time and how easily the change can be rolled back.

Evidence for a decision

Laptop and folders on a private desk
AI-generated illustration
Output
A bounded, readable change validated in the target repository.
Scientific papers connected by threads on a cyan and violet lit research table.
AI-generated illustration
Checks
Reproduction, diff, existing tests and adjacent behaviour.
Two screens show queued tasks and their results in a nighttime workspace.
AI-generated illustration
Evidence
Branch, instructions, final diff, commands and test results.
Participants comparing meeting notes around a conference speakerphone and tablet.
AI-generated illustration
Failure signal
A fix without reproduction or with out-of-scope changes.
Compare actual cost and keep a decision record →

Related guides

Questions before choosing

Is compiling code enough?

No. It must solve the requested case, preserve nearby behaviour and pass relevant tests.

What else should be measured besides generation speed?

Count rework, useful tests, out-of-scope changes and review time.

What access should an agent receive?

Limit access to the repository and actions needed for the test; keep secrets and sensitive operations out of reach.

Explore other tool families