We need a coding assistant that can run locally, inspect our repo, and make multi-file changes safely. What should we evaluate?

Evaluate it as a software-engineering system, not just a model:

• Repo understanding: navigation, dependency/context discovery, localization accuracy, and token efficiency.
• Multi-file correctness: cross-file refactors, imports, configs, tests, and preservation of unrelated behavior.
• Verification loop: runs tests/lint/build, interprets failures, iterates, and stops when genuinely blocked.
• Safety: filesystem boundaries, command/network permissions, secret protection, approval gates, rollback/checkpoints, and crash recovery.
• Diff quality: minimal, reviewable changes; no unrelated churn; clear changed-file accounting.
• Long-horizon reliability: tasks lasting 30+ minutes, context drift, partial failures, and resumption.
• Human control: preview/plan modes, explicit risky-action approvals, audit logs, and easy undo.
• Performance/cost: wall-clock time, tokens, model/API cost, tool-call count, and repeated work.
• Local operation: offline behavior, supported OSes/runtimes, model/provider flexibility, and resource consumption.
• Reproducibility: pin agent/model/runtime versions and run identical tasks against identical repository snapshots. Local benchmarking tools explicitly recommend comparing these variables rather than relying on headline leaderboards.

For a pilot, create 20–30 representative internal tasks: bug fixes, 3–10-file refactors, dependency migrations, failing-test diagnosis, security fixes, and deliberately impossible tasks. Measure task success + regression rate + unsafe actions + diff quality + time/cost, rather than one aggregate score.

Evaluating a local AI coding assistant that handles multi-file repository edits safely requires looking past the marketing hype and testing how the pieces fit together on your actual hardware . Because you need local execution, repository inspection, and safe multi-file edits, your evaluation framework should focus on four core pillars.

1. Model Capabilities & Local Hardware Fit

• Coding Intelligence: Look at state-of-the-art open coding models (such as variants of Qwen2.5-Coder or Llama-3-Instruct-based code models) that excel at function calling and structured output. A weak model will mangle multi-file refactors by hallucinating imports or breaking syntax.
• Context Window & VRAM: Multi-file editing means the model must ingest substantial parts of your repo. Ensure your local GPU/VRAM setup can handle a large context window (e.g., 3 2 k to 1 2 8 k tokens) without drastically slowing down generation times.
• Quantization Impact: Test quantized versions (like Q4_K_M or Q8_0 via Ollama ) to find the sweet spot between memory footprint and reasoning degradation.

2. Agent Harness & Repo Inspection

• Repository Indexing/RAG: How does the tool map your codebase? Check if it relies on a local vector database, AST (Abstract Syntax Tree) parsing, or raw file dumps to find relevant context across files.
• The Agent Loop: Multi-file changes require an iterative loop where the agent proposes a change, runs a verifier or linter, and self-corrects. Evaluate tools like Aider or terminal/IDE-based agents (such as Cline or OpenCode) that support structured tool-use and multi-file editing workflows.
• Local Configuration Overhead: Check how easily the harness connects to your local runtime (like Ollama or llama.cpp) and whether it requires tedious manual JSON configuration or supports seamless endpoint discovery.

3. Safety, Guardrails & Reversibility

• Diff Accuracy: The assistant shouldn't rewrite entire files if it only needs to change 5 lines. Look for precise search-and-replace block generation or unified diff application.
• Git Integration: A safe local agent should automatically commit checkpoints or stage changes in Git so you can easily run git diff or git reset if things go sideways.
• Sandboxing / Human-in-the-Loop: Ensure the tool prompts for explicit confirmation before executing terminal commands, overwriting files, or deleting code. Fully autonomous local execution without approval prompts can quietly corrupt a working build.

4. Developer Ergonomics & Integration

• Editor vs. Terminal: Decide whether your team prefers an IDE extension (VS Code/Neovim) or a terminal-centric coding partner.
• Latency & Responsiveness: Local inference can lag depending on your hardware specs. Measure time-to-first-token and overall task completion time to ensure it boosts productivity rather than frustrating developers.

To help narrow down your evaluation shortlist, tell me:

• What hardware/GPUs are available on the local machines or servers where this will run?
• Do you prefer a terminal-first tool (like Aider) or an IDE extension (like VS Code plugins)?

For a coding assistant that must run locally, inspect a repository, and safely make multi-file changes, evaluate it less like an autocomplete tool and more like a junior engineer with shell access. The key question is: does it reliably understand the repo, make bounded changes, verify them, and know when to stop? Tools in this space commonly differ in interface (terminal vs IDE), autonomy, model flexibility, and verification loops.

1. Repository understanding

Test whether it can build a useful mental model of your codebase.

Give it tasks like:

• “Explain the architecture of this repo.”
• “Find where authentication is implemented.”
• “Trace this API request from frontend to database.”
• “Identify all files affected by changing this interface.”

Measure:

• Does it find the right files quickly?
• Does it understand conventions?
• Does it avoid editing unrelated code?
• Does it handle large monorepos?

A good agent should use search intelligently rather than blindly reading everything.

2. Multi-file change quality

This is probably your highest-value test.

Create benchmark tasks that require:

• changing an API contract across services
• adding a feature touching frontend/backend/tests
• migrating a library usage
• refactoring a shared component
• updating configuration plus code

Score:

• Correct files changed
• Completeness
• Regression risk
• Diff size
• Whether the changes match existing style

Many agents look impressive on single-file fixes but degrade on coordinated multi-file work.

3. Safety and control model

A local agent should not have unlimited authority by default.

Evaluate:

• Can it run in a sandbox?
• Can you restrict filesystem scope?
• Does it ask before destructive commands?
• Can you review a diff before applying?
• Can you undo changes cleanly?

Look for:

• Git-aware workflows
• Atomic commits or checkpoints
• Clear tool permissions
• Command approval gates

Git integration is especially valuable because it makes experimentation safer. Tools such as Aider, for example, are designed around git-based workflows and local model support.

4. Verification loop

A strong coding agent should not stop after writing code.

Test:

“Implement this change and make sure the tests pass.” Measure whether it:

1. edits code
2. runs tests/lint/type checks
3. reads failures
4. fixes mistakes
5. reports remaining uncertainty

Poor agents often produce plausible patches but skip validation.

5. Model flexibility and local execution

Evaluate the whole stack, not just the UI.

Questions:

• Does it support local models?
• Which runtimes work? (Ollama, llama.cpp, vLLM, LM Studio, etc.)
• Can it use larger hosted models optionally?
• Can you swap models without changing workflows?

Some local-first tools support bringing your own model or connecting to local inference servers; this flexibility matters if privacy or air-gapped environments are requirements.

6. Hardware requirements

Benchmark on your actual developer machines.

Measure:

• Startup time
• First-token latency
• Time to complete a change
• RAM/VRAM usage
• Battery impact (laptops)
• Quality drop when using smaller models

A locally running weaker model may be slower overall if it requires many correction cycles.

7. Privacy and security

Ask:

• Does repository data ever leave the machine?
• Are telemetry and logs configurable?
• Can secrets accidentally enter prompts?
• Does it index .env, credentials, or private keys?
• Can it run offline?

Run tests with:

• private repositories
• internal naming
• secrets scanners enabled
• restricted network access

8. Developer workflow fit

Evaluate the interface your team actually wants:

Terminal-first

Good for:

• backend engineers
• CI workflows
• scripting
• Git-heavy workflows

IDE-first

Good for:

• interactive editing
• navigation
• reviewing inline changes

Autonomous agent workspace

Good for:

• larger tasks
• issue-to-PR workflows

Examples of commonly evaluated local/self-hostable options include Aider, Cline, Continue, and OpenHands, each emphasizing different workflows.

9. Create a realistic evaluation suite

I would avoid generic coding puzzles. Use your own repo and create 10–20 tasks:

Score each task:

• 0 = failed
• 1 = partially useful
• 2 = completed with minor fixes
• 3 = production-ready

10. Metrics worth tracking

For a team decision, measure:

• Task completion rate
• Human correction time
• Number of files incorrectly touched
• Test pass rate after agent changes
• Average diff size
• Developer trust rating
• Time saved vs manual work
• Cost per successful task

The most important metric is not “lines of code generated.” It is:

How often does a developer accept the change with minimal cleanup? A practical shortlist to evaluate would usually include:

• one terminal agent
• one IDE-native agent
• one highly autonomous agent
• one local-model stack

Then run the same repo tasks through each under identical constraints.

Evaluating a local, repo-inspecting, multi-file coding assistant requires looking at four core pillars: context engineering (how it reads your codebase), agentic safety (how it modifies files without breaking things), local hardware/model performance , and IDE/workflow integration.

When building your evaluation matrix, consider the following key areas:

1. Repo Inspection & Context Management
   • Repo Mapping: Does the tool generate a structural map or abstract syntax tree (AST) of your project (like Aider's repo-map or custom local indexers), or does it naively dump raw text into the context window?
   • Context Window & Efficiency: How well does it handle large codebases? Look for tools that support selective file-additions, smart symbol searching, or compressed summaries rather than forcing you to feed entire directories manually.
   • Local Embeddings & RAG: If it uses Retrieval-Augmented Generation for code search, can it run local embedding models (via Ollama or similar) to index your codebase locally without data leaks?
2. Multi-File Editing & Agent Capabilities
   • Diff Generation & Application: Can it output reliable unified diffs or block replacements across multiple files simultaneously, or does it frequently hallucinate line numbers and mangle files?
   • Autonomous Execution vs. Guardrails: Does it operate as a strict copilot (suggesting edits for approval) or an autonomous agent (like Cline or terminal agents)? Look for "approval gates" that require your sign-off before writing or executing shell commands.
   • Rollback & Git Integration: Does it automatically commit changes to a temporary git commit or stash so you can easily git reset if the model goes off the rails?
3. Model Compatibility & Local Performance
   • Instruction Following (IF) & Coding Smarts: Smaller local models (e.g., 7B–14B parameters) often struggle with complex multi-file refactors compared to frontier models. Evaluate how well open-weights models like Qwen2.5-Coder or DeepSeek-Coder perform under your specific architecture.
   • Backend Flexibility: Ensure the tool isn't locked to one inference engine. It should easily hook into local runners like Ollama, LM Studio , or vLLM.
4. User Experience & Workflow Friction
   • Interface Fit: Decide whether your team prefers an IDE extension (like the Continue extension for VS Code/JetBrains) or a terminal-native loop (like Aider).
   • Privacy & Air-Gapping: Verify that zero telemetry or code snippets leak to external third-party endpoints unless explicitly configured.

Popular open-source combinations to benchmark in your test include pairing Continue or Cline in VS Code with a local runner like Ollama , or testing terminal-based autonomous loops like Aider pointed at a strong local coding model.

To help narrow down your evaluation, what local hardware (GPU/RAM) are you working with, and do you prefer an IDE-integrated sidebar or a terminal-driven workflow?

Evaluating a local, repo-inspecting, multi-file-capable coding assistant requires looking at four critical layers: Model Intelligence, Context & Repository Mapping, Editing Mechanics & Safety , and Hardware/Infrastructure.

Popular open-source orchestration layers like Aider (terminal-based) or Continue (IDE-based) paired with a local backend like Ollama or LM Studio are standard starting points. Use this evaluation framework to test how they handle your specific codebase.

1. Repository Inspection & Context Management

An assistant is only as good as the code it can "see." Multi-file changes fail immediately if the model lacks structural awareness of your repo.

• Repository Mapping: Does the tool build a compressed, tree-like map of your codebase automatically (like Aider's repo map)? Or does it rely entirely on brute-force RAG (Retrieval-Augmented Generation) which might miss implicit cross-file dependencies?
• Context Window Handling: How does it behave when stuffing multiple files into context? Check if it supports large context windows (1 2 8 k to 2 5 6 k+ tokens) without severe degradation in attention or runaway memory consumption.
• Symbol & Definition Resolution: Can it accurately jump from a function call in file_a.py to its definition in file_b.py using local AST (Abstract Syntax Tree) parsing rather than guessing?

2. Multi-File Editing & Diff Reliability

Local models struggle significantly more than cloud giants (like Claude 3.7 Sonnet or GPT-4o) with outputting structured code edits across multiple files without breaking syntax.

• Edit Format Support: Test which diff/patch style the tool forces the model to use. Unified diffs (udiff ), search-and-replace blocks, or whole-file rewrites. Smaller local models often fail at complex diff formats and resort to "lazy coding" (truncating code with // ... rest of code).
• Simultaneous Multi-File Execution: Give the assistant a prompt that requires changing an interface in file A and updating three consuming modules in files B, C, and D. Does it modify all files correctly in one turn, or does it lose track and corrupt files?
• Architect/Editor Split: Look for tools that support an "Architect/Editor" pattern—where a smarter, slower reasoning model plans the multi-file changes, and a faster local model applies the raw text edits.

3. Safety, Guardrails & Git Integration

Local execution keeps your data private, but "safe" multi-file editing means you don't wake up to a mangled codebase.

• Git-Awareness & Auto-Commits: Does the tool automatically wrap changes in atomic Git commits so you can instantly git reset --hard if the AI hallucinates?
• Linter/Test Loop Integration: Can the assistant run your test suite or linter automatically after applying a multi-file edit, read the error output, and self-correct its mistakes before handing control back to you?
• Sandbox / Dry-Run Modes: Is there a way to preview unified diffs or review changes in a staging area before writing them to disk?

4. Local Model Performance & Hardware Constraints

Running a model smart enough to do multi-file reasoning locally requires heavy lifting.

• Model Quality: Evaluate top-tier local coding weights (e.g., Qwen2.5-Coder-32B, DeepSeek-Coder-V2, or Llama-3-Codestroyer variants) rather than smaller 7 B or 8 B models, which rarely possess the robust agentic loops required for safe multi-file changes.
• Tokens Per Second (t/s): Measure inference speed on your local hardware (Apple Silicon unified memory vs. NVIDIA enterprise/consumer GPUs). If generation drops below 1 5--2 0 t/s , long multi-file refactoring sessions become painful.

To help narrow down your toolchain evaluation, tell me:

• What hardware/GPUs do you have available locally?
• What primary programming languages make up your repository?

If your requirement is “runs on my machine, can understand a real repo, make coordinated multi-file edits, and not trash the working tree,” I’d evaluate the agent + execution harness, not just the underlying model. Current agents differ substantially in sandboxing, approvals, repo navigation, and long-running task behavior.

The evaluation dimensions

1. Repo understanding

Give it a deliberately unfamiliar repo and measure:

• Can it find the relevant code without being spoon-fed filenames?
• Does it understand architecture and dependencies?
• Does it inspect tests, configs, lockfiles, generated code, etc.?
• Does it avoid reading the entire repository unnecessarily?
• How does it behave on a 100K+ LOC monorepo?

Test: “Add feature X” without telling it which files to modify. Score the files it identifies before allowing edits.

2. Multi-file change quality

This should be your core benchmark.

Use tasks requiring changes across:

• implementation
• tests
• configuration
• API/schema/types
• documentation or fixtures

Prefer real historical issues from your backlog. Benchmarking sources specifically recommend 3+ file tasks because single-file benchmarks hide important failures.

Measure:

• Correctness
• Number of files unnecessarily touched
• Regressions
• Whether related tests are updated
• Whether it preserves existing conventions
• Whether it leaves dead/duplicate code

3. Edit safety and reversibility

This is almost as important as coding ability.

Look for:

• Git-aware operation
• Clear diffs before/after changes
• Ability to undo/revert an individual turn
• Worktree/branch support
• Workspace-scoped filesystem permissions
• Approval gates for destructive commands
• Network-access controls
• Protection against modifying files outside the repo

For example, current CLI agents expose different combinations of read-only/workspace-write/full-access modes and approval policies.

Red flag: an agent that expects you to give it unrestricted access to your home directory.

4. Command execution

Don't just ask whether it can run commands. Test how responsibly it does so.

Give it a failing test and see whether it:

1. reproduces the failure,
2. investigates,
3. makes the smallest reasonable fix,
4. reruns the relevant test,
5. runs the broader suite,
6. reports remaining failures honestly.

Also test:

• build systems
• linters
• formatters
• migrations
• package managers
• shell scripts
• flaky tests

5. Failure behavior

This is a surprisingly important differentiator.

Give it tasks that are impossible or underspecified:

• missing dependency
• unavailable credential
• broken upstream service
• ambiguous product requirement
• intentionally failing test with misleading error message

A good agent should say “I can't establish X”, not invent a solution and quietly change unrelated code. Failure-mode handling is worth scoring separately; published evaluations similarly recommend measuring whether agents surface blockers versus hallucinating fixes.

6. Security model

Treat the agent as an untrusted process with access to your source tree and shell.

Evaluate:

• filesystem sandboxing
• command approvals
• network restrictions
• secret/environment-variable exposure
• credential handling
• prompt injection resistance
• ability to prevent access to neighboring repositories
• audit logs

This matters because an agent that can read files and execute commands has a substantially larger attack surface than an autocomplete tool. Research has also found that functional correctness doesn't guarantee security: agents can produce working code that introduces vulnerabilities.

7. Long-running task reliability

Give it a 30–60 minute task, not ten five-minute demos.

Watch for:

• context drift
• forgetting earlier decisions
• repeatedly inspecting the same files
• overwriting its own good work
• test-fix-test loops
• accumulating unnecessary changes
• losing track of the original acceptance criteria

Long tasks expose weaknesses that simple “fix this function” demos don't.

8. Human review experience

The ideal workflow is:

Agent works → produces diff + tests + explanation → human reviews → merge Evaluate whether you can quickly answer:

• What changed?
• Why?
• Which files changed?
• Which commands ran?
• Which tests passed?
• Which tests weren't run?
• Did it touch anything unexpected?
• Can I revert just this task?

I'd heavily weight this. A slightly weaker agent whose output is easy to review can be much more useful than a stronger agent that produces opaque changes.

9. Locality/privacy

“Runs locally” can mean very different things.

Distinguish:

• Local UI, cloud model
• Local agent/runtime, cloud model
• Local agent + local model
• Fully offline

If proprietary code is involved, verify exactly what leaves the machine: source, tool output, telemetry, prompts, embeddings, logs, crash reports, etc. Self-hosting the agent runtime doesn't automatically mean the entire workflow is private.

10. Model flexibility

If you want to avoid vendor lock-in, test whether the harness can switch between:

• frontier hosted models
• cheaper hosted models
• local models
• OpenAI-compatible endpoints
• Ollama/LM Studio-style local inference

This lets you separate “is the agent good?” from “is the model good?”

---

A practical 100-point scorecard

I'd start with this:

And impose a hard gate: don't adopt anything that fails your safety requirements, regardless of its coding score.

The benchmark I'd actually run

Take 8–12 real tasks from your repository:

• 2 straightforward multi-file features
• 2 bug fixes requiring investigation
• 2 refactors
• 1 dependency/API upgrade
• 1 intentionally difficult/ambiguous task
• 1 task with a failing test
• 1 security-sensitive change
• 1 long-running task

For each candidate, use the same model where possible, same repo snapshot, same acceptance tests, and same permissions.

Record:

Task
├── Did it solve the problem?          yes/no
├── Tests passed                       %
├── Files changed
├── Unnecessary changes                count
├── Regressions                        count
├── Commands requiring approval
├── Unexpected filesystem access       yes/no
├── Time to useful patch
├── Tokens / model cost
├── Human review time
└── Would I merge this patch?          yes/no

The last metric—“Would an engineer actually merge this?”—is probably the most valuable.

Candidates I'd put in the first bake-off

For the workflow you've described, I'd start with Claude Code, OpenAI Codex CLI, Aider, and OpenHands. They represent useful variations in terminal agents, Git-centric workflows, and more autonomous/sandboxed agent environments.

I would not choose based primarily on SWE-bench or vendor demos. Your own repository is a much better benchmark, especially because multi-file coordination, security, and reviewability are precisely where agent behavior diverges.

If you tell me your language stack (e.g. TypeScript/Python/Go), repo size, and whether “local” means local runtime only or fully offline, I can turn this into a concrete bake-off with 10 benchmark tasks and a scoring rubric.

Task
├── Did it solve the problem?          yes/no
├── Tests passed                       %
├── Files changed
├── Unnecessary changes                count
├── Regressions                        count
├── Commands requiring approval
├── Unexpected filesystem access       yes/no
├── Time to useful patch
├── Tokens / model cost
├── Human review time
└── Would I merge this patch?          yes/no

The last metric—“Would an engineer actually merge this?”—is probably the most valuable.

Candidates I'd put in the first bake-off

For the workflow you've described, I'd start with Claude Code, OpenAI Codex CLI, Aider, and OpenHands. They represent useful variations in terminal agents, Git-centric workflows, and more autonomous/sandboxed agent environments.

I would not choose based primarily on SWE-bench or vendor demos. Your own repository is a much better benchmark, especially because multi-file coordination, security, and reviewability are precisely where agent behavior diverges.

Evaluating a local, repo-inspecting, multi-file coding assistant requires looking at three distinct layers: the agentic harness (the tool driving the edits), the local model (the brain), and your hardware constraints (the bottleneck).

When building or adopting this setup, focus your evaluation on these core criteria:

1. The Agentic Harness (Repo Inspection & File Safety)

A model alone cannot safely modify multiple files; it needs a harness (like Aider, Cline , or Continue ) to interact with your codebase. Evaluate the harness on:

• Context Retrieval & RAG: How does it discover relevant files? Look for tools that use tree-sitter or vector search rather than blindly dumping whole directories into the prompt.
• Diff/Patch Robustness: Can the harness reliably apply edits across multiple files without mangling indentation or syntax? Ask if it uses search-and-replace blocks or unified diffs, and how it handles failures.
• Sandbox & Git Integration: Does it automatically commit atomic changes, run local linters/tests before finalizing, and allow you to easily review or rollback diffs?

2. The Local Model (The "Brain")

Not all code models are built for agentic multi-file reasoning. When testing models (such as specialized coding variants of Qwen or Llama), evaluate:

• Instruction Following & Tool Use: Can the model strictly follow a multi-step JSON/tool-calling schema required by your harness, or does it hallucinate file paths?
• Effective Context Window: While a model might claim a 128k+ token window, test its practical coherence when fed a module containing 30k–60k tokens of your actual codebase. Does it lose track of variable definitions across files?
• Quantization Impact: Test quantized versions (e.g., Q4_K_M vs. Q8 GGUF formats) to find the sweet spot where speed doesn't compromise logic accuracy.

3. Hardware & Infrastructure

Local execution is bound strictly by your machine's architecture:

• VRAM Capacity: Multi-file tasks require larger context lengths, which scale VRAM usage rapidly. Evaluate whether a single GPU (like an RTX 3090 Go to product viewer dialog for this item./4090 Go to product viewer dialog for this item.) or a unified memory Mac Studio Go to product viewer dialog for this item. fits your model size comfortably without offloading to slower CPU RAM.
• Tokens-per-Second (t/s): An agent might take 10 turns to complete a multi-file refactor. If your generation speed drops below 15–20 t/s, the feedback loop becomes painfully slow. Look into backend engines supporting speculative decoding or multi-token prediction (MTP).

To help narrow this down, tell me:

• What hardware (GPU/VRAM or Mac Apple Silicon) do you plan to run this on?
• Which IDE or workflow interface (VS Code, CLI, JetBrains) is your team's standard?

I can recommend a specific combination of harness and model weights that fits your stack.

If the requirement is “runs locally, understands a real repo, makes coordinated multi-file changes, and doesn’t wreck the working tree,” I’d evaluate it less like an autocomplete tool and more like a junior engineer with shell access.

Evaluation criteria

A particularly important point is transactionality of multi-file edits. Some local agents use exact-match patches and validate the whole batch before writing, which is substantially safer than independently overwriting files.

The safety bar I'd require

For an internal/team tool, I'd make these effectively hard requirements:

1. Workspace confinement — the agent cannot read/write arbitrary paths outside the selected workspace without explicit approval.
2. Explicit command permissions — distinguish read-only commands (git diff, tests, searches) from mutations (rm, package installs, database commands, git reset, etc.).
3. Secret protection — don't feed .env, credentials, SSH keys, certificates, or similar material to the model unless explicitly authorized.
4. Pre-edit snapshot/diff — every run starts from a known working-tree state.
5. Atomic or validated multi-file patches — don't partially apply a plan because file 7 of 10 failed.
6. Approval gates for destructive actions — especially deletes, network access, installs, database mutations and dangerous Git operations.
7. Undo/checkpoints — one command should get you back to the pre-agent state.
8. Post-change verification — tests/lint/typecheck plus a final diff review.
9. Network isolation — ideally the default is no network access, with explicit per-operation permission.
10. Auditable execution — retain commands, approvals, model/provider, files changed, and test results.

These aren't theoretical concerns: current local-agent implementations are already converging on path confinement, approval policies, patch previews, backups, undo, Git safety, secret redaction, and dedicated security regression suites.

Build a bake-off rather than relying on demos

I'd give every candidate the same 8–10 repository tasks:

• Bug fix: identify and fix a subtle existing bug.
• 5-file feature: add an API field end-to-end.
• Refactor: rename/change an interface with 10+ consumers.
• Test-driven fix: start from a failing test.
• Migration: update code, config, fixtures and documentation together.
• Ambiguous task: see whether it asks useful questions instead of guessing.
• Dirty tree: give it unrelated uncommitted changes and see if they're preserved.
• Failure injection: make one test fail after its first attempt and see whether it diagnoses the failure.
• Security trap: put credentials and dangerous scripts in the repo and see what it touches.
• Large-repo task: measure latency and correctness when the relevant code isn't obvious.

Score task success, not just whether the agent produced code. I'd weight it roughly:

35% correctness · 20% safety · 15% multi-file consistency · 10% verification · 10% diff quality · 10% speed/cost

One other thing I'd specifically test is whether repository instructions actually help. Recent research found that automatically generated or overly elaborate repository context can reduce task success and increase inference cost; the practical lesson is to favor small, high-value repo instructions over a giant AGENTS.md.

A useful acceptance criterion

The strongest candidate should be able to take:

“Add support for X. Preserve backwards compatibility. Update the affected API, implementation, tests, fixtures and docs. Run the relevant checks and show me the final diff.” …and reliably produce a coherent, tested, reviewable multi-file change while preserving everything that was already in the working tree.

That is a much more meaningful benchmark than “how good is its autocomplete?”

For a local coding assistant that can inspect a repo and safely make multi-file changes, evaluate it less like an autocomplete tool and more like a junior engineer with shell access. The important questions are: Does it understand the codebase? Does it make correct changes? Can you trust the diff?

A good evaluation framework:

1. Repository understanding

Test: Give it tasks where the answer is not obvious from one file.

Examples:

• “Add support for a new API field. Find all affected layers.”
• “Replace this deprecated library usage throughout the repo.”
• “Where is authentication handled? Explain the flow before changing anything.”

Measure:

• Does it find the right files?
• Does it understand architecture?
• Does it reuse existing helpers instead of creating duplicates?
• Does it respect project conventions?

Failure signs:

• Editing only the file mentioned in the prompt
• Inventing APIs or patterns
• Missing existing utilities
• Ignoring configuration, migrations, generated files, or tests

Repository awareness is one of the biggest differentiators between agents that merely generate code and ones that can work inside a real project.

2. Multi-file change quality

Test with realistic tickets:

• A bug fix touching implementation + tests + docs
• A feature touching frontend + backend + schema
• A refactor across several modules

Score:

• Correct files changed
• Minimal unnecessary changes
• Consistent style
• No unrelated churn
• Clean git diff

A useful metric is not “did it finish?” but:

“Would I approve this pull request after a normal review?”

3. Planning and autonomy

Look at how it works, not just the final output.

Good behavior:

1. Inspects relevant files
2. Forms a plan
3. Makes incremental edits
4. Runs validation
5. Reports remaining uncertainty

Weak behavior:

• Immediately edits without understanding
• Runs broad destructive commands
• Keeps retrying the same failed approach
• Makes architectural decisions silently

Test different autonomy modes:

• Explain only
• Plan only
• Make changes with approval
• Make changes and validate independently

4. Safety model

For local tools, this is critical.

Evaluate:

File access

• Can it restrict itself to one repo?
• Can it read secrets accidentally?
• Does it respect ignore files?

Command execution

• Are shell commands approved?
• Can you block network access?
• Can you sandbox execution?

Change control

• Does it create clean commits?
• Can you review diffs before applying?
• Can you undo everything easily?

A capable agent with poor permissions is a liability.

5. Verification ability

Give it tasks where “it compiles” is not enough.

Examples:

• Introduce a subtle regression and ask it to fix it
• Change behavior with existing tests that need updating carefully
• Ask it to explain why a test proves correctness

Measure:

• Does it run the right tests?
• Does it interpret failures correctly?
• Does it avoid weakening tests just to get green?

An agent writing both implementation and tests can accidentally create tests that merely confirm its own assumptions, so independent verification matters.

6. Model and context handling

For local assistants, compare:

• Maximum repo size handled well
• Context refresh behavior after edits
• Search/indexing quality
• Performance on large monorepos
• Ability to handle stale or changing code

Test:

• 10k-line repo
• 100k-line repo
• Monorepo with multiple languages

A tool that works on a toy repo may fail when navigating a real one.

7. Developer workflow fit

Evaluate the interface:

• Terminal-first vs IDE-first
• Works with your Git workflow
• Supports existing linters/test runners
• Easy to hand off between humans and agents
• Clear logs of actions taken

A good agent should fit your engineering process rather than forcing a new one.

8. Cost, latency, and hardware

For local deployment:

Measure:

• RAM/VRAM requirements
• CPU/GPU utilization
• Response latency
• Battery impact (for laptops)
• Cost per developer per month
• Whether you can swap models

A slower but reliable assistant often beats a fast one that creates review burden.

9. Security and privacy

Check:

• Where code is processed
• Whether telemetry exists
• Data retention policy
• Model licensing
• Ability to run offline
• Secret handling

This can eliminate options before technical comparisons even begin.

10. Run a real-world bakeoff

I would avoid benchmark-only evaluation. Use your own repo and real tickets. A practical trial:

Week 1

• Pick 5–10 historical bugs/features
• Give each assistant the same tasks
• Record: - Time to solution
• Human intervention
• Diff size
• Test success
• Reviewer confidence

Week 2

• Let engineers use them naturally
• Measure: - Accepted changes
• Rework required
• Developer satisfaction
• Incidents caused

Real repository tasks reveal much more than coding puzzles.

Suggested scoring rubric

The biggest mistake is choosing the assistant that produces the most code. Choose the one whose changes your team can trust, review, and merge.

• Let engineers use them naturally
• Measure: - Accepted changes
• Rework required
• Developer satisfaction
• Incidents caused

Real repository tasks reveal much more than coding puzzles.

Suggested scoring rubric

When setting up a local, repository-aware coding assistant capable of safe multi-file edits, you need to evaluate tools across four core pillars: agentic orchestration, local model capability, context indexing , and safety/guardrails.

1. Agentic Orchestration (The Runner)

You need a harness or editor extension that doesn't just complete text, but can plan, execute terminal commands, and modify multiple files in a structured loop.

• Open-source extensions & IDE tools: Look at Continue or Aider , which connect your local editor (like VS Code or JetBrains) to local endpoints and support agentic file-editing loops.
• Autonomous CLI agents: Tools like OpenCode or local-first agent runners allow you to define multi-step workflows (planner→right arrow→ implementer→right arrow→ tester) directly on your machine.
• Key evaluation question: Can the tool perform atomic multi-file edits (rollback if step 3 of 4 fails), or does it just stream raw text and hope for the best?

2. Local Model Intelligence (The Brain)

Multi-file refactoring requires heavy reasoning. A small 7B model will often lose track of imports or break signatures across files.

• Model size vs. VRAM sweet spot: If you have a 24GB VRAM GPU (like an RTX 3090/4090), look into robust coder models in the 14B to 32B range (such as specialized Qwen-Coder or DeepSeek-Coder variants).
• Quantization & Context: Test 4-bit or 8-bit GGUF/EXL2 formats. Ensure the model supports a practical context window of at least 32k–64k tokens locally without choking your hardware.
• Key evaluation question: Does the model understand cross-file dependencies, or does it hallucinate functions that "should" exist in other modules?

3. Repository Inspection & Context (The Eyes)

An assistant is only as good as its understanding of your specific codebase structure.

• RAG vs. Full Context: Evaluate how the tool indexes your repo. Does it use a local vector database, AST (Abstract Syntax Tree) parsing, or a lightweight semantic search (like ripgrep + embeddings) to fetch relevant files?
• Scoping: Ensure you can explicitly point the agent to specific modules or provide an AGENTS.md / rules file to ground its scope, preventing it from choking on an entire monolithic repo at once.
• Key evaluation question: How fast does it index a freshly cloned repo, and does it correctly map out deep relative imports and project layout?

4. Safety, Permissions & Sandbox (The Guardrails)

Running an agent with file-write and terminal-exec capabilities locally presents a security and data-integrity risk.

• Diff previews & checkpoints: The assistant should always stage changes as a git diff for your approval rather than blindly writing to disk.
• Execution sandboxing: If the agent is allowed to run tests or linters (npm test, pytest ), check if it runs them in a sandboxed environment or directly on your host machine.
• Key evaluation question: Can you set strict permission tiers (e.g., "read-only for entire repo, write-only permitted in /src")?

To help narrow down the best stack, what are your team's hardware constraints (GPU VRAM / RAM) and what primary programming languages does your repository use?