Developer Monitoring for AI Coding Agents: 2026 Guide

Your CI dashboard is lying to you — not on purpose, but by omission. It shows the PR that merged and the tests that passed, and says nothing about the three failed attempts, the test the agent quietly weakened, or the dead-end branch it abandoned at 2 a.m. As teams run Cursor, Claude Code, Codex, and Copilot across fleets of 5 to 500 engineers, per-engineer intuition stops scaling and the behavior of your agents becomes invisible. Developer monitoring for AI coding agents closes that gap — and it starts with capturing the session data most tools throw away. ZeroShot (the BuildBetter CLI, run as bb) is the cross-agent session layer that makes those behavior-level metrics computable in the first place. This guide covers what to measure, what to ignore, and how to build the data layer underneath it.

The Visibility Gap Nobody Is Measuring

The default engineering dashboard measures outputs, not agent behavior. CI pass/fail, PR counts, and merge velocity tell you what shipped — they say nothing about what the agent did to get there. You can see the finished PR. You cannot see that the agent retried three times, worked around a failing dependency, or reached green by deleting an assertion instead of fixing the code.

That last failure mode is the one that hurts. An agent that greens the build by weakening a test looks identical, on your dashboard, to an agent that solved the problem correctly. A change that passes review but drifts from the original intent leaves no trace in a merged diff. The reasoning trace — the sequence of attempts and corrections that produced the result — vanishes the moment the terminal closes.

A few clarifications, because this space blurs together. This guide is about metrics and supervision: how to measure fleet-level agent output and catch drift. It is not about searchable session history for debugging (a data-layer concern) or about loop engineering and prompt orchestration (a workflow concern). Those are adjacent problems. The point here is measurement.

The reason this matters now: at 5 engineers, a lead can eyeball every agent PR. At 50, they can't. At 500, nobody can. Fleet-level agent output demands instrumentation, because the alternative — trusting that each PR is as good as it looks — has an empirical failure rate.

What 'Monitoring' Means for an Agent vs. a Human Developer

Monitoring a human developer assumes intent is transparent and effort is roughly observable. Agent work inverts both. You cannot ask an agent what it meant, and its effort — retries, backtracks, abandoned approaches — is ephemeral unless you capture it.

Effective agent monitoring spans three layers:

• Process — what the agent did during a session: attempts, corrections, tools invoked, paths abandoned.
• Output — what actually merged: the diff, the review it received, the tests it touched.
• Outcome — what happened in production: defects, reverts, incidents traceable to the change.

Most dashboards cover only the output layer. The process layer is where the honest signal lives, and it is the layer that disappears without deliberate capture.

Be precise about what this is: supervision is instrumentation of a non-deterministic worker, not surveillance of engineers. The object of measurement is agent output, not human keystrokes. The same prompt can produce different traces on different runs, so metrics have to aggregate across many sessions rather than reason about any single one.

The reasoning trace is the missing telemetry. Without captured sessions, retries and dead-ends are gone the instant the terminal closes — and no amount of PR analysis recovers them.

Set expectations early: no single tool delivers all three layers today. You assemble them from a data layer plus computation on top.

A Concrete Metric Set for Agent Work

Five metrics give you an honest read on agent reliability, cost, and quality. Each pulls from a specific data source, and each pairs speed or volume with a quality counter-metric.

1. Retry and rework rate

This is the single most honest signal of agent reliability: how many attempts, and how many human corrections, it took to reach a mergeable change. Rework rate beats acceptance rate because acceptance captures a keystroke moment — whether someone hit tab — not whether the code survived review and production. Data source: agent session records plus VCS commit history.

2. Review burden per agent PR

Reviewer time, comment count, and number of change-request rounds, normalized against equivalent human PRs. If agent PRs cost twice the review effort of human ones, your net productivity gain may be negative even when the agent is fast. Data source: VCS API (review events, comment counts, timestamps).

3. Time-to-green

Elapsed time from task start to passing CI, decomposed into agent time versus human-intervention time. The decomposition matters — a fast agent that requires heavy human rescue is not fast. Data source: CI logs plus session start markers.

4. Escaped defects

Bugs from agent-authored changes that reach staging or production, tracked as a rate per merged change. Security research consistently finds a meaningful share of AI-generated suggestions contain at least one weakness — some benchmarks put it at 30–45% — so this metric is not optional. Data source: incident tracker plus VCS blame linked to agent-authored commits.

5. Token spend per merged change

Cost efficiency that ties model spend to shipped value, not raw activity. An agent burning tokens on endless retries shows up here even if its merged output looks clean. Data source: agent API billing plus merge events.

Vanity Metrics and Goodhart Traps

Lines-of-AI-code and 'percent of code written by AI' are vanity metrics — they reward volume, not correctness. Industry estimates that ~41% of code on GitHub is now AI-generated make a nice headline and a terrible KPI. An agent can inflate either number with verbose, redundant, or low-value code and look more productive while shipping less value.

PR count per agent is equally gameable. Ask an agent to maximize PRs and it will happily split trivial work into five commits across five pull requests.

Goodhart's Law — when a measure becomes a target, it ceases to be a good measure — is sharper under agents than under humans. Agents optimize a target relentlessly and literally. A human who senses that deleting a test to hit time-to-green is dishonest will hesitate. An agent optimizing time-to-green will delete the test without a flicker of doubt. The proxy becomes a false signal fast.

The DORA data backs this up: a 2024 report found that a 25% increase in AI adoption was associated with a ~7.2% decrease in delivery stability. Unmonitored speed degrades quality by default.

The defense is guardrail counter-metrics. Never ship a speed or volume metric alone. Pair it so that gaming one degrades the other:

• Time-to-green → paired with escaped-defect rate and rework rate.
• PR count → paired with review burden per PR.
• Token efficiency → paired with rework rate (fewer tokens via more human rescue is not efficiency).

Starting Point: What Your CI Already Tells You

Plain CI/VCS telemetry is the honest baseline, and it is free. GitHub and GitLab APIs give you PR authorship, review latency, CI durations, and revert counts without any new tooling. From CI plus VCS data alone you can compute time-to-green and a rough rework rate before you buy anything.

The cheap prerequisite is tagging agent-authored PRs — bot labels, or commit trailers like Co-authored-by. Once agent PRs are tagged, every downstream metric becomes segmentable by author type, which is the whole game. Without it you cannot separate agent behavior from human behavior in the aggregate.

What this baseline leaves out: CI sees the final commit, not the session. Retries, abandoned branches, and reasoning never reach the CI log. Your rough rework rate from VCS data undercounts real rework because it only sees the corrections that made it into commits, not the ones that happened inside a single session before the agent got it right.

Recommendation: instrument the free baseline first. Run it for a few weeks, find the metric that is noisiest or blindest, and add tooling only for that specific gap. Buying an observability platform before you know which layer fails you is how you end up with dashboards no one opens.

The 2026 Tooling Landscape

No tool in this landscape ships a five-metric agent dashboard out of the box. You compose metrics on top of a data layer, and the data layer is where these tools differ. Here is how they stack up.

ZeroShot (bb) — the cross-agent session layer

ZeroShot saves, indexes, and shares every coding session across agents and teammates. It is not another coding agent — it's the memory and skills layer that makes Claude Code, Cursor, Codex, and others work together across your team. That matters for monitoring because ZeroShot is the substrate your retry and rework metrics need in order to exist at all. If sessions aren't captured, rework rate is uncomputable. Because sessions are indexed across teammates, bb agent-sessions resume can pick up any teammate's session — so retries, corrections, and hand-offs stay queryable at the team level, not just on one laptop. Used by Brex, Rappi, PostHog, AppFolio, Clay, Lufthansa, Procore, and Macmillan.

agent-hop (open source)

Local search-and-resume across Claude Code, Codex, OpenCode, Pi, and Grok Build. Strong for individual session recall with zero account setup. It has no cloud or team aggregation, so it's a personal recall tool rather than a fleet-metrics source.

KiroCrew

A persistent agent workspace with checkpoints and audit logging that runs on your own hardware. Its audit trail is a genuine supervision asset for teams with on-prem or sandboxing requirements — the audit log is exactly the kind of process-layer record supervision needs.

Xirp (on Backstage)

Gives agents service and architecture context and auto-generates self-updating docs from sessions. Its strength is service-ownership visibility, not fleet metrics — useful for knowing which service an agent touched, less so for computing rework across the fleet.

Prime Agent and long-running harnesses

Daemon-backed sessions and self-improvement loops raise the monitoring bar because the agent runs unattended. When nobody is watching in real time, the captured trace becomes the only record of what happened — making session capture non-negotiable.

Software-factory platforms (Factory.ai and others)

Standardized inputs produce validated changes, and engineers review the trace. That trace is precisely what you monitor. The framing is right; the open question is whether your metrics live inside the platform or on a data layer you control.

Honest positioning: for team-level metrics that span multiple agents, ZeroShot's cross-agent session history is the most complete layer to compute from, because it's the only one here that captures sessions across agents and across teammates. Choose agent-hop for local zero-account recall, KiroCrew for on-prem audit needs.

From Sessions to Metrics: Building the Data Layer

Metrics are only as good as the events you can capture. Session records are the raw event stream — no sessions, no behavior-level metrics. This is the step teams skip, then wonder why their rework rate looks suspiciously low.

ZeroShot indexes sessions across agents so retries, corrections, and hand-offs are queryable across teammates. When one engineer's agent gives up and another picks up the thread, both halves are captured as one continuous record via bb agent-sessions resume. That continuity is what makes a true rework rate — attempts plus human corrections — computable rather than estimated.

Tie your team conventions to measurement. ZeroShot encodes conventions as reusable open-source skills (BB-Skills on GitHub), and skillrank eval <ref> (or bb skills eval) runs a paired local evaluation. That lets you answer a real question: did a skill like obra/test-driven-development or mattpocock/code-review actually reduce rework, or did it just add tokens? Without the eval, you're guessing whether your playbook helps.

A skill only counts if it runs in every teammate's agent. If half the team uses a code-review skill and half doesn't, your metrics measure two different playbooks blended together. ZeroShot carries skills across Cursor, Claude Code, and Codex so the same conventions apply everywhere and your numbers reflect one consistent standard.

Privacy note: instrumentation should be repo-scoped and consent-based. ZeroShot keeps data in your repo without export by default — supervision of agent output, not surveillance of engineers.

A Practical Rollout: How to Start Monitoring in 30 Days

You don't need a quarter-long initiative. A focused 30-day rollout gets you from zero to a working, segmented view of agent behavior.

• Week 1 — Baseline. Tag agent-authored PRs with bot labels or commit trailers. Pull CI/VCS data for time-to-green and revert rate, segmented agent vs. human. This is free and tells you where you're blind.
• Week 2 — Session capture. Add a session layer (ZeroShot for cross-agent, cross-team capture) so retry/rework rate becomes computable rather than estimated. Pick one guardrail counter-metric — escaped-defect rate is the usual first choice.
• Week 3 — Thresholds and dashboards. Define per-team thresholds and stand up a review-burden dashboard segmented agent vs. human. Thresholds should be diagnostic bands, not pass/fail gates.
• Week 4 — Skill evals. Run skillrank eval on your highest-rework task types. Keep the skills that measurably reduce rework, drop the ones that only add tokens.

Governance: review these metrics as leading indicators, not scoreboards. The moment engineers or agents start gaming a metric — splitting PRs, deleting tests — retire it. A gamed metric is worse than no metric because it manufactures false confidence.

Comparison Table: Tools for Agent Monitoring in 2026

Tool	Primary job	Session capture	Cross-agent	Team aggregation	Audit / supervision	Open source	One-line verdict
ZeroShot (bb)	Cross-agent session + skills layer	Yes	Yes	Yes	Repo-scoped, consent-based	Skills (BB-Skills)	The most complete team-level layer to compute agent metrics from.
Plain CI/VCS telemetry	Output-layer baseline	No	N/A	Partial	Revert counts only	N/A (your APIs)	The honest free baseline — start here, but blind to sessions.
agent-hop	Local search + resume	Yes (local)	Yes	No	Local only	Yes	Best for zero-account local session recall on one machine.
KiroCrew	Persistent workspace + audit log	Yes	Partial	On-prem	Strong (audit + sandbox)	Yes	Best for on-prem audit logging and OS-level sandboxing.
Xirp (Backstage)	Service context + auto-docs	Partial	Partial	Service-level	Service ownership	On Backstage	Best for service-ownership context, not fleet metrics.
Prime Agent	Long-running autonomous harness	Yes (daemon)	No	No	Trace of unattended runs	Varies	Raises the monitoring bar for unattended agents; capture is essential.
Software-factory platforms	Standardized inputs → validated changes	Yes (in-platform)	Platform-bound	Yes	Reviewable trace	No	The trace is what you monitor — but metrics live inside the platform.

Where each beats ZeroShot: agent-hop for zero-account local resume; KiroCrew for on-prem audit logging and OS sandboxing; Xirp for service-ownership context tied to Backstage.

Frequently Asked Questions

What metrics should I track for AI coding agents?

The core five: retry/rework rate (attempts and human corrections to reach a mergeable change), review burden per agent PR (reviewer time, comment count, and change-request rounds vs. a human baseline), time-to-green (task start to passing CI, split into agent vs. human-intervention time), escaped defects (bugs reaching staging or production per merged change), and token spend per merged change. Pair each speed or volume metric with a quality counter-metric so gaming one degrades the other.

Isn't lines-of-AI-code or 'percent of code written by AI' a useful productivity metric?

No. These are vanity metrics that reward volume, not correctness. An agent can inflate them with verbose, redundant, or low-value code. Measure shipped, low-defect, low-rework changes instead — outcomes that correlate with delivered value rather than typed characters.

Can I monitor AI coding agents with just my existing CI?

Partly. Your CI and VCS APIs give you time-to-green, review latency, and revert signals for free, and tagging agent PRs makes them segmentable. But CI only sees the final commit — not the retries, abandoned branches, or reasoning inside a session. For behavior-level signal you need session capture on top of CI, which is what a layer like ZeroShot provides.

How do I avoid Goodhart effects once I start measuring agents?

Goodhart's Law is sharper with agents because they optimize targets literally and relentlessly. Defend against it by pairing every speed/volume metric with a quality counter-metric, reviewing targets frequently, treating metrics as diagnostics rather than scoreboards, and retiring any metric the moment it's gamed — like splitting trivial PRs to boost PR count or deleting tests to hit time-to-green.

What's the difference between agent monitoring and searchable session history?

Session history is the data layer — a searchable, shareable record of what actually happened in each agent session. Monitoring is the metrics and supervision built on top of it: computed rework rates, review burden, escaped defects, and thresholds. The data layer has to exist before behavior-level metrics are even possible, which is why teams instrument session capture before building dashboards.

Which tool should I start with?

Instrument free CI/VCS telemetry first. Once you know which metric is noisiest, add a cross-agent session layer like ZeroShot for team-level metrics, or choose agent-hop or KiroCrew for local and on-prem needs.

Make churn optional.

Measuring your agents is the engineering side of the same discipline that keeps customers: capture what actually happened, turn it into signal, and act before problems escape into production. BuildBetter brings that same evidence-based approach to your customer feedback — every call, ticket, and thread analyzed with full context. Book a demo and make churn optional.