Guide
Most AI coding tool rollouts are judged on seat counts and vibes. This is the measurement approach we use in every Neocortex pilot: two primary metrics, two guardrails and a six-sprint baseline locked before anything is installed.
An assistant is deployed, adoption is high and the honest answer to "did we get faster?" is that nobody knows. The metrics that were easy to instrument - commits, lines, suggestions accepted - all moved. The metrics the business cares about, how much shipped and how long it took, were never baselined.
Fixing that does not require a new observability platform. Everything below comes out of Git and Jira, which you already have and takes an afternoon to compute.
Two primary outcome metrics, two guardrails that stop the primaries from being gamed. Resist adding a fifth.
Six sprints of throughput - band is the interquartile range, dashed line the median
Merged work per engineer per sprint, normalized for ticket complexity - an LLM judge calibrated on the team's own history scores each ticket's size, so bigger tickets can't fake bigger output and smaller PRs can't inflate the count.
It only counts work that survived review, so it cannot be gamed by opening more PRs or writing more lines.
Git + Jira: merged PRs mapped to tickets, complexity-scored.
Median hours from first commit on a branch to merge, measured per PR and rolled up per sprint.
It captures the review-and-rework loop, which is where AI-assisted code usually loses the time it appears to save.
Git: first commit timestamp and merge timestamp.
Share of PRs that need two or more rounds of substantive review before merge.
A rising rework rate is the clearest early signal that an assistant is producing code that does not fit your codebase's conventions.
Git or your review tool: review submissions per PR.
Story points or issue count committed at sprint start versus completed at sprint end.
It keeps throughput honest - a throughput jump that comes with collapsing scope stability is a planning artifact, not a velocity gain.
Jira: sprint commitment and completion.
Step 1
Choose a single team of roughly four engineers plus a lead that owns a service end to end. Org-wide averages hide the effect you are trying to measure; a pod that size is small enough to attribute change to and still large enough to smooth out one person's bad sprint.
Step 2
Six sprints - roughly one quarter - is the shortest window that survives a holiday, an incident week and one person's vacation. Four is too noisy. Twelve reaches back into a different codebase and a different team.
Step 3
Take the median per sprint for each metric, plus the interquartile range. Report the spread alongside the number: if your six-sprint throughput ranges from 14 to 31 merged PRs, then a post-pilot 34 is not yet a result. This is why our success gate is +40%, not +10% - the result has to clear the noise, not just nudge the median.
Step 4
Write down the metric definitions, the query, the date range and the numbers before the tool is installed. A baseline agreed after the fact is the most common reason AI velocity claims get dismissed by finance.
Step 5
Run the identical query over the post-adoption sprints. Same pod, same repo, same definitions. Any change to the measurement is a change to the result.
Lines of code or commit count
AI assistants inflate both by design. Neither correlates with delivered value and both reward the behaviour you least want.
Self-reported time saved
Survey estimates of AI time savings consistently outrun what shows up in Git. Useful for adoption, useless as evidence.
Tool telemetry as the outcome
Suggestion-acceptance rate measures usage, not delivery. High acceptance with flat cycle time is the most common pattern in enterprise rollouts.
Comparing two different teams
Codebase age, on-call load and review culture swamp any tooling difference. Compare a pod to its own past, not to its neighbour.
This is exactly how the Neocortex pilot is structured. Week 0 locks the six-sprint baseline from your Git and Jira. Weeks 1–2 install and tune your knowledge layer alongside the pod. Weeks 3–4 the pod runs it on live sprints, independently - and these are the sprints we measure. The verdict comes at Week 4: same queries, same baseline.
Measured window
The target is +40% complexity-normalized throughput with cycle time flat or better and the guardrails holding. If the number is not there, the pilot failed - which is a far more useful outcome than a dashboard nobody trusts.
