16,885 changes in six months. How a fleet of AI agents builds our software.

01

The short version

In six months, 16,885 reviewed changes were merged into our main code base. By September, at least 89% of them named an AI model as a co-author.

Here is the whole idea in five sentences. A person says what they want and how we will know it worked. An AI front desk turns that into a job and hands it to an AI helper, which signs a shared sheet so no other helper touches the same files. The helper writes the change and a test that proves it, and two separate AI reviewers try to break it before a robot is allowed to merge it. Merged code goes live on its own, and then a check on the live system proves it did what it promised. Every one of those steps leaves a record, and those records are where every number on this page comes from.

02

Why we built it this way

We run call centers. We book appointments for roofers and solar companies, we answer phones, we train agents. Software is not what we sell, but almost everything we do runs on it: the dialer reports, the lead forms, the scheduling, the quality checks, the dashboards a floor manager looks at every morning. For years that meant a long list of things we wanted and a short list of things a small team could build.

The bet we made is simple to say. AI writes nearly all of the code. People plan and decide. Machines check everything, and everything is written down. We did not make that bet because AI code is perfect. We made it because the checking can be made strict and cheap, and checking is the part that keeps a business safe.

People decide what to build and what good looks like. Machines do the typing and the checking.

What "nearly all" means in numbers

We can measure a floor under it, not the exact share. In September, at least 89% of merged changes carried the line that names the AI model as a co-author (August: 65.5%). In the same month, about 86% of merged changes were opened by a robot account rather than a person's account. Both are lower bounds. Plenty of changes without the co-author line were also written by AI in a person's session, and our own git server drops that line when it squashes a change, which is why we do not show October for it.

Three rules that shaped everything

One: checks live in code, not in hope. An AI agent will happily say "done". So nothing counts as done because an agent said so. A change is done when a test that could fail did not, when two reviewers that could say BLOCK said SHIP, and when a check on the live system wrote down a real measurement.

Two: small and many beats big and few. A small change is easy to review, easy to undo and easy to prove. Our median change got smaller over the months, from 305 lines in August to 125 in October so far, while the number of changes went up.

Three: when a guard says no, stop. The fleet has automatic guards in front of risky actions. A helper that hits one stops and reports it. It never builds a new path around the guard.

What this is not

This is not a team of people pair-programming with an AI chat window, and it is not one giant agent left alone with the keys. It is closer to a well-run call center floor: a desk that hands out work, many workers who each do one job at a time, quality checks on every piece of work, and a log of everything. That is the shape we know how to run, so that is the shape we built.

03

The cast

Seven kinds of players do the work, and none of them is a person. The people stand at the two ends: the owner and our engineers plan the work, answer the questions the machines cannot answer, and decide what is risky enough to need a person. Everything in between is software, and most of it is AI.

• The front desk

  One AI window that plans the night, hands out jobs, reads the results and writes everything down. It is the only part that talks to the people.

  One window. Its log is the source for the night-shift numbers below.

• Helpers

  AI coding agents, each started for one job. A helper gets its own copy of the code, does the job, reports back and stops.

  Up to 40 at once (a cap, not an average).

• Strict reviewers

  Separate AI reviewers that never saw the work being built. Each one runs the tests, attacks the change and writes a SHIP or a BLOCK.

  4,327 verdicts, Oct 18 to 22.

• The merge robot

  A robot account that merges reviewed changes. For changes in the strict review kit, it merges only when two SHIP verdicts sit on the exact version in front of it.

  The two-SHIP rule since Oct 18.

• The task tracker

  Our project board. Every job is a task with an owner, the context, the hours and a pass-or-fail check, written before work starts.

  7,651 jobs reached review, Oct 6 to 22.

• The claim tool

  A shared sign-up sheet. A helper writes down which files it is working on; anyone else who asks for them is told who holds them.

  One sheet for the whole fleet. Its one big bug is a story below.

• Guards

  Small automatic checks in front of risky actions. A guard refuses, and the helper must stop and report. It is never argued with.

  28 guards in the shared set.

Two words will come up again and again. A front deskFront desk. One AI window that plans the work, hands out jobs to helpers, checks their results and writes down what happened. It does not write most of the code itself. All terms is the planner. A helperHelper. An AI coding agent started by the front desk for one job. It works on its own copy of the code, then reports back and stops. All terms is a worker. Work is grouped into lanesLane. One stream of work with a name, such as one feature or one fix. A lane can have a helper and its own sub-helpers. All terms, one stream of work each, so a night log can say "this lane merged, that lane is blocked" and a person reading it in the morning knows what happened without reading any code.

Notice what is missing. There is no person in the middle approving each change, and there is no single AI that does everything. The people stay at the two ends: they say what they want at the start, and they look at the proof at the end. In between, the work moves through machines that each do one narrow thing, and each machine checks the one before it.

04

The pipeline, step by step

This is the path a change walks today. Some steps are newer than others: the strict reviews, for example, started on Oct 18. The diagram builds as you scroll: each step lights up when you reach it. Under each step is the rule behind it and a real number for it.

Words you will meet here: a task trackerTask tracker. Our project board. Every job has a task with an owner, the context and a pass-or-fail check written before the work starts. All terms, a claimClaim. A line on a shared sign-up sheet that says "this helper is working on these files now". Another helper that asks for the same files is told who holds them. All terms, a precheckPrecheck. Fast machine checks that run before any reviewer: does it build, do the tests pass, did a secret or a private path slip into the change. All terms, a strict reviewStrict review. A review by a separate AI reviewer that has not seen the work before. It runs the tests, tries to break the change, and must write down what it attacked. All terms that ends in SHIP or BLOCKSHIP / BLOCK. The two answers a reviewer can give. SHIP means it may merge. BLOCK means it may not, with a list of what must be fixed first. All terms, the merge robotMerge robot. A robot account that merges reviewed changes. For changes in the strict review kit, it merges only when two strict reviews said SHIP on the exact same version of the code. All terms, go-liveGo-live. The step where merged code is built and put on the real servers. In our pipeline a robot does it after the merge. All terms and proofProof. A check on the real, live system after go-live that shows the change does what it promised, written down with what was measured. All terms. Tap any underlined word for its plain meaning.

Read the steps again and you will see the pattern. Each step is cheap compared with the one after it, and each one can send the work back. A precheck is fast; a strict review costs minutes and about a dollar; in our experience a bad merge that goes live costs far more to find and undo. So the cheap checks go first, and nothing reaches the expensive part of the line until it has passed everything before it.

05

One real change, end to end

Here is one change from a normal night, start to finish. It is about our own website, so it is safe to show. The problem: our own website's deploy kept deleting large video and audio files that were not in the site's new code home. The fix: the deploy protects media from deletion and refuses to run if that protection is removed.

A few things are worth pointing out. The helper measured before it changed anything: it counted the files at risk (269 of them, about 958 MB in all) with a check that could not change anything. The fix was the smallest one that works. And the new test was seen failing on the old code first. A test that only ever passes might be testing nothing.

The two reviewers had different jobs. One looked for safety problems (11 attack lines, 2.0 minutes, $0.61); the other looked for correctness (8 attack lines, 2.5 minutes, $0.63). Both said SHIP on the same exact version, so the merge robot merged it.

Then the change waited for the morning's go-live, which a robot ran with its own identity. The proof did not just say "deploy succeeded". It asked every page in the sitemap for an answer from outside, and checked that the set of media files was byte-for-byte the same. The last step made the problem easier to catch next time: the live site now says which exact version of the code it was built from, so a watcher can compare the two.

Gap: the ask time is the slot the job was handed out in; the code time is approximate (between 11:39 PM and 11:56 PM, as the log shows).

06

Six months, month by month

The pipeline did not arrive all at once. It grew one rule at a time, often right after something went wrong. The charts below show what that looked like from the outside: how many changes, how fast they moved, what kind they were, and how big. Tap or hover any bar or point for its value; every chart has its numbers in a table underneath.

The shape is not a smooth climb. May was busy, June was quiet, and then the line takes off in July and again in September. The July jump lines up with the month robot accounts started opening most of the changes themselves (the last chart in this section shows it).

How fast a change moved

The next chart is the one we are proudest of, and also the one that needs the most care. It shows the middle wait (the median) from the moment a change was opened to the moment it merged. In April it was 95.0 minutes. By August it was 2.5 minutes. The axis is a log scale (each gridline is a bigger step than the one below it), because the slowest changes waited hundreds of times longer than the fastest; on a plain axis the summer months would be flat lines at the bottom.

A fair reader will ask why the wait goes back up. We think it is partly that the checks got stricter, step by step: the median had already risen from 2.5 minutes in August to 4.1 in September, and from October more changes got full AI reviews that run the tests and attack the code, which take minutes, not seconds. We chose that trade on purpose. Speed is only worth having if the thing that arrives fast is right. October's numbers here also stop on Oct 3, when we moved to our own git server, so the later waits are measured differently (see the merge wait, below).

What kind of changes

Every change starts with a short type word: fix for repairing something, feat for something new, docs for writing, test for tests on their own, chore for housekeeping. Fixes lead every month we can count. That is not a bad sign. A fleet that ships small changes fast also finds its own mistakes fast, and the fixes are where those findings land.

Who opened the changes

The last chart in this set counts changes opened by a robot account instead of a person's account. It is a lower bound on AI work, not a measure of it: a person's account also opens changes its AI session wrote. Still, the share jumps in July, and by September robot accounts opened most of the changes.

07

The night shift

A call center knows about night shifts. So does this fleet. From Aug 1 to Oct 21, 5,412 of 12,191 merges, or 44%, landed from 7 PM to 7 AM Eastern. The busiest hour of the day is still 1 PM, when the people are awake and asking. But the night never goes quiet: even the slowest hour, 6 AM, had 237 merges over those weeks.

One night, hour by hour

Here is one night, from 6:00 PM on Oct 19 to 9:00 AM on Oct 20, read from the front desk's log. Every real action writes one line: a job handed out, a claim, a review verdict, a merge, a proof, a stop. The bars are the lines per hour. Move the slider or press Play to walk through the night.

Source and gap: The front desk's night log: one line per real action, with its time and its lane. Counts of lines that say merged, blocked, proved, or stopped and refused. Lines, not distinct events: one merge can log 2 or 3 lines. Lanes include sub-helpers. The hour bars add up to one line fewer than the total.

Look at the totals in the last line. 209 lines said merged, and almost exactly as many (203) said blocked. That is the night working as designed: a block is not a failure of the night shift, it is the night shift catching something before it ships. The 277 stop and refusal lines are the same idea one level down: a helper that hits a guard or a missing permission stops and says so, and the front desk decides what happens next in the morning.

08

Reviews that bite

A review that always says yes is worse than no review, because it makes people stop looking. So we count how often our reviewers say no. Since Oct 6, every review leaves a receipt: one file per reviewer per exact version of the code, with a verdict line the merge robot can read.

One job in six sent back is about right for us. It means reviewers find real problems, and it means most helpers get it right the first time. When a job is sent back, its helper fixes exactly what the reviewers named and the new version is reviewed again from scratch. The chart below shows how many versions each job needed before it was done.

What makes a review strict

Since Oct 18 the important changes go through a stricter kit. Two reviewers, each started fresh with no memory of the work, get the exact version of the change and the task it is meant to do. They must run every test that touches a changed file, and they must write attack linesAttack line. One written line in a review that names something the reviewer tried in order to break the change, and what happened. All terms: one line per thing they tried in order to break the change, and what happened. A SHIP with no attack lines does not count.

Before any reviewer reads the code, the kit does something sneaky on purpose. It makes small planted bugsPlanted bug. A small mistake we put on purpose into a throwaway copy of the code, such as turning "and" into "or". If the tests do not catch it, the reviewer knows where the tests are thin. All terms in a throwaway copy, like turning "greater than" into "greater than or equal", or "and" into "or", and runs the tests against each one. If the tests catch a planted bug, good. If they miss it, the reviewer is told exactly where the tests are thin, and looks there first.

A 55% catch rate sounds low. It is honest. The number is not there to look good: it is there to point the reviewer at the exact lines where a real bug could hide.

For a median of $1.07 a reviewer at list price, a change gets two careful readers who run the tests and try to break it. That is the economic heart of the whole bet: checking became cheap enough that we do not have to ration it. Since Oct 18, every change that goes through the kit gets two.

09

Guards and the sign-up sheet

A guardGuard. A small automatic check that sits in front of a risky action and refuses it, for example running a command that could delete files. A guard says no; it never asks the agent to argue. All terms is a small automatic check that sits in front of a risky action. When a helper tries something on the guard's list, the guard refuses before anything happens and says why. Think of the badge reader on a server room door: it does not argue and it does not make exceptions for people in a hurry.

Some guards stop a command that could delete files from the wrong place. Some refuse to let a secret appear in a command or a log. Some stop a helper from reaching into parts of the system it has no business in. We describe what they refuse, never how they work inside: a guard that is easy to explain is easy to get around.

A guard blocks you: stop and report. Never build around it.

We think that rule is the most important line in every helper's instructions. An AI agent is very good at finding another way to reach a goal. That is exactly what you do not want when the thing in the way is a safety check. So a helper that hits a guard stops, writes down what it tried and why it was refused, and leaves the decision to the front desk or a person. Sometimes the guard is wrong, and then we fix the guard, in the open, with a review. The helper never fixes it on the fly.

The sign-up sheet

The other quiet hero is the claim tool. With up to 40 helpers at work, two of them will sooner or later want the same file. Without a sheet, both would change it and one would undo the other. With the sheet, the first one writes its claimClaim. A line on a shared sign-up sheet that says "this helper is working on these files now". Another helper that asks for the same files is told who holds them. All terms and the second is told who holds it. A claim that runs out frees itself, so one stuck helper cannot hold work forever. It is the same trick a busy kitchen uses with order tickets on a rail: the ticket is the lock.

10

Tokens: what the work reads

AI models read and write in tokensToken. The unit AI models read and write in. A token is a short piece of a word; a page of text is several hundred tokens. All terms, short pieces of words. Every model callModel call. One request to an AI model and its one reply. An agent makes many calls to finish one job. All terms sends text in and gets text back, and an agent makes many calls to finish one job. Counting tokens is how you see what the fleet actually reads.

The cacheCache. Text the model has already read recently and can read again from a saved copy instead of fresh. Most of an agent conversation is the same text read again. All terms number is the one to understand. An agent working on a job re-reads the same instructions, the same files and its own earlier steps on every call. The cache means that repeated text is read again from a saved copy instead of fresh. Almost everything the fleet reads is a re-read.

The busiest single day in these records was Oct 17: 146,867 model calls and 72.6 billion tokens. Most calls went to the top model.

The leak we found

Counting tokens also caught a real mistake. After a tool update, many new helpers started their life with a full copy of their parent's conversation, a large block of text they did not need. In one measurement of the first call of every helper, 123 of 786 began at 100 thousand to 950 thousand tokens. Of 12.4 billion helper tokens in that measurement, 5.9 billion (47%) were that copy.

The fix was one setting. The lesson was bigger: the front desk now re-checks that setting after every tool update, because a quiet default change upstream can double what you read without anyone noticing.

11

What broke, and what we built

Many of the rules on this page exist because something went wrong first. Here are five of those moments, each with what happened, what we changed, and what is true now.

1. The guard that refused everything

   BeforeA memory check in the job launcher misread the machine and refused every job for 6 hours: 260 refusals, with 120 GB free.

   The fixThe check was corrected, and the desk counts refusals at every check-in.

   AfterRefusals are counted at every check; a refusal streak is now itself an alarm.

2. Reviews the gate could not read

   BeforeReviewers wrote receipts in several forms the merge gate could not count, and a SHIP written as prose looked like a yes.

   The fixOne strict receipt format the gate can read; a receipt in any other form, or a SHIP written as prose, does not count.

   After15,772 receipts with a readable verdict; 4,327 strict reviewer verdicts in the one format.

3. The sign-up sheet that said everything was taken

   BeforeThe claim tool (one sign-up sheet, so two helpers never edit the same thing) matched old 'everything under this folder' rows against any path, so stale rows blocked good jobs.

   The fixWildcard rows now match only inside their own folder.

   AfterA claim holds what it names, and only that.

4. Waiting 15 minutes for a robot that was already sure

   BeforeAfter the second SHIP, the timed merge sweeper took about 15 minutes (13.3 to 16 minutes on clean changes): a 10-minute grace period plus a 5-minute tick. Owners who merged their own reviewed change took a median of 1.7 minutes.

   The fixA shorter grace period and a 30-second tick that only does a full pass when something changed. Built with 38 + 57 tests, two strict reviews; live since October 22, 2:15 AM Eastern.

   AfterFirst 2 merges: 2.2 minutes for a developer change, 6.5 minutes for a robot change (it keeps a 5-minute grace period). Target: under 1 minute. Early, not a result.

5. Helpers that carried the whole conversation

   BeforeA tool update made many new helpers start with a full copy of their parent's conversation: 123 of 786 helpers began at 100 thousand to 950 thousand tokens; 5.9 billion of 12.4 billion helper tokens (47%) were this copy.

   The fixOne setting turned the copy off; the desk re-checks it after every tool update.

   AfterNew helpers start at 20 to 40 thousand tokens again.

The merge-wait story is the newest. Before, a reviewed change waited about 15 minutes for the robot (measured over 30 hours: 35 changes merged by the sweeper, 111 by their own owner, who took a median of 1.7 minutes). The faster sweeper went live on Oct 22 at 2:15 AM Eastern. Its first merges took 2.2 minutes for a developer change and 6.5 minutes for a robot change (which keeps a 5-minute grace period), from the second SHIP to merged. That is two merges: early, not a result.

12

What we still get wrong

A page full of numbers can make a system look finished. This one is not. Here is what we know is weak, in our own words.

Our numbers have holes

The merge totals count our main code base only. In October we split many apps into their own repos, and those are not in the 16,885. The token count comes from one account on one machine, so it is a floor. The strict-review numbers cover only Oct 18 to Oct 22, because the strict kit started on Oct 18. We would rather show a short, true window than a long, guessed one.

The tests miss too much

The tests caught 55% of planted bugs. That means they missed 1,699. The reviewers cover the gap, but a test is cheaper than a reviewer and it never gets tired. Raising that number is steady, unglamorous work.

The review kit itself fails sometimes

Of the strict review runs, 402 stopped on a problem with the kit or its setup rather than with the change. Each of those is a wasted few minutes and a job that waits. And 186 reviewers wrote no verdict at all. A missing verdict is not a SHIP, so the change waits for another review.

Jobs still bounce

From Oct 6 to Oct 22, 16% of jobs were sent back at least once, and 70 jobs needed 6+ versions. In our experience, part of the fix is on the people side: better asks, with the pass-or-fail check written first.

What is next

These are plans, not results. We will keep measuring the faster merge sweeper, now live, against its target of under one minute from the second SHIP to merged. We plan to count the app repos in the merge totals, and more accounts in the token count, so the floors in this story move closer to the real numbers. And we plan to keep doing what produced many of the rules on this page: when something breaks, measure it, fix the cause, add a check that would have caught it, and write it down.

13

Glossary

Every special word on this page, in plain language. A squash mergeSquash merge. Merging a change as one single step in the history, no matter how many small saves it had while it was being written. All terms is in here too.

Front desk

One AI window that plans the work, hands out jobs to helpers, checks their results and writes down what happened. It does not write most of the code itself.

Helper

An AI coding agent started by the front desk for one job. It works on its own copy of the code, then reports back and stops.

Lane

One stream of work with a name, such as one feature or one fix. A lane can have a helper and its own sub-helpers.

Claim

A line on a shared sign-up sheet that says "this helper is working on these files now". Another helper that asks for the same files is told who holds them.

Task tracker

Our project board. Every job has a task with an owner, the context and a pass-or-fail check written before the work starts.

Precheck

Fast machine checks that run before any reviewer: does it build, do the tests pass, did a secret or a private path slip into the change.

Strict review

A review by a separate AI reviewer that has not seen the work before. It runs the tests, tries to break the change, and must write down what it attacked.

SHIP / BLOCK

The two answers a reviewer can give. SHIP means it may merge. BLOCK means it may not, with a list of what must be fixed first.

Attack line

One written line in a review that names something the reviewer tried in order to break the change, and what happened.

Planted bug

A small mistake we put on purpose into a throwaway copy of the code, such as turning "and" into "or". If the tests do not catch it, the reviewer knows where the tests are thin.

Merge robot

A robot account that merges reviewed changes. For changes in the strict review kit, it merges only when two strict reviews said SHIP on the exact same version of the code.

Go-live

The step where merged code is built and put on the real servers. In our pipeline a robot does it after the merge.

Proof

A check on the real, live system after go-live that shows the change does what it promised, written down with what was measured.

Guard

A small automatic check that sits in front of a risky action and refuses it, for example running a command that could delete files. A guard says no; it never asks the agent to argue.

Token

The unit AI models read and write in. A token is a short piece of a word; a page of text is several hundred tokens.

Cache

Text the model has already read recently and can read again from a saved copy instead of fresh. Most of an agent conversation is the same text read again.

Model call

One request to an AI model and its one reply. An agent makes many calls to finish one job.

Squash merge

Merging a change as one single step in the history, no matter how many small saves it had while it was being written.

14

How we measured

Every number on this page was read from our own records on Oct 22. Here is how each was counted and where it falls short. Totals are aggregates; no person, client or system is named.

Each chart above already shows its own method and gap under it. The list below adds the numbers that have no chart, and repeats only the gap for the rest.

• Merged changes per monthGap: Main code base only. The app repos we split out in mid-October are not counted. April and October are partial months. One merged change is one reviewed change of any size.
• Opened changes per monthGap: October only up to the move to our own git server on October 3, so October is left off the comparison chart.
• Busiest dayMerges on the main branch grouped by day in US Eastern time, August 1 to October 21. The busiest day was Oct 15 with 462 merges; the next was Sep 24 with 445.Gap: Main code base only.
• Average per day12,191 main-branch merges divided by 77 days (August 1 to October 21).Gap: Main code base only.
• Night share and merges by hourShare of main-branch merges with an Eastern-time hour from 7 PM to 6:59 AM, August 1 to October 21.Gap: For a squash merge the commit time is the merge time; a few changes merged by hand show the author's time instead.
• Change types and sizeGap: August onward only: the local copy of the history starts July 31. Small types (build, ci, refactor, other) are left out. October is partial. Lines include generated files, data files and lock files, so read this as size, not as effort.
• AI co-author lineShare of merged changes whose message carries a co-author line naming the AI model.Gap: LOWER BOUND. Our own git server drops that line when it squashes a change, so October is not shown. Many changes without the line were also written by AI in a person's session.
• Opened by a robot account and minutes to mergeGap: LOWER BOUND. A person's account can also open a change its AI session wrote, so this undercounts AI work. Up to October 3 only. From October the reviews got stricter (review receipts from October 6, the strict two-reviewer kit from October 18), and later waits are measured another way: see the merge wait.
• Merge waitBefore: measured over 30 hours on the new pipeline (35 changes merged by the timed sweeper, 111 merged by their own owner): minutes from the second SHIP to merged. After: the same minutes for the first changes the faster sweeper merged once it went live (October 22, 2:15 AM Eastern).Gap: After is early: 2 merges so far (one developer change, one robot change), not a result. A robot change still waits a 5-minute grace period in case its owner merges it first.
• Review verdicts and strict reviewsGap: October 6 to October 22 only (receipts were first kept on October 6). 5,021 receipts do not name their code version, so they count in the totals but not in the rounds. The strict review kit started October 18, so this is October 18 to 22 only. Cost is the list-price figure the tool reports per run, not an invoice. Planted bugs are small deliberate code changes the tests must catch before any reviewer reads the code; a miss means the tests did not catch it, which tells the reviewer where to look.
• Tokens and the copied-conversation leakGap: LOWER BOUND and incomplete: one account on one machine; older records are pruned (nothing before September 9); other accounts, other engineers and other machines are not in it. The last week is partial and some work moved to accounts kept elsewhere, so the last bar is not a drop. No dollar total is given for the fleet. One measurement. After the fix (one setting) new helpers start at 20-40 thousand tokens again.
• One night, hour by hourThe front desk's night log: one line per real action, with its time and its lane. Counts of lines that say merged, blocked, proved, or stopped and refused.Gap: Lines, not distinct events: one merge can log 2 or 3 lines. Lanes include sub-helpers. The hour bars add up to one line fewer than the total.
• Helpers at onceThe fleet setting for how many helpers may run at once.Gap: A cap, not an average.
• Guards and refusalsThe number of guard checks in the shared guard set (tests not counted). Refusals: tool results in the main account's AI session records (September 9 to October 22) that carry a guard's standard refusal line, each counted once.Gap: LOWER BOUND. Only guards that print that exact refusal line are counted; other guards say no in other words.
• The traced changeThe front desk's night log, the review run's summary for that change, and the app's change log.Gap: The ask time is the work-order slot; the code time is approximate (between 23:39 and 23:56).