Automate the job, not the typing

The experiment

I wanted to know how far one software engineer could push the automation of software engineering.

Not in a demo repository. Not by asking an agent to generate a to-do app and declaring victory. I tested it on an independently operated product with more than 500 users, a real database, paid data, deployments, caches, browser journeys, and consequences when the system lies.

The boundary matters. I control this side project's product direction, repository, deployment path, and acceptable blast radius. I would not use the same autonomy in my primary job without that environment's security, review, legal, ownership, and incident controls.

The experiment started with a familiar question:

That question became obsolete surprisingly quickly. The agents could write plenty of code. In one stretch they delivered more than a dozen pull requests, a design overhaul, onboarding changes, dashboard work, and new product flows. I steered much of it from my phone. The machine did most of the typing.

The real constraint moved somewhere else.

That is the question this page answers.

The short version is that I stopped writing most of the code. I did not stop being the engineer. I kept the work that became more valuable as implementation became cheaper: framing the outcome, decomposing the system, deciding what evidence counts, judging the product, setting authority boundaries, and approving production.

The job after code gets cheap

Software engineering was never one activity. Typing code merely consumed enough time to hide the rest.

Once implementation accelerated, the job separated into responsibilities with different delegation boundaries:

The split is not human versus machine. It is judgment versus execution, and authority versus capability.

An agent may be capable of fetching a credential, merging a branch, changing a schema, or writing a plausible number into a card. Capability does not imply permission. A confident result does not imply proof. A green test does not imply a safe release sequence.

The useful unit of automation is therefore not a line of code. It is a responsibility with five parts:

1. an input;
2. permitted actions;
3. required evidence;
4. a stopping condition;
5. an accountable owner.

If any of those are missing, the task is not automated. It is merely moving quickly.

The operating model

The model has six moves. They form one loop, not six independent destinations.

1. Frame the outcome

The human decides what should change and why. The task brief names the visible outcome, what may change, what must not change, the evidence required, and the conditions that should stop the worker.

# Outcome

Remove the fabricated engagement count from every article.

## May change

- Article UI and focused tests

## Must not change

- Published article copy
- Analytics configuration

## Evidence required

- Search returns zero matches for the old label
- Production build passes
- Staging HTML does not contain the fabricated count

## Stop and ask

- A real count source is discovered
- The change requires a database or tracking decision

This is not prompt decoration. It is the first authority boundary. The worker can choose implementation details inside the lane. It cannot silently choose a new product direction.

2. Bound the responsibility

Before dispatch, identify the code, schema, data, workflow, cache, environment, and external actions the change can touch. If the blast radius is unclear, the task is still discovery.

The boundary should be physical where possible. Independent work gets an isolated worktree and a written ownership contract:

git worktree add ../lane-remove-metric -b task/remove-metric
cd ../lane-remove-metric
npm ci
npm test

In one live experiment I split five genuinely independent changes into five worktrees. They completed with zero file collisions. That was not luck. Collision freedom was a property of the decomposition.

The difficult part moved upward. Choosing slices that were truly independent became the scarce skill. Five agents in one shared lane is not orchestration. It is fleet cosplay.

3. Delegate execution

The worker receives the bounded goal and owns the build loop:

inspect
  -> propose
  -> implement
  -> run focused proof
  -> inspect the result
  -> repeat or stop

A useful agent loop has four controls:

• a measurable stopping condition;
• verification on every iteration;
• a hard budget on attempts, time, tokens, and external actions;
• an escalation path that produces a useful handoff instead of guessing.

The worker should stop at a named state. “Done” is not a state. “Built with focused tests green and no production authority exercised” is.

4. Verify independently

Every worker report is a claim.

The verifier should inspect the diff and the actual artifact without inheriting the builder's story as truth. It should search for a sequence the happy-path tests missed, derive at least one oracle independently, and prove the negative path of every new gate.

This separation caught two real release defects after more than 1,700 tests, a production build, and a successful staging walk. The bugs were not in the final state. They were in time.

5. Promote by evidence

Promotion is a state transition, not the builder's reward for finishing.

The release moves forward only when observable evidence exists for the exact target and exact diff. Staging proof does not authorize production. A human approves production after reviewing the release boundary, evidence, unresolved risk, and rollback.

6. Encode the lesson

Every meaningful failure should leave infrastructure behind.

The first time a class of bug reaches a human, the catch is useful. The second time, it is a process failure. Convert it into the cheapest durable owner:

• a unit or integration test;
• a data contract;
• a workflow gate;
• a browser or post-deploy canary;
• a versioned skill;
• a playbook;
• a monitoring assertion.

Prose creates recognition. Controls create reuse.

The control loop

The system is easier to reason about as two loops with different evidence contracts.

Loop one: product taste

The agent proves the route, state, viewport, and rendering. The human decides whether the language lands, the hierarchy feels calm, the number looks honest, and the interaction deserves to exist.

One measured UI change exposed why this separation matters. Required pull-request checks took a median 5 minutes 56 seconds. Staging added 3 minutes 35 seconds. The first proper look at the change arrived about 9 minutes 40 seconds after the edit. One test file spent 73.8 seconds patiently waiting through a real 1.2-second animation around forty times.

The answer was not fewer checks. It was earlier evidence:

for (const [name, viewport] of [
  ['desktop', { width: 1440, height: 1000 }],
  ['mobile', { width: 375, height: 812 }],
] as const) {
  await page.setViewportSize(viewport)
  await page.goto('/occupations')
  await page.getByRole('button', { name: 'Explain statuses' }).click()
  await page.screenshot({
    path: 'artifacts/taste/status-' + name + '.png',
    fullPage: true,
  })
}

Staging stopped being an expensive image viewer. It returned to its real job: proving integration in the deployed environment.

Loop two: promotion confidence

Static checks, sharded tests, relevant database gates, browser lanes, staging, and post-deploy checks can run in parallel where their facts are independent. Stable aggregators still fail closed at the policy boundary.

Do not start a database to check some CSS. Do not delete a safety check merely because it is slow. Classify the risk, run the relevant evidence lanes, and keep a scheduled full matrix as the backstop for a bad classifier.

Evidence from real failures

The framework did not emerge from a whiteboard. It came from repeated failures that changed the operating system.

A gate caught the person building the gate

Hardcoded routes had left dead links behind after a page moved. The first fix was a route registry. The real fix was banning the alternative:

export const ROUTES = {
  DASHBOARD: '/dashboard',
  MONEY_TOOLS: '/dashboard/money-tools',
  occupation: (code: string) => '/occupations/' + code,
} as const

A custom lint rule made hardcoded route strings an error. A duplication detector became a ratchet instead of an unrealistic zero-tolerance cleanup.

Partway through the migration, I was interrupted. Half the routes were still hardcoded. The build refused to go green. My own half-finished work could not pass the gate I was building.

That is the distinction:

The gate also taught a second lesson. Its first wrapper piped the detector output and swallowed the failure status. The log said “error” while CI said green. A check whose failure signal is laundered is worse than no check because it manufactures confidence. I now distrust a gate until I have deliberately watched its negative path fail.

A safety system refused a correct fix

A data-seeding worker encountered half-rotated credentials, diagnosed the problem correctly, and attempted to fetch a replacement secret. An independent permission system refused twice.

The worker stopped and produced the exact five-minute human repair. That refusal was not friction. It was the architecture working.

The guardrail had four properties worth copying:

1. separation of duties, because the requester could not approve itself;
2. context awareness, because it judged credential provisioning rather than blocking an entire API;
3. fail-closed behavior when judgment was unavailable;
4. denial with a useful human handoff.

Autonomy is capped by verification, not trust. The cap only counts if the thing it caps cannot lift it.

The reviewer found bugs in time

The release depended on a new dataset and cached its read model. Staging proved the final state:

data exists
schema exists
code is deployed
cache contains a valid result

Production was a sequence:

data workflow waits for approval
application deploy starts
application reads an empty table
cache stores the empty result
data workflow finishes

The independent reviewer blocked the release, then blocked the first fix too. A later copy-only commit could inherit an earlier commit's unmet data obligation while a file-diff check saw no relevant change.

The durable rule was a graph invariant. Let R be the latest dependency-bearing commit, W a successful data-workflow commit, and D the release:

R is an ancestor of W
W is an ancestor of D

git merge-base --is-ancestor "$required_sha" "$workflow_sha"
git merge-base --is-ancestor "$workflow_sha" "$release_sha"

“Run the data pipeline first” relies on memory. “No release can pass without a covering success in its ancestry” is a system.

The arithmetic was correct and the product was wrong

One product surface displayed 534. The source published 975. The code correctly summed every exact detailed row it could see, but many detailed cells were censored as <20. The exact source total existed at a coarser grain.

The bug was semantic, not arithmetic.

type PublishedCount =
  | { kind: 'exact'; value: number }
  | { kind: 'censored'; source: string }
  | { kind: 'floor'; value: number; omittedCells: number }

The repair stored the two grains separately and added an invariant:

export function assertCoarseTotalCoversDetail(coarseExact: number, detailedFloor: number) {
  if (coarseExact < detailedFloor) {
    throw new Error('Coarse total is below the detailed floor')
  }
}

Two screenshots became the input to the investigation. The team normalized the visible claim, traced one value backward from label to source, located the first divergence, and preserved the finding as a failing contract.

Build the system

Start with one bounded workflow. Do not begin with a general autonomous engineering platform.

Step 1: choose a real responsibility

Pick a task that repeats, has visible evidence, and has a cheap rollback. Good first candidates include:

• remove a deprecated UI pattern across known routes;
• turn a recurring review catch into a CI assertion;
• update a read model with a source-to-screen fixture;
• create a staging canary for a concrete failure class.

Avoid credential changes, destructive data operations, broad architecture migrations, and production promotion until the evidence system is already trustworthy.

Step 2: write the contract before dispatch

Use the outcome, may-change, must-not-change, evidence, and stop-and-ask structure shown earlier. Include the production boundary and any experimental work explicitly excluded from the release.

Step 3: separate builder and reviewer

The builder optimizes for completion. The reviewer optimizes for falsification. Give them separate context and separate authority.

Builder
  implement the scoped outcome
  prove focused behavior
  prepare staging evidence
  stop before production authority

Reviewer
  inspect the complete diff independently
  reconstruct dependencies
  find temporal and lineage failures
  exercise negative paths
  report blocked or ready

Step 4: build the verification wall

Use the cheapest owner for each fact:

Every new gate must demonstrate its own failure. A green-only history proves almost nothing.

Step 5: make state observable

Replace remembered instructions with evidence-bearing states:

scoped
  -> built
  -> staged
  -> independently reviewed
  -> ready for production
  -> approved
  -> released
  -> learned

Each transition should name the facts that make it true. “Staged” includes a deployed URL, passing checks, and a browser observation. “Approved” attaches to a specific diff and CI state. If the diff changes, the approval expires.

Step 6: measure the whole feedback system

When implementation becomes fast, every dependency it waits on becomes product architecture.

Measure:

• time to first human-visible evidence;
• pull-request critical-path time;
• runner time and expensive service startup;
• number of taste iterations before staging;
• number of staging promotions per product decision;
• failures caught by the builder, reviewer, gate, canary, and human;
• actions per human instruction;
• exceptions escalated to the human.

Optimize the constraint you can prove. A faster agent inside a serial pipeline is still waiting. A faster pipeline that deleted its evidence is merely failing sooner.

The autonomy boundary

The system does not have one global autonomy level. Every responsibility earns its own.

I use three questions before increasing it:

1. How quickly will I know the system is wrong?
2. How cleanly can I undo the action?
3. What evidence would prove the result is right?

If any answer is weak, the responsibility drops a level.

The human retains:

• product direction;
• ambiguous requirements;
• taste and language;
• changes to credentials and authority;
• destructive or irreversible operations;
• the acceptable blast radius;
• final production approval.

Agents may have broad authority to inspect, implement, test, challenge, and prepare. They have narrow authority to mutate external state. A reviewer may block a release but cannot merge it. A worker may diagnose a credential problem but cannot grant itself a stronger credential.

High autonomy is not removing the human. It is moving the human from performing every action to making the few decisions that change the direction or the risk.

Definition of done

The page is the model, but the model should end as a usable checklist.

A responsibility is ready for bounded automation when:

• the outcome and exclusions are explicit;
• the mutable surface is known;
• the worker runs in an isolated lane;
• the stopping condition and budget are measurable;
• every iteration produces evidence;
• a separate reviewer can falsify the result;
• each gate has a proven negative path;
• staging verifies the deployed artifact, not the builder's report;
• production approval belongs to a named human;
• rollback is understood;
• failures become durable controls;
• the evidence survives a session reset.

The aim is not maximum autonomy. It is the highest autonomy you can defend.

When this works, code becomes the inexpensive middle of a stronger system. The human frames the game and owns the irreversible calls. Agents supply speed, breadth, and persistence. Tests, gates, reviewers, and canaries make their claims expensive to fake.

That is how far I have pushed software-engineering automation on a real side project.

I would not dare copy the autonomy level blindly into my primary job.

I would copy the method: bound the responsibility, demand evidence, keep authority separate, and let every failure improve the machine.