The Last-Mile Benchmark

Can a coding agent get the app it built safely into a coworker's hands, and how often does it need a person to help? This page has everything: the full results and the rules we published before the first run.

Short version: read the article, with the charts.

Results

Download everything

Each download holds the agent's own timeline (RUN-LOG.md), when the person was asked and when they answered (person-log.txt), and the finished code (app/). The people's email addresses are replaced with approver@example.com and coworker@example.com, and account names with placeholders. No keys are included.

• Spryloom run (zip)
• Vercel + Supabase run (zip)
• Cloudflare run (zip)

Headline

Note 1, the Vercel route's reminder email. The job ran and found the pending request. Resend then refused to send it: on the free plan without a domain, it delivers only to the Resend account's own address, and the account had been created with a different address from the approver's. The person chose not to redo the sign-up and decided to count the reminder as received. Delivery was not verified in this run.

Safe-by-default checks on the live apps

These were run by the orchestrating agent on 8 October 2026 after the runs.

The live S2, S3 and S6 checks need a signed-in coworker session on each route. They were left out to keep the person's part small.

Free-plan limits each route hit

• Spryloom: none in this run. A job runs at most once every 10 hours, which suits a daily reminder.
• Vercel + Supabase:
  • Supabase's built-in email reaches only members of the Supabase organization, so coworkers can't get sign-in or invitation emails. That is why sign-in is Google only.
  • Resend without a domain sends only to the account's own address.
  • Google keeps the sign-in app in Testing unless it has a homepage and a privacy policy. Only listed test users (up to 100) can sign in, so each new employee has to be added in Google Cloud as well as in the app.
  • Vercel Hobby runs a scheduled job at most daily. Its terms are for non-commercial use, which a company's internal tool may fall outside.
  • A free Supabase project pauses after 7 days without activity.
• Cloudflare: no email without a domain on the account or the paid Workers plan. Zero Trust Free covers up to 50 users.

Other findings

• Spryloom:
  • spry apps showed "0 people" for an app two people had used. This is a bug.
  • A changed setting reaches the app only on the next publish. The agent handled it, because the skill now says so.
• Vercel + Supabase:
  • Every signed-in page load logs an error from Supabase's sign-in library under Next.js 16 caching, although the pages work.
  • After a rollback, new deploys don't go live until someone runs vercel promote.
  • The Supabase project reports region ca-central-1, where East US was intended.
• Cloudflare:
  • The agent registered the workers.dev subdomain itself after wrangler's own attempt failed.
  • Zero Trust gave the team a random name.
  • The agent added the one-time-PIN sign-in method.

What readers need to know

1. Spryloom ran the benchmark and is one of the routes.
2. Spryloom had a practice run on this exact app; the other routes had none. A dry run on 8 October found ten rough edges in Spryloom, which were fixed and released (spry 0.1.15) before the counted runs. Some of those fixes were to the instructions the agent reads. This should be said plainly.
3. The Spryloom agent read the plugin's skill from disk and used the spry CLI, because the plugin's MCP tools weren't connected in the subagent.
4. Accounts that already existed:
   • Spryloom and Vercel (Hobby).
   • A Supabase account, with a paused 2023 project, which the rules page wrongly said did not exist.
   • Cloudflare was new.
5. The person stopped the Cloudflare run at its second stop, then resumed it about 90 seconds later.
6. There was a break of about 70 minutes in the Cloudflare run, between the handover and the coworker signing in. It is excluded from the route time.
7. Cloudflare Zero Trust Free asked for a credit card. The person went ahead and activated it, at $0. That breaks the method's rule that a run stops if a free plan asks for a card. It is reported here and in the Cloudflare row, not hidden.
8. The Claude Code installation had a Vercel plugin installed. It may have helped the Vercel route.
9. One person played both roles, with two inboxes, on one machine.

How we tested

Good to knowThis section is the method exactly as we published it on 8 October 2026, before the first run. Nothing in it has been changed since. What happened differently during the runs is listed under "Changes" at the bottom of this page.

Published 8 October 2026, before any run. We set the rules first so anyone can see the test wasn't adjusted after the results came in. Any change to this page is listed at the bottom, with its date and the reason.

The question

AI coding agents can now build a small internal app in minutes. We want to know what happens next:

Can the same agent get that app safely into a coworker's hands, and how often does it have to stop and ask a person for help?

We are not testing how well agents write code. The agent and the app stay the same, and only the route from "it works on my machine" to "my coworker used it" changes.

The app

A purchase-request tracker for a ten-person company. The agent is given this, word for word:

Build a small purchase-request app for a 10-person company.

1. People sign in. The app is private: someone who is not signed in sees nothing.
2. Only people who have been invited can use it.
3. Two kinds of user: employees and approvers. The approver is me.
4. An employee can submit a request with: item, vendor, amount, reason.
5. An employee sees only their own requests.
6. An approver sees every request, and can approve or reject it.
7. Every status change is stored with who made it and when.
8. Data survives restarts and new versions of the app.
9. Once a day, email the approver a list of requests still pending after
   PENDING_REMINDER_HOURS hours (a setting, default 24).
10. The app has a real web address another person can open.

Keep the interface plain. Build it, run it locally so I can try it, and then
make it available to my coworker <coworker's address>, so they can sign in and
submit a request.

The three routes

Each route adds one line to the prompt:

Who does what

• The agent does the work. Claude Code (Opus 5.5) does everything itself, in a fresh session and an empty folder for each route: it writes the app, puts it online, and sets up the database, sign-in, email and the reminder.
• A person helps only when asked. When the agent reaches something only a person can do, it stops and asks, for example signing up for an account, solving a captcha or clicking an email link. The person does exactly that and nothing more. They never edit code and never suggest fixes.
• The coworker is a second real inbox. They open the invitation and submit a request in their browser.

Free plans only

Every route uses only free plans. No card is entered and no paid trial is started. If something the app needs isn't available on a free plan, we record that and carry on with the rest.

What we measure

• Stops: how many times the agent had to stop for a person, and why. This is the headline number.
• Time: from the prompt to a working app on the agent's machine, and from there to the coworker's request being approved online. Agent time and person time are counted separately.
• Moving parts: the accounts, services and secrets the finished app depends on, and the configuration files written to deploy it.
• Retries: deploys that failed and had to be tried again.

What we check

Every finished app gets the same fifteen checks.

What the app does:

• strangers see nothing;
• people who weren't invited can't get in;
• employees see only their own requests;
• the approver sees and decides on everything;
• every decision is recorded;
• data survives a new version;
• the reminder email arrives;
• the logs can be read;
• the app can go back to an earlier version.

Safe by default:

• the data can't be read without signing in;
• a coworker can't open someone else's request by changing its number;
• a coworker can't approve requests;
• nobody can pretend to be someone else;
• no secret shows up in the code, the browser or the logs;
• a coworker can't make themselves an approver.

Honest about the setup

• We make Spryloom, and Spryloom is one of the routes. That is why the rules are published first, and why every result will be published in full.
• No "agent's choice" route. The Claude Code installation running the test has a Vercel plugin installed, which would steer an agent left to choose. We left that route out rather than report a result the plugin decided.
• Accounts that already existed are disclosed. Spryloom and Vercel accounts existed before the test. Supabase and Cloudflare accounts did not, and are created during the run if the agent asks for them.

What we will publish

• each route's full results: timeline, every stop, the agent's transcript with secrets removed, check results and screenshots;
• a results table, and the raw numbers as a CSV.

There is no combined score. You can draw your own conclusions.

Changes to this page

8 October 2026, after the runs:

• The results are published, with the article.
• A Supabase account already existed. This page said it did not; it held a paused project from 2023, which was left untouched.
• Cloudflare's Zero Trust free plan asked for a credit card. These rules said a run stops there. The person entered one (nothing was charged) and carried on, and the results say so.
• The person stopped the Cloudflare run once, then resumed it a minute and a half later.
• The Spryloom route had a practice run on this app, and the rough edges it found were fixed before the counted runs. The other routes had none.