What running a team of AI agents taught me about applied AI in product management

One day, one request

At 1:03 pm on October 10, I sent the lead agent on Beast, my Linux workstation, an Instagram post of seven new nebula photos and one line: “Create sims of these new nebulas please.” Nine minutes later I added “VR version too please.” I wasn’t home, so I asked for phone stills instead of headset demos.

The lead didn’t render anything itself. It started a helper agent for the simulations, kept the plan, and reviewed what came back. At 1:44 the first Ring and Hourglass renders arrived. The lead held them back: next to the photos they looked like smooth 3-D models, not glowing gas. A rebuilt version reached my phone at 2:01.

At 5:15 the first finished loop landed: the Squid Nebula, in 4K, looping seamlessly, with zero flashes on the photosensitivity check. I replied that it just showed stars drifting in front of the nebula, and that the camera should circle it. Twenty minutes later the helper had built an orbiting camera for all four loops.

At 7:28 I said I’d rather have interactive sims than 360° films, because the films take too long to download. The lead stopped the 360° render that minute and started a second helper on interactive pages. At 8:25 the first one reached me: a Squid you can turn in your hand, 2.3 MB, opening in seconds. I wrote back “Hell yeah! That looks great.”

At 8:28 a third helper started porting the sims to Everdust’s native code. It built and tested on Cosmo, the Mac Studio, over the network, under the Mac’s shared rules. By 9:50 all four nebulas were ported, had passed their checks on the Mac, and were handed to the Mac’s lead agent with a row in our shared ledger. Three 4K orbit loops finished before midnight. The fourth is on hold until a GPU frees up.

I steered all of it from my phone in short Telegram messages. In the same hours, the same lead also fixed dead links on this site and started a paper.

Lesson learnedWith agents, my job became writing clear asks and making fast taste calls. The building was the easy part.

Design for interruption

At 11:40 am on October 2, nineteen helper agents were running on Beast at once: long-form films, VR scene packs, a film score. At 11:57 the weekly usage limit arrived. Within about half an hour, 18 of them had stopped mid-task.

Finished files on disk survived. Almost nothing that was running did, because the renders had been started from inside the agents and died with them. Everything that lived only in an agent’s working memory stopped too: plans, half-written pages, the next step. That evening I switched to another subscription, and each agent was resumed with a message. It picked up with its context intact.

The next evening another limit stopped the agents again. This time the GPU renders and video encodes kept going, because by then they ran as separate jobs. Limits hit again on October 4 and 5, and each time the same rule decided what survived.

We changed how work is built. Long work runs as a detached job with its own log, outside the agent. Long simulations save checkpoints, points they can restart from, and renders skip frames that are already on disk. Chains of jobs wait for a “done” file, not for a process name. That makes each step idempotent: running it twice does no harm, so a resumed agent can simply redo its last step. A resume message says where things stand. And there is now a cap of five live agents, testers included.

Lesson learnedAssume every agent will be stopped mid-sentence. Put the real work in jobs that survive it, and make every step safe to run twice.

Two AIs that can’t talk directly

The lead on Beast and the lead on Cosmo are separate sessions on separate machines. Neither can read the other’s mind or chat. For weeks, handoffs were note files dropped into a shared folder, and nobody acknowledged them. The Mac acted when I relayed things. On October 8 I told them: “I don’t feel like you guys have been in sync without me intervening.”

The fix went in that day. We created one ledger file with one row per open item: what it is, from whom to whom, which note has the details, its status, and who has acknowledged it. Both leads read the ledger at the start of every session. Nobody deletes a row. The Mac’s copy is the master, and it gets checked for changes before anyone overwrites it, because a stale copy once lacked three rows. The ledger grew to 27 rows in its first three days. The nebula port above is row 24.

Lesson learned“Done” means written down where the other side will look, and acknowledged. A handoff nobody confirmed hasn’t happened.

Ownership boundaries

On August 25, another session on Beast, following my instruction to free the GPUs, killed this session’s training job and image server. A minute later this session restarted its server, which crashed the other session’s freshly relaunched engine. Neither agent did anything wrong. Nobody owned the GPU.

The quieter version took longer to see. Cosmo pulls Beast’s web pages every ten minutes and commits them to the app. When Cosmo’s agents fixed a page, the next pull copied Beast’s older version back over the fix and committed it, silently. It happened at least twice before Cosmo paused the pull.

The fixes are all about ownership. A lane is a resource with one named owner: GPU 1 on Beast now belongs to Cosmo’s Unreal lane, with a lock folder anyone can check before using it. The Mac has two build slots, claimed by creating a lock folder, which only one agent can do at a time. There is a separate lock for the simulator, and a memory gate: no build starts while the Mac’s memory is under pressure. Processes are killed only by the ID their owner recorded, never by matching a name. And nobody touches what isn’t theirs.

October 10 showed it working. The render disk filled up and killed a VR render, because the Unreal lane’s working copies had grown past their share. It took one ledger row and a nudge from me. Cosmo’s agents pruned their copies and adopted a per-lane disk budget, and nobody had to guess whose files they were.

Lesson learnedEvery shared resource needs one named owner and a lock anyone can check. “Don’t touch what isn’t yours” only works once ownership is written down.

You can’t review 2,000 agents’ work, so build gates

No person can watch everything two thousand helper agents produce. So every render passes automatic checks before anyone looks. A photosensitivity gate counts flashes per second in every patch of the screen. Loop checks make sure the last frame flows back into the first. Other checks count particles that escape a simulation, scan for invalid numbers, and compare web and native images side by side. They catch regressions without anyone watching.

They don’t catch “wrong.” On October 10 the Horsehead Nebula look-dev scored 0.74 on our shape-overlap metric against the reference photo, with its colour within 2 degrees. It still looked like a cartoon sock puppet with an open, glowing mouth. The lead kept it out of the pack. Earlier that day, the lead had also caught a version that traced the photo’s outline into the simulation. That shortcut raised the score, and the metric couldn’t see it. The lead spotted it by reading the helper’s report closely.

Lesson learnedAutomatic gates catch regressions. Human taste catches “wrong.” You need both, and the gates should run before a person looks.

Where the human stays

I approve anything public. The leads filter before anything reaches me: that first Ring and Hourglass, the sock-puppet Horsehead. I make the taste calls, like circling the camera and choosing interactive pages over 360° films, and I set priorities across projects. Telegram is the control plane: short messages, phone stills, a yes or a no.

What ran out was neither compute nor ideas. Cosmo’s queue was always backfilled. The limit was how many subscriptions I had: 36 usage-limit stops in 21 days. So I now manage quota the way I’d manage headcount. I cap how many agents run at once, count testers toward that cap, run reviews one after another instead of in parallel, and wind down at night.

Lesson learnedKeep the human on taste, priorities and anything public, and budget AI quota like headcount. Quota runs out first, not compute.

Lessons learned

• Clear asks and fast taste calls are the manager’s real leverage. The building is the easy part.
• Assume every agent gets interrupted: put long work in jobs that survive it, and make every step safe to repeat.
• A handoff is done when it’s written where the other side looks, and acknowledged.
• Give every shared resource one named owner and a lock anyone can check.
• Let gates catch regressions automatically, and keep human eyes for “is this right?”
• Budget AI quota like headcount. It ran out before compute or ideas did.

By the numbers

A helper agent is a separate AI worker that a lead starts for one task. “At once” counts helpers that produced output in the same 10-minute window. Commits are unique across both machines.