Working with agents, the bottleneck is me

Claude has been doing great work for me, and I used up this week’s quota early. So I gave it a raise: I upgraded from Pro to Max.

After the raise, I didn’t want to waste the quota. I wanted it working flat out, so I opened three sessions at once and had it do three different things.

I knew right away something was off. It was fast. Before long all three were done and waiting for me. Every finished session became another item on my to-do list: what did it change, is it right, where do we go next. The three tasks had nothing to do with each other, and I kept switching between the three sessions, finishing one only to face the next.

My brain just couldn’t keep up.

The AI wasn’t slow at all. I was.

The slowest link

There’s a management approach called the Theory of Constraints. The idea is simple: how fast a line runs isn’t set by the average, but by its slowest link.

In The Goal, Goldratt gives an example. A troop of Boy Scouts goes on a hike. The boys walk at about the same average pace, yet the line keeps stretching out, and in the end the whole troop moves at the speed of the slowest boy. However fast the others walk, they just end up waiting for him up ahead. The book also says that a plant where everyone is busy all the time is not the most efficient plant.¹

Working with AI is the same. Writing code, looking things up, organizing documents: AI keeps getting faster at each of them. More and more often, the slowest link on the line is a person.

How the bottleneck moved to me

Trace logs I could never get through

At Blacklake I built a delivery assistant. Every time it handled a user’s question, it left a trace log: every step the AI took on that question, which tools it called, what it got back along the way, and how it answered in the end.

Later some users said the results weren’t great in places: they still had to touch up what the AI produced.

I made some small improvements based on experience, but I never really went through the trace logs. I opened a few in the browser and got tired before long. What the users raised ended up sitting there.

Between hand-written code and handing it all to AI

Back in January and February, while building the delivery assistant, a good part of the code was already written by AI. But I still read what it wrote: I’d open git diff, the old version on the left and the new one on the right, and look back and forth to see what had changed. Some parts I simply wrote by hand, including the core agent logic.

When it came down to it, I didn’t quite trust the AI. I always felt what it wrote might be wrong and needed a look. Though even after reading through it, I’m not sure I really understood it.

That was no longer coding entirely by hand, but it was far from handing everything to AI. I was about halfway there.

Then I joined a startup, and the world changed completely: all the code went to AI. I barely wrote any myself; when I wanted something changed, I just told it.

Yet I couldn’t drop the habit of reading git diffs. My colleagues had long stopped bothering. Only I was still sitting there, a glance to the left, a glance to the right. The AI finished in no time, and I took ages to read it.

That’s when I realized the thing holding us up was me: I was the one who insisted on checking how the AI changed things.

Human attention is limited, and I was spending mine comparing code line by line.

Not enough attention

Every time the AI finishes a job, someone has to look at it. When looking, there are two places where it gets too much.

Too many things to keep track of

There’s only so much a person can look at carefully in a day. That’s true of the trace logs AI produces, and of the three sessions at the start: in no time there are three to-dos waiting for me to check, and as soon as I finish one, there’s another.

Three sessions on the same task, I can manage; my head is holding one thing. Three completely different tasks, and it can’t keep up.

The psychologist Nelson Cowan’s research suggests the mind can hold only about four things at once.² And switching back and forth between tasks wears you out in itself.³

Fine-tuning takes patience

AI is fast at the start; a few rounds and the thing takes shape. Then, when it comes to fine-tuning, you have to slow down: look at one spot, fix one spot, settle down and close in bit by bit. What it takes then is patience, keeping your eyes on it and staying with it round after round.

So I now think that in the age of agents, attention is the most precious thing. The question becomes: what do I keep an eye on myself, and what do I let the AI watch? AI is a kind of intelligence; it has its own attention too. A person’s attention should go to just a few things, and the rest can go to AI.

Let AI keep watch

Hand the logs to AI

Only later did I learn that trace logs nobody can get through are a problem everyone building agents runs into. Early this year Harrison Chase, a co-founder of LangChain, wrote that once an agent is live it produces thousands of traces a day; some teams he’d talked to record more than a hundred thousand a day, and there’s no way for people to read them one by one.⁴

Thousands a day. Who could get through that?

The approach now is, broadly, to hand the reading of logs to AI:

• AI first tags and scores each trace: what the user was using it for, whether it succeeded, whether the user was happy.
• Then similar traces are automatically grouped, so you can see which kind of problem is most common and which has suddenly grown lately.
• People only look at the groups, open a few examples, and decide what to fix first; confirmed failures are kept as future test cases.

It’s a bit like AI quality inspection on a factory floor: AI checks every single product, and problems are passed up level by level according to how serious they are. People sit at the top and only deal with the few that are most serious and urgent. Reading logs works the same way.

A few days ago I heard a talk at DevFest Shanghai about this same kind of loop. One line on the slides put it bluntly: no need for people to read through mountains of logs.⁵

My sense is that people will read trace logs less and less: read some at the start yourself and decide what counts as right; once things are stable, hand it all to AI.

No more reading code line by line

On the coding side, I first talk the plan through with the AI, then let it write. The instructions file for the AI (AGENTS.md) also spells it out: after finishing each task, write unit tests, open a browser and walk through it, and fix any problems it finds.

That means fewer mistakes, and the fewest problems left for me. Then I check by feature, without reading code line by line. I’ve written about how I do this in I no longer read AI-written code line by line.

Don’t keep switching

You can open several sessions; the hard part is checking them. Even with three open, it’s best to handle them one at a time: finish with one, then look at the next, and let the long-running one keep going on its own. The worst is clicking here, then there, and getting dizzy right away.

The AI doesn’t get dizzy. We do.

How many to open, and whether I can check them all, I’m still figuring out.

Try handing it to AI first

When something looks like it’ll take a lot of time, I try handing it to AI first to see how long it takes. The result often surprises me.

Today’s portfolio for my blog was like that. The way I used to think about it, I’d have to dig up every project I’d done and every event I’d taken part in, then write a description for each. At least a week of work. Just thinking about it wore me out.

The AI sorted it all out in somewhere between a few minutes and half an hour, and I only polished the wording. An hour or two from start to finish, and the portfolio was live.

What surprised me even more was something else. When I was a Microsoft Learn Student Ambassador, I backed up my Outlook mailbox before the account was closed, thinking it might come in handy one day. I never opened it again and forgot it existed. The AI found it on my hard drive by itself, unpacked it, went through the emails, and pulled out the events I’d run back then, one by one. It even turned up my Teams chats with teammates and friends from that time.

So if you have a problem, just throw it at the AI and see whether it can do it.

Back to agent products

Back to the start. I gave Claude a raise, and it can do more and more, faster and faster. What’s actually in short supply is my own attention.

For agent products, I think two things matter.

First, memory: how deeply it knows the industry, and whether it remembers this customer’s own situation and habits. The more it knows, the less people have to explain.

Second, saving people’s attention: if an agent is really there to do the work, it should get things done, not keep people watching over it. Several new products are heading this way:

• In early November DeepSeek open-sourced a Super Agent prototype. Its description says the goal is to take as little human attention as possible and give as much value as possible.⁶ An article about it adds that DeepSeek’s research finds one engineer can effectively oversee at most three to five agent sessions at once.⁷
• xAI’s Grok Bot lets you set up a “CEO” bot. You only talk to it, and it hands out work to the other bots.⁸
• Meta’s Muse keeps running in the background after you close the app, and when there’s something new, it decides for itself whether it’s worth telling you.⁹

If people still have to keep watching, no attention has been saved at all.