The Goldilocks Problem, Two Years Later

Give an agent too little room and it interrupts so often that nothing gets done. Give it too much and you’re writing an apology to a customer.

In 2024, Sequoia called the middle ground “Goldilocks” agents. Back then the question was how much control flow to hand the model, somewhere between a hard-coded chain and an AutoGPT that wandered off. Two years later, agents send messages, update records, and move money. The Goldilocks question has moved from control flow to action.

Today’s models can do the work. Capability is no longer the limiting factor. It’s a question of how people, agents, and the systems around them work together. I find it helps to break it into three questions.

That’s what we’ll dig into at our next LA CTO Forum session.
Join Tony Karrer and four speakers who have shipped agents in production for Lessons Learned Building Agentic Systems, a free online session on Friday, November 13, 2026 (8–10 AM PT / 11 AM–1 PM ET). They’ll cover guardrails, context, what breaks, and what to do about it.
Can’t make it live? Register anyway and we’ll send you the recording.
Reserve your spot →

How much is the agent allowed to do?

Not every action carries the same risk. A recent paper on agents in regulated settings, from researchers at fortiss, a Munich-based research institute for software-intensive systems, lays out a ladder: the agent reasons over what it’s given, uses safe local tools, reads from external systems, writes things that are easy to undo (a draft, a pending record), and finally commits things that aren’t (a filing, a payment, an update to the system of record). As the authors put it, whether a human has to approve an action is a design choice, but whether the action can be undone is a property of the action itself.

A simpler test: is it reversible, and does it leave the building? Zachary Proser, who builds agent harnesses at WorkOS, lets his agents act alone on anything reversible and inbound, and stops for a human on anything irreversible or outbound.

The practical move is to shrink what the agent can touch before you add a human:

  • Fence the tool. Limit which records or fields it can reach, enforced in the tool itself, so an out-of-bounds call fails no matter what the model was told.
  • Stage the write. The agent drafts; something else commits.
  • Withhold the credential until it’s earned. An agent that doesn’t hold the “send” permission can’t send.

None of these depend on the model following instructions.

How often does a human check?

There are two basic patterns. A checkpoint is a review at a fixed step, every time. Escalation is the agent pausing when it’s unsure: low confidence, conflicting inputs, something out of scope. Checkpoints are predictable and easy to audit. Escalation scales better, but only if it’s specific. Vague or frequent escalations wear people out and erode trust.

The fortiss paper describes a document-routing agent that started with an expert reviewing every item. As accuracy improved, the team dropped the checkpoint and routed only low-confidence items to a person.

That’s the pattern to aim for: earn autonomy one action type at a time, against criteria you set in advance. An AWS reference design for graduated autonomy sums it up as “promote slowly, demote immediately.” Treat a model upgrade as a reason to look again.

Do humans help at all?

Here’s the uncomfortable part. In a recent study, people supervised an AI assistant through a simulated day that included out-of-scope actions like permanently deleting an archived timesheet and buying $12 of travel insurance nobody asked for. With approval required on every action, about 40% of those actions still got through. When a model flagged them first, people approved them anyway. When people wrote their own rules up front, they set most of them to “ask”, then said yes to almost every request that came back, even when the agent was overreaching.

It’s one small study, but it matches what anyone who has watched an approval queue already knows. When nearly everything is fine, people stop looking. A Data & Society report, The Oversight Fallacy, puts it well: “A loop can exist formally while failing substantively.”

Four ways to make the check work:

  • Check fewer, bigger things. Approve the staged draft, the batch, or the budget, not every call.
  • Show the work, not just the button. Put the evidence and the steps the agent took next to the decision, at the level of detail that reviewer needs.
  • Give reviewers authority, and measure them. Who can halt, revoke, and rerun? What’s the approval rate? When did someone last say no?
  • Plant things that should be caught.

What we’re doing

That last one is the practice we lean on most in our healthcare and legal work. We intentionally plant mis-identified results in what human reviewers see, then track whether they catch them. The idea isn’t new: airport baggage screening has projected fake threats into X-ray images for decades to keep screeners alert, and medical researchers have used deliberately wrong AI advice to test whether physicians catch it.

Roughly how it works for us:

  • Evals decide when a result needs a human, and that share drops as the system matures.
  • We also sample 1–5% of results for human review, depending on volume and maturity.
  • We plant fakes in roughly 5–10% of what reviewers see. We’re still testing what rate works best.
  • Our clients see results too, and they tell us when we get something wrong.

It has improved the accuracy of human review, especially over time. One lesson: we had to keep improving the fakes. People are very good at pattern recognition, and once the planted errors start to look alike, reviewers learn to spot the fakes instead of the errors.

The answer to the Goldilocks problem isn’t one setting. It’s deciding what the agent can touch, deciding where people check, and then checking the checkers.

Reading list

Sources cited