The Goldilocks Problem, Two Years Later

Give an agent too little room and it interrupts so often that nothing gets done. Give it too much and you’re writing an apology to a customer.

In 2024, Sequoia called the middle ground “Goldilocks” agents. Back then the question was how much control flow to hand the model, somewhere between a hard-coded chain and an AutoGPT that wandered off. Two years later, agents send messages, update records, and move money. The Goldilocks question has moved from control flow to action.

Today’s models can do the work. Capability is no longer the limiting factor. It’s a question of how people, agents, and the systems around them work together. I find it helps to break it into three questions.

How much is the agent allowed to do?

Not every action carries the same risk. A recent paper on agents in regulated settings, from researchers at fortiss, a Munich-based research institute for software-intensive systems, lays out a ladder: the agent reasons over what it’s given, uses safe local tools, reads from external systems, writes things that are easy to undo (a draft, a pending record), and finally commits things that aren’t (a filing, a payment, an update to the system of record). As the authors put it, whether a human has to approve an action is a design choice, but whether the action can be undone is a property of the action itself.

A simpler test: is it reversible, and does it leave the building? Zachary Proser, who builds agent harnesses at WorkOS, lets his agents act alone on anything reversible and inbound, and stops for a human on anything irreversible or outbound.

The practical move is to shrink what the agent can touch before you add a human:

• Fence the tool. Limit which records or fields it can reach, enforced in the tool itself, so an out-of-bounds call fails no matter what the model was told.
• Stage the write. The agent drafts; something else commits.
• Withhold the credential until it’s earned. An agent that doesn’t hold the “send” permission can’t send.

None of these depend on the model following instructions.

How often does a human check?

There are two basic patterns. A checkpoint is a review at a fixed step, every time. Escalation is the agent pausing when it’s unsure: low confidence, conflicting inputs, something out of scope. Checkpoints are predictable and easy to audit. Escalation scales better, but only if it’s specific. Vague or frequent escalations wear people out and erode trust.

The fortiss paper describes a document-routing agent that started with an expert reviewing every item. As accuracy improved, the team dropped the checkpoint and routed only low-confidence items to a person.

That’s the pattern to aim for: earn autonomy one action type at a time, against criteria you set in advance. An AWS reference design for graduated autonomy sums it up as “promote slowly, demote immediately.” Treat a model upgrade as a reason to look again.

Do humans help at all?

Here’s the uncomfortable part. In a recent study, people supervised an AI assistant through a simulated day that included out-of-scope actions like permanently deleting an archived timesheet and buying $12 of travel insurance nobody asked for. With approval required on every action, about 40% of those actions still got through. When a model flagged them first, people approved them anyway. When people wrote their own rules up front, they set most of them to “ask”, then said yes to almost every request that came back, even when the agent was overreaching.

It’s one small study, but it matches what anyone who has watched an approval queue already knows. When nearly everything is fine, people stop looking. A Data & Society report, The Oversight Fallacy, puts it well: “A loop can exist formally while failing substantively.”

Four ways to make the check work:

• Check fewer, bigger things. Approve the staged draft, the batch, or the budget, not every call.
• Show the work, not just the button. Put the evidence and the steps the agent took next to the decision, at the level of detail that reviewer needs.
• Give reviewers authority, and measure them. Who can halt, revoke, and rerun? What’s the approval rate? When did someone last say no?
• Plant things that should be caught.

What we’re doing

That last one is the practice we lean on most in our healthcare and legal work. We intentionally plant mis-identified results in what human reviewers see, then track whether they catch them. The idea isn’t new: airport baggage screening has projected fake threats into X-ray images for decades to keep screeners alert, and medical researchers have used deliberately wrong AI advice to test whether physicians catch it.

Roughly how it works for us:

• Evals decide when a result needs a human, and that share drops as the system matures.
• We also sample 1–5% of results for human review, depending on volume and maturity.
• We plant fakes in roughly 5–10% of what reviewers see. We’re still testing what rate works best.
• Our clients see results too, and they tell us when we get something wrong.

It has improved the accuracy of human review, especially over time. One lesson: we had to keep improving the fakes. People are very good at pattern recognition, and once the planted errors start to look alike, reviewers learn to spot the fakes instead of the errors.

The answer to the Goldilocks problem isn’t one setting. It’s deciding what the agent can touch, deciding where people check, and then checking the checkers.

Reading list

• “Goldilocks” Agents and the Power of Custom Cognitive Architectures (Sequoia, 2024) — Where the term comes from. Their question was how much control flow to hand the model; today’s is how much action.
• Autonomy and Agency in Agentic AI: Architectural Tactics for Regulated Contexts (Safin and Balta, fortiss, 2026) — The clearest map of the design space, with six concrete tactics. Read the document-routing example, where the checkpoint was retired once the agent proved out.
• The Oversight Fallacy (Data & Society, 2026) — Why an approve button isn’t oversight, with practical recommendations for both the teams building agents and the organizations deploying them.
• Closing the AI agent trust gap with graduated autonomy (AWS, 2026) — A concrete earned-autonomy design, including test cases planted in live traffic. It’s AWS-built, but the tiering pattern is portable.
• The Autonomy Boundary (Zachary Proser, 2026) — A simple rule for where an agent stops and asks, enforced by withholding credentials rather than by instructions.
• Perfecting Human-AI Interaction at Clinical Scale (Hippocratic AI, 2026) — How a patient-facing healthcare agent treats escalation and clarification as safety signals. It’s vendor research, but at real scale.

Sources cited

• Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach? (Ting Yan, August 2026) — The overreach study.
• Reliability and validity of threat image projection data as a measure of performance in X-ray baggage screening (Buser, Schwaninger, Rehor, and Sterchi, FHNW, 2025) — Airport screening.
• Do as AI say: susceptibility in deployment of clinical decision-aids (Gaube et al., npj Digital Medicine, 2021) — Physicians and wrong AI advice.

Often agentic loops are especially applicable in diverse domains. In healthcare, a patient chart review requires querying multiple systems, cross-referencing drug interactions, and reasoning across clinical history. In legal work, a contract review needs case law search, clause comparison, and conflict detection — not to mention that you may want to bring in significant client context. The loop exists because these problems are hard to solve with workflow-based designs.

Independent of workflow vs. agentic approach, scenario-based evals are the right approach for domain-specific applications. Each scenario is a specific input — a patient chart, a contract, a claims record — paired with an expert-defined expected outcome. Did the agent identify the medication interaction? Did it flag the indemnification clause? Did it reach the correct coverage classification? Did it ask the right questions and not ask ones it already had access to through context?

Why standard evals break for loops

Workflow-based implementations could test each step in isolation: given this input, does it produce the correct output to pass forward? In agentic loops, that no longer holds. The agent has autonomy over sequencing, and the order of operations may vary across sessions. Tool calls may fail or return unexpected results that the agent reasons past without flagging. An output-only eval catches none of it.

The non-determinism problem in domain evals

Here is where it gets genuinely hard. Evals are specific to a given scenario, but the agent’s path through that scenario is not fixed. Different sessions may pull different chunks from the retrieval system. The conversation may branch differently. The agent may reach the right conclusion through a different sequence of tool calls.

The way to handle this is to separate what you are actually testing. For the output layer, test the conclusion: did the agent identify the issue that matters? For the trajectory layer, test the key decision points: did it retrieve the document that contains the interaction? Did it call the drug lookup tool? Did it pass the correct patient ID? Trajectory evals do not require the agent to take an identical path every time. They require that the path passed through the checkpoints that justify the conclusion.

A case where the agent arrived at the right answer but skipped a critical tool call should not pass. That is not a correct evaluation. It is a lucky guess that will fail on the next prompt change.

How to start building this

You do not need a complete scenario library to get started.

• Start with production failures. Any trace where the agent reached a wrong conclusion, missed an issue, or called the wrong tool is a scenario waiting to be labeled. Get a domain expert to define correct behavior for that case and add it to the set. Five real failures are worth more than fifty synthetic happy-path cases.
• Write trajectory checkpoints, not just output assertions. For each scenario, identify the one or two tool calls or retrieval steps that are load-bearing. Test that those happened correctly and with the right parameters.
• Make “correct” explicit. Not a matching string, but a rubric: what must the output include, what must it not include, what tool calls must appear in the trace.
• Run evals on every significant change. Every prompt update, model upgrade, or tool integration should trigger a full scenario run. If scores drop, you do not ship.

The judge is itself a component to evaluate

Using LLM-as-judge to score outputs at scale is essential, but treat the judge as a component in your eval system, not a fixed oracle. Like every other component, it can drift. After a model update, your judge’s behavior may shift in ways you have not noticed. Build a calibration set for the judge: a sample of cases with human expert labels. Run the judge against them periodically and watch for divergence. When judge scores and expert scores disagree consistently, that is a signal. Either the judge has drifted, or your calibration set no longer reflects the scenarios the agent actually handles.

Treating the judge as infallible leads to calibration debt that accumulates invisibly and surfaces as production failures nobody predicted.

Reading list

• AI Agent Evaluations: A Developer’s Practical Guide (MLflow, 2025/2026) — Good on what specifically changes when you move from LLM to agent evals. The 75% judge-human agreement threshold as a recalibration trigger is a practical heuristic worth adopting. Apache-licensed tooling, no paywall.
• Evaluations for the Agentic World (QuantumBlack, January 2026) — Two detailed client case studies in regulated industries, a European telco and a European bank, covering trajectory validation, SME review as an eval layer, and continuous quality monitoring. The most concrete multi-agent eval case studies available in a public piece.
• A Pragmatic Guide to LLM Evals for Devs (Pragmatic Engineer, 2025) — The best starting point for teams new to systematic evals. Covers why LLM non-determinism breaks traditional testing and how to build a minimal eval suite that actually catches regressions. The golden dataset construction section is directly applicable to domain-specific scenario libraries.
• A Methodical Approach to Agent Evaluation (Google Cloud, November 2025) — The cleanest structural framework I have found: three pillars covering final output quality, trajectory analysis, and trust/safety. Worth reading specifically for the section on building a golden dataset when you do not yet have one. The Vertex AI mention at the end is a brief plug; the framework itself is tool-agnostic.
• Awesome-Evals (BenchFlow, GitHub) — A curated library of papers, blog posts, tools, and benchmarks organized by topic. Useful if you want to go deeper on any specific area: tool-calling evals, LLM-as-judge calibration, multi-agent evaluation. The “non-BS” framing in the description is accurate.