Your Next Bottleneck Is Code Review

November 10, 2026

Tony Karrer

AI is helping engineers write more code. But someone (or something) still has to review it.

We’re seeing reports of engineering teams merging up to four times as many PRs as they did two years ago, and those PRs are getting bigger. LinearB’s analysis of 8.1 million PRs found AI-assisted PRs are roughly 2.6 times larger than unassisted ones at the 75th percentile. Meanwhile, median review time has climbed from 33 minutes to 77 minutes. More code is getting written, but more of it is waiting for approval.

Adding another coding agent won’t fix that.

There are several ways to reduce the human effort in review without giving up much quality: keep PRs small, review the spec before the agent writes the code, let automated checks handle the mechanical work, ask for proof that a change works, and point reviewers at the parts of a change that need their attention.

This post focuses on a lever that comes before all of those: deciding which changes need human review in the first place. Most teams still treat a typo fix, a dependency update, and a change to authentication as variations of the same review process. They aren’t. The risks are different, and the review requirements should be too.

Give every PR the review it deserves

We need to start treating code review as a risk management problem, rather than a universal requirement that every change passes through in the same way.

Consider a small UI fix or a documentation change in a well-understood part of your application. If your automated tests pass, an AI reviewer doesn’t identify any issues, and the change doesn’t touch anything sensitive, do you really need an engineer to stop what they’re doing and approve it?

Probably not.

Now consider a change to billing logic, authentication, or a database migration. Even if it’s only five lines of code, I’d want someone who understands that system to look at it carefully. I’d also want to know that we have appropriate tests and a reasonable plan to roll it back if something goes wrong.

There’s a lot of work in between those two extremes. Everyday application changes where an engineer should still review the intent, behavior, and tests, but probably doesn’t need to scrutinize every line of code that an agent generated.

Traffic light risk assessments are essentially the approach I’d recommend: establish a fast track for low-risk changes, keep a normal review process for changes requiring engineering judgment, and apply more rigorous reviews to sensitive parts of the system.

Of course, the important part is defining what qualifies for each category. You don’t want an AI reviewer making that decision entirely on its own. Set hard boundaries around protected files, change size, test results, and other risk factors. The AI can identify additional concerns, but it shouldn’t be allowed to override those boundaries.

This isn’t theoretical

A few engineering teams are already putting versions of this into practice.

Will Larson recently wrote about how Imprint introduced a small-change policy that allows certain PRs to bypass traditional human review. They established a 1,000-line ceiling, restrictions on sensitive code, required clean AI review results, and made sure the author had experience with the affected service. About 10% of PRs initially qualified.

PostHog has taken a similar approach with its automated reviewer, StampHog. Their policy limits eligible changes to 500 lines across 20 files, excludes protected areas, and requires the AI reviewer to find no serious problems. In their published account, the agent provided the final approval on roughly a third of merged PRs.

Rewind has reported around 40% auto-approval in its early results. One detail I particularly like about their implementation is that they use a policy engine to determine which parts of the codebase require human review. The model can raise concerns, but it can’t decide to relax those requirements.

These numbers aren’t directly comparable, and I wouldn’t suggest that everyone should be targeting 30% or 40% automated approvals. Different applications have different risk profiles.

But there is a common theme worth paying attention to: deterministic rules establish what’s safe to automate, AI operates within those rules, and engineers focus on the changes that actually need their expertise.

Approval and shipping are different decisions

This is especially important for AI-generated code. A bot approving a PR doesn’t mean a bot should merge it. Rewind separates those permissions and still requires a human to merge agent-authored PRs.

Your policy should explicitly define who can author, approve, merge, and release each type of change. Without that separation, “AI-approved” can become an excuse for removing human accountability.

Start small. Measure both speed and safety.

Pick one well-understood repository and define the protected areas. Set the fast-track criteria. Then run the policy in shadow mode, where automation recommends decisions but humans still approve.

Before expanding, look at four things:

  • Review latency: Are PRs spending less time waiting?
  • Eligibility: How many changes actually meet the fast-track rules?
  • Safety: Are reverts, escaped defects, or incidents increasing?
  • Accuracy: Where did the automated reviewer disagree with humans, and who was right?

Give someone ownership of the policy and the authority to shut off automated approvals.

Review isn’t only about catching bugs. It’s also how engineers learn a codebase and catch architectural drift. If you remove routine reviews, make room for that learning elsewhere, through periodic reviews of related changes, for example.

The goal isn’t necessarily to maximize the percentage of PRs approved by AI. It’s to make better use of engineering judgment, letting low-risk changes move faster while helping engineers focus on the specific parts of a change that need their attention.

Reading list

  • Speeding up code review with small-change (Will Larson, October 2026) — The strongest practitioner anchor: Imprint’s eligibility rules, enforcement through CI, early 10% qualification rate, and the distinction between reviewing individual diffs and preserving organizational learning.
  • Review is the bottleneck now: How we let AI approve pull requests (safely) (Dave North, Rewind, June 2026) — The most detailed implementation account: specialist reviewers, deterministic policy enforcement, protected paths, shadow-mode calibration, audit evidence, and early results, with useful caveats about sample size and PR mix.
  • Stop being the code review bottleneck (Jina Yoon, PostHog, July 2026) — A concrete example of bounded auto-approval: StampHog’s size limits, deny list, escalation behavior, and fail-closed rule that the LLM cannot relax deterministic safeguards.
  • Everything We Talked About at O11yDay London 2026 (Ken Rimple, Honeycomb, October 2026) — Read Rebecca Murphey’s section, “Your AI Problem Is Your 2016 Problems Dressed in a Trenchcoat.” It provides Honeycomb’s 25% machine-only review goal and connects the review queue to flow efficiency and the theory of constraints.
  • The code review is dead; long live the code review (Cecilia Geraldo, Thoughtworks, June 2026) — A useful counterweight to the automation stories: the mechanics of line-by-line review may change, but shared understanding, mentorship, and technical rigor still need an explicit home.

The Goldilocks Problem, Two Years Later

Give an agent too little room and it interrupts so often that nothing gets done. Give it too much and you’re writing an apology to a customer.

In 2024, Sequoia called the middle ground “Goldilocks” agents. Back then the question was how much control flow to hand the model, somewhere between a hard-coded chain and an AutoGPT that wandered off. Two years later, agents send messages, update records, and move money. The Goldilocks question has moved from control flow to action.

Today’s models can do the work. Capability is no longer the limiting factor. It’s a question of how people, agents, and the systems around them work together. I find it helps to break it into three questions.

That’s what we’ll dig into at our next LA CTO Forum session.
Join Tony Karrer and four speakers who have shipped agents in production for Lessons Learned Building Agentic Systems, a free online session on Friday, November 13, 2026 (8–10 AM PT / 11 AM–1 PM ET). They’ll cover guardrails, context, what breaks, and what to do about it.
Can’t make it live? Register anyway and we’ll send you the recording.
Reserve your spot →

How much is the agent allowed to do?

Not every action carries the same risk. A recent paper on agents in regulated settings, from researchers at fortiss, a Munich-based research institute for software-intensive systems, lays out a ladder: the agent reasons over what it’s given, uses safe local tools, reads from external systems, writes things that are easy to undo (a draft, a pending record), and finally commits things that aren’t (a filing, a payment, an update to the system of record). As the authors put it, whether a human has to approve an action is a design choice, but whether the action can be undone is a property of the action itself.

A simpler test: is it reversible, and does it leave the building? Zachary Proser, who builds agent harnesses at WorkOS, lets his agents act alone on anything reversible and inbound, and stops for a human on anything irreversible or outbound.

The practical move is to shrink what the agent can touch before you add a human:

  • Fence the tool. Limit which records or fields it can reach, enforced in the tool itself, so an out-of-bounds call fails no matter what the model was told.
  • Stage the write. The agent drafts; something else commits.
  • Withhold the credential until it’s earned. An agent that doesn’t hold the “send” permission can’t send.

None of these depend on the model following instructions.

How often does a human check?

There are two basic patterns. A checkpoint is a review at a fixed step, every time. Escalation is the agent pausing when it’s unsure: low confidence, conflicting inputs, something out of scope. Checkpoints are predictable and easy to audit. Escalation scales better, but only if it’s specific. Vague or frequent escalations wear people out and erode trust.

The fortiss paper describes a document-routing agent that started with an expert reviewing every item. As accuracy improved, the team dropped the checkpoint and routed only low-confidence items to a person.

That’s the pattern to aim for: earn autonomy one action type at a time, against criteria you set in advance. An AWS reference design for graduated autonomy sums it up as “promote slowly, demote immediately.” Treat a model upgrade as a reason to look again.

Do humans help at all?

Here’s the uncomfortable part. In a recent study, people supervised an AI assistant through a simulated day that included out-of-scope actions like permanently deleting an archived timesheet and buying $12 of travel insurance nobody asked for. With approval required on every action, about 40% of those actions still got through. When a model flagged them first, people approved them anyway. When people wrote their own rules up front, they set most of them to “ask”, then said yes to almost every request that came back, even when the agent was overreaching.

It’s one small study, but it matches what anyone who has watched an approval queue already knows. When nearly everything is fine, people stop looking. A Data & Society report, The Oversight Fallacy, puts it well: “A loop can exist formally while failing substantively.”

Four ways to make the check work:

  • Check fewer, bigger things. Approve the staged draft, the batch, or the budget, not every call.
  • Show the work, not just the button. Put the evidence and the steps the agent took next to the decision, at the level of detail that reviewer needs.
  • Give reviewers authority, and measure them. Who can halt, revoke, and rerun? What’s the approval rate? When did someone last say no?
  • Plant things that should be caught.

What we’re doing

That last one is the practice we lean on most in our healthcare and legal work. We intentionally plant mis-identified results in what human reviewers see, then track whether they catch them. The idea isn’t new: airport baggage screening has projected fake threats into X-ray images for decades to keep screeners alert, and medical researchers have used deliberately wrong AI advice to test whether physicians catch it.

Roughly how it works for us:

  • Evals decide when a result needs a human, and that share drops as the system matures.
  • We also sample 1–5% of results for human review, depending on volume and maturity.
  • We plant fakes in roughly 5–10% of what reviewers see. We’re still testing what rate works best.
  • Our clients see results too, and they tell us when we get something wrong.

It has improved the accuracy of human review, especially over time. One lesson: we had to keep improving the fakes. People are very good at pattern recognition, and once the planted errors start to look alike, reviewers learn to spot the fakes instead of the errors.

The answer to the Goldilocks problem isn’t one setting. It’s deciding what the agent can touch, deciding where people check, and then checking the checkers.

Reading list

Sources cited

Hiring in the Age of Agentic AI

September 11, 2026

Tony Karrer

One thing we’ve been doing a lot lately is hiring senior engineers, both internally and for our clients. We’ve seen challenges with resumes that are finely tuned by AI based on the job description, candidates who are clearly reading from AI, initial screening by non-technical interviewers not providing much signal, and an even harder problem:

How do we change our interviews now that a key part of the job is working with coding agents? A senior engineer is now expected to manage coding agents to produce shippable products, not to write code. But many interviews still test closed-book problem-solving, the one activity the engineer won’t be doing!

The fix sounds simple: rebuild the interview around the actual work. Instead of asking a candidate to implement a quicksort, give them an ambiguous ticket, a real codebase, and the tools to solve it. Do that, and using AI stops being “cheating.” Using AI becomes the test. It’s easier said than done, but this is the direction we’ve been pushing our own interviews over the past six months.

Rebuilding technical interviews for the agentic era is one of the hardest problems on our plate right now, and we’re going deep on it live.
Join Tony Karrer and a panel of engineering leaders for Hiring Engineers in the Agentic AI Era, a free online mini-conference on Friday, October 9, 2026 (8–10 AM PT / 11 AM–1 PM ET). Can’t make it live? Register anyway and we’ll send you the recording.
Reserve your spot →

We’d like to hear what you’re seeing: Have you changed technical screens? What signal do you still trust and what’s changed? What surprised you when you tried AI-allowed rounds? Reply to this email or grab time on my Calendly if you want to compare notes.

Basically, it’s a measurement problem

AI didn’t break technical interviewing. It exposed how long we’d been measuring a proxy for the job instead of the job itself. Karat’s survey of engineering leaders found that 71% say AI makes it harder to assess technical skills, and their CEO argues the root cause isn’t cheating so much as interviews that haven’t kept up with the job. Candidates can produce correct LeetCode answers or polished take-homes, but struggle to debug a multi-file change or explain why they rejected a plausible diff.

What doesn’t work anymore: puzzles without context

Take-home assignments and automated code tests were never strong signals for senior hires. Nowadays they’re close to useless when everyday tools can one-shot them. Interviewing.io let candidates use ChatGPT on modified LeetCode problems. Pass rates jumped sharply and interviewers rarely noticed. Pragmatic Engineer’s reporting on hiring managers found startups dropping take-homes and algorithmic screens for that reason, while Big Tech interviewers report asking harder variants of the same questions instead.

Making the questions harder buys time, not relevance. One hiring manager described re-running their interview questions through the newest model every few months. Eventually it one-shotted their second-round question, and there was no harder one left to reach for. Even with one in hand, a puzzle the model can solve tells you little about how a candidate exercises judgment under real constraints. Templated system design interviews have the same flaw. Candidates learn a script that fits a sixty-minute slot, and the script rarely resembles how your team actually decides tradeoffs.

What survives: judgment in realistic conditions

The interviews that still produce signal look more like the job. Startups and some larger product companies are adding rounds where candidates use AI on a realistic task while an engineer watches how they work. Shopify’s engineering leadership has been public about embracing that model: candidates who don’t use an agent often lose to those who use one well, and the interesting part is what happens next. Do they spot garbage output? Fix an obvious one-line bug by hand or keep prompting? Explain tradeoffs when the model suggests the wrong abstraction?

Strong candidates treat generated code as a draft, validate edge cases, and stay coherent when follow-ups go off script. Weak candidates paste, ramble, or collapse when asked to defend a design choice without a fresh prompt. The shift is to make these skills the whole interview, not just a bonus round after a puzzle.

Two industries pretending to be one

Big Tech and many enterprises are largely fortifying the old process: screen sharing, cheat suspicion flags, harder algorithmic questions, and talk of returning to in-person screens. Startups are redesigning it: dropping take-homes, allowing Cursor or Copilot, or running paid trial days where AI use is expected. Both are rational responses to the same shock, but they optimize for different risks. Incumbents fear false positives from hidden assistance. Fast-moving product companies fear false negatives from banning the environment their engineers already use.

Leaders should name which failure mode they are actually solving. Prohibiting AI without redesigning questions often pushes assistance underground and rewards performers who are mostly good at looking honest on camera. Allowing AI without new rubrics and interviewer training just makes the old false positives easier to generate. Pedigree screening rises when resume signal collapses, which is understandable and still a bad long-term strategy for teams that need builders who can navigate messy codebases, not just familiar logos on a CV.

Recruiter and engineering misalignment makes the signal problem worse before a candidate ever reaches a technical round. When screening criteria are vague, recruiters optimize volume while engineers want depth. Promising profiles never reach the right conversation, or weak ones burn panel time on puzzles that AI already solves. The corrective move is shared ownership of the bar: senior ICs define what “honest talent worth a coding assessment” looks like in practice, recruiters screen to that definition, and both sides treat the early filter as a trust problem as much as a skills problem.

What we’re currently doing

Our own hiring practice has changed multiple times over the past year, and we expect it to change again. (What follows is the coding track, which every engineering candidate goes through regardless of level. Candidates for higher-level roles also get a separate architecture and system design interview.) Our first step is a background interview: a deep dive into work the candidate has done that has real technical complexity, plus a frank discussion of the kinds of challenges that come with the target environment.

Candidates who clear this first step go on to three exercises that mirror how the work actually looks.

  • Code analysis. We ask candidates to reason about somewhat obfuscated or obtuse legacy code and explain what it does. That rewards reading, tracing, and judgment, all critical skills in reviewing code that AI is creating.
  • A function body against a flimsy specification. It is a simple problem the candidate will encounter often, but the spec is incomplete on purpose. We watch whether they ask the interviewer the right clarifying questions, and then we watch them build the simple thing once the ask is clear. Communication under incomplete information is the skill. Clever algorithm recognition is the LLM’s job.
  • An AI-practices walkthrough, framed as day one on a project. The candidate has just cloned a codebase that has never been touched by an AI agent, and their first job is to explore it and get it ready for agentic development. We leave “ready” undefined on purpose. How they approach this task tells us how they think about setting an agent up to succeed. Next we hand them a ticket and ask how they would implement it with AI. Finally we show them the agent’s output and ask how they’d review it.

On that third exercise, we care how they work throughout: Do they understand AI basics and deeper SDLC habits? Do they explore the repo for docs and artifacts? Ask about existing tests, practices, and tech before prompting for code? How do they prompt to implement the ticket, and do they explore the generated code in depth with a critical eye? That last question is where judgment shows up, because the generated code looks plausible and the test is whether they can tell what’s wrong with it.

None of this is a finished template. It is our current bet on measuring the job we actually hire for: reading code, clarifying weak requirements, turning agentic assistance into trustworthy outcomes, and codifying what the agent gets wrong into rules so it doesn’t happen again.

AI is not ending technical interviewing. It is forcing honesty about what we have been optimizing for. The processes that survive will measure how engineers operate with assistance, incomplete information, and accountability for outcomes. That is closer to a PR review than a whiteboard puzzle, and for most product engineering teams, that is the job.

Reading list

  • Engineering Interviews in 2026: 3 Trends Hiring Leaders Must Prepare For (Karat, January 2026) — The source of the 71% figure, drawn from 400 engineering leaders across the US, India, and China. The more alarming number is buried further down: confidence that the right candidates are getting offers fell from 68% to 47% in a single year.
  • Three Ways AI is Reshaping Traditional Technical Interviews in 2026 (IEEE-USA InSight, 2026) — Where Karat CEO Mo Bhende makes the relevance argument directly: the fundamental job changed, technical interviews did not. Short, and the cleanest statement of the mismeasurement thesis we have found.
  • How hard is it to cheat with ChatGPT in technical interviews? We ran an experiment. (interviewing.io) — Thirty-seven interviews where candidates were told to use ChatGPT and interviewers were told nothing. Pass rates hit 73% on verbatim LeetCode and 67% on modified versions, dropping to 25% on fully custom questions. Not one interviewer suspected anything.
  • Tech hiring: is this an inflection point? (The Pragmatic Engineer) — Interviews with engineering managers at startups and mid-sized companies, most of whom say hiring is harder now than it has ever been. Useful for the practitioner quotes on why coding problems and take-homes both stopped working at the same time.
  • How AI is changing software engineering at Shopify (The Pragmatic Engineer, with CTO Farhan Thawar) — The source for the Shopify approach described above. Thawar shadows interviews himself and uses generated code as a probe, asking the candidate whether it is any good. Worth reading for how differently that round is scored.

Evals for Agentic Loop Applications

July 14, 2026

Tony Karrer

Agentic loops are not the same as the workflow-based LLM applications most of us started with. An agent takes multiple steps, calls external tools, reasons across retrieved context, and arrives at a conclusion through a path that is itself part of what needs evaluating. An eval suite that only checks the final output misses aspects that likely are important for overall quality of your agentic loop.

Often agentic loops are especially applicable in diverse domains. In healthcare, a patient chart review requires querying multiple systems, cross-referencing drug interactions, and reasoning across clinical history. In legal work, a contract review needs case law search, clause comparison, and conflict detection — not to mention that you may want to bring in significant client context. The loop exists because these problems are hard to solve with workflow-based designs.

Independent of workflow vs. agentic approach, scenario-based evals are the right approach for domain-specific applications. Each scenario is a specific input — a patient chart, a contract, a claims record — paired with an expert-defined expected outcome. Did the agent identify the medication interaction? Did it flag the indemnification clause? Did it reach the correct coverage classification? Did it ask the right questions and not ask ones it already had access to through context?

Evals are one of the hard lessons we’ve learned shipping agentic loops in production — and we’re going deep on it live.
Join four engineering leaders from Bridgewater Associates, Prudential Financial, and Morgan & Morgan for Real-World Lessons from Shipping LLM-Based Software, a free online mini-conference on Friday, August 14, 2026 (8–10 AM PT / 11 AM–1 PM ET). Can’t make it live? Register anyway and we’ll send you the recording.
Reserve your spot →

Why standard evals break for loops

Workflow-based implementations could test each step in isolation: given this input, does it produce the correct output to pass forward? In agentic loops, that no longer holds. The agent has autonomy over sequencing, and the order of operations may vary across sessions. Tool calls may fail or return unexpected results that the agent reasons past without flagging. An output-only eval catches none of it.

The non-determinism problem in domain evals

Here is where it gets genuinely hard. Evals are specific to a given scenario, but the agent’s path through that scenario is not fixed. Different sessions may pull different chunks from the retrieval system. The conversation may branch differently. The agent may reach the right conclusion through a different sequence of tool calls.

The way to handle this is to separate what you are actually testing. For the output layer, test the conclusion: did the agent identify the issue that matters? For the trajectory layer, test the key decision points: did it retrieve the document that contains the interaction? Did it call the drug lookup tool? Did it pass the correct patient ID? Trajectory evals do not require the agent to take an identical path every time. They require that the path passed through the checkpoints that justify the conclusion.

A case where the agent arrived at the right answer but skipped a critical tool call should not pass. That is not a correct evaluation. It is a lucky guess that will fail on the next prompt change.

How to start building this

You do not need a complete scenario library to get started.

  • Start with production failures. Any trace where the agent reached a wrong conclusion, missed an issue, or called the wrong tool is a scenario waiting to be labeled. Get a domain expert to define correct behavior for that case and add it to the set. Five real failures are worth more than fifty synthetic happy-path cases.
  • Write trajectory checkpoints, not just output assertions. For each scenario, identify the one or two tool calls or retrieval steps that are load-bearing. Test that those happened correctly and with the right parameters.
  • Make “correct” explicit. Not a matching string, but a rubric: what must the output include, what must it not include, what tool calls must appear in the trace.
  • Run evals on every significant change. Every prompt update, model upgrade, or tool integration should trigger a full scenario run. If scores drop, you do not ship.

The judge is itself a component to evaluate

Using LLM-as-judge to score outputs at scale is essential, but treat the judge as a component in your eval system, not a fixed oracle. Like every other component, it can drift. After a model update, your judge’s behavior may shift in ways you have not noticed. Build a calibration set for the judge: a sample of cases with human expert labels. Run the judge against them periodically and watch for divergence. When judge scores and expert scores disagree consistently, that is a signal. Either the judge has drifted, or your calibration set no longer reflects the scenarios the agent actually handles.

Treating the judge as infallible leads to calibration debt that accumulates invisibly and surfaces as production failures nobody predicted.

Reading list

  • AI Agent Evaluations: A Developer’s Practical Guide (MLflow, 2025/2026) — Good on what specifically changes when you move from LLM to agent evals. The 75% judge-human agreement threshold as a recalibration trigger is a practical heuristic worth adopting. Apache-licensed tooling, no paywall.
  • Evaluations for the Agentic World (QuantumBlack, January 2026) — Two detailed client case studies in regulated industries, a European telco and a European bank, covering trajectory validation, SME review as an eval layer, and continuous quality monitoring. The most concrete multi-agent eval case studies available in a public piece.
  • A Pragmatic Guide to LLM Evals for Devs (Pragmatic Engineer, 2025) — The best starting point for teams new to systematic evals. Covers why LLM non-determinism breaks traditional testing and how to build a minimal eval suite that actually catches regressions. The golden dataset construction section is directly applicable to domain-specific scenario libraries.
  • A Methodical Approach to Agent Evaluation (Google Cloud, November 2025) — The cleanest structural framework I have found: three pillars covering final output quality, trajectory analysis, and trust/safety. Worth reading specifically for the section on building a golden dataset when you do not yet have one. The Vertex AI mention at the end is a brief plug; the framework itself is tool-agnostic.
  • Awesome-Evals (BenchFlow, GitHub) — A curated library of papers, blog posts, tools, and benchmarks organized by topic. Useful if you want to go deeper on any specific area: tool-calling evals, LLM-as-judge calibration, multi-agent evaluation. The “non-BS” framing in the description is accurate.
As teams move to agentic dev, one question we’re hearing a lot is “Where does system/organizational truth live?” One interesting solution to this problem is to use the same git repository that engineers and agents already rely on.

Wikis and similar systems — Confluence, Notion, and the rest — are orthogonal to the system itself. Pages fall behind quickly, because updating them is secondary labor. In a traditional SDLC, this isn’t the end of the world. Humans can just ignore out-of-date documentation, and rely on the bits they know to be reliable, filling in the gaps by asking their manager or emailing the product team. An agent won’t do that! A copy and paste from outdated docs will be treated as 100% true by an agent.

Ultimately, this is less a model problem than a documentation and memory problem.

The repository as the system of record

One solution to this problem is to make the repository the system of record. Requirements and rationales live alongside the modules they describe, and git history and PR comments explain what changed along the way. Jira and Confluence shift from being the source of truth to useful reports generated from what lives in the repo.

Several forces stack in favor of this locality, especially with agents in the loop. The simplest is colocation: docs that live next to src/billing/ change when billing changes. Local docs aren’t orthogonal to the work the way wiki articles are. And because they ride the same branches, the honesty machinery you already run (pull requests, human and agent review, CI checks, linters) can govern documentation too.

Locality also changes what a diagram is worth. A box-and-arrow PNG in Confluence is opaque to an agent, but a Mermaid flowchart beside the service entrypoint shows up in the diff and can be read by the tools. Underneath all of this is a question of what the artifact is optimized for. When agents and engineers reason about a change, the question is usually what a module does and what must not break, not where the payments wiki lives. A repository is organized around that question; a topical wiki fights it.

Karpathy’s markdown library: compile, don’t chunk

We’re not the only ones thinking this way. Andrej Karpathy recently described an approach to building an LLM knowledge base that bypasses classic RAG for many real workloads. You dump raw material into a folder, let an agent compile it into interlinked Markdown, and run periodic lint passes so the library stays consistent. At team scale he argued structured text plus indices beats opaque embedding search that returns plausible fragments with weak provenance.

That pattern rhymes with system documentation too. You already have the raw material of a knowledge base scattered everywhere: git history, tickets, Slack threads, and code. You’re just not compiling it into agent-readable truth that lives beside the work. Karpathy’s point is that the model can be your librarian. Once the pattern is established, summaries, cross-links, and cleanup become ongoing maintenance, done entirely agentically, rather than a one-time migration project. This doesn’t mean vector databases are dead; it means the interesting bet for many product engineering teams is versioned Markdown in git, maintained with the same habits as code.

What you should do Monday

We are intentionally not prescribing a full-on “move your tickets into git” or “replace Jira with YAML tasks” project. Ticket workflow is a separate fight, and many teams are not ready for it. This thesis calls for different experiments around documentation and shared memory.

Try this: pick one bounded subsystem and require that its functional and architectural documentation live in Markdown/Mermaid stored alongside the code. Treat wiki pages as deprecated for that slice. Then add a lightweight gate that makes drift visible: a CI check that fails when certain paths change but their docs don’t, or an agent lint step that compares README claims to test names and public API surfaces.

Run a Karpathy-style compile pass to flesh out this new library: export related decisions from Slack into a raw/ folder, then run a bounded agent job to produce interlinked docs/ with backlinks. Review the agent’s work and tweak based on what it missed or under/overemphasized. Measure whether onboarding and agent sessions get faster, more accurate, or just plain easier.

Whatever experiment you run, teams will have to adjust. This change lands differently on each side of the house. Product leaders will need to build fluency with repo-resident artifacts (reading a spec in git, commenting on a PR). For engineering leaders, the change means owning freshness: if agents read the repo, stale local docs are now a production risk, not a technical-writing nuisance.

None of this requires replacing the wiki overnight. Plenty of orgs will keep it as the official face for auditors and executives, and that’s not wrong. Different consumers need different views. The bet worth running in 2026 is smaller: for the teams shipping with agents, stop asking them to treat a parallel documentation universe as true. Put truth beside the implementation, let git govern change, and let agents help maintain the library the same way we finally learned to let CI maintain tests.

Further Reading

Agentic Coding in Practice

May 15, 2026

Tony Karrer

Join us on June 12: Agentic Coding in Practice

Presenters will demo how their teams are actually wiring up agents, skills, rules, hooks, and review loops to make AI coding tools work inside real engineering processes, from spec to PR to QA. This session is designed for senior engineering, product, and QA leaders who want practical, ready-to-apply examples, not theory. Register here

Almost every CTO or VPE we talk to is asking some version of the same question: “Are we picking the right tools, building out the right workflows, and putting our people in the right places?” In other words: how should engineering teams actually operate now, with agents writing most of the code?

Teams are landing in pretty different places. Some are pushing toward full autonomy with minimal human review. Others are keeping tight control, using AI more as a reviewer or assistant. Most are somewhere in the middle.

But which workflow you pick isn’t really the challenge. The inputs are.

The bottleneck has moved. Writing code is getting cheaper, but building reliable systems requires more than just code. Specs, tests, and review are now the limiting factors, and agents amplify whatever you feed them. Strong inputs get strong results. Weak ones fail faster, at scale, and with less visibility into why.

One theme for the second half of 2026: engineering leaders need much better visibility into what other teams are actually doing. Too much of this is still being figured out in private, and some of the public conversation is noise.

What’s actually working

From what we’re seeing across teams and practitioner write-ups, a few patterns are emerging. Not as a single “right way,” but as things that consistently hold up.

Front-load the thinking: spec, tests, then code

The teams getting reliable output are putting more effort into shaping the problem up front (acceptance criteria, edge cases, tests) before letting agents implement. The shift is subtle but important: less time writing code, more time defining what “correct” looks like.

Parallelize aggressively, but with boundaries

Running multiple agents in parallel is becoming common: one exploring, one implementing, one cleaning up. (Stripe built a harness that ships >1,000 agent-driven PRs per week. ) But the teams doing this well isolate each agent in its own worktree, container, or sandbox, so mistakes don’t cascade. Parallelism helps, but only if it’s contained.

Where this breaks

A few failure modes show up just as consistently.

Tests written after the fact

This is amplification working against you. When you ask an agent to write tests after the implementation, the existing code becomes the spec. Agents don’t push back. They complete the task you gave them, even if the task is wrong. The result: tests that lock in whatever’s broken. Discipline matters more with agents, not less: tests first, then implementation.

Context can be a liability

Many teams are leaning on rules files (CLAUDE. md, AGENTS. md) to guide behavior. But they go stale quickly. Codebases evolve, and these files rarely keep up. Without regular review, they can make things worse, with agents following outdated or rigid instructions. Structure is necessary for good results, but bad structure gets amplified.

What actually changes for teams

Coding agents don’t remove the need for senior engineering judgment. They concentrate it. The work shifts toward defining problems clearly, validating outputs, and keeping the whole system reliable. Teams with weak specs, inconsistent tests, or overloaded review processes feel more pain, not less. Teams with strong fundamentals move faster.

The work moves up the stack, and the pressure moves with it. And the systems you build around the agents matter more than the agents themselves.

Reading list

  • Embracing the parallel coding agent lifestyle – by Simon Willison. Concrete patterns for running several coding agents at once. Covers worktrees and Docker for isolation, what kinds of work to delegate to parallel sessions, and what to supervise more tightly.
  • How Stripe built “minions” – by Steve Kaliski via Lenny’s Newsletter. Inside Stripe’s production agent harness shipping ~1,300 PRs/week from Slack reactions. Covers the harness layers, how they handle code review at that scale, and what they had to build vs. adopt off-the-shelf.
  • Why Testing After with AI Is Even Worse – by Matti Bar-Zeev. Why asking an agent to write tests after the implementation produces tests that validate existing bugs instead of catching them. Makes the case for TDD with agents, not against it.
  • How System Prompts Define Agent Behavior – by Srihari Sriraman and Drew Breunig (nilenso). A close read of system prompts across Claude Code, Cursor, Codex, Gemini, and others. Shows the same model producing dramatically different workflows depending on the prompt wrapped around it.
  • Your CLAUDE. md Is Making Your Agent Dumber – by Cordero Core. Recent research finding that LLM-generated CLAUDE. md / AGENTS. md files actively decrease agent success rates compared to having no context file at all. Practical guidance on what to do about it.
  • Ralph Wiggum as a “Software Engineer” – by Geoffrey Huntley. Walks through “Ralph,” a bash-loop technique for autonomous coding agents. Concrete on what kinds of projects it suits and where senior engineering judgment stays non-negotiable.
 
You may have heard some version of this question already: “If AI can generate code and tests so quickly, do we still need QA?”

Yes we still need QA, but what “QA” means is rapidly changing.

Teams using agentic coding are shifting quality both left and right. Left into work definition, acceptance criteria, and test architecture. Right into integration confidence, production signals, and release confidence.The shift matters because no downstream checkpoint can absorb the volume agentic engineering can produce.

QA has to stretch in both directions – into the decisions that shape the work, and into the signals that show it will hold up in production.

Where this is heading

The shape of QA is still in flux.

Some teams are pushing quality further left, with more rigor in specs, acceptance criteria, and engineering guardrails before agents ever start writing code. Fowler’s recent “harness” framing pushes even further in that direction: the goal is to create stronger guides, checks, and boundaries around agentic work so that quality is built into the system earlier, rather than rescued at the end.

At the same time, teams are also being pushed right. Faster code generation increases the need for integration confidence, release confidence, observability, and production feedback.

Both moves follow the same pattern: QA is no longer a stage in the pipeline. It’s a thread running through the whole SDLC.

Where AI helps

AI can be genuinely useful in QA work.

  • It can generate baseline unit tests quickly.
  • It can help fill gaps in integration coverage.
  • It can suggest edge cases.
  • It can help analyze failures faster.
  • It can reduce repetitive test-authoring work.

We worked with one client that had a 10+ year old Java application with almost no test coverage. Spending weeks writing tests by hand was not getting prioritized. AI generated a baseline suite in hours.

That was valuable. But “better than nothing” is not the same as “good enough to trust.” That is still the trap.

QA responsibilities still exist

Whether you call it QA or absorb the responsibility into other functions, the responsibilities still exist:

  • Helping define what “correct” means before work starts.
  • Shaping acceptance criteria, workflows, and edge cases.
  • Validating outcomes, not just checking whether scripts pass.
  • Improving release confidence across systems, not just within a single change.
  • Watching production signals and feedback after release.
  • Helping teams identify where AI-generated coverage looks good but misses the real risk.

QA’s future will vary by organization: some will keep a distinct function, some will shift testing into engineering, and others will refocus specialists on end-to-end validation, production risk, and release confidence.

What does not hold up is the idea that QA can remain a cleanup step at the end.

A more practical QA playbook

A practical approach for 2026 looks something like this:

  • Use AI aggressively to generate baseline unit and integration test code, followed by careful human review to finalize.
  • Involve QA earlier in shaping acceptance criteria and edge cases.
  • Put more energy into integration confidence and release confidence.
  • Spend less time maintaining brittle scripts and more time validating outcomes.
  • Treat QA as part of quality system design, not just test execution.

That last point is one of the biggest changes.

The QA teams that create the most value in an agentic world are not just writing or maintaining scripts. They are helping design the system that makes rapid change safe.

Additional reading

Every once in a while, the TechEmpower Framework Benchmarks surface something that makes everyone involved squint at the numbers and say, “Wait… what?”

Round 23 had one of those.

On the face of it, nothing particularly dramatic had changed about the idea of the benchmarks. Same philosophy as always:

Choosing a web application framework involves evaluation of many factors. While comparatively easy to measure, performance is frequently given little consideration. We hope to help change that.

Application performance can be directly mapped to hosting dollars, and for companies both large and small, hosting costs can be a pain point. Weak performance can also cause premature and costly scale pain by requiring earlier optimization efforts and increased architectural complexity. Finally, slow applications yield poor user experience and may suffer penalties levied by search engines.

What if building an application on one framework meant that at the very best your hardware is suitable for one tenth as much load as it would be had you chosen a different framework?

That’s the north star for the project, and it hasn’t really changed.

What did change in R23 was that one set of tests, the spring-mongo implementations, looked much worse than anyone expected. Not “eh, that’s a little low” worse, but “this is probably pointing at a real problem” worse.

And because the internet is sometimes better than we deserve, the lead performance engineer at MongoDB noticed, reached out, and we ended up on a little adventure together.

MongoDB’s Java story in the benchmarks is represented by spring-mongo (a Spring-based implementation) as well as a few Node.js variants. In Round 22, the spring-mongo tests were not the fastest things on the planet, but they were in a reasonable ballpark.

By round 23, some of the spring-mongo results in our continuous benchmark environment (Citrine) had fallen off a cliff.

For one of the core tests (query) the numbers looked roughly like this (requests per second, spring-mongo only):

  • Round 22: ~5.9k
  • Round 23 baseline: ~583

That’s not noise. That’s a smoking gun.

At the same time, other frameworks and other MongoDB-based implementations weren’t seeing that same kind of collapse. That’s usually a hint that something environmental or configuration-related is going wrong, rather than “MongoDB suddenly became slow” or “Spring suddenly can’t talk to the database.”

This is where Ger Hartnett from MongoDB enters the story. Ger is the lead performance engineer at MongoDB, and he did exactly what I wish more vendors would do: he treated the weird benchmark as a bug report.

He emailed me and, very politely, started asking all the questions you’d expect a performance engineer to ask:

  • “What’s the host hardware?” We’re running on a single Xeon 6330 box with 56 hyper-threaded cores.
  • “What OD and kernel versions were used for R22 and R23?” Ubuntu 22.04 for R22 and 24.04 for R23; the host OS had also moved forward over time, and we hadn’t been explicitly encoding that in the results yet.
  • “Can we get logs for spring-mongo?” We recommend against enabling logging for “round runs” (I/O eats CPU), but the benchmark harness and public status pages make it possible to turn them on temporarily (via pull request) and then correlate them with specific continuous runs.

That kicked off a back-and-forth where we coordinated on where and how their team could safely tweak the spring-mongo implementations.

Meanwhile, MongoDB pulled in Intel to help analyze what was happening on their own lab hardware, including runs with and without “performance mode” enabled at the platform level.

So we now had three axes of investigation:

  • The application and its configuration (the spring-mongo code and connection settings)
  • The database process and allocator behavior (MongoDB itself under Docker)
  • The platform-level configuration (power/performance modes, etc., analyzed with tools like PerfSpect)

The team ended up sending two pull requests:

  • #10514 – a workaround for a MongoDB server issue involving TCMalloc per-CPU memory pools when running inside Docker containers.
  • #10565 – changes to the way the spring-mongo client connects to the database, including adjustments to maxPoolSize and related options so the driver can actually make use of the available hardware.

Once both PRs were in and our continuous benchmarking pipeline had cranked through them, Ger sent a summary email with the numbers for spring-mongo:

Test R22 R23 Baseline TCMalloc Fix Only ConnString \+ TCMalloc
query 5.9k 583 837 14.2k
db 104k 68k 192k 205k
fortune 22k 59k 167k 188k
update 2.4k 528 795 10.2k

The query test went from 583 to 14.2k in our environment. That’s roughly a 25x relative to the Round 23 baseline, and significantly better than the Round 22 result. The other tests also improved substantially, especially once the allocator and pool configuration were both in a sane place.

This was not just about our benchmark setup. MongoDB and Intel ran the same workload in their own environments, with and without CPU “performance mode,” and saw similar patterns: the combination fixes make a huge difference.

I like stories where everyone comes out looking better in the end. MongoDB has a much stronger showing in the benchmarks, the benchmark implementations themselves improve as reference examples, and if you’re out there picking a stack for your next project, you get numbers that are closer to what’s actually possible when people who care about performance have had a chance to sharpen the edges.

Product meets Engineering in the AI Era

March 13, 2026

Tony Karrer

Join us on April 10: Product and Engineering Working Together in the Agentic Coding Era

We’ve assembled four product and engineering leaders to share exactly how they’ve retooled their processes. This virtual mini-conference is designed for CPOs, VPs of Product, CTOs, and Heads of Engineering who want practical, ready-to-apply examples — not theory. Register here

CPOs, VPs of Product, and CTOs are experiencing a common challenge: while agentic coding tools accelerate product development, they also introduce new friction between product and engineering. A product manager (PM) creates a spec that tells engineering what they want built, and then one of two things happens:

  • The engineer appropriately asks the agentic coding tool what questions it has. The agent immediately surfaces 15 questions, 12 of which need input from product. You have a cycle time hit and more context switching.
  • The engineer doesn’t surface the questions and builds it anyway. After PR reviews and QA, they realize the implementation does the wrong thing.

One theme for the first half of 2026: product and engineering leaders need to reduce this new friction.

What changed

A PM’s spec has two audiences.

First, people:

  • Reviewers (customers, leadership, other PMs) who need to confirm the product intent.
  • Engineers who need to reason about tradeoffs, durability, and how it fits the architecture.

Second, agents:

  • The agentic coding tool that will try to execute what you wrote, literally, at speed.

So what do we do?

PMs should use codebase-aware tools before handoff

I would highly recommend that product leaders and product managers try out the new Claude Desktop app, which bundles Claude, Claude Cowork, and Claude Code into a more PM-friendly interface. You can use it for a LOT more product needs than creating specs – see the additional reading below.

To get your PMs onboard, consider using the tool to ask:

“What does the product do today in scenario X?”

If you have Claude Desktop connected to your code, it often can answer those types of questions. It also will provide you the answer to:

“Given this draft spec, what questions do we need to answer before someone starts work?”

This helps PMs clarify ambiguity so you avoid the new friction points.

It’s time to change the default from “PMs don’t have visibility into the repo.” That policy actively works against speed and alignment. By giving the AI tooling access to the code base, PMs are empowered with insight while maintaining the separation of responsibilities with engineering.

Side note: Markdown is quickly becoming the shared format for specs because it’s easy to diff, easy to reuse, and plays nicely with repos and agent workflows. Pick a Markdown editor you like (Obsidian is a good choice) and make it part of the standard toolkit.

PRDs and Tickets => Specs

You may want to start calling PRDs / Tickets or other definitions of what’s to be built “specs” internally, not because PRD is wrong, but because it communicates a shift: the output is meant to be fed into an agentic coding tool w/ more specifics.

The upcoming virtual mini-conference and the additional reading has lots of help on this front, for example – acceptance criteria and edge cases are critical.

AI supports PMs but does not replace their judgment; it should enhance decision-making efficiency. Use AI to accelerate drafting, decomposition, and edge case discovery. But the final tradeoffs, priorities, and product decisions still belong to the PM. And us engineers still get to rely on PM judgment to know what to build.

Engineering still has to engineer

A clear spec does not eliminate engineering responsibilities. Strong teams do two things consistently:

  1. Architecture and technical planning: fit the spec into the system in a durable way (constraints, data flows, integration points, performance, security).
  2. Task shaping: break the spec into finer-grained development tasks that are independently testable, so agentic execution stays controlled and reviewable.

A good spec allows the engineers to focus on the work that actually requires engineering judgment.

Reading list

 

Red Teaming Gen AI

February 12, 2026

Tony Karrer

If you’re pushing LLM or RAG features into production, you already know the stakes: the models aren’t just code, they’re evolving systems that interact with unpredictable users and highly variable data. Traditional QA isn’t enough. To ship resilient AI and win confidence from customers and stakeholders, adversarial testing needs to move to the top of your playbook.

Adversarial testing: why it matters for LLM and RAG systems

Adversarial testing or “red teaming” is about trying to make your AI fail on purpose, before malicious actors or edge-case users do. For LLMs and RAG, that means probing for prompt injections, jailbreaks, hallucinations, data leakage, and subverted retrieval strategies.

LLM systems are vulnerable to cleverly crafted prompts that skirt safety limits and encourage harmful, biased, or unauthorized outputs.

RAG and hybrid architectures have unique takeover risks: manipulating the retrieval pipeline, poisoning source documents, or confusing context windows so the model behaves unpredictably.

Adversarial testing uncovers real issues that aren’t obvious until your model is live: privacy leaks, bias amplification, data extraction attacks, and unreliable inferences; all the stuff that keeps CTOs and CISOs up at night.​

How do tech leaders integrate adversarial testing for LLM/RAG?

  • Simulate attacks with both manual red teaming and automated tools and test vectors like prompt injections, data poisoning, and retrieval manipulation.
  • Chain attacks across model and retrieval layers; don’t assume vulnerabilities stop at the model boundary.
  • Use playbooks like MITRE ATLAS, OWASP ML Security Top 10, and keep logs for every test; they’re useful for team learning, postmortems, and compliance.
  • Layer in robust monitoring so adversarial scenarios are caught in real time, not just during scheduled security reviews. Real-time monitoring is essential for both security and reliability.
  • Involve domain experts and skeptics. Adversarial ideation is creative work, not just automation. It takes deep product knowledge and a healthy dose of adversarial thinking to imagine how your outputs could be abused.​
  • Simulate attacks with both manual red teaming and automated tools and test vectors like prompt injections, data poisoning, and retrieval manipulation.
  • Chain attacks across model and retrieval layers; don’t assume vulnerabilities stop at the model boundary.
  • Use playbooks like MITRE ATLAS, OWASP ML Security Top 10, and keep logs for every test; they’re useful for team learning, postmortems, and compliance.
  • Layer in robust monitoring so adversarial scenarios are caught in real time, not just during scheduled security reviews. Real-time monitoring is essential for both security and reliability.
  • Involve domain experts and skeptics. Adversarial ideation is creative work, not just automation. It takes deep product knowledge and a healthy dose of adversarial thinking to imagine how your outputs could be abused.​

Reading List