Often agentic loops are especially applicable in diverse domains. In healthcare, a patient chart review requires querying multiple systems, cross-referencing drug interactions, and reasoning across clinical history. In legal work, a contract review needs case law search, clause comparison, and conflict detection — not to mention that you may want to bring in significant client context. The loop exists because these problems are hard to solve with workflow-based designs.

Independent of workflow vs. agentic approach, scenario-based evals are the right approach for domain-specific applications. Each scenario is a specific input — a patient chart, a contract, a claims record — paired with an expert-defined expected outcome. Did the agent identify the medication interaction? Did it flag the indemnification clause? Did it reach the correct coverage classification? Did it ask the right questions and not ask ones it already had access to through context?

Why standard evals break for loops

Workflow-based implementations could test each step in isolation: given this input, does it produce the correct output to pass forward? In agentic loops, that no longer holds. The agent has autonomy over sequencing, and the order of operations may vary across sessions. Tool calls may fail or return unexpected results that the agent reasons past without flagging. An output-only eval catches none of it.

The non-determinism problem in domain evals

Here is where it gets genuinely hard. Evals are specific to a given scenario, but the agent’s path through that scenario is not fixed. Different sessions may pull different chunks from the retrieval system. The conversation may branch differently. The agent may reach the right conclusion through a different sequence of tool calls.

The way to handle this is to separate what you are actually testing. For the output layer, test the conclusion: did the agent identify the issue that matters? For the trajectory layer, test the key decision points: did it retrieve the document that contains the interaction? Did it call the drug lookup tool? Did it pass the correct patient ID? Trajectory evals do not require the agent to take an identical path every time. They require that the path passed through the checkpoints that justify the conclusion.

A case where the agent arrived at the right answer but skipped a critical tool call should not pass. That is not a correct evaluation. It is a lucky guess that will fail on the next prompt change.

How to start building this

You do not need a complete scenario library to get started.

• Start with production failures. Any trace where the agent reached a wrong conclusion, missed an issue, or called the wrong tool is a scenario waiting to be labeled. Get a domain expert to define correct behavior for that case and add it to the set. Five real failures are worth more than fifty synthetic happy-path cases.
• Write trajectory checkpoints, not just output assertions. For each scenario, identify the one or two tool calls or retrieval steps that are load-bearing. Test that those happened correctly and with the right parameters.
• Make “correct” explicit. Not a matching string, but a rubric: what must the output include, what must it not include, what tool calls must appear in the trace.
• Run evals on every significant change. Every prompt update, model upgrade, or tool integration should trigger a full scenario run. If scores drop, you do not ship.

The judge is itself a component to evaluate

Using LLM-as-judge to score outputs at scale is essential, but treat the judge as a component in your eval system, not a fixed oracle. Like every other component, it can drift. After a model update, your judge’s behavior may shift in ways you have not noticed. Build a calibration set for the judge: a sample of cases with human expert labels. Run the judge against them periodically and watch for divergence. When judge scores and expert scores disagree consistently, that is a signal. Either the judge has drifted, or your calibration set no longer reflects the scenarios the agent actually handles.

Treating the judge as infallible leads to calibration debt that accumulates invisibly and surfaces as production failures nobody predicted.

Reading list

• AI Agent Evaluations: A Developer’s Practical Guide (MLflow, 2025/2026) — Good on what specifically changes when you move from LLM to agent evals. The 75% judge-human agreement threshold as a recalibration trigger is a practical heuristic worth adopting. Apache-licensed tooling, no paywall.
• Evaluations for the Agentic World (QuantumBlack, January 2026) — Two detailed client case studies in regulated industries, a European telco and a European bank, covering trajectory validation, SME review as an eval layer, and continuous quality monitoring. The most concrete multi-agent eval case studies available in a public piece.
• A Pragmatic Guide to LLM Evals for Devs (Pragmatic Engineer, 2025) — The best starting point for teams new to systematic evals. Covers why LLM non-determinism breaks traditional testing and how to build a minimal eval suite that actually catches regressions. The golden dataset construction section is directly applicable to domain-specific scenario libraries.
• A Methodical Approach to Agent Evaluation (Google Cloud, November 2025) — The cleanest structural framework I have found: three pillars covering final output quality, trajectory analysis, and trust/safety. Worth reading specifically for the section on building a golden dataset when you do not yet have one. The Vertex AI mention at the end is a brief plug; the framework itself is tool-agnostic.
• Awesome-Evals (BenchFlow, GitHub) — A curated library of papers, blog posts, tools, and benchmarks organized by topic. Useful if you want to go deeper on any specific area: tool-calling evals, LLM-as-judge calibration, multi-agent evaluation. The “non-BS” framing in the description is accurate.

Almost every CTO or VPE we talk to is asking some version of the same question: “Are we picking the right tools, building out the right workflows, and putting our people in the right places?” In other words: how should engineering teams actually operate now, with agents writing most of the code?

Teams are landing in pretty different places. Some are pushing toward full autonomy with minimal human review. Others are keeping tight control, using AI more as a reviewer or assistant. Most are somewhere in the middle.

But which workflow you pick isn’t really the challenge. The inputs are.

The bottleneck has moved. Writing code is getting cheaper, but building reliable systems requires more than just code. Specs, tests, and review are now the limiting factors, and agents amplify whatever you feed them. Strong inputs get strong results. Weak ones fail faster, at scale, and with less visibility into why.

One theme for the second half of 2026: engineering leaders need much better visibility into what other teams are actually doing. Too much of this is still being figured out in private, and some of the public conversation is noise.

What’s actually working

From what we’re seeing across teams and practitioner write-ups, a few patterns are emerging. Not as a single “right way,” but as things that consistently hold up.

Front-load the thinking: spec, tests, then code

The teams getting reliable output are putting more effort into shaping the problem up front (acceptance criteria, edge cases, tests) before letting agents implement. The shift is subtle but important: less time writing code, more time defining what “correct” looks like.

Parallelize aggressively, but with boundaries

Running multiple agents in parallel is becoming common: one exploring, one implementing, one cleaning up. (Stripe built a harness that ships >1,000 agent-driven PRs per week. ) But the teams doing this well isolate each agent in its own worktree, container, or sandbox, so mistakes don’t cascade. Parallelism helps, but only if it’s contained.

Where this breaks

A few failure modes show up just as consistently.

Tests written after the fact

This is amplification working against you. When you ask an agent to write tests after the implementation, the existing code becomes the spec. Agents don’t push back. They complete the task you gave them, even if the task is wrong. The result: tests that lock in whatever’s broken. Discipline matters more with agents, not less: tests first, then implementation.

Context can be a liability

Many teams are leaning on rules files (CLAUDE. md, AGENTS. md) to guide behavior. But they go stale quickly. Codebases evolve, and these files rarely keep up. Without regular review, they can make things worse, with agents following outdated or rigid instructions. Structure is necessary for good results, but bad structure gets amplified.

What actually changes for teams

Coding agents don’t remove the need for senior engineering judgment. They concentrate it. The work shifts toward defining problems clearly, validating outputs, and keeping the whole system reliable. Teams with weak specs, inconsistent tests, or overloaded review processes feel more pain, not less. Teams with strong fundamentals move faster.

The work moves up the stack, and the pressure moves with it. And the systems you build around the agents matter more than the agents themselves.

Reading list

• Embracing the parallel coding agent lifestyle – by Simon Willison. Concrete patterns for running several coding agents at once. Covers worktrees and Docker for isolation, what kinds of work to delegate to parallel sessions, and what to supervise more tightly.
• How Stripe built “minions” – by Steve Kaliski via Lenny’s Newsletter. Inside Stripe’s production agent harness shipping ~1,300 PRs/week from Slack reactions. Covers the harness layers, how they handle code review at that scale, and what they had to build vs. adopt off-the-shelf.
• Why Testing After with AI Is Even Worse – by Matti Bar-Zeev. Why asking an agent to write tests after the implementation produces tests that validate existing bugs instead of catching them. Makes the case for TDD with agents, not against it.
• How System Prompts Define Agent Behavior – by Srihari Sriraman and Drew Breunig (nilenso). A close read of system prompts across Claude Code, Cursor, Codex, Gemini, and others. Shows the same model producing dramatically different workflows depending on the prompt wrapped around it.
• Your CLAUDE. md Is Making Your Agent Dumber – by Cordero Core. Recent research finding that LLM-generated CLAUDE. md / AGENTS. md files actively decrease agent success rates compared to having no context file at all. Practical guidance on what to do about it.
• Ralph Wiggum as a “Software Engineer” – by Geoffrey Huntley. Walks through “Ralph,” a bash-loop technique for autonomous coding agents. Concrete on what kinds of projects it suits and where senior engineering judgment stays non-negotiable.

CPOs, VPs of Product, and CTOs are experiencing a common challenge: while agentic coding tools accelerate product development, they also introduce new friction between product and engineering. A product manager (PM) creates a spec that tells engineering what they want built, and then one of two things happens:

• The engineer appropriately asks the agentic coding tool what questions it has. The agent immediately surfaces 15 questions, 12 of which need input from product. You have a cycle time hit and more context switching.
• The engineer doesn’t surface the questions and builds it anyway. After PR reviews and QA, they realize the implementation does the wrong thing.

One theme for the first half of 2026: product and engineering leaders need to reduce this new friction.

What changed

A PM’s spec has two audiences.

First, people:

• Reviewers (customers, leadership, other PMs) who need to confirm the product intent.
• Engineers who need to reason about tradeoffs, durability, and how it fits the architecture.

Second, agents:

• The agentic coding tool that will try to execute what you wrote, literally, at speed.

So what do we do?

PMs should use codebase-aware tools before handoff

I would highly recommend that product leaders and product managers try out the new Claude Desktop app, which bundles Claude, Claude Cowork, and Claude Code into a more PM-friendly interface. You can use it for a LOT more product needs than creating specs – see the additional reading below.

To get your PMs onboard, consider using the tool to ask:

“What does the product do today in scenario X?”

If you have Claude Desktop connected to your code, it often can answer those types of questions. It also will provide you the answer to:

“Given this draft spec, what questions do we need to answer before someone starts work?”

This helps PMs clarify ambiguity so you avoid the new friction points.

It’s time to change the default from “PMs don’t have visibility into the repo.” That policy actively works against speed and alignment. By giving the AI tooling access to the code base, PMs are empowered with insight while maintaining the separation of responsibilities with engineering.

Side note: Markdown is quickly becoming the shared format for specs because it’s easy to diff, easy to reuse, and plays nicely with repos and agent workflows. Pick a Markdown editor you like (Obsidian is a good choice) and make it part of the standard toolkit.

PRDs and Tickets => Specs

You may want to start calling PRDs / Tickets or other definitions of what’s to be built “specs” internally, not because PRD is wrong, but because it communicates a shift: the output is meant to be fed into an agentic coding tool w/ more specifics.

The upcoming virtual mini-conference and the additional reading has lots of help on this front, for example – acceptance criteria and edge cases are critical.

AI supports PMs but does not replace their judgment; it should enhance decision-making efficiency. Use AI to accelerate drafting, decomposition, and edge case discovery. But the final tradeoffs, priorities, and product decisions still belong to the PM. And us engineers still get to rely on PM judgment to know what to build.

Engineering still has to engineer

A clear spec does not eliminate engineering responsibilities. Strong teams do two things consistently:

1. Architecture and technical planning: fit the spec into the system in a durable way (constraints, data flows, integration points, performance, security).
2. Task shaping: break the spec into finer-grained development tasks that are independently testable, so agentic execution stays controlled and reviewable.

A good spec allows the engineers to focus on the work that actually requires engineering judgment.

Reading list

• Agentic workflows for software development – A good overview of agentic workflow design, especially the idea of deterministic checks and evaluation gates that keep speed from turning into chaos.
• How to write PRDs for AI Coding Agents – A practical reframing of PRDs as “spec architecting” so agents (and engineers) can implement reliably with fewer missing assumptions.
• How to Write a PRD with AI (Claude Code tutorial) – Template-forward guidance on assembling a PRD/spec from known context files (including a prd-template.md style approach) so your spec is consistent and agent-friendly.
• How to write a good spec for AI agents – The clearest explanation of “spec as prompt”: why big docs fail and how structure plus constraints drives correct execution.
• Claude Code for Product Managers – A PM-oriented walkthrough of exploring a repo in plain English to tighten tickets/specs and surface the right questions before handoff.
• Claude Code for PM work – Grounded notes on where PM workflows break today (docs are incomplete, reality lives in code) and how codebase-aware tools can reduce that gap.
• The Definition of Ready in the AI Era – A strong argument for shifting DoR from “can dev start?” to “if built exactly as written, would we be satisfied?”

AI coding tools are transforming how we make software. But measuring the impact of these tools is harder than it looks!

To address this pressing issue, we are excited to announce our upcoming webinar: AI Coding Tool Metrics: DORA and CTOs Deep Dive. This expert-led session aims to provide engineering leaders with the clarity and tools needed to navigate the complexities of measuring the impact of AI coding tools effectively.

For the first time, the LA CTO Forum is opening this session to a broader audience. Join us, along with fellow CTOs, VPEs, heads of engineering, senior product leaders, and IT leaders, to gain a practical and reality-based view of measuring AI coding tools in the real world.

Event Details

• Date: Friday, January 9, 2026
• Time: 8:00 AM – 10:00 AM PST / 11:00 AM – 1:00 PM EST

Reserve your spot

During this two-hour mini-conference, attendees can expect:

• Insights from a DORA researcher on how high-performing teams are adopting AI-assisted development and the key metrics that correlate with better outcomes.
• Real-world experiences shared by two CTOs on measuring AI tools in their organizations, including utilization, quality, satisfaction metrics, and handling non-code work.
• A moderated discussion among CTOs and attendees to address key questions and concerns.

Key Takeaways

• Discover the metrics used by leading organizations to measure the impact of AI coding tools and the tools that can help capture them.
• Learn how to assess where your team stands on the AI adoption curve and strategies to catch up if needed.
• Understand the hidden value AI tools provide beyond just increasing code output.

Don’t miss this opportunity to gain valuable insights and strategies to effectively measure the impact of AI coding tools in your organization.

All registrants will receive the slides and a full session recording.

AI is transforming how software gets built. Teams that integrate AI into their SDLC the right way are seeing faster delivery cycles, lower costs, and higher ROI.

To help teams make that transition effectively, TechEmpower is hosting a webinar:
Leveraging AI Tooling Across Your Software Development Lifecycle.

The session will be moderated by Tony Karrer, CEO of TechEmpower, with featured guest Brent Laster,
author of The AI-Enabled SDLC (O’Reilly). They’ll share practical strategies for integrating AI tools
across every stage of software development—from planning and coding to testing, documentation, and deployment.

This webinar will help attendees connect the dots and move from ad-hoc AI experiments to real-world, AI-driven workflows that scale.

Event Details

• Date:
  Thursday, November 13, 2025
• Time:
  11:00 AM – 12:00 PM PST /
  2:00 PM – 3:00 PM EST
• Reserve your spot

What You’ll Learn

• AI use cases across key SDLC phases: where to start and how to scale
• Real-world examples that work: AI-assisted coding, reviews, testing, documentation, and more
• Team enablement strategies: roles, prompting approaches, and workflows for adopting AI

All registrants will receive the slides and a full session recording.