Synthesis: Execution

The Big Picture

You’ve worked through five lessons: how to write AI PRDs with eval criteria, why AI products have a continuous lifecycle, how cross-functional teams collaborate on AI, why data quality is product quality, and how to adapt Agile for AI development.

Individually, these are tools for different aspects of execution. Together, they form an execution model built on one central insight: AI product execution is uncertainty management. Traditional product development minimizes uncertainty through upfront specification. AI product development embraces uncertainty and manages it through continuous evaluation and rapid iteration.

Connections

1. From PRDs to lifecycle — evaluation as the common thread

AI PRDs define eval criteria (Lesson 1). The lifecycle uses these criteria as a baseline for monitoring and improvement (Lesson 2). Without defined thresholds, you don’t know at launch whether quality is sufficient — and after launch, you don’t know if it’s declining.

For you as a PM: The eval criteria from the PRD aren’t just a launch gate. They’re the foundation for the entire lifecycle.

2. From cross-functional to Agile — rituals connect roles and processes

The eval review ritual (Lesson 3) is the bridge between cross-functional collaboration and the Agile process (Lesson 5). It gives the dual-track approach its rhythm: review experiment results, prioritize improvement actions, define next timeboxes.

For you as a PM: A single structured meeting per week can replace ad-hoc alignment and keep both tracks synchronized.

3. From data quality to PRDs — data as specification

Data quality requirements (Lesson 4) belong in the AI PRD (Lesson 1). What knowledge base quality is expected? How often are documents updated? What data governance policies apply? These questions belong in the specification — not in the retrospective after launch.

For you as a PM: If your AI PRD has no section on data quality, it’s incomplete.

4. From lifecycle to data quality — degradation has two causes

AI products degrade (Lesson 2) either through model drift or data drift (Lesson 4). In practice, data drift is the more common cause — outdated knowledge bases, new user queries outside the coverage, accumulated contradictions.

For you as a PM: When quality drops, check the data first, then the model.

5. From Agile to cross-functional — experiments need all perspectives

Experiment spikes (Lesson 5) only produce actionable results when all perspectives are involved (Lesson 3). An ML engineer experimenting alone optimizes for technical metrics. A PM evaluating alone misses technical constraints.

For you as a PM: Decision gates after experiment spikes should involve PM, tech lead, and (for UX-relevant experiments) designer.

The Meta-Insight

AI product execution follows a cycle: Specify (PRD), Manage (Lifecycle), Collaborate (Cross-Functional), Maintain Data (Quality), Adapt Process (Agile), Repeat. The teams that build the best eval infrastructure build the best AI products.

Your Execution Checklist

What you should now be able to do:

Write an AI PRD with eval criteria, quality thresholds, and failure modes — Lesson 1
Manage the AI product lifecycle from exploration to continuous improvement — Lesson 2
Translate between researcher and product mindsets and establish eval review rituals — Lesson 3
Define data quality as a product requirement and answer data governance questions — Lesson 4
Implement dual-track development with timeboxed experiments and kill criteria — Lesson 5
Diagnose quality drift: is it the model or the data? — Lessons 2 + 4

If any of these feel uncertain, go back to the relevant lesson. Execution is the bridge between strategy and results.

Continue with: AI Leadership

You deliver AI features. Chapter 9 shows how to build and scale AI organizations.

Self-Assessment

Three scenarios combining multiple concepts from this chapter. Think through your answer before revealing the solution.

Scenario 1: The Launch Without Eval Criteria

Your AI feature (an internal support bot) has been live for three months. Users are complaining about declining answer quality, but no one can quantify how bad it actually is — there are no baseline metrics from launch. The VP of Product wants an immediate fix. What do you propose?

Solution

This is a missing link between the PRD (Lesson 1) and lifecycle management (Lesson 2). Without eval criteria in the PRD, there’s no baseline — you can’t tell whether the model has degraded or the data has gone stale. Step one: retroactively define eval criteria and measure a baseline. Step two: check data quality (Lesson 4) — is the knowledge base current? Step three: establish an eval review ritual (Lesson 3) so the problem doesn’t go unnoticed again. An immediate fix without diagnosis would be blind action.

Scenario 2: The Solo Experiment Sprint

Your ML engineer ran a two-week experiment spike testing three prompt variants and wants to deploy the one with the highest accuracy. The UX designer wasn’t involved because “it’s just prompts.” The engineer presents impressive technical metrics. Do you approve the deploy?

Solution

This scenario connects Agile for AI (Lesson 5) with cross-functional collaboration (Lesson 3). Experiment spikes need all perspectives — technical accuracy alone says nothing about user experience. The prompt variant with the highest accuracy might produce the longest, least readable responses. As PM, you should convene a decision gate with PM, tech lead, and designer. Only deploy once user-facing metrics (response clarity, task completion) have also been reviewed. This delays the deploy by days but prevents a rollback after weeks.

Scenario 3: The Stale Knowledge Base

Your AI product answers customer questions about insurance policies. The knowledge base hasn’t been updated in six months, but there have been three policy changes in that time. Your data team says they have no capacity for updates. The hallucination rate is measurably rising. What do you prioritize?

Solution

This is where data quality (Lesson 4) meets lifecycle management (Lesson 2). A rising hallucination rate with an unchanged model is a clear signal of data drift — reality has moved on, the data hasn’t. This belongs in the AI PRD as a requirement (Lesson 1): how often must documents be updated? The priority is clear: knowledge base updates before any feature work. An AI product presenting outdated information as facts destroys trust faster than missing features do.

Sources: Building on Lessons 1–5. Anthropic Documentation (2025), Google Cloud Engineering Blog, Spotify Engineering Blog, LangChain Community Reports