Over the last several episodes of Talking Product, Collin and I have been exploring how we increase the probability that our AI investment will translate into business value.

Recent news has shown how organisations are having to confront the economics of AI more directly: token budgets being consumed faster than expected, tighter controls on usage, the challenges of moving from prototype to production, how to shape experiments around coherent hypotheses tied to a business outcome, and whether the use of GenAI is actually translating into customer value.

In Episode 15, we explore the mindset shift that I have experienced through working with the DSPy framework and becoming familiar with the work of Omar Khattab. This is not a technical discussion. Instead, Collin and I continue to explore the role non-technical senior leaders should play in shaping the AI journey.

I believe anyone in an AI leadership role should familiarise themselves with Omar’s work if they haven’t done so already. I have found him to be a grounded, coherent voice in a world inundated by hyperbole and breathlessness.

Omar Khattab is an Assistant Professor at MIT. His doctorate is from Stanford. His research focuses on how we program, evaluate and optimise systems built around language models. He created DSPy, an open-source framework for building and optimising LLM applications programmatically, rather than relying primarily on manually engineered prompts.

In this episode, Collin and I explore:

• What belongs to LLM inference and what belongs to good ol’ deterministic programming.
• Defining tight, measurable evaluations — yes, we’ve been banging on about this since covering Chip Huyen’s book.
• The power of optimisation — don’t just believe us, go and look at the use cases on the DSPy website.
• Building on that optimisation, how lower-cost models can sometimes deliver equal and even better performance.
• The role non-technical senior leaders should play in shaping the AI journey.

If you like the session, please subscribe through your preferred podcast service and leave us a smashing five-star review. It helps us build an audience and attract guests to the show.

And, as always, feedback is more than welcome.

Thanks.

Links referred to in the show:

Omar's home page: https://omarkhattab.com/

Two podcasts:

How Foundation Models Evolved: A PhD Journey Through AI's Breakthrough Era: https://podcasts.apple.com/us/podcast/how-foundation-models-evolved-a-phd-journey-through/id1740178076?i=1000737229448

DSPy: Transforming Language Model Calls into Smart Pipelines // Omar Khattab: https://podcasts.apple.com/gb/podcast/dspy-transforming-language-model-calls-into-smart-pipelines/id1505372978?i=1000637537521

I also found this post helpful: https://x.com/lateinteraction/status/1921565300690149759

In this episode of Talking Product, Collin and I discuss a tension we increasingly see inside organisations: digital capability is becoming an enterprise survival issue rather than a technology optimisation issue.

Unlike some of our previous episodes, this is less a discussion about a single topic and more a conversation between two practitioners trying to reconcile lived experience with organisational inertia. We believe this tension is surfacing more visibly now because of the disruption caused by generative AI.

Collin highlights an article I published on Substack at the end of last year. In it, I argued that the tolerance for organisational inefficiency and limited digital capability is likely to compress dramatically in the years ahead. Organisations that have invested time and energy into improving product delivery practices, architectures, infrastructure, data quality, and operating models are materially better positioned than those that have largely treated such initiatives as secondary concerns.

This led me to a broader question: if executives and boards are not asking deeper questions about AI, data, software capability, and operating models, who is? My conclusion is that institutional investors need to engage far more directly with digital capability risk because many portfolios are likely carrying significantly more exposure than is currently recognised.

We do not pretend to have definitive answers to many of these questions. In fact, part of the episode is an acknowledgement of just how difficult these topics are to navigate. But we once again argue for the importance of having these conversations inside organisations — because avoiding them does not reduce the risk. It simply delays the moment when those risks become visible.

Lastly, we once again argue that getting Chip Huyen’s book AI Engineering and starting conversations inside your organisation about the subjects she raises will be very good value for money — and may save you a great deal of money and grief further down the road.

Takeaways:

• An exploration of why digital capability is an enterprise-level risk rather than a delegated IT concern.

• A discussion about how generative AI accelerates the consequences of technical debt and organisational underinvestment.

• Questions around whether executives, boards, and institutional investors are asking sufficiently deep questions about AI, data, and operating models.

The Digital Laggard Thesis: https://johnyoung.substack.com/p/towards-an-investment-thesis-on-digital

In this episode, Collin and I continue building our list of questions to help increase the chances of your AI investments delivering a return. We focus in particular on the role non-technical C-level leaders should play in this effort.

We use Chip Huyen’s AI Engineering as a practical framework, exploring how it gives digital leaders and non-technical senior managers a structured way to engage more deeply with AI initiatives. In particular, it provides a way into the ecosystem and lifecycle of AI application development—helping leaders ask better questions around things like data, prompt design, fine-tuning, and evaluation—so they can have more meaningful discussions with technical teams about how these applications are built, where the risks sit, and what criteria should be used to define and measure success.

We walk through some of the key differences between traditional software and AI systems—particularly the shift from deterministic to probabilistic behaviour, and the central role of data in shaping outcomes.

From there, we build on the questions we believe leaders should be asking: What problem are we solving? How are we evaluating outputs? How are we managing risks around data quality, safety, and factual accuracy? What trade-offs are we making between quality, cost, and latency?

We spend some time looking at Chip’s section on evaluation criteria, using it as a springboard for non-technical senior leaders to delve deeper into the thinking behind—and expected outcomes of—AI applications. We also introduce the concept of “evals”—ongoing evaluation frameworks that extend beyond traditional testing—and why they require continuous iteration, collaboration, and oversight, even after deployment.

This episode continues our exploration of how leaders can better understand what they are funding, engage more effectively with product and delivery teams, and create the conditions for AI investments to deliver real value.

Links to Chip's book & interview on evals referred to in the episode:

Chip Huyen’s AI Engineering: https://www.oreilly.com/library/view/ai-engineering/9781098166298/

Lenny’s podcast - Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar: https://www.youtube.com/watch?v=BsWxPI9UM4c