Standard AI benchmarks measure speed and raw token throughput, but how do you measure whether artificial intelligence is actually completing enterprise work correctly?

In this episode of Utilizing AI, co-hosts Stephen Foskett and Brad Shimmin welcome Ryan Shrout, President and GM of Signal 65, to introduce "Pinnacle"—a new benchmark designed to evaluate real-world, agentic workflows.

Ryan breaks down why traditional hardware and model metrics miss the mark on business value, how Pinnacle evaluates performance across clean "governed" versus messy "as-found" data environments, and what recent test runs reveal about token efficiency, hallucination resistance, and hardware scaling.

From comparing frontier models to exposing the hidden costs of max-reasoning modes, this discussion offers a grounded look at quantifying true enterprise AI performance.

This and more on Utilizing AI, part of Futurum Media and the Futurum Podcast Network.

When cutting-edge AI capabilities shift overnight from exclusive enterprise clouds to permissive open-weights models accessible to anyone, IT leaders are left navigating a chaotic landscape of unprecedented power and heightened security risks.

In this episode of Utilizing AI, hosts Stephen Foskett, Brad Shimmin, and Olivier Blanchard unpack the realities of today’s expanding AI toolset—from high-performing open-source models coming out of China to Stripe's acquisition of OpenRouter.

The panel explores the stark tradeoffs between affordable accessibility, data privacy, and AI supply chain security. They also debate the rise of local hardware powerhouses for on-premise execution, the volatile "tokenomics" of reasoning models, and why the enterprise world desperately needs stable, Long-Term Support (LTS) release channels for AI software.

This and more on Utilizing AI, part of The Futurum Group Podcast Network.

Enterprise IT is currently facing a dizzying explosion of new AI models, frameworks, and platforms—making integration, rather than security or governance, the primary roadblock to actual business success.

In this episode of Utilizing AI, host Stephen Foskett is joined by Futurum Group analysts Brad Shimmin and Guy Currier to explore how organizations can effectively navigate this overwhelming "embarrassment of riches."

The panel breaks down the real-world friction of non-deterministic model behavior, sudden API performance shifts, and hidden tokenomics costs, evaluating whether emerging tools like model routers offer a true fix or merely add complexity.

Ultimately, they offer practical guidance for enterprise architects: double down on precise scope management, prioritize output quality and consistency over pure speed, consider open-source models for stability, and manage AI as a grounded IT asset rather than an all-knowing solution.

This and more on Utilizing AI, part of The Futurum Group Podcast Network.