『How to Increase Agent Accuracy & Reduce Cost with Context』のカバーアート

How to Increase Agent Accuracy & Reduce Cost with Context

How to Increase Agent Accuracy & Reduce Cost with Context

無料で聴く

ポッドキャストの詳細を見る

【Amazonプライム会員限定】今ならプレミアムプランが4か月 月額99円。

10月19日まで。※適用条件あり
What happens when an enterprise data agent asks you: “What is our churn this quarter?” The AI agent has access to the warehouse, the tables, and the query engine. It carries out the necessary calculations. However, the answer it provides turns out to be inaccurate. Why?It’s not because the model is inadequate. It’s because it didn’t know which definition of the term 'churn' the business had in mind, which table was out of date, or which join resulted in customers being counted twice. When you apply this situation to all the questions your enterprise directs at AI, you begin to understand why so many agent projects come to a halt at the demo stage.In this episode of the Don't Panic, It's Just Data podcast, host Shubhangi Dua, Podcast Producer and B2B journalist at EM360Tech, sits down with Suresh Srinivas, the CEO and Co-Founder of Collate and also a Co-Founder of the open-source project OpenMetadata. They talk about the context layer and the reasons why AI agents fail when it comes to enterprise data, even though the underlying models are continually getting better. He explains why the solution must be an open context layer rather than a proprietary one.Also Read: OpenAI's self-service AI data agent, built on OpenMetadataWhat OpenAI's internal data agent shows about scaleSrinivas cites a use case to Dua to explain how the solution can be deployed at scale. He talks about OpenAI's own internal data agent, Kepler, which is used by more than 3,500 people and carries out reasoning across about 70,000 datasets and over 600 petabytes of data, as proof that the issue is not particular to Collate's customers.OpenAI developed a seven-layered context system based on OpenMetadata, and is one of its largest users as well as a major open source contributor. Srinivas maintains that even though there are seven layers, they all come down to three basic elements: the context of the data (what exists), the ontology and semantics (what it means, expressed in business terms such as "revenue" or "churn"), and memory (the corrections and feedback that the agent learns from).Srinivas states that after OpenAI had invested in the context layer, query performance on Kepler decreased from 22 minutes to 90 seconds, representing a roughly 16-fold improvement, together with improvements in token efficiency.Anthropic too has published its own research on the importance of context layers for data agents, which shows that there is convergence among the leading research labs, not merely a point made by Collate.Key TakeawaysMcKinsey: fewer than 10 per cent of enterprises have scaled AI agents to value80 per cent of enterprises cite data limitations as the scaling barrierOn the Spider 2.0 benchmark, the top models achieved only 59 per cent accuracy when tested on enterprise data.Collate's context layer raised accuracy from 59 per cent to 94 per cent on the same benchmarkThat's an 86% reduction in wrong answers from context alone, not a bigger modelToken spend fell 75 per cent once a context layer was addedDatabase queries per question dropped from 190 to 27, an 86 per cent reduction in computeOpenAI's Kepler agent spans 70,000 datasets and 600+ petabytes, built on OpenMetadataOpenAI's query performance improved from 22 minutes to 90 seconds with contextContext layer reduces to three primitives: data context, ontology/semantics, memoryOpenMetadata has 15,000+ community members and 4,000 production deploymentsCollate curates context per persona rather than serving the full context to every agentCollate's AI Governance Studio audits what agents access and the risk they poseChapters00:00 Introduction to AI and Data Context Challenges02:16 The Root Cause of Data Problems: Missing Context04:20 Evolving Definition of Context in AI and Data06:52 Why Data Limitations Still Hinder AI Scaling07:22 The Impact of Context on AI Model Accuracy and Cost09:21 How Providing Context Improves AI Performance11:02 Verifying Data Accuracy in a Constantly Changing Enterprise12:25 AI Agents and Continuous Data Quality Management14:25 OpenAI's Use of Context Layers for Better AI Performance16:03 Avoiding Noise and Cost in Context Management18:47 The Role of Persona-Based Context Curation19:45 Open Source as a Foundation for Context Layers21:37 Accountability for Incorrect or Stale Context22:24 Collaborative Role of Data Teams and AI Agents in Context Management24:01 Guardrails and Deterministic Answers in Enterprise AI25:53 Reducing Token Consumption with Context Layers28:47 Key Takeaway: Building a Robust Context Layer for AI30:37 Upcoming Industry Discussions and Challenges at Big Data London32:28 The Future of AI Agents and the Importance of Context in Data Strategy33:12 Closing Remarks and Next Steps for AI and Data Leaders @CollateData @enterprisemanagement360 #AIagents #EnterpriseAI #DataGovernance #OpenMetadata #ContextLayer #AIaccuracy #AgenticAI #CIO #DataStrategy #EM360Tech #DontPanicItsJustData #opensourcesoftware #opensourceaiAI agents, ...
adbl_web_anon_alc_button_suppression_t1
まだレビューはありません