Skip to content

Why Enterprise AI Isn't Delivering ROI | Ahmed Bashir, DevRev

Eye On A.I.

Hosted by Craig S. Smith

37m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores why enterprise artificial intelligence initiatives frequently fail to deliver return on investment despite impressive initial demonstrations. The discussion focuses on the critical gap between powerful foundational reasoning models and the lack of a sophisticated enterprise memory layer. By examining industry benchmarks, token efficiency, and system architecture, the conversation highlights how traditional data fragmentation, complex permissions, and organizational silos impede AI adoption. The guest explains that successful enterprise deployments require an AI-native data format and bidirectional orchestration that connects diverse tools while respecting established security boundaries.

  • A more sophisticated model cannot compensate for the absence of a proper memory and governance layer. 0:29
  • Enterprise benchmarks must account for data complexity rather than just measuring raw task complexity. 12:30
  • Using deterministic databases for context retrieval instead of relying solely on LLMs saves significant tokens and money. 19:34
  • Shared memory layers provide the essential semantic understanding, orchestration, and permissions required to update enterprise systems automatically. 13:07

Chapters

  1. 0:00
    Enterprise AI Demo Gap

    Explains why impressive demos fail to deliver ROI in complex enterprise environments.

  2. 0:37
    Computer Product Overview

    Details the flagship product, Computer, and its role as an AI-native knowledge graph and conversational teammate.

  3. 3:30
    Levels of Autonomy in Tasks

    Defines the four levels of task autonomy used in the benchmark to measure AI capabilities across domains.

  4. 6:32
    Key Shared Memory Characteristics

    Outlines the essential properties of shared memory: cross-tool relation, bidirectionality, and permission handling.

  5. 8:46
    Conversational Interface Design

    Explains how Computer’s interface blends natural language with visible catalogs and collaborative features.

  6. 12:30
    Enterprise Benchmark Purpose

    Describes the creation of an enterprise benchmark that tests end-to-end AI systems under realistic data conditions.

  7. 18:36
    Benchmark vs. Model Metrics

    Contrasts traditional model-focused benchmarks with the new benchmark that emphasizes data complexity and orchestration.

Highlights

  • 0:12

    The ROI Gap in Enterprise AIinsight

    While many organizations adopt advanced AI models, few achieve real return on investment because of deficiencies in governance, orchestration, and semantic understanding at scale.

  • 1:24

    The Analogy of Redigitizationstory

    Music formats had to evolve from CDs to support portable devices and streaming. Similarly, enterprise data architecture must undergo redigitization to become AI-native.

  • 12:30

    The Flaw of Industry-Leading AI Claimsinsight

    Relying on the assumption that using the best underlying model equals industry-leading AI is a misleading premise that ignores the reality of organizational complexity.

  • 19:34

    Testing the Entire Agent Harnessstrategy

    Component benchmarks miss the mark by evaluating isolated models. Effective evaluation requires testing the complete logical unit, which includes the agent harness and data management.

  • 20:31

    Comparative Accuracy Deltadata

    Testing reveals a 50 percent relative accuracy improvement when using shared memory architecture compared to standard agent configurations using the exact same underlying model family.

  • 29:25

    Token Reduction via Databasesdata

    Using database queries for context instead of forcing agents to paginate through vast data via MCP results in a 75 to 80 percent reduction in token usage.

  • 31:30

    Combining Databases and LLMsstrategy

    Rather than relying entirely on probabilistic LLMs for context building, systems should leverage deterministic SQL databases and CPUs to handle structured relationships safely.

Quotes

  • “I think that the major observation is that a more sophisticated model cannot always make up for not having a sophisticated memory layer.”
    Unnamed speaker · 0:29
  • “You know, when we moved to the cloud, we didn't just take databases as they were on-premise and just move them over.”
    Unnamed speaker · 2:38
  • “Shared memory is what gives you the semantics, the governance and the orchestration.”
    Unnamed speaker · 21:09
  • “And, you know, it's something that existed for forty, fifty years.”
    Unnamed speaker · 32:02
  • G2 17:23–18:14
  • Mind Trip 22:43–24:43

About this episode

The pattern is familiar to every enterprise leader: the AI demo works perfectly, the enterprise deployment disappoints, and the CIO can't explain why. Ahmed Bashir, Chief Technology Officer of DevRev, joins Craig Smith with a specific and well-supported answer: the problem isn't the model. It's that enterprise data is organizationally complex, fragmented across dozens of tools, locked behind conditional permissions, organized by custom schemas, and every prominent AI benchmark tests models against flat files with pre-packaged data, which has nothing to do with how enterprise data actually lives. DevRev's Enterprise Bench is the first benchmark to test complete agent harnesses against a realistically distributed enterprise environment where the agent must find, relate, and validate information before performing any task, and the results expose a gap that pre-packaged benchmarks hide entirely. The most striking finding: DevRev's system achieves 50% higher accuracy than Claude Code using the identical Opus family model. The entire delta comes from the shared memory layer, the governance, semantic understanding, and orchestration that sits between the model and the enterprise data. Ahmed's analogy is precise: upgrading a CPU cannot compensate for insufficient RAM, because they perform fundamentally different functions. A more powerful model cannot make up for a weak memory layer for the same reason. The conversation also covers why Enterprise Bench aspires to be for AI agents what TPC benchmarks became for databases in the 1980s - the independent standard that ended vendor benchmark wars and gave buyers a genuinely apples-to-apples comparison - and why the build-vs-buy question in enterprise AI has a quantifiable answer for the first time, visible on a public leaderboard that puts DIY configurations directly alongside proprietary systems. Subscribe to Eye on A. I. for weekly conversations with the people building and deploying the future of AI.

Transcript

0:00
37:28

6,235 words · 2 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • AI
  • Computer
  • shared memory
  • MCP
  • Apple
  • iPhone
  • Databricks
  • iCloud
  • OpenAI
  • Snowflake

Page updated · transcript and summary are machine-generated