AI Won't Answer Everything for Your Team: It Will Answer What Matters Well [and That's Worth Its Weight in Gold]

AI Won't Answer Everything for Your Team: It Will Answer What Matters Well [and That's Worth Its Weight in Gold]

Text-to-SQL in production still fails more often than marketing admits. But conversational BI handles frequently asked questions well, and only 25% to 35% of employees use BI tools. The truth about AI analytics.

Summarize this article with

TL;DR

Although conversational AI still doesn't match human analysts on complex queries, its real value lies in effectively answering the repetitive, simple questions that make up most daily requests. This lets most employees, who don't use BI tools, access data, freeing analysts for more complex work. The evidence shows that people who use generative AI are, on average, 33% more productive in each hour they use it, and that AI improves the quality of work, making access to information more democratic and efficient.

On this page

This is the most uncomfortable post in the whole series. Because so far we have been taking apart other people's lies: the 98% of WhatsApp, the 80% of chatbots, the 99% of invoice readers. But today the number to be skeptical about is one that our own product would benefit from exaggerating.

The dream that the whole conversational business intelligence industry sells is seductive: "Ask your data in natural language and get the answer instantly. No SQL, no waiting for the analytics team to build you the report." It is the promise of Metrix, our conversational BI product. And it is a real promise, with an asterisk that almost nobody in the industry dares to add.

Today we add the asterisk. And by the end you will understand why we believe that, even with the asterisk (in fact, thanks to the asterisk), the opportunity is still huge.

The asterisk: translating language into SQL is still hard

The technical magic behind conversational BI is called "text-to-SQL": turning a question in plain language ("how much did we sell in Córdoba last quarter compared to the previous one?") into the database query that returns the right answer. The industry's marketing implies that this problem is solved. The independent academic evidence says otherwise.

The BIRD benchmark, presented at NeurIPS, one of the most demanding artificial intelligence conferences in the world, evaluated the best models on 12,751 question-query pairs across 95 real databases from 37 different domains. The result: GPT-4 reached an accuracy of 54.89%, compared to 92.96% for an expert human analyst (Li et al., NeurIPS, 2023).

Read that again. On real-world databases, the best model of its time got a little over half of the answers right, versus more than 90% for a human.

And when the benchmark gets even more realistic, the numbers get worse. Spider 2.0, designed to reflect real enterprise workflows, with their enormous schemas, badly named tables and real complexity, measured that GPT-4o solved just 10.1% of the problems, and o1-preview 17.1% (Spider 2.0, ICLR, 2025).

This is the reality that marketing hides: AI that queries data is still far from replacing an expert analyst on complex queries. Any vendor that promises you "100% accuracy, any question, instantly" either did not read these studies, or prefers that you not read them.

So, what is conversational BI good for?

Same move we made with the sales chatbot in the first post of the series: if the raw data discourages you, it is because you are asking the wrong question.

The question is not "can AI answer any data query as well as my best analyst?". The answer to that is no, not yet. The right question is: "how many of the questions your people ask every day are, in reality, simple?"

Because the truth at any company is that most questions to the data are repetitive and straightforward: how much we sold yesterday, what the stock of this product is, which sales rep is doing best this month, how collections are going. You don't need an analyst with a master's degree to answer that. You need the answer to arrive fast and without friction. And there, in the bulk of the volume, not in the tail of complex queries, conversational BI is transformative.

The real value is not in replacing the analyst on the hard question. It is in taking the hundred easy questions off their plate so they can focus on the hard one. And along the way, in something even bigger.

The data point that changes the size of the opportunity

Here is the figure that, for us, justifies the whole product. And unlike text-to-SQL, this one has solid independent backing.

Despite decades of investment in business intelligence tools, only between 25% and 35% of a company's employees actively use those tools. The 25% figure comes from a joint survey by BARC and Eckerson Group of 214 companies, which also pointed out something damning: that percentage "reflects minimal growth over the last seven years" (Eckerson Group / BARC, 2022). Gartner put it at 35% in 2019, according to TechTarget, 2022.

Think about what that means: your company probably already pays for BI licenses, already has the data, already built the dashboards. And most people don't touch them. Why? Because you have to learn how to use them, you have to know where to look, you have to understand the tool. The barrier is not the data: it is the friction of accessing it.

That is exactly what a conversational system tears down. Not because it answers impossible questions, but because it turns the people who don't use the data into people who can simply ask. The value is not in the sophistication of the answer; it is in the number of people who, for the first time, get one.

How much working with AI really pays off

And how much does productivity concretely improve for someone who uses generative AI to work with information? Here the independent evidence is solid again, and it is worth more than any vendor success story.

A controlled experiment with 758 Boston Consulting Group consultants, one of the most cited studies on AI and knowledge work, found that those who used GPT-4 completed 12.2% more tasks, did them 25.1% faster, and delivered quality more than 40% higher as measured by blind evaluators (Dell'Acqua et al., Organization Science, 2025; summary by Harvard Business School).

And at the macroeconomic level, research from the Federal Reserve Bank of St. Louis based on a national survey estimated that workers who use generative AI are, on average, 33% more productive in each hour they use it (Federal Reserve Bank of St. Louis, 2025).

McKinsey, for its part, estimates that generative AI has the potential to automate between 60% and 70% of the time that many work activities consume today (McKinsey, 2023). It is modeled potential, not a guaranteed result (and we say so, as always), but it shows the scale of what is at stake.

A word about the "88% faster"

We can't close without mentioning the star figure of the conversational BI sector. You will see cases of an "88% reduction in time to get an insight," typically tied to a ThoughtSpot case with the customer Cox 2M (ThoughtSpot, Cox 2M case).

Is it a lie? No. It is a real case, from a named customer. But it is a single case, told by the vendor, with no external audit. Taking that 88% and presenting it as a general promise would be repeating exactly the mistake this series is here to correct. That is why you won't find that number on our site as if it were a guarantee for you. You will find a claim we can stand behind: frequently asked questions go from hours or days of waiting to seconds. Without attaching a percentage that isn't ours.

What you will read on the Metrix site

After all this research, these are the figures we chose for Metrix, our conversational business intelligence product:

  • "Only 25% to 35% of your employees use data tools. Democratize access." Source: BARC and Eckerson Group, 2022, and Gartner, 2019 (via TechTarget).

  • "25% faster and more than 40% higher quality in analysis work with AI." Source: experiment with 758 consultants, Organization Science, 2025.

  • "33% more productivity in every hour of work with AI." Source: Federal Reserve Bank of St. Louis, 2025.

  • "From hours to seconds on your business's frequently asked questions": no invented percentage, because complex questions still need a human, and we prefer to tell you so.

What you won't read: that Metrix answers anything with perfect accuracy. It doesn't. No system in the world does today, and the benchmarks prove it. What it does is something more useful and more honest: it gives your data a voice for the questions your people ask every day, and it gives hours back to your analytics team for the questions that really matter.

That is the difference between selling a dream and delivering a tool. We chose the second.


The full series: The real numbers behind enterprise AI

  1. The AI industry lies to you with numbers: we investigated which ones are real

  2. Sellium: Your chatbot doesn't resolve 80% of inquiries, and we lie to you less

  3. Revio: The "98% open rate" of WhatsApp is made up, here is the real number

  4. Metrix: AI won't answer everything for your team (you are here)

  5. Arconte: Reading invoices with AI, where 99% is true and where it is marketing

  6. Trazzo: Last mile with AI, how much fuel, time and calls you really save

Every figure in this series is backed by its source, with date and methodology when available. If you find a figure you think we should review, write to us: we take this seriously.

Frequently asked questions

How accurate is conversational AI at answering complex questions about data?

Academic evidence shows that current AI models, such as GPT-4, have limited accuracy on real-world databases. On demanding benchmarks, they get it right a little over half the time or even less, far from the performance of an expert human analyst.

What is the real value of conversational BI if AI is not 100% accurate?

The main value is answering quickly and without friction the vast majority of simple, repetitive questions asked every day. This frees analysts to focus on complex queries and lets more people access the information they need.

Why don't most employees use existing Business Intelligence tools?

Despite investments in BI, only 25% to 35% of employees actively use these tools, according to the study. This is due to access friction: you have to learn how to use them, know where to look and understand the tool, which creates a barrier for most people.

How can conversational BI increase a company's productivity?

By removing the friction of accessing data, conversational BI lets most employees, who don't use these tools, simply ask and get answers. This democratizes access to information and improves decision-making across the organization.

What impact does using generative AI have on worker productivity?

Controlled studies show that using generative AI significantly increases productivity. Consultants who used GPT-4 completed more tasks, faster and with higher quality, and a survey from the Federal Reserve Bank of St. Louis estimates that people who use generative AI are, on average, 33% more productive in each hour they use it.

Topics

Juan Manuel Garrido

Written by

Juan Manuel Garrido

Co-founder, Vantegrate

Co-founder of Vantegrate and founder of EGA Futura (1994). He has spent more than 30 years building business software for Latin America and has been a Salesforce partner since 2009. He writes about CRM, business processes and the Salesforce and Oracle ecosystems.

150+ integrations

SalesforceMercado PagoOpenpayPaywayOracleServiceNowSlackSAPStripeFiservSalesforce Marketing CloudMicrosoft Dynamics 365HubSpotWhatsApp BusinessSalesforceMercado PagoOpenpayPaywayOracleServiceNowSlackSAPStripeFiservSalesforce Marketing CloudMicrosoft Dynamics 365HubSpotWhatsApp Business
How many sales slipped away while you read this?

Every minute a WhatsApp message goes unanswered is an opportunity moving to your competitor. Book a demo and find out how much Sellium can sell for your company.

Book a demo

Francisco Morales, co-founder of VantegrateFrancisco Morales, co-founder, takes your call. We reply on WhatsApp within 4 business hours, no strings attached.

Salesforce
ISV Partner
AppExchange Partner
Salesforce · Since 2009
Oracle
OCI Partner
Marketplace & OCI Partner
Oracle Cloud Infrastructure
Share