This afternoon I made the statement, “I have never had an exec ask a data question for which I didn’t need to ask a clarifying follow-up question.”

To the dozens of executives I’ve worked with over the last 12 years: please don’t take this as a criticism! Your job is to ask the questions; my job is to know the dozens of different ways to interpret and answer that question with your data, and help you understand which of them are actually useful to you. As far as I’m concerned, this is all as it should be.

I made this statement in the context of Snowflake Intelligence, Snowflake’s new agentic platform. Snowflake Intelligence provides answers from your data using natural language, with a semantic layer sitting in the midst to help interpret your question and translate it into SQL. To be clear, I am very excited about Snowflake Intelligence, and the opportunities it presents. I’ve experienced a lot of features that will help bridge the gap between you and your data, and give Intelligence an advantage over other AI tools, e.g. a RAG powered chatbot. Among these advantages are:

• The “golden queries” that a developer provides for the agent
• The sample questions a developer provides for business users
• The ability to input synonyms for each field, getting you from “opinc_less_opex” to “Net Operating Income”
• The ability to auto-generate a draft of the semantic model so you aren’t starting from zero
• The visibility into steps and transparent statements of assumption provided as part of the result set

So what keeps me from recommending that every exec who has the good fortune of being a Snowflake customer uses Intelligence for all of their questions? In other words…

What Should Guide Your Decision About When to Use AI?

First: Cost and Reproducibility

If you repeatedly view key performance indicators with clear definitions and goals visualized in a certain way — it’s inefficient to use Snowflake Intelligence. You’ll definitely still want a reporting solution for those repeated and consistent questions. It’s more cost-effective, produces more consistent results, and ensures everyone asking the same questions gets the same result.

Second: The Human Touch

Let’s go back to my original statement. Say an exec says “I want to know what percentage of subscribers had high-priority tickets in North America last year”. This may sound like a fairly clear request. It includes a metric, with a specified unit of measure, a region, a timeframe, and a restriction.

Based on this question, I might ask things like:

• Do you want this for last calendar year or last fiscal year?
• Should we base the date on ticket creation date or ticket close date?
  • If they say create date, I’ll follow up to make sure they understand the implications of doing reporting with lagging results.
• Shall we include free trial users who never converted to paid users in our subscriber totals?
• Does North American mean super_region “US and Canada”, or country_name in (‘US’,’Canada’)? (For a memorable former client, the first included Puerto Rico and the second did not. And nobody realized this for years.)
• Is assignment of a subscriber’s region based on customer’s billing address or mailing address? Or the version of the app they signed up with? Or the version of the app they used last? Or the version of the app they’ve used most frequently? Or the version of the privacy policy they signed? Or the country of the IP address they used at signup?

Hopefully some of these questions won’t be necessary because they will have already been answered — one of my favorite stages of any DAS42 project is when I get to help our clients’ senior leadership align on terminology and standard metrics and definitions across the organization.

Sometimes our prospects are skeptical of the value of this work. “Oh, we already have standard metric definitions,” they’ll say. Then, time and time again, we find that:

• They don’t understand the nuances of data capture — what a null value can indicate, how test records are identified, or when in the process a “high priority” status is conferred upon a ticket
• The marketing team has a definition of “subscribers” — but it’s not quite the same as the sales team or customer success teams’ definitions
• Metrics are being created using start dates, create dates, etc, meaning that if something changes upon close, the metric can change retroactively — and your books can never close

These are not problems that today’s AI is yet equipped to resolve, and this is where a tool like Snowflake Intelligence is best used in partnership with smart, savvy AI engineers.

Back to our example executive, asking about US and Canadian subscribers with high-priority tickets opened in the last year. Snowflake Intelligence doesn’t currently ask for clarifications; it proceeds with assumptions. Say that it returns a result that shows the previous complete calendar year. Our exec notices this, and decides she wants something different, because it’s already October and January 2024 was a long time ago. “Please return a result for the last 12 complete months,” she says. And it does! Amazing, she thinks.

She looks at the assumptions that can be expanded from the carat near the top of her answer, and:

• She sees that it’s using the ticket creation date, but she doesn’t know the nuances of when a ticket is assigned its “high priority” status, and she hasn’t been warned that the results will change retroactively as more tickets opened last month continue to close
• She sees that it interprets North America as billing address super_region “US and Canada”, but she doesn’t know what the alternatives are
• It doesn’t mention the nuance with inclusion vs exclusion of trialists

She brings her results to the next exec team meeting. Then, hearing her conclusion and realizing it doesn’t jive with their own experience, a second exec asks, “How many paid subs did we have in the last 12 complete months in the US or Canada who had a ticket that was closed as high priority? What percentage of all subs is this?”

This sounds like almost the same question, right? But let’s think about a few ways that result will differ from the first:

• It excludes free trialists (used the term “paid subs”)
• It specifies the ticket was closed as high priority vs opened
• It does not include Puerto Rico
• It is unclear whether the denominator includes any or all of the limitations of the first part of the clause. Is it truly all subs? All paid subs? All paid subs in the US and Canada? All paid subs in the US and Canada with activity in the last 12 months? All paid subs in the US and Canada who opened a ticket in the last 12 months?

Like many previous advancements, this is a powerful tool to unlock exploration of your data. But also like its predecessors, it can be dangerous if used naively. If you’re taking business actions based on the results – run them by your engineering team.

Or call DAS42. We’re happy to take a look and help you understand the things you don’t know that you don’t know about your data.