Craig Stoss AI is incredibly iterative. I think people forget that humans are iterative too. So if you change your model, or you add a new category of ticket, you need to teach your model that.
One of the biggest troubleshooting things we do with our customers is edge cases. So you want a tool that can hand off to a human seamlessly to handle that stuff.
Jen Weaver Welcome back to Live Chat with Jen Weaver, where we break down the exact moves support pros are making to simplify operations and lead with impact.
Today's guest is Craig Stoss, VP of partner solutions at Kodif — and if that sounds familiar, you're right, he's back for a second episode. Today we tackle a deceptively tricky topic: using AI to make your data useful.
Craig has sharp takes on everything from cleaning your data and connecting scattered tools to spotting patterns your team might otherwise miss. Whether you're drowning in untagged tickets or trying to figure out what on earth is even worth automating, Craig is giving you a game plan.
Before we get started though, our QA tool, Supportman, is what makes this podcast possible. Supportman sends real-time QA from Intercom to Slack with daily threads, weekly charts, and done-for-you AI-powered conversation evaluations. It makes it so much easier to QA Intercom conversations right where your team is already spending their day, in Slack.
All right, on to today's episode. So we're talking about using AI to actually utilize your data. Can you tell me a bit more about what you mean by that?
Craig Once your data is in a clean and healthy state, the next thing is to actually use it.
I always joke that data is a lot like an exercise bike — it's only really useful if you use it. I could put an exercise bike beside my bed; if I never get on it, I'm not going to lose any weight. And that's the same with data. Collecting data is easy. It's using it that's the hard part.
I think that in today's world, this topic ultimately has to talk about AI. I'm not saying that AI tools are the only way to gather insights and analytics. There are plenty of tools — there are even free tools online that I've used where, for example, you can upload a whole bunch of text and it builds word clouds. It just scans the document, figures out what words are there, and builds you a word cloud that gives you an idea of what this data is about. That was how I did it five, six years ago.
The value of data is making it work for your business. And there are a few key things you can do to enhance that.
The first, as we talked about in our previous episode, is make sure your data is clean, that it's all apples to apples, that you don't have any legacy data that's going to skew results or confuse any sort of AI model that you might be using.
But the more important thing, I think, is the connectivity and the correlation of data. You likely have data in half a dozen systems at least. Your help desk has data. You probably have some sort of survey tool. You probably have a product tool. If you're a SaaS platform you probably have a tool like Pendo that measures how customers use your product. You might have a CRM that has the contract values of your customers, or metadata about where they are, their demographics, the vertical that they're in. If you are e-commerce you have certainly Shopify and all the metrics that they have around consumers and order values and shipping details.
You have data in all these systems, and that's not abnormal. What is abnormal, and needs to become more normal, is connecting that data.
For example, your CRM data — the customer name, demographic information, location information, certainly vertical if you're in a B2B scenario — has to connect to the help desk. So that if someone has feedback you connect the two. It should connect into your product to say, well, these customers in this vertical are having this product problem and they're complaining about these support issues.
It needs to be connected, and there are multiple ways of doing that. First and foremost, the most common way is to have some sort of business intelligence tool in the background that pulls this data in and organizes it in a meaningful way. Or there are AI tools out there where you feed your credentials for these tools and it will figure it out — you can say here are the unique identifiers between these different tools, and it parses it together.
But that is probably your number one starting point in my opinion: get your data clean, and then correlated.
Jen All these different places that you're using a business intelligence tool to gather this data together — are there any of those tools that you just don't think it's worth connecting and getting insights from? Or that you see support teams use and you just think, this is a waste of time?
Craig The purist of the voice of customer side of me — the guy that loves voice of customer and thinks that customer feedback should be the main driver for a business — wants to say no, no data is irrelevant in this scenario.
I always joke with my team that if you're calling a customer and they sigh — if you say, "Hey Jen, can you please do this thing?" and you go [sighs] — that's feedback. That's showing that, yeah, okay, you've answered my question, but whatever you've told me to do is frustrating to me.
That's really hard to capture. Incredibly hard to capture. AI could probably capture it in some sentiment analysis way, but that's useful feedback. And how do you train a team to capture that and relay it — hey, this thing that I said caused this reaction for a paying customer?
So the purist in me wants to say all feedback is relevant, and the filter stage is post that. Feed it all into whatever your system is, and then afterwards figure out, okay, well, there may be some noise over here that we should eliminate.
The realist in me recognizes that's not necessarily possible. And it depends on the nature of your business.
A great example: I have a friend, years ago, back when AI and sentiment analysis were just becoming a thing — this is well before ChatGPT was out — they scanned Twitter for movie reviews and then aggregated them, kind of like a real-time Rotten Tomatoes. The idea was that as a movie was reviewed, they would categorize the review as positive or negative, and they could give you real-time analysis. Literally you could watch it on a Friday night as a new movie premiered and see the real-time sentiment about it.
But it proved to be very difficult. Because what if your tweet mentions 10 movies — the top 10 movies that you saw this weekend — and they reviewed each of them separately? How do you start to parse that out? And then there's obviously English language, where it's like, oh, that movie was sick. Well, is that good or bad?
And it's only gotten worse, in the sense that the amount of data that comes in is huge. It's petabytes of data being transferred on everything. And if you have a business that has a common name, or is compared to other businesses, it's really hard to filter through — is this actually feedback about me, or is this just some AI-generated article that happens to talk about my stock price?
It becomes very difficult. So the realist in me knows that there is data that's not relevant, even though the purist in me wants to say take it all in.
Jen Do you think AI at this point can effectively filter those things out? You just sighed, like a customer asked you an annoying question.
Craig No, no, it's not an annoying question. It's a great question, it's just a difficult one.
AI in its current form — and by the way, I'm a futurist. I love where AI is going, I work at an AI company because I believe in what we're doing and I think we're doing it in a really ethical and interesting way. But I also have to take a step back sometimes and realize what AI is not.
The current form of AI is really just really, really awesome pattern recognition. Even when it generates an image, it doesn't know that it's generating a picture of a parrot. It just knows that when something called a parrot is inserted into my system, there's some red here, there's some yellow here, there's some white over here, there's this poofy thing over here that's blue. It has no clue. It's just a pattern.
There's a really good YouTube video from a creator called CGP Grey — he uses B's and threes. He says, if I give an AI millions and millions of examples of what a B looks like and millions and millions of examples of what a three looks like, it's going to be pretty good at telling the difference between B's and threes. But then if I give it an upside down three, or a backwards three, and that's the only one I give it, it's not going to say that's a three. Even though a human will obviously say, well, that's just an upside down three.
It's a really good video. I'd highlight it for anyone who wants to learn a bit more about how AI works — I think it's a good starter video.
So to take all that back to your question: if there is a pattern that you can get from this content — if you have one troll that every day posts something stupid about your company just to be a troll, and that pattern is something you can recognize and teach the AI — ignore this pattern, or ignore things that look like this — yeah, AI can do that.
The problem is I don't think most data comes in that form. Think about Google. Google uses some AI — obviously it's proprietary — to spam filter, and they're pretty good at spam filtering. But it's because Google controls so much of the email market. Their AI can see that this exact same email has gone out to millions and millions of their users, and it's like, oh well, that's a pattern, that's unlikely to be a real email. Because when's the last time you sent an email to a million users, Jen? It's just not a use case.
So Google is really good at recognizing that pattern and saying, well, this is probably spam.
So that's the answer to your question. If there's a pattern, you could teach the AI the pattern and account for it. But in general, no — you just sometimes will have these anomalies.
That's the other thing: there's a statistical relevancy too. In theory you should have a long tail of anomalies on either side, and then your bell curve of here's the stuff that you should actually focus on in the middle. So there's also an element of maybe you just chop off the bottom 20% and top 20% and say, okay, this is the real stuff.
Which ironically is actually why data the old way is a problem. Because we know, for example, statistically that people who fill out surveys are either in that top 20% of the happiest people or in the bottom 20% of the angriest people. So you never get that middle opinion. Whereas AI is better at the middle opinion, and therefore in theory it should be more accurate to what the average customer is experiencing.
Jen That makes me think of a data problem that I've often thought about in support, which is we're hearing from the people who are upset about something.
I worked at a company where there was one feature that people really would get hugely upset about, and I would deal with it all day long as a specialist. And my support leader reminded me, well, this is 0.6% of all users. So that middle of the bell curve — what we're hearing about the most may not be the most relevant data for the business. It may be what product teams are bringing in.
Craig That's a really good point. What's interesting about that is it's a volume-based question.
For example, if you have one small bug that annoys a small percentage of your customers, but it's a big impact — however you define impact —
Jen Been there, done that.
Craig How does that weigh in a weighted average against something that is in the middle of that bell curve, that annoys a lot of customers but is lower impact?
At a previous company we actually did a really good job of this. Every bug was categorized exactly by that — categorized by frequency and by impact.
In the auto industry this is common. It's called failure mode analysis or something, I forget the exact acronym. But the idea is that if you have a defect in your brakes that only happens one in a million times, but it kills you — is that more or less important than something that happens one in a thousand times that might break your finger or your arm?
I'm not in any moral or educational way allowed to make a judgment there, but which is it? And which should you prioritize from a fix standpoint? I think that support and product are similar — you need to have that deeper analysis.
And that leads into a good distinction. There are really two types of analytics you can get out of this data.
You can search data for something you already want to search for. For example, if there is a particular area of your product that you are interested in understanding, you can say, AI, surface feedback about this area of the product, and it can find those needles in the haystack and surface that to you.
But I would argue the more interesting use case is the stuff that you're not looking for.
Kodif has a feature where if a spike happens and some new category is discovered, you can set a threshold to notify you. A great example: we had a food company that saw a spike in rotten sweet potatoes being mentioned in emails and chat that had never been mentioned before.
They had a threshold set so that within the day, a notification went to the support manager saying, hey, there's this new topic, we've seen this topic multiple times today, we've never seen it before, something's up.
And the support manager was able to look at those tickets, look at the customers, figure out that all these customers came from the same batch from one supplier, and realized that that supply had some rotten sweet potatoes in it. And then was able to go and say to the support team, hey listen, this was our fault, there's definitely a problem here, this is the wrong batch — give them a refund, or whatever it is.
That's interesting from a data perspective, because not only is that real time, as opposed to waiting till the end of the month to gather some data, but it's something that you may never have caught. Especially if you have a large support team that's remote. I handle the rotten sweet potato ticket today, never thought about it. Person two handles it, never thinks about it. The correlation isn't there to combine that knowledge and say this is actually a trend.
Jen AI can build that trend for you. It's that pattern recognition that you were talking about, on the large scale.
Craig So those are the two types of analytics you can drive from it. And often that tail — where there's one specific issue that isn't in that bubble — that's where that comes from. That analytics where it says, hey, there's something here that doesn't feel right based on the rest of the trends we were aware of. And it could pop that up.
Jen Is there anything else that you think support teams absolutely need to know, especially from the perspective of a support leader who may not have implemented AI yet for their data? What's an entry point?
Craig That's a great question. I've always pitched the crawl, walk, run concept with AI.
It's important to understand — if I were to go buy a VoIP telephone system for my business, and I were to put every logo on a wall, Aircall, RingCentral, Five9, all of them, and throw a dart, I'm going to get a VoIP telephone system. At least 10 years ago this was true. Five, ten years ago this was true — I know Five9 has evolved quite a bit, RingCentral has evolved quite a bit, Dialpad has evolved quite a bit. But say five, ten years ago, throw a dart at them, I was going to get a VoIP telephone system.
AI isn't like that. The tool market is incredibly complex. And not only is it complex from a use case standpoint, it's complex in that two companies that maybe are categorized the same — conversational AI, chatbot AI, agentic AI is that big new term — even two companies that are in that same category, or would result from the same Google search, do things so differently that it could actually change the way the model does or does not impact your business.
I think step one in an AI journey is really clearly document what you're trying to do. And I mean write it down, and make sure that it resonates, and even use numbers to prove that it does.
So are you trying to deflect volume — deflection being a dirty word in support — is that ultimately your goal? Are you trying to increase the efficiency of your human support, so you don't want to deflect volume, but when something gets to your humans you want to make them more efficient? Are you trying to increase personalization?
Maybe you're a luxury brand and you want to make sure that the agent knows not only that you are Jen Weaver, but that your last order was this certain size of shoes, and that you typically order purple clothes, so they can make smarter recommendations to you. In the food and beverage world, maybe it's, are you a vegan, are you gluten-free? Personalization is huge in support.
There are tons of these use cases, and you just need to be very specific with what you want, and then find a product that solves for those specific use cases. Because it's not so much about "I want to buy AI." It's "I want to buy a solution to these specific problems." And you may want to solve all of them and find a tool that can do all of them — they exist. You want to be sure that your tool matches your problem set.
And it's true with analytics as well, because whatever tool you get is going to solve this based on the data and analytics that you provide it. What are your top 50 ticket types? What are the areas of your business that are inefficient that need to be automated? What are the key metadata pieces that will provide that personalization? You need to inform the AI of that. So your data has to be in a way that you could provide that information to the model.
That would be step one, the crawl phase. And then we could probably talk about walk and run a little bit later.
Jen If I've done that and employed an AI tool of some kind, what does it look like for me to determine that it's doing what I need it to do? I've decided on these metrics that I want to change. Does it go wrong sometimes and need adjustment?
Craig AI is incredibly iterative. I think people forget that humans are iterative too.
If a new feature is released in a product, or if you have a new physical product that you're selling as an e-commerce brand, people just don't magically know that exists. It's not like that old movie The Computer Wore Tennis Shoes, where he injected everything into his brain. Or The Matrix, more recently. Humans need to learn things, and in the same way AI tooling needs to learn things.
So if you change your model, or you add a new category of ticket, you need to teach your model that.
I was just speaking to a customer of Kodif this morning, and I said one of the biggest troubleshooting things we do with our customers is edge cases. If you have an automation that works in the 99% case, and then all of a sudden you see one person that came in missing a piece of information, or maybe some anomaly — maybe they moved before the shipping occurred. It's hard to automate those types of things, and so the human would probably need to be involved. So you want a tool that can hand off to a human seamlessly to handle that stuff.
Think about the last time you had a complex situation that you needed to call an airline or a telecom company about that wasn't a standard question. Sometimes you have to go through two or three humans to find the person that actually knows that answer. You will never automate that use case fully. So the key for AI tooling is to make sure it can get to the right person as quickly as possible.
Jen That's impactful. I think that's often forgotten.
Craig When you talk about measuring success, it's certainly before and after comparisons of the number you're trying to impact. So efficiency — that might be handle time. Customer satisfaction, or customer sentiment, might be a measure of personalization. Containment is always a big one when it comes to AI: what are the answers that you're giving, are they accurate, are they solving people's problems?
You want a system that can measure those things and tell you. But you have to understand that this is iterative. This is not a set-it-and-forget-it tool.
You could set up Zendesk and probably never change it for two years and it would still likely do its job, more or less. With AI, you need to have someone that's monitoring in some capacity. I'm not saying it's a full-time job, but you need someone to look at it and assess it.
The better tools will — as with my sweet potato example — notify you when something goes a little bit weird, or maybe an answer wasn't known. It'll provide suggestions saying, here are ways to enhance your system. The better tools will do that. So it's guided, it's less onerous or manual a task, but it's still a task.
Jen I know you need to hop out in a minute, so just one more question. What's a tool or technology that you think every support leader should be looking into now?
Craig I think that in the future, this concept of personalization and the demand for real-time answers are going to be the two biggest trends in CX.
We live in a world where even my six-year-old knows that he gets his TV shows if he clicks on his picture on my Netflix. We live in a world where we get the content we want, we get the products we want. Amazon tells us what we should be buying. Food delivery companies are suggesting orders for you. Fridges are now telling you when you're going to run out of milk.
We live in a world where everything is telling us this is what you want and this is when you want it. That is personalization, and that is real time. And that is coming into any sort of B2C and a lot of B2B businesses.
So I think the technology you need to look into is something that solves those issues as it pertains to your business. That's the number one recommendation I have.
Some of that doesn't have to be AI, let me be very clear. You don't necessarily need AI for personalization — it could be an integration inside your help desk to pull in information. It doesn't require artificial intelligence at all. So just to be clear, I'm not saying that AI is the only way forward. But personalization and real-time access to information.
On that front, I would say tools that are in knowledge management and are able to give you a help center that is real time. Certainly conversational AI can do that, AI agents can do that. When it comes to personalization, that could be integration, that could be AI, that could be any tool that correlates your data set — we talked about correlating your data set. Any tool that correlates your data set is going to help you there.
Those are the types of tools and technologies that I would say every support leader at least needs to know what's in the market today, if not starting to build that strategy. Because this trend is not going away. We are not going to become people that want less personalization and want to wait longer for our answers. That's just not going to happen.
Jen I really appreciate you being here. I know you need to hop out, I don't want to push you right up against your next meeting time. But thank you so much.
Craig I hope this was useful. Thank you for having me.
Jen Thanks again to Craig for being here and sharing his data expertise.
Just to review those steps again. One, clean and connect your data first. Two, define your AI use case — don't just "buy AI." I'm making air quotes with my hands now as I do that. Three, choose a tool that matches your problem specifically. Four, expect iteration and maintain oversight. Five, use AI for known questions and unknown trends. That one, I think, is especially important.
Craig breaks down the realities of working with support data. If you'd like us to touch on data again in the future, please do reach out. Or if this episode sparked an idea or helped you rethink how you're using — or ignoring — your data, share it with a fellow support pro. It would mean a lot to us. Hit that subscribe button, drop us a rating, and join the newsletter, the link is in the description.
And remember, our sponsor Supportman makes this podcast possible. Please check it out — there's a 14-day free trial, and if you're using Intercom and Slack, it's an absolute no-brainer to deal with your QA automatically.
Until next time, keep it practical, keep it human, and give your data something to do.