Amanda Drws Hey, I'm not here to catch your mistakes. I'm trying to get customers to stop yelling at you. That's my only goal — for you to have as smooth of a day as possible, with as few customers who are angry and stressed out.
And this is how it gets done. Because all of these errors, you're not treating it as "the agent made a mistake." It's, how do I set up systems that make it impossible for this to happen?
Jen Weaver Hey friends, welcome to Live Chat with Jen Weaver.
Today I'm chatting with Mandy Drws, an absolute expert in QA and a support leader who treats quality assurance less like a compliance drill and more like a calming force for anxious teams.
She argues that QA over 10% of tickets is really just a signal, not a solution. Instead, she focuses on amplifying the QA wins — what's discovered during your QA process — to many different teams at your company. She walks us through her weighted, point-based rubric that dings a stray typo lightly but flags systemic errors loudly.
Even better, she shows how a lean QA loop can surface trends like sudden spikes in certain types of questions — specifically menopause and sleep connections, we'll get into that — before any dashboard knows how to track them.
So grab your headphones. Let's learn how less QA can drive more insight, more safety, and more trust on your team.
Before we get started though, our QA tool Supportman is what makes this podcast possible. Supportman sends real-time QA from Intercom to Slack with daily threads, weekly charts, and done-for-you AI-powered conversation evaluations. It makes it so much easier to QA Intercom conversations right where your team is already spending their day, in Slack.
All right, on to today's episode.
We're here with Amanda Drws, who is close to my heart, a QA expert. I've been so inspired by Amanda's work.
We're doing a feature — not really new any more — about a week in the life. And I love that you can give us a perspective on a week in the life of a support leader in your previous role.
Amanda Sure. At my last role I was the head of support for a consumer tech product, a wellness device. It was definitely pretty chaotic — I think anybody who works at a startup knows just how different week to week can go.
But at the same time, there were some things that were pretty regular. In a normal week, Monday was always kind of dedicated to what went wrong over the weekend and getting caught up.
Tuesdays and Wednesdays were typically the days where I was meeting with other departments, giving them updates about what we had found in the previous week, and hearing their questions for the next week. And also meeting with my team to just touch base and make sure that everybody knew what was coming down the pipeline.
And then Thursday was always my QA and CSAT overview day. I would meet with my quality assurance specialist, I would talk to all of my supervisors, and we would all sit down and say, okay, how is QA going? How is CSAT going? What are we seeing in social media? And what questions did the other departments have that maybe we can focus on answering in the next week?
Jen I know we jumped the gun a little bit with the week in the life. For folks who don't know you, could you share a little bit about your previous role? You were the senior manager and head of support, and I remember you telling me you grew the team quite a bit during your time.
Amanda At my most recent role we had a crazy backlog of tickets. We were building this department, and also at the same time finding out what even were the technical issues that our customers were running into. What were the issues that we were running into with e-commerce?
This was June of 2020. So USPS was losing like half of our international orders. What do we do about that?
Jen As you grew that team, you also worked on QA quite a bit and became pretty opinionated. What do you think the most important part of a QA program is?
Amanda My impression of QA, from conversations with other people, is that a lot of it is really just stress soothing. People are anxious. It's an anxiety tax.
Jen An anxiety tax. I love that.
Amanda And it's not about necessarily improving training or improving our messaging or anything like that. It's really just a case of, people are worried that there's something going wrong in the background and they want to soothe that anxiety to make sure there's nothing going wrong. But they're leaving it at that.
Jen Because the question that they're trying to answer is, is there something going wrong in the background — which they're anxious about. And so if I just give myself the illusion that I'm doing due diligence here.
Amanda And then there's also not really a set metric or understanding of, well, what does it mean if things are going wrong? What's an acceptable error rate? They don't go into QA trying to think about that or trying to answer that question, and so it never gets answered.
Jen You have to know what questions you want to answer as you go into it.
Amanda Exactly. And they have to be pretty specific, otherwise you're not going to end up answering them.
So when I go into QA, a lot of the things that I'm thinking about are: how good is my training? And is this new hire that is getting more QA than the rest — are they really internalizing and understanding the training, or do I need to make a change?
Things like, we just had this new big feature release. Do my agents understand that feature? Did I fully adequately prep them for this big feature launch, or no?
Jen And if you're asking that question in QA, it's kind of a last catch of whether that worked — but really you should be answering that question in training.
Amanda Exactly. That's the last calibration to say, yes, I got it right.
And then the other part of that is you can't just say yes or no. You have to say, okay, if the answer is no, what do I do next time so that we are not catching it after we have already made this mistake?
Jen Right. If you just go, well, we caught this and the specialist just needs to do better — then that doesn't really solve the systemic problem.
Amanda Exactly.
Additionally, one of the things that I've been seeing a lot is companies talking about being customer obsessed. That's something that gets thrown around a lot. But then I also have to ask, okay, well, how is your QA supporting that?
Because there are plenty of opportunities in QA to not only be finding out what is going wrong, but also what's going really well — and making sure that one customer who has this amazing experience, that's maybe not being captured in your data collection or not being captured in CSAT, is still being distributed to the rest of the team, to marketing, to product, to give a better understanding of what makes a truly exceptional experience for a customer.
Jen A lot of teams are doing a small number of QA conversations, a small percentage of their total conversations. And even if you do a large percentage, you're not guaranteed to catch every issue.
Amanda Exactly. I'm definitely of the opinion that QA should be like 10% or less of your tickets.
It should be really a spot check of confirming anything new is working well. It should be a little bit of a check to make sure that old ideas and old concepts haven't gotten rusty. And there should be that last quarter or so of QA that's like, okay, what's working well? What happened in these really good CSATs?
I love when my agents flag conversations that say, hey, this went really well. I want to brag about it — put it in a brag box. And that's when QA goes through and can give real specifics of, this is what went so well.
Jen So you say that QA isn't just for support. That's one of the things that kind of blew my mind about your perspective — that support QA has implications for every team. Can you break that down? What are some of the teams that really should be utilizing QA data?
Amanda If marketing is doing A/B testing, they know based on click rates, based on open rates, which of their A/B worked better. But they don't necessarily know why. And that's data that support can provide — they can say, this is what resonated.
But there will be times where there are concepts or things that really resonate with customers, where either it's not easy to collect automatically in any kind of sidebar, or we don't know that we should be collecting it.
That's where QA comes in. Working for this wellness company, a big concept was sleep. So we did have some general data collection happening around, is this helping people sleep, yes or no?
But it was QA that came to us and said, hey, I'm seeing a lot more questions around sleep for women in menopause.
Why would we collect that data? We wouldn't know to collect that data. But QA was the one who said, I'm looking at all of these general tickets, and here's a trend that I'm seeing.
And that means I can go to marketing and say, hey, now that I'm looking for this, menopause is being mentioned in like 10% of our tickets around sleep — just doing a word search for menopause.
And then marketing can turn around and say, okay, let's have a webinar about this, let's write some articles about this. And then we ended up having like a 500-person webinar talking exclusively about how to manage stress and sleep for women in menopause.
Jen That blows my mind. Because what you've done there is — marketing, maybe that's not even your ICP, that age group of women particularly — but marketing has learned from customer support interactions what actual customers are really interested in. Which means potential customers would be interested in that. And that's something you can't really get from anywhere else.
Amanda Exactly. They weren't our ICP. They're not even our main purchasing demographic. But they are a main demographic of people reaching out to support.
So here's this whole segment of our market that was not being utilized, and whose questions were not being answered, needs not being met. It was great. Super interesting too.
Jen That's a great example. I know we also talked about the support training team a little bit, and how they can benefit from QA.
A lot of times people go from training to graduating their onboarding, and then that data is not returned to the training team. What kind of a program would you recommend for getting that information back to the training team to iterate?
Amanda My trainer was part of the QA meetings. It's not just about experience or practicality, it's also getting that data and getting that feedback. And training doesn't ever stop.
It's really important to make sure that when you're going through your training — hopefully your training is good, you're looking at example tickets, you're looking at example questions — that doesn't need to stop when training ends. It's really helpful for agents to see how other agents are answering those questions, how they're handling de-escalation, how they're handling issues where they don't know how to answer it.
And when you set up this rolling system where your trainer gets to see how these new hires are doing in QA and CSAT, and roll that into training, you can start doing interesting things like listening sessions.
Every other week we had a CS-only listening session, and that was for calls that went really well. And again, QA would be supplying very specific reasons of what exactly went really well, and what exactly did this agent pick up on that made that experience better.
It also gave agents an opportunity — we set out bonuses. We were like, hey, did you have a call that went a little wonky and you don't know why? We'll give you a bonus if you share it with the team and you describe what you're not sure about.
Jen That's great, I love that. It incentivizes something that is usually disincentivized, which is being vulnerable in public about something I'm not sure I did well.
Amanda Exactly. I always say low stakes, high quality. Because when you say things like, this went wrong, and the stakes are really low, then you can bring the quality of everybody up.
One of the things I was always trying to tell my agents is, we don't all need to have the same bad phone call. We don't all need to make the same mistake. One person could do it once and we could learn from that, and that would be fine.
So we had these listening sessions. Even the ones where the conversations went well — you open the floor and you say, hey, have you had a call that went like this? What worked well for you? This worked really well in this situation — when would it not work? What are other times?
And then once a month we also did whole-company listening sessions. We gave bonuses for those too, because the agents were horrified. They were like, what are you talking about, the whole company is going to listen to my call? And we're like, this is a call where we think you did so great though.
Jen "We don't care, we hate this."
Amanda Yeah.
Jen I totally understand that. But on the other hand, it's the customer information that the rest of the company really needs. So often in support we have this real closeness to the customer, and it's hard to get that across to other teams.
Amanda And there's certainly something about — you can tell people, oh, this problem is a real pain point for us. But then if you hear a customer on the phone with their voice breaking because they are so stressed out, it is a different experience.
And when you hear an agent picking up on these clues and these undertones, it also does a lot for really hammering home the level of skill, experience, and professionalism that customer support agents have to bring.
I think it's really considered an entry-level job. It can be really hard to convince other departments that this is an exceptionally skilled role and a highly technical role. So having that opportunity for these agents to really show off the level of things that they are picking up on — that other departments don't even hear in that first listen to the call — I think that's a fabulous opportunity for support to really show its value.
Jen We think of it as just another day, but often other teams are blown away by the ability we have to deal with issues. It's a complex job.
So with QA and data collection, are there interesting trends or feedback that you want to talk through?
Amanda I think that ops runs into the same issue as QA. Okay, but what are you doing with that data? Are you just collecting it because it's there and you feel like you should be collecting it, and you have all this data around your customer to feel good about it — but what are you doing with it?
Are you customer obsessed, or are you customer good enough?
Jen A lot of teams would be grateful to shoot for customer good enough, unfortunately. But a lot of teams talk about being customer obsessed.
QA is not just the data, and it's not even just those projects where you prove value to marketing. It's also a culture tool. When you have a culture of ongoing learning as a team instead of just coaching mistakes — how does QA support that? How can you structure a QA program so that it does that?
Amanda One of the other commentaries that I've seen around QA and AI in particular is, using AI means that QA is no longer a popularity contest.
And that immediately put my hackles up, because I was like, absolutely not. You have learned something very valuable there — which is that you have a department problem, not a QA or a tool problem.
Certainly QA should not be a popularity contest. If it is, I want to know. I don't want AI to fix it.
It's the idea of, well, do you need to fix QA, or do you need to fix your department or your customer journey as a whole? These are very different things. Very band-aid, treat the symptoms, versus treat the root cause.
And QA can be a system that is set up as support — particularly if you go into it with the assumption that your agents are professionals, that they are the experts in their field, and your QA is there not to catch their mistakes but to align their expertise with your training. And to make sure that when things are going particularly well, when an agent does go a little bit off script and a customer loves it, that everybody does find that out, that that's celebrated.
Like I said, when there is something behind these hard numbers that really points to the humanity and the soft skills that come in with support that are so valuable — that's something that QA can be doing.
And then it turns into this much more open and supportive culture, where agents are less concerned about making errors. They're a lot more open when the idea is, hey, I'm not here to catch your mistakes, I'm trying to get customers to stop yelling at you. That's my only goal — for you to have as smooth of a day as possible, with as few customers who are angry and stressed out.
And this is how it gets done. Because all of these errors, you're not treating it as "the agent made a mistake." It's, how do I set up systems that make it impossible for this to happen?
Jen So QA really needs to ally itself with the customer support reps, so that it's on their side, not here to nitpick at them.
Amanda Yes, please.
Jen I like what you've shared with me about your rubric — a very beautiful spreadsheet, that is amazing, that I was just like, can I have that?
One thing about it is, sometimes what can make QA feel adversarial is if I make one typo, then I get dinged for that. And on the reverse, sometimes it can miss serious errors that do happen only one time, but it was about billing or it was really difficult.
So what's your solution for keeping QA on our side — here for me as a specialist, but making sure it finds the right problems and not just randomly some things?
Amanda I have looked at some QA tools, and my process is certainly that of a startup with no money. We had Google Docs, and that was what we could afford when we were first building this QA template.
Jen I think a lot of teams are in that position for sure. QA is scrappy more often than not for small teams, I think.
Amanda Yes. So we built this template — instead of 100% and ticking down, what we did was this weighted QA with points that added up, and we made it pretty granular.
There are things that, like you said — a typo. I don't care about a typo. One typo is not a big deal unless it's coming up in every single ticket you QA. Some typos are a one-point error. It's not a big deal unless it's trending across a lot of tickets.
Other things, like accidentally leaving in a macro insert that you forgot to edit out — maybe you weight that more as a four.
Jen I have done that and it's so painful.
Amanda Who among us has not done this? I always tell my agents, if you haven't done it, don't worry, you will soon. Not a big deal unless you're doing it all the time.
And then there might also be things that you would consider either auto-fails or much more significant errors that you want to weight a little heavier.
It also gives you the option to decide, okay, what's your hard cut-off of how many errors can occur in a ticket before you get properly concerned? And if you're also tracking, are they happening multiple times over the course of the month, are they happening in any particular type of ticket — then you can get this really good understanding of exactly how your team is functioning.
Jen For someone listening who's on a team that maybe does have a QA program, but it's really just scorecards and compliance — what would you say is one thing they can do to really make QA have more impact than just testing whether a rep is doing a good job?
Amanda Particularly with the fact that we are all on Slack all the time — or whatever we use, so that's my example — if you want to show value, don't wait to be asked.
And I would say don't necessarily put it in an email, but put it in a Slack message. Maybe if you're part of the marketing general Slack channel or something like that. Find those channels, if they're public join them, and just say, hey, here is something that I found, here is something that might be useful for you.
And if you are consistently doing that, if you are consistently paying attention to the questions that they're asking in their meetings, in their Slack channels, and being able to come back the next week and say, our QA found this answer — even if they weren't asking you.
You have to just kind of announce it. That's one of the things that we did with these whole-company listening sessions. Nobody asked for it, we were just — we think that this would be useful for you.
And sometimes, our first meeting, I think one person from another department showed up. But we recorded it, we went through it, we posted a list of some of the findings from that call and said, here's what we found, here's some things that might be useful, here's the recording.
And the next month, a lot more people showed up.
Jen I love that, that's a huge win.
Is there anything else you wish more leaders understood about the real impact of QA?
Amanda I think that QA is treated very much as optional. Particularly with more experienced leaders, there can be this idea that because of their experience they have this innate sense of what's going on and why.
I think that's a trap that I can raise my hand and say I have fallen into as well — and in the process of doing data analytics and QA got humbled.
But that's really what I wish that they understood. Investment in QA, and well-done QA, will save you so much time and money in the long run. Maybe this month it does not impact your profits. Maybe this quarter it's net neutral. But as time goes on, if you really do want to be customer obsessed, you have to invest in QA. You have to invest in data analytics.
Jen Yeah, it's necessary.
Amanda It truly is. And assuming you're doing it right, it will pay you back tenfold.
Jen I can't thank you enough for being on the podcast. You're a delight to chat with.
Amanda Thank you, Jen. You're such a great interviewer, I will come back any time you want me.
Jen Thanks, I appreciate that.
Thanks for listening. Before you head out, here's a quick recap of Mandy's proven QA playbook — because small, repeatable moves will help your team more than giant wishlist items.
First, audit under 10% of your tickets. Keep reviews below a tenth of total tickets so QA stays a spotlight, not a bottleneck. This is what Mandy recommends for teams who are doing manual QA.
Two, aim at what's new. Point a spotlight at fresh launches or training rollouts so you can catch problems as they crop up.
Three, weight what matters. A granular points-based rubric lets a missed macro or accuracy problem outweigh something like a stray typo.
Four, automate the basics. Dashboards can track first response, and other easy metrics are also important, so humans can hunt for other insights.
Five, broadcast what's gold. Put findings back to product, marketing, training, so that wins and fixes can scale.
Six, recalibrate your QA often. When your patterns shift — new regular typos, or new stressors for customers — tweak your systems before tickets start to pile up.
If you put just those six habits into action, you'll trade catching mistakes for empowering your specialists to solve problems ahead of time. I hope that's super helpful. I'll see you in the next Live Chat episode.