Prevention-first QA

Chloé Koers-Bourrat It's cheaper to prevent issues than to apologize for them. So if we're able to actually review ahead of time everything that is currently wrong with our support process, we can work on that, we can improve it. And maybe it will cost some time and some money to actually do that, but it will save us a lot in the future.

Jen Weaver Welcome back to Live Chat with Jen Weaver, I'm so glad you're here.

Today I'm sitting down with Chloé Koers-Bourrat, the mastermind who turned a fast-growing ad tech support team into a 97-plus CSAT powerhouse.

We'll unpack how her tiny QA squad reviews just a handful of chats each week yet drives product change, why "QA to save the day" is more than just her motto, and how she keeps a very human touch even while letting AI handle the heavy lifting.

We also dig into what it was like for her team when the support QA tool Klaus eventually stopped working for them after it was acquired by Zendesk. That's something I've wondered about for a really long time.

So whether you're launching quality from scratch or levelling up an existing program, I hope this podcast is full of practical ideas that you can actually steal and use.

Before we get started though, our QA tool Supportman is what makes this podcast possible. Supportman sends real-time QA from Intercom to Slack with daily threads, weekly charts, and done-for-you AI-powered conversation evaluations. It makes it so much easier to QA Intercom conversations right where your team is already spending their day, in Slack.

All right, on to today's episode. We're going to talk about QA, but first — I heard that you have a really unconventional path to support. Can you tell us a little bit more about that?

Chloé Thank you for having me.

I started as a customer success manager, and after almost five years overall in customer success manager positions — we had this amazing value, that everybody does support. So as a customer success manager we would have to be in support, handle customer chats. And I kind of fell in love with it.

I really enjoyed solving customers' issues and understanding the root cause, and how we could get there so that it wouldn't happen again.

At the time the support operations team lead was a French colleague, and I talked to him almost every week about different support topics. He said, I really want to have you in my team. And I said, well then, I'm going to join.

So here I am, four years later — joined right after COVID — and enjoying it very, very much.

Jen And your role is support operations, is that right?

Chloé Support operations is a team that I'm part of, and I'm mainly focusing on QA tasks, and then internal processes between support operations and all the other teams within our company.

Jen We're huge proponents of QA, because that's what Supportman, our tool, does. So I love that we get to dig into that further. I just recently did a talk at the Support Driven Expo about QA, so this follows on that really nicely.

But before we get into your work with quality, I would love to hear — what does a week in your life look like?

Chloé I would say the number one focus, other than QA, is being in support. We still have support shifts. I am still in support handling chat, facing customers, and working on our internal processes through the support chat.

So I have three shifts of four hours every week that I need to be in and handle our customers' cases. During those support shifts I usually also train new joiners and help them get onboarded within the support role overall.

That takes let's say three mornings out of my week. Outside of that, I'm focusing on QA review, and I'm actually right now working on an AI project for QA. That's a big project that is taking a lot of my time.

In terms of meetings, I have a lot of meetings with product managers, but also engineering leads, to make sure that the support process overall is good for them — how we are escalating cases, how we're sending feedback, what can be improved, and specific topics that we need to focus on, or roadmaps, to understand exactly what is about to come for our customers. So we can also be prepared within our team to train our support agents, or to get all the documentation ready.

And then internal meetings as well with the rest of the team, because we are a global team. We have people based in — well, I can't give you all the locations — but we have Manila, Singapore, Dubai, Helsinki, Berlin. I'm based in Madrid, and then we have New York, Chicago, Guatemala, and we also have somebody in San Francisco.

So we're everywhere, which helps us cover 24/7 support. But I'm juggling different working hours so I can really talk to everybody and make the most out of it.

Jen You mentioned to me that QA helps you identify broken processes. That might be a good place to start.

Chloé One of the main processes I actually started working on when I joined the support operations team was technical escalation.

As I said before, we had this value in the company that everybody does support. We would have engineers sitting in support with us and just handling some chats.

The problem is that within our platform we have so many features and so many parts of the tool that not all engineers are familiar with everything. They each have their special feature that they're responsible for. So we would have a chat coming in about, let's say, reporting, and the engineer sitting in support would be about creative. So they wouldn't even know how to answer those requests, and the escalation wasn't very quick and efficient.

So the first thing that we did is actually take engineers out of support, and see how we could reach out to each individual team better so that the escalation process would be a bit more seamless.

We started with what we call the support ticket, and we integrated the Jira process within our escalation process. We had it already for our bug reports, but now it was what we call the support ticket, coming directly from support escalations.

We revamped all of this so that whenever they get a ping about a support ticket that has been created for this specific feature, they know that it's for them. That was time-saving for engineers, not having to be in support — so it was a huge saving of money — but also we're able to better track exactly the issue that was actually happening.

Our engineers are only located in EMEA working time zones, so mainly in Helsinki and Berlin. But because our support is 24/7, our teams in the US or in APAC have technical issues that they need to escalate, and whenever we did that before, we didn't always have engineers to actually help us because they were not always in support.

So now we are able to just create a support ticket, and whenever they get online the next working day they just go into their support ticket board and check exactly what was created during the night. They're able to work on that investigation and the issue that was reported, and whenever they leave comments it just gets back to the Americas. So we can also do a follow-up from the EMEA time zone.

That gives us real time tracking and task tracking, for them and for our teams, to make sure we're giving the customers all the information based on the escalation that has been done.

Jen That sounds like a really great system. And I wonder if the engineers are almost relieved that they don't have to be in support any more.

Chloé Oh yeah. Every time we have new engineers, we tell them a little bit about support, how it works, and how the engineering organization is involved. And one question that comes back, I would say at every session, is: do I need to be talking to customers and be customer-facing? Because I don't want to.

And I'm like, no worries, it's not going to happen, we're going to manage that for you. I don't mind handling the communication style towards our customers, and leaving the engineers to just solve the issues so they don't happen again.

Jen As a support person, I find that really validating — that customer support is a skill that not everyone has or wants to develop. And as support people we've maybe been developing it for years, and sometimes it feels like it's not even a thing. But it really is a thing that sometimes other people are even afraid to deal with.

I know you have a lot of other cross-functional wins from your QA system, but I wonder if first we can go back and talk a little bit about the history of QA on your team.

Chloé The support organization overall has existed since the company was founded 12 years ago. When we started QA we had 20, 22 agents, something like that. Now we're up to 37, 38.

We started experimenting with QA in 2021, but based on the scale that we were at and the size of our team and the number of customers that we had, from a leadership perspective it was still not the right time to actually go into QA. We were still reviewing from time to time, but not very thoroughly and not with a specific detailed process and category.

And then in 2022 my current team lead told me, we need to scale this — because we have been still for a long time in terms of CSAT and we have never gone above 97. We were always around 96.5, 97, but never going above. And we were starting to see from leadership that it was better if we could increase that a little bit more towards 98 than remaining at 97.

So we started prospecting QA platforms to see how we could do it, because internally we were dealing with spreadsheets. We started using Klaus at the time, before it was acquired by Zendesk QA. And we were reviewing between 100 and 120 chats per week.

It was very interesting at first, because when we were able to pick chats based on specific characteristics that we were filtering in the Klaus platform, we really started to see some patterns that we could improve very easily.

For example, what we call visual aid. Whenever we're passing on information to our customers on a step-by-step basis — what you need to do — we were sometimes forgetting to include a screenshot of where they could find a specific button, or a screen recording, small GIFs, something like a one-, two-, three-second snippet. It doesn't have to be too long, just to illustrate what we're telling them.

And by just doing that, we started seeing customers were a little bit more responsive.

Especially because we had a few people in the team that got the bad habit of just sending a link to the knowledge base. Like, you want information? Yeah, you can find it in here.

And then we made the reflection: if customers are reaching out to us through the support chat, it's because they want to talk to somebody. They don't want to go through a knowledge base article and just read something that they could find there obviously, but nobody's telling them exactly what to do, where to put it.

So we started changing things like the visual aid and the step-by-step guide that we're sending. Also reviewing the welcoming messages, to have something a bit standardized — not something super friendly, or over-strict, very cold.

Because we have a live chat with our customers we still wanted to be friendly, but not too friendly, not going like "yo." We wanted to be just, we're here to help, we're like an extension of your team, and we're here to guide you to how you can solve the issue and help you out through the process.

So tone and welcoming messages were also something that we worked out very easily. But I would say the biggest change was the visual aid part, definitely.

Jen How did you identify that problem?

Chloé At first I started picking the chats that got negative ratings, to understand where the frustration from the customers was coming from. Whether it was something about the support that we provided, or more something about the answers that we gave them, or the solution that was not correct.

And we started seeing that actually the solutions that we're providing were correct, were quite on point — but the way we're communicating it was very cold, and just like "do this, do that," and not always showing them where they have to go to actually solve it.

Jen So you started out without a particular tool and you were using your common sense to get some big wins. Did you see that CSAT change? You said it was like 96, 97.

Chloé In six months' time after implementing Klaus, we already saw an increase — even if it was just a few percentage points, we already saw that, because on a monthly basis we were never below 97.2, 97.3. Which was already good.

You say, just 0.2, 0.3 points, not that much. But at least we were making a change. And now for the past two years we've never gone below 97.5. Or if we have, it was a specific month where we had a lot of new integrations and releases coming from external APIs, so we had a lot of frustration coming from the customer.

But we have also had a peak at 98.9, which is just crazy. We were never expecting that. Year to date we have been at 97.6, and we haven't gone down — it's only increasing.

Jen You mentioned Klaus and how wonderful that was for your team. I also really loved using Klaus. Can you tell us a little bit more about how long you used it and what that was like?

Chloé So we had — and I have to be very honest, I don't want to call them out publicly — but we had an amazing onboarding team when we joined Klaus. I had two people, one was based in Malaga in Spain and the other one in Amsterdam, and we had weekly calls. It was amazing.

They really guided me through the platform. They helped me create all the different categories and scorecards that we had, gave me so many ideas on what we could review and how we could really use the tool and take the most out of it.

So our onboarding with Klaus was just amazing, and we had, I would say, one really good year.

And then, unfortunately — and maybe it's only my point of view — Klaus was acquired by Zendesk, and the follow-up and the way that our company was handled from Zendesk was not the same. The experience was a bit different, but also the tool started to lack a few things that were very interesting, a few features starting to go away, which was not optimal for us.

So this year we decided — we're still in contract, but we decided that we're going to shut down the QA program with them, because we're also internalizing all of it. And because internally we're pushing for a lot of AI, we're going to see if there's a way that we can use AI to automate our QA process.

Jen So how does your QA process work?

Chloé When we were using Klaus we were reviewing between 100 and 120 chats per week. On a monthly basis, roughly 400 chats more or less. And obviously that rate went down when we stopped using the platform.

What we prioritize mainly is obviously the negative reviews that we're getting from customers. I'm very lucky, to be honest, in that whenever I get a negative rating and it's during EMEA working time zone, I can just jump in the support chat and talk to the customer. Hey, I'm part of the QA team, I reviewed the chat.

I actually had a case like this this morning, so it's very fresh. I went over the chat and said, okay, I see that my colleague actually offered you a solution, was very thorough in explaining it. So I'm trying to understand why it was a neutral rating — three out of five — and what we could do better.

And the customer just bluntly told me, it's not about the person who helped me in support, but more about the platform, because it's a limitation on our end. Like, good job for the support team, but it's definitely something we need to improve on our end.

So that was also an opportunity to pass on feedback to our product team.

But going back to choosing a chat — negative ratings is the first chat that we're going to review. And then what I'm trying to do personally right now is pick some of our recent joiners in the team and just go over their chats. I want to see if there's a way that we could standardize a few things — the visual aid, tone, the communication whenever they're escalating things to engineers, what they're doing, if they're missing information.

So I'm picking those chats first. I'm not reviewing more than 10, maybe 20 chats per week. The drop is huge, and we're aware of this. But we're really focusing on, instead of having a quantity of chats, just really quality.

And whenever we review a chat, we try to be as fair as possible, and the feedback that we're going to pass is constructive, it's actionable. I'm going to grab you by the hand, you're going to sit with me, and we're going to go over the chat together and see if there's something that we can improve so that next time we have a similar case it doesn't happen again. But really coming from a constructive place, and not blaming, like, hey, you did this badly. Not our intention at all.

Jen You mentioned your product feedback loop, and that it's helping you to uncover product issues during QA. Do you have tips for other teams on how to create that feedback loop, how support can work with product teams?

Chloé Something that has been a bit difficult for us to do was actually to create a good relationship with the product team. Because the product managers are still doing support nowadays — they're still being in support, so they can also see the platform and what customers are telling them.

But because the product team has always worked very closely with engineers, which is expected — from the support side we've never reached out and said, hey, maybe we can also help. Whenever we needed to file feedback we just forgot about it. Like, hey, yeah, we tell the customer, we file the feedback to our product team, they'll be in touch. And that was it.

It was actually raised in a conversation a few months ago by some of our customers: I gave feedback about this a few weeks ago, so do you know if it was taken into account? Do you know if the product manager read it?

Jen Customers want to know.

Chloé Oh yeah, they want to know. And I completely understand, I would want to know as well. This feature could be super important, I'm not the only one asking for it, so why isn't it available yet? Which is totally fair.

So we started reaching out to our product managers and building a strong relationship with them. We know that you're in support, you're always seeing everything that support is doing and all the feedback that we're filing.

And sometimes we're filing feedback out of a gap in knowledge from our end, because there's something that was done in the tool that actually answers that specific feedback, but maybe we're not informed about it, or maybe the knowledge base doesn't have enough information about it.

So we're always asking for visibility on roadmaps and new features. Even if it's just one small button at the top right corner that is going to help clone something super easily.

Jen That sounds like a really great relationship between product and support.

You mentioned revamping how knowledge is shared internally, and that's really related to QA — it goes back to training and your knowledge base. Do you have a sense of what doesn't work as far as knowledge related to QA?

Chloé Not sharing it.

It can be funny, but it's something that we've been going through for the past year. Because our platform is so wide and we have so many features, we're starting to get some support agents specialized in only specific features and specific tools.

I was in support last week, and it was a very funny case, but a customer came in asking about something and I was reading it and I was like, I've never heard about this before. I was actually writing in the notes of the chat that some of my colleagues were checking, like, I've never heard about this before, can we actually do that? And then somebody jumped in and said, yeah, we can do this for this specific platform.

So there are so many places where the knowledge has been shared that we don't even know about, and we're not gathering it.

So something that we did internally was create a specific channel for all the knowledge shares that we are gathering from support sessions. Every time we have a support session we just share something new that we've learned — because maybe it's not going to be something new for others, but it will be something new for a colleague that is in Guatemala that I'm not going to talk to in the next few weeks because of time zones.

Jen Back to QA — do you do peer-to-peer reviews? Do you find that's absolutely essential?

Chloé Yeah, that was actually one of the features that was removed by Zendesk, and we were kind of sad to see it go, because it was something we were really using a lot. It really helped us improve, especially from the support agent perspective.

We know — and it's going to sound rough, maybe, from how I'm going to say it — but we know that when feedback comes from a support lead or a team lead in general, it can be perceived as really rough. Like, yeah, it's my manager telling me I'm doing a bad job. And then you feel bad about it, and then you really pay attention to it next time, and then you seek approval and just make sure that your manager is seeing that you're making an effort and taking into account what they're telling you. Which obviously is completely understandable and completely logical, I abide by that.

But when it comes from a peer — somebody that you've been working with either in the same office or in another office, but that you talk to on a weekly, monthly basis — it's more of a friendly chat. Like, hey, I saw this chat that you had and I was reviewing this, and personally maybe I would have done this a bit differently. Because if you actually had sent a screenshot to the customer, you would have avoided like four or five messages and the answer would have been there in the first place.

I would say in eight out of 10 cases, whenever we have peer-to-peer review, the agents are always telling me, I work better when I see the review from my peers, because I know that they're in the same situation as me. And if the case was reversed it would have happened the same way — I would have given exactly the same feedback.

So we can really see that it has improved on our end, and support agents are more happy to just have a session and say, okay, I'm actually going to do support with another person sitting next to me and we're going to review the support chats that we're working on together.

So I'm just going to open up my laptop, start working on my chats, they're going to work on their chats, and I have a question or a doubt at some point — not about the issue itself, but more about how would you say that to a customer? We have a difficult customer, or it's a difficult answer you need to give them, and you don't always know how to do it.

So peer-to-peer for us has been huge, and it has also helped build closer relationships between our support agents, especially when we had new joiners in the team.

Jen It's kind of like mentorship. Do all your specialists offer QA, or is that something they train into?

Chloé They all train into it, and it's not mandatory for them to actually do it. Some of them really want to give reviews to their peers, some of them just want to receive it, and some of them just don't want to do it — they want to receive the review directly from me or from some of their support leads.

So I would say it's mostly on a case by case basis. Obviously when we have a negative rating from our customers we're always going to give the review, and it's always going to come from the support lead level. But for the peer-to-peer it's more on a wanting basis. If they're willing to get the review and to also give it, then let's just do it.

So right now, even when we have a negative rating, we have people just jumping in to see how the conversation was handled and participating in the review, on top of what the support leads are saying.

Jen It sounds like you're really willing to expand or adapt the QA program to what specialists need — who wants to offer QA, and what they need.

So I have to wrap up. I have a number of quick questions for you. What's something you wish that more executives knew about QA, but most of them don't know or don't care about?

Chloé One thing that comes to mind is, one of our co-founders always told us: it's better to ask for forgiveness than for permission. And I think we can apply exactly the same, especially towards the customer whenever we want to do the right thing by them.

And I think we can apply that to QA — which is, it's cheaper to prevent issues than to apologize for them. So if we're able to review ahead of time everything that is currently wrong with our support process, we can work on that, we can improve it. And maybe it will cost some time and some money to do that, but it will save us a lot in the future.

Jen Good point, I like that.

Automation and AI are definitely growing. Do you have thoughts about how QA can make sure, as we automate more things, that we keep the human touch?

Chloé We're using AI in our support process currently, and some of our customers are not super happy to be in touch with a bot and would rather be in touch with a human. So they're always asking, human please, human please, because they want to talk to somebody. Which I completely understand.

And it's true that the tone and the empathy that maybe a human will put into it will not be done by AI. But because it's a tech-heavy environment on our end, and our platform is very much tech heavy, it's always good to have that human touch.

So yes, we can definitely use AI — and we're actually doing QA on all the chats that are being handled by AI, to make sure that the answers being provided are correct and accurate and we're guiding the users towards the right resources.

But we always have that human touch, even if it's AI handling it. There's a human behind it just reviewing it and saying, yes, this is correct, no, this is incorrect, we need to improve this. So we shouldn't lose our human touch even if we have AI in the support process.

Jen That's a really good sentiment. What's one thing you would never do again when it comes to QA?

Chloé Not having it.

Jen Not having it. Which you did for a long time.

Chloé Yeah, we did for a long time. And I felt really bad at first about why we didn't put the process in before.

Because even if we've been a very fast-growing company, and we've gone from a smallish number of customers to a really big number of customers — I don't even know how many customers we have right now — our team size scaled so much, and we could have made a lot of changes internally to our processes and to the way we handle support way before we did. Even before COVID maybe, if we actually had the right tools and were talking to the right people about it.

So yeah — implement QA as soon as you can in a support process, even if it's just through spreadsheets. Keep that process in mind and just implement it, because it's going to save a lot of time and a lot of money.

Jen Good point. Last question — if your QA process had a tagline or a motto, what would it be?

Chloé For me it's: QA to save the day.

Jen I love that. Thank you so much for being here and sharing your insights. I'm excited for other teams to get QA going the same way that you have.

Chloé Thank you for having me, that was amazing. I had fun.

Jen Oh good, I'm glad.

That's a wrap on our deep dive into QA with Chloé. If you're ready to level up your own quality program, here's our quick start checklist from Chloé.

First, lead with the tough stuff. Do negative CSAT first and get it out of the way.

Two, trade quantity for quality. Like our other guests have mentioned, 10 to 20 thoughtful reviews beats 100 drive-bys.

Three, make feedback peer-led, to soften the sting and speed adoption. Peer-to-peer QA is a huge win.

Four, add screenshots to every answer and watch your handle time drop.

Five, escalate through a dedicated Jira board so engineers see only what matters to them.

And finally, six, let Slack nudges keep tickets moving after day three. If a ticket gets stale, nudge your team in Slack — but definitely automate that.

So I hope you try this. If you do, even just for a week, let me know how it goes, because I'd like to see if launching a program like this has an effect on your team. See you next time.