The ten-ticket method

Stacy Justino Consistent quality review, I think, is better than no quality review. Since we are a small group of people who are carving out time to do this consistently, we decided to go with a cadence of 10 support interactions a month.

Jen Weaver Hey friends, welcome back to Live Chat with Jen Weaver.

It's not often that I get to do a podcast episode with somebody who I admire as a professional in our field, but who also has been a great friend and a mentor to me. So I'm super excited to talk today with Stacy Justino, who's the product support manager at PetDesk.

Stacy arrived last fall and spun up a brand new quality program for their veterinary support in just a few weeks. Now, you know, we're all about QA here at Supportman — that's what we do. So I'm also super excited to unpack her topic: her ten-ticket cadence, the four-point rubric her team actually uses, and the simple Google Sheets trick that keeps her QA reviews fair and random.

Stacy spun up this program to be lightweight so that she could actually keep it going over time with minimal overhead. I think it'll be really helpful for support leaders who maybe don't have a ton of resources for building a QA program. Let's be real, that's probably most of us.

Before we get started though, our QA tool Supportman is what makes this podcast possible. Supportman sends real-time QA from Intercom to Slack with daily threads, weekly charts, and done-for-you AI-powered conversation evaluations. It makes it so much easier to QA Intercom conversations right where your team is already spending their day, in Slack.

All right, on to today's episode.

So let's talk about this new feature we're doing on the podcast — it's sort of like a week in the life of a support leader. What do you do? What are your meetings like? How do you structure your work? What is it like for Stacy in a week at work?

Stacy Starting with things that are multiple-times-a-day sort of things: checking on the chat queue, checking our ticket queues. Is there a ticket that's been sitting there at the top for a while? Let's make sure that somebody gets on that. Oh, we have six people waiting in chat, we have wait times above three minutes — let's post in Slack to say, hey, we need anybody who's not on break, at lunch, or in a meeting to grab a chat.

So that's something that's happening multiple times throughout the day.

Beginning of the week, there's some weekly reporting that needs to get done, so doing that on Monday. In terms of other stuff, there are projects to be done — we call our quarterly projects, or OKRs, "rocks." So making sure that I have time, and I try to book at least a few hours in my calendar every week to make sure I can focus on those things.

Jen And where do you keep that? Is that on paper?

Stacy In a notebook.

Jen Nice. Same, girl. Colored pens?

Stacy And then I have another colored pen where I check off the boxes.

Jen Of course. Colored pens are where it's at.

So you're talking about planning out your week. I'm going to try that out — that simple little just writing down what needs to be done today, and then this week, and with due dates. I love that.

Stacy That's helpful for me. If I need a reminder, like — today, what is on the list to do today that needs to get done?

So Mondays are sort of for weekly reporting. And then I have 30-minute one-on-ones with each of my direct reports each week, sprinkled throughout. I try to put them in blocks of two or three, so they're not too long of blocks, but that I am in that mindset — okay, this is my time one-on-one with the people who report to me. So I really aim to be present and not be distracted during those times.

And then of course we have our team meetings on Tuesdays and Thursdays. I also have a weekly meeting with our senior product support specialists and our product solutions advisor on Tuesdays, as well as a Monday and Wednesday meeting with the support leadership team — with our boss, the senior director of global support, and the support managers of all of the PetVisor and PetDesk support teams.

Jen And the other support teams serve other parts of the business, other tools?

Stacy Yes. We collaborate a bunch because of the way the products interact with each other. We now have a weekly pet communication support and customer success leadership meeting, which has been really great.

Jen Today we're going to take a deep dive into your QA process for your support team. Will you get me started on where you work and what the context for the conversation is?

Stacy I am a product support manager at PetDesk. PetDesk is a company that has multiple products, but I specifically support our PetDesk communications product, which is basically kind of like a CRM for veterinary providers that integrates with their practice management system. And we also have an accompanying app for pet parents.

So a veterinary provider has clients that have booked appointments, and if they have a PetDesk communications subscription we can handle appointment reminders, health service reminders — oh, Pixie is overdue for a rabies vaccine — and we can send those kinds of reminders.

And then the pet parent can see all their appointments in the app, see a lot of other information. If the clinic has it set up, they can earn loyalty points or reorder prescriptions for their pet.

So that's the scope of what my team does. And we hadn't had a quality program before I got here. I started at PetDesk at the end of October 2024, and so we spun one up.

Jen Can you tell me a little bit more about how you got started? It must have been that you saw the absence of a quality program and thought, this is an initiative I can run.

Stacy Yes. We already had some loose quality standards. One of our senior product support specialists — those are the folks who do training and onboarding of new people — David, he would always go over sort of quality. What does quality mean when it comes to PetDesk support, and how do you do that?

So we already had a basis for that, and I was lucky to come into an organization where quality was already emphasized. It wasn't straight up like, answer as many tickets as you can. So that foundation was already there. I wasn't starting from a point of, oh, our quality needs tons and tons of work, we need to put this in place because the gap between where we're at versus where we need to be is pretty big.

That was a good starting place. And I think all of the specialists on the team were eager for more regular feedback. Because the feedback mechanisms up until we launched the quality program were: oh, this ticket got escalated by a CSM, or negative CSAT. Or in the cases where specialists on the team, in their one-on-ones with their manager, would be like, hey, I think I could have done better on this, can we talk about this ticket?

Jen So a little haphazard.

Stacy And not consistent, and very reactive.

Jen So you saw a big opportunity to systematize that. Where did you start?

Stacy I drew on a lot of my previous experiences. When I was at Big Fish Games, I ran the quality team within the customer support team.

Jen So you have some experience.

Stacy Yes. And also at Wistia I spun up a quality program there.

So I was able to take a lot of what I found success with in the past. We decided to start with what the quality guidelines were already in place for the team. What did we cover in training? What did we have in our one Guru card around quality? And rather than totally rework it, I incorporated some of the things that I found pretty important and foundational in a quality rubric.

And then — since the folks doing the reviews are the two managers and three senior product support specialists who, spoiler alert, we all have a lot of responsibilities already on our plate — I really wanted to focus on something that we could do and deliver on consistently.

I know a lot of organizations, especially ones with a dedicated quality team, or that are using software that enables you to do things a little more easily, tend to do a percentage of total interactions. But since we are a small group of people who are carving out time to do this consistently, we decided to go with a cadence of 10 support interactions a month. So two QA reviews each month — five tickets in the first review, five tickets in the second.

So that people get regular feedback, but it's not too cumbersome for the folks doing the reviews.

Jen That makes total sense, because you're trying to initiate this program and get buy-in, and you don't want people to feel overwhelmed and feel like it's an addition to their workflow.

But that brings me to a burning question a lot of people have about QA. How do you choose which conversations? Is it the ones with negative CSAT, or is it random?

Stacy That is a great question. I am in the camp of, it should be random, but with some parameters to make sure that you are selecting tickets that are representative of the work that they do.

In our case we have live channels. So we have folks who do phone support and tickets, and folks that do live chat support and tickets. And the phone volume and the chat volume is a greater percentage than the ticket volume they handle.

So for folks who are working live channels, over the course of the month, for the first QA we will pick three chats or phone calls — depending on which channel they're on — and then two email tickets. That's how we make sure that it's representative of their work over the course of the month.

And then in terms of other things we look at — recency is pretty important. I always want the person whose ticket is being reviewed to be able to remember that ticket.

Jen Of course, that makes sense. If you pick a ticket they handled at the beginning of the month and this is their second QA review, it's not as impactful.

Stacy We're using Zendesk, and so I've created a report in Zendesk. Unfortunately I have to create two reports, one for chats and one for tickets, because of how the data sets work.

But we look at tickets that were solved within the past seven days, and exclude some ticket types — we have a "not for support" ticket type where it's meant for another team, or we're just escalating to the CSM. I haven't gone too far into adding more filters.

And then I make sure that the ticket created date is the day after they were emailed their feedback, so that we're reviewing tickets that they interacted with after their last round of feedback.

Jen So it's not reaching back into the past. You have these sets of time that you're QA'ing.

The filters you're using are essentially — it needs to be a recent conversation, and it needs to be representative of wherever they're working, so email and chat, or email and phone. That makes total sense.

Just to get a handle on what you're working with, what's the typical volume of conversations that your team is handling in an average month?

Stacy A single support specialist on our team is probably doing between 350 to 400. So it's a pretty small percentage that we're looking at over the course of the month. But consistent quality review, I think, is better than no quality review.

Jen Absolutely. And especially it makes sense to me that you're starting this program — you don't have a QA team. So you're being very mindful of the workload of the managers, you said it's team leads and managers who are doing QA.

Stacy Senior product support specialists. They're kind of like a tier two type role.

Jen That makes sense. They've been there a while, they know what a good ticket looks like.

So as far as the people who are doing QA, it's the senior specialists. What about who receives QA? Is it everyone, or is it new people?

Stacy All specialists. We have level one, level two, and then we have our senior product support specialists. So all level one and level two folks get QA.

The way we've mapped it out is, you don't get QA during your training and onboarding. And then we recently decided that for that first month post-onboarding, the QA will be done by the senior product support specialists, because those are the folks that they've been interacting with the most. And they'll also deliver that feedback.

And then the next month, it depends on how we've assigned who's going to QA who for the month — but then the manager will be the one who has that feedback session going forward, because by then they'll already have a couple of one-on-ones under their belt with their manager and have built some rapport before the manager takes over that QA feedback.

Jen So I'm getting a picture of a really gentle process for everyone. It's gentle for the folks who have onboarded — it eases them into it, and then it allows them some time with their manager to gain rapport and get to know them before they start doing QA with their manager.

Stacy Exactly.

Jen And is it the senior specialist who does training, who's doing onboarding of the new team member? So it's an extension of training.

Stacy Exactly. That first month getting folded into the QA process is that extension of training.

Jen So you've just launched this program. If I'm a customer support leader looking ahead to a team that doesn't have a quality program, what would you say is the biggest mistake I'm likely to make that you could help me avoid?

Stacy I think the first thing would be focusing on negative CSAT, because I think that's what people think they should do — but that's not really representative of probably the majority of their work.

I think it's still important to review those. I was actually just having a conversation earlier today with someone about this. If you want to review those against your quality rubric, that's perfectly good and okay. But if their quality score is going to be part of their performance guidelines and performance expectations, that is not fair and equitable — to have your ticket selection criteria be all negative CSAT plus a handful of random ones. Because that's going to tank their score, and like I said, it's not representative.

So to me, one of the keys to success is that the tickets you're reviewing really should be as close as you can get to representative of what they're handling in a given month, on a percentage scale.

I didn't cover this earlier, but one of the other things — since we don't have software to do this, I exported this into Google Sheets. And Google Sheets has a really cool feature where you can randomize a range.

So you just copy all of the rows that you want to do this to, and you right-click, and in the very bottom there's additional options, and it says "randomize range." So I just filter by the one specialist, randomize range, and then click through the tickets top down, because they're in a random order. And then I just look into them.

I figured that out like two months ago when I was first doing this. I was like, there's got to be a way to do this. I was trying to use the RAND function, but that's annoying because it refreshes, so you have to use the random function, copy and paste the value, and then sort. So this was much easier to do.

Jen You copy it into a new sheet so you're working with separate data?

Stacy Yes.

And what I also do is I'll click through those tickets, because — because of certain situations with our software, we get a lot of tickets from providers who are reaching out to support to say, hey, can you update this client's phone number, can you update this client's name?

So that's a big percentage. And that is the one case where it's like, the tickets I'm selecting aren't quite representative. 30% of our tickets are these account modification tickets, but I don't want to QA 30% of their tickets being those, because those are very much "use the macro, update the thing" in our tools.

So I try to make sure that there's a balance of not all of one ticket type, because that's not helpful to them. Still random in terms of still going top down.

But what I'm working through now, having gone through this for a couple of months — let me add to our Guru card about support interaction selection criteria. Because I think one of the things, to your question, is transparency. Being really transparent about the process.

So the cadence, the support interaction selection criteria, all of that stuff is in our Guru card about our quality program, that everybody has access to.

Jen Speaking of transparency — do you feel like it's always a good idea to do QA more privately? If I'm giving a specialist feedback, what are the pros and cons to doing that in a DM or in a meeting, versus maybe sometimes shouting out that the specialist has done a great job and putting that in a team channel?

Stacy I think there are situations where both of those are probably good mechanisms.

The one-on-one session gives them face-to-face time to talk through it. And one of the things that I think is really important is you need to give mechanisms — and your team really needs to feel — that if they don't agree with something, they can bring it up. So that to me is better, and you can establish that rapport in a one-on-one session.

But I think as a team culture, shouting out when people do a really great job is something that should be happening. We have two team huddles every week, and in the first one of the week I always do a CSAT shout-out.

And one of the great things about having a quality program when you do that is you can call out things that are in your quality rubric about what that person did really well. So that's the other thing I think to make it successful — everybody needs to really understand and buy into your rubric.

Jen I love that you mentioned this rubric. I'd like to dive into what your rubric is and how you determined it. Is that something that came from your previous work, or did you tailor it to your new company?

Stacy Both. David, the senior support specialist I mentioned, had been working on a rubric with the previous manager, and so I took some of that, because that was based on the quality training that new hires on our team go through. And then I incorporated some of the elements of the criteria that I've used in the past that I think are pretty foundational. Incorporated those together to come up with what our PetDesk communications team quality criteria are.

So our four criteria are accuracy, completeness, customer excellence, and empathy/tone.

We're dealing with veterinary providers. It's a very high stress job, they're very busy. So empathy and tone might not be a full criterion in some places, but for us it was really important.

There are some things that are universal in terms of a quality support experience. But I would always recommend to someone that one of your criteria should be really rooted in your company values, or what is specific to your company.

So one of the elements of customer excellence is guidance — controlling the communication, guiding to the best options, effective solution, providing additional information to the customer to address their next question or issue based upon the original issue. We put it under this "guidance" heading, because we need to serve as guides sometimes for folks — whereas in other support orgs that I've led it hasn't been as important.

Jen That makes perfect sense. So it's almost the customer profile that guides that rubric.

Do you have any other advice for customer leaders who are working on quality programs?

Stacy One piece of advice is, if you're incorporating the quality score into monthly or quarterly performance for individuals on your team, making sure that whatever benchmark you've set for your internal quality score, and what you're looking for in your rubric, is complementary to your productivity expectations or your workflows, or your business needs.

I can give a real life example of where this happened. When I was at Big Fish Games we had monthly performance, and we had "meets expectations," "exceeds expectations," and "needs improvement." And we had monthly bonuses for our support team.

To earn a level one bonus you had to meet expectations in productivity, quality, and CSAT. And we had a maximum threshold for average number of replies per ticket, to make sure that they're really productive — but if on average it's taking three replies to solve a ticket, we should not be giving a bonus for that behavior.

We had level one and level two criteria for quality. So if you got an 85% to 90% QA score you were meeting expectations; 90% or above, you're exceeding expectations. And then we also had minimum standards, which if you missed one of those, you were automatically at "needs improvement" for quality.

So we basically created this system, with the best of intentions, where somebody can miss a minimum standard on their first QA, they know they're not going to get a bonus — so why would they be incentivized to exceed expectations in productivity? They'll still want to meet expectations, because they want to get a raise in the annual performance review cycle and they don't want to have to go on a performance improvement plan. But they have no incentive to try to exceed expectations on productivity.

And I actually got this feedback surfaced to me by a specialist on the team in a skip level meeting. And I was like, oh wow, this is really great.

Because one of the minimum standards was around selecting the correct ticket fields, like ticket type, because reporting is very important. But we still believed that it was really important, so we actually adjusted it — we folded that into our regular standards rather than a minimum standard.

And then also the fact that if you missed a minimum standard — not all the minimum standards were the same weight, but we weren't going to weight them differently.

So I was like, okay, that's also a good point. Because that's also something like — okay, my first QA I missed a minimum standard, I'm not going to bust my butt to exceed expectations on productivity. And I was like, totally understand that, that's valid.

What we did — we were using MaestroQA there, so I had a little bit more flexibility in terms of getting creative with the scorecard. So I figured out how we should weight a minimum standard. I figured out the math that a minimum standard would deduct 3% from your overall score, because that's what it would come out to be, kind of double what a regular standard miss would.

So somebody could still miss one minimum standard on a QA, but they'd have to really knock it out of the park on everything else — and it wouldn't disqualify them.

That change was really well received, made sure we were incentivizing the right behavior and not shooting ourselves in the foot.

To me that's always a really powerful example of a really good learning, an aha moment. Best of intentions, but in practice it was actually not serving the right purpose.

Jen What really stands out to me about that is that you found this out in a skip level meeting. It really highlights the importance of leadership listening to frontline specialists, and really taking time with intention digging into: what is your working life like? What are these well-intentioned programs that we're creating doing to your daily life, and how does that motivate or not motivate you?

Stacy Exactly.

In terms of cadence — if somebody is fully trained and doing well and they meet their internal quality score, the next month we'll only look at five tickets. So it helps keep the workload manageable, but it also is something that somebody could be working towards.

Jen So it's elastic. You grow the number of tickets as needed, or shrink them as people demonstrate.

Stacy Yeah. And if we had a big fundamental change in the product, or new features, then we would potentially feel like, okay, this month everybody's going to get 10 tickets QA'd, because there's been a lot of change. We made some significant shifts in process and we want to make sure that everybody's moving forward with the new processes. So that's a way to do that.

And then one other thing that I could foresee in the future is, certain months we might do a sprint — where maybe we know that our account modification tickets are always really good, so we're going to focus on technical issues this month. But you're doing that for everyone.

And the thing that you might want to do though is, okay, if it's part of their performance metrics, maybe we say, hey, let's look at how everybody did. And if the team average is way lower, then maybe for this month we adjust what that internal quality score goal is. Because it goes back to being fair and equitable and transparent.

Jen And if everyone is experiencing the same thing, then something's clearly going on.

I love the emphasis on fairness and making it work for your team's workflow as is. I think that's probably why it's been successful.

Thank you so much for being here. I really appreciate you, I have adored working with you, you're one of my favorite people.

Stacy Aw, thanks.

Jen That's Stacy's setup story in a nutshell. Here are the steps that you can take back to your team.

One, ten interactions, two reviews every month. Keep it lightweight.

Two, random, recent, and representative ticket picks.

Three, a four-pillar rubric. Keep it lightweight, make it simple. Hers is accuracy, completeness, customer excellence, and empathy.

Four, private coaching for anything that maybe doesn't go perfectly, plus public kudos for what goes well.

Five, shrink to five tickets if an agent is crushing it, bump back up if things change. So only offering QA where it's really needed.

If an idea landed as you listened to this podcast, or if you're using this process in the future, please share this episode with a fellow support leader and drop us a quick review and a subscribe. Until next time, keep leading with clarity and with care. See you soon.