hero image

Ep. 48 - The Philosophies of Data Science in Klaviyo with Michael Lawson and Woody Austin

Michael Lawson and Woody Austin are on the Data Science team at Klaviyo, the leading growth marketing platform chosen by more than 20,000 online businesses.

In This Conversation We Discuss:

  • [1:10] Michael and Woody’s time before Klaviyo
  • [1:41] The guys’ journey to eCommerce
  • [2:49] What made Michael and Woody fascinated with Klaviyo?
  • [4:30] The guys’ roles at Klaviyo
  • [5:16] Klaviyo’s Smart Send Time feature
  • [7:43] Sponsor: Simplr simplr.ai/honest
  • [8:33] Is Smart Send Time going to require more effort to set up?
  • [9:50] Klaviyo’s “predicted gender” feature
  • [12:21] Underutilized features of Klaviyo
  • [14:20] How do the underused features work on Klaviyo?
  • [15:11] Sponsor: Gorgias gorgias.link/honest
  • [16:00] Klaviyo’s data science philosophy that the guys liked
  • [17:37] Is Klaviyo considered AI?
  • [19:51] Tools are just for help, you still have to make an effort.
  • [21:13] More new features from Klaviyo: SMS and Customer Analytics
  • [19:51] Klaviyo’s philosophy of Owned Marketing and Owned Growth

Resources:

If you’re enjoying the show, we’d love it if you left Honest Ecommerce a review on Apple Podcasts. It makes a huge impact on the success of the podcast, and we love reading every one of your reviews!

 

Transcript:

 

Woody Austin

All of our customers really should own their data, own their relationship with their customers.

 

Chase Clymer

Welcome to Honest eCommerce, where we're dedicated to cutting through the BS and finding actionable advice for online store owners.

 

I'm your host Chase Clymer, and I believe running an online business does not have to be complicated or a guessing game.

 

If you're struggling with scaling your sales, Electric Eye is here to help. To apply to work with us visit electriceye.io/connect to learn more. Now let's get on with the show.

 

Alright, everybody, welcome back to an episode that I've been looking forward to for a very long time because I keep getting dodged by my partners over there.

 

But finally, they came through I had to track them down at KIaviyo Boston the other day, so I finally got some commitments.

 

Welcome to the show, Woody and welcome to the show, Michael from the data science team at Klaviyo, my favorite email marketing platform What's up, guys?

 

Michael Lawson

Hey! How are you doing?

 

Chase Clymer

Awesome. Cool.

 

Woody Austin

Excited to be here.

 

Chase Clymer

Oh, yeah. And I'm excited to have some actual scientists that I'm talking to on the show. That's pretty cool.

 

Michael Lawson

Yeah.

 

Chase Clymer

Awesome. Alright, so what are you guys doing before you ended up at Klaviyo?

 

Woody Austin

Yeah. I'm Woody. Before Klaviyo, I was in grad school for the last six years at UT Austin for computer science and I was specifically working on high-performance computing and machine learning algorithm development in recommendation systems.

 

Alex Ikonn

Yeah, this is Michael. I was also in grad school. Also for six years. I was at the University of North Carolina, doing an algorithm development and experimental design mostly in medical research.

 

Chase Clymer

That's so cool. So what made you guys get into the eCommerce space? What appealed about Klaviyo?

 

Woody Austin

Well, I was kind of looking for a classic computer science job and I thought that eCommerce was kind of a nice place to go because you get to work with so many customers who really see the product of your work.

 

And after being in that ivory tower for so long, I was really wanting to be able to interact with real people and develop something real. I found Klaviyo because I moved up here to Boston to be with my girlfriend who's going to school here. And I was kind of looking around for jobs.

 

And when I stumbled upon Klaviyo I just had fun interviewing and the culture was really great. We got to mess around with the product a little bit. And I was surprised by just how nice it was to use.

 

Michael Lawson

It's somewhat similar for me. I was looking for a little bit of a change after being in grad school. And one of the things I liked the most about Klaviyo was the pace. Things happen at a very quick pace. Which can... That was a nice change after.

 

There will be times that I'd have to wait to start on a project for three months because I couldn't get access to a data set in grad school because health data is just --for a very good reason-- protected so strongly.

 

Chase Clymer

Absolutely. So before you guys... As you guys were joining the team, what were some of the features of Klaviyo that really stuck out to you and be like, "Wow! This product is way smarter than people think."

 

Michael Lawson

Hmm, there's a lot. It's tricky. I think the thing that really stuck out the most to me was being able to automate in an intelligent way. So to set up a flow and say, "Only people who have bought a certain type of product should receive this type of experience."

 

That's exactly the sort of personalization that was at the core of what I studied in grad school and medical research. And it was interesting to see a lot of the same ideas getting applied to guide customers through their experience with your brand.

 

Woody Austin

Yeah, I was really impressed with the segmentation engine at Klaviyo. It's dynamic and you can get really specific and it's fast. I was also really surprised --coming from the machine learning world-- just how much powerful personalization you can get out of Klaviyo without even diving into the data science features to begin with. So I thought the tool is just laid out really nicely. And yeah.

 

Chase Clymer

Oh, yeah. It is amazing. We used a few different platforms for clients previously and then once we found Klaviyo, we're like, "Oh, man, this thing has it and it's easy to use." I think that's the best part. With some of the more legacy automation engines out there --Infusionsoft is one that comes to mind-- it's kind of scary to get in there and understand how it works. With Klaviyo, it's very user-friendly.

 

Woody Austin

Yeah, we try to make it that way. (laughs)

 

Michael Lawson

(laughs) Mm-hmm. One of our goals.

 

Chase Clymer

So what are you guys responsible for at Klaviyo? What's your day-to-day?

 

Woody Austin

So I guess my official title is machine learning engineer. On the data science team, we are involved in the entire development of the product. So we actually sit down and act as our own product managers. We get to decide what the direction of each of our groups does.

 

I'm working specifically on product recommendations. The recommendation systems for populating the product feed and everybody's emails and trying to make them be as good as possible for the users that get them eventually.

 

Michael Lawson

Yeah, I'm working on improving the experiment experience in Klaviyo. So if you want to run an AB test in any area of our software, we wanted to make it easy to use and as powerful as possible. We're working on improving that.

 

Chase Clymer

That's so cool. So there are some few features that you guys shared with me in these notes. Let's talk about the first one here. I'm sure that some of the listeners that have Klaviyo aren't even using this. Smart Send Time. What is it? How does it work? And why should I be using it?

 

Michael Lawson

So Smart Send Time, I guess, as the name indicates... The first thing I'd say about it is that it's smart. (laughs) It is a way to intelligently target the time that you send your emails to when your customers will actually want to receive and open them. The way that we do it is a little different from how kind of the industry-standard approach has been. The industry-standard approach has been pretty much (like), "Look at historic data, see when your customers have opened their emails and go with that time."

 

There are some inherent issues with that. It brings in some survivor biases. And because of that, Christina, on our team, --who developed this algorithm-- came up with an approach where, "Well, this is an eCommerce platform, let's actually test what works best. Let's do science to figure out the right time to send your emails."

 

And what we do is we test all the 24 hours of the day against each other, so that you can actually see if 7:00 PM is the best. It'll outperform everything. It won't just outperform 6:00 PM. It won't just outperform 3:00 PM. It will outperform all.

 

Woody Austin

A couple of other things that I think are really cool about this algorithm is, Christina actually started out by trying to do what the rest of the industry did. And she got the pretty graphs that everybody has showing what your open... What your send time should be, what the open rate should be. And then she just found out that in practice, we didn't get any lift at all from those. On open rates or click-through rates or anything like that.

 

So she went back to the drawing board and came up with something completely different that actually does provide a lift to our customers.

 

Michael Lawson

In terms of "Why should we use it?” I guess the proof is pretty much in what's happened after it's been out there in the world. Smart Send Time, was released...

 

Woody Austin

A little over a month ago, I think?

 

Michael Lawson

...a little over a month ago, I believe, in web release. And since then, over 1000 customers have used it and they've seen a median 8% lift in open rates.

 

Woody Austin

Yeah, and the median there is really important because it means that we're not letting huge outliers drag us one way or another. So when people report the average lip, that isn't quite as true or as telling what the median does.

 

Chase Clymer

That's awesome.

 

Simplr Ad

Support for today's podcast comes from our friends at Simplr: a new way to staff 24/7 sales and customer service on your eCommerce store.

 

It works with your existing email and chat platforms, so setup is quick and easy.

 

Simplr's network of on-demand, US-based, Simplr specialists are standing by to answer your customers' most common questions.

 

Set it up for free today and then turn it on or off depending on your customer volume. You only pay $2.25 for every resolution. There are no hidden fees, contracts or minimums.

 

Close more sales with Simplr by staffing your email and live chat around the clock with Simplr specialists. Start your free seven-day trial at simplr.ai/honest.

 

Chase Clymer

So when I'm going to send a new email blast, first of all, I'm going to be split testing my subject line for sure.

 

Woody and Michael

Mm-hmm.

 

Chase Clymer

But is there any extra work that I need to do to use this Smart Send Time as well?

 

Woody Austin

So yeah, it's pretty easy. Any other users could have used Klaviyo before, all you do is you go in and you select your scheduled send time, but now there's an option that I... I forgot what it's actually called in the UI? It's like Spread Send Times...

 

Michael Lawson

Uhm, Exploratory Send. When you're discovering your specific Smart Send Time, you choose Exploratory Send. That's when they do the scattershot across all 24 hours. And then once you've found your best send time, you can just choose Focused Send at that time.

 

And then it also splits a few emails into a few hours before and a few hours after so that if characteristics of your audience change, --Maybe your optimal send time was 7 pm at a certain point, but now it's 5 pm because your customer base has shifted their preferences a little bit-- the algorithm could still pick up on that.

 

Woody Austin

Yeah, and that's really cool. At Klaviyo, we're trying to make things really easy to use. That means, we're still exploring for you so if there's something that does change, you don't have to worry about it. You can kind of set it and forget it.

 

Chase Clymer

That's super cool. Another really cool feature that has just come out recently is "predicted gender." How does that work? That's just so cool to me.

 

Woody Austin

Yeah, so it's actually pretty straightforward. We just look at the first name and then correlate that with the way that names are distributed across the entire US. So actually, it's kind of funny because on the data science team, we have a wall where every single member of our team, we printed out our Wolfram Alpha entries for our names, and you can see the distribution for our names. But we use that same kind of thing.

 

We just look at the distribution across the US and we say, "Are you likely male? Are you likely female? Or are we not sure?" And so that allows you to segment your audience into those three segments. And you can target either male and unknown, or female and unknown, or just male, just female. And it really helps you target your message towards your audience and make them feel like you're really talking directly to them.

 

Michael Lawson

And one thing that's nice about this feature being in Klaviyo is, you get everything that's already there. So if you are, at some point in your process, asking people for their gender, you can also have that override our predictions.

 

Because if someone says they're male, you know. So you can just build that. You can set that up in your segmentation logic as well which is a nice drag-along feature that you just get for free.

 

Woody Austin

Yeah. And you can add other segments that you're already using. So if you want to only send to your VIP customers who are likely male or who have identified as male, then you also can do that,

 

Michael Lawson

In terms of how to use it, there are definitely some best practices. I think there are some very obvious "Don'ts" which... It's almost not worth even spelling enough. Obviously, don't use any data stereotypes that's going to turn off your audience. You don't want to do that.

 

Don't assume too hard either. I mean, I personally... My name is Michael. It's a very common male name. It's one of the most common. But I've met female Michaels. If you just assume because the name is likely male and you call them... If you say, sir, in your salutation in the email, that might be going too far.

 

What we've seen some customers do is just try to personalize some of the content and make it more relatable. So certain colors that are chosen or the choice of, "Do you use a male or female model in a picture to try to get them to see themselves in the picture?" Things like that.

 

Chase Clymer

Oh yeah, that's that's some great advice. So while it is predictive, it's still... A computer is just giving its best guess. (laughs) So I wouldn't... I hadn't thought about it that hard. So, what are what are some of the other data science features that you think people are underusing on the platform right now?

 

Woody Austin

I'm a little bit biased since I work directly on the recommendation systems but when we look at the analytics for all of the emails that are sent across Klaviyo, very few people are actually using Personalized Product Recommendations in their emails, or at least through Klaviyo anyway. So, I would say that one is really underutilized.

 

I know a lot of people, whenever they're sending campaigns, are going to want to tell a story so it makes sense that they wouldn't necessarily use personalized recommendations there. But within flows, if it's like an abandoned cart flow, then we can send them products that would be personalized for them.

 

Michael Lawson

Yeah, that's definitely a good one. Another one that I think flies under the radar a little bit is Expected Date of Next Purchase. And I think a really good illustration of that is, people tend to buy in their own case for certain types of products.

 

So you might have a customer that, very regularly buy, something from you every three months. And if you set up your Customer Winback Flow to email them after two months, you're pressuring them and they won't like that. If instead, you know that they've predicted the date of the next order is three months from now, then you can wait until that point, give them a little extra time, make sure that they're actually... Maybe they were on vacation.

 

Being able to actually use something predictive rather than just a rule that you came up with --maybe in a few seconds while you were setting up the flow-- I think that's also a very powerful one.

 

Woody Austin

And also super related to that is the Churn Risk as well. So you can also predict how likely your customers are to churn. And you don't really need to badger somebody with the Customer Winback Flow if they're highly unlikely to churn but maybe whenever they're getting more into that orange or red category.

 

Chase Clymer

And now within my customer settings in Klaviyo, am I setting up automations that trigger when people get into certain segments or is it more of a, "You got to think a little bit more outside the box of how you're going to target these people." How does that work?

 

Michael Lawson

Over expected date of next order, for instance, that's actually a setting in your flow. So that's a trigger that can trigger a flow. It's when someone reaches their expected date of next order, send them through this automated sequence.

 

Woody Austin

I think for things like Customer Lifetime Value and Churn Risk and that kind of thing, you can segment on some of those properties within Klaviyo. And then, I guess, for product recommendations, that one should be pretty straightforward. Whenever you're creating your template and you go to the product block, you can click "Use Personalized Recommendations."

 

Chase Clymer

That's awesome.

 

Gorgias Ad

Let's be honest today. All of your customers are going to have questions.

 

What are you doing to manage all those questions? Do you have a helpdesk for your business?

 

One of our sponsors of today's episode is Gorgias. Gorgias is the number onerated helpdesk for eCommerce. It integrates seamlessly with Shopify. We have installed it in a bunch of stores. It's also used by brands like MVMT and Rothy's.

 

What it does is it takes all of your customer insights and information, brings it into one amazing dashboard so you can solve their problem as quickly as possible. If you want to give Gorgias a try, visit gorgias.link/honest to get your second month free.

 

Chase Clymer

So I know that you guys are big fans of the product. Does Klaviyo have any particular philosophy towards data science that you guys are happy about?

 

Michael Lawson

Yes, and yes, I guess, is the short answer. So one of the things that... I guess one of the big takeaways from Klaviyo Data Science is we actually are doing science here. As we talked about Smart Send Time, we didn't just settle for something that would work okay or something that is done by many people.

 

We found the approach that actually had demonstrated value and actually helped people. And that's a big part. The rigor that goes into making sure that our methods work.

 

Woody Austin

Yeah. I'm kind of piggybacking on that a little bit. We really hire good people. Like, Michael has a Ph.D. in statistics. Christina got very far into grad school and her degree in statistics and then I studied machine learning for a really long time. But you know, even though we aren't the bosses, our bosses really listened to us and they want to get the math right as well.

 

So, they aren't just going to put out some subpar feature just because the industry wants it, we're going to make sure whatever we put out actually works. Another part of our philosophy that I really like is that we try to focus on automating the parts of marketing that humans aren't good at.

Humans are really good at coming up with a narrative. Humans are really good at talking to each other and communicating with each other. So we want to try to make it easier for our customers to reach their audience.

 

Reach their audience in a way that makes them feel like they're being talked directly to, not just being sprayed with like a mass email. And so I really like working on this part of the platform as well.

 

Chase Clymer

Absolutely. Now, machine learning is like a really big buzzword as of late, along with AI or... There's a bunch of buzzwords in that same vein. Where do you think Klaviyo lands as far as the technology itself? Is it actually artificial intelligence or what is it? I'm curious about your guys' take would be on that, coming from a scientific perspective.

 

Woody Austin

Yeah. We approach it in an ad hoc manner, depending on what project we're working on. So we want to have really rigorous statistics on a lot of our projects and so those tend to be more on the pure statistics side of things. Or I guess you could call it data science. For things like machine learning or... Sorry.

 

For recommendation systems, those typically fall more in the “machine learning side” of things. And specifically for a recommendation system that's really hard to perform statistics on because you're showing somebody a product and you aren't sure whether they're going to like it or not.

 

And so your historical data, just like with Smart Send time, really doesn't matter at all. So you have to really throw the kitchen sink on it, but do it in a principled way. And yeah, I also had one more comment about the previous question on our philosophy I kind of forgot to throw in there as well. But now I'm blanking on that too. Sorry.

 

Chase Clymer

(laughs)

 

Woody Austin

I'll come back to that. I'm sorry. (laughs)

 

Michael Lawson

I guess following up on that thread, I think one of the things about, I guess, calling the platform AI... First of all, I personally think that the term AI is overused in many places. And it's a very buzzword thing right now. And I think many things are called AI that maybe shouldn't be.

 

But in terms of Klaviyo as AI, I would say, in some ways maybe (and) in some ways, no. Again, the platform is about making human connections. So the AI parts --and this is going to sound very similar to what he said-- are about doing the things that are hard for a human to know, like, "When should I actually send my email? When should I know to stop my AB test?" Because I've reached a point where I actually know what the result is. And it's really enabling humans to use their intelligence more effectively.

 

Chase Clymer

Absolutely. And I just want to follow up on that. Signing up for Klaviyo or signing up for whatever software of the month is buzzing isn't going to make or break your business. You know what I mean?

 

If you have a good email strategy, and you're coming over from a less feature-rich platform, (then) yeah. I think Klaviyo is going to do it for you and do it well, and probably make you more money. But you still got to do the work. You still got to write these flows. You still got to write these campaigns. The machines aren't doing all the work. They're just doing the really, really technical stuff.

 

Woody Austin

Yeah, I think one of the best examples that I have (is) a machine is not going to be writing your emails better than your marketing team or you yourself can write your emails but...

 

Michael Lawson

Or at least if it is, that's in the... That's a little science fiction, and it'll be a while, right? (laughs)

 

Woody Austin

At least if you want to do it well. Even if you... Even if the machine could generate it for you, --because there are algorithms that can do that-- you're gonna have to go in and you're gonna have to edit it and it's probably not going to be the right tone for you.

 

But something that I'm sure everybody would love to be able to do is to know every single one of their friend's customers and really like to target the information to them. And so we can find the groups of customers that are really similar, that have kind of obvious characteristics where they might like a particular product from you. So that's the kind of stuff that we're trying to start with automation.

 

Chase Clymer

That's awesome. And now, is there anything else that you guys see it's coming out on the horizon here shortly that is going to get you guys excited or features that are going to come into Klaviyo? I know SMS dropped at your event in Boston the other day. Was there anything else that you guys can share?

 

Michael Lawson

Yeah. Actually, both of the big announcements at Klaviyo Boston were... I'm super excited to work on them from the data science side. So first of all, there's SMS. I think that opens up a bunch of really interesting questions.

 

I think AB testing there is really important because the space constraints are so strong in SMS and really finding ways to squeeze something out of every single character is going to be crucial. And also it opens up the question of, "Should I be sending an SMS or should I be sending an email and what's the best way to even know that." So there's a lot of just channel testing, like, "What channels should I be using for this message?" That is going to be really cool to work on, I think, and then customer analytics.

 

Customer analytics is going to be... I think that's going to be huge because it just gives... It's a feature that allows humans to learn something that it is hard for them to know. For instance, "How are people who came to me from this particular channel doing?" Like, "What are they buying? How often are they buying?" It makes it possible to just understand things like that.

 

And I think when you're able to answer questions like that, you can much, much more effectively use the data science features that we already have and think of. We've already got some that were thinking of that are going to piggyback off of that as well.

 

Woody Austin

Yeah. And it's nice as data scientists, too because we also have access to this new report builder and analytics tool so we're going to be able to leverage the data from our customers and our customers' customers even more.

 

Michael Lawson

Yeah.

 

Chase Clymer

That's fantastic. I can't thank you guys enough for taking the time to hang out with me today and get into the more technical side of --which I'm not joking when I say this-- one of my favorite apps to use in the ecosystem right now. So is there anything that you want to say before we get off here?

 

Woody Austin

I did remember my other point that I was going to make earlier (laughs)...

 

...about the philosophy that we have. It kind of goes along with the people who went to Klaviyo Boston. We introduced this idea of Owned Marketing and Owned Growth and how all of our customers really should own their data, own their relationship with their customers. And that really extends into the data science side of things as well. So we aren't sharing data between our customers.

 

We're trying to build the best models that we possibly can. But those models kind of belong to our customers, our customers' data belongs to only that customer. And so we try to keep the idea of Owned Marketing throughout the entire company and this really shows the culture out here.

 

Chase Clymer

Nice.

 

Michael Lawson

I guess I'd like to say, keep your eyes peeled. I think there's some really cool stuff coming from data science in the near horizon.

 

Chase Clymer

Awesome. Thank you guys so much for the time.

 

Michael Lawson

Yeah, thank you.

 

Woody Austin

Thank you.

 

Chase Clymer

I cannot thank our guests enough for coming on the show and sharing their journey and knowledge with us today. We've got a lot to think about and potentially add to our businesses. Links and more information will be available in the show notes as well.

 

If anything in this podcast resonated with you and your business, feel free to reach out and learn more at electriceye.io/connect. Also, make sure you subscribe and leave an amazing review. Thank you!