1 00:00:00,000 --> 00:00:03,630 How much is media contributing relative to customer base is a really nice place 2 00:00:03,630 --> 00:00:04,410 to start. 3 00:00:04,410 --> 00:00:09,240 And the benefit of running incrementality and media mix modeling is 4 00:00:09,240 --> 00:00:12,000 informing the model with some of that causal data. 5 00:00:24,780 --> 00:00:29,040 Well, hello and welcome to another edition of the E-Commerce Evolution podcast. 6 00:00:29,280 --> 00:00:32,700 I'm your host, Brett Curry, CEO of OMG Commerce. 7 00:00:32,700 --> 00:00:37,170 And today we have got a doozy of an episode. 8 00:00:37,170 --> 00:00:42,030 We're talking about the three horsemen of measuring your 9 00:00:42,030 --> 00:00:45,900 marketing effectiveness. We're talking MTAs Multitouch attribution. 10 00:00:45,900 --> 00:00:49,470 We're talking M'S. Media mixed modeling. We're talking incrementality. 11 00:00:49,740 --> 00:00:50,670 It's going to be nerdy, 12 00:00:50,670 --> 00:00:55,530 but I also promise you it's going to be practical and it will make you more 13 00:00:55,740 --> 00:00:58,560 money. And so we'll hopefully make it fun as well. 14 00:00:58,560 --> 00:01:01,740 And so my guest today is Tom Leonard. 15 00:01:01,740 --> 00:01:04,290 We are LinkedIn friends first. 16 00:01:04,290 --> 00:01:09,270 So I saw Tom on LinkedIn posting about incrementality, talking about MMM, 17 00:01:09,600 --> 00:01:13,650 throwing shade on certain tools and stuff like that on LinkedIn. And I'm like, 18 00:01:14,010 --> 00:01:18,480 this is my type of guy. So I reached out, we had a call, and then we're like, 19 00:01:18,480 --> 00:01:20,370 Hey, we got to record a podcast. 20 00:01:20,370 --> 00:01:24,660 Let's create some insights for people on the pod. 21 00:01:25,050 --> 00:01:29,340 And so Tom is a fractional marketing leader. 22 00:01:29,730 --> 00:01:33,630 He's operationalizing MMM and incrementality testing, 23 00:01:33,630 --> 00:01:37,590 and I'm delighted that he's my guest today. So Tom, with that intro, 24 00:01:37,890 --> 00:01:39,270 how's it going? And welcome to the show. 25 00:01:39,780 --> 00:01:42,690 Good. Thanks for having me, Brent. Excited to be here. And yeah, 26 00:01:42,810 --> 00:01:46,730 some of my favorite things to talk through, so excited to do it. Good stuff. 27 00:01:46,830 --> 00:01:49,110 It's good stuff, man. So briefly, 28 00:01:49,110 --> 00:01:51,870 before we dive into the meat of the content here, 29 00:01:52,200 --> 00:01:56,220 what's your background and how did you become a guy who's 30 00:01:56,520 --> 00:01:58,770 operationalizing MMS and incrementality? 31 00:01:58,840 --> 00:02:00,170 Yeah. And what does that even mean? 32 00:02:01,830 --> 00:02:02,690 That's a good point. 33 00:02:02,710 --> 00:02:05,490 For sure. Yeah, totally. Yeah. 34 00:02:05,490 --> 00:02:09,990 So spent most of my career thus far on the agency side at performance agencies. 35 00:02:10,830 --> 00:02:15,240 And I'd say the crux of how I got to where I'm now, 36 00:02:15,240 --> 00:02:18,720 or I've been reflecting back a little bit more on the why I have such a passion 37 00:02:18,720 --> 00:02:22,320 for measurement. And I was at a pretty hardcore DR agency, 38 00:02:23,340 --> 00:02:28,050 and it was right shortly after TRUBY for Action came out when YouTube was 39 00:02:28,050 --> 00:02:30,150 starting to invest in, DR. 40 00:02:31,350 --> 00:02:35,850 Moved into a new role we had created with a centralized group of basically 41 00:02:35,850 --> 00:02:39,750 people who had different areas of subject matter expertise and a few analysts 42 00:02:39,750 --> 00:02:42,920 that ran tests across a pretty large client base. 43 00:02:42,920 --> 00:02:44,940 And I was our YouTube SME, 44 00:02:45,930 --> 00:02:48,600 and worked with a couple analysts to run a bunch of tests. 45 00:02:48,840 --> 00:02:51,540 And really it was to evangelize how to, 46 00:02:51,540 --> 00:02:56,520 and is YouTube a platform to drive growth? And it was really interesting 47 00:02:56,790 --> 00:02:59,800 because I started spending a lot of time on YouTube and then also connect to TV 48 00:02:59,800 --> 00:03:04,150 and broader programmatic video. And it was this really interesting, 49 00:03:04,780 --> 00:03:07,600 for me, the biggest learning was less about how to make YouTube as effective as 50 00:03:07,600 --> 00:03:08,080 possible, 51 00:03:08,080 --> 00:03:12,400 but more how to help brands think about demand creation as opposed to just 52 00:03:12,400 --> 00:03:13,900 demand capture. And frankly, 53 00:03:14,230 --> 00:03:19,090 the difficulty of getting brands to leverage YouTube relative 54 00:03:19,090 --> 00:03:20,170 to connected tv, 55 00:03:20,170 --> 00:03:23,950 because YouTube sat so close to Google ads and therefore last click attribution 56 00:03:24,220 --> 00:03:27,100 and see tv, you couldn't click and was sexier in a deck. 57 00:03:28,060 --> 00:03:31,450 And it was just this sort of recognition of the 58 00:03:32,380 --> 00:03:37,330 irrational kind of human behavior just in any sort of industry or any thing 59 00:03:37,330 --> 00:03:38,163 in life. 60 00:03:39,130 --> 00:03:43,930 But it sort of helped frame up this idea of you really have to do more than 61 00:03:43,930 --> 00:03:45,880 just, I don't know, 62 00:03:45,880 --> 00:03:50,680 represent logic or rational arguments. You really have to also 63 00:03:50,680 --> 00:03:54,790 bring the easy to understand clear data. And that's, 64 00:03:54,790 --> 00:03:59,770 I think what draws me to incrementality testing specifically and why 65 00:03:59,770 --> 00:04:02,380 that's sort of the backbone of a lot of what I do now. 66 00:04:02,380 --> 00:04:06,850 And I think I use the word operationalizing, NMM and incrementality testing. 67 00:04:06,850 --> 00:04:11,530 And really what I mean by that is a lot of people will run medium mix models or 68 00:04:11,530 --> 00:04:12,880 run incrementality tests, 69 00:04:13,150 --> 00:04:17,290 but oftentimes they'll sit in a slide or in a report to be shown once, 70 00:04:17,290 --> 00:04:18,670 but never to be looked at again. 71 00:04:18,970 --> 00:04:22,870 And so what I'm really trying to do with brands now is how do you build a 72 00:04:22,870 --> 00:04:26,620 framework and a repeatable methodology to get insights from tests, 73 00:04:26,620 --> 00:04:29,470 but not just leave them as insights but to take action? 74 00:04:29,500 --> 00:04:33,160 Because the only way that you create value from any of these sort of testing 75 00:04:33,160 --> 00:04:36,820 methodologies and measurement methodologies is by acting on the insights. 76 00:04:37,630 --> 00:04:41,230 And so that's sort of what I mean by my funky little headline of those words. 77 00:04:41,710 --> 00:04:42,880 Yeah, it's so good, man. 78 00:04:42,880 --> 00:04:46,360 And it's one of those things where data really doesn't matter if you don't take 79 00:04:46,360 --> 00:04:49,510 the right actions from it. And what's so interesting, 80 00:04:49,510 --> 00:04:54,430 and our paths are similar in that I got my start in actually TV and 81 00:04:54,430 --> 00:04:57,940 radio and doing traditional media, and then I got into SEO and paid search, 82 00:04:58,570 --> 00:05:02,710 but I loved video. Video was my thing, but I love paid search as well. 83 00:05:02,710 --> 00:05:06,040 And then when TrueView and TrueView for Action came out, I was like, whoa, 84 00:05:06,460 --> 00:05:07,840 these are all my world's colliding. 85 00:05:07,840 --> 00:05:08,673 This is. 86 00:05:09,070 --> 00:05:11,350 Video and there's some search components, 87 00:05:11,500 --> 00:05:14,830 at least some search intent involved there. And it's direct response. 88 00:05:14,830 --> 00:05:16,570 I've always been a direct response guy. 89 00:05:17,290 --> 00:05:19,990 I believe that marketing should drive an outcome, right? 90 00:05:19,990 --> 00:05:22,840 Advertising should drive a measurable outcome, 91 00:05:23,290 --> 00:05:26,350 and that should be measured in terms of new customers and profitable new 92 00:05:26,350 --> 00:05:28,990 customer acquisition. And what's really interesting, Tom, 93 00:05:28,990 --> 00:05:31,150 and I think this kind of feeds into the conversation we're having today. 94 00:05:31,750 --> 00:05:35,350 There was a period of time, so I grew up reading some of the classics. 95 00:05:35,350 --> 00:05:39,820 So David Ogilvy of course, but John Cap's tested advertising methods, 96 00:05:40,180 --> 00:05:42,160 Claude Hopkins Scientific Advertising. 97 00:05:42,160 --> 00:05:45,760 And they would do things like they would run and add in a newspaper or magazine 98 00:05:45,760 --> 00:05:48,370 and people would clip a coupon and bring it in, 99 00:05:48,370 --> 00:05:51,970 or they would call a certain number and they would track it and they would have 100 00:05:51,970 --> 00:05:52,803 codes and stuff. 101 00:05:53,200 --> 00:05:58,040 And I remember thinking once I got into e-commerce, I was like, oh man, 102 00:05:58,370 --> 00:06:03,170 we've got so many tools. The world is so clear now we have every piece of 103 00:06:03,170 --> 00:06:04,790 data at our disposal. 104 00:06:05,030 --> 00:06:08,030 And now the more I've gotten into it and the more I've matured, I'm like, 105 00:06:08,750 --> 00:06:11,120 we've got more data. But I don't know that we've got more insights, 106 00:06:11,120 --> 00:06:13,130 and I don't know that we've got any more clarity. In fact, 107 00:06:13,130 --> 00:06:14,420 there's maybe more confusion. 108 00:06:14,420 --> 00:06:17,540 And I think it goes back to what you said a minute ago, 109 00:06:17,840 --> 00:06:21,740 this idea of demand generation versus demand capture. 110 00:06:22,100 --> 00:06:26,450 We're really good at measuring channels and campaigns that are demand capture, 111 00:06:26,450 --> 00:06:28,550 meaning they're capturing demand that's already out there. 112 00:06:29,390 --> 00:06:31,700 That's harder to measure the demand generation, 113 00:06:31,880 --> 00:06:34,880 which is usually where the magic happens. 114 00:06:35,270 --> 00:06:37,670 And so super excited to dive in here. 115 00:06:37,670 --> 00:06:42,590 I think what might be useful is let's talk about what 116 00:06:42,710 --> 00:06:46,460 are these kind of three horsemen that I laid out there, MTAs, multitouch, 117 00:06:46,460 --> 00:06:51,080 attribution, and incrementality. So let's start with MTAs first. 118 00:06:51,080 --> 00:06:54,020 So Multitouch attribution tools, 119 00:06:54,230 --> 00:06:56,420 what are they and what is your take on them? 120 00:06:57,770 --> 00:07:01,580 Yeah, big question. Great question. Yeah, I mean, 121 00:07:01,580 --> 00:07:04,520 MTA been around for a while, 122 00:07:04,820 --> 00:07:07,610 different flavors and ways of trying to make it work, 123 00:07:07,790 --> 00:07:12,080 especially as so much has changed in privacy and the tech and tracking 124 00:07:12,080 --> 00:07:12,800 landscape. 125 00:07:12,800 --> 00:07:16,280 But ultimately the goal is to try to give fractional credit to all the 126 00:07:16,280 --> 00:07:20,810 touchpoints along a customer journey with a recognition that the last touchpoint 127 00:07:22,010 --> 00:07:26,930 click or last impression is ultimately not what drove that person 128 00:07:26,930 --> 00:07:27,950 to purchase. 129 00:07:27,950 --> 00:07:32,930 That may be the last or the only thing that you might see in something like 130 00:07:32,930 --> 00:07:35,960 Google Analytics or your analytics suite. 131 00:07:36,830 --> 00:07:40,640 But there's this general recognition that that is not what drove the purchase. 132 00:07:40,640 --> 00:07:45,110 So MTA, the kind of promise, which I ultimately think is a failed promise, 133 00:07:45,110 --> 00:07:50,090 is whether all the different touch touchpoint and then how can you 134 00:07:50,480 --> 00:07:53,270 value those differently. So maybe you use first touch, 135 00:07:53,270 --> 00:07:57,830 maybe you use even distribution. The idea of data-driven attribution was the 136 00:07:57,830 --> 00:07:59,870 holy rail or the Promise many years ago, 137 00:08:00,320 --> 00:08:03,370 and I guess still to a degree for some is like, 138 00:08:03,370 --> 00:08:07,700 how do you know this channel was more additive or more necessary and therefore 139 00:08:07,700 --> 00:08:10,580 should get more credit than that channel? 140 00:08:11,690 --> 00:08:13,670 Which I think makes a ton of sense in promise. 141 00:08:14,630 --> 00:08:19,520 I think in reality it's really hard and I would argue impossible to do, 142 00:08:19,520 --> 00:08:23,870 especially as a lot of the ability to track users at a one-to-one level degrades 143 00:08:24,680 --> 00:08:27,770 generally my perspective, I'm very bearish on MTA, 144 00:08:27,950 --> 00:08:29,810 so that'll probably come through pretty strongly. 145 00:08:30,410 --> 00:08:33,110 But I guess I don't think the toothpaste is going back in the tube in terms of 146 00:08:33,110 --> 00:08:36,620 the ability to track a customer across all these different touchpoints, 147 00:08:37,790 --> 00:08:42,500 especially as the ability to track through or impression based 148 00:08:42,500 --> 00:08:47,240 touchpoint erodes. And then you really get reliant on clicks, 149 00:08:47,240 --> 00:08:50,780 which I think then leads to a lot of all the issues that just last click in 150 00:08:50,780 --> 00:08:51,613 general has. 151 00:08:52,580 --> 00:08:57,090 So I think it's really hard to make a compelling case for MTA. 152 00:08:57,090 --> 00:08:59,490 I've seen too many brands, 153 00:08:59,490 --> 00:09:02,190 especially trying to build MTA tools internally 154 00:09:03,750 --> 00:09:08,130 and just be a huge time and resource suck. And then when you ask to compare, 155 00:09:08,670 --> 00:09:12,570 show the multi-touch view versus last click, it's like, I don't know, 156 00:09:12,600 --> 00:09:17,130 80 or 90% only had one touch point anyways, that's all that MTA model could see. 157 00:09:17,310 --> 00:09:20,580 So is it really that much more useful than last click? 158 00:09:22,320 --> 00:09:26,580 It's sort of multi-touch when that can be measured, but usually it can't be. 159 00:09:26,580 --> 00:09:30,720 Yeah, and It never really answers the causality question either, 160 00:09:30,720 --> 00:09:32,650 which we'll get to when we talk about incrementality. 161 00:09:32,670 --> 00:09:33,840 And I always kind of tell this, 162 00:09:34,860 --> 00:09:39,510 I think the short story of why MT A isn't really viable anymore as all the 163 00:09:39,510 --> 00:09:41,040 tracking and privacy changes. 164 00:09:41,580 --> 00:09:44,910 But I think the slightly longer story is the kind of recognition that just 165 00:09:44,910 --> 00:09:49,320 because an ad was shown or a click occurred doesn't mean that 166 00:09:49,680 --> 00:09:52,380 that medium was needed or that channel was needed. 167 00:09:52,380 --> 00:09:54,210 It doesn't answer the causal question, 168 00:09:54,210 --> 00:09:56,100 what would've happened without this ad running? 169 00:09:56,100 --> 00:10:00,210 Did somebody just happen to use multiple touchpoints as navigation or was it 170 00:10:00,210 --> 00:10:03,390 more convenient to click on one of these ads that happened to be served? 171 00:10:03,390 --> 00:10:08,250 But if you're not comparing that to some sort of control group to really hard 172 00:10:08,700 --> 00:10:11,490 to assign causality to the fact that there just was a touchpoint. 173 00:10:12,750 --> 00:10:16,530 Yeah, it is so good. And it's one of those things where I remember again, 174 00:10:16,530 --> 00:10:17,280 early on, 175 00:10:17,280 --> 00:10:20,890 you would look inside of Google ads or you look inside of Meta or was back when 176 00:10:20,890 --> 00:10:24,630 it was Facebook only, and you were like, the data's here. 177 00:10:24,630 --> 00:10:27,300 I see row ads and I see clicks and I see performance and all that. 178 00:10:28,380 --> 00:10:30,630 Then you realize, well, wait a minute, this isn't fully accurate. 179 00:10:30,630 --> 00:10:33,900 If I add the two together, that's double my total revenue, 180 00:10:34,110 --> 00:10:36,030 so I can't just rely on what's in the platform. 181 00:10:36,030 --> 00:10:40,170 And that got worse as I was 14 was introduced and other privacy changes were 182 00:10:40,170 --> 00:10:44,010 made. But then MTA came along and it's like, oh, 183 00:10:44,010 --> 00:10:48,120 finally we're going to get to see the full picture. It's going to decipher, 184 00:10:48,120 --> 00:10:50,940 decode the shopping journey, 185 00:10:50,940 --> 00:10:55,770 and we're going to finally see with a keen eye in perfection exactly what caused 186 00:10:55,770 --> 00:11:00,540 this ad or what caused this purchase to happen. And then we finally realized 187 00:11:01,650 --> 00:11:04,710 MTA is maybe just a third option. It's like, okay, 188 00:11:04,710 --> 00:11:09,420 Google's imperfect, Meta's data's imperfect, and then mt A, 189 00:11:09,420 --> 00:11:10,410 it's just imperfect too. 190 00:11:10,410 --> 00:11:15,330 So now we just got three imperfect things to look at and make 191 00:11:15,360 --> 00:11:16,230 decisions from. 192 00:11:16,230 --> 00:11:20,190 And in some ways it leads to more confusion than it leads to clarity. 193 00:11:20,640 --> 00:11:25,290 And now I don't want to wholesale discard 194 00:11:25,290 --> 00:11:28,950 MTAs because I do believe there's some helpful insights that can be gained 195 00:11:28,950 --> 00:11:29,783 there, 196 00:11:30,840 --> 00:11:34,200 but it's incomplete and incomplete at best. 197 00:11:34,200 --> 00:11:37,260 And one of the best analogies I've heard, and this actually comes from Ben Ter, 198 00:11:38,490 --> 00:11:41,400 who's also a LinkedIn friend, but I met him in person as well, 199 00:11:41,820 --> 00:11:43,470 but he talks about this analogy of, Hey, 200 00:11:43,470 --> 00:11:48,030 if we're trying to measure what caused people to watch this 201 00:11:48,030 --> 00:11:49,860 movie at our movie theater, 202 00:11:50,220 --> 00:11:55,030 and we look at all these results and 30% say they saw a 203 00:11:55,030 --> 00:11:59,170 billboard for our movies, 20% say they saw a TV ad, 204 00:11:59,380 --> 00:12:03,880 but you know what? A hundred percent say they saw the poster on the 205 00:12:03,880 --> 00:12:06,940 door. So we're like, let's just cut everything. 206 00:12:06,940 --> 00:12:09,760 Let's just do the poster at the door and that's it. And you're like, well, 207 00:12:09,760 --> 00:12:13,000 wait a minute. Everybody saw it. Everybody was walking in the door. 208 00:12:13,300 --> 00:12:16,990 But the movie poster is not what caused someone to purchase. 209 00:12:17,440 --> 00:12:19,720 It was the billboard and the TV and some of the other things, 210 00:12:19,720 --> 00:12:21,760 word of mouth and other things that caused them to come in. 211 00:12:21,760 --> 00:12:26,620 And so this idea of causality, super, super valuable. 212 00:12:26,620 --> 00:12:31,480 So that really leads us to incrementality. So talk about incrementality. 213 00:12:31,780 --> 00:12:35,680 What is it and why are you on a quest to operationalize it? 214 00:12:37,060 --> 00:12:40,210 Yeah, it's really the best way, 215 00:12:40,210 --> 00:12:44,380 if not the only way to establish that a causal 216 00:12:44,920 --> 00:12:49,570 portion that we've been talking about. It has a distinct control group, 217 00:12:49,570 --> 00:12:51,190 so it has a counterfactual, 218 00:12:51,370 --> 00:12:54,340 it has what would've happened without this intervention, 219 00:12:54,340 --> 00:12:56,200 whatever that intervention is. 220 00:12:56,200 --> 00:13:00,820 And there's a handful of ways to derive that counterfactual that control. 221 00:13:00,820 --> 00:13:04,570 The most common would be geographic based. So like a match market test. 222 00:13:04,720 --> 00:13:08,530 I've got this market over here that historically has behaved similarly to this 223 00:13:08,530 --> 00:13:11,500 market over here. I can see that in an AA test, 224 00:13:11,500 --> 00:13:14,080 the lines sort of move similar to one another. They're not, 225 00:13:14,740 --> 00:13:17,050 if they're influenced by outside factors, they're influenced. 226 00:13:17,050 --> 00:13:18,580 In what's an AA test for those who don't know. 227 00:13:19,090 --> 00:13:20,590 Before an intervention happens. 228 00:13:20,620 --> 00:13:24,130 So just over time are those lines essentially moving together? 229 00:13:24,610 --> 00:13:29,320 Are external factors or stimuli equally impacting both sides of that test 230 00:13:29,650 --> 00:13:33,490 so that you can feel confident that when you do intervene and it becomes 231 00:13:33,490 --> 00:13:34,720 comparing A to B, 232 00:13:35,320 --> 00:13:39,400 the delta is what was a result of that intervention. 233 00:13:39,400 --> 00:13:44,050 So oftentimes it's my Atlanta 234 00:13:44,110 --> 00:13:45,760 and I don't know Memphis, 235 00:13:45,760 --> 00:13:49,780 maybe some other midsize city that you've done this market matching for. 236 00:13:50,230 --> 00:13:52,390 Historically, they both look like this on a line, 237 00:13:52,450 --> 00:13:55,840 all of a sudden you turn off ads on Facebook in Atlanta, 238 00:13:55,870 --> 00:14:00,700 what happens to your top line that Delta is what was attributed or 239 00:14:00,700 --> 00:14:04,810 should be attributed to advertising in Atlanta. 240 00:14:05,110 --> 00:14:08,590 Whereas the flip side of that would be attribution would say basically anything 241 00:14:08,590 --> 00:14:12,130 that was attributed to that could be attributed to that would really, 242 00:14:12,250 --> 00:14:16,630 it should just be the gap between a world where that ad does not exist 243 00:14:16,900 --> 00:14:19,810 compared to a world where that ad does exist. We can't take credit for 244 00:14:19,810 --> 00:14:20,643 everything. 245 00:14:20,890 --> 00:14:24,460 We can only take credit for as much above and beyond what would've happened 246 00:14:24,460 --> 00:14:28,840 anyways. And so that's the basis of incrementality testing. 247 00:14:28,840 --> 00:14:30,010 There's other ways to do it. 248 00:14:30,910 --> 00:14:35,770 If you use a Facebook or Google conversion lift study because they own 249 00:14:35,770 --> 00:14:38,380 that auction or anybody that owns an auction, 250 00:14:38,380 --> 00:14:41,080 they can do that hold out for you at a user level. 251 00:14:41,260 --> 00:14:45,130 They can track all of those users regardless of if you serve an ad. 252 00:14:46,420 --> 00:14:51,410 Good examples are maybe easier to describe in a first party data capacity. 253 00:14:51,410 --> 00:14:55,670 If you're running email, you may blast all of your customers and say, Hey, 254 00:14:55,970 --> 00:14:58,970 I sent an email to all my customers and this many purchased. 255 00:14:59,210 --> 00:15:03,110 They went back to the website or clicked it. But if you just said, Hey, 256 00:15:03,110 --> 00:15:07,850 I'm going to serve just to odd number of customer IDs and not to 257 00:15:07,850 --> 00:15:11,240 even number customer IDs, I can then just compare, 258 00:15:11,240 --> 00:15:12,650 forget about who clicked on ads, 259 00:15:12,650 --> 00:15:14,390 who did anything. I'm just going to look at my backend. 260 00:15:14,660 --> 00:15:18,530 I know I exposed these users, but not these users 50 50 split. 261 00:15:18,620 --> 00:15:20,360 They've historically kind of done the same thing. 262 00:15:20,360 --> 00:15:24,080 All I did was even an odd and just measuring the difference between those two 263 00:15:24,080 --> 00:15:24,440 groups. 264 00:15:24,440 --> 00:15:29,150 So really any way that you can establish a true control that 265 00:15:29,150 --> 00:15:33,590 passes that AA test. So before you intervene, do they continue to look similar? 266 00:15:33,770 --> 00:15:37,190 Are they influenced at the same rate so that you can feel confident that when 267 00:15:37,190 --> 00:15:41,270 you do intervene with new media, retracting media, 268 00:15:41,510 --> 00:15:46,040 some new sort of test that you are confidently comparing to what would've 269 00:15:46,040 --> 00:15:47,960 happened in a world without that intervention? 270 00:15:48,350 --> 00:15:49,010 Yeah, yeah. 271 00:15:49,010 --> 00:15:53,570 It's applying the scientific method with some rigor behind 272 00:15:53,870 --> 00:15:56,750 what happens when I turn this channel on, 273 00:15:57,110 --> 00:15:59,480 or what happens when I turn this channel off? 274 00:15:59,840 --> 00:16:02,540 What is the actual impact of this channel? 275 00:16:02,540 --> 00:16:07,100 And what's interesting is I remember back in my early days 276 00:16:07,130 --> 00:16:09,770 of being in the advertising world, 277 00:16:09,770 --> 00:16:12,290 this was when online stuff was just getting kind of warmed up. 278 00:16:12,590 --> 00:16:16,010 I was talking to this furniture store owner and I'm like, Hey, what do you do? 279 00:16:16,100 --> 00:16:19,700 Do you invest in radio ads? Do tv, do you do newspaper? 280 00:16:19,910 --> 00:16:21,800 And so as I went through Themm like, Hey, do you do radio ads? And he is like, 281 00:16:21,800 --> 00:16:25,580 yeah, I mean, yeah, I sort of do. And I'm like, newspaper's like, yeah, 282 00:16:26,180 --> 00:16:28,670 there's a big sale, something will happen. I'm like, well, what about tv? 283 00:16:28,670 --> 00:16:31,550 And he said, yes. And his eyes lit up and he is like, 284 00:16:31,550 --> 00:16:36,260 when I run TV ads, I feel it. People walk in the door, 285 00:16:36,260 --> 00:16:40,430 it happens. And I remember early on in my online career thinking, man, 286 00:16:40,430 --> 00:16:43,070 that was so unsophisticated. Did that guy really know what's going on? 287 00:16:43,340 --> 00:16:46,100 But now looking back, I'm like, yeah, that's maybe all that matters. 288 00:16:46,820 --> 00:16:51,810 That is incrementality in a real loose easy just to observe with your eyes think 289 00:16:51,810 --> 00:16:52,650 because you had one. Totally. 290 00:16:53,180 --> 00:16:55,410 Which I think people take for granted. Yeah. 291 00:16:55,410 --> 00:16:56,243 They do. 292 00:16:56,540 --> 00:16:56,990 Yeah. 293 00:16:56,990 --> 00:16:59,240 That's not exciting. That's not like, where's all your data? 294 00:16:59,960 --> 00:17:02,150 It's in my cash register. That's where all the data. 295 00:17:02,150 --> 00:17:04,670 Is, especially for smaller brands, 296 00:17:05,540 --> 00:17:10,310 when you have the ability to feel if something's 297 00:17:10,310 --> 00:17:11,360 working or not working, 298 00:17:11,540 --> 00:17:15,230 if you double spend in something that you think is working really well because 299 00:17:15,230 --> 00:17:16,910 attribution says it's working really well, 300 00:17:17,180 --> 00:17:18,770 and all of a sudden your cash just doubles, 301 00:17:18,770 --> 00:17:22,730 even though your attributed number scales linearly, something has to give, 302 00:17:22,970 --> 00:17:23,300 right? 303 00:17:23,300 --> 00:17:27,950 And what has to give is it wasn't really causing any additional top line growth. 304 00:17:27,950 --> 00:17:30,230 It was just really good at getting the attributed credit. 305 00:17:30,530 --> 00:17:34,010 So I think the feeling it in the p and l is 306 00:17:35,180 --> 00:17:36,020 definitely overlooked. 307 00:17:36,800 --> 00:17:39,020 It's valid, and it is overlooked though. You're a hundred percent, 308 00:17:39,170 --> 00:17:40,940 especially now that we have so many tools at our disposal. 309 00:17:40,940 --> 00:17:43,880 And I think another way to look at this, and look, I'm a Google guy, 310 00:17:44,090 --> 00:17:48,890 YouTube and Google is kind of where I really got my start in online. 311 00:17:48,900 --> 00:17:49,733 Marketing. 312 00:17:49,950 --> 00:17:53,160 But listen, branded search is a perfect example here. What happens, 313 00:17:53,160 --> 00:17:53,970 we see this all the time. 314 00:17:53,970 --> 00:17:58,050 What happens if you turn branded search completely off? Now, I believe, 315 00:17:58,050 --> 00:18:00,000 and this is top of front of the podcast, 316 00:18:00,240 --> 00:18:03,510 there are strategic ways to use branded search and there's ways to run it and 317 00:18:03,510 --> 00:18:07,890 not waste money, but a lot of people could shut it off and nothing happens, 318 00:18:07,950 --> 00:18:10,020 nothing. Maybe sales get in a little bit, 319 00:18:10,020 --> 00:18:14,130 but you take meta meta's really working and you shut it off and you feel it. 320 00:18:14,400 --> 00:18:17,970 Sales go down and that's an incrementality. 321 00:18:17,970 --> 00:18:22,140 Same is true for YouTube if you're doing YouTube the right way. And so yeah, 322 00:18:22,140 --> 00:18:26,250 I really like this. And one kind of anecdote here to share, 323 00:18:26,250 --> 00:18:31,050 we just did a test with Arctic, Arctic coolers, Yeti competitor, 324 00:18:31,290 --> 00:18:35,190 my favorite cooler, my favorite drinkware as well. And so they wanted to see, 325 00:18:35,190 --> 00:18:40,170 Hey, can YouTube drive an incremental lift at Walmart? So they had just 326 00:18:40,230 --> 00:18:42,690 gotten into most Walmart stores, coast to coast. 327 00:18:43,050 --> 00:18:47,940 So we did exactly what you laid out there. We had a 19 test markets, 328 00:18:48,180 --> 00:18:51,600 19 matched control markets. So similar markets. 329 00:18:51,600 --> 00:18:54,480 So think like a Denver and a Kansas City or the example, 330 00:18:54,480 --> 00:18:57,750 use Atlanta and whatever else that's kind of comparable. And hey, 331 00:18:57,750 --> 00:19:00,180 let's run YouTube in one and not in the other. 332 00:19:00,210 --> 00:19:03,480 And let's measure then the growth in Walmart sales, 333 00:19:03,480 --> 00:19:05,970 and let's do a comparison between the two in Walmart sales. 334 00:19:05,970 --> 00:19:08,760 And it was remarkable. It was about an eight week test. 335 00:19:09,210 --> 00:19:13,470 We had three test regions, so 19 markets, but three test regions, 336 00:19:14,040 --> 00:19:19,020 test region. One, we saw an average of 12% lift in Walmart sales. 337 00:19:19,800 --> 00:19:23,940 The test region two was like 15% lift. 338 00:19:23,940 --> 00:19:28,020 And then our final test region was 25% lift. 339 00:19:28,020 --> 00:19:29,400 And there were some standouts, 340 00:19:29,400 --> 00:19:33,990 like Oklahoma City was up 40% and Salt Lake City was up 48%. But it was one of 341 00:19:33,990 --> 00:19:37,290 those things where, okay, now we look at that and we can say, okay, 342 00:19:37,350 --> 00:19:40,110 YouTube had a big impact. And what's also interesting, Tom, 343 00:19:40,110 --> 00:19:42,450 is we just ran the YouTube portion at OMG. 344 00:19:42,900 --> 00:19:47,130 They also did a connected TV test in other markets, not related, 345 00:19:47,820 --> 00:19:49,920 didn't see a lift, didn't see a measurable lift. 346 00:19:49,920 --> 00:19:54,570 And so it could be lots of that was not to throw shade on 347 00:19:54,570 --> 00:19:56,490 CTVI like CTV, 348 00:19:56,550 --> 00:19:59,100 so maybe they just did a wrong or wrong creatives or who knows what. 349 00:19:59,100 --> 00:20:01,350 But it's one of those things where it's like, okay, 350 00:20:01,350 --> 00:20:04,050 if you do this the right way, you should see an impact. 351 00:20:05,350 --> 00:20:07,410 And I think touching on the piece that I didn't mention, 352 00:20:07,410 --> 00:20:11,460 the other beauty or value of incrementality testing relative to 353 00:20:11,820 --> 00:20:16,650 attribution or mt a is the ability to see beyond your.com to be able to 354 00:20:16,650 --> 00:20:20,970 see what's happening on third parties like Amazon, what's happening in store. 355 00:20:20,970 --> 00:20:24,600 If you get that data own an operated store or if you can get that through 356 00:20:24,600 --> 00:20:27,870 wholesale data, it really simplifies. 357 00:20:28,620 --> 00:20:30,870 There's so much complexity. And I think that's, again, 358 00:20:30,870 --> 00:20:33,930 one of the rubs that I have with MTA is all of them, 359 00:20:35,130 --> 00:20:38,790 all of the data you have to wrangle together to try to 360 00:20:39,810 --> 00:20:42,060 patchwork this kind of story together. 361 00:20:42,990 --> 00:20:45,870 Whereas in incrementality testing, it's pretty straightforward. 362 00:20:45,870 --> 00:20:50,860 It's what did I spend and how did I run that spend in these by 363 00:20:50,860 --> 00:20:55,390 market by day or by week, and what was my sales? What were my sales? 364 00:20:55,630 --> 00:21:00,370 What were my new customers or whatever metric I'd want to look at with that same 365 00:21:00,370 --> 00:21:02,650 granularity and same dimension. 366 00:21:04,030 --> 00:21:06,550 And that's really it because you're really just trying to understand the 367 00:21:06,550 --> 00:21:10,690 relationship that calls the relationship between spend and outcomes, 368 00:21:11,740 --> 00:21:15,100 all that kind of muddy middle in the middle, trying to get it at the user level, 369 00:21:15,100 --> 00:21:19,240 which again, not going back into the tube really simplifies things. 370 00:21:19,720 --> 00:21:20,710 Yeah, it does. 371 00:21:20,710 --> 00:21:23,770 And another thing that was kind of interesting that came a light doing this test 372 00:21:23,770 --> 00:21:28,660 for Arctic is all of the ads we tagged with available at Walmart, 373 00:21:28,810 --> 00:21:32,260 shop at Walmart, find on the shelves and Walmart, whatever. 374 00:21:34,360 --> 00:21:36,430 We measured everything though in those markets. 375 00:21:36,610 --> 00:21:40,780 So you could look at Walmart sales, online sales, so the.com and Amazon. 376 00:21:40,910 --> 00:21:43,930 And what's interesting is the push to Walmart really worked. 377 00:21:44,110 --> 00:21:48,820 It's a reminder of what you ask someone to do in an ad is what they're going to 378 00:21:48,820 --> 00:21:51,070 lean towards. Because in some of the markets, 379 00:21:51,070 --> 00:21:52,720 we didn't see that much of an online lift. 380 00:21:52,720 --> 00:21:56,110 We saw some clicks and stuff like that, but the Lyft was at Walmart. 381 00:21:56,980 --> 00:21:59,380 But we also saw a pretty strong lift at Amazon as well, 382 00:21:59,380 --> 00:22:00,820 because I think that just speaks to, 383 00:22:01,210 --> 00:22:03,400 there's some people that are just going to buy everything from Amazon right 384 00:22:03,400 --> 00:22:07,810 there, tell 'em to go online value pro proposition. Is it on Amazon? Yeah, yeah. 385 00:22:08,560 --> 00:22:10,990 Yeah. Here in a day or two, it's hard. 386 00:22:10,990 --> 00:22:14,560 To beat, dude. It's hard to beat same price in a couple days. 387 00:22:14,560 --> 00:22:19,060 I don't have to leave my house. But yeah, really, really interesting. 388 00:22:19,720 --> 00:22:22,060 And so we'll circle back to that of course, 389 00:22:22,060 --> 00:22:26,080 but let's talk about then MMM or media mix modeling. 390 00:22:26,500 --> 00:22:29,320 What is that? How are you using that? 391 00:22:29,320 --> 00:22:33,370 And then how does that kind of relate to incrementality testing? Because again, 392 00:22:33,370 --> 00:22:37,770 going back to your tagline, Tom, you did not say operationalizing NTAs. 393 00:22:38,110 --> 00:22:40,840 You said operationalizing m and ms and incrementality. 394 00:22:40,840 --> 00:22:44,440 So what is MM and how does that pair with incrementality? 395 00:22:44,590 --> 00:22:44,950 Yeah, 396 00:22:44,950 --> 00:22:49,510 basically a big correlation exercise trying to suss out without a true kind of 397 00:22:49,510 --> 00:22:50,380 holdout group, 398 00:22:50,380 --> 00:22:55,270 what is the impact and contribution of each media channel and also what would 399 00:22:55,270 --> 00:22:56,350 happen without media. 400 00:22:56,350 --> 00:22:59,830 So trying to suss out a lot of the same questions as incrementality, 401 00:22:59,830 --> 00:23:03,970 but basically using correlation as opposed to having a true holdout group. 402 00:23:04,930 --> 00:23:06,160 So basically, 403 00:23:06,160 --> 00:23:10,870 and I'm sure all the hardcore MMM people and data scientists will thumbs down 404 00:23:12,040 --> 00:23:16,540 this or whatever you can do to podcast, but hey, in this period of time, 405 00:23:17,290 --> 00:23:20,380 sales went up and nothing could really explain that other than the fact that 406 00:23:20,380 --> 00:23:25,180 TikTok spend went up and essentially doing that at a mass scale over longer 407 00:23:25,180 --> 00:23:29,500 periods of time trying to take into account anything that could explain that. 408 00:23:29,500 --> 00:23:33,130 So you'll always kind of flag it with these are promotions that happen, 409 00:23:33,130 --> 00:23:35,980 it should because you're going to give a model at least like two years worth of 410 00:23:35,980 --> 00:23:37,390 data or two years worth of data, 411 00:23:38,170 --> 00:23:41,080 it'll bring in seasonality and try to understand those sort of trends. So it's 412 00:23:41,080 --> 00:23:44,740 trying to pull out if not seasonality, if not promotions, 413 00:23:44,740 --> 00:23:47,480 if not some other things that we are flagging. 414 00:23:48,560 --> 00:23:52,340 And it wasn't price reductions, it wasn't all these pieces, 415 00:23:52,490 --> 00:23:55,130 what was happening in media that could explain that change. 416 00:23:55,310 --> 00:23:58,610 And so that's ultimately what MMM is doing. 417 00:23:58,820 --> 00:24:00,410 It's a big correlation exercise, 418 00:24:00,410 --> 00:24:05,240 figuring out roughly what is the channel contribution to a top line revenue or 419 00:24:05,240 --> 00:24:07,550 order number and what's really important. 420 00:24:07,550 --> 00:24:12,440 I think the nicest part or the best first step with M is trying to get an 421 00:24:12,440 --> 00:24:13,970 understanding of a base, 422 00:24:14,000 --> 00:24:17,390 which is what it's going to be called or intercept what without the presence of 423 00:24:17,390 --> 00:24:17,810 ads, 424 00:24:17,810 --> 00:24:22,550 does this model think that my sales would be such that I can then calculate not 425 00:24:23,150 --> 00:24:27,410 a total CAC of just looking at total new customers divided by cost, 426 00:24:27,410 --> 00:24:32,240 but incremental to media or remove base from 427 00:24:32,240 --> 00:24:33,080 that equation, 428 00:24:33,290 --> 00:24:37,550 how many conversions were contributed because of media as this model sees, 429 00:24:37,550 --> 00:24:39,500 which no model is going to be perfect, 430 00:24:39,500 --> 00:24:41,480 no measurement method is going to be perfect, 431 00:24:41,480 --> 00:24:43,280 but it's a really nice place to start to say, 432 00:24:43,940 --> 00:24:47,690 I knew I couldn't account all new customers to advertising, 433 00:24:47,690 --> 00:24:51,170 but what's a good number to use or to start with? Well, it looks like, 434 00:24:51,740 --> 00:24:55,310 and this will depend on the maturity of the brand, but a really mature brand, 435 00:24:55,310 --> 00:24:56,360 I mean super mature brand, 436 00:24:56,360 --> 00:25:01,310 the big CPGs might be like 99% base smaller brand might be something 437 00:25:01,310 --> 00:25:03,980 like 50% because you've got this word of mouth flywheel, 438 00:25:03,980 --> 00:25:05,240 you've got product market fit, 439 00:25:05,390 --> 00:25:09,380 but trying to get an understanding of how much is media contributing relative to 440 00:25:09,380 --> 00:25:11,390 customer base is a really nice place to start. 441 00:25:11,660 --> 00:25:16,520 And the benefit of running incrementality and media mix modeling is 442 00:25:16,520 --> 00:25:19,280 informing the model with some of that causal data. 443 00:25:19,730 --> 00:25:24,320 You see that a lot and there's a really powerful feature of media mix 444 00:25:24,320 --> 00:25:27,650 modeling is saying, Hey, yes, that's a correlation exercise, 445 00:25:28,580 --> 00:25:29,780 can't pull everything out, 446 00:25:30,260 --> 00:25:33,920 but let me inform the model or at least restrict the priors it can use or the 447 00:25:33,920 --> 00:25:35,660 coefficient, whatever you want to call 'em, 448 00:25:36,110 --> 00:25:39,590 what it's searching for to try to find a fit in this model and say, well, 449 00:25:39,590 --> 00:25:42,170 I did a hold out test. I know you don't have the causal data, 450 00:25:42,170 --> 00:25:46,340 but we ran this in this channel and that channel and helping that restrict the 451 00:25:46,340 --> 00:25:50,750 model and giving it data that it can't have without that human intervention can 452 00:25:50,750 --> 00:25:52,130 be a really powerful flywheel. 453 00:25:52,790 --> 00:25:55,460 So using your incrementality test data, 454 00:25:55,460 --> 00:25:59,960 feeding that back into your MMM model to make it more accurate and 455 00:26:00,440 --> 00:26:02,930 more causal and make that correlation. 456 00:26:02,930 --> 00:26:03,763 Stronger. 457 00:26:03,770 --> 00:26:07,250 Because the two things that are really like you're really trying to get, 458 00:26:07,430 --> 00:26:11,030 but you don't get with Multi-Tech attribution or attribution in general. 459 00:26:11,030 --> 00:26:14,840 And you do get with the combination of media mix modeling and incrementality 460 00:26:14,840 --> 00:26:17,150 testing is the incremental impact, 461 00:26:17,150 --> 00:26:20,960 the causal impact of what would've happened without the presence of ads as well 462 00:26:20,960 --> 00:26:22,340 as the diminishing returns curve, 463 00:26:22,340 --> 00:26:24,740 which we know can be really powerful and important too, 464 00:26:25,130 --> 00:26:29,450 is what has happened over time as I spend in that sort of a feature of big 465 00:26:29,450 --> 00:26:33,050 feature of media mix modeling is understanding where are you on a diminishing 466 00:26:33,050 --> 00:26:35,570 returns curve? Is there if I keep spending more, 467 00:26:35,570 --> 00:26:37,310 I know it's not going to scale linearly, 468 00:26:37,550 --> 00:26:39,440 but are there channels that diminish faster? 469 00:26:39,440 --> 00:26:41,600 Is there more headroom in other channels? 470 00:26:41,600 --> 00:26:46,020 And it really becomes this true optimization game of where do I put the next 471 00:26:46,020 --> 00:26:48,870 dollar? Ultimately the question that every marketer, 472 00:26:48,870 --> 00:26:51,600 every finance team is trying to answer is, Hey, 473 00:26:51,600 --> 00:26:54,900 if I find $20,000 into couch cushions, where do I put it? 474 00:26:55,260 --> 00:27:00,150 And if I need to give back $20,000, where do I pull from to have. 475 00:27:01,140 --> 00:27:04,110 I want to hang out at your house and look at your couch cushions and find 20 476 00:27:04,110 --> 00:27:04,680 grand? That's. 477 00:27:04,680 --> 00:27:08,910 Great. Yeah, it's easy to give it back, but yeah, right. 478 00:27:08,910 --> 00:27:11,820 We're trying to figure out what is going to be the least impactful if I have to 479 00:27:11,820 --> 00:27:15,990 give the money back and cut budgets and where is it going to be the most 480 00:27:15,990 --> 00:27:18,270 impactful if I have another $20,000? 481 00:27:18,270 --> 00:27:21,810 Because the answer is not going to be found in what has the highest or the 482 00:27:21,810 --> 00:27:25,440 lowest ROAS in an attributed view. And in fact, 483 00:27:25,440 --> 00:27:28,950 that can have the complete opposite impact that you want. 484 00:27:29,400 --> 00:27:30,990 Yeah, yeah, it's really great. 485 00:27:30,990 --> 00:27:34,950 So I want to actually talk about that point in a minute where 486 00:27:35,850 --> 00:27:38,490 if you've got cut budgets, which hey, listen, 487 00:27:38,490 --> 00:27:42,120 there's been some uncertainty even as we record this, tariffs up, tariffs down, 488 00:27:43,110 --> 00:27:46,980 markets up, market down, whatever consumer sentiment is all over the place. 489 00:27:47,340 --> 00:27:50,070 So if things get a little bit tight, what are we going to do? 490 00:27:50,070 --> 00:27:53,100 We can't slash marketing, we can't slash growth. 491 00:27:53,100 --> 00:27:54,750 I think that sends you into a death spiral, 492 00:27:55,170 --> 00:27:58,020 but we might have to get pull back and get more efficient. 493 00:27:58,020 --> 00:28:02,070 And so let's talk about that actually for a little bit. 494 00:28:02,070 --> 00:28:05,340 So where can you be led astray? 495 00:28:05,370 --> 00:28:07,650 I think you just made a post on LinkedIn about this, right? 496 00:28:07,650 --> 00:28:12,120 Where you start looking at performance, which feels like the smart thing to do, 497 00:28:12,330 --> 00:28:14,250 looking at ROAS and whatnot, and you're like, well, great, well, 498 00:28:14,250 --> 00:28:17,670 let's just cut the lowest ROAS campaigns and channels. We'll be fine. 499 00:28:19,440 --> 00:28:21,480 How does that lead you astray? 500 00:28:21,480 --> 00:28:24,900 And if you want to talk about your specific example to help illustrate these 501 00:28:24,900 --> 00:28:25,733 points, that'd be great. 502 00:28:26,220 --> 00:28:26,820 Yeah, totally. 503 00:28:26,820 --> 00:28:29,190 I think the other one you're referring to is I think branded search, 504 00:28:29,190 --> 00:28:31,830 which we were talking about earlier. And I love using both a, 505 00:28:32,100 --> 00:28:35,940 because it can be really, if a brand is spending a lot of money there, 506 00:28:35,940 --> 00:28:39,270 it can be a really great place to go find those savings without impacting top 507 00:28:39,270 --> 00:28:42,540 line. But also frankly, it's really easy to understand. 508 00:28:42,840 --> 00:28:47,640 I think most people understand that up and down the organizational chart 509 00:28:48,360 --> 00:28:51,870 across departments, everybody sort of understands the idea of, Hey, 510 00:28:51,870 --> 00:28:53,640 if somebody's already searching for my brand, 511 00:28:53,970 --> 00:28:56,940 do I need to pay to get that click and that conversion? 512 00:28:56,940 --> 00:29:01,650 And I found that just the fact that it's easy to understand can be a 513 00:29:01,650 --> 00:29:06,060 really good gateway to incrementality testing because it's easy to get buy-in. 514 00:29:06,060 --> 00:29:07,590 Everybody understands that idea, 515 00:29:07,740 --> 00:29:12,720 whereas it may be more challenging to express that idea in 516 00:29:12,720 --> 00:29:15,870 other types of campaigns. But branded search is a good example, 517 00:29:16,050 --> 00:29:19,110 and the example that you're referring to, 518 00:29:20,040 --> 00:29:22,950 kind of a midsize brand that I was working with went through that exact 519 00:29:22,950 --> 00:29:24,270 exercise, had to cut budgets. 520 00:29:26,190 --> 00:29:30,480 They looked at up and down the campaigns they were running. It was like, Hey, 521 00:29:30,480 --> 00:29:33,600 we just got to make the best decision we can with the best available data. 522 00:29:34,620 --> 00:29:39,210 They were basically running p max non-branded search and branded search and 523 00:29:39,720 --> 00:29:44,050 p max and branded search where had the best attributed roas Best CPA 524 00:29:44,950 --> 00:29:49,420 non-brand was really hard to justify in a lower budget kind of environment based 525 00:29:49,420 --> 00:29:53,770 off the attribution data cut that leaned a little bit more into branded search 526 00:29:53,770 --> 00:29:57,910 as a percentage of their budget. And over the next couple months, 527 00:29:58,600 --> 00:30:03,550 new customers in total revenue was declining despite the 528 00:30:03,550 --> 00:30:06,820 attributed ROAS and CPA looking even better than ever. 529 00:30:07,900 --> 00:30:12,010 And that's where was brought in, looked at all these things, 530 00:30:12,010 --> 00:30:16,870 saw the loose correlation to non-brand and new customer 531 00:30:16,870 --> 00:30:18,100 acquisition and top line, 532 00:30:18,760 --> 00:30:21,670 just the general skepticism that many have around branded search, 533 00:30:21,670 --> 00:30:24,160 especially in a low competition environment, 534 00:30:24,160 --> 00:30:29,110 which they were in. There weren't many competitors in the auction that we 535 00:30:29,110 --> 00:30:32,140 could see in Auction Insights. So yeah, 536 00:30:32,140 --> 00:30:35,380 ran a very blunt instrument match market test, 537 00:30:35,380 --> 00:30:40,360 which at a brand of that size and for a branded search I don't think is ever a 538 00:30:40,360 --> 00:30:43,990 bad idea. And yeah, no impact to branded search. 539 00:30:43,990 --> 00:30:45,520 It was about 20% of their budget, 540 00:30:45,520 --> 00:30:49,090 which was substantial that you can either make the decision, 541 00:30:49,090 --> 00:30:53,860 I'm going to put that 20% back in my pocket or save it for a rainy day 542 00:30:53,860 --> 00:30:58,480 or give it to some other place in the org or say, Hey, 543 00:30:58,480 --> 00:31:03,220 I'm going to redistribute this to something that I see in correlation 544 00:31:03,220 --> 00:31:06,130 data that might help drive top line backup. 545 00:31:06,130 --> 00:31:10,660 Let's reinvest that in non-brand as opposed to keeping it in branded. Again, 546 00:31:10,810 --> 00:31:13,450 complete opposite of what attribution would say. 547 00:31:14,170 --> 00:31:17,380 And you see that a lot frankly with branded search is an easy one to pick on. 548 00:31:17,800 --> 00:31:18,940 Same with retargeting, 549 00:31:20,110 --> 00:31:24,490 but really anything that's especially challenging with the black box 550 00:31:24,700 --> 00:31:25,780 solutions that blend, 551 00:31:25,780 --> 00:31:29,560 and I'm sure we could do a whole talk show on p max Advantage plus some of the 552 00:31:29,560 --> 00:31:33,730 things that bundled together historically radically different levels of 553 00:31:33,730 --> 00:31:36,850 incrementality can be a real challenge when you're then measuring on 554 00:31:36,850 --> 00:31:41,800 attribution. But yeah, a ranty way of saying yes, 555 00:31:42,460 --> 00:31:47,440 finding areas to cut oftentimes if you follow the attribution kind 556 00:31:47,440 --> 00:31:51,760 of data can lead to really kind of impactful in a negative way 557 00:31:52,300 --> 00:31:56,440 business outcomes because the attribution view just does not take into account 558 00:31:57,190 --> 00:32:00,610 what would've happened without the presence of those ads like Incre Ality does. 559 00:32:01,510 --> 00:32:05,110 And so can definitely lead brands astray as they're looking to cut. 560 00:32:05,770 --> 00:32:07,690 Yeah, really interesting. And yeah, 561 00:32:07,900 --> 00:32:11,830 max notorious for leaning into remarketing or branded search. 562 00:32:11,830 --> 00:32:15,100 If you're not diligent about that, it can lean into both of those things. 563 00:32:15,100 --> 00:32:17,260 And so got to be mindful of that. 564 00:32:17,650 --> 00:32:20,080 You also quoted something that totally ties into this. 565 00:32:20,320 --> 00:32:24,580 It's from a shop talk talk that you went to shop Talk the show, 566 00:32:24,880 --> 00:32:29,050 and I can't remember who said it, but if you see high roas, 567 00:32:29,800 --> 00:32:34,660 I know something is wrong and that the auto targeting is just finding existing 568 00:32:34,810 --> 00:32:39,530 customers. Do you remember actually who said that and unpack a little bit? 569 00:32:40,040 --> 00:32:45,020 Yeah, I forget his name and I could look real quick. He worked for. 570 00:32:45,740 --> 00:32:46,573 Mic. 571 00:32:47,120 --> 00:32:51,350 The Post Dan Danone, the big CPG. 572 00:32:52,580 --> 00:32:54,140 Yeah, I just really appreciated that quote because I 573 00:32:56,060 --> 00:33:00,530 mean always wonder if I live in sort of a bubble of being super passionate about 574 00:33:00,770 --> 00:33:02,720 incrementality versus attributed metrics, 575 00:33:03,140 --> 00:33:06,620 but that was just really refreshing to hear because I don't think that's the 576 00:33:06,620 --> 00:33:07,453 natural. 577 00:33:08,810 --> 00:33:09,643 It's not. 578 00:33:10,190 --> 00:33:11,060 Thought in people's. 579 00:33:11,060 --> 00:33:12,230 Head spend more. 580 00:33:13,160 --> 00:33:16,310 But I really think it should kind of spark some skepticism, 581 00:33:16,310 --> 00:33:20,120 especially when your goal really is to try to drive new customers. 582 00:33:22,040 --> 00:33:22,730 My first, 583 00:33:22,730 --> 00:33:27,440 especially if you think about both incrementality in the context of a SC 584 00:33:27,440 --> 00:33:31,280 or pex that's blending retargeting and prospecting by default 585 00:33:32,600 --> 00:33:34,610 and knowing diminishing returns 586 00:33:36,290 --> 00:33:39,320 are my first dollars, yes, they're going to be the most effective, 587 00:33:39,320 --> 00:33:44,240 but if they are focused on people that are already buying from me and my goal in 588 00:33:44,240 --> 00:33:45,290 my head is new customers, 589 00:33:45,440 --> 00:33:50,330 I should be shocked that I can spend a hundred dollars and drive 590 00:33:51,110 --> 00:33:53,240 this amazing new customer revenue 591 00:33:55,100 --> 00:33:59,300 and not think that something is up or even over time as I continue to spend 592 00:33:59,810 --> 00:34:03,380 our BS meters should probably go up a little bit more. 593 00:34:04,100 --> 00:34:07,790 And I don't think they do by default. So I found that comment really refreshing. 594 00:34:09,080 --> 00:34:09,860 Yeah, I think that really illustrates that, 595 00:34:09,860 --> 00:34:13,130 right where it's like most of us would think, oh, ROAS is going up great, 596 00:34:13,130 --> 00:34:14,120 we're printing money. 597 00:34:14,540 --> 00:34:19,490 Whereas maybe you should say BS detector, something's wrong here. 598 00:34:19,490 --> 00:34:23,060 This campaigns leaning into customers that we're going to buy anyway. 599 00:34:23,150 --> 00:34:26,600 And I'll give two examples here to illustrate this a little bit more. 600 00:34:26,860 --> 00:34:29,480 And I'll also, since we've been picking on branded search so much, 601 00:34:29,480 --> 00:34:32,870 I'll share a couple of ways I think we should use it. One. 602 00:34:33,380 --> 00:34:33,710 If. 603 00:34:33,710 --> 00:34:35,570 Other competitors are aggressively bidding on, 604 00:34:36,680 --> 00:34:40,460 just know that if you're not Nike and you're not Adidas and you're not like Ford 605 00:34:40,460 --> 00:34:44,300 or something, it's not a lock. If it's a new customer, 606 00:34:45,140 --> 00:34:46,850 they could be swayed by a competitor. 607 00:34:46,850 --> 00:34:49,760 And that's generally how we like to separate it out is like, 608 00:34:50,090 --> 00:34:54,530 let's have branded search for returning customers and let's make that crazy 609 00:34:54,530 --> 00:34:56,570 efficient or just turn it off altogether. 610 00:34:56,750 --> 00:34:56,990 If. 611 00:34:56,990 --> 00:35:00,530 It's a new customer, then again, we want it to be very efficient, 612 00:35:00,530 --> 00:35:03,890 but maybe we want it on because we don't want our competitor to come in and 613 00:35:03,890 --> 00:35:08,690 swipe us to give and swipe our customer. And so one example of this, 614 00:35:08,690 --> 00:35:12,920 I did a podcast with Brian Porter, he's the co-founder of Simple, modern, 615 00:35:12,920 --> 00:35:17,690 great Drinkware brand has become a friend and they did a study incrementality 616 00:35:17,690 --> 00:35:19,640 study and they found, I'll get these numbers off, 617 00:35:19,640 --> 00:35:23,150 but it was like branded search was 10% incremental. 618 00:35:23,510 --> 00:35:28,220 So basically what that means is if it shows that I got a hundred new customers 619 00:35:28,220 --> 00:35:29,180 from Branded Search, 620 00:35:29,810 --> 00:35:33,080 I probably would've gotten 90 of those if I had shut it off, right? 621 00:35:33,080 --> 00:35:35,000 Only 10% were incremental. 622 00:35:35,570 --> 00:35:39,720 So then what you would need to do there is you need a 10 x row as on branded 623 00:35:39,720 --> 00:35:42,030 search for it to even make sense. If it's below that, 624 00:35:42,030 --> 00:35:45,840 you're completely wasting money. Pair that with, 625 00:35:45,840 --> 00:35:50,130 and you and I were commenting on the House analytics, HAUS, 626 00:35:50,130 --> 00:35:54,750 Olivia Corey and team did 190 incrementality studies involving 627 00:35:54,780 --> 00:35:59,430 YouTube and they showed with tremendous amounts of rigor 628 00:35:59,430 --> 00:36:00,660 that hey, 629 00:36:00,660 --> 00:36:05,220 YouTube is probably 342 times more 630 00:36:05,220 --> 00:36:08,340 incremental, meaning if you see a one in platform, 631 00:36:08,340 --> 00:36:11,880 it's actually like a 3 42 in terms of incremental impact. 632 00:36:11,880 --> 00:36:16,170 And so wildly different between those two. But again, 633 00:36:16,740 --> 00:36:19,800 we're just so drawn to in platform row as man, we'll just say spin, 634 00:36:19,800 --> 00:36:23,280 spin spend on p max and branded search when really we should be saying, 635 00:36:23,820 --> 00:36:27,270 let me lean into YouTube or let me lean into top of funnel meta. 636 00:36:27,870 --> 00:36:31,350 I think both those examples too are really good examples. 637 00:36:31,560 --> 00:36:33,480 To me it also speaks though to the importance of 638 00:36:35,070 --> 00:36:39,540 cost per incremental almost being more important than incremental 639 00:36:40,290 --> 00:36:43,200 percent incremental. And that's something I always use with branded search. 640 00:36:43,200 --> 00:36:45,750 I think you and I have a very similar feeling around branded search. 641 00:36:46,590 --> 00:36:48,090 There's definitely a time and a place for it, 642 00:36:48,090 --> 00:36:51,420 and it's one of those things where it might not matter that it's 10% 643 00:36:51,420 --> 00:36:55,680 incremental, 10% incremental relative to what Google's attributing. 644 00:36:57,230 --> 00:37:01,350 If your attributed CPA is a dollar and now it's 645 00:37:01,860 --> 00:37:02,580 $10, 646 00:37:02,580 --> 00:37:07,290 but your margin when you sell a product is a thousand dollars like 647 00:37:07,860 --> 00:37:08,880 hammer that all day long, 648 00:37:09,210 --> 00:37:12,420 that cost per incremental is still extremely profitable and valuable. 649 00:37:13,560 --> 00:37:15,000 And same with the YouTube piece. 650 00:37:15,570 --> 00:37:20,310 If YouTube was four times as incremental as Google said, 651 00:37:20,580 --> 00:37:22,440 but your YouTube was crazy expensive, 652 00:37:22,590 --> 00:37:24,780 it still might not be worth it even though it's four times. 653 00:37:24,780 --> 00:37:25,500 More. 654 00:37:25,500 --> 00:37:27,900 Incremental than the platform was making. 655 00:37:27,900 --> 00:37:32,880 And that's how I think a lot about this with connected tv where 656 00:37:34,800 --> 00:37:39,660 connected TV can be super powerful and maybe more so than linear tv, 657 00:37:39,840 --> 00:37:44,730 but if you can buy scatter linear TV for a 10th 658 00:37:44,730 --> 00:37:46,800 of the cost of CTV, 659 00:37:47,250 --> 00:37:51,540 well it just has to be more than a 10th as effective and 660 00:37:52,350 --> 00:37:54,180 it's accreted, it's a positive. 661 00:37:54,180 --> 00:37:58,890 So it becomes more of comparison of a cost per than just a 662 00:37:58,890 --> 00:37:59,280 blanket. 663 00:37:59,280 --> 00:38:03,780 How incremental is something which I always think is important to focus on and 664 00:38:03,780 --> 00:38:04,110 call out. 665 00:38:04,110 --> 00:38:05,460 To. Yeah, it's so good. 666 00:38:05,460 --> 00:38:09,870 I mean measuring something in terms of percentages can provide insights and help 667 00:38:09,870 --> 00:38:13,020 make decisions, but ultimately it's the cost per right. 668 00:38:14,220 --> 00:38:17,460 Translate that into real dollars to see if it makes sense. 669 00:38:17,940 --> 00:38:19,380 100% agree with you, 670 00:38:19,380 --> 00:38:22,560 but I think this also goes back to and use your linear TV example, 671 00:38:22,560 --> 00:38:25,770 and I still love TV and connected TV and stuff. Again, 672 00:38:25,770 --> 00:38:27,900 I'll use YouTube just because I've got the numbers in my brain, 673 00:38:27,900 --> 00:38:32,310 but with YouTube sometimes we'll see a $5 CPM or a 674 00:38:32,310 --> 00:38:36,370 $7 CPM in certain audiences compared to other channels that are 675 00:38:36,370 --> 00:38:39,940 15, 20, 30, 50, whatever. Totally. And I'm like, well, 676 00:38:39,940 --> 00:38:44,680 if we're reaching the right person and if the message and offer are 677 00:38:44,680 --> 00:38:49,270 good, how could this not work? And it's one of those things where it's like, 678 00:38:49,270 --> 00:38:52,030 okay, we're either one of those is off, we're talking to the wrong person, 679 00:38:52,030 --> 00:38:52,960 that's the wrong message, 680 00:38:53,170 --> 00:38:56,320 or we're just not measuring it properly and that's where we need to look at it. 681 00:38:56,350 --> 00:38:58,690 So did you have a thought on that? 682 00:38:58,960 --> 00:39:00,490 You another question on MM here in just a second. 683 00:39:00,700 --> 00:39:03,520 Yeah, yeah, totally. But it made me think of the idea of, 684 00:39:05,200 --> 00:39:08,710 I think the reason I'm starting to become way more bullish on any channel that's 685 00:39:08,710 --> 00:39:11,620 historically been hard to measure where I think there's that arbitrage 686 00:39:11,620 --> 00:39:15,970 opportunity of costs are still relatively low because people haven't all moved 687 00:39:15,970 --> 00:39:17,590 in because it's easy to attribute. 688 00:39:17,980 --> 00:39:20,800 It'll be really interesting with a house example, 689 00:39:20,950 --> 00:39:23,110 does that inspire a lot more YouTube buyers? 690 00:39:23,470 --> 00:39:26,710 That's something that Google should have put out way long ago, 691 00:39:26,710 --> 00:39:30,940 but I think it would undermine undermine search and that's their bigger 692 00:39:30,940 --> 00:39:34,330 business. And I could do a whole kind of rant and I'll save you that, 693 00:39:34,330 --> 00:39:38,410 but the idea of incrementality first measurement probably wouldn't be great for 694 00:39:38,410 --> 00:39:40,360 the search business. So probably exactly, 695 00:39:40,360 --> 00:39:43,660 haven't been able to make such a good point that case on YouTube. 696 00:39:44,020 --> 00:39:46,750 But you think about all the channels that have historically been harder to 697 00:39:46,750 --> 00:39:47,583 attribute, 698 00:39:47,830 --> 00:39:51,700 that's where costs are deflated just from a supply and demand perspective. 699 00:39:51,700 --> 00:39:56,560 So when you can move in and get CPMs at five to $7 and it's really effective, 700 00:39:56,560 --> 00:39:59,260 but most people that are measuring through attribution don't know it's really 701 00:39:59,260 --> 00:40:04,210 effective, that's a huge win for certain period of time until everybody's flood, 702 00:40:04,210 --> 00:40:05,290 everybody and the costs go. 703 00:40:05,290 --> 00:40:06,123 Up the market. 704 00:40:06,760 --> 00:40:10,270 I'm sure there's a lot of people that were not excited to see that study from 705 00:40:10,270 --> 00:40:14,770 house like dang it, that means my costs are going up. I don't like that at all. 706 00:40:14,890 --> 00:40:16,540 So really good man. 707 00:40:16,540 --> 00:40:19,900 So we talked about incrementality testing and I think you can use tools like 708 00:40:19,900 --> 00:40:21,100 House and then there are others. 709 00:40:21,100 --> 00:40:24,130 We're just talking about work magic and there's a number of others you can lean 710 00:40:24,130 --> 00:40:27,310 into. Full disclosure, they're pretty expensive, 711 00:40:27,880 --> 00:40:29,860 but you can also do stuff on your own too. 712 00:40:30,490 --> 00:40:32,320 If you've got someone that can measure this stuff, 713 00:40:32,320 --> 00:40:36,700 you can do a little bit of it on your own. What about the MMM side of things? 714 00:40:36,880 --> 00:40:41,770 What's kind of the easy way to start there? Is there an easy way to start? 715 00:40:42,640 --> 00:40:43,750 What do you recommend to people. 716 00:40:43,750 --> 00:40:48,730 There? I don't know. I dunno if there's an easy way to do anything. 717 00:40:48,730 --> 00:40:52,870 I think, well, I guess that's not totally true. 718 00:40:52,870 --> 00:40:55,900 I think there's some ways to run relatively easy incre tests. 719 00:40:56,470 --> 00:40:58,270 So I think that's the easier place to start. 720 00:40:58,690 --> 00:41:01,570 Certainly you can always ratchet up the scientific rigor. 721 00:41:01,570 --> 00:41:05,650 I think the problem with looking for an easy MM solution is 722 00:41:06,940 --> 00:41:11,380 anybody could run a model with Robin or there's a lot of open source packages, 723 00:41:11,380 --> 00:41:13,180 but just because you can run a model, 724 00:41:16,150 --> 00:41:16,983 it could say anything. 725 00:41:18,400 --> 00:41:22,810 It's not necessarily rooted in this can all of a sudden predict the future 726 00:41:23,620 --> 00:41:26,110 and tell you exactly the contribution from media. 727 00:41:26,110 --> 00:41:28,660 Whereas incrementality can do that a little more out of the box. 728 00:41:28,660 --> 00:41:30,730 You may have wildly wide confidence intervals, 729 00:41:31,750 --> 00:41:36,350 but it answers the question. It gives you the comparison. 730 00:41:36,800 --> 00:41:37,970 I didn't do it in this market, 731 00:41:37,970 --> 00:41:40,940 I did it in this market. What is the Delta Media mix modeling? 732 00:41:40,940 --> 00:41:44,120 You could build a model to tell sort of any story. 733 00:41:45,110 --> 00:41:48,560 The proof is sort of in the pudding of if I do the thing that the model says, 734 00:41:48,980 --> 00:41:51,830 does it change my top line? 735 00:41:52,010 --> 00:41:55,340 Can I see over time that when I listen to the model 736 00:41:57,140 --> 00:41:58,370 that improves my top line? 737 00:41:58,370 --> 00:42:03,140 So it's a lot easier to get started with incrementality testing. 738 00:42:03,140 --> 00:42:08,060 You can run poor man's match market tests as I sort you can just 739 00:42:08,060 --> 00:42:08,750 sort of pick, 740 00:42:08,750 --> 00:42:12,200 some markets historically behave similarly and there's certainly some risk 741 00:42:12,200 --> 00:42:15,710 there, but with a model you might think that it's an amazing model. 742 00:42:16,760 --> 00:42:21,230 I just don't feel like there's a great place to DIY that together without some 743 00:42:21,230 --> 00:42:25,220 real scientific or statistical rigor. Or if you do, 744 00:42:25,220 --> 00:42:29,900 you've just got to try to prove it over and over by taking some big swings. And 745 00:42:29,900 --> 00:42:30,733 that's really, 746 00:42:31,430 --> 00:42:35,120 I sort of feel like you can get away with the kind of feel it sort of tests 747 00:42:35,750 --> 00:42:38,630 without really running a true incrementality test or model. 748 00:42:38,990 --> 00:42:42,290 If you're a small enough business and you spend a decent amount on Facebook, 749 00:42:42,980 --> 00:42:44,780 maybe you're not willing to turn off Facebook, 750 00:42:44,780 --> 00:42:48,080 but are you willing to drastically increase spend and see if you can feel 751 00:42:48,080 --> 00:42:51,530 something at the top line? Okay, then what happens if you cut it in half? 752 00:42:51,530 --> 00:42:52,070 What happens? 753 00:42:52,070 --> 00:42:56,840 And start to understand those curves on your own is probably a less risky way 754 00:42:56,840 --> 00:43:01,430 than trying to, I've never done anything in R and I'm going to run 755 00:43:02,480 --> 00:43:05,090 or done any sort of medium amount. I'm going to try to run one. 756 00:43:05,090 --> 00:43:06,500 That's probably a risky proposition. 757 00:43:06,680 --> 00:43:09,980 Yeah, it's a really good insight. I'm glad you answered the question that way. 758 00:43:09,980 --> 00:43:10,850 I think, yeah, 759 00:43:10,850 --> 00:43:15,020 leaning into the poor man's incrementality test or just leaning really heavily 760 00:43:15,020 --> 00:43:18,800 into a channel and measuring your top line if you've got a small enough business 761 00:43:18,800 --> 00:43:22,280 to look at that, but probably if you're going to lean into MM M1, 762 00:43:22,280 --> 00:43:26,360 you need a couple years of data and so to be able to make some correlations and 763 00:43:26,360 --> 00:43:31,280 you probably need to lean in to someone or a tool with quite a bit of 764 00:43:31,280 --> 00:43:32,750 experience because you can do that astray. 765 00:43:33,080 --> 00:43:35,270 And on your comment on cost too. 766 00:43:35,270 --> 00:43:39,710 I mean it's all relative and a lot of times where you're going to need a medium 767 00:43:39,710 --> 00:43:42,710 mix modeling is when you're spending a significant amount in a significant 768 00:43:42,710 --> 00:43:43,430 number of channels, 769 00:43:43,430 --> 00:43:46,550 which you're probably only doing if you are spending a lot total, 770 00:43:46,550 --> 00:43:50,060 which you're probably only doing if your revenue can support that high level of 771 00:43:50,060 --> 00:43:50,390 spend, 772 00:43:50,390 --> 00:43:55,190 which means that a tool may not be all that expensive relative to the 773 00:43:55,190 --> 00:43:58,850 opportunity you could derive from it, which is where I always net out. 774 00:43:59,600 --> 00:44:01,790 So I'm paying 10 or 20 grand for a tool monthly, 775 00:44:01,790 --> 00:44:05,750 but it's allowing me to redeploy millions in ad spend. 776 00:44:05,750 --> 00:44:09,860 And it totally in completely makes sense. So Tom, 777 00:44:09,860 --> 00:44:12,080 this has been fantastic. I'm just watching the clock. 778 00:44:12,080 --> 00:44:15,530 I know we're kind of coming up against it, but one, 779 00:44:15,530 --> 00:44:18,530 I recommend people follow you on LinkedIn. You put out some awesome content. 780 00:44:18,530 --> 00:44:19,400 I love reading it. 781 00:44:19,700 --> 00:44:19,700 Thank. 782 00:44:19,700 --> 00:44:23,540 You. People should definitely follow you on LinkedIn and you are, is it Tom, 783 00:44:23,840 --> 00:44:27,980 what is your handle on LinkedIn? You are Thomas B. Leonard. 784 00:44:28,190 --> 00:44:29,990 Thomas B. Leonard. That's probably confusing. 785 00:44:30,200 --> 00:44:33,300 I'm very self-conscious of LinkedIn, so I'm glad to thank you for saying that. 786 00:44:34,530 --> 00:44:36,120 I think it's good, man. I think it's really good. I like it a lot. Yeah. 787 00:44:36,420 --> 00:44:39,720 Yeah, it's been fun to start doing connecting with folks. 788 00:44:40,710 --> 00:44:45,480 Definitely an area that had a lot of excitement and passion for, 789 00:44:45,480 --> 00:44:47,610 it's fun to have these sort of conversations, 790 00:44:47,610 --> 00:44:51,420 so I appreciate you reaching out a while ago and that we could connect. 791 00:44:51,420 --> 00:44:52,253 Absolutely. 792 00:44:52,380 --> 00:44:55,080 Man. Absolutely. So then if other people were like, Hey, 793 00:44:55,080 --> 00:44:58,890 I just want to talk to Tom because maybe you can help my brand or my business, 794 00:44:59,550 --> 00:45:04,110 how can they connect with you and who are you looking to or who do you feel like 795 00:45:04,110 --> 00:45:04,943 you can help? 796 00:45:05,310 --> 00:45:09,210 Yeah, definitely appreciate that. Yeah, reach out on LinkedIn. 797 00:45:10,890 --> 00:45:14,640 I spend time there. I love reading everybody's thoughts and content. So yeah, 798 00:45:14,640 --> 00:45:19,590 reach out on LinkedIn mostly we work with consumer facing brands that 799 00:45:19,590 --> 00:45:24,300 are trying to understand where to put the next dollar or where to pull 800 00:45:24,840 --> 00:45:29,520 in the scenarios. They have to really kind of rescue people from attribution, 801 00:45:30,480 --> 00:45:35,310 trying to better understand where they can get more with their ad dollars. 802 00:45:35,310 --> 00:45:38,520 I think to your point that you teed up now is such an interesting time or 803 00:45:38,520 --> 00:45:40,890 anytime that there's margin pressure, 804 00:45:42,180 --> 00:45:44,280 there's more scrutiny on a marketing budget. 805 00:45:45,090 --> 00:45:49,350 Really want to try to help empower marketing teams to feel more confident with 806 00:45:49,350 --> 00:45:52,800 what they're doing and ultimately the finance teams to feel more confident with 807 00:45:52,800 --> 00:45:57,330 what marketing team is doing. Hundred percent. That's where I love to plug in, 808 00:45:57,750 --> 00:46:01,440 but also just love to talk about this stuff probably more than I should. 809 00:46:01,440 --> 00:46:03,150 So always open to the conversation. 810 00:46:04,050 --> 00:46:05,730 Yeah, I talk about that a lot. 811 00:46:05,730 --> 00:46:10,620 I've read analytics and measurement books on vacation and my wife 812 00:46:10,620 --> 00:46:14,400 is like, what is wrong with you? And I'm like, it's interesting. I don't know. 813 00:46:14,400 --> 00:46:18,960 I like it. And so totally, we are just a different breed I suppose, 814 00:46:18,960 --> 00:46:20,640 but I love that. 815 00:46:20,640 --> 00:46:24,630 And then I think this is a great way to end it where if I've got an extra dollar 816 00:46:24,900 --> 00:46:29,490 to spend on marketing, where do I put it? If I need to cut a dollar of spend, 817 00:46:29,550 --> 00:46:30,930 where do I cut it from? 818 00:46:31,260 --> 00:46:36,060 And that's really what this approach is about MMM 819 00:46:36,120 --> 00:46:39,570 and incrementality. And so I think their necessities, 820 00:46:39,570 --> 00:46:43,620 I think attribution is broken and or misleading in so many different ways. 821 00:46:44,550 --> 00:46:48,360 There's some correlations there, so we don't have to throw it out completely, 822 00:46:48,690 --> 00:46:51,840 but I do believe you need to lean into MMM and incrementality for short. 823 00:46:51,840 --> 00:46:56,670 So connect with Tom on LinkedIn. And with that, we'll wrap. 824 00:46:56,670 --> 00:47:01,500 Tom's been fantastic. Thanks for the time, the insights and the energy. Yeah. 825 00:47:02,340 --> 00:47:04,590 Thanks so much Brett time. Glad to connect. 826 00:47:05,040 --> 00:47:09,900 Absolutely. And as always, thank you for tuning in. We'd love to hear from you. 827 00:47:09,900 --> 00:47:12,060 If you found this episode helpful, 828 00:47:12,660 --> 00:47:15,690 someone else in the D two C space or marketing space, and you think, man, 829 00:47:15,690 --> 00:47:18,570 they got to listen to this, please share it. We mean the world to me. 830 00:47:19,020 --> 00:47:21,510 And with that, until next time, thank you for listening.