1 00:00:14,441 --> 00:00:18,321 Justin Norris: welcome to rev ops FM attribution remains one of the 2 00:00:18,321 --> 00:00:20,081 hardest challenges in marketing. 3 00:00:20,091 --> 00:00:20,631 It affects. 4 00:00:21,111 --> 00:00:24,831 Literally everything, how we're perceived as a discipline, the strategies, the 5 00:00:24,831 --> 00:00:29,441 activities we decide to do, how we justify our existence, all of it. 6 00:00:29,501 --> 00:00:33,411 And at the end of the day, there's still so little rigor in most 7 00:00:33,421 --> 00:00:34,951 companies around attribution. 8 00:00:35,431 --> 00:00:38,181 Most teams I've worked with are doing something either very basic 9 00:00:38,181 --> 00:00:42,671 like first or last touch or using methodologies pushed by a vendor. 10 00:00:43,091 --> 00:00:45,371 Uh, that are kind of just invented out of thin air that aren't 11 00:00:45,371 --> 00:00:47,011 backed by a lot of evidence. 12 00:00:47,011 --> 00:00:50,051 I don't see that to be critical because I've used those methodologies too. 13 00:00:50,371 --> 00:00:52,691 I've been a consultant for those methodologies. 14 00:00:53,184 --> 00:00:56,054 but I think there is a growing awareness as we scratch a little bit beneath the 15 00:00:56,054 --> 00:01:00,264 surface that a lot of the ways that we're doing attribution and historically 16 00:01:00,264 --> 00:01:04,944 have done attribution, especially in B2B, Don't have a very solid grounding. 17 00:01:05,498 --> 00:01:10,152 so today we're joined by Pranav Piyush is the CEO of Paramark, which 18 00:01:10,152 --> 00:01:15,163 is a company that provides media mix modeling software, incrementality 19 00:01:15,163 --> 00:01:19,514 testing software, and we actually got connected through a linkedin discussion. 20 00:01:19,514 --> 00:01:20,904 I was posting about attribution. 21 00:01:20,914 --> 00:01:24,045 He was very, politely pushing back on some of the things that I was saying. 22 00:01:24,045 --> 00:01:26,755 So I thought it was actually a great opportunity to bring him onto the 23 00:01:26,755 --> 00:01:29,075 show and talk about what he's seeing 24 00:01:29,085 --> 00:01:31,375 so for now, thank you so much for joining us today. 25 00:01:31,615 --> 00:01:32,825 Pranav Piyush: Thanks for having me. 26 00:01:32,845 --> 00:01:38,755 And I have to give you your flowers for being engaging and really sort of engaging 27 00:01:38,755 --> 00:01:41,655 in discussion and debate on LinkedIn. 28 00:01:41,685 --> 00:01:43,005 I think that's the way it should roll. 29 00:01:43,005 --> 00:01:45,115 So excited to chat live about it. 30 00:01:45,456 --> 00:01:45,795 Justin Norris: Agreed. 31 00:01:45,795 --> 00:01:47,735 So, I mean, Maybe let's just start. 32 00:01:47,756 --> 00:01:51,817 I've set the stage a little bit with, some of the problems out there, but 33 00:01:52,377 --> 00:01:54,592 give us your view from 1000 feet. 34 00:01:54,592 --> 00:01:56,832 what is the state of B to B attribution today? 35 00:01:56,832 --> 00:01:59,712 How is it going from your perspective in the industry? 36 00:02:00,147 --> 00:02:04,467 Pranav Piyush: things to be excited about is the higher level of 37 00:02:04,487 --> 00:02:10,836 conversation around the concepts of incrementality and even just the 38 00:02:10,905 --> 00:02:12,785 concepts of correlation and causation. 39 00:02:13,305 --> 00:02:16,225 And that's really good. 40 00:02:16,535 --> 00:02:18,645 I'm glad that we're having that conversation. 41 00:02:18,645 --> 00:02:21,825 I'm glad that these types of podcasts are coming across. 42 00:02:22,975 --> 00:02:27,935 At the same time, there's almost a tale of two cities or two worlds or 43 00:02:27,935 --> 00:02:35,605 whatever, where there's a whole spectrum of conversation that is either still 44 00:02:35,615 --> 00:02:42,565 stuck in the multi touch attribution world or is inventing pseudoscience. 45 00:02:43,425 --> 00:02:48,415 In the name of incrementality, and that gets me really upset because 46 00:02:48,855 --> 00:02:51,915 I have this point of view that the word attribution has been maligned. 47 00:02:52,035 --> 00:02:54,995 It's actually a really beautiful word, but it's literally just been 48 00:02:55,035 --> 00:02:58,945 maligned and something very similar is going to start happening with 49 00:02:58,945 --> 00:03:02,665 incrementality because Of a whole host of reasons that we can talk about that. 50 00:03:02,995 --> 00:03:07,815 So I'm both excited and optimistic, but at the same time, a little bit disappointed 51 00:03:07,815 --> 00:03:09,675 by some of the conversations going on. 52 00:03:10,126 --> 00:03:12,705 Justin Norris: Yeah, I am seeing this play out the scene 53 00:03:12,705 --> 00:03:14,265 that you've described exactly. 54 00:03:14,265 --> 00:03:19,710 And I go back to, 10 years ago or maybe even a little bit more when Bizible first 55 00:03:19,710 --> 00:03:23,660 came on the scene and it was like this amazing thing you could track all these 56 00:03:23,660 --> 00:03:27,370 different touch points and then you could choose all these different ways to divide 57 00:03:27,370 --> 00:03:30,358 them up take your opportunity credit and you just kind of slice it up like a pizza, 58 00:03:30,368 --> 00:03:35,128 like this touch point gets a slice and we do w shape or equal weight or whatever. 59 00:03:35,430 --> 00:03:38,591 and we never really stopped a question, I guess, as a marketer and like, I'll 60 00:03:38,591 --> 00:03:40,528 take ownership for this for myself. 61 00:03:40,528 --> 00:03:43,206 We didn't stop to question like, is this justifiable? 62 00:03:43,206 --> 00:03:44,466 Does it make sense to do this? 63 00:03:44,466 --> 00:03:49,004 Does it actually produce like a, good outcome in terms of decision 64 00:03:49,004 --> 00:03:52,914 making if we do this and now, I guess there is a lot more pushback. 65 00:03:53,079 --> 00:03:53,728 against this. 66 00:03:54,148 --> 00:03:57,478 Can you just walk us through like, is that sort of historical? 67 00:03:57,537 --> 00:03:59,227 multi touch attribution methodology. 68 00:03:59,527 --> 00:04:01,537 Is it a valid way of looking at the world? 69 00:04:01,537 --> 00:04:02,707 And if not, why not? 70 00:04:03,170 --> 00:04:08,590 Pranav Piyush: I give you three points of evidence that will hopefully 71 00:04:09,130 --> 00:04:11,010 help inform this conversation. 72 00:04:11,620 --> 00:04:17,829 So the first one, if you look at a whole variety of channels, and this is 73 00:04:17,860 --> 00:04:22,960 arguably sort of more biased towards larger brands, whether you're B2B or B2C. 74 00:04:22,990 --> 00:04:24,470 But if you look at channels like. 75 00:04:24,835 --> 00:04:29,730 Social video podcasts, Like the one that we're talking about right now. 76 00:04:30,480 --> 00:04:38,170 These do not generate clicks or touches by definition. 77 00:04:38,530 --> 00:04:42,840 You are going to exclude a significant part of marketing 78 00:04:42,840 --> 00:04:44,690 and media from your models. 79 00:04:45,170 --> 00:04:47,960 If you rely on multi touch attribution models to assess the 80 00:04:47,960 --> 00:04:49,430 impact of these types of channels. 81 00:04:50,305 --> 00:04:53,505 And well, some people will tell me, well, like, what about view 82 00:04:53,505 --> 00:04:54,885 through attribution, right? 83 00:04:54,915 --> 00:04:55,975 Isn't that the solution? 84 00:04:57,055 --> 00:05:01,095 And I'm like, yes and no, because that brings me to the second point, 85 00:05:01,295 --> 00:05:04,705 which is there is coincidence. 86 00:05:05,085 --> 00:05:07,565 There is correlation and there is causation. 87 00:05:07,965 --> 00:05:08,785 The three C's. 88 00:05:09,705 --> 00:05:13,425 Everything that we are talking about in MTA world is coincidence. 89 00:05:14,615 --> 00:05:17,225 A Latin term that I just recently came across. 90 00:05:17,890 --> 00:05:20,500 Post hoc ergo proctor hoc. 91 00:05:20,930 --> 00:05:25,470 All this means is, after this, therefore because of this? 92 00:05:26,690 --> 00:05:28,570 That's a logical fallacy. 93 00:05:29,195 --> 00:05:32,534 That is a very popular fallacy that just because something happened right 94 00:05:32,625 --> 00:05:35,825 prior to something else happening that we assume that there's a cause 95 00:05:35,825 --> 00:05:39,315 and effect relationship there where there isn't even a correlation. 96 00:05:40,185 --> 00:05:41,175 so that's the second point. 97 00:05:41,865 --> 00:05:45,215 And the third is when you add all the privacy changes that have happened 98 00:05:45,215 --> 00:05:49,905 in the last five years, people are realizing that even the touch based 99 00:05:49,905 --> 00:05:56,365 data that they do have is incomplete because guess what, 20 to 30 to 50 100 00:05:56,365 --> 00:05:58,525 percent of people do not accept cookies. 101 00:05:58,930 --> 00:06:01,930 And most of your first click, last click data is coming through 102 00:06:02,220 --> 00:06:04,430 either cookies or UTM codes. 103 00:06:04,700 --> 00:06:07,340 And now there's increasing evidence that UTM codes are 104 00:06:07,580 --> 00:06:11,240 probably going to get stripped out from pretty much every browser. 105 00:06:11,420 --> 00:06:14,860 Safari is already doing that in multiple cases, Firefox is already 106 00:06:14,860 --> 00:06:16,120 doing that in multiple cases. 107 00:06:16,790 --> 00:06:20,560 So, when you look at all three of those points, it's like, how 108 00:06:20,610 --> 00:06:22,750 can multi touch attribution work? 109 00:06:23,220 --> 00:06:26,350 And that's how I generally think about this conversation. 110 00:06:26,945 --> 00:06:30,265 Now, it doesn't mean that you shouldn't track clicks and touches. 111 00:06:30,315 --> 00:06:31,305 That's not what I'm saying. 112 00:06:31,955 --> 00:06:37,485 There's perfect legitimate use cases for tracking a user journey. 113 00:06:37,835 --> 00:06:39,625 And that's how I think about that data. 114 00:06:39,625 --> 00:06:41,325 It's behavioral analytics. 115 00:06:41,585 --> 00:06:42,525 It's not attribution. 116 00:06:43,415 --> 00:06:45,345 Attribution is very simply cause and effect. 117 00:06:45,765 --> 00:06:48,015 If we're not talking about cause and effect, you can't call it attribution. 118 00:06:48,789 --> 00:06:51,019 Justin Norris: So that is an important distinction to make tracking the 119 00:06:51,019 --> 00:06:56,795 touch points, understanding, these are observable facts that we can detect. 120 00:06:56,795 --> 00:06:59,255 We're not saying it's the entirety of everything that happened. 121 00:06:59,255 --> 00:07:01,305 It's just, we're saying that this happened and we could track it. 122 00:07:02,000 --> 00:07:06,230 That can be useful taking it and then, you know, starting to dole out credit 123 00:07:06,230 --> 00:07:11,130 and saying, therefore this channel drove X million dollars in pipeline. 124 00:07:11,150 --> 00:07:12,160 That's a, that's a fallacy. 125 00:07:12,160 --> 00:07:13,040 And I think that makes sense. 126 00:07:13,064 --> 00:07:17,534 I I've yet to see a strong argument against, I think those three things that 127 00:07:17,534 --> 00:07:18,814 you mentioned are kind of devastating. 128 00:07:19,308 --> 00:07:23,957 so let's then like, I've created a nice foil now for, MMM, or 129 00:07:24,138 --> 00:07:26,658 I've heard it say both media mix modeling and marketing mix modeling. 130 00:07:26,658 --> 00:07:28,488 I don't know which of those you prefer. 131 00:07:28,818 --> 00:07:31,618 Maybe just introduce us to that and why is it different and not 132 00:07:31,618 --> 00:07:33,488 suffer from those same limitations. 133 00:07:33,902 --> 00:07:35,072 Pranav Piyush: I don't have a preference. 134 00:07:35,072 --> 00:07:37,272 I think, you know, people can call it whatever it is. 135 00:07:37,282 --> 00:07:39,762 It's we invent so many new terms. 136 00:07:39,802 --> 00:07:43,372 what's interesting about MMM is that it's predated all of us. 137 00:07:43,622 --> 00:07:44,942 It's predated the Internet. 138 00:07:45,422 --> 00:07:49,452 So this was actually invented, I believe, in the 70s or 80s 139 00:07:49,482 --> 00:07:52,092 by, you know, folks at PNG. 140 00:07:52,552 --> 00:07:53,382 This is folklore. 141 00:07:53,382 --> 00:07:56,332 I don't think there is good attribution for this, but I 142 00:07:56,352 --> 00:08:00,983 think it was literally built by a partnership of academia folks at M. 143 00:08:00,983 --> 00:08:01,113 I. 144 00:08:01,123 --> 00:08:01,373 T. 145 00:08:01,373 --> 00:08:04,613 And Howard and practitioners that companies like P. 146 00:08:04,613 --> 00:08:04,743 N. 147 00:08:04,743 --> 00:08:05,053 G. 148 00:08:05,583 --> 00:08:06,943 And they had a hard job, right? 149 00:08:06,943 --> 00:08:11,483 Because you didn't have any touch and click data. 150 00:08:11,873 --> 00:08:15,273 So how do you know which of your ads are working or not working? 151 00:08:15,273 --> 00:08:16,663 And they were working with, you know, T. 152 00:08:16,663 --> 00:08:16,983 V. 153 00:08:16,983 --> 00:08:21,783 And radio and newspapers and these types of media channels. 154 00:08:22,213 --> 00:08:23,763 And so they had to invent something new. 155 00:08:24,223 --> 00:08:28,043 And so you had a bunch of these statisticians, who we now call data 156 00:08:28,043 --> 00:08:33,203 scientists, who in the 70s and 80s are like, Hey, there's actually a way that 157 00:08:33,203 --> 00:08:42,164 we can model the data about readership and listenership and find correlations 158 00:08:42,164 --> 00:08:50,004 between the increase or decrease in readership and Of PNG products. 159 00:08:50,744 --> 00:08:56,344 So as you increase the number of ads in the wall street journal, maybe 160 00:08:56,344 --> 00:09:03,244 that's a bad example, how many more sales can be attributed to that region 161 00:09:03,364 --> 00:09:06,124 that that newspaper is distributed in. 162 00:09:06,124 --> 00:09:08,094 And that was the beginning of MMM. 163 00:09:08,774 --> 00:09:11,184 You look at time series data. 164 00:09:11,639 --> 00:09:13,359 So you're looking at day by day. 165 00:09:13,689 --> 00:09:16,359 You have 10, 000 people reading a newspaper. 166 00:09:16,459 --> 00:09:18,529 You have 15, 000 people reading a newspaper. 167 00:09:18,539 --> 00:09:20,439 You have 20, 000 people reading a newspaper. 168 00:09:20,699 --> 00:09:24,409 And as that trend goes up and to the right, do you have a 169 00:09:24,409 --> 00:09:28,189 corresponding increase in your amount in your sales and vice versa? 170 00:09:28,239 --> 00:09:32,268 When you decrease the distribution of ads through newspapers, do 171 00:09:32,269 --> 00:09:34,859 you see a decline in your sales? 172 00:09:35,464 --> 00:09:40,084 And now when you do that across multiple channels at the same time, you 173 00:09:40,084 --> 00:09:41,834 can build very sophisticated models. 174 00:09:41,834 --> 00:09:44,234 And these are all being done through spreadsheets and 175 00:09:44,234 --> 00:09:46,084 manual work back in the 80s. 176 00:09:46,424 --> 00:09:50,004 And now it's been digitized and we can talk about sort of what that's been like. 177 00:09:50,014 --> 00:09:54,904 So when you think of it that way, right, let's talk about all three issues. 178 00:09:55,684 --> 00:09:58,954 You no longer have to sacrifice channels. 179 00:09:59,274 --> 00:10:00,994 This can work for newspapers. 180 00:10:00,994 --> 00:10:01,974 It can work for Google. 181 00:10:01,974 --> 00:10:03,164 It can work for podcasts. 182 00:10:03,164 --> 00:10:05,064 It can work for pretty much any channel out there. 183 00:10:05,899 --> 00:10:14,209 Second, you are no longer making an assumption about one thing happening and 184 00:10:14,209 --> 00:10:15,589 therefore the second thing happening. 185 00:10:15,589 --> 00:10:19,759 You're actually looking at the statistical correlation between a 186 00:10:19,819 --> 00:10:24,289 quantity increasing and its impact on your sales increasing or not. 187 00:10:24,659 --> 00:10:26,529 So you can imagine if those two numbers are going like 188 00:10:26,529 --> 00:10:28,299 this, there's no correlation. 189 00:10:28,739 --> 00:10:31,799 But if the both numbers are going like this, there is a correlation. 190 00:10:32,244 --> 00:10:33,274 So that's the second piece. 191 00:10:33,534 --> 00:10:38,984 And third, because you are not actually tracking individual users, you're 192 00:10:38,984 --> 00:10:41,879 making a, analysis on aggregated data. 193 00:10:42,259 --> 00:10:43,889 You have no privacy concerns. 194 00:10:44,339 --> 00:10:48,219 We're not trying to spy on a certain user and say, did you click or touch on 195 00:10:48,229 --> 00:10:50,529 this particular piece of advertising? 196 00:10:51,079 --> 00:10:55,289 We're saying as overall numbers have increased in terms of 197 00:10:55,289 --> 00:10:57,029 impressions or reach or frequency. 198 00:10:57,029 --> 00:10:57,069 Okay. 199 00:10:57,364 --> 00:10:59,344 Has your sales numbers increased? 200 00:10:59,344 --> 00:11:03,204 It's a very different approach and completely privacy 201 00:11:03,414 --> 00:11:05,041 friendly and future proof. 202 00:11:05,308 --> 00:11:08,258 Justin Norris: So that makes total sense to me and let me ask you a question 203 00:11:08,268 --> 00:11:11,468 like I'm going to try to poke a hole not out of skepticism because I would 204 00:11:11,468 --> 00:11:12,738 just like to hear how you feel it. 205 00:11:13,058 --> 00:11:15,458 And this is probably coming from a place of ignorance because I am not 206 00:11:15,458 --> 00:11:17,138 a statistician or a mathematician. 207 00:11:17,328 --> 00:11:20,208 so we've solved the post hoc fallacy because we've seen that two things 208 00:11:20,208 --> 00:11:23,688 trend in a similar direction, let's say, placing ads in the wall street 209 00:11:23,688 --> 00:11:28,103 journal, like you said, and sales, what it doesn't solve for, like, let's say 210 00:11:28,103 --> 00:11:32,303 that the reason why we decided to place more ads was because, you know, economic 211 00:11:32,313 --> 00:11:35,813 forecast, consumer confidence is high as we predict there'll be more demands. 212 00:11:35,813 --> 00:11:38,983 We're going to place some more ads so we have correlation, but we don't 213 00:11:38,993 --> 00:11:41,533 necessarily have causation because the increasing spending could have been. 214 00:11:41,928 --> 00:11:42,698 Caused by that. 215 00:11:42,918 --> 00:11:44,908 How do you resolve that problem in MMM? 216 00:11:45,332 --> 00:11:50,022 Pranav Piyush: There is a certain amount of correlation that you will not be able 217 00:11:50,022 --> 00:11:52,552 to convert into causation through MMMs. 218 00:11:52,632 --> 00:11:54,672 MMMs are not causal. 219 00:11:56,392 --> 00:12:00,714 I'm probably the only MMM vendor that will that out loud and be 220 00:12:00,744 --> 00:12:02,424 willing to, you know, stand by it. 221 00:12:02,879 --> 00:12:07,439 So just because I'm talking about MMM being better than MTA doesn't 222 00:12:07,439 --> 00:12:10,799 mean that I'm saying that MMM is the end all be all and gives you 223 00:12:10,799 --> 00:12:13,199 perfect causality in your models. 224 00:12:13,259 --> 00:12:13,639 It doesn't. 225 00:12:14,049 --> 00:12:15,339 It is a correlation. 226 00:12:15,669 --> 00:12:18,739 It's an estimate of causality, but it is not causality. 227 00:12:19,509 --> 00:12:23,739 To get to causality in marketing, you have just one option. 228 00:12:24,294 --> 00:12:25,444 That is experimentation. 229 00:12:26,624 --> 00:12:29,594 By the way, that is also the only known way of getting to 230 00:12:29,594 --> 00:12:31,224 causality in any other field. 231 00:12:31,577 --> 00:12:33,967 Justin Norris: I was just thinking that it sounds a lot like medical 232 00:12:33,997 --> 00:12:36,577 tests, like you can do long term studies and establish correlation. 233 00:12:36,577 --> 00:12:39,547 But if you want causation, you have to do a controlled trial. 234 00:12:39,862 --> 00:12:40,992 Pranav Piyush: Precisely. 235 00:12:41,012 --> 00:12:44,252 So random control trials are CTS. 236 00:12:44,912 --> 00:12:50,062 Everything in all other fields of science, rests on the shoulders of our CTS. 237 00:12:50,132 --> 00:12:51,852 And there is no difference in marketing. 238 00:12:52,112 --> 00:12:55,452 If you want to get to causality, you have to run experiments. 239 00:12:55,832 --> 00:12:59,512 So one of the things that we talk about at paramark all the time is the 240 00:12:59,512 --> 00:13:01,912 purpose of your marketing mix models. 241 00:13:01,912 --> 00:13:02,942 Media mix models. 242 00:13:02,992 --> 00:13:06,512 Attribution models is not to establish causality. 243 00:13:06,552 --> 00:13:13,302 It is to understand the The hints of causality, it's estimates of causality. 244 00:13:13,742 --> 00:13:17,612 And if you really want to get precise about causality, that 245 00:13:17,612 --> 00:13:21,152 should inform a series of experiments that you are running. 246 00:13:21,342 --> 00:13:23,592 And we can talk about how experiments can be run in media. 247 00:13:23,902 --> 00:13:27,702 It's harder than running an A B test on your website because you 248 00:13:27,712 --> 00:13:28,992 don't control the surface, right? 249 00:13:28,992 --> 00:13:32,342 You don't control TV, you don't control radio, you don't control meta. 250 00:13:32,802 --> 00:13:37,317 But there are, Reasonable ways of doing experimentation on that channel 251 00:13:37,327 --> 00:13:39,437 that gives you a sense of causality. 252 00:13:39,437 --> 00:13:42,037 And I can talk about that if there's, you know, if you think 253 00:13:42,037 --> 00:13:43,517 that's a good time to jump into it. 254 00:13:43,994 --> 00:13:44,274 Justin Norris: Yeah. 255 00:13:44,274 --> 00:13:46,493 I mean, let's, go there let's say a podcast. 256 00:13:46,493 --> 00:13:47,383 Here's a perfect example. 257 00:13:47,383 --> 00:13:48,753 Obviously this is my own podcast. 258 00:13:48,753 --> 00:13:51,163 Let's say, uh, a company's looking to run a podcast. 259 00:13:51,783 --> 00:13:56,142 How should we understand causally if it's affecting, you know, Sales or not. 260 00:13:56,519 --> 00:13:57,399 Pranav Piyush: That's an interesting one. 261 00:13:57,649 --> 00:14:02,859 Podcast advertising far easier to test causality, right? 262 00:14:02,859 --> 00:14:08,349 Because if you're advertising on Spotify or on YouTube or any other channel where 263 00:14:08,349 --> 00:14:12,559 you have sort of podcast advertising, almost all those platforms are going 264 00:14:12,559 --> 00:14:14,449 to give you geotargeting capabilities. 265 00:14:15,019 --> 00:14:20,539 So if you're doing testing on podcast ads, the way to construct your test 266 00:14:20,539 --> 00:14:23,159 and control groups is by geography. 267 00:14:23,589 --> 00:14:28,399 And you'll tell Spotify that, Hey, target my ads just to, I'm just making this 268 00:14:28,399 --> 00:14:34,139 up San Francisco and Miami, and let's keep New York and Austin as the control. 269 00:14:34,699 --> 00:14:38,959 And you're going to monitor your conversions from those locations 270 00:14:39,159 --> 00:14:43,519 over the next six weeks to see if there was a statistically valid 271 00:14:43,529 --> 00:14:45,319 increase in your conversions. 272 00:14:45,919 --> 00:14:51,709 As a result of you running those podcast ads, that is pretty much 273 00:14:51,709 --> 00:14:53,289 the blueprint for any channel. 274 00:14:53,819 --> 00:14:57,319 The hardest parts about this are identifying the right geographies, 275 00:14:57,659 --> 00:15:01,599 identifying the time frame, and identifying the amount of 276 00:15:01,609 --> 00:15:05,239 budget that you have to dedicate to be able to see an effect. 277 00:15:05,664 --> 00:15:06,884 That's pretty simple. 278 00:15:07,784 --> 00:15:09,494 Podcasts by themselves? 279 00:15:10,274 --> 00:15:12,524 I think of them as an asset, not a channel. 280 00:15:13,064 --> 00:15:15,734 So let's talk about this, You and I are recording this podcast. 281 00:15:16,254 --> 00:15:18,104 Well, how does it make it out to people? 282 00:15:18,214 --> 00:15:19,954 There's only one way you have to distribute it. 283 00:15:21,024 --> 00:15:25,264 So the real question is how are you distributing it? 284 00:15:25,464 --> 00:15:28,584 And does that distribution plan have a positive effect? 285 00:15:28,954 --> 00:15:29,734 On your pipeline. 286 00:15:30,624 --> 00:15:34,864 So whether that's splicing it up on social, whether that's cutting it up 287 00:15:34,864 --> 00:15:39,874 into social ads, whether it's through email to my list of email subscribers, 288 00:15:40,224 --> 00:15:45,114 but that's where you get into testing, not the, whether me producing a 289 00:15:45,114 --> 00:15:48,563 podcast has an effect or not, it's like asking, you know, I created an 290 00:15:48,563 --> 00:15:50,213 ebook, but like that doesn't matter. 291 00:15:50,213 --> 00:15:53,523 What really matters is how did you deliver your ebook to your audience? 292 00:15:53,523 --> 00:15:54,393 And you can test that. 293 00:15:54,682 --> 00:15:57,122 Justin Norris: If the tree falls in the forest and no one hears it, 294 00:15:57,652 --> 00:15:59,382 does it make a sound kind of thing? 295 00:15:59,772 --> 00:16:02,992 But let's get into content because this is a huge question. 296 00:16:03,422 --> 00:16:07,094 I, work with a content team, and I've worked with, with many content 297 00:16:07,094 --> 00:16:10,624 teams over time, and everybody wants to know, like, is this working? 298 00:16:10,634 --> 00:16:11,344 Is it helping? 299 00:16:11,734 --> 00:16:15,316 And, I saw a post, I think it was Dale Harrison who also posts a lot on these 300 00:16:15,316 --> 00:16:17,225 topics, maybe it was on a post of yours. 301 00:16:17,225 --> 00:16:20,235 It might have been around, around incrementality, like seeing a blog 302 00:16:20,525 --> 00:16:22,595 or not seeing a blog, and it's like, well, of course, the people that 303 00:16:22,595 --> 00:16:25,525 are reading your blog are people that are already more likely to buy. 304 00:16:25,525 --> 00:16:25,665 Okay. 305 00:16:26,220 --> 00:16:31,680 And I could see that argument, but then you're also saying, well, we're investing 306 00:16:31,680 --> 00:16:35,750 a ton, like hundreds of thousands, even millions in a bigger company in content, 307 00:16:36,020 --> 00:16:39,200 there has to be some way to be able to tell aside from the distribution channel. 308 00:16:39,490 --> 00:16:42,290 Whether that content is actually valuable, whether it influences 309 00:16:42,290 --> 00:16:46,370 people's decisions in any way, how would you go about trying to do that? 310 00:16:46,759 --> 00:16:52,099 Pranav Piyush: So first off, I would say, if you think about any 311 00:16:52,409 --> 00:16:58,085 creative production, at the end of the day, content is at the heart 312 00:16:58,125 --> 00:17:00,745 of what we do as marketers, right? 313 00:17:00,765 --> 00:17:04,315 If you don't have content, you have nothing, you're distributing nothing. 314 00:17:04,865 --> 00:17:10,974 So the idea that you have to justify your investments and 315 00:17:10,974 --> 00:17:14,515 content come from a place of severe. 316 00:17:15,484 --> 00:17:20,035 Anxiety and under confidence in you as a marketer. 317 00:17:20,915 --> 00:17:21,204 Okay. 318 00:17:21,244 --> 00:17:23,714 So that's the first thing that I'm going to say, like, that's the 319 00:17:23,714 --> 00:17:25,444 wrong conversation to be having. 320 00:17:26,055 --> 00:17:29,975 Now you might still get forced into having that conversation because you have 321 00:17:29,975 --> 00:17:34,115 a finance team and you have a CEO who maybe don't understand that and you're 322 00:17:34,125 --> 00:17:37,975 having to talk about the justifications of why you're investing hundreds 323 00:17:37,975 --> 00:17:39,445 of thousands of dollars in content. 324 00:17:40,245 --> 00:17:40,675 I get it. 325 00:17:41,335 --> 00:17:42,775 I understand where that's coming from. 326 00:17:43,865 --> 00:17:50,005 I would flip the script and flipping the script means if you can explain. 327 00:17:50,755 --> 00:17:54,485 That a hundred percent of your marketing budget, right? 328 00:17:54,575 --> 00:17:59,585 If you're spending 5 million a year are bringing an incremental 329 00:18:00,425 --> 00:18:05,527 50 million in pipeline, then nobody cares how you spend the 5 million 330 00:18:06,077 --> 00:18:07,887 between content versus distribution. 331 00:18:09,157 --> 00:18:14,197 The problem and the question arises because you don't have good math on 332 00:18:14,197 --> 00:18:19,187 your side about how much incremental pipe has been generated as a result 333 00:18:19,187 --> 00:18:20,547 of that 5 million in marketing. 334 00:18:21,197 --> 00:18:24,697 And I don't care if it's brand, performance, content, creative, paid, 335 00:18:24,717 --> 00:18:26,457 earned, owned, it doesn't matter. 336 00:18:27,217 --> 00:18:29,857 So, that's my answer to that question. 337 00:18:30,107 --> 00:18:34,057 Trying to pinpoint the efficacy of every single piece of 338 00:18:34,057 --> 00:18:36,987 content is a losing proposition. 339 00:18:37,777 --> 00:18:43,467 If you had to do it, I would do it based on the engagement metrics of that content. 340 00:18:44,457 --> 00:18:45,077 What do I mean? 341 00:18:45,582 --> 00:18:48,592 You know, I had a conversation with Ashley Faust from Atlassian 342 00:18:48,592 --> 00:18:50,212 and I pitched this concept to her. 343 00:18:50,222 --> 00:18:56,092 I was like, look at the total volume of consumption of a content. 344 00:18:56,242 --> 00:19:00,042 So if you're talking about a blog post, the total minutes that have ever 345 00:19:00,042 --> 00:19:01,962 been read about a certain blog post. 346 00:19:02,432 --> 00:19:04,432 If it's a video, the total view time. 347 00:19:04,487 --> 00:19:06,267 Guess what? 348 00:19:06,287 --> 00:19:10,157 If you go and talk to YouTube creators, that's what they're going to talk about. 349 00:19:10,397 --> 00:19:12,747 The lifetime viewership of their content. 350 00:19:13,157 --> 00:19:15,827 They're going to look at the percentage of people who make it 351 00:19:15,827 --> 00:19:17,147 to the end of the video, right? 352 00:19:17,157 --> 00:19:18,537 That's the video retention rate. 353 00:19:18,977 --> 00:19:24,887 So you look at the consumption metrics of the pieces of content to understand 354 00:19:24,907 --> 00:19:26,617 if the content is good enough or not. 355 00:19:27,187 --> 00:19:29,717 But again, it's You still have to figure out how you're going 356 00:19:29,717 --> 00:19:30,837 to distribute the content. 357 00:19:30,847 --> 00:19:32,497 And that's a different metric altogether. 358 00:19:32,762 --> 00:19:33,842 Justin Norris: that makes sense to me. 359 00:19:33,852 --> 00:19:36,602 And I think I agree with you, but I want to, I want to drill down on it one more 360 00:19:36,602 --> 00:19:40,402 level, just because this really, and I'm sure you have a similar experience. 361 00:19:40,402 --> 00:19:42,392 This really is a conversation we have all the time. 362 00:19:42,762 --> 00:19:44,802 And I think part of it, you're quite right. 363 00:19:44,812 --> 00:19:45,452 Is the. 364 00:19:45,852 --> 00:19:49,492 insecurity, I think we've developed as, as marketers in the face of like 365 00:19:49,512 --> 00:19:52,592 these purely short term activation, performance driven metrics. 366 00:19:53,071 --> 00:19:55,721 and the way that executive teams ask questions to marketers. 367 00:19:55,721 --> 00:19:59,384 So there's that, but then I think there's also, you want to know 368 00:19:59,444 --> 00:20:01,767 sometimes like, is my content any good? 369 00:20:01,767 --> 00:20:03,047 Like I put out podcast episodes. 370 00:20:03,047 --> 00:20:05,967 I want to understand are these episodes good or the ones with 371 00:20:05,977 --> 00:20:08,337 the highest viewership, the best episodes, or is it, you know, there's 372 00:20:08,337 --> 00:20:09,677 lots of factors that come into it. 373 00:20:10,047 --> 00:20:12,777 So to give a tangible example to just try to test. 374 00:20:13,102 --> 00:20:13,812 What you're saying. 375 00:20:14,055 --> 00:20:16,505 we have a, a blog post in my company. 376 00:20:16,505 --> 00:20:18,565 It's like a very, uh, top of funnel. 377 00:20:18,565 --> 00:20:19,485 It's a blog post in NASA. 378 00:20:19,485 --> 00:20:22,805 It's something, it's a piece of content, very top of funnel, very 379 00:20:22,805 --> 00:20:25,061 general, not really product related. 380 00:20:25,061 --> 00:20:26,491 So of course it gets a ton of traffic. 381 00:20:26,501 --> 00:20:28,211 It's about like onboarding or something like that. 382 00:20:28,211 --> 00:20:29,721 Something that's very relevant to everyone. 383 00:20:30,181 --> 00:20:33,551 So it's a lot of people, but there's no guarantee that the. 384 00:20:33,836 --> 00:20:36,936 People who are consuming that content, you know, have any sort 385 00:20:36,936 --> 00:20:40,576 of like product related intent are ever going to buy anything, connect 386 00:20:40,576 --> 00:20:42,866 that content back to our product. 387 00:20:43,146 --> 00:20:46,796 So I, totally buy into the engagement metrics and I, and that is the 388 00:20:46,796 --> 00:20:48,246 same feedback I give to my team. 389 00:20:48,256 --> 00:20:50,826 And yet I struggle sometimes when you say it could be engaging, but is it 390 00:20:50,826 --> 00:20:54,396 engaging the right people that eventually will lead to the outcomes that you want? 391 00:20:54,406 --> 00:20:55,596 Like, how do you control for that? 392 00:20:55,887 --> 00:20:58,460 Pranav Piyush: You know, It's the same topic as the question 393 00:20:58,460 --> 00:21:00,030 of a marketing qualified lead. 394 00:21:00,549 --> 00:21:02,409 This is the same exact conversation. 395 00:21:02,879 --> 00:21:07,969 And if you think about the word qualified, who are we to qualify our prospects? 396 00:21:09,099 --> 00:21:13,179 They are qualifying us for whether we solve their need or not. 397 00:21:13,949 --> 00:21:17,319 So, that to me is not a measurement question, that's a strategy 398 00:21:17,319 --> 00:21:21,309 question of why are you putting that content out in the first place. 399 00:21:21,859 --> 00:21:26,299 And if you're putting the content out in the first place is to, you know, boost 400 00:21:26,299 --> 00:21:31,299 your engagement metrics, then obviously it's going to attract the wrong traffic. 401 00:21:31,539 --> 00:21:32,259 You see what I'm saying? 402 00:21:32,709 --> 00:21:36,929 Because your strategy is focused on juicing a number rather 403 00:21:36,929 --> 00:21:38,659 than to serve your audience. 404 00:21:39,359 --> 00:21:43,529 If your strategy was to serve your audience, you would immediately think 405 00:21:43,529 --> 00:21:48,484 about a different way of measuring it, Or your measurement would be more real, 406 00:21:48,668 --> 00:21:53,968 so, I think we conflate different things when we talk about content strategy, like 407 00:21:54,458 --> 00:21:56,568 I don't care for the qualified metric. 408 00:21:56,588 --> 00:21:58,478 I don't care for qualified as a term. 409 00:21:58,918 --> 00:22:03,288 We don't do any of that at paramark itself is because it's very clear who we serve. 410 00:22:03,848 --> 00:22:07,618 And if you don't find that out on your first visit to 411 00:22:07,618 --> 00:22:09,555 paramark, we did something wrong. 412 00:22:09,585 --> 00:22:12,535 So it doesn't matter what engagement metrics I had from that number. 413 00:22:13,045 --> 00:22:15,895 and again, it goes back to like having that conversation with your leadership. 414 00:22:15,895 --> 00:22:16,805 It was like, no, no, no, I want to. 415 00:22:17,215 --> 00:22:20,990 Increase my organic visits to the website by 20%, And you're 416 00:22:20,990 --> 00:22:23,220 like making shit up to do that. 417 00:22:23,510 --> 00:22:26,190 But why are we doing that? 418 00:22:26,280 --> 00:22:28,740 Like, why does 20, why is 20 percent the right number? 419 00:22:28,831 --> 00:22:31,311 Justin Norris: basically The content engagement is a useful proxy 420 00:22:31,311 --> 00:22:32,561 of how valuable the content is. 421 00:22:32,561 --> 00:22:34,671 But if you're putting out content about like how to pick the winning 422 00:22:34,671 --> 00:22:37,751 lottery numbers, you shouldn't necessarily expect more people to buy 423 00:22:37,751 --> 00:22:39,401 your contract management software. 424 00:22:39,411 --> 00:22:40,711 Like it just doesn't work that way. 425 00:22:40,711 --> 00:22:40,871 And. 426 00:22:41,290 --> 00:22:44,271 that's logical reframing it as a strategy rather than a measurement 427 00:22:44,271 --> 00:22:46,411 question, I think is interesting. 428 00:22:46,711 --> 00:22:50,809 So digging a little bit into this, and don't expect you to, like, unpack complex 429 00:22:50,869 --> 00:22:55,216 math here in a conversation, but how does it actually work under the hood? 430 00:22:55,351 --> 00:22:56,101 are the inputs? 431 00:22:56,101 --> 00:22:56,881 What are the outputs? 432 00:22:56,881 --> 00:22:58,293 And then how are teams? 433 00:22:58,293 --> 00:23:01,093 How are your customers actually making decisions using this information? 434 00:23:01,309 --> 00:23:02,359 Pranav Piyush: Yeah, there's a lot there. 435 00:23:02,359 --> 00:23:03,999 No, I think we should, we should talk about it. 436 00:23:03,999 --> 00:23:11,339 So if you visualize the question of marketing measurement as a formula, that's 437 00:23:11,339 --> 00:23:13,719 the best sort of, that I have found. 438 00:23:14,259 --> 00:23:17,219 So on the right hand side, you've got your. 439 00:23:18,104 --> 00:23:23,944 Success metric might be pipeline, might be sales, might be orders, might be leads. 440 00:23:24,015 --> 00:23:26,155 whatever the metric is that you're optimizing towards, And 441 00:23:26,155 --> 00:23:28,975 on the left hand side, you've got every single marketing channel. 442 00:23:30,345 --> 00:23:34,045 You've got a multiplier on that marketing channel that represents 443 00:23:34,045 --> 00:23:36,125 the strength of the correlation. 444 00:23:36,575 --> 00:23:36,915 Okay. 445 00:23:37,355 --> 00:23:42,625 And you have a few other variables that are representing seasonality. 446 00:23:43,115 --> 00:23:48,584 That are representing, the organic trend of your business that are representing 447 00:23:48,644 --> 00:23:51,674 other factors that may be outside of your marketing team's control. 448 00:23:52,384 --> 00:23:55,494 When you sum up all of those things on the left hand side. 449 00:23:55,834 --> 00:23:57,874 You're trying to predict the right hand side. 450 00:23:58,494 --> 00:24:00,074 That's what's happening under the hood. 451 00:24:00,384 --> 00:24:01,444 And I'm not even kidding. 452 00:24:01,454 --> 00:24:04,554 If you actually go look up the academic work, that's literally 453 00:24:04,554 --> 00:24:05,574 what's happening under the hood. 454 00:24:05,764 --> 00:24:08,294 So it's just a way to solve that formula. 455 00:24:08,794 --> 00:24:13,274 And you're applying a whole bunch of machine learning to predict or to 456 00:24:13,284 --> 00:24:18,154 simulate, based on all the data that you have, the answer to that equation. 457 00:24:18,960 --> 00:24:21,890 So how do customers make use of this? 458 00:24:21,920 --> 00:24:27,420 When you get output out of a, an MMM, you essentially get a few 459 00:24:27,420 --> 00:24:31,370 different sort of interesting, tidbits of information you understand. 460 00:24:31,670 --> 00:24:37,240 The percentage of your success metric that came from a certain channel, or that can 461 00:24:37,240 --> 00:24:39,570 be attributed back to a certain channel. 462 00:24:39,580 --> 00:24:40,850 And again, this is an estimate. 463 00:24:41,590 --> 00:24:45,760 You understand the cost, obviously, of acquiring an incremental 464 00:24:45,760 --> 00:24:47,210 conversion from that channel. 465 00:24:47,931 --> 00:24:49,521 so you get that across every single channel. 466 00:24:49,551 --> 00:24:51,581 And you also get what is known as a baseline. 467 00:24:52,101 --> 00:24:52,981 baseline. 468 00:24:53,001 --> 00:24:55,191 is essentially your organic demand. 469 00:24:56,011 --> 00:24:58,891 Your word of mouth, your brand equity. 470 00:24:59,011 --> 00:25:03,091 These are all sort of things that were not driven by marketing 471 00:25:03,091 --> 00:25:04,761 or sales in the short term. 472 00:25:04,771 --> 00:25:07,501 These are sort of longer term things that are happening in your business. 473 00:25:07,977 --> 00:25:13,667 And then you also get a sense of what a future might look like if you were 474 00:25:13,667 --> 00:25:15,977 to invest more in every single channel. 475 00:25:16,177 --> 00:25:18,487 Again, this is like, you can think of it as a forecast. 476 00:25:18,837 --> 00:25:23,427 If I put another 10, 000 into Metta, what will that do to my success metric? 477 00:25:23,427 --> 00:25:23,477 Okay. 478 00:25:24,732 --> 00:25:29,292 And you can imagine all of this being shown to you on a monthly or a 479 00:25:29,292 --> 00:25:33,222 weekly basis, depending on how often you are refreshing your analysis. 480 00:25:33,591 --> 00:25:34,471 So that's what you get. 481 00:25:34,561 --> 00:25:39,391 So now you have a very rich understanding of, as you have increased or decreased 482 00:25:39,391 --> 00:25:43,641 your spend and your strategies in different channels, What has that done 483 00:25:43,641 --> 00:25:45,551 in terms of contribution to your metric? 484 00:25:46,041 --> 00:25:49,811 And if you invest more, what is the likelihood of that 485 00:25:49,821 --> 00:25:51,551 increasing even further or not? 486 00:25:52,971 --> 00:25:56,651 So our customers will use that to inform a series of experiments. 487 00:25:57,431 --> 00:26:02,452 Where you take the most efficient channels and you figure out, Oh, if 488 00:26:02,452 --> 00:26:06,732 this channel is looking so good, can I just dump another million dollars here? 489 00:26:08,082 --> 00:26:09,682 What is the point of diminishing return? 490 00:26:09,732 --> 00:26:10,002 Right? 491 00:26:10,222 --> 00:26:13,632 So you run an experiment, run an actual incrementality test to test 492 00:26:13,632 --> 00:26:18,112 that hypothesis, and that becomes an ongoing set of experiments that 493 00:26:18,112 --> 00:26:21,622 you're constantly running every month, every quarter, every year. 494 00:26:22,122 --> 00:26:24,292 And that's essentially your marketing roadmap. 495 00:26:25,397 --> 00:26:29,427 For other channels where it's highly inefficient, you've spent a lot of 496 00:26:29,427 --> 00:26:32,707 money, but it's not statistically correlated with your success metric, 497 00:26:32,877 --> 00:26:34,097 you might have a different hypothesis. 498 00:26:34,097 --> 00:26:38,057 Maybe we need to pull back on spending, or maybe we need to change 499 00:26:38,057 --> 00:26:42,717 the creative execution in that channel completely, Maybe static ads 500 00:26:42,717 --> 00:26:45,817 are not good, maybe we need video ads, maybe thought leadership ads. 501 00:26:45,817 --> 00:26:49,336 I'm just making this stuff up, So, the question is not necessarily 502 00:26:49,336 --> 00:26:52,716 to cut spend, the question is to find the winning channel. 503 00:26:53,021 --> 00:26:56,361 Combination that'll help you extract even more growth. 504 00:26:56,791 --> 00:27:00,261 And the only way to do that is to constantly be experimenting. 505 00:27:00,931 --> 00:27:04,851 So summarizing, when you run an MMM, you get a whole bunch of output. 506 00:27:05,151 --> 00:27:09,771 Think of those outputs as informing your experimentation roadmap, and then go 507 00:27:09,771 --> 00:27:11,721 out and experiment on a monthly cadence. 508 00:27:12,382 --> 00:27:15,585 , Justin Norris: I want to steel man your case here because on, on many 509 00:27:15,585 --> 00:27:19,624 levels, I would, I would love for this to be like the answer not, not that 510 00:27:19,624 --> 00:27:22,964 I have a horse in this race, but the, just the status quo was so bad for 511 00:27:22,964 --> 00:27:26,124 marketing attribution that it would be wonderful if this was the solution. 512 00:27:26,124 --> 00:27:28,276 So It feels like a big company thing. 513 00:27:28,336 --> 00:27:30,076 It feels like, yeah, all right. 514 00:27:30,076 --> 00:27:32,866 If I've got millions of dollars to play around, we're like an extra million 515 00:27:32,866 --> 00:27:34,046 dollars and spend for an experiment. 516 00:27:34,056 --> 00:27:34,616 Like, sure. 517 00:27:34,956 --> 00:27:38,186 But if I'm a smaller company or even I'm at like a scale upstage 518 00:27:38,216 --> 00:27:42,186 company, 400 people, 50 million ARR, I'm picking a fake number. 519 00:27:42,206 --> 00:27:42,976 It's not the real number. 520 00:27:42,986 --> 00:27:46,019 Let's say a 5 million budget or even a 10 million marketing budget. 521 00:27:46,019 --> 00:27:48,598 You don't necessarily have that sort of wiggle room. 522 00:27:48,598 --> 00:27:49,458 So what do 523 00:27:49,458 --> 00:27:53,194 Pranav Piyush: first off, you're absolutely right that if you are 524 00:27:53,194 --> 00:27:57,224 just spending 100, 000 a year, all of the stuff that I said 525 00:27:57,244 --> 00:27:58,684 is way too complicated for you. 526 00:27:58,924 --> 00:28:00,084 And I do not recommend it. 527 00:28:01,354 --> 00:28:05,024 So this is not the go to methodology. 528 00:28:05,349 --> 00:28:11,139 For seed stage startups or even series a startups or, you know, your mom and pop 529 00:28:11,139 --> 00:28:13,659 store around the corner, that's not it. 530 00:28:13,759 --> 00:28:20,229 This is meant for when you have lots of channels and a lot of spend to optimize. 531 00:28:21,159 --> 00:28:22,389 That's the reality. 532 00:28:22,949 --> 00:28:24,269 Now, what is the, threshold? 533 00:28:24,719 --> 00:28:29,913 And, probably somewhere around a million or two, where you start to 534 00:28:29,913 --> 00:28:35,473 see that transition from mostly one channel to now many channels and 535 00:28:35,473 --> 00:28:37,763 this number will be different for different types of businesses, right? 536 00:28:37,763 --> 00:28:40,693 You could have an e commerce store where all you do is Facebook. 537 00:28:41,273 --> 00:28:41,703 That's it. 538 00:28:41,753 --> 00:28:44,783 That's your only distribution method and you have 10 million in spend 539 00:28:44,783 --> 00:28:48,503 on Facebook and you don't need to do attribution modeling because 540 00:28:48,503 --> 00:28:49,853 that's the only channel you have. 541 00:28:50,373 --> 00:28:51,413 There is nothing else. 542 00:28:51,753 --> 00:28:55,307 So, there's a little bit of like just fake, you know, I'm giving you some like 543 00:28:55,327 --> 00:28:57,037 lines, but they're not precise lines. 544 00:28:57,718 --> 00:29:02,678 Now, having said that, if I were, and Paramark doesn't do advertising 545 00:29:02,678 --> 00:29:08,428 just yet, we are going to start in Q4, maybe in Q1, and the fundamental 546 00:29:08,438 --> 00:29:10,898 way that we're going to do this is through an incrementality test. 547 00:29:12,888 --> 00:29:14,198 So here's the fun fact. 548 00:29:14,278 --> 00:29:17,498 You don't need to have MMM to be able to do incrementality testing. 549 00:29:17,508 --> 00:29:20,238 You can run an incrementality test, a geo test. 550 00:29:21,702 --> 00:29:22,772 So how does one do it? 551 00:29:22,832 --> 00:29:26,322 If I'm spending for the first time ever, I can look at all of 552 00:29:26,322 --> 00:29:30,132 my traffic today and where that traffic comes from by geography. 553 00:29:30,532 --> 00:29:33,552 I don't need any privacy type of software to do that. 554 00:29:33,592 --> 00:29:37,461 That's just IP address matching with geolocation, Now I can see 555 00:29:37,461 --> 00:29:40,051 that, Oh, like 30 percent of my traffic is coming from California. 556 00:29:40,661 --> 00:29:44,031 The remaining is split across these five states, New York, 557 00:29:44,051 --> 00:29:46,121 Texas, Washington, whatever. 558 00:29:47,041 --> 00:29:52,051 I'm going to run this campaign just in Texas, and see if my traffic goes up, 559 00:29:52,801 --> 00:29:58,881 do my demos go up, does my pipeline go up, based on the location of the 560 00:29:58,891 --> 00:30:00,136 people who are entering the country. 561 00:30:00,676 --> 00:30:01,146 The funnel. 562 00:30:02,276 --> 00:30:02,736 That's it. 563 00:30:02,786 --> 00:30:04,746 I don't need any complicated software to do this. 564 00:30:05,316 --> 00:30:08,626 So the interesting thing is, everything that I just talked about, 565 00:30:08,806 --> 00:30:13,096 you don't need software to do it on your own at a smaller scale. 566 00:30:13,116 --> 00:30:14,156 You can do it yourself. 567 00:30:14,206 --> 00:30:18,786 You need basic math skills and basic, you know, understanding of how to do 568 00:30:18,796 --> 00:30:21,066 geotargeting in all the ad platforms. 569 00:30:21,626 --> 00:30:24,126 so that's my answer for smaller stage companies. 570 00:30:24,776 --> 00:30:25,716 Justin Norris: That's really interesting. 571 00:30:25,770 --> 00:30:29,401 is there anything to be said about confounding variables as we 572 00:30:29,401 --> 00:30:31,023 record this, you know, terrible 573 00:30:31,023 --> 00:30:34,018 events with Hurricane Helene on the east coast. 574 00:30:34,042 --> 00:30:34,682 of, the U. 575 00:30:34,682 --> 00:30:34,842 S. 576 00:30:34,842 --> 00:30:38,092 that's gonna affect demand to some degree for some period of time. 577 00:30:38,282 --> 00:30:40,485 How do you adjust for that in these models? 578 00:30:40,789 --> 00:30:44,710 Pranav Piyush: listen, I think the, for smaller businesses, when you're 579 00:30:44,720 --> 00:30:48,630 just starting, you literally will know what's happening in your test market. 580 00:30:48,929 --> 00:30:51,659 you're not going to set it and then not look at the news for the next six 581 00:30:51,659 --> 00:30:52,219 weeks, Right? 582 00:30:52,749 --> 00:30:58,099 so you can always kind of reset if you run into any issues that have a 583 00:30:58,149 --> 00:30:59,999 potentially negative effect on your test. 584 00:31:00,079 --> 00:31:01,009 This happens all the time. 585 00:31:01,029 --> 00:31:02,989 Even in A B testing, you launch an A B test, like. 586 00:31:03,659 --> 00:31:06,269 Oh shit, there's a bug in our test version. 587 00:31:06,649 --> 00:31:09,389 Okay, we're gonna have to fix that bug and then relaunch the test. 588 00:31:10,159 --> 00:31:13,819 So, that's a very acceptable answer to that question. 589 00:31:14,225 --> 00:31:17,965 It gets a little bit more interesting when you're at a large scale. 590 00:31:18,415 --> 00:31:22,915 When you're at a large scale, you may not have your eyes on every single DMA, 591 00:31:22,925 --> 00:31:24,725 and every single state and location. 592 00:31:25,372 --> 00:31:26,232 That's impossible. 593 00:31:26,622 --> 00:31:32,552 And so the art is in constructing your test and control groups in a way that you 594 00:31:32,562 --> 00:31:35,022 minimize for the confounding variables. 595 00:31:35,492 --> 00:31:38,739 So you're not just looking at one whole, one DMA, one city. 596 00:31:38,739 --> 00:31:41,209 You're looking at a collection of test states. 597 00:31:41,499 --> 00:31:46,199 You're grouping them together that sometimes will avoid the 598 00:31:46,199 --> 00:31:50,980 noise that might come from like location specific variables. 599 00:31:51,570 --> 00:31:55,610 Even having said that, if you were doing this test in March of 2020. 600 00:31:56,905 --> 00:31:59,415 I would probably not look at the results of that test. 601 00:31:59,673 --> 00:32:04,403 So the point that I'm making there is you always have to use judgment. 602 00:32:04,663 --> 00:32:09,553 That's why us humans will have jobs for a very, very long time is you're applying 603 00:32:09,553 --> 00:32:14,263 human judgment on top of the data and not letting the data do the judgment for you. 604 00:32:14,416 --> 00:32:18,696 And that's how I think about, you know, confounding variables is you have to 605 00:32:18,696 --> 00:32:22,266 have a good hypothesis of what else could have happened that impacted this 606 00:32:22,266 --> 00:32:25,206 result that doesn't pass your intuition. 607 00:32:25,797 --> 00:32:26,877 Justin Norris: That makes a lot of sense. 608 00:32:26,996 --> 00:32:28,826 I want to bring it back. 609 00:32:28,826 --> 00:32:32,666 to the original posts that got us chatting on this topic. 610 00:32:32,666 --> 00:32:33,726 I was sharing. 611 00:32:33,726 --> 00:32:36,016 it was kind of like a history of one opportunity. 612 00:32:36,136 --> 00:32:37,016 One opportunity. 613 00:32:37,696 --> 00:32:39,216 Look at all these touch points. 614 00:32:39,226 --> 00:32:40,336 This is interesting. 615 00:32:40,656 --> 00:32:42,796 And I can't even remember what exactly I was saying about it, but 616 00:32:42,796 --> 00:32:44,606 I think I was saying, you know, this isn't representative, but 617 00:32:44,606 --> 00:32:45,746 it's still kind of interesting. 618 00:32:46,136 --> 00:32:48,246 It's useful as a communication tool for sales. 619 00:32:48,363 --> 00:32:49,513 it's 1 opportunity. 620 00:32:49,513 --> 00:32:52,907 It's not statistically valid now, let me ask you, am I deluding 621 00:32:52,907 --> 00:32:55,037 myself when I look at that and be like, Oh, this is interesting. 622 00:32:55,037 --> 00:32:56,137 Like, is it just irrelevant? 623 00:32:56,137 --> 00:32:56,717 Should we ignore it? 624 00:32:56,727 --> 00:33:00,917 What's your take on these like very local datasets and are they worth looking at? 625 00:33:02,057 --> 00:33:03,157 Pranav Piyush: It's a great question. 626 00:33:03,968 --> 00:33:06,063 humans are interesting creatures, right? 627 00:33:06,280 --> 00:33:07,560 we're visual in nature. 628 00:33:07,670 --> 00:33:08,990 We like to see things. 629 00:33:08,990 --> 00:33:10,370 We like to be able to touch things. 630 00:33:10,370 --> 00:33:11,850 We like to be able to feel things. 631 00:33:12,290 --> 00:33:15,620 And there's a natural tendency for us to do the same thing 632 00:33:15,630 --> 00:33:16,920 when it comes to analytics. 633 00:33:16,920 --> 00:33:21,641 If it's not on a, chart, it's really hard for us to visualize, right? 634 00:33:21,641 --> 00:33:25,411 Which is why MMM struggled so much because there's such like. 635 00:33:25,856 --> 00:33:31,106 Probabilistic statistical sort of type of things that it's people's intuitive mental 636 00:33:31,116 --> 00:33:33,146 models don't map to that way of thinking. 637 00:33:33,776 --> 00:33:37,596 so, that's what's going on in your brain when you see that laid out very 638 00:33:37,626 --> 00:33:41,466 neatly on a chart that this thing happened, this thing happened, this 639 00:33:41,466 --> 00:33:44,316 thing happened, this thing happened, and it gives you a sense of comfort, 640 00:33:44,456 --> 00:33:45,766 I know that this is what happened. 641 00:33:46,826 --> 00:33:47,266 So. 642 00:33:47,852 --> 00:33:50,302 are you deluding yourself? 643 00:33:52,914 --> 00:33:54,134 Maybe that's a strong word. 644 00:33:54,414 --> 00:34:01,414 what I would say is it's I don't think that it adds anything to your 645 00:34:01,464 --> 00:34:03,444 reality is what I'm going to say. 646 00:34:03,774 --> 00:34:06,304 think about the additional information that you got out of 647 00:34:06,304 --> 00:34:07,874 that that you didn't already know. 648 00:34:08,614 --> 00:34:13,944 So my way of thinking about this is if you are looking at that data, 649 00:34:14,034 --> 00:34:18,774 you're doing that to understand the behavioral journey of somebody. 650 00:34:19,239 --> 00:34:21,989 And that's literally like, this is what I did, and then this is what 651 00:34:21,989 --> 00:34:23,489 I did, and then this is what I did. 652 00:34:23,949 --> 00:34:27,159 If you view it from that lens, it's perfectly reasonable. 653 00:34:27,219 --> 00:34:31,248 Hey, we understand our customer's journey through the buying process. 654 00:34:31,859 --> 00:34:35,879 Here's the typical things that are involved in the behavior journey. 655 00:34:36,579 --> 00:34:40,029 But does that mean that there is cause and effect? 656 00:34:40,268 --> 00:34:40,978 Probably not. 657 00:34:41,768 --> 00:34:45,579 And those two things are very distinct things. 658 00:34:46,109 --> 00:34:49,569 And you have to just have an intellectually honest conversation 659 00:34:49,569 --> 00:34:51,159 about what are you actually looking at. 660 00:34:51,774 --> 00:34:53,081 Justin Norris: I think that's reasonable. 661 00:34:53,129 --> 00:34:55,708 the way you put it about the way people process information. 662 00:34:55,799 --> 00:34:58,754 William Carlos Williams, a modernist poet from the United States. 663 00:34:58,754 --> 00:35:01,544 He had this famous expression, no ideas, but in things. 664 00:35:02,054 --> 00:35:05,944 and I, I often think about that because I actually have a lot of trouble dealing 665 00:35:05,944 --> 00:35:07,914 with like mathematical abstractions. 666 00:35:08,384 --> 00:35:10,784 Uh, I really find that my insights and my understanding 667 00:35:10,784 --> 00:35:12,714 come from concrete particulars. 668 00:35:12,714 --> 00:35:13,054 So. 669 00:35:13,509 --> 00:35:16,739 Would it be valid to say, I'm looking at this journey, obviously there's nothing 670 00:35:16,739 --> 00:35:22,229 even correlative about one opportunity's journey, let alone causative, but I say, 671 00:35:22,229 --> 00:35:25,999 oh look, they're like, before they buy, all these people started attending these 672 00:35:25,999 --> 00:35:28,829 workshops we were having, maybe there's something to that, and like what you 673 00:35:28,829 --> 00:35:32,859 said for MMM, it could inform a larger scale experiment, is that an okay? 674 00:35:33,159 --> 00:35:34,279 Way in your opinion to look at it. 675 00:35:34,623 --> 00:35:35,403 Pranav Piyush: Absolutely. 676 00:35:35,723 --> 00:35:39,813 And I would also look at the individual things that are happening in that journey. 677 00:35:39,943 --> 00:35:40,213 Right? 678 00:35:40,213 --> 00:35:42,683 So, the one that you described, workshops. 679 00:35:43,233 --> 00:35:45,613 Workshops, webinars, events. 680 00:35:46,003 --> 00:35:49,563 These are real things that are happening in the world. 681 00:35:49,643 --> 00:35:53,203 As opposed to, I sent an email to my entire email database. 682 00:35:54,583 --> 00:35:59,353 And email showed up as a touch point in that journey, so you have to apply a human 683 00:35:59,353 --> 00:36:03,883 judgment when you're evaluating these journeys of like, what is really going? 684 00:36:04,763 --> 00:36:09,023 And I say this to many marketers, if you're not talking to your 685 00:36:09,033 --> 00:36:10,893 audience on a daily basis. 686 00:36:11,238 --> 00:36:12,398 What are you even doing? 687 00:36:12,748 --> 00:36:13,018 Right? 688 00:36:13,038 --> 00:36:14,658 You can't call yourself a marketer. 689 00:36:14,978 --> 00:36:20,814 So, the other part of this is like, let's break out of our charts and visuals and 690 00:36:20,814 --> 00:36:22,654 let's go talk to actual human beings. 691 00:36:23,024 --> 00:36:26,954 That's going to tell you a lot more about whether it was the email or 692 00:36:26,954 --> 00:36:30,754 the workshop that got them excited to engage with your buying process. 693 00:36:31,029 --> 00:36:33,149 Justin Norris: So even those conversations, it's another example 694 00:36:33,149 --> 00:36:37,569 of something that it doesn't really scale mathematically, but it's very 695 00:36:37,569 --> 00:36:41,709 rich in terms of giving ideas, get like it feeds the sort of intuitive, 696 00:36:41,964 --> 00:36:44,574 emotional side of your brain, you could say for lack of a better word. 697 00:36:44,766 --> 00:36:45,146 Pranav Piyush: Totally. 698 00:36:45,146 --> 00:36:47,016 And I go back to the PNG example, right? 699 00:36:47,036 --> 00:36:51,263 So tying it all the way back, MMM started with this partnership between, 700 00:36:51,263 --> 00:36:53,338 um, operators and academics at P. 701 00:36:53,338 --> 00:36:53,428 N. 702 00:36:53,428 --> 00:36:53,638 G. 703 00:36:54,008 --> 00:36:54,518 Guess what? 704 00:36:54,528 --> 00:36:54,668 P. 705 00:36:54,668 --> 00:36:54,778 N. 706 00:36:54,778 --> 00:36:54,878 G. 707 00:36:54,878 --> 00:36:59,508 Was also a pioneer in how you do customer research. 708 00:36:59,918 --> 00:37:04,368 They were spending time in people's homes, understanding how they used P. 709 00:37:04,368 --> 00:37:04,478 N. 710 00:37:04,478 --> 00:37:04,618 G. 711 00:37:04,618 --> 00:37:08,468 Products, and they still, to this day, have a huge team 712 00:37:08,468 --> 00:37:09,978 of people who just does that. 713 00:37:10,493 --> 00:37:16,973 So you can have both methodologies to understand the impact of your products 714 00:37:16,973 --> 00:37:21,763 and your customers lives and how they make decisions at an individual level, 715 00:37:21,783 --> 00:37:26,263 but then really get to understand them and talk to them and observe them and their 716 00:37:26,263 --> 00:37:32,293 reality and also look at the aggregate impact of your marketing strategies on 717 00:37:32,423 --> 00:37:34,418 buying behaviors and your sales metrics. 718 00:37:34,798 --> 00:37:36,980 It's not an either or it's a both. 719 00:37:37,515 --> 00:37:38,894 so great line of questioning 720 00:37:39,089 --> 00:37:40,879 Justin Norris: So turning to incrementality again, you've, 721 00:37:40,899 --> 00:37:42,169 you've mentioned it a few times. 722 00:37:42,199 --> 00:37:45,709 I see some vendors using incrementality probably in ways 723 00:37:45,719 --> 00:37:48,659 that you would disapprove of, I'm sure that I've used incrementality. 724 00:37:49,179 --> 00:37:51,469 probably in ways that, that you would disapprove of in terms of, 725 00:37:51,962 --> 00:37:54,212 communicating about it internally and the. 726 00:37:54,797 --> 00:37:58,097 The example I wrote down, which I think was the same, uh, maybe the same example 727 00:37:58,097 --> 00:38:01,897 from your blog post, all right, two charts, people that viewed a blog content 728 00:38:02,037 --> 00:38:05,540 and people that didn't, look, the people that viewed blog content have a greater 729 00:38:05,540 --> 00:38:09,510 chance, greater likelihood of converting, like, that's kind of seems reasonable, 730 00:38:09,510 --> 00:38:12,080 it's better than multi touch, we're not just saying, like, because they viewed a 731 00:38:12,080 --> 00:38:15,000 blog, we're giving it some credit, which is probably the most reductive, we're at 732 00:38:15,000 --> 00:38:19,962 least dividing them into groups, why is this, not useful, let's say, not valid? 733 00:38:20,274 --> 00:38:25,114 Pranav Piyush: Yeah, it's a great question in that analysis you are comparing 734 00:38:25,144 --> 00:38:32,479 two cohorts or two groups both groups had the option of Viewing the content 735 00:38:32,549 --> 00:38:37,549 or not viewing the content one group chose to view the content one group 736 00:38:37,559 --> 00:38:42,774 chose not to view the content So these are very different Different groups 737 00:38:42,804 --> 00:38:49,674 by themselves, and therein lies the challenge of claiming incrementality. 738 00:38:51,814 --> 00:38:53,974 If you take a step back and think about the concept of 739 00:38:53,974 --> 00:38:55,214 incrementality testing, right? 740 00:38:55,234 --> 00:39:00,154 We talked about test and control groups for test and control groups for RCTs. 741 00:39:00,174 --> 00:39:01,834 Your groups have to be identical. 742 00:39:03,844 --> 00:39:10,434 So in an RCT world, in an incrementality testing world, you would have a test group 743 00:39:10,784 --> 00:39:15,054 where it's made up of both people who are viewing the blog and not viewing the blog 744 00:39:15,474 --> 00:39:18,114 and a control group where there is no blog 745 00:39:20,954 --> 00:39:25,064 because you're testing the effect of the existence of the blog. 746 00:39:26,179 --> 00:39:27,699 In getting to the conversion, 747 00:39:27,789 --> 00:39:30,229 Justin Norris: Not just did they, did they choose to do it because 748 00:39:30,229 --> 00:39:32,089 already they've self selected into, 749 00:39:32,809 --> 00:39:33,089 it's like 750 00:39:33,389 --> 00:39:38,839 Pranav Piyush: bias is exactly the term, so that's precisely it. 751 00:39:38,859 --> 00:39:41,999 Anytime you have a question of incrementality, the immediate question 752 00:39:41,999 --> 00:39:45,909 you have to have is what is the control and the control has to be the non 753 00:39:45,909 --> 00:39:50,729 existence of the marketing or the media or the asset that is in the test bucket. 754 00:39:51,669 --> 00:39:55,565 Otherwise, it wasn't a causal analysis at all. 755 00:39:55,919 --> 00:39:59,059 Justin Norris: and still on the subject of incrementality, I saw something, 756 00:39:59,107 --> 00:40:01,817 on LinkedIn the other day, which I actually found disturbing on some level. 757 00:40:02,117 --> 00:40:03,107 So I'd like you to comment on it. 758 00:40:03,537 --> 00:40:09,617 It was the comment that, or let's say you spend, you bid on AdWords, 759 00:40:10,277 --> 00:40:11,947 you acquire a customer that way. 760 00:40:12,417 --> 00:40:15,247 We're not saying that AdWords was the only thing responsible for that, but at least 761 00:40:15,247 --> 00:40:18,007 like common sense that you would feel, I'm going to give AdWords a little bit 762 00:40:18,007 --> 00:40:21,377 of credit and then someone in the, in the comment, again, I can't remember who's 763 00:40:21,377 --> 00:40:24,477 like, actually there could have been no incrementality whatsoever from doing that. 764 00:40:25,247 --> 00:40:28,527 And in the sense that spending that money actually gave you some 765 00:40:28,557 --> 00:40:31,147 sort of incremental lift that you weren't otherwise going to get. 766 00:40:31,657 --> 00:40:34,707 And that's disturbing, I think, because if you can't rely on the fact that this 767 00:40:34,707 --> 00:40:38,497 person came, we have a trackable touch point here that came in from this channel. 768 00:40:38,497 --> 00:40:41,967 If we can't rely on that to say that at least We're getting something from this. 769 00:40:42,634 --> 00:40:43,364 what can we rely on? 770 00:40:43,374 --> 00:40:43,984 What can we trust? 771 00:40:43,984 --> 00:40:46,764 So, you know, it undermines a lot of the ways of thinking. 772 00:40:46,794 --> 00:40:49,724 In other words, that I think are very standard, very accepted. 773 00:40:50,204 --> 00:40:52,174 tell us about incrementality from that point of view. 774 00:40:52,843 --> 00:40:54,453 Pranav Piyush: Yeah, so I'm going to play it back to you, right? 775 00:40:54,523 --> 00:40:59,253 So the example is somebody bids on an AdWord, somebody clicks on 776 00:40:59,253 --> 00:41:03,263 that AdWord, comes and converts on your, you know, website. 777 00:41:03,673 --> 00:41:04,583 Is that incremental? 778 00:41:04,583 --> 00:41:05,673 Is it not incremental? 779 00:41:06,078 --> 00:41:06,358 Justin Norris: Yeah. 780 00:41:06,463 --> 00:41:08,673 Pranav Piyush: I think it's the devil's in the details a little bit. 781 00:41:08,673 --> 00:41:12,253 I'll give you two examples, and I can prove incrementality or lack 782 00:41:12,253 --> 00:41:13,843 thereof in both examples, okay? 783 00:41:14,349 --> 00:41:18,833 If that AdWord purchase was for a branded keyword, And nobody 784 00:41:18,833 --> 00:41:20,453 else was bidding on that keyword. 785 00:41:20,740 --> 00:41:23,641 It's actually probably not incremental at all, right? 786 00:41:23,641 --> 00:41:25,511 Because there's no competition for that keyword. 787 00:41:25,561 --> 00:41:30,011 And if you hadn't bid on that keyword, your organic search result would 788 00:41:30,011 --> 00:41:32,216 probably have taken the top result. 789 00:41:32,636 --> 00:41:33,886 Now, here's the caveat. 790 00:41:34,146 --> 00:41:37,906 Maybe your organic is awful because you just are new and you haven't done 791 00:41:37,906 --> 00:41:43,256 anything and you know that your organic result is on page three, then obviously 792 00:41:43,256 --> 00:41:45,376 that AdWord click is incremental. 793 00:41:45,626 --> 00:41:47,466 There's no way somebody is going to page three. 794 00:41:48,086 --> 00:41:52,957 So the devils in the details, unfortunately, you can't have 795 00:41:52,957 --> 00:41:56,497 this conversation on LinkedIn post without all the additional context. 796 00:41:56,656 --> 00:41:57,896 Now let's take another example, right? 797 00:41:57,896 --> 00:42:03,306 If it's a non branded search keyword, It's a long tail keyword, non branded 798 00:42:03,326 --> 00:42:04,476 has nothing to do with your brand. 799 00:42:05,356 --> 00:42:07,766 I would argue it's very hard. 800 00:42:08,441 --> 00:42:11,271 To not be incremental in that bucket. 801 00:42:12,084 --> 00:42:14,564 Again, the same conversation would apply, right? 802 00:42:14,984 --> 00:42:18,804 For it to have been non incremental means that that person would have 803 00:42:18,804 --> 00:42:24,804 done a non branded search, found your organic page, click through. 804 00:42:25,134 --> 00:42:26,204 And then bought something. 805 00:42:26,654 --> 00:42:32,924 So your organic has to be a plus and generally speaking, that's like the 806 00:42:32,934 --> 00:42:34,504 chances of that happening are pretty low. 807 00:42:35,284 --> 00:42:39,434 So you have to view these things from the subjective lens. 808 00:42:40,349 --> 00:42:41,599 Now, here's my argument. 809 00:42:41,839 --> 00:42:44,559 Run that as an incrementality test, and you will know the answer. 810 00:42:45,069 --> 00:42:46,219 It's not that hard. 811 00:42:46,549 --> 00:42:52,089 If you limit your AdWord by to a certain geography, and you can literally 812 00:42:52,089 --> 00:42:55,569 compare and contrast that with a test geography and a control geography, 813 00:42:55,569 --> 00:42:59,489 and you can see the incrementality of whatever strategy that you're employing. 814 00:42:59,499 --> 00:43:00,349 If you're in doubt. 815 00:43:00,633 --> 00:43:03,093 If you're not in doubt, and you can have a reasonable conversation 816 00:43:03,103 --> 00:43:06,093 like this, you should be able to get to that pretty quickly. 817 00:43:06,725 --> 00:43:08,833 Justin Norris: That answer seems commonsensical, at least, and I 818 00:43:08,853 --> 00:43:12,143 think, in part, I'm probably being unfair as I'm asking you to justify 819 00:43:12,143 --> 00:43:15,529 somebody else's comment, which doesn't make a lot of sense, but perceiving, 820 00:43:15,529 --> 00:43:18,969 I suppose, we're chatting a little bit before the show, just about how 821 00:43:19,009 --> 00:43:22,749 MMM and sort of related comments are becoming much more mainstream. 822 00:43:22,779 --> 00:43:23,814 I suppose they already do. 823 00:43:23,974 --> 00:43:27,729 We're, mainstream for decades, as you were saying, but sort of unknown, 824 00:43:27,739 --> 00:43:30,559 like running along this parallel track to the world of B2B SaaS, all 825 00:43:30,559 --> 00:43:32,179 of a sudden, they start poking in. 826 00:43:32,179 --> 00:43:36,269 And even over the last six months, I see more and more other podcasts, more and 827 00:43:36,269 --> 00:43:37,889 more people popping up, more vendors. 828 00:43:37,899 --> 00:43:39,089 So it's really interesting time. 829 00:43:39,659 --> 00:43:44,596 And, in a lot of these discussions, MMM and brand, um, The notion of 830 00:43:44,596 --> 00:43:46,386 brand seemed to be fellow travelers. 831 00:43:46,386 --> 00:43:47,816 I don't suggest the 832 00:43:47,826 --> 00:43:51,436 same thing, but there's a, on the one hand, we have sort of 833 00:43:51,436 --> 00:43:55,036 like multi touch attribution, trackable short term activation. 834 00:43:55,036 --> 00:43:59,216 On the other hand, we have like brand long term and MMM. 835 00:43:59,216 --> 00:44:01,346 And so that's why I sort of correlate those things together. 836 00:44:01,786 --> 00:44:04,756 And I think it's really interesting conversation is for the longest time 837 00:44:04,766 --> 00:44:08,476 brand spend, maybe it was either derided or people were afraid to do 838 00:44:08,476 --> 00:44:09,866 it because you couldn't justify it. 839 00:44:10,231 --> 00:44:12,371 And now there's like a lot of people coming out, or at least a 840 00:44:12,371 --> 00:44:15,691 small group of vocal people coming out and saying unapologetically, 841 00:44:16,041 --> 00:44:17,381 brand is super important. 842 00:44:17,391 --> 00:44:18,831 We really need to do this. 843 00:44:18,851 --> 00:44:21,731 People are forming the consideration set before they even search. 844 00:44:21,731 --> 00:44:24,701 And maybe that's where that notion of is the search even incremental if they've 845 00:44:24,701 --> 00:44:25,951 already decided who they want to buy. 846 00:44:26,521 --> 00:44:27,841 so maybe just talk about brand. 847 00:44:27,841 --> 00:44:30,871 How do you view brand and how does it relate to the MMM discussion? 848 00:44:31,271 --> 00:44:35,233 Pranav Piyush: brand is another one of those words that has been maligned a lot. 849 00:44:35,383 --> 00:44:38,473 and the first conversation I have about brand is what is brand? 850 00:44:38,483 --> 00:44:39,543 What is branding? 851 00:44:39,553 --> 00:44:40,813 And what is brand marketing? 852 00:44:41,313 --> 00:44:43,983 These are three completely different things. 853 00:44:44,528 --> 00:44:45,748 And we conflate them. 854 00:44:45,848 --> 00:44:48,498 So when people, you know, ask me about brand is like, which version 855 00:44:48,498 --> 00:44:49,308 of brand are you talking about? 856 00:44:49,308 --> 00:44:54,858 Are you talking about brand as in the concept of brand, which is to me, the 857 00:44:54,858 --> 00:44:59,238 perception that an audience of people has about your product or service. 858 00:44:59,418 --> 00:45:00,678 It's merely a perception. 859 00:45:01,188 --> 00:45:06,244 or are you talking about brand marketing, which again, I hate as a term because it, 860 00:45:06,529 --> 00:45:09,489 way people think about brand marketing is, oh, it's just harder to measure marketing 861 00:45:09,746 --> 00:45:13,803 and I'll call it brand marketing, And so to me, brand marketing is no such thing. 862 00:45:13,983 --> 00:45:15,703 Every piece of marketing performs. 863 00:45:15,996 --> 00:45:18,166 You just didn't have a way of measuring it in the past. 864 00:45:18,496 --> 00:45:22,296 And so what MMMs and incrementality testing have done have made it 865 00:45:22,326 --> 00:45:26,366 possible for you to measure any type of marketing, whether, you 866 00:45:26,366 --> 00:45:29,066 know, you call it brand marketing or performance marketing or what have you. 867 00:45:29,066 --> 00:45:32,536 It doesn't matter to me if it's a billboard, if it's TV, if it's radio, if 868 00:45:32,536 --> 00:45:38,796 it's, you know, a feel good, hilarious ad on YouTube, whatever it is can be measured 869 00:45:38,796 --> 00:45:40,386 through incrementality testing or MMM. 870 00:45:41,406 --> 00:45:45,946 Now, if you flip the script and say, Oh, but I want to understand 871 00:45:45,956 --> 00:45:50,786 the impact of this type of marketing on my brand perception. 872 00:45:51,561 --> 00:45:55,086 Now, that's a very different thing because you're talking about 873 00:45:55,356 --> 00:45:58,916 peeking into somebody's brain and quantifying their perception. 874 00:45:59,511 --> 00:46:03,441 This is an incredibly hard thing to do, and it's almost not worth doing. 875 00:46:04,271 --> 00:46:09,571 So until Elon Musk commercializes Neuralink, and we all have direct access 876 00:46:09,571 --> 00:46:14,431 to each other's brains, I would recommend not trying to measure brand perception. 877 00:46:14,801 --> 00:46:18,811 and there are better things to measure than that to understand 878 00:46:18,811 --> 00:46:20,051 the impact on the long term. 879 00:46:20,271 --> 00:46:23,981 So, again, not a satisfying answer, but if you're talking about brand marketing, 880 00:46:24,481 --> 00:46:26,121 It's a myth that you can't measure it. 881 00:46:26,161 --> 00:46:27,381 You can absolutely measure it. 882 00:46:27,821 --> 00:46:30,971 If you're talking about brand perception, brand awareness, it 883 00:46:30,971 --> 00:46:32,941 gets a little bit more finicky. 884 00:46:32,941 --> 00:46:35,281 And I think it's very, very hard to do well, a 885 00:46:35,476 --> 00:46:38,647 Justin Norris: On the subject of brand based on your experience in this field 886 00:46:38,647 --> 00:46:43,104 and working with your clients, as a marketer, you know, I'm in operations. 887 00:46:43,104 --> 00:46:44,304 I work adjacent to marketers. 888 00:46:44,564 --> 00:46:45,544 You feel trapped. 889 00:46:45,899 --> 00:46:51,539 You feel like, all right, AdWords is safe, directly attributable, 890 00:46:51,549 --> 00:46:53,719 direct response channels are safe. 891 00:46:53,739 --> 00:46:54,989 I can justify them. 892 00:46:55,329 --> 00:46:58,005 As soon as I go into brand, it's very, very difficult. 893 00:46:58,065 --> 00:46:59,405 I don't have an MMM today. 894 00:46:59,845 --> 00:47:02,925 and yet if you don't invest in brand, what you find is everyone's 895 00:47:02,925 --> 00:47:04,365 going to fish at the same spots. 896 00:47:04,395 --> 00:47:06,825 There's the, whatever, 5 percent of people that are in market. 897 00:47:08,075 --> 00:47:10,875 Everyone's competing, throwing money and you can't scale those channels 898 00:47:10,945 --> 00:47:12,255 all of a sudden they're like, all right, we're going to increase 899 00:47:12,255 --> 00:47:14,685 your targets by 20 percent and here's another 20 percent budget. 900 00:47:14,995 --> 00:47:17,515 You know, I can't turn this tap open any wider. 901 00:47:17,515 --> 00:47:19,155 There's no more water coming out. 902 00:47:19,965 --> 00:47:21,275 So you need, you do. 903 00:47:21,275 --> 00:47:23,535 I believe strongly you do need to do something, whether you 904 00:47:23,545 --> 00:47:26,425 call it brand marketing or demand creation or whatever the terms are. 905 00:47:26,425 --> 00:47:28,885 I think they're all referring to a similar thing of trying to reach 906 00:47:28,885 --> 00:47:32,895 people that aren't in market today, get in their head so that they want to 907 00:47:32,955 --> 00:47:34,445 come to you when they are in market. 908 00:47:35,063 --> 00:47:37,153 How do you perceive the value of that? 909 00:47:37,163 --> 00:47:40,453 Does that, do you have any data that suggests that like, yes, this really 910 00:47:40,453 --> 00:47:41,703 is a very important thing to do. 911 00:47:41,943 --> 00:47:42,196 So 912 00:47:42,196 --> 00:47:42,866 Pranav Piyush: hundred percent. 913 00:47:43,236 --> 00:47:49,256 I may not like the word brand or brand marketing, but the concept is, a plus 914 00:47:49,346 --> 00:47:54,492 you have to be able to talk to people who are not actively searching for a thing 915 00:47:54,492 --> 00:47:59,082 in your category, because to your point, that's three or 5 percent of the market. 916 00:47:59,562 --> 00:48:01,602 For certain SAS categories, it's even lower. 917 00:48:02,142 --> 00:48:06,072 So for the remaining 95, 97%, what are you going to do? 918 00:48:06,082 --> 00:48:07,472 Wait until they get into market. 919 00:48:07,987 --> 00:48:11,847 And have already sort of decided and you're going to be playing from, behind 920 00:48:11,887 --> 00:48:17,247 the starting line, or do you want to be 10 or 20 percent ahead of the starting line? 921 00:48:17,647 --> 00:48:20,857 And that's the perfect way to talk about brand, right? 922 00:48:20,887 --> 00:48:25,787 Is you could be starting from ahead of the starting line if you spent enough time and 923 00:48:25,787 --> 00:48:27,987 energy sort of creating that perception. 924 00:48:28,725 --> 00:48:30,865 So I think it's incredibly important. 925 00:48:30,895 --> 00:48:34,235 I think more people need to figure out a way to measure those things. 926 00:48:34,255 --> 00:48:35,355 It's not that hard. 927 00:48:35,355 --> 00:48:40,479 Frankly, you know, the one thing that I would say is if you can get a great 928 00:48:40,529 --> 00:48:45,949 analyst on your team that is not being pulled in 1000 different directions at 929 00:48:45,949 --> 00:48:48,549 the same time, it can be a game changer. 930 00:48:49,104 --> 00:48:53,526 Or, there's so many consultants and vendors out there in the ecosystem, 931 00:48:53,836 --> 00:48:59,946 wink, wink, that you should really have somebody who can be an extension to your, 932 00:49:00,046 --> 00:49:04,606 you know, marketing analyst, marketing operations team, that's giving you 933 00:49:04,606 --> 00:49:08,333 that, firepower to go test new channels. 934 00:49:08,698 --> 00:49:12,058 That you don't have the internal sort of infrastructure to be able to do. 935 00:49:12,608 --> 00:49:15,038 And if you can make the case to go build that internal 936 00:49:15,038 --> 00:49:17,228 infrastructure, hell yes, go for it. 937 00:49:17,728 --> 00:49:19,408 But I find that that's a much harder. 938 00:49:20,308 --> 00:49:26,708 You know, hill to climb than getting an external solution just because 939 00:49:27,008 --> 00:49:29,678 hiring people is significantly harder in this new environment. 940 00:49:30,471 --> 00:49:32,161 Justin Norris: there are a lot of different vendors popping 941 00:49:32,161 --> 00:49:33,471 up into the space right now. 942 00:49:33,518 --> 00:49:34,128 you're one of them. 943 00:49:34,138 --> 00:49:36,258 Like, tell us about your vision for Paramark. 944 00:49:36,322 --> 00:49:37,147 why is it different? 945 00:49:37,147 --> 00:49:38,797 what are you doing that's special in this field? 946 00:49:39,074 --> 00:49:39,974 Pranav Piyush: I'd say three things. 947 00:49:40,054 --> 00:49:43,604 One is, I have a former VP of marketing myself. 948 00:49:43,999 --> 00:49:45,839 I've been in the hot seat. 949 00:49:46,009 --> 00:49:47,739 I've had to defend budgets. 950 00:49:47,779 --> 00:49:51,369 I've had to present to the board, uh, reported into the CMO. 951 00:49:51,749 --> 00:49:57,979 I've managed large budgets and I know how anxiety inducing that week before QBR is. 952 00:49:58,499 --> 00:50:01,529 When you have to stand up in front of everybody and talk about how 953 00:50:01,529 --> 00:50:04,579 marketing is driving the business, but you yourself are a little bit 954 00:50:04,579 --> 00:50:09,339 unsure about how it's exactly going, and it's not for the lack of trying. 955 00:50:09,339 --> 00:50:10,529 It's not for the lack of creative. 956 00:50:10,529 --> 00:50:12,369 It's not for the lack of strategy and vision. 957 00:50:12,649 --> 00:50:16,279 It's because your hands are tied behind your back because you don't 958 00:50:16,289 --> 00:50:19,759 have the internal infrastructure to do the measurement that you ought to. 959 00:50:19,994 --> 00:50:21,850 So that's our whole purpose. 960 00:50:22,285 --> 00:50:24,295 We are built for CMOs. 961 00:50:24,600 --> 00:50:27,590 Literally the name Paramount comes from being on the side of the marketer. 962 00:50:28,510 --> 00:50:33,090 We're never going to go and sell to CROs and CEOs and CPOs. 963 00:50:33,180 --> 00:50:37,600 Our single mission is to make CMOs successful and. 964 00:50:38,345 --> 00:50:42,495 Really position them back into the leadership role that they should 965 00:50:42,495 --> 00:50:48,275 have always been in through better measurement, through calmer measurement. 966 00:50:48,703 --> 00:50:50,753 not a death by a thousand paper cuts. 967 00:50:50,973 --> 00:50:52,933 So that's our vision. 968 00:50:52,933 --> 00:50:54,833 I don't get into features and capabilities. 969 00:50:54,833 --> 00:50:56,393 Like people can figure that out on their own. 970 00:50:56,633 --> 00:50:58,033 Justin Norris: I love features and capabilities. 971 00:50:58,033 --> 00:51:01,494 So I'm just going to ask, like, if I come to you, is it sort of like 972 00:51:01,864 --> 00:51:04,594 self service, like give me your data and, and all right, here's your 973 00:51:04,594 --> 00:51:05,949 bottle, go have fun with it, or. 974 00:51:06,059 --> 00:51:08,814 Or is it sort of blended with a, professional service, like 975 00:51:08,814 --> 00:51:11,694 to help interpret the data, to help run experiments. 976 00:51:11,704 --> 00:51:13,774 How are you partnering with clients in that way? 977 00:51:13,973 --> 00:51:15,003 Pranav Piyush: Complete white glove. 978 00:51:15,053 --> 00:51:17,433 We get you up and running in about four weeks. 979 00:51:17,613 --> 00:51:19,183 We hook into your warehouse. 980 00:51:19,223 --> 00:51:21,043 We hook into your ad platforms. 981 00:51:21,043 --> 00:51:22,183 We hook into spreadsheets. 982 00:51:22,893 --> 00:51:26,653 You have slack access or team's access or whatever you use. 983 00:51:27,133 --> 00:51:31,013 And every two weeks you have a call with a dedicated customer success rep that is 984 00:51:31,053 --> 00:51:35,243 literally white glove hand holding you through the entire process, interpreting 985 00:51:35,243 --> 00:51:39,643 the data, designing experiments, recommendations, And obviously you have 986 00:51:39,673 --> 00:51:41,583 all the data that you can possibly need. 987 00:51:41,733 --> 00:51:46,003 So dashboards and reports that help you slice and dice every single campaign, 988 00:51:46,013 --> 00:51:48,793 every single channel, every single time frame that you can possibly need. 989 00:51:48,996 --> 00:51:50,866 Justin Norris: setting realistic expectations, let's say, are 990 00:51:50,876 --> 00:51:51,986 people signing up and it's like. 991 00:51:52,851 --> 00:51:53,511 Oh my gosh. 992 00:51:53,511 --> 00:51:55,781 I found the way the world makes sense now. 993 00:51:55,781 --> 00:51:59,211 Like life as a marketer is amazing now or are there still, and it's 994 00:51:59,211 --> 00:52:02,591 fine if there are, but are there still like challenges, ambiguities, 995 00:52:02,631 --> 00:52:06,601 uncertainties, does all that just go away or is it still a fact of life? 996 00:52:06,849 --> 00:52:11,399 Pranav Piyush: Think of it as the calmness because you know the way to 997 00:52:11,399 --> 00:52:13,539 get to an answer is running experiments. 998 00:52:14,619 --> 00:52:18,369 So that confidence of if there is uncertainty or ambiguity 999 00:52:19,194 --> 00:52:20,194 Let's go run an experiment. 1000 00:52:21,104 --> 00:52:24,124 And that uncertainty will be removed in about six weeks. 1001 00:52:24,724 --> 00:52:30,555 So it's a different way of operating, which is coming from a, position of 1002 00:52:30,575 --> 00:52:35,785 strength and confidence and predictability and a way of knowing the answer. 1003 00:52:35,785 --> 00:52:38,655 Because you have experimentation as part of the platform. 1004 00:52:39,225 --> 00:52:43,585 If you were just doing marketing, mixed modeling or attribution modeling, which 1005 00:52:43,585 --> 00:52:45,045 is much more sort of looking back. 1006 00:52:45,970 --> 00:52:46,920 Well, I don't believe it. 1007 00:52:47,930 --> 00:52:51,230 How should I believe it and you get into these like philosophical 1008 00:52:51,230 --> 00:52:53,930 debates rather than no, no, no, let's go run an experiment. 1009 00:52:54,550 --> 00:52:57,550 It will be very clear what's happening at the end of that. 1010 00:52:57,906 --> 00:52:58,526 Justin Norris: It's interesting. 1011 00:52:58,526 --> 00:53:02,456 I've, uh, and I've, I've seen, you know, a good handful of vendors in this space. 1012 00:53:02,496 --> 00:53:07,286 Now you seem to be emphasizing the incrementality testing in your go 1013 00:53:07,286 --> 00:53:10,186 to market in a different way, or at least more, much more prominently than 1014 00:53:10,186 --> 00:53:11,476 others are, and there is something. 1015 00:53:11,996 --> 00:53:14,806 To your point about like, all right, I've got this model, but do I believe it? 1016 00:53:14,806 --> 00:53:16,996 But the, the testing, it feels active. 1017 00:53:17,026 --> 00:53:20,609 It feels like you're, going on, on offense, so to speak. 1018 00:53:20,609 --> 00:53:23,400 And like you said, you're resolving and puts agency in your 1019 00:53:23,400 --> 00:53:25,250 control, which is very comforting. 1020 00:53:25,703 --> 00:53:26,213 Pranav Piyush: Exactly. 1021 00:53:26,223 --> 00:53:27,473 You can do something about it. 1022 00:53:27,493 --> 00:53:29,643 You don't have to sit there and debate models. 1023 00:53:29,893 --> 00:53:30,863 Justin Norris: So let me just last question. 1024 00:53:30,863 --> 00:53:34,320 I'm just curious, you know, your own go to market, you have an unfair advantage. 1025 00:53:34,320 --> 00:53:37,130 You see all the data, you see, you see what's working in theory. 1026 00:53:37,670 --> 00:53:39,630 You know, you should be able to like, do anything. 1027 00:53:39,630 --> 00:53:41,700 Does that, of course I'm being somewhat facetious, but, how are you 1028 00:53:41,700 --> 00:53:44,913 thinking about, how you're getting the word out about what you're doing? 1029 00:53:45,325 --> 00:53:46,882 Pranav Piyush: I say this all the time. 1030 00:53:46,962 --> 00:53:47,842 Measurement is. 1031 00:53:48,287 --> 00:53:51,227 Is Robin creative is Batman. 1032 00:53:52,097 --> 00:53:56,262 So the secret to go to market, I could be sitting at a trove of data, 1033 00:53:56,602 --> 00:54:01,902 but to differentiate and to make a place in the audience's mind, 1034 00:54:01,912 --> 00:54:03,312 you have to have great creative. 1035 00:54:03,832 --> 00:54:09,632 So for us up until now, that has been our organic social activity and having 1036 00:54:09,652 --> 00:54:11,492 good high quality conversations. 1037 00:54:12,082 --> 00:54:16,902 On social some, you know, warm outbound and that's been sufficient to get us 1038 00:54:16,902 --> 00:54:19,502 from zero to one to go from one to five. 1039 00:54:19,502 --> 00:54:22,632 We're probably going to need to do something different and it's going to 1040 00:54:22,642 --> 00:54:28,632 be figuring out new ways of reaching B2B audiences that are not the cookie cutter. 1041 00:54:29,132 --> 00:54:30,902 And that's hard, right? 1042 00:54:30,982 --> 00:54:34,422 Everyone is kind of doing the same thing and we have some tricks 1043 00:54:34,422 --> 00:54:37,442 up our sleeve, but you know, I won't talk about that just yet. 1044 00:54:37,452 --> 00:54:38,852 You'll, you'll see it in market. 1045 00:54:38,872 --> 00:54:40,892 So hopefully you'll see it in market and you'll tell me if 1046 00:54:40,892 --> 00:54:41,742 you liked it or you didn't. 1047 00:54:42,011 --> 00:54:42,541 Justin Norris: that's great. 1048 00:54:42,571 --> 00:54:45,631 And I will say it's it's an exciting time just as an idea. 1049 00:54:46,499 --> 00:54:50,344 it makes me happy to see, what feels like more substantial conversations. 1050 00:54:50,344 --> 00:54:53,574 And I'm sure there's, there's a lot more debate to be had around measurement. 1051 00:54:53,574 --> 00:54:57,954 It's not just going to resolve itself, but at least bringing marketing a little bit 1052 00:54:57,954 --> 00:55:02,084 further, at least B2B marketing further along the maturity curve to a place where 1053 00:55:02,084 --> 00:55:07,039 it isn't like, Starting from zero where every day is kind of day zero of how 1054 00:55:07,039 --> 00:55:09,699 should we measure this thing, which really does feel like for the last 10 years. 1055 00:55:10,293 --> 00:55:11,063 it's been like that. 1056 00:55:11,063 --> 00:55:13,775 So, but thank you so much for coming on the show. 1057 00:55:13,775 --> 00:55:15,015 This was really, really interesting. 1058 00:55:15,395 --> 00:55:18,345 wish you well, we'll watch yearly to see how Paramount does going forward. 1059 00:55:18,656 --> 00:55:22,206 Pranav Piyush: Thank you for having me and great work on this podcast. 1060 00:55:22,206 --> 00:55:22,586 It's awesome.