1 00:00:14,590 --> 00:00:17,060 Justin Norris: Every marketing team wants attribution. 2 00:00:17,080 --> 00:00:21,340 But the weird thing is that it's often not that satisfying when they actually get it. 3 00:00:21,660 --> 00:00:25,340 I did a lot of multi-touch attribution projects as a consultant, and we got 4 00:00:25,340 --> 00:00:29,380 really good at implementing these tools, technically creating taxonomies, 5 00:00:29,520 --> 00:00:31,060 making sure that data was clean. 6 00:00:31,640 --> 00:00:35,740 But I found that when you actually show these reports to a c-level executive, 7 00:00:36,080 --> 00:00:37,540 it can be kind of underwhelming. 8 00:00:37,890 --> 00:00:40,180 They don't always pass the common sense test. 9 00:00:40,280 --> 00:00:45,500 People want to pick on details, and maybe that's because the idea of dividing up an 10 00:00:45,500 --> 00:00:49,580 opportunity like a pizza between different touch points isn't actually the way. 11 00:00:49,590 --> 00:00:52,540 Maybe that's not going to give us the answers we need, 12 00:00:53,120 --> 00:00:54,340 but where does that leave us? 13 00:00:54,340 --> 00:00:58,300 Because we still need to know what's working in marketing, and today's guest 14 00:00:58,300 --> 00:01:01,860 thinks there's a better way, and he's founded a company to make marketing mix 15 00:01:02,020 --> 00:01:07,100 modeling or MMM available to B2B marketers and he's gonna tell us all about it. 16 00:01:07,160 --> 00:01:10,620 So I'm super excited to welcome Mark Stouse, CEO of 17 00:01:10,725 --> 00:01:12,250 proof analytics to the show. 18 00:01:12,390 --> 00:01:13,890 Thanks a lot for being here, mark 19 00:01:14,265 --> 00:01:15,205 Mark Stouse: Hey, thank you so much. 20 00:01:15,655 --> 00:01:17,135 Justin Norris: Mark, maybe one clarifying question. 21 00:01:17,445 --> 00:01:21,985 I've seen MMM, spelled out as media mix modeling and marketing mix modeling, which 22 00:01:21,985 --> 00:01:23,505 is the right way from your point of view? 23 00:01:24,265 --> 00:01:24,386 Mark Stouse: actually. 24 00:01:24,606 --> 00:01:30,416 It really represents the evolution of it over the last say, 40 to 45 years. 25 00:01:31,036 --> 00:01:37,336 Back when Procter and Gamble first brought out what was then called econometric, I. 26 00:01:37,713 --> 00:01:40,089 Analysis it was advertising. 27 00:01:40,263 --> 00:01:45,259 that's what it was really all about, hence the media mix modeling reference. 28 00:01:45,299 --> 00:01:51,503 As, time moved on channels proliferated, it became still very much within 29 00:01:51,613 --> 00:01:54,207 kind of B2C a reference point. 30 00:01:54,741 --> 00:01:56,443 It became marketing mix modeling. 31 00:01:57,130 --> 00:02:04,949 Today it is really go to market mix modeling because it includes not 32 00:02:04,979 --> 00:02:09,999 just marketing data and channels and investments and all that kinda stuff, but. 33 00:02:10,750 --> 00:02:17,307 Sales and customer success and product data outside data externalities, you 34 00:02:17,307 --> 00:02:22,657 know, the economy, your competitor actions, really is today much 35 00:02:22,687 --> 00:02:25,737 broader canvas as it should be. 36 00:02:25,867 --> 00:02:26,216 Right? 37 00:02:26,466 --> 00:02:33,200 That's the long and the short of why people say that MMM means media mix 38 00:02:33,440 --> 00:02:34,880 modeling or marketing mix modeling. 39 00:02:35,060 --> 00:02:39,019 And actually both of 'em are kind of A little bit passe today 40 00:02:39,399 --> 00:02:42,668 Justin Norris: Maybe let's take a step back before that and talk about 41 00:02:43,102 --> 00:02:46,797 what was your professional experience leading up to founding proof analytics 42 00:02:46,857 --> 00:02:48,637 and what brought you to this direction? 43 00:02:49,156 --> 00:02:49,277 I. 44 00:02:49,583 --> 00:02:54,283 Mark Stouse: So I was actually pretty much a classic marketer and communicator. 45 00:02:54,933 --> 00:02:59,293 I've worked across all of the different subsets of marketing at one time or 46 00:02:59,293 --> 00:03:04,590 another, and I've been a large company, CMO, . About a little less than 20 47 00:03:04,590 --> 00:03:09,373 years ago I was at HP and we were all kind of in the middle of an existential 48 00:03:09,373 --> 00:03:17,808 crisis because the then CEO of hp, mark Hurd was a very Operations focused 49 00:03:17,908 --> 00:03:19,928 and a very customer focused CEO. 50 00:03:20,458 --> 00:03:27,028 And he wanted to know why there wasn't more evidence of what marketing was 51 00:03:27,548 --> 00:03:29,498 actually delivering to the company. 52 00:03:29,908 --> 00:03:32,567 It was actually incredibly unpleasant. 53 00:03:32,825 --> 00:03:36,808 And about the only good thing that I can say about that whole experience that I had 54 00:03:36,808 --> 00:03:43,312 and that other peers of mine had was that it was, was highly motivating to change. 55 00:03:44,116 --> 00:03:48,250 I kind of got to a point where I said to myself, look, I either have 56 00:03:48,250 --> 00:03:53,890 to do something to fix this or I just need to like go do something 57 00:03:53,890 --> 00:03:58,203 else, 'cause it's not just about budget issues and all that kinda stuff. 58 00:03:58,273 --> 00:04:01,265 It's about credibility, am I actually doing here? 59 00:04:01,932 --> 00:04:08,125 So in my particular case for whatever reason, herd gave me a project. 60 00:04:09,370 --> 00:04:12,990 To actually see what could be done to resolve this. 61 00:04:13,540 --> 00:04:18,779 And I didn't know Jack at that time about analytics. 62 00:04:19,240 --> 00:04:24,470 I was not even a math enthusiast, and and so I, I remember I went 63 00:04:24,540 --> 00:04:30,170 home that next Friday I went through all the stuff in my garage. 64 00:04:30,690 --> 00:04:36,693 . And found a couple of old math textbooks from university and started 65 00:04:36,693 --> 00:04:40,053 to read them in ways that I never read them when I was in school. 66 00:04:40,713 --> 00:04:45,913 ' cause I was actively seeking an answer and all of a sudden I got to this and it 67 00:04:45,913 --> 00:04:48,873 was all about multi-variable regression. 68 00:04:49,488 --> 00:04:56,038 Which is absolutely the cornerstone of causal analytics ? It is the cornerstone 69 00:04:56,658 --> 00:04:58,438 of the scientific method of inquiry. 70 00:04:59,015 --> 00:05:04,285 It has so much credibility that, it'd be hard to have more credibility 71 00:05:04,665 --> 00:05:06,435 than multi-variable regression has. 72 00:05:07,145 --> 00:05:12,422 Not because it is perfect, but it is absolutely the best that we have. 73 00:05:13,042 --> 00:05:18,303 And today, is actually the bedrock of causal ai. 74 00:05:18,879 --> 00:05:25,899 So I started working on this project for Herd, and he liked it a lot. 75 00:05:26,159 --> 00:05:30,916 And I guess my prize was he set up a mentoring session. 76 00:05:32,056 --> 00:05:35,366 With the CFO of hp, Bob Leman. 77 00:05:35,643 --> 00:05:42,793 The whole idea was, I want you guys to talk you from your end and he from his end 78 00:05:42,843 --> 00:05:47,903 and see if we can't get to a meeting of the mines between marketing and finance. 79 00:05:48,543 --> 00:05:51,343 This whole experience it just changed me. 80 00:05:51,963 --> 00:05:55,473 And before he passed away, I had the opportunity to talk to Mark 81 00:05:55,473 --> 00:05:57,593 again and thank him for it, right. 82 00:05:57,593 --> 00:06:03,811 Because it was, not fun at all, but he did me the hugest favor. 83 00:06:03,996 --> 00:06:08,299 So I started climbing the analytics ladder, right? 84 00:06:08,399 --> 00:06:10,099 The stairway to heaven, so to speak. 85 00:06:10,189 --> 00:06:10,539 Right? 86 00:06:10,669 --> 00:06:15,309 By the time I was CMO at Honeywell aerospace, pre any kind of 87 00:06:15,759 --> 00:06:17,749 automation for analytics, right? 88 00:06:17,986 --> 00:06:22,986 meant we had to hire a ton of people in order to get the latency on the 89 00:06:22,986 --> 00:06:29,786 recalculation down, the scalability up the cost be damned at this point, right? 90 00:06:30,066 --> 00:06:30,946 'cause it was so important. 91 00:06:31,596 --> 00:06:36,096 And get the understandability of the outputs to the point where the 92 00:06:36,296 --> 00:06:40,213 business goes, yeah, I not only get it and believe it, but I can make a 93 00:06:40,213 --> 00:06:44,476 better decision today than I could make before as a result of this. 94 00:06:44,616 --> 00:06:50,656 And then you kind of say, there's not very many companies that are willing to invest 95 00:06:51,106 --> 00:06:57,786 seven, eight, $9 million a year just in marketing analytics particularly in B2B. 96 00:06:58,596 --> 00:07:01,959 That point you start to realize that, that automation was gonna 97 00:07:01,959 --> 00:07:03,799 be absolutely indispensable. 98 00:07:04,130 --> 00:07:09,253 To solving the underlying root problems, if Data science has has a number 99 00:07:09,253 --> 00:07:13,583 of major problems in the business context, but they can kind of be summed 100 00:07:13,583 --> 00:07:16,343 up by the fact that it's too slow. 101 00:07:17,013 --> 00:07:20,810 So by the time they get you the insights, everything has moved on, 102 00:07:20,830 --> 00:07:22,410 you've already had to make the decision. 103 00:07:23,020 --> 00:07:27,840 The information is not comprehensible many times by normal people. 104 00:07:29,303 --> 00:07:30,433 it's not scalable. 105 00:07:31,053 --> 00:07:35,325 So you end up with three or four mega models that get updated 106 00:07:35,515 --> 00:07:37,525 once, maybe twice a year. 107 00:07:38,185 --> 00:07:40,145 And so not agile, right? 108 00:07:40,303 --> 00:07:47,136 But big opportunity to appropriately automate and bring ai into that 109 00:07:47,146 --> 00:07:51,696 whole thing and really elevate all of this to a new level of 110 00:07:51,696 --> 00:07:54,416 accessibility and meaningfulness. 111 00:07:54,956 --> 00:07:56,536 And so that's why we built proof. 112 00:07:56,987 --> 00:08:00,627 Justin Norris: And so I want to contrast MMM and regression based 113 00:08:00,627 --> 00:08:02,187 analytics that you're describing with. 114 00:08:02,437 --> 00:08:06,927 What most B2B marketers are probably familiar with, which is 115 00:08:06,927 --> 00:08:10,277 multi-touch attribution, that's what pretty much everybody does today. 116 00:08:10,907 --> 00:08:14,507 What is the problem with this form of attribution from your point of view? 117 00:08:14,507 --> 00:08:15,147 Or is there a problem? 118 00:08:15,147 --> 00:08:15,867 Is it still okay? 119 00:08:15,967 --> 00:08:17,387 Can they be complimentary? 120 00:08:17,759 --> 00:08:21,539 Mark Stouse: as long as it's accurate data, there's absolutely nothing wrong 121 00:08:21,809 --> 00:08:26,329 with having a really good understanding of the customer journey, patterns in the 122 00:08:26,489 --> 00:08:29,009 customer journey, things like that, right? 123 00:08:29,669 --> 00:08:34,182 And indeed, a lot of our customers will include customer journey 124 00:08:34,251 --> 00:08:36,261 data in regression models. 125 00:08:36,912 --> 00:08:37,642 It's valuable. 126 00:08:38,462 --> 00:08:46,449 The problem is that it is not fit for purpose for what it's being sold to do. 127 00:08:47,679 --> 00:08:49,779 We can just start with the idea of bias. 128 00:08:50,485 --> 00:08:52,735 There's a lot of bias in that data. 129 00:08:53,265 --> 00:08:57,165 There's a lot of bias in the weightings of that data. 130 00:08:57,826 --> 00:09:01,806 So marketing teams set their own weightings in advance. 131 00:09:02,226 --> 00:09:07,196 It is antithetical to the mathematical principles, to say that you can 132 00:09:07,336 --> 00:09:11,036 use MTA data to optimize anything. 133 00:09:11,716 --> 00:09:12,956 Particularly spend 134 00:09:13,491 --> 00:09:15,881 Justin Norris: I just want to unpack that to understand why 135 00:09:16,415 --> 00:09:18,975 Mark Stouse: mTA data is an effect. 136 00:09:19,155 --> 00:09:20,415 You're measuring an effect. 137 00:09:21,012 --> 00:09:22,052 not capturing. 138 00:09:22,842 --> 00:09:23,802 The cause of that. 139 00:09:24,482 --> 00:09:28,222 And more specifically, the time lagged cause. 140 00:09:28,862 --> 00:09:35,262 So how are you going to optimize the money that you spent the past at this 141 00:09:35,262 --> 00:09:41,392 point, based on what you think you're seeing from effects in the present when 142 00:09:41,392 --> 00:09:43,432 you have no idea what the time lag is? 143 00:09:43,442 --> 00:09:43,792 Right? 144 00:09:43,852 --> 00:09:47,452 So there is this assumption, that the time lag is near zero, 145 00:09:48,128 --> 00:09:50,428 but that's just so not the case. 146 00:09:50,523 --> 00:09:50,943 Mm-hmm. 147 00:09:51,068 --> 00:09:54,610 I B2B, the time lags can be extensive. 148 00:09:54,800 --> 00:09:56,250 What do I mean by extensive? 149 00:09:56,840 --> 00:09:57,860 Two, three quarters. 150 00:09:58,220 --> 00:10:03,323 we get into brand investments, it's double that, exactly are you optimizing 151 00:10:03,383 --> 00:10:08,883 and how do you know which part of your budget back in time you needed to optimize 152 00:10:09,133 --> 00:10:12,433 based on this effect in the present? 153 00:10:12,683 --> 00:10:12,859 Right? 154 00:10:12,859 --> 00:10:15,019 Justin Norris: If we take an MTA scenario, just to make it like a 155 00:10:15,019 --> 00:10:18,139 little bit more clear, let's say we have an opportunity and we have 156 00:10:18,669 --> 00:10:21,376 three people in the buying committee that are part of that opportunity. 157 00:10:21,396 --> 00:10:24,816 And between those three people, there were sacred brown numbers, 30 158 00:10:24,816 --> 00:10:28,336 different touch points that they had with different marketing initiatives. 159 00:10:28,336 --> 00:10:30,816 And those could be like, they downloaded eBooks, they attended 160 00:10:30,816 --> 00:10:32,336 webinars, they clicked on ads. 161 00:10:32,716 --> 00:10:36,254 And so the, traditional way is like, we're gonna take those 30 touchpoints 162 00:10:36,514 --> 00:10:38,054 and we're gonna take that opportunity. 163 00:10:38,144 --> 00:10:41,414 Let's say it's a 3 million opportunity, and we're just gonna divvy it up. 164 00:10:41,434 --> 00:10:43,094 And the way that we divvy it up could vary. 165 00:10:43,154 --> 00:10:46,454 We may pick some touch points, is being more influential or not influential? 166 00:10:46,454 --> 00:10:49,654 And I totally agree with you there, that it's kind of like arbitrary that 167 00:10:49,654 --> 00:10:52,934 they give you different models that you can choose from and it's like, pick 168 00:10:52,934 --> 00:10:55,894 the one you want, like on what basis? 169 00:10:56,394 --> 00:10:59,034 Mark Stouse: I don't think that marketers understand how 170 00:10:59,944 --> 00:11:06,486 transparently untrue the premise of MTA is to everyone else but them. 171 00:11:07,206 --> 00:11:10,246 Justin Norris: I wanna get to the reality of the situation because this 172 00:11:10,246 --> 00:11:11,686 is the, the industry that we're in. 173 00:11:12,371 --> 00:11:15,711 Mark Stouse: The reason why it went through adoption is that most. 174 00:11:16,511 --> 00:11:22,801 Marketers don't have enough knowledge of math to be able to recognize 175 00:11:23,091 --> 00:11:24,451 when something doesn't work. 176 00:11:24,714 --> 00:11:31,094 And number two, it was presented to them as a way of mining data that they 177 00:11:31,094 --> 00:11:34,054 already were generating in their stacks. 178 00:11:34,584 --> 00:11:37,564 So it was easy in quotes, right. 179 00:11:38,114 --> 00:11:38,734 To do this. 180 00:11:39,324 --> 00:11:43,824 And I've been around it long enough and talked to enough marketers about 181 00:11:43,824 --> 00:11:50,026 it, everybody was shocked when they went into their first meeting with 182 00:11:50,266 --> 00:11:55,186 MTA data, Thinking that they were just gonna be greeted as conquerors 183 00:11:55,246 --> 00:11:59,346 and victors and to your earlier point at the top of the show, right. 184 00:11:59,526 --> 00:12:01,306 It was anything but that. 185 00:12:01,807 --> 00:12:05,457 it was very innervating for a lot of marketers. 186 00:12:06,187 --> 00:12:09,355 You can actually do it in Excel if you only have to do 187 00:12:09,355 --> 00:12:10,715 it on a very limited basis. 188 00:12:11,149 --> 00:12:20,217 But if B2B would just adopt B2C analytics and also I would just add market research 189 00:12:20,347 --> 00:12:26,946 priorities they would see a very rapid transformation of their credibility and 190 00:12:26,946 --> 00:12:28,266 their precision and everything else. 191 00:12:28,606 --> 00:12:28,826 And. 192 00:12:29,961 --> 00:12:35,441 can always modify it as needed, but the core principle is ready to 193 00:12:35,461 --> 00:12:37,968 go and has been a very long time. 194 00:12:38,238 --> 00:12:41,634 Justin Norris: So if I recap your perspective as I understand it, touch 195 00:12:41,634 --> 00:12:44,754 points, the customer journey data, what the person did and when that can 196 00:12:44,754 --> 00:12:46,474 be interesting, that can be valuable. 197 00:12:46,734 --> 00:12:50,634 The notion of then doling out credit to those touch points for revenue 198 00:12:51,074 --> 00:12:55,099 outcomes, not mathematically, statistically defensible. 199 00:12:55,364 --> 00:12:56,724 Mark Stouse: would just add this, right? 200 00:12:56,951 --> 00:12:59,951 MTA at the end of the day is about pattern matching. 201 00:13:00,221 --> 00:13:03,481 It's about identifying large scale, repeating patterns 202 00:13:03,941 --> 00:13:05,041 in the customer journey. 203 00:13:06,071 --> 00:13:10,594 And then the assumption is that if a bunch of people are doing the same thing 204 00:13:10,664 --> 00:13:12,874 over and over again, it must be causal. 205 00:13:13,297 --> 00:13:19,137 That is a complete fallacy, the same math is used for example, 206 00:13:19,237 --> 00:13:24,057 and in fact, pattern matching too is used to study climate change. 207 00:13:24,977 --> 00:13:29,747 Some are causal, some are absolutely not causal. 208 00:13:30,647 --> 00:13:33,977 They happen all the time, but they don't mean anything. 209 00:13:35,257 --> 00:13:39,827 You can't just assume that because, a thousand people do it. 210 00:13:40,377 --> 00:13:40,667 Okay. 211 00:13:41,287 --> 00:13:44,797 One of your channels, one of your feedback loops, you know, it just keeps happening 212 00:13:44,797 --> 00:13:48,430 that ipso facto means that it's causal. 213 00:13:48,920 --> 00:13:51,353 Justin Norris: I've always had this unease as well where you a great 214 00:13:51,353 --> 00:13:55,123 example from earlier in my career, we had a self-serve SaaS app, what 215 00:13:55,123 --> 00:13:56,483 would be called product led today. 216 00:13:57,023 --> 00:14:00,783 And everybody that signed up for that app got a welcome email. 217 00:14:01,003 --> 00:14:03,873 So from that point of view, if you're looking at touchpoints, like the 218 00:14:03,873 --> 00:14:05,393 welcome email is very influential. 219 00:14:05,393 --> 00:14:07,713 It's very important but, but everybody got it. 220 00:14:07,813 --> 00:14:11,313 So just because people got it and some people opened it doesn't mean you 221 00:14:11,313 --> 00:14:16,539 could necessarily infer that that was driving the outcome in a particular way, 222 00:14:16,976 --> 00:14:20,116 . So everything we've been discussing is kinda like the current state. 223 00:14:20,416 --> 00:14:26,329 Let's contrast it now with MMM and why does it not have these different 224 00:14:26,329 --> 00:14:27,689 shortcomings that we've described? 225 00:14:28,075 --> 00:14:31,635 Mark Stouse: Number one, let's start with the shortcomings that it does have, right? 226 00:14:31,895 --> 00:14:36,915 It is very dependent, as is every analytic of any kind. 227 00:14:37,205 --> 00:14:42,425 Very dependent on data quality, There's that old saying about gigo, right? 228 00:14:42,425 --> 00:14:43,705 Garbage in, garbage out. 229 00:14:44,439 --> 00:14:48,599 Everyone is going to be victimized by gigo equally. 230 00:14:49,179 --> 00:14:50,199 Beyond that though. 231 00:14:51,154 --> 00:14:53,814 It has a lot of major advantages. 232 00:14:54,260 --> 00:14:57,360 It is a lean data mathematical process. 233 00:14:57,940 --> 00:15:03,560 So you do not have to have big data or even a lot of lean data order 234 00:15:03,580 --> 00:15:06,560 to run these models accurately. 235 00:15:07,150 --> 00:15:09,130 And that is actually a huge factor. 236 00:15:09,370 --> 00:15:13,850 One of the things that CDOs chief digital officers or data officers are 237 00:15:13,850 --> 00:15:19,270 grappling with right now on private LLMs is that they big data stocks. 238 00:15:19,900 --> 00:15:22,230 They don't have enough training data and they don't have enough 239 00:15:22,300 --> 00:15:25,070 operational data to run private LLMs. 240 00:15:26,222 --> 00:15:30,526 Everyone's been so understandably fascinated with what they can do 241 00:15:30,556 --> 00:15:36,683 with public LLMs that they haven't really thought it through in terms 242 00:15:36,743 --> 00:15:39,883 of the limitations on private stuff. 243 00:15:39,883 --> 00:15:40,203 Right? 244 00:15:40,370 --> 00:15:42,083 regression does not have this problem. 245 00:15:42,333 --> 00:15:47,233 If you say, oh, I don't have enough data to run regression analytics. 246 00:15:47,860 --> 00:15:51,260 not even remotely able to do any kind of ai. 247 00:15:52,240 --> 00:15:55,686 And I think that a lot of c-suites now Are starting to say, your 248 00:15:56,126 --> 00:16:01,126 creation and maintenance of a high quality data pool that's relevant to 249 00:16:01,126 --> 00:16:04,916 your function is a core competency. 250 00:16:05,463 --> 00:16:10,043 And if you tell us that you don't have the right data, you don't trust your 251 00:16:10,043 --> 00:16:15,469 data that's on you, The other thing is you have to have some basic capability 252 00:16:15,609 --> 00:16:17,989 on the human side of the equation. 253 00:16:18,412 --> 00:16:23,956 So with proof, for example, we have simplified the whole thing, dramatically. 254 00:16:24,356 --> 00:16:26,915 Automated it significantly in the right places. 255 00:16:27,346 --> 00:16:31,306 you don't need a full-blown data scientist to run it. 256 00:16:32,046 --> 00:16:35,727 You need a competent data analyst and you really don't 257 00:16:35,727 --> 00:16:37,887 even need that person full time. 258 00:16:38,147 --> 00:16:41,407 That's probably a half FTE, right? 259 00:16:41,728 --> 00:16:45,168 Justin Norris: Could we give a Fisher Price version definition of 260 00:16:45,168 --> 00:16:49,328 what MMM is like, just so people can conceptualize it in their minds. 261 00:16:50,423 --> 00:16:55,020 Mark Stouse: we live in a multi-variable world there's tons of potential 262 00:16:55,200 --> 00:16:59,327 causes everything that you control and everything that you don't control. 263 00:16:59,787 --> 00:17:02,487 That's kind of a really super easy way to bucket it. 264 00:17:03,092 --> 00:17:08,352 And these all have interactions with each other across time and space 265 00:17:08,698 --> 00:17:11,188 that produce particular outcomes. 266 00:17:11,948 --> 00:17:14,998 Causally speaking this is a probabilistic calculation. 267 00:17:15,738 --> 00:17:22,708 So outside of certain physical laws like gravity, There is no deterministic outcome 268 00:17:22,728 --> 00:17:25,268 that's possible to determine, right? 269 00:17:25,448 --> 00:17:29,678 You'd have to know everything that there is to know about what 270 00:17:29,678 --> 00:17:33,655 contributes to a particular outcome to get to a deterministic answer. 271 00:17:34,435 --> 00:17:37,695 That's just not the way that operates, particularly when you're 272 00:17:37,695 --> 00:17:38,735 talking about human behavior. 273 00:17:39,428 --> 00:17:46,985 So this is the same math that's being used routinely to study climate 274 00:17:46,985 --> 00:17:49,265 change, to study epidemiology. 275 00:17:50,235 --> 00:17:53,655 To study economics, it captures time lag. 276 00:17:54,115 --> 00:17:58,951 So in the end, the report will tell you historically the stack rank 277 00:17:59,211 --> 00:18:04,191 of everything that you control and don't control, that's in the model. 278 00:18:04,649 --> 00:18:07,099 It's relative effect on this outcome. 279 00:18:07,808 --> 00:18:12,011 It will then forecast all that into the future so that you can 280 00:18:12,011 --> 00:18:13,171 then make different choices. 281 00:18:13,631 --> 00:18:18,611 And because of automation, we've sped that up to the point 282 00:18:18,621 --> 00:18:23,552 where you can run multi-variable regression exactly like GPS. 283 00:18:23,972 --> 00:18:29,592 say that you're recalculating the model every week, new data comes in and is 284 00:18:29,592 --> 00:18:34,352 presented to the model automatically, it automatically recalculates. 285 00:18:34,652 --> 00:18:39,739 And you see how the present now is comparing with the forecast 286 00:18:39,929 --> 00:18:41,339 that you have for the same period. 287 00:18:41,742 --> 00:18:44,029 if there's a growing delta, right? 288 00:18:44,079 --> 00:18:46,499 You see, hey, okay, I need to make some changes. 289 00:18:46,699 --> 00:18:48,769 Stick with the GPS analogy. 290 00:18:49,769 --> 00:18:50,569 need to reroute. 291 00:18:51,069 --> 00:18:52,179 Or man, this is great. 292 00:18:52,249 --> 00:18:53,459 It's just totally tracking. 293 00:18:53,879 --> 00:18:59,009 And it will tell you how long, longer it's going to take for you to reach your 294 00:18:59,129 --> 00:19:01,369 objective, your goal, your destination. 295 00:19:01,882 --> 00:19:02,942 So there's a countdown. 296 00:19:03,160 --> 00:19:08,817 Regression is a huge part of the actual GPS that's on your 297 00:19:08,817 --> 00:19:10,237 phone, you use every day. 298 00:19:10,684 --> 00:19:14,304 And if you stop and think about it for a second, most business 299 00:19:14,584 --> 00:19:19,024 questions, and indeed a lot of life questions are navigation questions. 300 00:19:19,594 --> 00:19:20,207 Where am I? 301 00:19:20,257 --> 00:19:21,327 Where do I want to go? 302 00:19:21,787 --> 00:19:23,007 What's the best way to get there? 303 00:19:24,047 --> 00:19:28,367 I have enough time and resources to achieve my destination? 304 00:19:29,447 --> 00:19:31,947 Am I gonna run outta gas before I get there? 305 00:19:32,437 --> 00:19:34,701 I gonna run out of time before I get there? 306 00:19:34,801 --> 00:19:37,287 Am if I have to be there at nine o'clock for a meeting? 307 00:19:37,747 --> 00:19:39,967 And the GPS says, you're not getting there until 11. 308 00:19:40,657 --> 00:19:43,637 gonna have to make some choices, that's part of it. 309 00:19:44,587 --> 00:19:49,677 also captures all the headwinds and tailwinds that may be speeding 310 00:19:49,737 --> 00:19:50,997 you up or slowing you down. 311 00:19:51,257 --> 00:19:55,471 So in many ways, it it gives you the answer to your questions. 312 00:19:56,174 --> 00:19:59,014 Justin Norris: Take a practical example along the lines of one of those questions, 313 00:19:59,104 --> 00:20:04,533 let's say a company spends a hundred thousand dollars a month on display 314 00:20:04,633 --> 00:20:08,757 ads as an act of faith because those can be notoriously difficult to track. 315 00:20:08,757 --> 00:20:11,517 They're not always resulting in a click, but they could have 316 00:20:11,517 --> 00:20:13,117 impressions that could have an impact. 317 00:20:13,497 --> 00:20:16,677 So they're spending that a hundred K month over month, and the CFO 318 00:20:16,877 --> 00:20:18,837 challenges the CMO and says, why. 319 00:20:19,622 --> 00:20:21,482 Are we spending this 100 K every month? 320 00:20:21,482 --> 00:20:22,202 What's it doing? 321 00:20:22,662 --> 00:20:24,882 And that's the question the CMOs trying to answer. 322 00:20:25,502 --> 00:20:31,282 How would MMM, like what does it compare that spend versus a particular outcome? 323 00:20:31,382 --> 00:20:33,943 And try to show a causal relationship between them. 324 00:20:34,525 --> 00:20:34,796 Mark Stouse: yeah. 325 00:20:35,356 --> 00:20:36,436 Multiple variables, right? 326 00:20:36,616 --> 00:20:39,476 So it's highly context oriented. 327 00:20:39,816 --> 00:20:40,876 All of these models are. 328 00:20:41,641 --> 00:20:45,011 Seek to capture as much of a known context as possible. 329 00:20:45,847 --> 00:20:52,257 what a lot of marketing teams figure out using exactly that scenario, 330 00:20:52,940 --> 00:20:58,686 is they've been trying to justify that display ad budget in terms of 331 00:20:58,946 --> 00:21:04,825 demand gen, in terms of, performance marketing, And that is just not the 332 00:21:04,825 --> 00:21:07,305 way that that typically plays at all. 333 00:21:07,815 --> 00:21:11,385 That is a brand reputation investment. 334 00:21:12,004 --> 00:21:15,844 know, you kind of think about marketing expense as being 335 00:21:15,994 --> 00:21:17,924 essentially two big chunks, right? 336 00:21:18,074 --> 00:21:24,931 Branded demand brand is easily two to three times time lagged in its effect. 337 00:21:25,646 --> 00:21:27,016 Than demand is. 338 00:21:27,562 --> 00:21:29,822 It also sticks around a lot longer. 339 00:21:30,052 --> 00:21:34,822 The effects of brand investment doesn't deteriorate very quickly 340 00:21:35,602 --> 00:21:40,292 unless there is a major scandal of some sort, something like that, that all 341 00:21:40,292 --> 00:21:41,972 of a sudden breaches the trust wall. 342 00:21:42,472 --> 00:21:47,532 But absent that right, it hangs around The halflife is quite extended, 343 00:21:47,922 --> 00:21:52,292 whereas the Halflife on demand investment is highly perishable. 344 00:21:52,702 --> 00:21:58,052 Lasts maybe a couple months and then it's gone, so the snows melt 345 00:21:58,372 --> 00:22:00,892 a lot faster with demand than brand. 346 00:22:01,752 --> 00:22:08,072 You have to then say to finance, never discussed time lag with you 347 00:22:08,192 --> 00:22:10,552 before specifically, but we all know. 348 00:22:11,287 --> 00:22:13,937 That marketing takes time to have an effect. 349 00:22:14,747 --> 00:22:20,377 That simple statement is insufficient because if we don't know the 350 00:22:20,377 --> 00:22:22,377 time lag, we will never know. 351 00:22:22,837 --> 00:22:29,707 The ROI, so we now have the ability to say that this investment over 352 00:22:29,707 --> 00:22:35,507 here doesn't really drive demand, doesn't really pay off on this side. 353 00:22:35,517 --> 00:22:41,647 There isn't a quick return a demand perspective, but from a brand perspective, 354 00:22:42,477 --> 00:22:47,147 this is how it is improving average deal size and average deal velocity. 355 00:22:47,237 --> 00:22:51,187 Those are the two big ones that we see again and again on brand investment. 356 00:22:52,257 --> 00:22:54,637 It is grease on the wheel of the deal. 357 00:22:55,792 --> 00:22:59,572 That's what brand reputation really is, right? 358 00:22:59,592 --> 00:23:04,052 It makes people buy more than they otherwise would and buy 359 00:23:04,192 --> 00:23:06,332 faster than they otherwise would. 360 00:23:07,262 --> 00:23:12,112 that That is key in B2B because it's a higher cost, higher 361 00:23:12,302 --> 00:23:14,152 risk by decision to begin with. 362 00:23:15,052 --> 00:23:20,072 And then today, if we layer in all the risk factors that people are worried 363 00:23:20,072 --> 00:23:21,372 about it's even more important,? 364 00:23:22,092 --> 00:23:27,239 I mean, If you really want to understand how trusted you are by your customer 365 00:23:27,269 --> 00:23:29,799 base, look at your average deal velocity. 366 00:23:30,579 --> 00:23:35,859 If you really want to understand how confident people are in your product's 367 00:23:35,859 --> 00:23:41,259 ability to generate massive amounts of value for them, look at their average 368 00:23:41,329 --> 00:23:43,804 deal size, particularly year one, right? 369 00:23:43,804 --> 00:23:46,852 Certainly year two, your, a lot of people, just as a matter 370 00:23:46,852 --> 00:23:50,652 principle today, are buying small in year one and testing it out. 371 00:23:51,872 --> 00:23:59,332 If you see a major uptick in year two and deal size, that is a 372 00:23:59,342 --> 00:24:01,332 major vote of confidence in you. 373 00:24:01,872 --> 00:24:06,352 you don't, you need to really find out why that is. 374 00:24:07,152 --> 00:24:08,472 'cause I guarantee there's a reason. 375 00:24:09,162 --> 00:24:12,092 Justin Norris: Looking at our example, let's say the dataset that 376 00:24:12,092 --> 00:24:16,012 you had, the company had always been running those display ads. 377 00:24:16,072 --> 00:24:20,652 Do you need a period where those display ads are not there in order to demonstrate 378 00:24:20,792 --> 00:24:22,572 the impact of having them or not? 379 00:24:22,952 --> 00:24:27,212 Or is there a way, even though they've always been there, to somehow demonstrate 380 00:24:27,392 --> 00:24:28,572 the impact that they're having? 381 00:24:29,079 --> 00:24:34,099 Mark Stouse: I think that the answer to that question is very much contextual. 382 00:24:35,124 --> 00:24:40,614 if we were talking about a correlation analysis, meaning this ad buy right 383 00:24:40,764 --> 00:24:44,921 against revenue, Then yeah, you would definitely need some interruptions 384 00:24:45,381 --> 00:24:48,121 to be able to essentially create a 385 00:24:48,976 --> 00:24:50,916 Justin Norris: Oh, control, that's the word I was looking for too. 386 00:24:50,976 --> 00:24:51,196 Yep. 387 00:24:51,931 --> 00:24:54,951 Mark Stouse: If you're talking about multi-variable though, where a lot of 388 00:24:54,951 --> 00:24:59,811 things are changing all the time around it, it's not necessary that's actually 389 00:24:59,811 --> 00:25:03,717 one of the things that's really, really cool about MVR multi-variable regression 390 00:25:03,817 --> 00:25:08,397 is that because you're bringing in so many different variables and because 391 00:25:08,397 --> 00:25:13,054 everything changes, particularly the stuff that you don't control you're gonna 392 00:25:13,274 --> 00:25:14,974 see exactly what you're talking about. 393 00:25:14,974 --> 00:25:17,596 You're gonna see implicit AB test. 394 00:25:19,071 --> 00:25:19,911 Evolving across time. 395 00:25:20,747 --> 00:25:21,977 Justin Norris: We're talking about limitations. 396 00:25:21,997 --> 00:25:25,707 And one of the things we're talking about was being an aggregate 397 00:25:25,707 --> 00:25:27,677 versus tracking individual users. 398 00:25:28,347 --> 00:25:31,877 Mark Stouse: Yeah, so one of the things that's really important to say about 399 00:25:32,377 --> 00:25:37,637 multivariable regression slash go to market, mixed modeling, et cetera, right? 400 00:25:38,117 --> 00:25:43,337 Is that it is looking at all the causal relationships in aggregate. 401 00:25:44,377 --> 00:25:52,027 So it's not possible to say that all these things cause this one identified 402 00:25:52,027 --> 00:25:54,917 person to take the steps that they took. 403 00:25:55,567 --> 00:25:58,217 That's not what go to market mix modeling is all about. 404 00:25:58,927 --> 00:26:05,637 It's about looking at a population wide trend that's causal for a lot of people 405 00:26:06,079 --> 00:26:10,039 one of the things that companies really appreciate though about this is that. 406 00:26:10,434 --> 00:26:15,001 There's no PII in proof or in, a regression model. 407 00:26:15,271 --> 00:26:20,614 So you don't have the security concerns that you would have in 408 00:26:20,704 --> 00:26:26,927 MTA or any of the other touch based approaches, which are trying to tie. 409 00:26:27,418 --> 00:26:33,158 Broad investments, programmatic investments to impact on a particular 410 00:26:33,158 --> 00:26:35,358 human being or particular company. 411 00:26:35,738 --> 00:26:39,248 And to do that, you have to pierce the veil on identity. 412 00:26:39,248 --> 00:26:40,928 And that's a problem. 413 00:26:41,248 --> 00:26:42,528 Justin Norris: So it's looking at the big picture. 414 00:26:42,528 --> 00:26:47,488 So cookies, not an issue, GDPR, not an issue. 415 00:26:47,498 --> 00:26:50,348 It's really comparing bulk data sets and the 416 00:26:50,608 --> 00:26:56,518 Mark Stouse: In fact, the only GDPR thing that we ever have to encounter is 417 00:26:57,178 --> 00:27:00,168 the login information for users of proof. 418 00:27:00,972 --> 00:27:06,498 So we keep that secured, But in terms of the data that's in the 419 00:27:06,498 --> 00:27:09,238 computations, it's a non factor. 420 00:27:09,868 --> 00:27:11,018 It's just non issue. 421 00:27:11,805 --> 00:27:14,225 Justin Norris: Is there a way to explain that multi-variate regression 422 00:27:14,225 --> 00:27:17,798 to someone who maybe they've got high school math, first year university 423 00:27:17,798 --> 00:27:18,911 math, 'cause maybe that's a challenge. 424 00:27:18,971 --> 00:27:22,151 As you said, marketers, some of them are more quantitatively oriented. 425 00:27:22,211 --> 00:27:23,391 Not all of them are. 426 00:27:23,738 --> 00:27:26,796 And they're gonna need to trust, this data, so they'll need to be able 427 00:27:26,796 --> 00:27:29,046 to make sense of how it's produced. 428 00:27:29,146 --> 00:27:31,210 You know, If it's just a kind of computer and it spits out an 429 00:27:31,210 --> 00:27:33,210 answer like, yep, do X or do Y. 430 00:27:33,558 --> 00:27:38,094 Mark Stouse: They're already effectively thinking in this way because they're 431 00:27:38,094 --> 00:27:40,814 spreading out their investments across all kinds of different 432 00:27:40,814 --> 00:27:46,057 channels, So they're acknowledging the multi-variable world in which they live. 433 00:27:46,207 --> 00:27:52,016 What they are not necessarily acknowledging is the fact that it's not 434 00:27:52,056 --> 00:27:58,913 a list of one-to-one correlations, My investment in display ads versus revenue, 435 00:27:59,213 --> 00:28:03,353 my investment in social versus revenue, my investment here versus revenue. 436 00:28:03,773 --> 00:28:04,873 That's not it. 437 00:28:05,433 --> 00:28:11,203 It's a tapestry that weaves back and forth with different time lags associated with 438 00:28:11,278 --> 00:28:11,358 hmm. 439 00:28:11,428 --> 00:28:16,388 Literally the only way capture that is with multi-variable regression 440 00:28:16,608 --> 00:28:20,718 or econometrics or, marketing mix modeling, go to market mix 441 00:28:20,878 --> 00:28:21,918 modeling, right there, it's all 442 00:28:21,918 --> 00:28:22,398 the same thing, 443 00:28:22,780 --> 00:28:24,160 Justin Norris: the fir, the first thing that you mentioned 444 00:28:24,160 --> 00:28:25,800 Is what we're conditioned to look for. 445 00:28:25,800 --> 00:28:29,640 Like we were conditioned to say like, I invested a dollar here and it produced 446 00:28:30,060 --> 00:28:32,560 two, like these definitive statements. 447 00:28:33,300 --> 00:28:35,740 I think if I'm interpreting you correctly, you're saying reality is not 448 00:28:35,740 --> 00:28:37,640 that straightforward most of the time. 449 00:28:37,995 --> 00:28:40,415 Mark Stouse: The final output can be that straightforward. 450 00:28:40,845 --> 00:28:41,135 Okay? 451 00:28:41,395 --> 00:28:45,521 But the causality elements on it, are going to vary with the 452 00:28:45,631 --> 00:28:50,131 wind, so a great example of this is uh, Johnson Controls, right? 453 00:28:50,351 --> 00:28:56,991 So even before Covid really became a story, uh, their analytics started to 454 00:28:57,011 --> 00:28:58,994 say in terms of forecasting, right? 455 00:28:59,667 --> 00:29:01,375 for reasons we don't understand. 456 00:29:01,915 --> 00:29:06,555 All these different channel investments that we have historically made are 457 00:29:06,948 --> 00:29:11,016 showing that they're not gonna be as effective in six months as they are now. 458 00:29:11,199 --> 00:29:13,079 A lot of these were physical events. 459 00:29:13,689 --> 00:29:17,839 , this was not due to marketing data in these models. 460 00:29:17,989 --> 00:29:22,721 This was all about external data was already up these early signals, 461 00:29:23,498 --> 00:29:24,638 they decided, you know what? 462 00:29:24,738 --> 00:29:27,798 We are gonna fly by instrument. 463 00:29:27,848 --> 00:29:30,158 We're not going to fly by what we can see. 464 00:29:30,416 --> 00:29:36,189 And so they started removing investments early a lot of these things and they were 465 00:29:36,189 --> 00:29:38,162 able to claw back quite a bit of money. 466 00:29:38,662 --> 00:29:43,848 And then all of a sudden, real life proved the analytics accurate. 467 00:29:44,021 --> 00:29:45,868 And so they were very happy about that. 468 00:29:46,823 --> 00:29:52,643 And then finance came later and said, Hey, so sorry, we're gonna have to cut you. 469 00:29:53,036 --> 00:29:53,973 30 or 40%. 470 00:29:54,053 --> 00:29:56,333 I can't remember exactly what it was, but it large. 471 00:29:56,760 --> 00:29:59,034 And they said, let us show you something. 472 00:29:59,084 --> 00:29:59,434 Right? 473 00:29:59,434 --> 00:30:05,063 And they modeled the effects of those kinds of cuts over the next three 474 00:30:05,063 --> 00:30:10,252 years, and it was so detrimental to the business that finance took 475 00:30:10,732 --> 00:30:14,726 a lot less and went elsewhere to get whatever they needed, right? 476 00:30:15,467 --> 00:30:22,552 So this is an example, not only of how a marketing team used it to avoid 477 00:30:22,562 --> 00:30:27,876 waste, but also to avoid cuts that would be ultimately bad for the business, 478 00:30:28,899 --> 00:30:31,759 Justin Norris: and the sort of external data you just described, those are things 479 00:30:31,759 --> 00:30:33,759 that's not specific to any one business. 480 00:30:33,829 --> 00:30:38,639 It's common to the macro environment that many businesses are working in. 481 00:30:39,049 --> 00:30:42,342 Like if I come to you, do I need to bring that external data on my own? 482 00:30:42,342 --> 00:30:46,582 Or do you have sort of global external data that you can match 483 00:30:46,582 --> 00:30:49,902 up with my business specific data and include in the model? 484 00:30:50,494 --> 00:30:55,434 Mark Stouse: So we are not a data provider, but we can and do 485 00:30:55,454 --> 00:31:01,951 all the time help people locate data sets that are usually free. 486 00:31:02,661 --> 00:31:05,991 That kind of stuff is usually free, either from the government 487 00:31:06,091 --> 00:31:10,221 or major financial institutions or universities, things like that. 488 00:31:10,954 --> 00:31:12,174 So that's not a problem. 489 00:31:12,624 --> 00:31:13,184 And it's great. 490 00:31:13,184 --> 00:31:15,024 Usually just fantastic data. 491 00:31:15,224 --> 00:31:20,344 I mean, It's been totally scrubbed, so that's not a big deal at all. 492 00:31:20,574 --> 00:31:24,574 In fact, we live in the golden age of data availability. 493 00:31:25,104 --> 00:31:28,597 Even if it's not free, someone is measuring it. 494 00:31:29,137 --> 00:31:33,877 Purely speculatively in the belief that somebody is gonna wanna buy it. 495 00:31:34,397 --> 00:31:39,457 And so today with a credit card, you can subscribe to all kinds of 496 00:31:39,487 --> 00:31:41,784 data sources, cost effectively. 497 00:31:42,431 --> 00:31:47,615 Particularly given how important it is to maximizing the investment. 498 00:31:48,050 --> 00:31:52,423 The other thing that I would just point out is ROI is really 499 00:31:52,423 --> 00:31:55,343 important, but you only know RROI. 500 00:31:55,776 --> 00:31:57,226 Looking backwards, right? 501 00:31:57,786 --> 00:32:00,485 ROI is a historical assessment. 502 00:32:00,673 --> 00:32:05,523 What is really important is forecasted, ROI, and then the 503 00:32:05,523 --> 00:32:08,083 comparison between it and actuals. 504 00:32:08,463 --> 00:32:12,963 So again, this is very much like public companies issue guidance 505 00:32:13,343 --> 00:32:16,603 and then they issue regular updates against that guidance. 506 00:32:17,833 --> 00:32:23,303 That is exactly what the C-suites of many companies are, demanding today, right? 507 00:32:23,353 --> 00:32:26,975 It's like an investment uh, to start a new company. 508 00:32:27,235 --> 00:32:30,999 The first question the investor's gonna ask is, what do you expect 509 00:32:30,999 --> 00:32:33,479 this to do in year 1, 2, 3, 4, 5? 510 00:32:33,889 --> 00:32:36,829 And what's the basis for that projection? 511 00:32:37,399 --> 00:32:42,979 If you're just extrapolating from hope and best wishes and all that kind of 512 00:32:42,979 --> 00:32:44,899 stuff, that's not much of an argument. 513 00:32:45,119 --> 00:32:51,167 But if you're doing regression based analysis, that starts to mean something. 514 00:32:51,817 --> 00:32:54,360 Justin Norris: And listening to the examples that you described, it does feel 515 00:32:54,750 --> 00:32:56,960 like a very enterprise oriented solution. 516 00:32:57,070 --> 00:33:01,596 If I work at a company, 400 people, 50 million a RR, is this 517 00:33:01,596 --> 00:33:04,556 something that can work for us, or do I need to be of a certain size 518 00:33:04,996 --> 00:33:06,396 threshold for it to be useful? 519 00:33:07,411 --> 00:33:09,131 Mark Stouse: No, actually it totally scales. 520 00:33:09,391 --> 00:33:14,011 The reasons for investing in it are going to be different in a small, 521 00:33:14,271 --> 00:33:16,931 medium, or enterprise type business. 522 00:33:17,521 --> 00:33:18,781 But it totally scales. 523 00:33:19,181 --> 00:33:23,081 And the cost is totally approachable, even for mom and pop, right? 524 00:33:24,111 --> 00:33:28,371 So the reasons for doing it as a small, and let's say the lower 525 00:33:28,401 --> 00:33:30,371 half of the medium sized business. 526 00:33:30,981 --> 00:33:37,106 Are mainly because if you make a bad investment in whatever, It has a almost 527 00:33:37,336 --> 00:33:40,346 immediately negative effect on cashflow. 528 00:33:41,298 --> 00:33:45,058 Too much risk, so they are modeling to avoid that. 529 00:33:45,198 --> 00:33:49,898 We do have some small customers and that's their main reason for doing it. 530 00:33:50,464 --> 00:33:53,804 If you are an enterprise and you're spending, I don't know, $200 531 00:33:53,804 --> 00:33:55,724 million a year on marketing, right? 532 00:33:56,164 --> 00:34:03,124 you spend it wrong, what the CFO is most concerned about is things 533 00:34:03,274 --> 00:34:05,578 like opportunity cost, right? 534 00:34:05,715 --> 00:34:09,975 That's the comparison that's going on in their mind all the time, particularly 535 00:34:10,105 --> 00:34:17,065 today, is what are all the different ways that I can spend this dollar and what 536 00:34:17,065 --> 00:34:22,376 are the most effective, or what's most likely to give me the biggest impact? 537 00:34:23,186 --> 00:34:24,326 That's the competition. 538 00:34:24,436 --> 00:34:29,045 That is the Game of Thrones, particularly right now in budget season. 539 00:34:29,846 --> 00:34:32,936 budget season right now is sort of year round now, right? 540 00:34:32,936 --> 00:34:35,146 Because everything is so pressed. 541 00:34:35,525 --> 00:34:42,416 So if you are a marketing leader, you are in competition to show that money 542 00:34:42,466 --> 00:34:45,496 spent with you is a better return. 543 00:34:46,034 --> 00:34:50,447 Than if they spend it in it or HR or whatever, right 544 00:34:50,913 --> 00:34:55,253 you also need to really understand this, you marketing by definition, 545 00:34:55,393 --> 00:35:00,385 is a non-linear multiplier of areas of business performance that are 546 00:35:00,445 --> 00:35:02,491 linear, one of which is sales. 547 00:35:02,871 --> 00:35:08,891 So in simple language, your leverage, you are bringing huge amounts of 548 00:35:09,211 --> 00:35:15,504 leverage to sales performance that sales cannot create for itself, right? 549 00:35:15,597 --> 00:35:20,957 not a ding on sales, it's just the nature of the reality of the situation, what 550 00:35:20,957 --> 00:35:23,617 do I mean by linear and non-Linear, okay. 551 00:35:23,837 --> 00:35:28,644 It means that if I were to go to my CRO and double goal for. 552 00:35:28,692 --> 00:35:32,758 For 2024, the first conversation that they're gonna want to have 553 00:35:32,758 --> 00:35:37,878 with me is about essentially doubling their Salesforce, right? 554 00:35:37,878 --> 00:35:44,520 Because the relationship between revenue coming in and the cost of that 555 00:35:44,520 --> 00:35:51,778 revenue in the form of sales team is it's known it's a linear function. 556 00:35:52,288 --> 00:35:55,998 And that's because it's the collective performance of a 557 00:35:55,998 --> 00:35:57,838 lot of individual performances. 558 00:35:58,828 --> 00:35:59,118 Okay? 559 00:35:59,738 --> 00:36:04,084 So a bell curve your sales team's performance is gonna be on a bell curve. 560 00:36:04,584 --> 00:36:06,234 It's just fundamentally linear. 561 00:36:06,698 --> 00:36:11,544 The whole reason why modern marketing was created in 1890, whatever, 562 00:36:11,997 --> 00:36:18,017 to bring non-linear leverage to that whole equation, even in B2C. 563 00:36:18,584 --> 00:36:19,684 So what does that mean? 564 00:36:19,824 --> 00:36:24,636 It means that because of the way marketing is, were to have the 565 00:36:24,636 --> 00:36:28,593 same conversation with the CMOA, we're doubling the revenue goal. 566 00:36:29,643 --> 00:36:35,056 Might have to increase marketing spend by 25%, maybe 20%, there's already 567 00:36:35,256 --> 00:36:37,296 a ton of leverage built into it. 568 00:36:37,476 --> 00:36:43,587 And if you want an easy understanding of ROI for marketing, it is the 569 00:36:43,727 --> 00:36:46,955 extent of that multiplier, so we'll talk about it this way. 570 00:36:47,465 --> 00:36:52,464 You are basically saying that marketing's mission is to help 571 00:36:52,674 --> 00:36:56,188 sales more product to more people. 572 00:36:56,728 --> 00:36:58,226 That's revenue faster. 573 00:36:58,901 --> 00:37:01,939 That's cash flow from revenue more profitably. 574 00:37:02,239 --> 00:37:07,081 That's margin impact than sales could do by itself. 575 00:37:07,621 --> 00:37:08,601 That's the key phrase. 576 00:37:09,250 --> 00:37:13,150 The extent to which that is true is the ROI to the business. 577 00:37:13,589 --> 00:37:16,261 So does that actually look like in real life? 578 00:37:16,451 --> 00:37:22,051 Well, and a lot of really, really great B2B marketing go-to 579 00:37:22,051 --> 00:37:23,491 market kinds of operations. 580 00:37:24,377 --> 00:37:27,967 ratio is somewhere between 10 and 20 x. 581 00:37:28,296 --> 00:37:34,116 So that means that if you take marketing away entirely, which I think would be a. 582 00:37:34,286 --> 00:37:39,451 Something that not even the most draconian CFO would ever contemplate, but let's 583 00:37:39,451 --> 00:37:41,971 just say it is the ultimate AB test. 584 00:37:42,081 --> 00:37:44,411 So we're gonna completely shut down marketing. 585 00:37:44,648 --> 00:37:46,816 You're gonna see a massive fall off. 586 00:37:46,876 --> 00:37:50,176 It may take a year, but you're gonna see a massive fall off in sales 587 00:37:50,176 --> 00:37:54,736 productivity that is going to reveal the extent of the marketing multiplier. 588 00:37:54,970 --> 00:37:59,614 It's like literally guarantee able because sales create the leverage for itself. 589 00:37:59,724 --> 00:38:01,174 It's just not the way it works. 590 00:38:01,842 --> 00:38:04,782 Justin Norris: If a company is getting started with this solution 591 00:38:04,782 --> 00:38:06,232 what would the process be like? 592 00:38:06,232 --> 00:38:08,032 What data sources would we need to bring? 593 00:38:08,732 --> 00:38:11,312 How much time does it take to build the model, that sort of thing. 594 00:38:12,475 --> 00:38:16,375 Mark Stouse: the very first step is, and this is part of the onboarding for us, 595 00:38:16,555 --> 00:38:20,655 is that we sit down with the customer and we say, look, what are your top 596 00:38:21,017 --> 00:38:26,257 questions that you most want to know The answer to this is a mixed audience, 597 00:38:26,487 --> 00:38:28,737 usually of marketers and business leaders. 598 00:38:29,997 --> 00:38:31,787 And doesn't matter what your job is. 599 00:38:32,097 --> 00:38:35,717 Nobody has any problem rattling off their list of questions. 600 00:38:36,297 --> 00:38:40,897 Those questions then generate what's called a model framework, could easily 601 00:38:41,127 --> 00:38:47,327 analogize that to a recipe card., So this is gonna be a punch list of data types 602 00:38:47,477 --> 00:38:54,737 that you're going to need to be able to supply to the model to compute the answer. 603 00:38:55,464 --> 00:38:58,784 then it's gonna outline the model itself, algorithmically speaking. 604 00:38:59,404 --> 00:39:04,214 When you actually make the dish from that recipe, that is the model. 605 00:39:04,774 --> 00:39:08,624 Usually the way this actually works is it's very fast. 606 00:39:08,774 --> 00:39:11,944 It's usually a matter of weeks, like less than a month, 607 00:39:12,254 --> 00:39:14,394 that kind of timeframe where. 608 00:39:15,638 --> 00:39:18,378 The analyst and the business user. 609 00:39:18,870 --> 00:39:21,513 Could be a marketer, could be finance guy, could be whatever. 610 00:39:22,033 --> 00:39:26,493 Are collaborating in the tool on a minimum viable model. 611 00:39:26,794 --> 00:39:32,255 And at some point business user says, that answers my question. 612 00:39:32,624 --> 00:39:35,164 We need to put this model in production. 613 00:39:35,420 --> 00:39:38,280 You hit the big red button, it goes into production. 614 00:39:38,770 --> 00:39:41,720 After that, it's pretty autonomous. 615 00:39:42,380 --> 00:39:46,727 There's kind of some DevOps type work, you you know, you have to maintain the 616 00:39:46,727 --> 00:39:48,127 model and all that kind of stuff, right? 617 00:39:48,227 --> 00:39:52,247 But for the most part, it is doing its thing on an automated 618 00:39:52,257 --> 00:39:58,054 basis and you're getting whatever cadence is right for your business. 619 00:39:59,514 --> 00:40:03,854 Daily, weekly, monthly, you're getting an update on demand. 620 00:40:04,674 --> 00:40:04,834 Justin Norris: the. 621 00:40:04,849 --> 00:40:07,619 . Analyst that you just described, is that someone that you're supplying 622 00:40:07,619 --> 00:40:10,379 from your team or is this kind of a bring your own process? 623 00:40:10,654 --> 00:40:15,814 Mark Stouse: we We have a large partner ecosystem that, we can recommend from. 624 00:40:16,194 --> 00:40:23,514 We also occasionally can do it ourselves on a managed service basis that is 625 00:40:23,834 --> 00:40:28,914 actually increasingly popular as teams get thinned out and they know 626 00:40:28,914 --> 00:40:32,754 that they need this capability, but they don't have the bandwidth or the 627 00:40:32,824 --> 00:40:36,314 expertise to manage it internally. 628 00:40:36,904 --> 00:40:42,614 And so that's actually extremely popular these days and very cost effective. 629 00:40:43,160 --> 00:40:45,371 the key thing is that there's lot of. 630 00:40:46,561 --> 00:40:53,329 You could easily spend 2020 5K upfront in time, not in licenses. 631 00:40:53,599 --> 00:40:53,889 Okay. 632 00:40:53,909 --> 00:40:58,102 But in time to get everything set up but once you've done that, 633 00:40:58,152 --> 00:41:02,065 right, again, the models, unless you need more models, right? 634 00:41:02,065 --> 00:41:04,485 The models are doing their thing, right? 635 00:41:04,485 --> 00:41:10,292 You don't have ongoing major, investment, know, every month gotta redo the 636 00:41:10,292 --> 00:41:12,052 whole thing from the ground up. 637 00:41:12,312 --> 00:41:17,252 That's, exactly the kind of thing that proof was built to eliminate, 638 00:41:17,572 --> 00:41:19,372 Justin Norris: if you add a new, you add a new channel, do 639 00:41:19,372 --> 00:41:20,292 you have to update the model? 640 00:41:20,312 --> 00:41:22,732 Or the model can be flexible enough to just say like, oh, you're 641 00:41:22,732 --> 00:41:25,452 doing LinkedIn advertising now. 642 00:41:25,452 --> 00:41:26,052 That's fine. 643 00:41:26,072 --> 00:41:28,412 We can just incorporate that as we go. 644 00:41:29,023 --> 00:41:30,583 Mark Stouse: you could do it either way, right? 645 00:41:30,863 --> 00:41:36,183 I think be the best practice is that you clone the model that you have and 646 00:41:36,183 --> 00:41:42,903 then you add the new data streams to it that you're maintaining the integrity 647 00:41:43,123 --> 00:41:45,143 of the original model for comparison. 648 00:41:46,003 --> 00:41:48,863 And yet you are updating it that way, right? 649 00:41:48,923 --> 00:41:53,543 And then at some point you're gonna dispense with the oldest version 650 00:41:53,563 --> 00:41:58,200 of the model altogether, So it's not like you just see a massive 651 00:41:58,800 --> 00:42:00,840 proliferation of models across time. 652 00:42:01,487 --> 00:42:04,867 But you do need to do this in an orderly way so that everybody 653 00:42:04,997 --> 00:42:06,437 doesn't lose reference points. 654 00:42:07,102 --> 00:42:09,502 Justin Norris: you mentioned competitors and I've seen competitors 655 00:42:09,502 --> 00:42:13,132 popping up in the market, even . A multi-touch attribution vendor 656 00:42:13,132 --> 00:42:15,652 that's adding MMM to their mix. 657 00:42:16,072 --> 00:42:20,292 What's your outlook on how easily other vendors could recreate these capabilities? 658 00:42:20,352 --> 00:42:23,772 Do you feel that you have a fairly strong competitive moat around the offering 659 00:42:23,772 --> 00:42:27,392 you've built, or will other people be able to jump into this environment and 660 00:42:27,392 --> 00:42:28,952 offer similar things fairly easily? 661 00:42:29,763 --> 00:42:34,023 Mark Stouse: So it's really important to say this, the competitive advantage 662 00:42:34,133 --> 00:42:36,623 that anybody has is not in the math. 663 00:42:36,760 --> 00:42:42,988 So if people propose that they have some kind of Super Cal Flagal algorithm, 664 00:42:43,510 --> 00:42:45,560 you need to be very careful about that. 665 00:42:46,114 --> 00:42:49,914 ' cause the odds also are that it's not transparent, it's a black box. 666 00:42:50,454 --> 00:42:51,114 And also. 667 00:42:52,220 --> 00:42:56,200 If it is transparent, you probably don't have the ability to evaluate it. 668 00:42:56,587 --> 00:43:04,941 In our particular case, the IP that really matters is in how we automate it, how we 669 00:43:04,991 --> 00:43:10,465 scale it, how we make it consumable and approachable and understandable and how 670 00:43:10,465 --> 00:43:12,145 we do it at a particular price point. 671 00:43:12,765 --> 00:43:17,441 And I think that this is the other thing that is highly relevant today, 672 00:43:17,821 --> 00:43:22,161 not just in this area, but across SaaS, is that prices are coming down. 673 00:43:22,741 --> 00:43:26,938 Uh, We're gonna see a fundamental change in SaaS pricing. 674 00:43:27,512 --> 00:43:32,249 The days of, you know, every year having a pricing increase are over. 675 00:43:32,729 --> 00:43:33,419 It's just done. 676 00:43:34,569 --> 00:43:37,469 We are very well fixed competitively speaking. 677 00:43:38,266 --> 00:43:40,466 Do have a, I think, a really solid moat. 678 00:43:40,686 --> 00:43:46,336 So there are a lot of competitors today that have some great products, but these 679 00:43:46,336 --> 00:43:53,016 are products that were conceived of and written by and for data scientists. 680 00:43:53,654 --> 00:43:55,354 and, And a lot of 'em are gorgeous by the way. 681 00:43:55,404 --> 00:44:01,534 Like, graphically, they're gorgeous, but if you expose them to a normal business 682 00:44:01,534 --> 00:44:07,957 user, a marketer, sales leader, whatever gonna stare at that screen and go, I have 683 00:44:08,137 --> 00:44:10,277 not a clue in the world what that means. 684 00:44:10,787 --> 00:44:14,117 Like how do I make a better decision based on that? 685 00:44:14,297 --> 00:44:18,437 You've just now thrown a lot of friction into it. 686 00:44:19,107 --> 00:44:20,277 It's taking more time. 687 00:44:20,937 --> 00:44:26,877 You've gotta spend a lot of time translating the data science outputs 688 00:44:27,027 --> 00:44:29,397 into something that is usable. 689 00:44:30,087 --> 00:44:37,289 And we don't do that, we built it with the end user in mind violating any. 690 00:44:38,194 --> 00:44:39,404 Data science principles. 691 00:44:39,674 --> 00:44:40,674 That's our big advantage. 692 00:44:40,844 --> 00:44:45,244 Think one of the biggest things I can say about this, just to sum up, is 693 00:44:45,244 --> 00:44:53,281 that there's a reason why large B2C marketing teams, CPG uh, hotel and 694 00:44:53,561 --> 00:44:56,364 hospitality, retail, whatever, right? 695 00:44:56,374 --> 00:45:05,732 They control all four pss of marketing and they have, at worst, a defacto authority 696 00:45:05,922 --> 00:45:08,132 over and responsibility over the p and l. 697 00:45:08,432 --> 00:45:10,092 Of their product with their marketing. 698 00:45:10,799 --> 00:45:16,202 There's a reason for that, they are using econometrics slash Go-to market 699 00:45:16,262 --> 00:45:22,609 mix modeling slash MMM, using that to optimize and to understand causality 700 00:45:22,629 --> 00:45:24,889 and to optimize based on that causality. 701 00:45:25,549 --> 00:45:29,809 And they're also investing a ton of money in market research, which a lot 702 00:45:29,809 --> 00:45:32,129 of that ends up in the models, right? 703 00:45:32,609 --> 00:45:40,763 So if B2B marketers want to have the same attributes their B2C brothers and 704 00:45:40,763 --> 00:45:44,957 sisters, gonna have to do what B2C does. 705 00:45:45,390 --> 00:45:51,124 This is one of those situations where it's like gravity, You can disagree all you 706 00:45:51,124 --> 00:45:54,789 want to with gravity, and if you throw yourself off a building, it's not gonna 707 00:45:55,109 --> 00:45:57,676 end well, the same thing is true here. 708 00:45:57,746 --> 00:46:00,356 This is a mathematical principle at work. 709 00:46:00,546 --> 00:46:04,876 It's mathematical law of gravity with quotation marks around it. 710 00:46:05,052 --> 00:46:08,731 And so everything that I've said today, I've, and I really 711 00:46:08,991 --> 00:46:11,011 try really super hard to do. 712 00:46:11,011 --> 00:46:12,491 This is not my opinion. 713 00:46:13,071 --> 00:46:19,631 I'm just representing a level of fact that people can either accept or reject, 714 00:46:19,931 --> 00:46:22,591 but it doesn't change the truth of it. 715 00:46:22,923 --> 00:46:26,019 Justin Norris: We will include a link to your website so people can uh, check it 716 00:46:26,019 --> 00:46:28,579 out, learn more, look at your resources. 717 00:46:29,559 --> 00:46:31,199 And I'm excited to see where this goes. 718 00:46:31,709 --> 00:46:33,999 Mark, thank you so much for chatting with me today. 719 00:46:34,466 --> 00:46:35,046 Mark Stouse: You're welcome. 720 00:46:35,136 --> 00:46:35,566 Thank you.