Welcome, Luminaries.

Thank you so much for joining our episode this week.

You have chosen wisely as always.

We are digging deeper this week into the subject of consultants and how they get, well, how can we say, hung up on certain kinds of evidence, particularly numbers.

We're super happy to welcome to the show our friend and colleague, Ann Fraser.

Anne, great to have you with us.

Hello, Ann.

Tell us a bit about yourself and the work that you do.

Thanks, Ian.

My name is Ann and my background is very analytical.

I was an engineer by education and I did mba, which got me into consulting.

And I did management consulting for a number of years.

And it was very analytical consulting, using evidence and data and numbers to help our clients make better decisions and take better actions.

And then after working in consulting for a while, I got into training and helping consultants learn how to better consult and use numbers and evidence to help them help their clients make better decisions.

Thanks so much for joining us here.

It's great to have you on Luminaries.

Ian and I have been talking about how it's easy for consultants to get very focused on getting numbers correct.

And I'd love to hear a little bit about the difference between how you approach quantitative work now compared with the beginning of your career.

Thanks, Mike.

It's an interesting question and I want to talk about numbers.

And a lot of the work I did was using numbers and a lot of data.

But consulting work encompasses more than numbers and it's really evidence and information.

You're not always just working on numbers.

But it's very hard to handle all of that at the beginning.

And what I've noticed when starting out, just coming out of university, is that in school we learned that numbers were important.

Your evidence is important, you have to be perfect.

And that's the only way to get the high grades and the high marks.

But working in consulting, a, we never have time to be perfect, and B, the client doesn't need perfect.

They need what we call good enough.

In fact, we have an acronym called gmo, G E M O meaning good enough.

Move on.

We need to get the numbers or the evidence good enough that it helps our clients make the right decisions or take the right actions.

And that's what they're expecting if we aimed.

And I think, Mike, I've heard you always say perfect is the enemy of the good, meaning that if we try and aim for perfect, we'll never get the good.

So besides being comfortable with numbers, what else are we going to have to get comfortable with?

Well, as a junior consultant, you want to start to get comfortable ambiguity and the fact that you're not going to have everything you need before you're required to have a point of view and make a judgment on things.

And in a lot, I find a lot of junior consultants, myself included, are really not comfortable dealing with less than perfect data.

In fact, Mike, I think you have a story about working with some junior consultants.

Well, not only working with junior consultants, I remember working with an entire analytics firm, a global analytics firm that decided to move into consulting.

And really, pretty much to a person, the folks that were coming into consulting having gone from really putting all their time, effort, and energy into getting as perfect a number as possible.

Now we're in this situation that you were just talking about, Ann, where we don't have time to be perfect anymore.

And sitting there with a team of folks who go, how can we actually have any really valid insights or make any recommendations given that we only have Precise data on 95% of the market?

And I was just dumbfounded.

I was just struck like, what are you talking about?

So it was a real transition for everybody, high to low, who came out of that other environment to say, we're in consulting now, ladies and gentlemen, the people that we're consulting for have perhaps imperfect data on what they do in the market and not a whole lot of data on what everybody else is doing.

We need to help them make a better decision or take decide on taking an action yet.

Don't have to have better than 95% to do that.

Yeah.

And I've worked on many projects where the data just didn't exist, the evidence didn't exist, and you have to deal with less than 100% of that certainty all the time.

Yeah.

Yeah.

I'm pretty sure that the government only has good data about 95% of what I do for money, which.

But they're still perfectly fine giving me a tax bill like, I think it's okay.

That was not a confession, by the way.

So speaking of getting comfortable with it not being perfect, I think as a consultant, you could worry a lot about making mistakes.

You could worry about what's in that missing 5%, maybe even more so when you're feeling that having a definitive number might somehow be able to be purely right or purely wrong.

How do we handle mistakes when they come along?

Or at least how do we handle the potential for mistakes?

Yeah, I think that's another big fear that some junior consultants have, is making mistakes.

Because when we do consulting, we're dealing with evidence and information, and we know it's not Perfect.

Not only that, we know we're not perfect.

And inevitably, if you're dealing with analysis and building up models and any kind of thing, we're going to make mistakes.

So it's not completely preventing mistakes.

It's how you handle when a mistake is made.

I think is a big learning for junior consultants.

And it must be terrifying.

The first time.

So I can remember the first time that one of my mistakes got discovered front and center in front of the client.

I thought my career was over.

Yeah, me too.

In fact, I remember distinctly, I was not a junior consultant, but a junior manager, first time managing.

So my team, we had made a mistake and it had to do with reports that went out that meant a person wasn't getting their bonus.

So it was a very scrutinized, deliverable we were having.

Luckily, we caught the mistake, but it had already gone out.

And for the first time, I actually had to call the client.

I was so nervous and had to let them know that we had missed a lot of data.

That meant all the reports had people not making their bonus.

Yeah.

And I was so nervous talking to the client and must have read and my voice must have rest.

She must have been able to hear that in my tone because she actually spent most of the call calming me down.

But I love what she said.

She said, ann, we're dealing with a lot of data and mistakes happen.

We know that what matters is how you deal with it afterward.

And you take responsibility and accountability and you figure out how to correct it as quick as you can.

And she appreciated that I cared and that came off in the phone.

So I think that's the lesson I learned is we all know everyone makes mistakes.

It's you own up to them and take responsibility for it, be accountable, and that's all they can ask for.

Right.

And clients are surprisingly, I'm not going to say forgiving, but they're surprisingly generous, I think, in their interpretation of these kind of situations, especially if they've been around the block once or twice with their own work with this kind of data.

Yeah.

And in fact, Ian, I know you do a great exercise in training, asking people, put yourselves in as the customer.

When you've had a service provider like cell phone and they've made mistake, what has made you feel more comfortable with the mistake that's happened and.

Right.

We all know that answer.

Yeah, exactly.

And it's really, really natural to defend in those situations.

You know, like you found the cell phone company to complain about the charges or something, and they get defensive and they Try to explain to me why I'm wrong to think what I think.

And it really annoys me.

I find it much, much easier to accept a conversation with somebody who's made a mistake, who, as you say, and just says, we accept that we've messed up here, and I'd like to be the one who fixes it for you and let us take care of that.

Yeah, exactly.

Wow.

And we've been talking a lot about kind of handling big chunks of what you might call secondary data.

But we can have the same mindset, I think, when it comes to primary research and evidence that we get from people who we interview.

Right.

And it was a lesson I learned as well as the person I was interviewing to learn.

So the project entailed me asking some experts on their opinion of what might happen.

So there was no real concrete data about it.

I knew to use statistical terms, the confidence interval is quite wide.

The data wasn't perfect by any means, but I had to ask a bunch of them what they thought about certain things and put numbers to it, their best guess on things.

And one of the experts I was interviewing said, hold on, this is all garbage in, garbage out type of thing.

And I just kind of said, okay, well, I understand that.

And the client understands this is not perfect data as well, but if we don't have this, they have to make a decision with zero data.

And he did pause, stop, and think about it.

And I said, I think your expert opinion is better than zero data, and it's better than my information, my thoughts and my guesses.

And he did respond to that, and we continued the interview.

So I was happy about that.

But it is that fact.

Sometimes we're helping our clients where the data quality isn't very good, but it's better than nothing.

And as long as the client realizes that it's.

That it is not the perfect data, but it gives them some sense or some guidance on making their decisions or taking some actions, then it's something of.

Value that's really great.

We spend a lot of our time closer to no information than we do to 100% perfect information.

Yes, unfortunately, yes.

And that story reminds me of sitting in another primary research example.

And the example was of a consultant interviewing an insurer.

And the insurer was talking about the likelihood of reimbursement and any restrictions that would be applied based upon the characteristics of a certain forthcoming drug.

And the insurer came out with all these things that would have to happen based on their best guesstimate, if you will.

And fascinatingly, the Consultant who had a little bit of experience said, oh, that's very interesting.

Tell me the last time you did that to a drug because it was pretty highly restrictive set of things.

And the person went and thought back and said, not sure we ever have.

Which kind of led me to thinking as you were talking about that, that you know, we have to handle mistakes.

And I know you're so good about this to avoid mistakes in the first place.

What kind of things can consultants do to make sure that we're not making mistakes in this area?

That's the ideal.

And of course you learn the hard way a lot of times to say, oh, that hurt me in this project I'm going to do some extra planning at the beginning to make sure that happens.

But even if you are junior and don't have that experience, you can do some things and you can reach out to those who have experience that will help you do some planning upfront so that you could reduce and possibly eliminate a lot of potential mistakes.

Again, we're not expected to be 100% perfect 1% of the time ever.

But you can do some planning.

And I know I did some one on one coaching actually.

I had a client who had lost a lot of their managers and so the quality of their work was going down and they asked me if I could come just do a little bit of coaching.

And one thing was on quality control because a lot of mistakes were being missed.

And so all I did was have conversations with the people I was coaching at the beginning of a project.

And what we talked about was saying, okay, you're going to get this data coming in that you're going to analyze.

And one rule I always had that I learned the hard way was never trust any data that's coming into you, even if it's within your company that data is being passed.

But it could be data or evidence that's coming in from another client.

Always do some sanity checks on it.

And a person who's not experienced might say, oh, I don't know, I, I wouldn't know what to check.

I don't know what to expect.

But when I worked with them, we realized a few things.

There are some sanity checks that are basic things everybody knows.

So say if you're working with country level data, you could say, okay, there's certain states or provinces, counties, et cetera, that you expect to have the largest portion of the sales.

And we talked about that to say, what are some sanity checks you can perform on that data?

Well, you'd expect this area to have the Largest portion of sales.

So we started building up a QA plan.

In other words, you can look at trends.

To say is do I expect the sales to be steady or not steady or the grand totals are just looking at totals year after year.

It shouldn't have an erratic pattern.

If you do see an erratic pattern, you can't explain it.

That's when you go back to the client and it saves you so much time because if you'd worked with that data and continued on without doing the sanity checks, then you don't catch the mistakes until the end.

Hopefully catch them and then you do all this rework.

So it's good to have a plan up front.

Just the sanity checks.

It's great.

I think that combined with your point about being inherently skeptical about any data that comes towards you, I think that's really good advice.

Yeah.

In fact, I have some stories of things we caught in doing this that the sanity checks, things we didn't know was going on in the market, that got explained early on and we knew we, in some cases we had to remove the data.

So one example is we were doing forecasting and I'm based in Canada, and we noticed that the sales for one province dipped significantly.

And if we had just fed that into their model, we would have this horrible forecast for that province.

But we went back to the client and said, okay, we see this huge dip in the five years of data that we have.

Is that normal?

Is that expected for the future?

And they said, oh no, that was a supply issue, a once off.

And so we were able to just remove that data and forecast it out and it didn't impact the client.

You also get to find interesting trends that you didn't realize.

Certain markets.

We were doing some analysis for the oral contraceptive market and we did that trend check to say, okay, let's look at the sales month by month.

And we saw this huge surge in September and we didn't fully put together what that meant.

But of course, when students are heading back to college and university, there was a surge in that they need to re up.

That was normal data.

We did double check that with the client and they said, absolutely, that's normal data and you can use that data, it's valid.

But there's even, there's even little things like a rep, a sales representative might be on maternity if that happened.

And we noticed that the trends for her territory went down.

The client didn't think to tell us that when they were feeding us that data, but we checked it and we were able to make sure that all the data going into our analysis and modeling is correct by just doing some of those sanity checks.

On another check we did, I was working with a client who sold smoking cessation products across Canada.

And in Canada, it wasn't a prescription medication, but if you wanted reimbursement, you needed a prescription, so people could just buy it over the counter and pay upfront, or if they wanted to claim reimbursement from the province, they needed prescription.

So when we looked at it and we did it at a physician level, and all of a sudden there was this one physician in Quebec who pretty much sold an enormous amount of sales, like, by far, like, maybe 30 times higher than the next physician.

Maybe even higher.

Maybe 100 times.

I can't remember.

But when we brought that up, the president of the company wasn't aware of this.

And, oh, my goodness, he said, we should be spending a lot of time with the physician and making sure that physician is very happy.

But it was actually the sales director for the province.

She knew exactly what was happening.

And it wasn't that the physician, per se, because it's a drug that doesn't need a prescription but needs enrichment.

He was involved with multiple clinics across the province, and so they just wrote scripts under his name for reimbursement purposes.

The physician themselves probably wasn't making those decisions on who was going to get the script or where the script was.

Going and that kind of thing.

Yeah, it's interesting when you break out the data and you really examine and do these sanity checks, you learn a lot about the market as well.

Now, analyzing and predicting numbers seems to be something that's very prone to that.

Not only the kind of natural quirks about what happens in the world and supply chains and channels and stuff, but also the quirks of people manipulating or gaming the numbers.

There's this saying, isn't there?

Figures don't lie, but sometimes liars figure.

Now, Mike and I have been talking a little bit about this, about Campbell's Law and Goodhart's Law, and the way that once you make a measure into a target, it ceases to be a good measure.

Have you ever come across that people starting to game what they know, the consultants are starting to measure?

I think there is bias that on how you interpret figures.

I would say you definitely need some objective viewpoints in your data.

I know the clients have bias and they know what they want.

Even if the evidence is bad and you have to tell the client the baby is ugly.

Right.

And you can't be manipulated.

Yeah.

And I know, Mike, you have some interesting stories when dealing with giving negative news with.

To a client.

I think, yeah.

Knowing that you've.

As you're doing your analysis, stopping to check along the way, particularly when the trends are moving away from what the client's expecting to hear and making sure to ground that it's.

It serves two purposes.

One, it does part of the QA that you were talking about and that I'm going to make sure that all these numbers are right.

We're signing off on this.

We all agree to this.

It also is a little bit of the cats on the roof we used to call it.

I'm babysitting for my friend's cat.

Something terrible has happened to the cat.

Do I tell them on their first day of vacation what's happening?

How's the cat?

I'm not sure the cat's actually up on the roof.

And ultimately there's going to be a sad end to that story, but for the first day.

So sometimes we'd look at each other and the client would walk in and as the client was coming in, we'd say, remember, the cat's on the roof.

Let's not be talking about all how well this is going to go, because preliminary analysis suggests that it's not so.

And on the one hand, you've got that kind of thing going on.

I remember doing some mergers and acquisition work with financial institutions and had a number of small, relatively new financial institutions that were all going to become kind of a predominant regional force by coming together.

And there were these teams of accountants and investment bankers and everybody who were killing each other with numbers.

And everybody was all in this big conference room in a hotel and going nuts and crossing eyeballs with the precision of something that was just not that precise.

Because everybody had limited amounts of data over time and all sorts of arguments.

And finally a very senior ex banker who was the CEO of one of these, turned to his top finance guy and he said, george, thank you.

I think that's exactly right.

Gentlemen, ladies, the number is 1.4273 that we're looking for.

And everybody's teams huddled and they came back and they all agreed that was the number.

That's exactly it, down to four decimal places.

And I was walking out of the room with George later.

I said, by God, George, how did you get that number?

And I said, was it your finance guy who is an absolute wizard?

He said, no, it was the extension number to the phone that was sitting by me in the conference room.

Oh, dear.

But it had four Digits.

And so obviously it was the number.

And we took hours and hours and hours of conversation and boiled them down to a final 15 minutes.

Geez, Mike, that must have been a very profitable project as well.

Consulting.

So much easier when you know the number and you can work backwards.

Isn't that exactly right?

You make up numbers?

Well, we do.

I think.

I think we do have a tendency to want to give the client what they want to hear, especially if you're working on a forecast.

They have a goal in mind.

And there is a tendency, and I can tell a story where I was helping a client determine whether a program they did was profitable.

And we did some analytics on it.

It was a control and test study.

And essentially we found it was not profitable at all.

In fact, it was horrible.

And I wanted to be careful about telling my client who spent all that money and their neck was on the line.

That didn't work.

So I kind of tried to say it in a softer way and say, oh, we could not prove that it was profitable.

And then what happened was they started talking to say, wow, we need to do more so that you can prove it.

And I had to say, let me be more direct and clear.

Do not do this again.

You will be losing money.

So we want to make sure they get the message based on evidence of what actions to take, what decisions to make.

So let me tell you one time, we were doing a piece of strategy work, and to your point, and about clients knowing what they want and saying, okay, sometimes we do have the evidence, and we've done all the work that we can do to make sure this is right.

It's as good as we believe it can be, given that we don't have data on the future.

So it can be perfect.

But we had some projections that, in fact, were not what they wanted to see.

And we had done all the prep up to there.

We'd done all the QA as much as we could upfront.

We'd done all the back and forth, and we ultimately had to deliver that report.

Report.

The client said, we just don't think this is right.

We've tried to make our ideas clear to you.

These numbers just don't work for us.

And we're certain that you have not done a very good job.

And we apologized that we did not meet their needs.

We thanked them and said this was the point where we should kind of part ways.

And they said that's exactly what they wanted to do.

They wanted to find another firm, which they did.

And we got a call about a Year later, that said, we're launching another major initiative and we need some forecasting work done and we thought we'd see if you were interested.

And I said, well, that didn't turn out real well last time.

He said, you don't know the half of it.

It didn't turn out well with us working for you, going into the market turned out to be a complete disaster and your work really would have prevented us from doing that.

So we'd like you to come back if you would, and help us this time.

And I said, we'd be delighted.

It will probably cost a little bit more, man.

Yeah, it's a good lesson, right.

Telling them that what they want to hear makes them happy for a week.

Telling them what's really going on and helping them make a decision makes them happy for years.

Yeah.

As a junior consultant or as any consultant, it's really hard to tell a client something opposite to what they want to hear.

And we do have a tendency of wanting to tell them what they want to hear because we're people pleasers, we're service oriented.

And so, as Mike's story shows, we really need to be able to do that because we won't be getting asked back.

They need to be able to trust us and trust that we give them advice based on evidence, objective evidence.

One of the things that Ian and I talked about earlier in the main episode was analysis paralysis, that one of the things that happens sometimes is that we've got such a fixation on numbers.

We're into the numbers and we can't pull ourselves up out of it.

You mentioned earlier, guimo, good enough, move on.

And I've seen you working with some teams late in a project when the presentation delivery is imminent.

And I think there are a couple of stages up from that that you introduced to the team.

That's right, yeah.

And GUIMO is similar to that 8020 rule.

Like 80% of the work or 80% of the time, what is it?

80% of the results come from 20% of the work.

Right.

And to get that extra 20% perfected, it's going to take a lot of time anyhow.

Sorry, back to your prompt.

Yes, Mike.

As I'm working with teams and as it's getting later and later and the deadline is looming, we sometimes move from Gimo, good enough, move on.

The first next stage is Sumo S U M O which means we gotta shut up and just move on.

We gotta stop talking about it and have that analysis paralysis and we have to move on.

And the final one Gets a little ruder.

It's FIMO F I M O.

We just gotta, we just gotta go effort, move on.

We gotta get going and finish this up.

Yes.

And hopefully I hope those listeners out there don't actually get to FIMO sometimes.

I know I have, unfortunately, but hopefully it's rare with every.

For everyone.

Awesome.

Well, Al, hopefully a fair ways before we get to FIMO today.

Thank you already for joining us.

It's been loads of fun talking this through with you.

Tell us about your close of the year.

What are you up to, what's coming next for you?

Thanks, Ian.

I've been just.

The latest stuff that I've been working on is helping clients with critical thinking, which is quite a broad term, but we focus on that critical thinking that happens when you're really defining what the client's need is.

So I don't know about you, but I've experienced a lot of clients will ask me for something very specific and it's not truly what they need.

And you have to apply critical thinking to ask them a bunch of questions to understand what they need, how they're using it, why they want it, all those things so that you deliver what they need, not necessarily what they ask from you in the first place.

And that's been really interesting, working on critical thinking with clients.

Fantastic.

I think we might have to bring you back pretty soon.

Here's some more about that.

This has been great, Ed.

We really appreciate you being here.

You have forgotten more about models and forecasting than I will ever know.

So it's great to see somebody who while so deep into this, also can pull back and take that higher view over this.

We're so glad to have had you listeners here at Luminaries to talk about what do we do with numbers, what do we do with evidence, what do we do with the decision making and action taking that all goes to adding value to our clients Next week, networking, adding value to ourselves and thereby adding value to our clients.

Both help on projects and certainly career planning and navigating.

We hope you'll join us next week on Luminaries.