1
00:00:00,080 --> 00:00:03,840
In this special holiday themed episode, we're diving into a topic

2
00:00:03,840 --> 00:00:07,620
that's as classic as Christmas Carols, but just as divisive as fruitcake.

3
00:00:08,080 --> 00:00:11,620
And that topic is the future of SQL Server Integration

4
00:00:11,759 --> 00:00:15,285
Services, SSIS. But wait, there's a

5
00:00:15,285 --> 00:00:19,045
twist. This episode was recorded live, so if you notice

6
00:00:19,045 --> 00:00:22,885
a different vibe, some festive banter, and maybe even a change in

7
00:00:22,885 --> 00:00:26,699
our usual musical interludes, that's why. Think of it as

8
00:00:26,699 --> 00:00:30,320
the holiday party version of our usual data driven discussions.

9
00:00:31,099 --> 00:00:34,780
Together, we'll explore why SSIS, despite its vintage

10
00:00:34,780 --> 00:00:38,219
status, remains a cornerstone of data engineering and why

11
00:00:38,219 --> 00:00:42,045
dismissing it might just be a data driven mistake. So grab your

12
00:00:42,045 --> 00:00:45,345
cocoa, settle in by the fire or your nearest CPU,

13
00:00:45,725 --> 00:00:48,145
and let's get festive with some data talk.

14
00:01:04,340 --> 00:01:07,479
Well, hello, and welcome to franksworld.comstream.

15
00:01:10,245 --> 00:01:14,085
And, with me today is Andy, and I'm

16
00:01:14,085 --> 00:01:17,604
looking for the lower third that has us both. There we

17
00:01:17,604 --> 00:01:21,205
go. Frank and Andy Frank Lavinia and Andy Leonard, host of Data

18
00:01:21,205 --> 00:01:24,850
Driven, which I might turn this into a podcast. I might take the

19
00:01:24,850 --> 00:01:27,170
audio and and turn it into a podcast. What do you think about that? That'd

20
00:01:27,170 --> 00:01:30,930
be good kind of festive stream and also kind

21
00:01:30,930 --> 00:01:33,810
of up to date on things. And it gives me some more time that to

22
00:01:33,810 --> 00:01:37,405
put together another episode that I had a really great

23
00:01:37,405 --> 00:01:40,785
conversation with a guy who does red teaming for LLMs.

24
00:01:41,405 --> 00:01:44,945
Nice. So which I think is a growth industry

25
00:01:45,085 --> 00:01:47,585
and certainly a wise career move.

26
00:01:48,525 --> 00:01:52,220
Speaking of career moves. Good thing.

27
00:01:52,840 --> 00:01:56,460
Oh, we have a first comment. SQL dev d b a.

28
00:01:56,920 --> 00:02:00,600
Hey, SQL dev. Awesome. So

29
00:02:00,600 --> 00:02:04,200
this is this may be the first time we've done this. This feature's been around

30
00:02:04,200 --> 00:02:07,604
for a while. No. We did it once or twice before. Did we do it

31
00:02:07,604 --> 00:02:11,385
on recent? Like, months. Yeah. That we've done. So we're sharing

32
00:02:12,004 --> 00:02:15,685
our so Frank's audience, people that are connected to Frank,

33
00:02:15,685 --> 00:02:19,390
they're seeing this. People connected to me are seeing this. It's like it'll

34
00:02:19,470 --> 00:02:23,310
because it told me Frank started this, and, then he sent me

35
00:02:23,310 --> 00:02:27,070
the link. And as I joined in, it it said, hey. You can

36
00:02:27,070 --> 00:02:30,750
share this with with your on your channels as well. So I

37
00:02:30,750 --> 00:02:33,965
was like, oh, yeah. Click that. Oh, you know what it is? We did it

38
00:02:33,965 --> 00:02:37,805
the other way. You were the main, and then I shared it on my channels.

39
00:02:37,805 --> 00:02:41,565
That's what happened. That's what happened. Yeah. Yeah.

40
00:02:41,565 --> 00:02:44,785
Well, it's cool, though. If you've never met me before hello?

41
00:02:45,245 --> 00:02:48,870
That's Frank. Frank digs data on the socials and,

42
00:02:49,110 --> 00:02:52,810
franksworld.com, datadriven.tv, which hopefully you know about that,

43
00:02:53,030 --> 00:02:56,489
and impactquantum.com. So that's me.

44
00:02:57,190 --> 00:03:00,955
And, yeah. So back to the segue.

45
00:03:01,655 --> 00:03:05,255
Yeah. I was talking about how security and AI is a

46
00:03:05,255 --> 00:03:08,775
good career move. And we were talking about, speaking of

47
00:03:08,775 --> 00:03:12,455
career moves, 'tis the season for SSIS is the title of the

48
00:03:12,455 --> 00:03:15,410
stream. And this kind of goes,

49
00:03:17,070 --> 00:03:20,750
I'm sorry. Come on, man. It's fun. Right? It is.

50
00:03:20,750 --> 00:03:24,050
It's awesome. So so and I had kind of done,

51
00:03:24,670 --> 00:03:28,270
2 livestreams on this already, but one of them for, like, 10 minutes, I didn't

52
00:03:28,270 --> 00:03:31,965
catch the fact that I had no audio. And then yesterday, I did one for

53
00:03:31,965 --> 00:03:34,445
2 minutes, so I didn't catch the fact that I didn't do the audio. So

54
00:03:34,445 --> 00:03:38,045
I figured I'd bring the troublemaker himself onto here. Although, strictly

55
00:03:38,045 --> 00:03:41,489
speaking, you're not the original troublemaker on this. Well,

56
00:03:41,489 --> 00:03:44,930
I participated in it. I'll I'll own my my part of the

57
00:03:44,930 --> 00:03:48,610
trouble. You'll own your part of the trouble. So so I definitely will. Yeah.

58
00:03:48,610 --> 00:03:52,370
What's the background here? Well and and I'll

59
00:03:52,370 --> 00:03:55,445
I'll do a plug for, for andylehner.blog.

60
00:03:56,145 --> 00:03:59,505
And if you go there, you can sign up for my newsletter over on the

61
00:03:59,505 --> 00:04:02,865
right side. It's kinda hard to read because the widget is a little

62
00:04:02,865 --> 00:04:06,385
narrower than it needs to be. But if you if you do that or if

63
00:04:06,385 --> 00:04:10,060
you just look up engineer of data, I think it's

64
00:04:10,060 --> 00:04:11,840
engineer of data dot substack.com.

65
00:04:14,300 --> 00:04:17,980
But I I put a newsletter out today kinda talking about it. Yeah.

66
00:04:17,980 --> 00:04:21,740
There's the site. Thanks, Frank. No problem. And, you see the subscribe

67
00:04:21,740 --> 00:04:25,545
to my newsletter down there on the right, and there's a box on

68
00:04:25,545 --> 00:04:28,585
the left where you type your email address and then on the right, you click

69
00:04:28,585 --> 00:04:32,265
it's free. And it'll it should take you, right over to

70
00:04:32,265 --> 00:04:35,885
Subsec, which by the way, I started using this year. And so far,

71
00:04:36,105 --> 00:04:39,669
I'm pretty impressed. It's it's been a a

72
00:04:39,669 --> 00:04:43,190
really interesting, experience for me. So the

73
00:04:43,190 --> 00:04:46,870
trouble here here's, here's where the

74
00:04:46,870 --> 00:04:50,470
trouble happened. I I have been, reading. I

75
00:04:50,470 --> 00:04:53,895
caught a couple of articles just every now here and then, mostly

76
00:04:54,435 --> 00:04:58,194
on LinkedIn, where people

77
00:04:58,194 --> 00:05:01,875
would express an opinion about, you know,

78
00:05:01,875 --> 00:05:05,640
SSIS stinks. I don't like it. It's old. It's was

79
00:05:05,640 --> 00:05:09,340
so much trouble. And, you know, and they would just

80
00:05:09,480 --> 00:05:13,080
kind of kind of poo poo share their their negative thoughts about

81
00:05:13,080 --> 00:05:16,140
Azure sorry. About SSIS. And

82
00:05:16,920 --> 00:05:20,295
I've, of course, I've worked in SSIS since

83
00:05:20,595 --> 00:05:24,435
really before it came out, I got to work on that Rocks book project

84
00:05:24,435 --> 00:05:28,115
with Brian Knight and I remember that book. Yeah. Yeah. 10

85
00:05:28,115 --> 00:05:31,475
of us. Yes. Back when Rocks would put your picture on the cover of the

86
00:05:31,475 --> 00:05:34,750
book. And have a copy around here somewhere. Yeah. That

87
00:05:35,050 --> 00:05:37,990
yeah. Thank you, Frank. You know, it just but it's

88
00:05:39,190 --> 00:05:42,650
so I got yes. I got kind of a boost out of my career,

89
00:05:43,190 --> 00:05:46,975
and I did an awful lot in SSIS for a long time. And

90
00:05:46,975 --> 00:05:50,655
every now and then, I still do. I used to

91
00:05:50,655 --> 00:05:54,015
deliver training, as part of solid,

92
00:05:54,335 --> 00:05:58,015
solid quality learning is what it was called when I joined it. Solid

93
00:05:58,015 --> 00:06:01,780
queue. After that, I worked with them for a few years and I delivered

94
00:06:01,780 --> 00:06:05,539
training developed by Eric Veerman and

95
00:06:05,539 --> 00:06:08,500
also did consulting gigs. And I learned a lot,

96
00:06:08,900 --> 00:06:12,259
about both data engineering and SSIS while I was

97
00:06:12,259 --> 00:06:16,005
doing both those things. When I left solid q, I

98
00:06:16,005 --> 00:06:18,985
think I put about a year or 2 between me and,

99
00:06:20,165 --> 00:06:23,045
you know, and the business. Actually, it was about two and a half years because

100
00:06:23,045 --> 00:06:26,780
I went to work for Unisys then as a ETL architect. I remember

101
00:06:26,780 --> 00:06:29,900
that. You're up in Reston quite a bit because that's where it was. Oh, yeah.

102
00:06:29,900 --> 00:06:33,660
Yeah. Frank. Now an apartment complex now, that building. Oh, is

103
00:06:33,660 --> 00:06:37,420
it? Okay. I think so. Yeah. Okay. So Frank and I

104
00:06:37,420 --> 00:06:40,780
have been friends since the before times, even before SSIS came

105
00:06:40,780 --> 00:06:44,445
out. And, Well, no. I think you had just written the book at the

106
00:06:44,445 --> 00:06:48,285
time. I I'm trying to remember. So Just moved to Richmond just when I

107
00:06:48,285 --> 00:06:51,905
met. It was November of 2005. December 2005.

108
00:06:52,125 --> 00:06:55,880
Yeah. Yeah. November 2005 is when we met. And,

109
00:06:56,280 --> 00:06:59,980
another mutual friend that I won't name, but we're all still friends now.

110
00:07:00,280 --> 00:07:03,020
And the book actually was published in

111
00:07:03,880 --> 00:07:06,860
January, I think, of 2006.

112
00:07:07,560 --> 00:07:11,275
Yes. That's right. So it wasn't it wasn't quite

113
00:07:11,275 --> 00:07:14,955
ready for for prime time. But oh, sorry. The the

114
00:07:14,955 --> 00:07:18,635
book wasn't out. It was going through the process, and it takes a couple of

115
00:07:18,635 --> 00:07:21,695
months from the from the time all of the drafts are finished

116
00:07:22,155 --> 00:07:25,615
until they they make a book out of it. It was my very first,

117
00:07:26,229 --> 00:07:29,990
book project. And, yeah, I I'm pretty sure I was I was so

118
00:07:29,990 --> 00:07:33,349
excited. I was telling everybody, I worked on a book. Oh, yeah. Yeah. Because it

119
00:07:33,349 --> 00:07:37,189
was for the Richmond Code Camp, which was in May, April of

120
00:07:37,349 --> 00:07:40,785
Yep. 2006. Yeah. 2000

121
00:07:41,565 --> 00:07:45,325
Yeah. It was 2006. You and I. Where I did A team. A team.

122
00:07:45,405 --> 00:07:49,165
Developers on a plane, and I had the guy I photoshopped the

123
00:07:49,165 --> 00:07:52,685
guy carrying your book. That's right. I do remember that.

124
00:07:52,685 --> 00:07:56,500
Yeah. I have to find that picture somewhere. I've been I've been using

125
00:07:56,560 --> 00:08:00,320
SSIS for a a long time. I would say I

126
00:08:00,320 --> 00:08:04,160
learned more about data engineering, the

127
00:08:04,160 --> 00:08:07,985
field and did more projects probably

128
00:08:08,205 --> 00:08:11,824
in, in data warehousing where I used SSIS

129
00:08:12,044 --> 00:08:15,805
for for the data engineering, data integration. I think it's important to to

130
00:08:15,805 --> 00:08:19,505
to, 1, explain for those who may not know what SSIS

131
00:08:19,565 --> 00:08:23,210
is, and 2, explain that data engineering was not always seen as a

132
00:08:23,210 --> 00:08:23,710
discrete,

133
00:08:26,729 --> 00:08:30,250
profession or or Yeah. It's a data engineering's a

134
00:08:30,250 --> 00:08:33,630
relatively new word to describe what we do. It was called

135
00:08:34,010 --> 00:08:35,709
the part of business intelligence.

136
00:08:37,395 --> 00:08:40,835
Back even before all that, I think the first term I

137
00:08:40,835 --> 00:08:43,495
heard was data acquisition,

138
00:08:44,355 --> 00:08:48,115
and it was in it was sometimes that was that phrase

139
00:08:48,115 --> 00:08:51,040
was used standalone. The most often,

140
00:08:51,900 --> 00:08:55,040
at the time when I and this is what got me into databases

141
00:08:55,580 --> 00:08:59,200
was doing system control and data acquisition or SCADA

142
00:08:59,740 --> 00:09:03,440
systems. These were manufacturing systems where you collected data from

143
00:09:03,820 --> 00:09:07,475
instruments on the floor. You gotta remember, IoT

144
00:09:07,475 --> 00:09:11,235
was, you know, still somebody's dream back, you know, in

145
00:09:11,235 --> 00:09:14,915
the 19 nineties. IoT. It was just OT back then. It just was

146
00:09:14,915 --> 00:09:18,694
OT. You're right. It's funny. Yeah. But

147
00:09:18,800 --> 00:09:21,939
but we still did acquire, data from,

148
00:09:23,120 --> 00:09:26,800
plant floors and instruments that were mounted all over, but they weren't

149
00:09:26,800 --> 00:09:30,560
Internet enabled at that time. They were, most of them were hardwired. A

150
00:09:30,560 --> 00:09:34,295
few were using wireless. And so that's kinda what led me

151
00:09:34,295 --> 00:09:38,055
into this whole this whole field. And the idea of the

152
00:09:38,055 --> 00:09:41,815
field, is of data engineering, data

153
00:09:41,815 --> 00:09:45,390
integration as we called it back then, is we do that data acquisition part.

154
00:09:45,610 --> 00:09:48,910
We go find wherever the data lives, we go find it there.

155
00:09:49,209 --> 00:09:52,910
And sometimes the data is a very static

156
00:09:52,970 --> 00:09:56,490
list. It it could be even a text

157
00:09:56,490 --> 00:10:00,185
document, created in notepad that

158
00:10:00,965 --> 00:10:04,345
is tab separated or, you know, delimited

159
00:10:04,405 --> 00:10:08,165
by character position or something like that. And a lot of old

160
00:10:08,165 --> 00:10:12,005
old lookups, lookup data was that way. And I'm not making

161
00:10:12,005 --> 00:10:15,770
that up. It was maintained in a a text EDI.

162
00:10:15,770 --> 00:10:19,610
EDI. Yeah. So electronic data interchange. Yeah.

163
00:10:19,610 --> 00:10:23,370
So, yeah, EDI is I have an interesting stories about EDI, but but one of

164
00:10:23,370 --> 00:10:26,330
the things that really kept me away from the data space for a long time

165
00:10:26,330 --> 00:10:30,134
was I didn't wanna be DBA. And this work, I think, had traditionally

166
00:10:30,134 --> 00:10:33,595
been kind of merged with DBAs. Oh, absolutely.

167
00:10:33,815 --> 00:10:37,194
But at some point, I don't know exactly when it really

168
00:10:38,134 --> 00:10:41,630
evolved into its own discipline. And I remember.

169
00:10:41,930 --> 00:10:45,149
Go ahead. Because I remember I tried to get you a job at a particular

170
00:10:45,769 --> 00:10:49,290
company. I remember that. And what do they do? And what was

171
00:10:49,290 --> 00:10:52,190
it? Why do we need a DBA? You don't need a DBA.

172
00:10:52,805 --> 00:10:56,565
Right. And I think that I'm not DBA. That

173
00:10:56,565 --> 00:11:00,245
was the funny part. Well, that was the fun. Well, we clearly did because at

174
00:11:00,245 --> 00:11:02,904
the time there was a project going on,

175
00:11:04,404 --> 00:11:07,610
and I think the term data architect is what you just said. You were I'm

176
00:11:07,610 --> 00:11:11,450
not a DBM data architect. And then that fell

177
00:11:11,450 --> 00:11:15,210
on deaf ears. And, ironically, like,

178
00:11:15,210 --> 00:11:18,990
not like a couple months later, there was a project that we worked on that,

179
00:11:23,445 --> 00:11:26,665
so many stories, and I'm just trying to protect the innocent and the guilty,

180
00:11:27,525 --> 00:11:31,365
and myself, from from from libels. But, basically,

181
00:11:31,365 --> 00:11:35,125
there was a project going on that when it was basically kind of

182
00:11:35,125 --> 00:11:38,580
behavioral analysis of social networks. Right? This is before

183
00:11:38,580 --> 00:11:41,380
Facebook. I think Myspace was around that sort of thing. But it was basically the

184
00:11:41,380 --> 00:11:45,060
idea of organizational networking as a discipline. And it turned out that the

185
00:11:45,060 --> 00:11:48,820
the software that we bought off the shelf would actually query the

186
00:11:48,820 --> 00:11:51,640
database, bring everything back in from the database,

187
00:11:52,785 --> 00:11:56,385
and then run through the filtering on the C Sharp

188
00:11:56,385 --> 00:11:59,925
components on the web server. Gotcha. So

189
00:12:00,385 --> 00:12:04,145
long story short, there was 0 optimization, hardly an

190
00:12:04,145 --> 00:12:07,365
index. I mean, it was just a mess. A data architect

191
00:12:07,990 --> 00:12:11,590
will use the terms of the day, would have slot spotted this right away. We

192
00:12:11,590 --> 00:12:15,430
didn't. And it was just a massive disaster. And it's kind of one of those

193
00:12:15,430 --> 00:12:18,790
things where there were a number of projects that that company was taking

194
00:12:18,790 --> 00:12:21,915
on. Basically, one of their one of their core

195
00:12:22,295 --> 00:12:26,055
business models was was brilliant actually was software maintenance. So you have an

196
00:12:26,055 --> 00:12:29,735
existing application offshore or outsource it outsource it to

197
00:12:29,735 --> 00:12:33,350
us, and we'll take care of it for you. And,

198
00:12:33,350 --> 00:12:36,490
you know, it was really like an an an education

199
00:12:36,870 --> 00:12:40,709
in kind of Jenga programming. Right? Where you had they wanted updates

200
00:12:40,709 --> 00:12:44,389
to this stuff, but they didn't wanna pay to redo it. So you kinda, like,

201
00:12:44,389 --> 00:12:47,435
had to replace rip and replace stuff. And there's one particular

202
00:12:47,815 --> 00:12:51,575
instance where there was a SQL query that took like 14

203
00:12:51,575 --> 00:12:55,415
minutes to bring back an answer. And I'm like, it's only like

204
00:12:55,415 --> 00:12:58,955
like 30,000 records. Like, what what's the deal here?

205
00:12:59,175 --> 00:13:01,835
And turns out there was no indexes, no nothing.

206
00:13:03,320 --> 00:13:06,540
Well, you know, those indexes take up space. Right? Exactly.

207
00:13:06,680 --> 00:13:09,560
Exactly. I mean, this is like why you should save space.

208
00:13:11,560 --> 00:13:14,680
Joke. That's a joke. For one reason or the other, like, there was there was

209
00:13:14,680 --> 00:13:17,835
no index. And I was like, well, let's add indexes. And like, no, no, no.

210
00:13:17,835 --> 00:13:21,675
We can't change the schema. Okay. So what I end up what I end

211
00:13:21,675 --> 00:13:25,135
up doing was creating temporary tables with indexes

212
00:13:27,399 --> 00:13:30,779
and then copying all the data, and I still got it down to 2 minutes.

213
00:13:31,080 --> 00:13:34,779
Nice. Which 1 minute and 59 seconds was copying the data,

214
00:13:35,800 --> 00:13:39,560
and then one second was actually changing. Yeah. So, like, it was it was kind

215
00:13:39,560 --> 00:13:43,015
of like what I call Jenga programming or Jenga architecture. You had to like they

216
00:13:43,015 --> 00:13:46,695
wanted updates, couldn't touch too much, couldn't change anything,

217
00:13:46,695 --> 00:13:50,535
couldn't improve anything because it was just it was a

218
00:13:50,535 --> 00:13:54,295
time in my career that I think back of and I've kind of learned

219
00:13:54,295 --> 00:13:57,949
many lessons, both hard lessons and soft skill

220
00:13:57,949 --> 00:14:01,490
lessons. But Sure. But we digress. But, I'm just

221
00:14:02,589 --> 00:14:06,370
gonna I'm just gonna take that answering your question in my usual

222
00:14:06,430 --> 00:14:09,889
long winded way. SQL Server Integration

223
00:14:10,029 --> 00:14:13,504
Services, came along, and it was probably

224
00:14:13,524 --> 00:14:17,165
the, again, it was the thing that

225
00:14:17,165 --> 00:14:20,764
spanned the longest part of my career. Before that, I worked with something called

226
00:14:20,764 --> 00:14:24,570
data mirror. That was the first, I'd I'd

227
00:14:24,570 --> 00:14:27,930
say the the first system like that. First bit of software that

228
00:14:27,930 --> 00:14:31,690
way. Before that, I was writing my own. So I

229
00:14:31,690 --> 00:14:35,450
was reading from these plant networks and writing to all

230
00:14:35,450 --> 00:14:38,925
sorts of stuff. And I got into SQL Server because

231
00:14:39,625 --> 00:14:43,324
I crashed access back in the nineties. So I ran,

232
00:14:44,264 --> 00:14:48,024
I collected a 1,000 points of data every second for a long

233
00:14:48,024 --> 00:14:51,730
weekend. And I wanna say the access file grew to about 4 gigs.

234
00:14:51,870 --> 00:14:54,990
When I went to open it and start doing some analysis on it, it turned

235
00:14:54,990 --> 00:14:58,750
out it wouldn't open. So 4 gigs is nothing now. Right? You

236
00:14:58,750 --> 00:15:02,270
can do that on a smartwatch. But back then, a server

237
00:15:02,270 --> 00:15:05,904
struggled, to open the file system. If you go back far

238
00:15:05,904 --> 00:15:09,685
enough, would have freaked out or anything over certain size unless it was, like, NTFS

239
00:15:09,745 --> 00:15:13,264
or something like that. Right? Yeah. And this this

240
00:15:13,264 --> 00:15:16,725
wasn't. This was, one of the other OSs. But

241
00:15:18,450 --> 00:15:20,950
so, you know, I went I went to, altavista.digital.com

242
00:15:22,370 --> 00:15:25,970
and typed in Microsoft database, and I saw this

243
00:15:25,970 --> 00:15:29,110
listing for something called SQL Server, and that's how it all started.

244
00:15:29,615 --> 00:15:33,214
Well, then I I got got in as, working

245
00:15:33,214 --> 00:15:36,735
on a data warehouse, and part of my job moved

246
00:15:36,735 --> 00:15:40,495
into the database part of it. I actually was hired to do the reporting piece

247
00:15:40,495 --> 00:15:44,230
of it, and lots of cool lessons learned there as

248
00:15:44,230 --> 00:15:48,070
well. But on the database side, they use Data Mirror. I

249
00:15:48,070 --> 00:15:51,830
think that company is still around. I'm not sure. But this is like 25 years

250
00:15:51,830 --> 00:15:55,524
ago. And it was it was so cool,

251
00:15:55,524 --> 00:15:59,125
and I was fascinated that somebody had built software to

252
00:15:59,125 --> 00:16:02,904
orchestrate this collection of data. I was like, wow.

253
00:16:03,045 --> 00:16:06,485
That is a good idea. You know, it always makes me feel better, Frank, when

254
00:16:06,485 --> 00:16:10,110
smart people come up with an idea that I've also come up with independently.

255
00:16:10,490 --> 00:16:14,330
It makes me feel like, okay. Maybe I'm onto something. Go through all of

256
00:16:14,330 --> 00:16:18,110
that, data transformation services or DTS, and then finally, SSIS

257
00:16:18,170 --> 00:16:20,190
and this big block. And

258
00:16:21,785 --> 00:16:25,625
what I've noticed and I kinda noticed this trend started

259
00:16:25,625 --> 00:16:29,305
maybe 4 or 5 years ago. I people complained about SSIS before

260
00:16:29,305 --> 00:16:32,665
that. Don't get me wrong. And a lot of it is

261
00:16:32,665 --> 00:16:36,510
because are you sitting down? It's

262
00:16:36,510 --> 00:16:40,190
hard. We're not making it up.

263
00:16:40,190 --> 00:16:43,710
Comparatively, though, like, I I remember when I was at

264
00:16:43,710 --> 00:16:47,390
barnesandnoble.com and which just goes

265
00:16:47,390 --> 00:16:50,875
back a ways. So if you bought a magazine at Barnes and

266
00:16:50,875 --> 00:16:54,635
Noble between 1996 and probably about

267
00:16:54,635 --> 00:16:58,335
2012, 13, you

268
00:16:58,395 --> 00:17:01,760
you interacted with the system I wrote, nice in the late

269
00:17:01,760 --> 00:17:05,599
nineties or at least part of it anyway. So, you

270
00:17:05,599 --> 00:17:09,440
know, that's how I learned EDI, right? Because we get these feeds

271
00:17:09,440 --> 00:17:13,040
from publishers, literally a mainframe would dial up another

272
00:17:13,040 --> 00:17:16,724
mainframe, download the file over a modem.

273
00:17:17,585 --> 00:17:21,185
And, and this is how it worked. And what we did was we pulled down

274
00:17:21,185 --> 00:17:24,945
the raw EDI files and I parsed it

275
00:17:24,945 --> 00:17:28,620
and I had to do that and drop it into an informix database. So it

276
00:17:28,620 --> 00:17:32,380
was a cool writing for GL scripts to to to take

277
00:17:32,380 --> 00:17:36,060
that data in text format and then dump it

278
00:17:36,060 --> 00:17:39,820
into an actual. You were doing data engineering. I was

279
00:17:39,820 --> 00:17:43,660
doing data engineering, which is kind of funny. But like, you know, data engineering as

280
00:17:43,660 --> 00:17:47,445
a discipline is not easy. Right? So SSIS being hard.

281
00:17:47,445 --> 00:17:51,225
I mean, you know, brain surgery brain surgery is hard too. Right?

282
00:17:51,445 --> 00:17:54,645
You you make a good point about it. And it, you know, it took me

283
00:17:54,645 --> 00:17:58,450
a while, especially teaching it. And I would do

284
00:17:58,450 --> 00:18:01,890
4 or 5 day course, originally with solid q and then

285
00:18:01,890 --> 00:18:05,730
eventually on my own. I I wrote my own course. I

286
00:18:05,730 --> 00:18:09,490
found myself adding to Eric's content when I would deliver the

287
00:18:09,490 --> 00:18:12,754
material here. And don't get me wrong. Eric

288
00:18:12,934 --> 00:18:16,654
is still a genius. He was then and he still

289
00:18:16,654 --> 00:18:20,434
is. I just I I had a way of approaching

290
00:18:20,575 --> 00:18:24,140
some, demos and examples that I felt kinda added

291
00:18:24,140 --> 00:18:27,820
to the clarity of the information we were sharing. I

292
00:18:27,820 --> 00:18:31,659
kind of expanded that out and wrote all my own material, my own I

293
00:18:31,659 --> 00:18:35,500
use my own data, that I collect as part of my, weather station

294
00:18:35,500 --> 00:18:39,325
here. And to this day, there are students that

295
00:18:39,325 --> 00:18:43,165
are going through, recordings of that class.

296
00:18:43,165 --> 00:18:46,785
The last recordings I made were back in December 2020,

297
00:18:47,325 --> 00:18:50,760
and I recorded 3 courses on SSIS. The 4

298
00:18:51,140 --> 00:18:54,660
day from 0 to SSIS course was, you know, will take you

299
00:18:54,660 --> 00:18:58,020
from if you can spell SSIS to being a

300
00:18:58,020 --> 00:19:01,860
functional, advanced beginner, low

301
00:19:01,860 --> 00:19:05,375
end intermediate developer. And it was built for

302
00:19:05,375 --> 00:19:09,135
that. It's got labs 13 12, 13 labs

303
00:19:09,135 --> 00:19:12,975
that you do in 2 days, of that course. And then it talks

304
00:19:13,054 --> 00:19:16,835
it kinda changes gears and goes to the care and feeding of SSIS

305
00:19:16,975 --> 00:19:19,580
and ancillary topics. So

306
00:19:20,679 --> 00:19:24,299
I learned a ton about the concepts

307
00:19:25,159 --> 00:19:28,440
of data engineering on while as

308
00:19:28,600 --> 00:19:32,039
while doing SSIS training and consulting and

309
00:19:32,039 --> 00:19:35,735
development. So when I teach it, Frank,

310
00:19:36,355 --> 00:19:40,115
I share these concepts that

311
00:19:40,115 --> 00:19:43,315
I learned. Because you gotta keep in mind, this all came out around the same

312
00:19:43,315 --> 00:19:46,915
time as a data warehouse toolkit book, by,

313
00:19:47,315 --> 00:19:51,029
Kimball and his crew. And the in

314
00:19:51,029 --> 00:19:54,730
fact, I don't know what the relationship was between Microsoft

315
00:19:54,870 --> 00:19:58,570
and Kimbell, but I do know from the horse's mouth

316
00:19:58,950 --> 00:20:02,650
that the, data flow task in SSIS

317
00:20:03,029 --> 00:20:06,395
was modeled to load, Kimball data

318
00:20:06,395 --> 00:20:10,175
warehouses. There's just a lot of functionality baked right in

319
00:20:10,555 --> 00:20:14,315
that, you know, targets those star schemas, and, you know,

320
00:20:14,315 --> 00:20:17,980
it's it's built to do that. There's so, you know, there

321
00:20:17,980 --> 00:20:21,760
was that aspect of it. So at the same time, I'm reading

322
00:20:22,220 --> 00:20:25,520
and learning and, you know, and then going out and teaching

323
00:20:26,059 --> 00:20:29,655
and, you know, and and consulting. There's

324
00:20:29,655 --> 00:20:33,355
this nice amalgam going on. I'm getting information from books.

325
00:20:34,055 --> 00:20:37,575
I'm applying that information on consulting gigs. I'm

326
00:20:37,575 --> 00:20:41,335
figuring out new ways to solve, you know, problems I hadn't

327
00:20:41,335 --> 00:20:44,830
seen before, And then I'm training. So I'm just

328
00:20:44,830 --> 00:20:48,510
rolling all that together. When I do the training, I'm sharing with people, hey. Here's

329
00:20:48,510 --> 00:20:52,290
some first principles, if you will Right. Of data engineering.

330
00:20:52,910 --> 00:20:56,530
And we call it data integration and BI back then.

331
00:20:57,695 --> 00:21:01,415
And star schemas and why you use them and how they work and, you

332
00:21:01,415 --> 00:21:05,175
know, kind of the trade offs that you get. Data explodes a little

333
00:21:05,175 --> 00:21:08,935
bit. Talking about concepts like staging, data,

334
00:21:08,935 --> 00:21:12,750
the benefits of it, why you like, how

335
00:21:12,750 --> 00:21:15,650
you would wanna build your staging tables.

336
00:21:16,670 --> 00:21:20,270
If you're reading from a flat file, everything in

337
00:21:20,270 --> 00:21:23,825
that file is text. Now the text may be

338
00:21:24,045 --> 00:21:27,725
numbers. It may be dates, but it's really just text.

339
00:21:27,725 --> 00:21:31,105
So you built the stage tables with and bar charts.

340
00:21:31,565 --> 00:21:35,085
So at, you know, stuff like that because you wanna get in and get out

341
00:21:35,085 --> 00:21:38,879
just quickly. Memory than the way to do it would be in memory and then,

342
00:21:38,879 --> 00:21:42,320
like, do validation as you do the insert and things like that. There's a there's

343
00:21:42,320 --> 00:21:45,700
a 100 different ways to slice that. Yeah. There really are.

344
00:21:46,000 --> 00:21:49,039
But, you know, when you did, that was that was just pieces and parts of

345
00:21:49,039 --> 00:21:52,875
saying, okay. You know, Tim, I'm teaching you how to use this mechanism,

346
00:21:53,095 --> 00:21:56,615
if you will. Right. SSIS. But I'm also sharing with

347
00:21:56,615 --> 00:22:00,455
you how you would use it and then why you would

348
00:22:00,455 --> 00:22:04,160
use it that way. And, you know, so there's more to

349
00:22:04,160 --> 00:22:07,840
it than just the data engineering. And the point I wanted to make thanks,

350
00:22:07,840 --> 00:22:11,280
Hector. Merry Christmas to you too, Hector. The data

351
00:22:11,280 --> 00:22:14,980
engineering all by itself, just that world,

352
00:22:15,745 --> 00:22:19,265
that's hard all by itself. Yeah. Absolutely. And then the tool

353
00:22:19,265 --> 00:22:22,565
itself was extremely

354
00:22:22,784 --> 00:22:26,465
flexible. And, you know, from the years that you and I have been sharing

355
00:22:26,465 --> 00:22:29,890
about stuff, anytime you say it's it's flexible, you're

356
00:22:29,890 --> 00:22:33,490
also saying, the the it's a sonic way of

357
00:22:33,490 --> 00:22:37,250
saying it's complex. Right. And if

358
00:22:37,250 --> 00:22:41,090
it wasn't flexible, people would say that it's too simple. And, like, it's just

359
00:22:41,090 --> 00:22:43,690
one of those things where now that I'm in a job where I am in

360
00:22:43,690 --> 00:22:47,045
a on the product group, what Microsoft would call PG, a product group or or

361
00:22:47,045 --> 00:22:50,805
team. Yeah. We call it a BU. I I understand. Like, there's

362
00:22:50,805 --> 00:22:54,325
only so many hours in a day that you have engineers and

363
00:22:54,325 --> 00:22:57,480
there's time to market. You have to kind of make these trade offs.

364
00:22:58,040 --> 00:23:01,799
And, you know. That's it. I mean, that's that that I mean,

365
00:23:01,799 --> 00:23:05,240
I had this real eye opening moment with with I think suspect was the guy

366
00:23:05,240 --> 00:23:08,540
who introduced us, who was an evangelist at Microsoft back in the day.

367
00:23:08,919 --> 00:23:12,534
And, you know, I wanted some new shiny feature in Visual

368
00:23:12,534 --> 00:23:16,375
Studio 2005. And, you know, I was complaining about it.

369
00:23:16,375 --> 00:23:20,075
And he kind of pointed out like, look, even Microsoft has limited

370
00:23:20,215 --> 00:23:23,975
resources in terms of people, time and testing and material and

371
00:23:23,975 --> 00:23:27,050
things like that. And I was like, you know, I mean, my god, if Microsoft

372
00:23:27,050 --> 00:23:30,410
has that problem, then I guess everyone has that problem. You know? It turns out

373
00:23:30,410 --> 00:23:33,770
they're just a bunch of software developers just like the rest of us. Turns out

374
00:23:33,770 --> 00:23:36,910
they're all humans. Although maybe now it's mostly AI. Who knows? But,

375
00:23:38,090 --> 00:23:41,525
it's getting there. So so so, you know, I think we both kind

376
00:23:41,525 --> 00:23:45,205
of set the stage for the controversy here. Right. SSI has

377
00:23:45,205 --> 00:23:47,545
been around for at least 20 years,

378
00:23:48,725 --> 00:23:52,485
maybe 25 and SQL Server itself. Let's let's remind folks the

379
00:23:52,485 --> 00:23:56,110
history of SQL Server. It was originally who was it a

380
00:23:56,110 --> 00:23:59,870
partnership with? Sybase? Yes. I believe it was a

381
00:23:59,870 --> 00:24:03,310
Sybase product, completely. And I don't know if it was like And it was like

382
00:24:03,310 --> 00:24:07,090
version 6. Got into the mix, and there was a collaboration

383
00:24:07,630 --> 00:24:10,335
or something, and then they ended up with it,

384
00:24:11,294 --> 00:24:14,975
owning it. That's my best guess on it. I actually

385
00:24:15,054 --> 00:24:18,335
I I know I haven't spoken to her in a while, but I was I'm

386
00:24:18,335 --> 00:24:22,174
friends with and and have co worked with, with

387
00:24:22,174 --> 00:24:25,710
Caitlin Delaney. And she was

388
00:24:25,710 --> 00:24:29,390
with Sybase. Oh, okay. Yep.

389
00:24:29,390 --> 00:24:33,230
So, you know and did we have her as a guest on the show? I

390
00:24:33,230 --> 00:24:36,669
know we wanted to. We we totally need to because that would be Yeah. Interesting

391
00:24:36,669 --> 00:24:39,870
story because I first heard of SQL Server when I was at Barnes and Noble

392
00:24:39,870 --> 00:24:43,225
because at the time we were ready to launch in 19 this is why I

393
00:24:43,225 --> 00:24:46,905
left Barnes and Noble. We're ready to launch by Christmas of 96 with a

394
00:24:47,065 --> 00:24:50,905
Yeah. Linux or Unix based based system based on Spark, Oracle,

395
00:24:50,905 --> 00:24:54,610
and a few other things. No. I'm sorry. 4 g l. It was Ultimate

396
00:24:54,610 --> 00:24:58,290
Formics. And, you know, we had the hardware. We had

397
00:24:58,290 --> 00:25:02,130
everything set up. And then as the story goes, Bill

398
00:25:02,130 --> 00:25:05,650
Gates and, one of the Riggio brothers who was the CEOs

399
00:25:05,650 --> 00:25:09,145
of kind of co CEOs of Barnes and Noble at the time.

400
00:25:11,125 --> 00:25:14,585
Bill Gates had kind of I don't know what he'd done, Jedi Mind Trick.

401
00:25:15,525 --> 00:25:18,425
In August, September of, like, 96,

402
00:25:19,580 --> 00:25:23,420
basically said, no, we're ripping everything we've built so far and we're moving

403
00:25:23,420 --> 00:25:27,180
it over to Microsoft tooling, which at the time was not really mature. I

404
00:25:27,180 --> 00:25:30,300
mean, it was this is like inter dev. I think we had a beta version

405
00:25:30,300 --> 00:25:33,280
of visual inter dev. Yeah. Yeah. Which

406
00:25:33,965 --> 00:25:37,805
was not the best product at the time. Right? It was, you know You know,

407
00:25:37,805 --> 00:25:41,025
I used it At the time. At the time.

408
00:25:41,325 --> 00:25:44,845
Yeah. I I used it, and if you came

409
00:25:44,845 --> 00:25:48,225
from, like, cold fusion or some other development platform.

410
00:25:48,529 --> 00:25:52,070
Yes. Was also awful. Yes. But yeah. So

411
00:25:52,289 --> 00:25:55,990
So I started on inter dev. In fact, that was the first tool

412
00:25:56,529 --> 00:25:59,889
that I I remember downloading for, Visual

413
00:25:59,889 --> 00:26:03,575
Studio. I don't think I downloaded it. I think I went somewhere and bought a

414
00:26:03,575 --> 00:26:07,255
CD or something. Yeah. Yeah. I think I found it in her dev 97 CD,

415
00:26:07,255 --> 00:26:11,015
which was the the second or third version. But, I mean, I we we had

416
00:26:11,015 --> 00:26:14,520
everything written in per on CGI Pearl scripts. Like, we had everything,

417
00:26:14,820 --> 00:26:18,580
and it was just a very different era. But my

418
00:26:18,580 --> 00:26:22,100
take was and this was my I was at the meeting with the CEO and

419
00:26:22,100 --> 00:26:24,920
everyone else. Like, if we don't launch by this Christmas,

420
00:26:26,195 --> 00:26:29,735
people are not going to use us as a habit. Amazon will

421
00:26:30,115 --> 00:26:33,875
take the mindshare and this and that. And then then the

422
00:26:33,875 --> 00:26:37,715
CEO said, sit down, s t f u. Basically, you don't know

423
00:26:37,715 --> 00:26:41,550
how to sell books. You may know technology, but you don't know how

424
00:26:41,550 --> 00:26:45,390
to sell books. Now we can look back at Jeff

425
00:26:45,390 --> 00:26:48,990
Bezos' super yacht and his, you know, moon

426
00:26:48,990 --> 00:26:52,590
missions and all that. These guys have super yachts and moon missions.

427
00:26:52,590 --> 00:26:56,205
Right? They do not, actually. Oh. And my well, I

428
00:26:56,205 --> 00:26:59,825
mean, I'm pretty sure they live in an oceanfront thing in Long Island. But,

429
00:27:01,645 --> 00:27:05,485
he didn't know anything about selling books online either. So I can kinda I

430
00:27:05,485 --> 00:27:09,160
can sit back here, you know, some, you know, good God almost 30 years

431
00:27:09,160 --> 00:27:12,140
later and kind of be smug about it. Right? Right. But

432
00:27:14,200 --> 00:27:17,240
it's just it's just funny. Right? Like, so so what's interesting is and I think

433
00:27:17,240 --> 00:27:21,000
this really cuts to the bone of what this controversy is. And I

434
00:27:21,000 --> 00:27:24,325
have the thing queued up. I can kind of show the screen where you posted

435
00:27:24,325 --> 00:27:26,424
it, where

436
00:27:29,125 --> 00:27:32,184
the fundamentals haven't really changed. Not at all.

437
00:27:32,804 --> 00:27:36,105
Right. Yeah. Binary is still binary.

438
00:27:38,340 --> 00:27:42,100
The debates about schema optimization and things like that are still

439
00:27:42,100 --> 00:27:45,700
very much the same today as they were

440
00:27:45,700 --> 00:27:49,480
20 years now. The numbers are bigger. The stakes are arguably bigger.

441
00:27:49,845 --> 00:27:53,684
But for the most part, the fundamentals haven't changed. And and

442
00:27:53,684 --> 00:27:56,485
I would say this is really where it kind of boiled down to. And this

443
00:27:56,485 --> 00:27:59,544
is this is where the controversy starts. So buckle up, kids.

444
00:28:00,325 --> 00:28:03,740
Let's see. I will share the screen. There's actually

445
00:28:03,740 --> 00:28:07,340
2. I think you talked about one of them. The choices? I'm only

446
00:28:07,340 --> 00:28:10,880
aware of 1. This is the this is the one post,

447
00:28:11,419 --> 00:28:15,100
and I'll drop you the link, to to one of

448
00:28:15,100 --> 00:28:18,399
the others. Right? Yeah. I'll put it in a chat.

449
00:28:18,755 --> 00:28:22,355
I'll I'll send that to you here. Just a second. Along can can understand. So

450
00:28:22,355 --> 00:28:26,035
this is what I saw. And it was basically Kendra

451
00:28:26,035 --> 00:28:29,255
Little, who was a I would say legendary. Scary

452
00:28:29,475 --> 00:28:33,180
smart. She's legendary in in in in the sequel

453
00:28:33,240 --> 00:28:36,920
kind of family, right? Hashtag sequel family. Is that still a thing? I

454
00:28:36,920 --> 00:28:40,360
think so. She's legendary. She used to work at Redgate. I think she worked at

455
00:28:40,360 --> 00:28:43,100
Microsoft, too, at a time.

456
00:28:44,120 --> 00:28:47,495
I think so. But I'm not positive. Well, we can look at LinkedIn. If only

457
00:28:47,495 --> 00:28:50,075
we had that information. But anyway,

458
00:28:51,255 --> 00:28:54,934
so you basically so if you read this and she says

459
00:28:55,015 --> 00:28:58,615
so it says strong disagree. Don't run after every shiny

460
00:28:58,615 --> 00:29:02,460
thing. Again, that is good advice. But, Lord, I would assume

461
00:29:02,840 --> 00:29:06,280
that is her saying. But Lord, don't learn SQL Server and

462
00:29:06,280 --> 00:29:09,960
SSIS if you want to be a data engineer. That's 2 decades too

463
00:29:09,960 --> 00:29:13,720
out of date. Sincerely, a SQL Server expert. I think that's

464
00:29:13,720 --> 00:29:17,495
a bit harsh. She's right about this part. Don't try to change chase out

465
00:29:17,495 --> 00:29:21,255
there if you show anything. So apparently, I can't, and I

466
00:29:21,255 --> 00:29:23,995
can't select a thing. So I read that,

467
00:29:24,855 --> 00:29:28,410
and and I know there's more controversies that are in there as I as

468
00:29:28,410 --> 00:29:30,990
I look at the thing. And you said I humbly submit

469
00:29:32,010 --> 00:29:35,630
data engineering may be accomplished even in the year of our Lord, 2024,

470
00:29:36,330 --> 00:29:39,930
using T SQL, this foul year of our

471
00:29:39,930 --> 00:29:43,675
Lord, 2024. To borrow a phrase from Hunter Thompson,

472
00:29:44,375 --> 00:29:48,075
T SQL, SIS, ADF, Fabric Data Factory

473
00:29:48,215 --> 00:29:51,675
and other technologies supported by Microsoft, which I thought

474
00:29:52,375 --> 00:29:56,150
clearly Microsoft's not going anywhere. Right? Yeah.

475
00:29:57,010 --> 00:30:00,550
And so I basically said

476
00:30:01,090 --> 00:30:04,390
fundamentals never grow out of style. Then I think I wrote again somewhere

477
00:30:04,450 --> 00:30:08,290
like when I looked at the context of it because that's

478
00:30:08,290 --> 00:30:11,155
not what you're supposed to do apparently in social media. You're supposed to react right

479
00:30:11,155 --> 00:30:14,915
away. I did that, by the way, Frank.

480
00:30:14,915 --> 00:30:18,515
I'm guilty. I did not go look at the context. So this is the

481
00:30:18,515 --> 00:30:22,195
original context. I well and, you know,

482
00:30:22,195 --> 00:30:25,800
you pointed that out and I'll I'll be honest, I I'm still

483
00:30:25,800 --> 00:30:29,640
running on second hand information. I have not yet clicked it and gone back

484
00:30:29,640 --> 00:30:33,400
to, to our guest post. Now I can see

485
00:30:33,400 --> 00:30:37,075
it. Now you can see. So so this is what struck me is, well. This

486
00:30:37,075 --> 00:30:39,635
is what struck me as odd. And I know we had talked about it and

487
00:30:39,635 --> 00:30:42,195
I had talked about it. You talked about it. We talked to each other about

488
00:30:42,195 --> 00:30:45,475
it. You know, we talked to our dogs about it. I don't know. Like, but

489
00:30:45,475 --> 00:30:49,075
like, it was kind of like so so when I read the thing, it gets

490
00:30:49,075 --> 00:30:52,830
even stranger. Right? So Yeah. He was talking to

491
00:30:52,830 --> 00:30:56,530
someone, and I guess strictly speaking, even this is secondhand knowledge. Right?

492
00:30:56,670 --> 00:31:00,290
But, so that's the data

493
00:31:00,430 --> 00:31:04,030
scientist in there. Like, well, strictly speaking, this data is also all right. So

494
00:31:04,030 --> 00:31:07,385
so look looking to someone to get a job as a data engineer. Okay?

495
00:31:07,525 --> 00:31:11,145
Right. Unfortunately, he was learning about LLMs and other ML stuff.

496
00:31:11,605 --> 00:31:15,445
I'm like, that's not data engineering. That's a

497
00:31:15,445 --> 00:31:19,290
AI engineering or data science type work. That's more like I think he's he's trying

498
00:31:19,290 --> 00:31:22,890
to set him straight from that. He's like, you're learning the wrong things. That's how

499
00:31:22,890 --> 00:31:26,650
I read those two sentences. I mean, I would say you're learning the right things

500
00:31:26,650 --> 00:31:29,630
if you wanna be an AI practitioner.

501
00:31:30,394 --> 00:31:33,774
Yeah. But I wouldn't call I wouldn't, you know, read up on Langchain,

502
00:31:34,475 --> 00:31:38,315
you know, Ollama and anything LLM and all that stuff

503
00:31:38,315 --> 00:31:42,075
and then call myself a data engineer. I mean, that's Yeah.

504
00:31:42,154 --> 00:31:46,000
That's like a cardiologist cutting up you know, doing your taxes. You

505
00:31:46,000 --> 00:31:49,780
know what I mean? Like Sure. Or or cutting open your brain. Like,

506
00:31:50,160 --> 00:31:53,920
I mean, I suppose there's some similarities, but it's not

507
00:31:53,920 --> 00:31:57,380
the same. Well, I I do like bullet number 1.

508
00:31:57,520 --> 00:32:01,325
Yeah. You know, let's see that. Yeah. This is something I think

509
00:32:01,325 --> 00:32:04,685
that you point out quite a bit. So when you give your talks, either on,

510
00:32:05,165 --> 00:32:08,925
SSIS or ADF, you ask

511
00:32:08,925 --> 00:32:12,645
people, like, how many people here have workloads running in the

512
00:32:12,645 --> 00:32:16,290
in the cloud or right? And then only a quarter of the hands

513
00:32:16,290 --> 00:32:19,970
go up. Well, it's it grew to about

514
00:32:19,970 --> 00:32:23,730
40% the last time I did it, but it's been over a year

515
00:32:23,730 --> 00:32:27,490
since I since I spoke live and asked that question, ran

516
00:32:27,490 --> 00:32:30,855
that little survey. There's a slide usually hidden in,

517
00:32:32,355 --> 00:32:35,955
all of my presentations that has survey up near the very

518
00:32:35,955 --> 00:32:39,655
top. Right. You know, it just and that's that's what the survey is about.

519
00:32:40,435 --> 00:32:43,880
And often, especially

520
00:32:44,100 --> 00:32:47,940
say the last, I said it's been over a year. So let's say from a

521
00:32:47,940 --> 00:32:51,299
year ago and then back maybe 4 years of asking that

522
00:32:51,299 --> 00:32:54,980
question. Almost every time I did that and people

523
00:32:54,980 --> 00:32:58,775
didn't see everyone else's hand go up with theirs,

524
00:32:59,475 --> 00:33:03,155
the those people would come up to me at the end. And usually, their

525
00:33:03,155 --> 00:33:06,135
first comment was, I didn't know

526
00:33:06,515 --> 00:33:10,115
that it was most of the people here were not doing

527
00:33:10,115 --> 00:33:13,820
production jobs in the cloud at this point point with data. I thought we

528
00:33:13,820 --> 00:33:17,580
were way behind and we're the only ones. And my response would be

529
00:33:17,580 --> 00:33:21,200
2 fold. The first would be, that's because Microsoft

530
00:33:21,340 --> 00:33:25,120
marketing is doing an astounding job. That is not a swipe

531
00:33:25,425 --> 00:33:28,565
at Microsoft Marketing. If anything, they deserve a raise

532
00:33:29,185 --> 00:33:32,405
because they were so effective at communicating

533
00:33:32,945 --> 00:33:36,465
how cool this is Right. And how these larger

534
00:33:36,465 --> 00:33:40,005
companies are doing it. You all of the big shows, keynotes,

535
00:33:40,610 --> 00:33:44,210
There's some list of big companies, and they're almost all of them or

536
00:33:44,210 --> 00:33:47,110
companies that you'd wanna work for because it's prestigious.

537
00:33:47,970 --> 00:33:50,930
That's so I don't know if you want it on my personal market. Seem like

538
00:33:50,930 --> 00:33:54,370
everybody's doing it. And I I know I know for a fact it's not always

539
00:33:54,370 --> 00:33:58,105
true because when I worked in the sales for Microsoft, we

540
00:33:58,105 --> 00:34:01,945
would encounter them and there was a pejorative term used internally called server

541
00:34:01,945 --> 00:34:05,785
huggers. Okay. Right. Because like, oh, they're

542
00:34:05,785 --> 00:34:08,205
server huggers. They'll never go to Azure. Right.

543
00:34:09,369 --> 00:34:12,409
So so now, you know, I used to see that it's server hugger as a

544
00:34:12,409 --> 00:34:16,250
pejorative. Now in light of kind of maturity and

545
00:34:16,250 --> 00:34:20,010
working, with more customers and being

546
00:34:20,010 --> 00:34:23,725
more aligned in the open source kind of realm and dealing with

547
00:34:23,725 --> 00:34:27,025
international customers who have very real regulatory concerns.

548
00:34:27,485 --> 00:34:30,945
You're right. Call them smart. Right. It's not, you know,

549
00:34:31,804 --> 00:34:35,565
I didn't so much drink the Kool Aid is I became one with

550
00:34:35,565 --> 00:34:38,020
the Kool Aid. You couldn't tell where I ended and where it began, where I

551
00:34:38,020 --> 00:34:41,480
kind of had this deep programing experience

552
00:34:41,619 --> 00:34:45,300
of. Yeah. That's not always the answer. Right. And I

553
00:34:45,300 --> 00:34:48,819
think that dealing with LLMs and AI and things like

554
00:34:48,819 --> 00:34:52,375
that, I think really makes that more obvious.

555
00:34:52,375 --> 00:34:56,214
Right? Yeah. I totally agree with that. And, you know,

556
00:34:56,214 --> 00:34:59,654
to be fair, and I wanna start with, you know, with being as

557
00:34:59,654 --> 00:35:03,095
positive as I can about this. If I was It's not a negative on any

558
00:35:03,174 --> 00:35:06,535
from scratch. Wasn't. No. I I'm just saying. But if I'm starting

559
00:35:06,535 --> 00:35:10,290
today, day 1, and I wanna go, be a

560
00:35:10,290 --> 00:35:14,130
starter company and and work with data, I it

561
00:35:14,130 --> 00:35:17,490
would be foolish. Foolish to start

562
00:35:17,490 --> 00:35:20,309
today and not go to the cloud. Absolutely.

563
00:35:20,915 --> 00:35:24,435
So and and the reasons are numerous. Yeah. Here's the

564
00:35:24,435 --> 00:35:27,955
thing. The companies there are a handful of

565
00:35:27,955 --> 00:35:31,494
companies, really large companies, mind you,

566
00:35:31,955 --> 00:35:35,700
that have started sent in the cloud age. Let's just call it

567
00:35:35,700 --> 00:35:39,460
that, or the Internet age. There's a small number of them

568
00:35:39,460 --> 00:35:42,920
that have gone on to be huge, but they are really huge.

569
00:35:43,380 --> 00:35:46,980
They're overpowering, oversized. They're larger than the

570
00:35:46,980 --> 00:35:50,599
companies that are previous to the Internet age companies

571
00:35:51,105 --> 00:35:54,945
that have made their way into the Internet. And that's that's

572
00:35:54,945 --> 00:35:58,785
not an accident. However, those

573
00:35:58,785 --> 00:36:02,545
companies, the brick and mortar companies, are the companies calling consultants like

574
00:36:02,545 --> 00:36:05,859
me and asking me to help them either

575
00:36:06,160 --> 00:36:08,900
transition from a purely on premises

576
00:36:09,599 --> 00:36:12,740
environment, managing their data into a cloud environment

577
00:36:13,200 --> 00:36:16,799
or the and back before that, in 20 years ago, when I was first

578
00:36:16,799 --> 00:36:20,405
getting called to do this kind of work, they were just trying to figure out

579
00:36:20,405 --> 00:36:24,245
how to collect their data and then analyze it. And

580
00:36:24,245 --> 00:36:28,005
so, you know, SSIS was a great way to do that. T

581
00:36:28,005 --> 00:36:31,685
SQL was everywhere. Azure Data Factory didn't exist. Yes.

582
00:36:31,685 --> 00:36:35,510
Much less Fabric Data Factory. And so we were just trying to solve

583
00:36:35,510 --> 00:36:39,210
this business problem. And I was trying to couch couch my responses,

584
00:36:39,350 --> 00:36:43,110
especially there was a thread that that got combative, I

585
00:36:43,110 --> 00:36:46,870
would say. And, you know, as we went went down

586
00:36:46,870 --> 00:36:50,225
through that, and I kept trying to say, and

587
00:36:50,685 --> 00:36:54,445
I did. I said over and over again that, you know,

588
00:36:54,445 --> 00:36:57,825
my job is to go help solve these business problems.

589
00:36:58,285 --> 00:37:01,885
And what I meant by that opening line,

590
00:37:01,885 --> 00:37:05,630
that T SQL, SSIS, Azure Data Factory, Fabric Data

591
00:37:05,630 --> 00:37:09,170
Factory, even in 2024 of viable ways to accomplish

592
00:37:09,630 --> 00:37:13,470
data engineering. I I meant that, and I'm not back backing

593
00:37:13,470 --> 00:37:17,170
off that for one minute. I I misunderstood the context of the question,

594
00:37:17,975 --> 00:37:21,435
and I didn't really understand until I listened to your stream

595
00:37:21,815 --> 00:37:25,575
last night where you had gone back and done what I should have done and

596
00:37:25,575 --> 00:37:29,355
read the original post. And you said, yeah. It's kind of a mixed mesh

597
00:37:29,940 --> 00:37:33,720
post. The guy's talking about data engineer, but he's also talking about LLMs

598
00:37:34,339 --> 00:37:37,940
and machine learning. And in the middle of that, he

599
00:37:37,940 --> 00:37:41,560
throws out, you know, this comment about SSIS,

600
00:37:41,619 --> 00:37:45,375
how 90 99.5% of the companies are still using. I

601
00:37:45,375 --> 00:37:49,135
think that estimate is high. I I think it was more of

602
00:37:49,135 --> 00:37:52,975
a, let's make this point that there's still a lot of companies out there

603
00:37:52,975 --> 00:37:56,495
using, T SQL and SSIS to

604
00:37:56,495 --> 00:38:00,015
accomplish this. And this is something that I can't find the comment that I put

605
00:38:00,015 --> 00:38:03,610
in there. I'm looking for it now, but. Yeah. Some of the

606
00:38:03,610 --> 00:38:07,370
comments I can't get to anymore. I don't know why. Maybe they were

607
00:38:07,370 --> 00:38:11,210
reported or maybe they're. Who knows? Right. I mean, social media

608
00:38:11,210 --> 00:38:14,830
does weird things to people psychology. But the

609
00:38:16,835 --> 00:38:20,135
point that I think that I wanna say

610
00:38:21,155 --> 00:38:24,375
that Kendra overlooks. I think everyone overlooks it.

611
00:38:25,954 --> 00:38:29,714
Data and back end systems have a

612
00:38:29,714 --> 00:38:33,530
longer shelf life. And I say

613
00:38:33,530 --> 00:38:37,210
this as someone who was, what, 10, 15 years ago,

614
00:38:37,210 --> 00:38:40,570
strongly ensconced in client development. Right? Whether it was your

615
00:38:40,570 --> 00:38:44,410
Windows, Windows Phone, or other types of Windows

616
00:38:44,410 --> 00:38:47,335
based devices. Right. Or web development. Right.

617
00:38:48,835 --> 00:38:52,055
Those technologies turn over pretty quickly.

618
00:38:52,275 --> 00:38:55,954
Right. You know, you're likely to get

619
00:38:55,954 --> 00:38:59,714
multiple updates per year on a device phone, like an app on

620
00:38:59,714 --> 00:39:03,220
a device, but you're likely to never see,

621
00:39:05,040 --> 00:39:08,720
a radical change or redesign. You'll you'll see a

622
00:39:08,720 --> 00:39:12,420
radical change or web redesign of a website or portions of a website

623
00:39:13,045 --> 00:39:16,745
couple times a year maybe. Right? But you're never gonna see

624
00:39:17,365 --> 00:39:20,905
a radical redesign of a data back end system,

625
00:39:21,845 --> 00:39:25,605
but once or twice a decade. And It's true.

626
00:39:25,605 --> 00:39:28,585
Yeah. And mostly what drives that is scale,

627
00:39:29,100 --> 00:39:32,940
not features. Right. Not features. It's just date or just tend yeah.

628
00:39:32,940 --> 00:39:36,460
Exactly. Right? So if you a 100 x and who could who could accounted for

629
00:39:36,460 --> 00:39:39,980
that, you know, going to the project started. It's a problem. Still a

630
00:39:39,980 --> 00:39:43,575
wonderful problem to have, but but a problem nonetheless. Well, and there's

631
00:39:43,575 --> 00:39:47,335
also the fact that, you know, it's 2,000 whatever now, and

632
00:39:47,335 --> 00:39:50,935
there's still mainframes running. Right? There are still not not not to to

633
00:39:50,935 --> 00:39:54,715
knock on IBM too hard because they are the company of Red Hat. But,

634
00:39:55,655 --> 00:39:59,200
d b 2 is still around, still getting updates. Still backbone of

635
00:39:59,200 --> 00:40:02,960
many Fortune 100 companies that also share the stage with Satya

636
00:40:02,960 --> 00:40:06,640
at these big Microsoft events too. Right? Like Which was mind blowing

637
00:40:06,640 --> 00:40:10,480
for people from the old days of Microsoft. Right? Well,

638
00:40:10,480 --> 00:40:14,315
that's a whole other thing. But, like, you know but, I

639
00:40:14,315 --> 00:40:18,155
mean, it it really boils down to, like, these technologies have a longer shelf

640
00:40:18,155 --> 00:40:21,915
life. So if something is 20 I think we get

641
00:40:21,915 --> 00:40:24,954
hung up. 1 of the threads sub threads in here gets hung up on, you

642
00:40:24,954 --> 00:40:28,740
know, 30, 20 year old technology. We're thinking that, well, you know,

643
00:40:28,740 --> 00:40:32,120
there's a meme of the the little monkey puppet, like, you know,

644
00:40:32,420 --> 00:40:35,940
giving a side eye and then goes like a cringe face, like, and a side

645
00:40:35,940 --> 00:40:39,460
eye where it's like, oh, Windows is, you know, I don't know, 40 year old

646
00:40:39,460 --> 00:40:43,265
technology. And I'm thinking, like, some, you know, Unix people or Linux slash

647
00:40:43,265 --> 00:40:46,785
Linux people are, like, 40 years old is old. You

648
00:40:46,785 --> 00:40:50,545
know? I mean, this stuff goes back much further. So it's but

649
00:40:50,545 --> 00:40:54,224
it's still like and that's not a knock. It's just

650
00:40:54,339 --> 00:40:57,120
No. It's just now that we're in this

651
00:40:57,980 --> 00:41:01,500
industry now for as long as we've been in it and the

652
00:41:01,500 --> 00:41:05,100
industry's been around longer this long, there's just

653
00:41:05,100 --> 00:41:08,780
stuff that is gonna just start aging out, but it doesn't age out as

654
00:41:08,780 --> 00:41:11,955
quickly as we think it does. It's not like it's not like the iPhone. Sure.

655
00:41:11,955 --> 00:41:15,665
Right? Where you the iPhone I don't know what number they up

656
00:41:15,665 --> 00:41:19,375
to. 16, 17. Right? Oh, well, suddenly my iPhone 15 looks bad, and that

657
00:41:19,375 --> 00:41:23,085
happens every year or 2. You this you don't see that in

658
00:41:23,085 --> 00:41:26,839
database systems. Right? The only impetus to really move, say, from, like, SQL Server

659
00:41:26,839 --> 00:41:30,680
2,005 to 2019 is updates stop going. Right?

660
00:41:30,680 --> 00:41:34,519
And that's a whole big project. Yeah. The maintenance cycle. So it goes out

661
00:41:34,519 --> 00:41:38,299
of maintenance. And then you worry that if something crazy happens,

662
00:41:39,305 --> 00:41:42,825
you can't get support for it. And that's

663
00:41:42,825 --> 00:41:46,665
kinda like, you know, it's it it's sort of it I'll say this.

664
00:41:46,665 --> 00:41:50,185
It's analogous to your phone starting to run slow for some unknown

665
00:41:50,185 --> 00:41:53,800
reason. That's funny. Something something on SQL.

666
00:41:53,800 --> 00:41:57,640
Yeah. Something something. Sybase something. Well, and you think about all the

667
00:41:57,800 --> 00:42:01,400
I mean, I mean, and contrary to this, contrary to that statement of these things

668
00:42:01,400 --> 00:42:04,920
have long life shelf lives. Yeah. Is the fact that I mentioned

669
00:42:04,920 --> 00:42:08,695
Informix earlier. Raise your hand if you heard of Informix. Right?

670
00:42:08,755 --> 00:42:12,515
So I've heard of Informix. You've heard of Informix? I mean, we don't count.

671
00:42:12,515 --> 00:42:16,035
But but, like no. But, like, I remember my first

672
00:42:16,035 --> 00:42:19,770
experience with Informix was because some alum of Fordham had

673
00:42:19,770 --> 00:42:23,130
because it was a big shot at Informix. And, and I think we had somebody

674
00:42:23,130 --> 00:42:26,590
who was also a big shot at Silicon Graphics. So we had SGI machines

675
00:42:27,050 --> 00:42:30,810
running at Formix. Right. So I remember my first UNIX I used was an

676
00:42:30,810 --> 00:42:34,565
IRIX system. Right. Which most people today

677
00:42:34,565 --> 00:42:38,404
wouldn't even know what what that means. Right. And, you know, but Informix is

678
00:42:38,404 --> 00:42:40,265
out of business. Sybase is gone.

679
00:42:42,484 --> 00:42:46,325
I can't even think of other names. I know there's more.

680
00:42:46,325 --> 00:42:49,940
Right. But really, the only things that it those have probably been

681
00:42:49,940 --> 00:42:52,760
migrated to SQL Server or Oracle. Well,

682
00:42:53,299 --> 00:42:57,140
or some form of Postgres or something like that. And I

683
00:42:57,140 --> 00:43:00,920
I hear you. You know, there's there's an argument to be made

684
00:43:01,299 --> 00:43:04,744
for, you know, the the cost of maintaining

685
00:43:05,125 --> 00:43:08,185
old software. Right. There there definitely is.

686
00:43:10,085 --> 00:43:13,705
I'll say this about SSIS. I if you learned

687
00:43:14,565 --> 00:43:18,180
SSIS in probably in 2,005 era, between

688
00:43:18,180 --> 00:43:21,460
2,005 and 2,008, that engine

689
00:43:21,940 --> 00:43:25,540
I I don't know how many lines of code were

690
00:43:25,540 --> 00:43:29,035
changed before it was upgraded to 2008

691
00:43:29,175 --> 00:43:32,855
or r two, but it changed. There were some performance tweaks in there. It was

692
00:43:32,855 --> 00:43:36,695
obviously, faster. And then again, that happened in

693
00:43:36,695 --> 00:43:40,075
the 2012, error when we saw

694
00:43:40,599 --> 00:43:42,220
I love that SQL tab.

695
00:43:44,599 --> 00:43:48,119
You know, it's dead. Long live Crystal. So was Crystal ever database or was it

696
00:43:48,119 --> 00:43:51,720
just Crystal Reports? Reports is all I knew. I didn't know about it as a

697
00:43:51,720 --> 00:43:54,700
database. That's all I I I use Crystal Reports and

698
00:43:55,484 --> 00:43:59,184
my favorite thing was it filling up the drive because

699
00:43:59,325 --> 00:44:02,684
it kept caching things. But I

700
00:44:02,684 --> 00:44:06,444
remember the whole idea of just because you place it somewhere, it doesn't

701
00:44:06,444 --> 00:44:09,484
mean it's actually gonna end up there. Like, the whole thing is

702
00:44:10,220 --> 00:44:14,000
but sorry to cut off. That's okay. But SSIS in general,

703
00:44:14,059 --> 00:44:17,819
if you learned it even in, you know, 2006, came out

704
00:44:17,819 --> 00:44:21,660
in November, I think, of 2,005. Even if

705
00:44:21,660 --> 00:44:25,435
you learned it then, it at at a fundamental level, it

706
00:44:25,435 --> 00:44:29,275
hasn't changed that much. And whereas you'll see other software

707
00:44:29,275 --> 00:44:32,815
you Visual Studio is, you know, a software development platform

708
00:44:33,195 --> 00:44:36,315
that allows you to do c sharp and v v and, you know, all of

709
00:44:36,315 --> 00:44:39,569
the stuff. And it allows it still supports for us. I know. I haven't tried

710
00:44:39,569 --> 00:44:43,170
v v in 9 years now. So

711
00:44:43,170 --> 00:44:46,869
it's been a while. But if you look at

712
00:44:47,730 --> 00:44:51,295
how much most software changes from a developer

713
00:44:51,295 --> 00:44:54,895
perspective, and SSIS is software development. So as your

714
00:44:54,895 --> 00:44:58,515
data factory, and any data engineering, that software development,

715
00:44:58,895 --> 00:45:02,595
SSIS is probably in the 95%

716
00:45:03,849 --> 00:45:06,510
of what it was. If if you knew the fundamentals

717
00:45:07,369 --> 00:45:10,810
in 2006, you know those fundamentals in

718
00:45:10,810 --> 00:45:14,589
2024. And Right. Part of the decision

719
00:45:14,730 --> 00:45:18,194
to go make the upgrade, we talked about, you know, maintenance wonders and stuff, and

720
00:45:18,194 --> 00:45:21,675
I I get it. And it's not the same as your phone slowing down. I

721
00:45:21,675 --> 00:45:25,355
said that, but that's a bad analogy. But Well, it is also

722
00:45:25,355 --> 00:45:29,195
it's also, I think, also very relevant to Windows 10. Right? If you're

723
00:45:29,195 --> 00:45:32,950
on Windows 10, your updates are gonna stop in October. That's

724
00:45:32,950 --> 00:45:36,790
true. I don't wanna get on that soapbox and rant. Sure. No. But I

725
00:45:36,790 --> 00:45:40,550
I mean, there's I get reasons for that as well. I don't

726
00:45:40,550 --> 00:45:44,230
like that it's gonna change because I like Windows 10. But,

727
00:45:44,950 --> 00:45:48,615
but yeah. Well, there's there's I'm gonna join you in not

728
00:45:48,615 --> 00:45:51,755
going down that road. But I'll say this. Hey,

729
00:45:52,215 --> 00:45:55,895
Maddie. How are you? The, the

730
00:45:56,055 --> 00:45:59,895
just the fundamentals of data engineering haven't changed. And the

731
00:45:59,895 --> 00:46:03,350
tool itself, you know, if you knew it back then.

732
00:46:03,350 --> 00:46:06,950
And it's, you know, you know it now. And so if you

733
00:46:06,950 --> 00:46:10,710
learned it now, you could go back then and still work

734
00:46:10,710 --> 00:46:14,470
in the previous versions of it with, very little headache. And

735
00:46:14,470 --> 00:46:18,165
that speaks a lot speaks volumes to the,

736
00:46:18,645 --> 00:46:22,345
the team that designed and built that. And

737
00:46:23,285 --> 00:46:27,045
so in addition to the technical reasons for doing this, the business

738
00:46:27,045 --> 00:46:30,840
reasons, kind of revolve around one of my favorite

739
00:46:30,840 --> 00:46:34,460
phrases. I mentioned this in today's newsletters. It's a compelling

740
00:46:34,520 --> 00:46:38,220
reason. Do you have a compelling reason to make this change?

741
00:46:38,360 --> 00:46:41,420
And business people think about this all day every day because

742
00:46:42,200 --> 00:46:45,845
the amount of money that they make, the profit is based directly

743
00:46:45,985 --> 00:46:49,345
on the amount of money that they, you know, they spend and are they getting

744
00:46:49,345 --> 00:46:52,865
this value for it. So if they can improve the performance of

745
00:46:52,865 --> 00:46:56,305
something, say 10 times, and a result of

746
00:46:56,305 --> 00:46:59,925
that is they get, 5 times as many customers,

747
00:47:00,410 --> 00:47:03,710
then that's not a bad investment. That'll just work. But

748
00:47:04,330 --> 00:47:07,550
if you're coming to me and I'm a business that existed

749
00:47:08,010 --> 00:47:11,390
before the Internet, if you're coming to me and saying,

750
00:47:11,930 --> 00:47:15,550
I want you to change to this completely different model,

751
00:47:17,345 --> 00:47:20,244
where, you know, and and the way it's presented

752
00:47:20,704 --> 00:47:24,385
often is you can save money. And that's

753
00:47:24,385 --> 00:47:28,065
true because if I start a new business today, I'm I

754
00:47:28,065 --> 00:47:31,770
couldn't even compete. I'm not gonna be able to stand up the

755
00:47:31,770 --> 00:47:35,530
servers, you know, take that time and buy that hardware and float

756
00:47:35,530 --> 00:47:39,310
that that inventory that I need to manage all that. Whereas,

757
00:47:39,370 --> 00:47:43,070
I can I can pay rent essentially every month

758
00:47:43,610 --> 00:47:47,388
on that service? Right? Right. I I always like to say I

759
00:47:47,388 --> 00:47:50,815
always like to say if I wanted to start a bookstore today,

760
00:47:52,235 --> 00:47:54,415
right, versus 1996,

761
00:47:55,835 --> 00:47:59,055
right, or 1995, depending on when you wanna say when they started.

762
00:48:00,039 --> 00:48:02,520
I mean, Barnes and Noble spent a ton of money. I don't have the exact

763
00:48:02,520 --> 00:48:05,720
number, but I can there's probably tens of 1,000,000 of dollars, probably closer to a

764
00:48:05,720 --> 00:48:08,940
$100,000,000 to just before they had their first customer.

765
00:48:09,720 --> 00:48:13,405
Right? Wow. And that but that was the heyday of the

766
00:48:13,405 --> 00:48:16,224
dotcom. Right? Because they were you know? But then

767
00:48:17,165 --> 00:48:20,204
but if you wanted to start a bookstore today, whether or not it's a good

768
00:48:20,204 --> 00:48:23,805
idea, let's let's just suspend our disbelief for a

769
00:48:23,805 --> 00:48:27,570
second. You can probably do it on a on an average credit

770
00:48:27,570 --> 00:48:30,390
card limit. Because

771
00:48:31,329 --> 00:48:35,170
because your IT is enabled. Right. And and you pay like I

772
00:48:35,170 --> 00:48:38,685
said, you pay fractions of what you would have to do in the brick and

773
00:48:38,685 --> 00:48:42,525
mortar. And most of the initial spend isn't gonna be your servers or hardware.

774
00:48:42,525 --> 00:48:46,205
It's gonna be in development and marketing. Right? Getting the word out

775
00:48:46,205 --> 00:48:49,980
because it's such a noisy market. It did the the market has radically changed.

776
00:48:50,460 --> 00:48:53,980
And I also think imagine go ahead. I'm sorry.

777
00:48:53,980 --> 00:48:57,500
What? Imagine? I was gonna say imagine that you've built

778
00:48:57,900 --> 00:49:01,660
Right. This infrastructure on premises already. You've got all of this done.

779
00:49:01,660 --> 00:49:05,485
It's a sunk cost. We can debate about how to feel about sunk cost.

780
00:49:05,545 --> 00:49:09,145
Right. But it's there. You spent the money and it's there. And

781
00:49:09,145 --> 00:49:12,985
you're not gonna get that 5 x income boost when you move to

782
00:49:12,985 --> 00:49:16,610
the cloud. In fact, in some cases, not

783
00:49:16,610 --> 00:49:20,450
all by by any stretch, but in enough cases, you

784
00:49:20,450 --> 00:49:24,290
move to the cloud and it costs you money. Because when

785
00:49:24,290 --> 00:49:28,050
you're getting the presentation about starting using the metrics of this

786
00:49:28,050 --> 00:49:31,765
new company being started today Right. You're, you know, you're told the

787
00:49:31,765 --> 00:49:35,225
truth. You're not being lied to at all in in any of this.

788
00:49:35,605 --> 00:49:38,825
But often, systems that were designed software

789
00:49:39,365 --> 00:49:43,090
and front end back end systems that were designed, you know, from

790
00:49:43,090 --> 00:49:46,310
the nineties through the mid early 2000.

791
00:49:47,170 --> 00:49:50,869
Those systems were architected in a whole different mindset

792
00:49:51,010 --> 00:49:54,849
of what's the prevalent mindset for today. And as a result of

793
00:49:54,849 --> 00:49:58,625
that Yeah. Yeah. As a result of that, one of the things missing from

794
00:49:58,625 --> 00:50:02,405
the spreadsheet calculation that you're gonna get the ROI

795
00:50:02,465 --> 00:50:06,245
from moving off your on premises servers to the cloud is

796
00:50:06,385 --> 00:50:09,825
that couple of $1,000,000 and about 18

797
00:50:09,825 --> 00:50:13,220
months, of the hit that you're gonna have to

798
00:50:13,220 --> 00:50:17,060
spend rearchitecting Yep. All of your systems so that they

799
00:50:17,060 --> 00:50:20,600
now fit today's paradigm. And frankly,

800
00:50:20,980 --> 00:50:24,765
if you are interested in doing that, you you could go do that

801
00:50:24,765 --> 00:50:27,565
at any you could have done that at any time in the last 10 years

802
00:50:27,565 --> 00:50:31,405
and made that shift. But people didn't do

803
00:50:31,405 --> 00:50:35,085
it because the business people didn't do it because the ROI was not dead. There

804
00:50:35,085 --> 00:50:38,700
was not enough return on that investment. If they wanted to, they

805
00:50:38,700 --> 00:50:42,540
would have spent that money then, but it wasn't gonna improve the bottom line.

806
00:50:42,540 --> 00:50:45,980
In fact, it was gonna hurt the bottom line. And so you see

807
00:50:45,980 --> 00:50:49,420
companies now make this move into the cloud and

808
00:50:49,420 --> 00:50:53,105
then, yeah. Yes. That is a

809
00:50:54,305 --> 00:50:57,984
that that's an astute question to ask. So for those who may be

810
00:50:57,984 --> 00:51:01,744
listening and not viewing this, it says is SQL SQL dev

811
00:51:01,744 --> 00:51:05,559
d b a says, I use Brent's and I'm assuming Brent Ozarks. Brent

812
00:51:05,559 --> 00:51:09,000
Ozars. Problem are you trying to solve by changing this for justifying

813
00:51:09,000 --> 00:51:12,440
upgrades? Brilliant. That's that is brilliant. And he's

814
00:51:12,440 --> 00:51:16,039
right. And the you know, but it's compelling to hear and

815
00:51:16,039 --> 00:51:19,484
read the case studies of of companies that, you

816
00:51:19,484 --> 00:51:22,925
know, were able to do use to to access

817
00:51:22,925 --> 00:51:26,605
$10,000,000 worth of hardware, like you said, on a credit card. And think,

818
00:51:26,605 --> 00:51:30,045
wow, what would that do for us? And the answer is sometimes, yeah, it'll

819
00:51:30,045 --> 00:51:33,869
revolutionize your business. You'll 10 x coming out of this. But other

820
00:51:33,869 --> 00:51:36,769
times, it's like, no. You'll point 8 x.

821
00:51:37,630 --> 00:51:41,390
You know, this isn't as compelling. So it's interesting because, like, I think

822
00:51:41,390 --> 00:51:44,109
there's a number of and I found the article. I'll pull it up. But but

823
00:51:44,109 --> 00:51:47,710
one of the examples, it was either Dropbox or Box. I forget which company it

824
00:51:47,710 --> 00:51:51,075
was. But but they had basically started off, I think, in

825
00:51:51,075 --> 00:51:54,675
AWS. Mhmm. And then they got to a certain size. They actually

826
00:51:54,675 --> 00:51:58,115
figured out it's cheaper for them to design their own servers that are optimized for

827
00:51:58,115 --> 00:52:01,795
mass storage Mhmm. Than doing it. So they started building their own hardware and their

828
00:52:01,795 --> 00:52:04,940
own stuff. But I could tell you, if they were a startup and they went

829
00:52:04,940 --> 00:52:08,060
to a VC saying, we want to start with this on prem, they would have

830
00:52:08,060 --> 00:52:11,900
been laughed out of the building. Yep. Today. Yes. Today they would

831
00:52:11,900 --> 00:52:15,520
have. They mean, you know, and it's

832
00:52:16,475 --> 00:52:20,235
it just shows that the the shifting economics of cloud versus on

833
00:52:20,235 --> 00:52:23,915
prem and and other types of things that I don't think people really have figured

834
00:52:23,915 --> 00:52:27,755
out yet. So this was a really interesting I'm gonna share this tab if

835
00:52:27,755 --> 00:52:31,460
I can show it on the screen. Sources. But that that

836
00:52:31,460 --> 00:52:34,760
use case is you can't, you know, having the compelling

837
00:52:34,900 --> 00:52:38,680
reason to migrate to the cloud, and you can do that upfront.

838
00:52:38,980 --> 00:52:42,500
It's harder. But exactly what you're showing there, you're

839
00:52:42,500 --> 00:52:46,175
sharing that that idea of leaving the cloud, that's

840
00:52:46,795 --> 00:52:50,175
growing. And it it's growing across the board. And I

841
00:52:51,355 --> 00:52:54,875
one of the, metrics for that that's

842
00:52:54,875 --> 00:52:58,555
directly related to what we're talking about here today with

843
00:52:58,555 --> 00:53:02,120
with data engineering, is that there

844
00:53:02,280 --> 00:53:06,040
there's been an increase in 2024 in the

845
00:53:06,040 --> 00:53:09,640
number of, people that reach out to me to talk

846
00:53:09,640 --> 00:53:12,220
about, SSIS help with their systems.

847
00:53:13,985 --> 00:53:17,745
And, I mean, I do consulting in, you know, ADF and fabric, and

848
00:53:17,745 --> 00:53:21,525
most of my consulting has been in ADF. When SSIS was involved,

849
00:53:21,825 --> 00:53:25,365
it was in lifting and shifting SSIS into an Azure SSIS,

850
00:53:26,109 --> 00:53:29,450
integration runtime. But all of a sudden, after

851
00:53:30,070 --> 00:53:33,590
2, 3, 4 years of that, that shifted this

852
00:53:33,590 --> 00:53:36,950
year. And people started reaching out to me with SSIS on

853
00:53:36,950 --> 00:53:40,790
premises consulting things, and I kept up with it. So I was

854
00:53:40,790 --> 00:53:44,204
able to do it. And but there's other

855
00:53:44,204 --> 00:53:47,805
evidence that I will not share. I probably I may be able to, but I'm

856
00:53:47,805 --> 00:53:51,165
just not going to. But it's even better evidence than my

857
00:53:51,165 --> 00:53:54,930
anecdotes about people more people reaching out to me. Right. That

858
00:53:55,410 --> 00:53:59,170
the amount of SSIS being executed in

859
00:53:59,170 --> 00:54:02,950
the world has increased, and it's a double digit percentage

860
00:54:03,089 --> 00:54:06,550
increase just in the past few months.

861
00:54:07,205 --> 00:54:10,965
And I I think I now this is where I start speculating, and I

862
00:54:10,965 --> 00:54:14,805
don't know the answer to that. But we have a our mutual

863
00:54:14,805 --> 00:54:18,645
friend that we, another mutual friend you and I connected with

864
00:54:18,645 --> 00:54:22,270
in November of 2025. Sorry. 2005. Like in the

865
00:54:22,270 --> 00:54:25,730
future. Recently recently worked for a

866
00:54:26,030 --> 00:54:29,710
year and a half, 2 years for this large agency that's

867
00:54:29,710 --> 00:54:33,170
not part of the government, but does money supply stuff.

868
00:54:33,385 --> 00:54:37,225
Oh, okay. I know. I know. Yeah. After getting his MBA from Sloan, you

869
00:54:37,225 --> 00:54:40,985
know, which no. Sloan. No. Sloan is

870
00:54:40,985 --> 00:54:43,885
important. The, you know MIT. School with MIT.

871
00:54:44,425 --> 00:54:48,045
Right. He's a graduate with that. Super smart.

872
00:54:48,420 --> 00:54:51,960
He shares with me when I'm telling him this story, I give him that stat,

873
00:54:52,260 --> 00:54:55,800
and he says, here's what's going on. Economically,

874
00:54:56,260 --> 00:54:59,940
money is more expensive today than it was. And

875
00:54:59,940 --> 00:55:03,745
so he said he said that as he's telling me this as a

876
00:55:03,745 --> 00:55:07,585
cautionary tale because he says it's gonna change. It's good. Money's gonna get

877
00:55:07,585 --> 00:55:11,185
cheap again, and people are gonna flock back to the cloud. That's his

878
00:55:11,185 --> 00:55:14,750
theory that it's all being driven by money, and I don't think he's wrong,

879
00:55:14,750 --> 00:55:18,510
especially I think that's one level. I think that's one lever. I think there's

880
00:55:18,510 --> 00:55:22,349
more than one lever. That is certainly a big one. But I you

881
00:55:22,349 --> 00:55:26,190
know, as someone who I you know, my previous role at Red

882
00:55:26,190 --> 00:55:29,755
Hat and my current role at Red Hat, I have to think globally. Right? And

883
00:55:29,755 --> 00:55:33,355
we don't again, not a commercial for Red Hat even though the

884
00:55:33,355 --> 00:55:36,955
fedora is there. You know, one of the things we do

885
00:55:36,955 --> 00:55:40,715
is we basically provide a data platform end to end that

886
00:55:40,715 --> 00:55:44,240
can run-in any cloud on

887
00:55:44,240 --> 00:55:48,079
prem or, you know, one of the hyperscale. Or hybrid. Yeah. Yeah. Or

888
00:55:48,079 --> 00:55:51,760
hybrid. Right? Where and there was one customer that I spoke with

889
00:55:51,760 --> 00:55:55,540
before I won opportunity to leave. They were, big government agency.

890
00:55:55,839 --> 00:55:59,535
And this big government agency, you know, they have

891
00:55:59,535 --> 00:56:03,375
their own data centers, even though there was a push to

892
00:56:03,375 --> 00:56:06,974
get rid of them all. But they also have because of way contract

893
00:56:07,055 --> 00:56:10,734
government contracts work in the US, they had, foot

894
00:56:10,815 --> 00:56:14,109
you know, money to spend in AWS, money to spend on Azure, and I think

895
00:56:14,109 --> 00:56:17,950
even money to spend on Google Cloud. So the one

896
00:56:17,950 --> 00:56:21,310
advantage that we had that the other ones couldn't is that the he called them

897
00:56:21,310 --> 00:56:24,910
the soft costs of training people how to do he'd do the same

898
00:56:24,910 --> 00:56:28,615
thing to do linear regression in SageMaker and

899
00:56:28,615 --> 00:56:31,915
push them out of production in SageMaker and Azure and in Google.

900
00:56:32,295 --> 00:56:36,055
Right? Yeah. And this was one tool. You learn it

901
00:56:36,055 --> 00:56:39,035
once. You administer it once. The same glass.

902
00:56:40,540 --> 00:56:43,819
It was the same thing. I think those environments are very real. Now those are

903
00:56:43,819 --> 00:56:47,260
probably limited to large customers or kind of the government

904
00:56:47,260 --> 00:56:50,960
agencies that have these kind of contracts and things like that.

905
00:56:51,339 --> 00:56:55,154
Yeah. But also Mhmm. You have a number of

906
00:56:55,694 --> 00:56:59,535
countries that it's just not a good look to move

907
00:56:59,535 --> 00:57:03,214
your data out of country. Right now, in the

908
00:57:03,214 --> 00:57:06,674
US and Canada, we don't have this issue because there's plenty of all the hyperscales

909
00:57:06,734 --> 00:57:10,440
have footprints in Canada and the US. But if you're in

910
00:57:10,440 --> 00:57:14,200
Latin America, which is this where this example comes from, right, there's only

911
00:57:14,520 --> 00:57:18,360
at least as Red Hat defines Latin America, includes Mexico, and basically all the

912
00:57:18,360 --> 00:57:22,205
way down to Antarctica. Mhmm. And only just,

913
00:57:22,205 --> 00:57:25,565
like, 30 some odd countries. Right? Someone's gonna write me hate mail saying that this

914
00:57:25,565 --> 00:57:29,325
is the exact number, but let's just keep the math simple. It's 30. We

915
00:57:29,325 --> 00:57:33,165
love those mail. We do. We love we love the mail. We learn things every

916
00:57:33,165 --> 00:57:36,970
time. Right. I talk about them personally when I get corrected at

917
00:57:36,970 --> 00:57:40,250
at the dinner table because I wanna share that with No. I mean, it's it's

918
00:57:40,250 --> 00:57:42,810
good. I'm not saying don't do it. I'm just trying to keep the math simple

919
00:57:42,810 --> 00:57:46,650
because it's Friday before, you know, basically, we're sure holidays. Sorry, Frank.

920
00:57:46,650 --> 00:57:49,870
I I derailed you. No. That's fine. Only 3 countries

921
00:57:50,490 --> 00:57:54,075
from Mexico down to Antarctica have hyperscaler presences.

922
00:57:54,135 --> 00:57:57,895
Now, the 4th one in the but out of 30.

923
00:57:57,895 --> 00:58:01,575
Right. So it's actually 10% or less realistically. I think it's like

924
00:58:01,575 --> 00:58:04,955
37 countries. I asked Wikipedia and stuff like that.

925
00:58:06,500 --> 00:58:10,340
So less than 10%. Right? Right. If you're in a

926
00:58:10,340 --> 00:58:13,880
country that doesn't have a footprint. If you're in that 90%.

927
00:58:16,260 --> 00:58:19,220
You have to ship it out as a country. You have to be okay with

928
00:58:19,220 --> 00:58:22,984
that or do roll your own solution on a thing. So there was a

929
00:58:22,984 --> 00:58:26,744
government we we won a big contract because they

930
00:58:26,744 --> 00:58:30,505
wanted to do advanced AI and they wanted

931
00:58:30,505 --> 00:58:33,704
to keep it in country. Right? Doesn't necessarily have to be on prem. Could just

932
00:58:33,704 --> 00:58:37,210
be, like, you know, an Equinox data center down the street or something like that.

933
00:58:37,210 --> 00:58:40,750
Right. But within their thing. And it was a government agency, so it wasn't

934
00:58:40,970 --> 00:58:44,650
computer science or even data science. It was political science that really kinda was the

935
00:58:44,650 --> 00:58:48,329
driver there. Right? Because if I'm in country x and I have to move

936
00:58:48,329 --> 00:58:51,994
my and I'm a government agency in country x, I have to move my data

937
00:58:51,994 --> 00:58:55,375
to a sovereign country y. Not a good look.

938
00:58:55,674 --> 00:58:57,055
Yeah. Right? And,

939
00:58:59,674 --> 00:59:03,430
you know, would it really matter? I don't think so. Like, in a but

940
00:59:03,430 --> 00:59:06,630
from a legal point of view, it kinda does. Like, where the data resides in

941
00:59:06,630 --> 00:59:10,390
the residency. And I think if you go to the Azure website

942
00:59:10,390 --> 00:59:14,230
now, they'll actually tell you where the data resides. And they actually interestingly

943
00:59:14,230 --> 00:59:17,695
enough, they get down to granular, at least on the US side to the

944
00:59:17,695 --> 00:59:21,535
state. Right? So, like, it'll say, you know, Virginia and

945
00:59:21,535 --> 00:59:25,295
stuff like that. We have a comment from, or not. I'm gonna hide my

946
00:59:25,295 --> 00:59:28,970
screen so I can look up the Azure map and kind of

947
00:59:28,970 --> 00:59:32,410
demonstrate that. And then I have to figure out where the sources are. There we

948
00:59:32,410 --> 00:59:36,250
go. So you wanna read the comment while I do that while I'm distracted? Sure.

949
00:59:36,250 --> 00:59:40,089
So so many comments, but there are clients who are

950
00:59:40,089 --> 00:59:43,914
considering the technology used in the software package, and they

951
00:59:43,914 --> 00:59:46,734
may escape when they see the old school,

952
00:59:47,355 --> 00:59:51,135
stuffs. I'm so maybe.

953
00:59:51,994 --> 00:59:54,875
And and that may be, you know, I hate to be that guy that says,

954
00:59:54,875 --> 00:59:58,680
oh, that use case is invalid. I don't think so. I'm not aware of it.

955
00:59:58,680 --> 01:00:01,579
That doesn't mean it's invalid. I'm not aware of a lot of things.

956
01:00:02,920 --> 01:00:06,760
But but maybe. And, you know, definitely

957
01:00:06,760 --> 01:00:10,540
have mixed emotions, about that. If you're buying

958
01:00:10,760 --> 01:00:14,595
because the the if you're a client and you're

959
01:00:14,595 --> 01:00:18,275
buying from, some company and you

960
01:00:18,275 --> 01:00:22,115
decide to go to a different company because of the

961
01:00:22,115 --> 01:00:25,875
technology stack that's being used behind there, I

962
01:00:25,875 --> 01:00:28,835
don't know. I think that says more about you as a client than it does

963
01:00:28,835 --> 01:00:32,650
about the company. If they're delivering the service and it's, you know,

964
01:00:33,030 --> 01:00:36,390
the the the rules of data engineering are you get accurate

965
01:00:36,390 --> 01:00:40,089
data as fast as possible, and those priorities

966
01:00:40,150 --> 01:00:43,875
are in that order. Yeah. I don't think the old school stuff second.

967
01:00:44,115 --> 01:00:47,315
The only risk of the old school stuff is it's still maintained or there's still

968
01:00:47,315 --> 01:00:50,835
security packages. That would I mean, if I were in that

969
01:00:50,835 --> 01:00:54,035
position, I would be like, oh, I mean I mean, if you're still using, say,

970
01:00:54,035 --> 01:00:57,630
Sybase 6. Right? You know, like Yeah.

971
01:00:57,869 --> 01:01:01,650
You know, got a little there's definitely a line there, and it's drawn

972
01:01:01,869 --> 01:01:05,230
based on it's drawn different places first. And some of the

973
01:01:05,230 --> 01:01:08,450
reasons, that it is drawn in different places is

974
01:01:09,150 --> 01:01:12,750
security is huge these days. I mean, that's gotta be your number one

975
01:01:12,750 --> 01:01:16,385
concern. And and, you know, it it

976
01:01:16,385 --> 01:01:20,085
goes from there. But if you're delivering the service securely,

977
01:01:20,865 --> 01:01:24,625
I'll just pick that one, then I would say

978
01:01:24,625 --> 01:01:28,200
that, you know, that if if you

979
01:01:28,200 --> 01:01:31,660
lose a client because you're not using the new shiny,

980
01:01:32,840 --> 01:01:35,560
I don't know what you can do about that. I'm trying to think I'm not

981
01:01:35,560 --> 01:01:39,400
gonna say the client's wrong for feeling that way. They probably have valid reasons

982
01:01:39,400 --> 01:01:42,985
for feeling that way. But if, you know, if they wanna if they

983
01:01:42,985 --> 01:01:46,825
wanna make that decision based on that. And I'm looking at Frank's graphic here

984
01:01:46,825 --> 01:01:49,865
of the is that the data centers? This is the one I was telling you

985
01:01:49,865 --> 01:01:52,825
about. Right? So this is this is just Azure, but I would say it's a

986
01:01:52,825 --> 01:01:56,529
pretty good proxy for the other hyperscalers. Right? Mhmm. I would

987
01:01:56,529 --> 01:02:00,210
say Azure at one point had more. I I haven't kept up.

988
01:02:00,210 --> 01:02:02,849
Well, that was one of our talking points when I worked at Microsoft. Right? We

989
01:02:02,849 --> 01:02:06,529
have more than to be honest. Right? But I would say if it's not exactly

990
01:02:06,529 --> 01:02:10,184
more, it's close enough. Right? So there's Mexico. You see the United

991
01:02:10,184 --> 01:02:14,025
States is pretty well covered. So is Canada. Right? So if you were a Canadian

992
01:02:14,025 --> 01:02:17,724
company, you had to keep it in Canada. You had an option. Right?

993
01:02:18,265 --> 01:02:21,885
Yeah. If you're an American company, you have pretty good choices.

994
01:02:22,280 --> 01:02:25,900
If you're Mexico, yep. But if you're in any of these countries in Latin America,

995
01:02:26,920 --> 01:02:30,140
down through South Wales. We only had okay. So now there's Chile.

996
01:02:30,520 --> 01:02:34,040
Okay. Gotcha. Right? So I'm sorry. Now there's

997
01:02:34,040 --> 01:02:34,540
4.

998
01:02:38,045 --> 01:02:41,805
So my math is going to get more complicated right away. Right. So, there's

999
01:02:41,805 --> 01:02:45,645
Brazil. Actually, no, there's still 3.

1000
01:02:45,645 --> 01:02:48,625
So there is no footprint for Azure in Argentina.

1001
01:02:49,869 --> 01:02:53,170
These little blue things you see, that is Colombia. Those are networking

1002
01:02:53,230 --> 01:02:56,830
pops. So basically, from a

1003
01:02:56,830 --> 01:03:00,670
networking point of view, if computer science were the only thing that would matter,

1004
01:03:00,670 --> 01:03:04,405
then that be that would be acceptable. But data residency is

1005
01:03:04,405 --> 01:03:07,545
the issue. So if I go here, the US East 1.

1006
01:03:08,325 --> 01:03:12,085
Right? It'll tell you that its

1007
01:03:12,085 --> 01:03:15,545
location is Virginia and it's stored at rest in the United States.

1008
01:03:16,165 --> 01:03:17,145
Like like here.

1009
01:03:21,370 --> 01:03:25,210
Is there any more details? There isn't. US.

1010
01:03:25,210 --> 01:03:28,990
Yeah. They used to. Yeah. They don't talk much about that,

1011
01:03:29,050 --> 01:03:32,625
about where they are. But one of the

1012
01:03:32,625 --> 01:03:36,225
US East Georgia. 1 of the Yep. The US

1013
01:03:36,225 --> 01:03:39,765
East 2 is down in Danville, isn't it?

1014
01:03:40,465 --> 01:03:44,145
They are or Mecklenburg. I forget which. Well, essentially, they chose a

1015
01:03:44,145 --> 01:03:46,325
picture of Richmond. Right.

1016
01:03:47,900 --> 01:03:50,860
I mean, these are, you know, with

1017
01:03:51,660 --> 01:03:55,360
we kinda touched briefly on on politics in a geopolitical

1018
01:03:56,620 --> 01:03:59,920
slash, sovereignty strategic way.

1019
01:04:00,444 --> 01:04:04,285
Right. These are huge. And I I know I'm not the one, thinking

1020
01:04:04,285 --> 01:04:08,125
about it. But Look at this. Yeah. There's one in Israel. Those satellites, Frank,

1021
01:04:08,125 --> 01:04:11,724
are getting it, by the way. Those those

1022
01:04:11,724 --> 01:04:15,325
satellites on the graphic, those things are moving at way faster than

1023
01:04:15,325 --> 01:04:19,140
normal satellites. Oh, yeah. Yeah. I mean, satellites are going to

1024
01:04:19,140 --> 01:04:22,260
change things, but, like, in terms of where the data sits at rest is really

1025
01:04:22,260 --> 01:04:25,860
where because ultimately, I think it really boils down to when are the

1026
01:04:25,860 --> 01:04:29,620
local police going to barge down a door with a with a court hopefully

1027
01:04:29,620 --> 01:04:33,194
with a court order and basically copy everything. Right. That's really what

1028
01:04:33,194 --> 01:04:37,035
matters. Right. That turns out that's really what ended up mattering. But

1029
01:04:37,035 --> 01:04:40,494
if you look at the Middle East, right, like 1, 2,

1030
01:04:41,675 --> 01:04:45,000
3 countries have it.

1031
01:04:45,140 --> 01:04:48,980
Right. And that entire region, you

1032
01:04:48,980 --> 01:04:52,500
know, obviously with geopolitical tensions being what they have been for a number of

1033
01:04:52,500 --> 01:04:56,260
years. Yeah. Moving your data center to any one of these countries may be an

1034
01:04:56,260 --> 01:04:59,825
issue for you for your organization or your

1035
01:04:59,825 --> 01:05:03,425
regulatory. Right? Europe is kind of the same thing, right, where, you

1036
01:05:03,425 --> 01:05:07,265
know, there's Switzerland, there's Italy. And I

1037
01:05:07,265 --> 01:05:10,885
know that there's different kind of things in terms of Germany.

1038
01:05:13,280 --> 01:05:17,120
It was actually I don't know if it still is now, but it

1039
01:05:17,120 --> 01:05:20,880
might have been a, I used to live in

1040
01:05:20,880 --> 01:05:24,605
Frankfurt, actually. Yeah. There was actually what they call a

1041
01:05:24,605 --> 01:05:28,445
sovereign cloud because there was concern that if it was a US company owning

1042
01:05:28,445 --> 01:05:32,285
a data center, that US courts would have jurisdiction there, which is a

1043
01:05:32,285 --> 01:05:36,045
brilliant move by a a past administration. I say

1044
01:05:36,045 --> 01:05:39,185
brilliant sarcastically in case you're didn't get pick up on that.

1045
01:05:39,849 --> 01:05:43,530
Where they thought that they could basically issue a a court order to

1046
01:05:43,530 --> 01:05:47,290
demand something from here in Ireland. And

1047
01:05:47,290 --> 01:05:50,730
Microsoft fought that because they realized, like, wait a minute. That would mess up our

1048
01:05:50,730 --> 01:05:54,415
entire that would cause a lot of problems. Yeah.

1049
01:05:54,415 --> 01:05:58,095
And, ultimately, they dropped the case before it was finally

1050
01:05:58,095 --> 01:06:01,535
decided. But in order that they could, thing, at one point,

1051
01:06:01,535 --> 01:06:05,000
anyway, this is actually owned by a German

1052
01:06:05,000 --> 01:06:08,839
company, managed by a German company, and it's leased to Microsoft to to

1053
01:06:08,839 --> 01:06:12,380
to have that concern. I think China also operates the same way.

1054
01:06:12,760 --> 01:06:16,520
Frank, I was commenting on the, satellites on the graphic

1055
01:06:16,520 --> 01:06:20,145
there. Oh, that they were moving around. Yeah. Yeah. They are moving very, very

1056
01:06:20,145 --> 01:06:23,825
fast. And it keeps they're moving with us.

1057
01:06:23,825 --> 01:06:27,664
But, I mean, keep in mind, though, like, keep in mind, though, that

1058
01:06:27,664 --> 01:06:31,130
we're just talking about data residency. There's other things that if you're building a real

1059
01:06:31,130 --> 01:06:34,970
solutions, other things to consider. Yeah. Right? Like And there's

1060
01:06:34,970 --> 01:06:38,490
a whole lot to that. And Yeah. Yeah. Oh, absolutely. And,

1061
01:06:38,490 --> 01:06:42,030
you know, part of the part of it is,

1062
01:06:42,490 --> 01:06:46,225
part of what what happens when you start kinda going back to

1063
01:06:46,225 --> 01:06:50,005
the data engineering, platforms and stuff that you use.

1064
01:06:50,225 --> 01:06:54,065
There are sound business reasons for not making a change,

1065
01:06:54,065 --> 01:06:57,585
and there are some unsound business reasons that will

1066
01:06:57,585 --> 01:07:01,240
confine you to not making a change. And I I think about this. I'll put

1067
01:07:01,240 --> 01:07:04,940
it in context of, of SQL Server.

1068
01:07:06,920 --> 01:07:10,680
Companies will come up and they this has happened, and I still have clients

1069
01:07:10,680 --> 01:07:14,059
running applications on old servers

1070
01:07:14,904 --> 01:07:18,365
because the company that so they

1071
01:07:18,585 --> 01:07:21,005
they serve, their clients include

1072
01:07:22,904 --> 01:07:26,444
enterprises that care an awful lot about

1073
01:07:26,505 --> 01:07:30,290
checking boxes and auditability and all of that stuff. Regulatory

1074
01:07:30,430 --> 01:07:34,110
type things, which is not bad. It's just the

1075
01:07:34,110 --> 01:07:37,790
way that it is. That's their their business demands something

1076
01:07:37,790 --> 01:07:41,630
like this. These companies were formed. They were stood up, and they've got SQL

1077
01:07:41,630 --> 01:07:45,414
Server 2,005 running or 2,008 or stuff that's been

1078
01:07:45,414 --> 01:07:49,095
out of the maintenance cycle at Microsoft for a long,

1079
01:07:49,095 --> 01:07:52,855
long time. And unknown it's also not a well known

1080
01:07:52,855 --> 01:07:56,375
fact that if you don't wanna upgrade to version x or

1081
01:07:56,375 --> 01:07:59,974
y, you can pay extra money, and Microsoft will maintain

1082
01:07:59,974 --> 01:08:03,720
and provide you patches. Right? There there's rumors that

1083
01:08:03,720 --> 01:08:07,340
there's at least as of a few years ago, there were still Windows 95 systems

1084
01:08:07,400 --> 01:08:11,020
that were, you know and that sounds absurd.

1085
01:08:11,800 --> 01:08:15,545
Yeah. But Well, you walked down the, entry to,

1086
01:08:15,785 --> 01:08:19,465
Delta flight, and there was one that's what was it? One is 97, I

1087
01:08:19,465 --> 01:08:23,225
think, sitting there. 98. Yeah. 98, was it? Yeah. One is 98.

1088
01:08:23,225 --> 01:08:27,064
Sorry. Yeah. One is 98, boxes sitting there for the longest time. They're still

1089
01:08:27,225 --> 01:08:30,850
I believe they're 1 to 7 now. Still. I I

1090
01:08:30,850 --> 01:08:34,390
saw XP. XP. You're right. It is XP. Yeah.

1091
01:08:34,689 --> 01:08:38,529
So, you know, you just I kinda noticed this, like, wow. I hadn't seen

1092
01:08:38,529 --> 01:08:41,590
that in a while. But it's it's not about

1093
01:08:42,335 --> 01:08:45,715
will the new technology run that

1094
01:08:46,094 --> 01:08:49,795
SQL Server 2,005 database. The answer is clearly yes.

1095
01:08:50,335 --> 01:08:53,950
Well, you can always virtualize something. Right? Like, that's something that, like I

1096
01:08:53,950 --> 01:08:57,790
mean, there's that compatibility levels help. Right. There's a number of things that

1097
01:08:57,790 --> 01:09:01,310
do it. But here's the kicker. If the application is

1098
01:09:01,310 --> 01:09:05,149
not certified to run on that

1099
01:09:05,149 --> 01:09:08,955
and you're for you can change it, and you be maybe you have changed

1100
01:09:08,955 --> 01:09:11,755
it and tested it and go, yeah. It works. We'll just move it to, you

1101
01:09:11,755 --> 01:09:15,435
know, SQL Server 2019 or 2022. We

1102
01:09:15,435 --> 01:09:19,115
know it works. But the people you're serving,

1103
01:09:19,115 --> 01:09:22,960
people who care way more about checking all the boxes and the regulations

1104
01:09:23,020 --> 01:09:26,620
being a 100% and auditable, they won't

1105
01:09:26,620 --> 01:09:30,460
allow you to. And it gets even more complex when that company that

1106
01:09:30,460 --> 01:09:33,899
originally sold you that software 20 years ago is no longer in

1107
01:09:33,899 --> 01:09:37,404
business. Right. So you have no path forward.

1108
01:09:38,024 --> 01:09:41,545
I mean, the only Or they get bought by another company that you

1109
01:09:41,545 --> 01:09:44,604
don't really like. Exactly. That's happened too.

1110
01:09:45,064 --> 01:09:48,904
Exactly. Then A lot of mainframe companies were brought up by I don't wanna

1111
01:09:48,904 --> 01:09:52,460
name names, but, like, were brought up, and they it really was, like,

1112
01:09:53,640 --> 01:09:57,080
ironically, because they what they do, they they knew they had them. And

1113
01:09:57,080 --> 01:10:00,680
ironically, a lot of mainframe migrations happened because of

1114
01:10:00,680 --> 01:10:04,375
that. Like, it was And so you've got, you know, you've got that

1115
01:10:04,375 --> 01:10:07,995
angle where people are sticking with older systems for whatever reason.

1116
01:10:08,135 --> 01:10:11,435
And it's, you know, it goes like I'm saying, my point is that this goes

1117
01:10:11,735 --> 01:10:15,195
beyond just the data engineering realm. There are

1118
01:10:15,335 --> 01:10:18,290
there are compelling reasons to use,

1119
01:10:19,070 --> 01:10:22,510
older software. It may not be anybody's, you

1120
01:10:22,510 --> 01:10:26,190
know, satisfactory answer, but it is, you know, those

1121
01:10:26,190 --> 01:10:29,950
reasons exist. And if you're the, you know, if you're a

1122
01:10:29,950 --> 01:10:33,345
developer who likes using the new shiny and learning the new

1123
01:10:33,345 --> 01:10:36,865
stuff, I'm one of those. That's why I'm teaching courses on fabric data

1124
01:10:36,865 --> 01:10:40,545
factory right now and watching as it kind of some

1125
01:10:40,545 --> 01:10:44,225
days it works and some days it doesn't. We've had that happen on a

1126
01:10:44,225 --> 01:10:47,739
number of deliveries this year, with that.

1127
01:10:47,739 --> 01:10:51,340
So if you read what I wrote about this and

1128
01:10:51,340 --> 01:10:55,100
you come away with Andy's against the new stuff, well, you're just

1129
01:10:55,100 --> 01:10:58,639
as wrong as wrong can be. That's not the case at all.

1130
01:10:58,860 --> 01:11:02,385
You know, it's an

1131
01:11:02,385 --> 01:11:06,065
interesting I mean, so back to the lecture at hand, what kind of kicked this

1132
01:11:06,065 --> 01:11:08,965
all off and inspired the stream was

1133
01:11:10,225 --> 01:11:13,950
this post where I think the short

1134
01:11:13,950 --> 01:11:17,470
answer is everybody's a little right. Everybody's a little

1135
01:11:17,470 --> 01:11:21,150
wrong. And as a consultant, you can appreciate these two

1136
01:11:21,150 --> 01:11:22,770
words. It depends.

1137
01:11:25,150 --> 01:11:28,945
Right? Because, like, you may want to upgrade to the new shiny. I know every

1138
01:11:28,945 --> 01:11:32,145
developer wants to do that. And I think the comment for some reason I can't

1139
01:11:32,145 --> 01:11:35,765
say is like basically hiring managers will put in a job description. All

1140
01:11:36,545 --> 01:11:39,765
there's also the other matter of job descriptions and, you know, job requirements

1141
01:11:39,985 --> 01:11:43,210
are. They're always a 100% accurate. Disconnecting

1142
01:11:43,670 --> 01:11:47,290
from consensus reality. Yes. I like to say.

1143
01:11:48,790 --> 01:11:52,390
But they may want someone with, like, say, ADF

1144
01:11:52,390 --> 01:11:56,085
and and and this, but then actually have them working on systems and SSIS

1145
01:11:56,145 --> 01:11:59,425
because the hiring manager knows that he or she may not have that open req

1146
01:11:59,425 --> 01:12:03,185
for a while and has a in the back of

1147
01:12:03,185 --> 01:12:06,324
the mind the idea of moving to that someday.

1148
01:12:06,880 --> 01:12:10,180
Sure. But realistically, for the next 2 years, you're gonna work in this site.

1149
01:12:10,560 --> 01:12:14,100
Yeah. I mean And there are still large

1150
01:12:14,160 --> 01:12:18,000
consulting companies out there that develop brand new

1151
01:12:18,000 --> 01:12:21,375
applications in SSIS, brand brand new data

1152
01:12:21,375 --> 01:12:25,215
engineering data warehouse. One worse than that or one better depending on your

1153
01:12:25,215 --> 01:12:28,975
point of view. A few years ago, I think it was on dotnetrocks. They

1154
01:12:28,975 --> 01:12:32,770
were talking about telemetry from Visual Studio. And this back

1155
01:12:32,770 --> 01:12:34,949
when I cared about Windows client development.

1156
01:12:36,370 --> 01:12:39,350
Basically, WPAF came out in 2006,

1157
01:12:39,969 --> 01:12:43,429
2007. Right? XAML.

1158
01:12:44,115 --> 01:12:47,635
No. Not XAML. Metro or modern

1159
01:12:47,635 --> 01:12:50,055
applications, UWP came out in 2011,

1160
01:12:50,995 --> 01:12:54,355
2012. Right? So there's been multiple

1161
01:12:54,355 --> 01:12:58,195
frameworks to write when and it's been a number since. But, again, don't really care

1162
01:12:58,195 --> 01:13:00,020
about those client development anymore.

1163
01:13:03,680 --> 01:13:07,360
Windows Forms is still the number one of

1164
01:13:07,360 --> 01:13:11,200
all those, like, ways to develop Windows applications that run on Windows

1165
01:13:11,200 --> 01:13:14,340
Mac. Windows Form is still accounts for 80% of development.

1166
01:13:15,375 --> 01:13:19,215
Wow. Something someone's going to like, please email me in

1167
01:13:19,215 --> 01:13:22,335
hate with hate mail, not hate mail, but like tell me the exact number. But

1168
01:13:22,335 --> 01:13:26,095
it was still. He's off. Well, and they kept saying in Visual

1169
01:13:26,095 --> 01:13:29,180
Studio 2005 came out. They said, this is it. This is the end of the

1170
01:13:29,180 --> 01:13:32,400
line for Windows Forms. We're not updating. We're not adding anything.

1171
01:13:33,100 --> 01:13:36,480
And the future is from now on. Right?

1172
01:13:37,260 --> 01:13:41,100
Until the future became something else. And then when

1173
01:13:41,100 --> 01:13:44,925
I last installed, I think it was Visual

1174
01:13:44,925 --> 01:13:46,625
Studio 2016, 2019.

1175
01:13:49,245 --> 01:13:53,005
There was improved. They added stuff to Windows Forms, which is kind of funny because

1176
01:13:53,005 --> 01:13:56,765
they said they never would. Yeah. But it just because they.

1177
01:13:56,765 --> 01:14:00,500
Demand. Right? Customer demand. Ultimately, customer demand

1178
01:14:00,500 --> 01:14:04,100
is what pays the bills. So you've got to be very mindful of that.

1179
01:14:04,100 --> 01:14:07,860
And, you know, if you if it's 2024 and you're writing

1180
01:14:07,860 --> 01:14:11,560
a Windows Forms app, I have questions.

1181
01:14:12,905 --> 01:14:16,205
You know, I'm not saying I disagree, but I have many questions.

1182
01:14:16,665 --> 01:14:20,125
Arguably, you could say the same thing for UWP or

1183
01:14:21,625 --> 01:14:25,245
WPF. Right? You know,

1184
01:14:25,750 --> 01:14:29,510
But, again, it really depends. Like, in the last time I had worked with

1185
01:14:29,510 --> 01:14:33,350
WPF professionally was it was for when I

1186
01:14:33,350 --> 01:14:37,190
was at a, between my stints

1187
01:14:37,190 --> 01:14:40,905
at Microsoft, and there was a, you know, there was

1188
01:14:40,905 --> 01:14:44,744
a customer who was a mortgage company, and they basically had their mortgage

1189
01:14:44,744 --> 01:14:47,324
intake form written in WPF. Right?

1190
01:14:48,985 --> 01:14:52,585
And they were having performance issues with it. And I I looked at the code,

1191
01:14:52,585 --> 01:14:56,090
and I was like, this is a good lesson, I think, is

1192
01:14:56,090 --> 01:14:59,850
that, you know, they loaded up, like, some 6, 700 controls

1193
01:14:59,850 --> 01:15:03,690
all at once. Right? Wow. Because there were a lot of

1194
01:15:03,690 --> 01:15:06,650
fields and but they were all collapsed and things like that. And I was like,

1195
01:15:06,650 --> 01:15:10,405
well, I'm looking at this, and I'm, like, testing it. And I'm like, oh,

1196
01:15:10,405 --> 01:15:13,445
dear god. This is gonna be a nightmare because it's 600 controls. You could do

1197
01:15:13,445 --> 01:15:16,825
lazy loading and things like that, but then Sure. There could be unintended

1198
01:15:16,885 --> 01:15:20,405
consequences there. And then then I happen to notice when I load the

1199
01:15:20,405 --> 01:15:23,980
app, the CPU spikes, but the GPU was

1200
01:15:23,980 --> 01:15:27,820
hardly touched, which the whole promise of WPF was

1201
01:15:27,820 --> 01:15:30,719
that it would offload as much of the rendering

1202
01:15:31,500 --> 01:15:35,260
Yeah. Over to the GPU as possible because it was basically built on

1203
01:15:35,260 --> 01:15:38,785
XNA, which was a gaming framework. But that, again,

1204
01:15:39,885 --> 01:15:43,585
different different sidetrack, and different lifetime ago.

1205
01:15:45,405 --> 01:15:49,165
So I'm like, what the heck is going on? So then I'm like, I looked

1206
01:15:49,165 --> 01:15:52,380
at the some of the machines. I'm like, they had the generic

1207
01:15:52,760 --> 01:15:56,440
GPU driver. So I'm like, just

1208
01:15:56,440 --> 01:16:00,120
for grins, let me see if I

1209
01:16:00,120 --> 01:16:02,940
can get the proper driver for this device.

1210
01:16:04,205 --> 01:16:07,725
All of a sudden the 30, 40 seconds it took to load that initial

1211
01:16:07,725 --> 01:16:09,985
screen went down to 5 seconds.

1212
01:16:11,085 --> 01:16:14,765
Wow. And that's a big jump. That was a

1213
01:16:14,765 --> 01:16:17,585
big jump. And that was like, and I said to the guy,

1214
01:16:18,605 --> 01:16:22,360
it's like, look, just installing this driver, you get

1215
01:16:22,360 --> 01:16:26,120
a, you know, massive increase in it. Right? Do you

1216
01:16:26,120 --> 01:16:29,240
really wanna architect it or are you trying to just you want this to be

1217
01:16:29,240 --> 01:16:32,920
faster? Like, what's considered acceptable? And he said, well, under 10 seconds would be

1218
01:16:32,920 --> 01:16:36,145
acceptable. And I was like, I could do you better. How about 5 and a

1219
01:16:36,145 --> 01:16:39,364
half? Right. And I showed him

1220
01:16:39,825 --> 01:16:43,585
and he goes, well, not everybody has a GPU in their device. And I was

1221
01:16:43,585 --> 01:16:47,344
like, well, like, you know, this GPU costs

1222
01:16:47,344 --> 01:16:50,700
about, I think, 189. For some reason, 189 is stuck

1223
01:16:50,700 --> 01:16:54,240
under $200. Yeah. And I'm like,

1224
01:16:54,460 --> 01:16:57,680
so you'd have to install it. You have to think about the labor of installing,

1225
01:16:58,380 --> 01:17:01,900
like, this cheap GPU. Right? And this, he goes, that's

1226
01:17:01,900 --> 01:17:05,685
fine. He goes, look. The cost of redeveloping and

1227
01:17:05,685 --> 01:17:09,525
rearchitecting and retesting this versus $200 per

1228
01:17:09,525 --> 01:17:13,145
box, plus whatever it takes for somebody to go in with a screwdriver

1229
01:17:13,205 --> 01:17:16,960
and update the drivers. Right. It was so like we

1230
01:17:16,960 --> 01:17:20,080
ended up not having to touch the code at all. Right? Yeah. It was just

1231
01:17:20,080 --> 01:17:23,920
a matter of a driver update, which nice. Because it was like a fixed

1232
01:17:23,920 --> 01:17:27,520
price kind of support contract. I was the hero because I

1233
01:17:27,520 --> 01:17:31,185
solved the problem with about 3 out 3 to

1234
01:17:31,185 --> 01:17:34,965
5 hours of work. Nice. And the customer was happy because

1235
01:17:35,105 --> 01:17:38,725
they didn't have to re architect anything. Everybody wins.

1236
01:17:38,945 --> 01:17:42,619
Everybody wins. I love it. Happens. But but it it's just it

1237
01:17:42,619 --> 01:17:46,060
just goes to show you, like, sometimes the most cost

1238
01:17:46,060 --> 01:17:49,739
effective approach isn't is to not

1239
01:17:49,739 --> 01:17:53,179
touch anything. You know, and it's although it was a

1240
01:17:53,179 --> 01:17:56,535
relatively small amount of money, and and,

1241
01:17:56,735 --> 01:17:58,835
manageable amount of time for the client,

1242
01:18:00,494 --> 01:18:04,175
sometimes in if you look at the, you know, kind of the the

1243
01:18:04,175 --> 01:18:06,994
big, performance tuning

1244
01:18:08,310 --> 01:18:12,150
picture. And I I ran into some of this, in the past where

1245
01:18:12,470 --> 01:18:15,770
Well, it's not always a happy ending. As as an engineer,

1246
01:18:16,630 --> 01:18:20,390
I want to, you know, to fix the the thing that I'm engineering. And if

1247
01:18:20,390 --> 01:18:24,045
it's software, then I wanna make the software perform better. And

1248
01:18:24,045 --> 01:18:27,805
if it's, you know so I went through one of those experiences and

1249
01:18:27,805 --> 01:18:31,325
then, number of circumstances where

1250
01:18:31,485 --> 01:18:35,325
but the end result was we we

1251
01:18:35,325 --> 01:18:38,465
threw money at it and

1252
01:18:39,450 --> 01:18:41,950
bought, better disks.

1253
01:18:43,610 --> 01:18:47,450
And I remember telling me about this. I remember you telling me this. Oh, it's

1254
01:18:47,450 --> 01:18:51,290
a specific long story, but, yeah, it's back from about 12 years ago. Yeah.

1255
01:18:51,290 --> 01:18:54,885
You were, like, just a long SSDs, and it got fast

1256
01:18:55,185 --> 01:18:58,705
enough. And so when I did the math on that was the

1257
01:18:58,705 --> 01:19:02,304
enterprise level project, there were dozens of people

1258
01:19:02,304 --> 01:19:05,985
tuning on a tiger team, and we did make it

1259
01:19:05,985 --> 01:19:09,640
go faster. And as a test, what I did was I

1260
01:19:09,640 --> 01:19:13,320
rolled the code back to where it was when we

1261
01:19:13,320 --> 01:19:17,000
started. I will say now though, tiger the term tiger

1262
01:19:17,000 --> 01:19:20,805
team after my Nobody knows what I mean. Sorry. I know what you mean. I

1263
01:19:20,805 --> 01:19:24,565
know what you mean. Of individuals, dozens of individuals focused on solving

1264
01:19:24,565 --> 01:19:28,105
a particular problem. It's it's like, Although in some cases

1265
01:19:28,565 --> 01:19:32,405
slurring. Focused on not solving a particular problem, but that's a story

1266
01:19:32,405 --> 01:19:36,219
for another day. Different different story. You're right. But, yeah, we

1267
01:19:36,219 --> 01:19:39,739
we did and and I rolled everything back and ran the old

1268
01:19:39,739 --> 01:19:43,340
code that we had optimized. You know, it ran super

1269
01:19:43,340 --> 01:19:46,640
fast on the SSDs, and it was running okay. It was acceptable,

1270
01:19:46,780 --> 01:19:50,595
barely, on, the spindles. But when we rolled it back, it

1271
01:19:50,595 --> 01:19:54,115
was, it was the same difference. We actually got a touch more

1272
01:19:54,115 --> 01:19:57,955
performance just off the SSDs. And when it you know, you do the math on

1273
01:19:57,955 --> 01:20:01,770
that. At that time, SSDs were rather new, and the amount

1274
01:20:01,770 --> 01:20:05,230
that I wanted was, not trivial.

1275
01:20:06,090 --> 01:20:09,530
I asked for it on a lark, and I was surprised when it showed

1276
01:20:09,530 --> 01:20:13,370
up. So she did work for Microsoft. Okay. I'm not

1277
01:20:13,370 --> 01:20:16,490
surprised. Kendra is, scary as much. She did consulting. We

1278
01:20:16,835 --> 01:20:20,195
Brent's name came up a a few times. She worked with Brent for a while.

1279
01:20:20,195 --> 01:20:23,315
She worked at Redgate. Not everyone knows what Redgate is, but they're kind of a

1280
01:20:23,315 --> 01:20:26,034
big deal in this in this situation. Yeah. Yeah. They're a big deal. So, like,

1281
01:20:26,034 --> 01:20:29,800
clearly, like, I I just find I'd love to get her, like, initial

1282
01:20:29,800 --> 01:20:32,940
opinion after factoring in all kind of all of this is that,

1283
01:20:33,640 --> 01:20:37,420
you know, ironically, Sync don't run after every shiny thing.

1284
01:20:38,040 --> 01:20:41,800
But the the thing that that guy that was originally learning was is

1285
01:20:41,800 --> 01:20:45,324
the new shiny thing, ironically. Like, so there's a lot of layers to this.

1286
01:20:45,324 --> 01:20:49,165
There's even if you take this kind of at face value of not knowing the

1287
01:20:49,165 --> 01:20:52,844
context, there's a lot of layers. But, like, beyond that, there's even more layers. Like,

1288
01:20:52,844 --> 01:20:56,685
it becomes this multidimensional problem. It's true. It's like an

1289
01:20:56,685 --> 01:21:00,510
ogre. Many layers. Many I was wondering when we're

1290
01:21:00,510 --> 01:21:03,550
gonna have a movie reference because it's been a while. There we go. Boom. Shrek's

1291
01:21:03,550 --> 01:21:07,230
a classic. Shrek is a classic. The

1292
01:21:07,230 --> 01:21:10,770
sequel is not so much, but that's a common case.

1293
01:21:11,385 --> 01:21:14,585
Sure. Very, very few Matrix movies looking at here. It was

1294
01:21:15,224 --> 01:21:19,065
you know, I I saw the, activity on this and it's best because

1295
01:21:19,065 --> 01:21:22,184
since I posted this, I can see how many people looked at that. The the

1296
01:21:22,184 --> 01:21:25,530
link I sent you for the other one is kinda the one that started it.

1297
01:21:25,690 --> 01:21:28,969
I don't know if you have that link in chat. I think If you wanted

1298
01:21:28,969 --> 01:21:32,650
to click on that. I have That was that was a

1299
01:21:32,650 --> 01:21:36,409
few days, maybe a week before this. That was the first one I

1300
01:21:36,409 --> 01:21:40,245
commented. It was similar sentiment and I shared, you know, again, I

1301
01:21:40,245 --> 01:21:44,045
joined the conversation and then, I can't believe is

1302
01:21:44,045 --> 01:21:47,844
SSIS dead? Yeah. That's the one. So that

1303
01:21:47,844 --> 01:21:51,525
particular one has gotten, like, probably 12

1304
01:21:51,525 --> 01:21:55,260
to 15 times as many views. It's happened to You know, you mentioned And

1305
01:21:55,260 --> 01:21:58,460
it was the first time I had piped up about anything like this. I'm I

1306
01:21:58,460 --> 01:22:01,980
commented on the original one. LinkedIn will pop this up if you write a

1307
01:22:01,980 --> 01:22:05,200
lengthy comment like like that one. And

1308
01:22:06,060 --> 01:22:09,580
I said, that window exist. Looks like Windows XP

1309
01:22:09,580 --> 01:22:13,215
era. Well, that's SSIS. That is

1310
01:22:13,215 --> 01:22:16,895
a funny. Like, it's just funny. Yeah. But, And I

1311
01:22:16,895 --> 01:22:20,735
commented when this LinkedIn popped up and said, hey. Do you wanna repost this

1312
01:22:20,735 --> 01:22:24,409
since you wrote this comment? And I said, sure. And it just pastes the

1313
01:22:24,409 --> 01:22:27,370
comment up at the top of the repost. But I can see, like, the number

1314
01:22:27,370 --> 01:22:30,750
of people that and that one drew a lot more,

1315
01:22:31,610 --> 01:22:35,449
a lot more comments. And like I said, 12 to 15 times

1316
01:22:35,449 --> 01:22:39,255
the views. And, and and I communicated

1317
01:22:39,255 --> 01:22:42,935
with the original author just to touch about that at all. It it

1318
01:22:42,935 --> 01:22:46,615
went nowhere near as I'm

1319
01:22:46,615 --> 01:22:50,455
trying to think of the right word. No nowhere. It it

1320
01:22:50,455 --> 01:22:54,199
didn't get nearly as heated. I'll say it that way. And it

1321
01:22:54,199 --> 01:22:57,980
may just be that, you know, that I'm saying heated, but there was

1322
01:22:58,040 --> 01:23:01,640
there was one individual in particular who just very passionate about the tools that they

1323
01:23:01,640 --> 01:23:04,940
that they used for. I I I wanted to ask,

1324
01:23:05,320 --> 01:23:08,965
that individual, you know, how

1325
01:23:08,965 --> 01:23:12,505
many SSIS packages they developed and put into production.

1326
01:23:13,685 --> 01:23:17,445
Because I I I I would again, I think I know the answer

1327
01:23:17,445 --> 01:23:20,670
to that. This in the this individual conversations like that.

1328
01:23:21,870 --> 01:23:25,550
Say that again? Which individual? Not in that one.

1329
01:23:25,550 --> 01:23:29,310
Oh. Not it's in the other range. The one we've been looking at. This angle

1330
01:23:29,310 --> 01:23:33,010
here, though. Right? Nothing against Kiwi

1331
01:23:33,625 --> 01:23:37,405
ETL pools. What do you Yeah. Appreciate so this is this

1332
01:23:38,105 --> 01:23:41,865
is an argument of GUI versus Yeah. Straight

1333
01:23:41,865 --> 01:23:45,305
code. I think that's also an interesting concept. Low code, no

1334
01:23:45,305 --> 01:23:49,130
code versus you know? I like them both, but,

1335
01:23:49,270 --> 01:23:52,489
you know I do too. It's for me, it's which solution

1336
01:23:52,710 --> 01:23:56,469
matches best and as a consultant who goes to work with other companies a

1337
01:23:56,469 --> 01:24:00,230
lot Right. A big factor for me is What they have. You know, what

1338
01:24:00,230 --> 01:24:03,855
happens when I'm gone? Can you support it? What what are you most

1339
01:24:03,855 --> 01:24:07,395
familiar with? Even if I'm working in a tech a technology

1340
01:24:07,615 --> 01:24:11,375
technician, a technology like SSIS, there's really a

1341
01:24:11,375 --> 01:24:14,790
couple of ways you can, float it. It. That's good, John.

1342
01:24:14,990 --> 01:24:18,830
2 was good. I'm referring to a 3, and I think there was a 4th

1343
01:24:18,830 --> 01:24:22,270
one. Oh, really? I didn't even know there was a 4th one. So there you

1344
01:24:22,270 --> 01:24:25,945
go. But in SSIS, you can take a more And

1345
01:24:25,945 --> 01:24:28,905
the matrix 2 was good as well. Matrix 2 was good as well. I was

1346
01:24:29,225 --> 01:24:32,525
That's true. Yeah. You you could take, like, a more

1347
01:24:32,585 --> 01:24:36,345
DBA approach, a more T SQL driven approach, or you

1348
01:24:36,345 --> 01:24:39,890
can take a more dot net driven approach because you have both

1349
01:24:40,110 --> 01:24:43,650
execute SQL tasks and SQL sources and destinations and transformations,

1350
01:24:44,030 --> 01:24:47,170
and you have a script task and a script component.

1351
01:24:47,870 --> 01:24:51,150
And, you know, which way do you go? Which is the right way to go?

1352
01:24:51,150 --> 01:24:54,975
Well, it depends. If you've got a bunch

1353
01:24:54,975 --> 01:24:57,475
of dot net developers being tasked with maintaining

1354
01:24:58,415 --> 01:25:02,175
the, ETL after I leave, no way. 5.

1355
01:25:02,175 --> 01:25:05,920
Good god. Shrek 5 or matrix 5.

1356
01:25:06,219 --> 01:25:09,739
Either one sounds like a terrible idea. You know,

1357
01:25:09,739 --> 01:25:13,420
whoever's gonna be left behind support net, you gotta make sure that Right. You've done

1358
01:25:13,420 --> 01:25:17,040
a a fair enough job of representing their preferences,

1359
01:25:18,219 --> 01:25:21,744
on that. And, you know, I I know

1360
01:25:22,284 --> 01:25:26,045
I know consultants who come in and say, you know, we gotta do it

1361
01:25:26,045 --> 01:25:29,405
this way and you're bad and wrong and you're, you know, you're gonna go out

1362
01:25:29,405 --> 01:25:33,210
of business in 18 months if you don't listen to me. Right. That

1363
01:25:33,210 --> 01:25:36,850
if I also think too, like, when I first read it I'm sorry. I don't

1364
01:25:36,850 --> 01:25:40,690
say hire somebody else. Right. I mean, I when I first read

1365
01:25:40,690 --> 01:25:42,869
it, I first read it as should you learn SQL?

1366
01:25:44,865 --> 01:25:48,705
Okay. Right? And that's a big debate in the data science community is should should

1367
01:25:48,705 --> 01:25:52,545
data scientists learn SQL because, you know, Python can do everything. Just your no. No

1368
01:25:52,545 --> 01:25:56,385
knock on my Python can do everything. Should you use Python for everyone everything, I

1369
01:25:56,385 --> 01:26:00,100
think, is another question. Right? And I think the answer is no. Yeah. I think

1370
01:26:00,100 --> 01:26:03,780
if you're a data scientist if you're a

1371
01:26:03,780 --> 01:26:07,540
data scientist or even an AI kinda engineer, whatever the word is

1372
01:26:07,540 --> 01:26:11,219
this week, you should still learn SQL

1373
01:26:11,219 --> 01:26:14,925
because this one, it's way less complicated than anything else you're doing. And

1374
01:26:14,925 --> 01:26:18,525
2, it's kind of the the the the lingua

1375
01:26:18,525 --> 01:26:22,205
franca of anything data or data interaction. Right? And

1376
01:26:22,205 --> 01:26:25,965
again, Frank, the context comes into play here. Right. There's the there's the

1377
01:26:25,965 --> 01:26:29,760
mechanical tool. It's describing a software, you

1378
01:26:29,760 --> 01:26:33,360
know, a language is mechanical is one one way of looking at it. But Right.

1379
01:26:33,520 --> 01:26:37,280
There's also the problem you're trying to solve. And if if you're gonna do

1380
01:26:37,280 --> 01:26:40,420
data exploration or managing or whatever you wanna call it,

1381
01:26:40,640 --> 01:26:44,315
then I don't really care which mechanism you use. But don't

1382
01:26:44,315 --> 01:26:47,835
tell me one of these mechanisms is better than the other. It

1383
01:26:48,395 --> 01:26:51,755
Without a qualifier. For this particular task.

1384
01:26:51,755 --> 01:26:55,590
Yeah. So it could be that, you know, one

1385
01:26:55,590 --> 01:26:59,190
of them is is good at this one particular thing, and I I would argue

1386
01:26:59,190 --> 01:27:03,030
this. I did in my newsletter. Oh, strike 5. Okay. In my

1387
01:27:03,030 --> 01:27:06,870
newsletter, I argued, every single tool has something

1388
01:27:06,870 --> 01:27:10,665
that they're stronger and there's there's some

1389
01:27:10,665 --> 01:27:14,425
feature or some aspect of it that's better than all of

1390
01:27:14,425 --> 01:27:18,045
the others. And they also have some weakness

1391
01:27:18,185 --> 01:27:21,829
that's worse than all of the others. It's true. And and so

1392
01:27:21,829 --> 01:27:24,969
you gotta, you know, you gotta strike that balance. It's gonna depend

1393
01:27:25,829 --> 01:27:29,449
on your use case, the parameters, the things that are important to you.

1394
01:27:29,510 --> 01:27:32,329
Sometimes it's cash on hand.

1395
01:27:32,949 --> 01:27:36,725
Sometimes it's the servers, hardware that you're forced to work

1396
01:27:36,725 --> 01:27:40,165
with because you're owned by somebody and they they're not upgrading. They're not

1397
01:27:40,165 --> 01:27:43,605
giving you that staffing concerns. The people, their

1398
01:27:43,605 --> 01:27:47,365
experience, their languages, that their Well, how easy is it to find someone that

1399
01:27:47,365 --> 01:27:50,870
necessarily knows SQL versus knows Python?

1400
01:27:51,730 --> 01:27:54,690
Or, the bill rate for someone that knows SQL is probably gonna be different than

1401
01:27:54,690 --> 01:27:57,670
the person who knows Pandas. I think so.

1402
01:27:58,850 --> 01:28:02,390
I I think there are differences there, but, again, it's gonna depend on the company.

1403
01:28:02,610 --> 01:28:06,155
That's true. You're worth our job market. Right. You're worth

1404
01:28:06,155 --> 01:28:09,994
more to, you know, this company as a SQL developer than, you

1405
01:28:09,994 --> 01:28:13,675
know, maybe than a Pandas developer is to that that company. That's a

1406
01:28:13,675 --> 01:28:17,114
possibility. Go go where you get paid the

1407
01:28:17,114 --> 01:28:20,949
most, but realize especially if you're new, and I think that

1408
01:28:20,949 --> 01:28:24,710
person that in Kendra's original, post that

1409
01:28:24,710 --> 01:28:28,170
she was, she was jamming on, that

1410
01:28:28,550 --> 01:28:32,070
that was a different scenario. You're right. There's a number of squirrelly things about that

1411
01:28:32,070 --> 01:28:35,805
original post, but if you're

1412
01:28:35,805 --> 01:28:39,565
a young developer and just getting started, you're in college. I've got a

1413
01:28:39,565 --> 01:28:43,165
daughter studying computer science in college right now. And my

1414
01:28:43,165 --> 01:28:46,525
recommendation to her is first, learn everything that you

1415
01:28:46,525 --> 01:28:50,360
can. Do as much as absolutely as you can.

1416
01:28:50,420 --> 01:28:52,520
Pick up that knowledge. But,

1417
01:28:54,020 --> 01:28:57,860
be aware that there's more to it than just the, you

1418
01:28:57,860 --> 01:29:01,435
know, the the the brain exercise you get and the

1419
01:29:01,435 --> 01:29:05,114
thrill you get from seeing the code execute. Keep that thrill. Keep that spark

1420
01:29:05,114 --> 01:29:08,635
alive. Right. Right. Right. Right. But, you know, real

1421
01:29:08,715 --> 01:29:12,074
realize there's often more to it. And some of those factors you have no control

1422
01:29:12,074 --> 01:29:15,775
over, and the person you're working for has no control over.

1423
01:29:16,080 --> 01:29:19,760
And, Ed, you know, there's just a number of things totally external to the experience

1424
01:29:19,760 --> 01:29:23,380
of writing code that often

1425
01:29:23,440 --> 01:29:27,219
impact the experience of writing code. That's very true.

1426
01:29:27,360 --> 01:29:31,165
That's very true. K. Actually, going on for,

1427
01:29:31,165 --> 01:29:34,625
like, 90 minutes. So Wow.

1428
01:29:34,845 --> 01:29:37,965
Goodness. It doesn't seem like it. I've had this much coffee, and I still don't

1429
01:29:37,965 --> 01:29:41,725
have to go to the bathroom. That's amazing. Christmas miracle.

1430
01:29:41,805 --> 01:29:45,310
Of you. It's a festive miracle or a Christmas miracle. I don't

1431
01:29:45,310 --> 01:29:49,150
know. Awesome. But, so

1432
01:29:49,150 --> 01:29:52,910
this is this has been great. I think we kinda got to the bottom

1433
01:29:52,910 --> 01:29:56,690
of this is that basically, it's a nuanced conversation.

1434
01:29:56,910 --> 01:30:00,415
There's no simple answer. I have many

1435
01:30:00,415 --> 01:30:04,015
questions though. Why if you're learning LLMs and now, why are you

1436
01:30:04,015 --> 01:30:07,775
calling yourself a data engineer? But Yeah. That's a different thing.

1437
01:30:07,775 --> 01:30:11,075
Could just be semantics at that point. Could be.

1438
01:30:13,450 --> 01:30:15,870
But thanks, John. Thanks, Merdad.

1439
01:30:17,050 --> 01:30:20,890
Thanks, Hector and SQL Dev

1440
01:30:20,890 --> 01:30:24,650
DBA. I'm sure that's not the

1441
01:30:24,650 --> 01:30:27,485
name on on the driver's license, but

1442
01:30:29,065 --> 01:30:32,824
you never know. You never know. I wanna get a license plate holder

1443
01:30:32,824 --> 01:30:36,505
that has, like a, like, drop table.

1444
01:30:36,505 --> 01:30:40,340
Like, so that way when they

1445
01:30:40,340 --> 01:30:44,100
take a picture, they do the OCR. Boom. I've seen

1446
01:30:44,100 --> 01:30:47,940
the bumper stickers with longer Right. Right. Right. Right. Right. Secret

1447
01:30:47,940 --> 01:30:51,625
hacks on them, you know, secret injection attacks. No. Hey, man. Thanks

1448
01:30:51,625 --> 01:30:55,304
for tuning in. And, he legally changed his name. That's

1449
01:30:55,304 --> 01:30:59,065
cool. Apparently, there's a number of people

1450
01:30:59,065 --> 01:31:02,824
who have, like, a last name, and their

1451
01:31:02,824 --> 01:31:06,670
last name is Null. Oh, wow. And I

1452
01:31:06,670 --> 01:31:10,510
was looking up on YouTube. Like, there's, like, a number of, like, problems that people

1453
01:31:10,510 --> 01:31:14,190
have. And my first thought was, wait. Wouldn't that only be the case if

1454
01:31:14,190 --> 01:31:17,745
they encased it in quotes or single quotes? Like, wouldn't it?

1455
01:31:17,985 --> 01:31:21,825
Apparently, no. Some of these systems Depends. I mean, you

1456
01:31:21,825 --> 01:31:25,125
wanna talk old systems. I'm sure DMVs have some pretty

1457
01:31:25,665 --> 01:31:29,445
ancient technologies that are still running.

1458
01:31:29,505 --> 01:31:33,170
I mean, I can only imagine. Robert

1459
01:31:33,230 --> 01:31:36,750
Tables. Yes. Little Bobby Tables. Little Bobby Tables. That's the

1460
01:31:36,750 --> 01:31:40,530
one. I tried to name my kids something like that. But

1461
01:31:41,390 --> 01:31:45,145
You got overruled if I remember correctly. I did in so many ways. I'm

1462
01:31:45,145 --> 01:31:48,824
glad I didn't go with the, initials of, x

1463
01:31:48,824 --> 01:31:52,344
a m l. Yeah. That

1464
01:31:52,344 --> 01:31:55,705
that You could have. Xavier Anthony

1465
01:31:55,705 --> 01:31:59,130
Marcus was, on the table. See and

1466
01:31:59,130 --> 01:32:02,969
Lavinia? That would just Yep. That would just flow. That would be

1467
01:32:02,969 --> 01:32:06,730
your role. Though, since XAML kinda died,

1468
01:32:06,730 --> 01:32:10,190
it's probably a good thing. That's true.

1469
01:32:10,575 --> 01:32:14,335
So XML is even not as in vogue as it once

1470
01:32:14,335 --> 01:32:17,775
was. Well, you know our mutual friend, mister Kevin

1471
01:32:17,775 --> 01:32:21,375
Hazard, aka the Duke of Hazard. The

1472
01:32:21,375 --> 01:32:24,675
Duke. He says he says JSON is just hipster

1473
01:32:24,735 --> 01:32:28,510
XML. He's right, though. And there's another format called

1474
01:32:28,510 --> 01:32:32,130
JSONL. What? Yeah. JSONLines.

1475
01:32:32,510 --> 01:32:36,030
Never even heard of that. It I it's only the

1476
01:32:36,270 --> 01:32:39,870
I didn't hear about it until the product I work on, Rel AI, actually

1477
01:32:39,870 --> 01:32:43,555
uses it to store our data. And, basically, it's a different

1478
01:32:43,555 --> 01:32:47,155
between JSON and JSONL? Basically, it's a long line

1479
01:32:47,155 --> 01:32:50,915
where each line is a record of or a chunk of

1480
01:32:50,915 --> 01:32:54,650
JSON. That's how I interpret it. I'm sure I'll get

1481
01:32:54,650 --> 01:32:58,489
corrected, but please correct me on that one. Interesting. Yeah. I'm like, Jason

1482
01:32:58,489 --> 01:33:00,670
now? Like, what the heck?

1483
01:33:02,010 --> 01:33:05,655
Jason is an acceptable one. That's true. Jason

1484
01:33:05,655 --> 01:33:09,255
would've been a good name. But I didn't want him to get

1485
01:33:09,255 --> 01:33:11,675
teased on Friday 13th for the rest of his life.

1486
01:33:13,094 --> 01:33:16,855
So so,

1487
01:33:17,175 --> 01:33:20,215
thanks everyone for tuning in, man. This was awesome. We should do it more often.

1488
01:33:20,215 --> 01:33:23,230
Thumbs up on, or, you know, be sure to like, share, subscribe. I gotta do

1489
01:33:23,230 --> 01:33:26,670
this to all the things. Like, share, and subscribe. I can show off,

1490
01:33:26,989 --> 01:33:27,969
this little graphic.

1491
01:33:34,255 --> 01:33:38,015
You you know, Frank, we could do this, we could do this

1492
01:33:38,015 --> 01:33:41,775
a lot more often if there anytime I stir up some trouble, we do a

1493
01:33:41,775 --> 01:33:45,615
a live stream Oh, god. We do it every day. I Yeah. That's what I'm

1494
01:33:45,615 --> 01:33:49,239
saying. You know? No. I think it's interesting. I think it's good because, like, you

1495
01:33:49,239 --> 01:33:52,699
know, it the controversy

1496
01:33:55,719 --> 01:33:58,840
I think people are starting I just look last time I looked at the thread

1497
01:33:58,840 --> 01:34:02,455
thoroughly, people were talking past each other. Yeah. Which I

1498
01:34:02,455 --> 01:34:05,355
guess defines all of Reddit, mostly Internet.

1499
01:34:07,495 --> 01:34:11,335
You know, it is what it is. Yeah. But

1500
01:34:11,335 --> 01:34:14,055
cool, man. I gotta actually, now that I mentioned it, I do have to go

1501
01:34:14,055 --> 01:34:17,850
to the restroom. See. You did it to yourself. I did do it to myself.

1502
01:34:17,850 --> 01:34:20,909
And thank you, Miranda, for turning in. And

1503
01:34:22,570 --> 01:34:24,989
I will play the outro graphic. Excellent.