In this special holiday themed episode, we're diving into a topic
that's as classic as Christmas Carols, but just as divisive as fruitcake.
And that topic is the future of SQL Server Integration
Services, SSIS. But wait, there's a
twist. This episode was recorded live, so if you notice
a different vibe, some festive banter, and maybe even a change in
our usual musical interludes, that's why. Think of it as
the holiday party version of our usual data driven discussions.
Together, we'll explore why SSIS, despite its vintage
status, remains a cornerstone of data engineering and why
dismissing it might just be a data driven mistake. So grab your
cocoa, settle in by the fire or your nearest CPU,
and let's get festive with some data talk.
Well, hello, and welcome to franksworld.comstream.
And, with me today is Andy, and I'm
looking for the lower third that has us both. There we
go. Frank and Andy Frank Lavinia and Andy Leonard, host of Data
Driven, which I might turn this into a podcast. I might take the
audio and and turn it into a podcast. What do you think about that? That'd
be good kind of festive stream and also kind
of up to date on things. And it gives me some more time that to
put together another episode that I had a really great
conversation with a guy who does red teaming for LLMs.
Nice. So which I think is a growth industry
and certainly a wise career move.
Speaking of career moves. Good thing.
Oh, we have a first comment. SQL dev d b a.
Hey, SQL dev. Awesome. So
this is this may be the first time we've done this. This feature's been around
for a while. No. We did it once or twice before. Did we do it
on recent? Like, months. Yeah. That we've done. So we're sharing
our so Frank's audience, people that are connected to Frank,
they're seeing this. People connected to me are seeing this. It's like it'll
because it told me Frank started this, and, then he sent me
the link. And as I joined in, it it said, hey. You can
share this with with your on your channels as well. So I
was like, oh, yeah. Click that. Oh, you know what it is? We did it
the other way. You were the main, and then I shared it on my channels.
That's what happened. That's what happened. Yeah. Yeah.
Well, it's cool, though. If you've never met me before hello?
That's Frank. Frank digs data on the socials and,
franksworld.com, datadriven.tv, which hopefully you know about that,
and impactquantum.com. So that's me.
And, yeah. So back to the segue.
Yeah. I was talking about how security and AI is a
good career move. And we were talking about, speaking of
career moves, 'tis the season for SSIS is the title of the
stream. And this kind of goes,
I'm sorry. Come on, man. It's fun. Right? It is.
It's awesome. So so and I had kind of done,
2 livestreams on this already, but one of them for, like, 10 minutes, I didn't
catch the fact that I had no audio. And then yesterday, I did one for
2 minutes, so I didn't catch the fact that I didn't do the audio. So
I figured I'd bring the troublemaker himself onto here. Although, strictly
speaking, you're not the original troublemaker on this. Well,
I participated in it. I'll I'll own my my part of the
trouble. You'll own your part of the trouble. So so I definitely will. Yeah.
What's the background here? Well and and I'll
I'll do a plug for, for andylehner.blog.
And if you go there, you can sign up for my newsletter over on the
right side. It's kinda hard to read because the widget is a little
narrower than it needs to be. But if you if you do that or if
you just look up engineer of data, I think it's
engineer of data dot substack.com.
But I I put a newsletter out today kinda talking about it. Yeah.
There's the site. Thanks, Frank. No problem. And, you see the subscribe
to my newsletter down there on the right, and there's a box on
the left where you type your email address and then on the right, you click
it's free. And it'll it should take you, right over to
Subsec, which by the way, I started using this year. And so far,
I'm pretty impressed. It's it's been a a
really interesting, experience for me. So the
trouble here here's, here's where the
trouble happened. I I have been, reading. I
caught a couple of articles just every now here and then, mostly
on LinkedIn, where people
would express an opinion about, you know,
SSIS stinks. I don't like it. It's old. It's was
so much trouble. And, you know, and they would just
kind of kind of poo poo share their their negative thoughts about
Azure sorry. About SSIS. And
I've, of course, I've worked in SSIS since
really before it came out, I got to work on that Rocks book project
with Brian Knight and I remember that book. Yeah. Yeah. 10
of us. Yes. Back when Rocks would put your picture on the cover of the
book. And have a copy around here somewhere. Yeah. That
yeah. Thank you, Frank. You know, it just but it's
so I got yes. I got kind of a boost out of my career,
and I did an awful lot in SSIS for a long time. And
every now and then, I still do. I used to
deliver training, as part of solid,
solid quality learning is what it was called when I joined it. Solid
queue. After that, I worked with them for a few years and I delivered
training developed by Eric Veerman and
also did consulting gigs. And I learned a lot,
about both data engineering and SSIS while I was
doing both those things. When I left solid q, I
think I put about a year or 2 between me and,
you know, and the business. Actually, it was about two and a half years because
I went to work for Unisys then as a ETL architect. I remember
that. You're up in Reston quite a bit because that's where it was. Oh, yeah.
Yeah. Frank. Now an apartment complex now, that building. Oh, is
it? Okay. I think so. Yeah. Okay. So Frank and I
have been friends since the before times, even before SSIS came
out. And, Well, no. I think you had just written the book at the
time. I I'm trying to remember. So Just moved to Richmond just when I
met. It was November of 2005. December 2005.
Yeah. Yeah. November 2005 is when we met. And,
another mutual friend that I won't name, but we're all still friends now.
And the book actually was published in
January, I think, of 2006.
Yes. That's right. So it wasn't it wasn't quite
ready for for prime time. But oh, sorry. The the
book wasn't out. It was going through the process, and it takes a couple of
months from the from the time all of the drafts are finished
until they they make a book out of it. It was my very first,
book project. And, yeah, I I'm pretty sure I was I was so
excited. I was telling everybody, I worked on a book. Oh, yeah. Yeah. Because it
was for the Richmond Code Camp, which was in May, April of
Yep. 2006. Yeah. 2000
Yeah. It was 2006. You and I. Where I did A team. A team.
Developers on a plane, and I had the guy I photoshopped the
guy carrying your book. That's right. I do remember that.
Yeah. I have to find that picture somewhere. I've been I've been using
SSIS for a a long time. I would say I
learned more about data engineering, the
field and did more projects probably
in, in data warehousing where I used SSIS
for for the data engineering, data integration. I think it's important to to
to, 1, explain for those who may not know what SSIS
is, and 2, explain that data engineering was not always seen as a
discrete,
profession or or Yeah. It's a data engineering's a
relatively new word to describe what we do. It was called
the part of business intelligence.
Back even before all that, I think the first term I
heard was data acquisition,
and it was in it was sometimes that was that phrase
was used standalone. The most often,
at the time when I and this is what got me into databases
was doing system control and data acquisition or SCADA
systems. These were manufacturing systems where you collected data from
instruments on the floor. You gotta remember, IoT
was, you know, still somebody's dream back, you know, in
the 19 nineties. IoT. It was just OT back then. It just was
OT. You're right. It's funny. Yeah. But
but we still did acquire, data from,
plant floors and instruments that were mounted all over, but they weren't
Internet enabled at that time. They were, most of them were hardwired. A
few were using wireless. And so that's kinda what led me
into this whole this whole field. And the idea of the
field, is of data engineering, data
integration as we called it back then, is we do that data acquisition part.
We go find wherever the data lives, we go find it there.
And sometimes the data is a very static
list. It it could be even a text
document, created in notepad that
is tab separated or, you know, delimited
by character position or something like that. And a lot of old
old lookups, lookup data was that way. And I'm not making
that up. It was maintained in a a text EDI.
EDI. Yeah. So electronic data interchange. Yeah.
So, yeah, EDI is I have an interesting stories about EDI, but but one of
the things that really kept me away from the data space for a long time
was I didn't wanna be DBA. And this work, I think, had traditionally
been kind of merged with DBAs. Oh, absolutely.
But at some point, I don't know exactly when it really
evolved into its own discipline. And I remember.
Go ahead. Because I remember I tried to get you a job at a particular
company. I remember that. And what do they do? And what was
it? Why do we need a DBA? You don't need a DBA.
Right. And I think that I'm not DBA. That
was the funny part. Well, that was the fun. Well, we clearly did because at
the time there was a project going on,
and I think the term data architect is what you just said. You were I'm
not a DBM data architect. And then that fell
on deaf ears. And, ironically, like,
not like a couple months later, there was a project that we worked on that,
so many stories, and I'm just trying to protect the innocent and the guilty,
and myself, from from from libels. But, basically,
there was a project going on that when it was basically kind of
behavioral analysis of social networks. Right? This is before
Facebook. I think Myspace was around that sort of thing. But it was basically the
idea of organizational networking as a discipline. And it turned out that the
the software that we bought off the shelf would actually query the
database, bring everything back in from the database,
and then run through the filtering on the C Sharp
components on the web server. Gotcha. So
long story short, there was 0 optimization, hardly an
index. I mean, it was just a mess. A data architect
will use the terms of the day, would have slot spotted this right away. We
didn't. And it was just a massive disaster. And it's kind of one of those
things where there were a number of projects that that company was taking
on. Basically, one of their one of their core
business models was was brilliant actually was software maintenance. So you have an
existing application offshore or outsource it outsource it to
us, and we'll take care of it for you. And,
you know, it was really like an an an education
in kind of Jenga programming. Right? Where you had they wanted updates
to this stuff, but they didn't wanna pay to redo it. So you kinda, like,
had to replace rip and replace stuff. And there's one particular
instance where there was a SQL query that took like 14
minutes to bring back an answer. And I'm like, it's only like
like 30,000 records. Like, what what's the deal here?
And turns out there was no indexes, no nothing.
Well, you know, those indexes take up space. Right? Exactly.
Exactly. I mean, this is like why you should save space.
Joke. That's a joke. For one reason or the other, like, there was there was
no index. And I was like, well, let's add indexes. And like, no, no, no.
We can't change the schema. Okay. So what I end up what I end
up doing was creating temporary tables with indexes
and then copying all the data, and I still got it down to 2 minutes.
Nice. Which 1 minute and 59 seconds was copying the data,
and then one second was actually changing. Yeah. So, like, it was it was kind
of like what I call Jenga programming or Jenga architecture. You had to like they
wanted updates, couldn't touch too much, couldn't change anything,
couldn't improve anything because it was just it was a
time in my career that I think back of and I've kind of learned
many lessons, both hard lessons and soft skill
lessons. But Sure. But we digress. But, I'm just
gonna I'm just gonna take that answering your question in my usual
long winded way. SQL Server Integration
Services, came along, and it was probably
the, again, it was the thing that
spanned the longest part of my career. Before that, I worked with something called
data mirror. That was the first, I'd I'd
say the the first system like that. First bit of software that
way. Before that, I was writing my own. So I
was reading from these plant networks and writing to all
sorts of stuff. And I got into SQL Server because
I crashed access back in the nineties. So I ran,
I collected a 1,000 points of data every second for a long
weekend. And I wanna say the access file grew to about 4 gigs.
When I went to open it and start doing some analysis on it, it turned
out it wouldn't open. So 4 gigs is nothing now. Right? You
can do that on a smartwatch. But back then, a server
struggled, to open the file system. If you go back far
enough, would have freaked out or anything over certain size unless it was, like, NTFS
or something like that. Right? Yeah. And this this
wasn't. This was, one of the other OSs. But
so, you know, I went I went to, altavista.digital.com
and typed in Microsoft database, and I saw this
listing for something called SQL Server, and that's how it all started.
Well, then I I got got in as, working
on a data warehouse, and part of my job moved
into the database part of it. I actually was hired to do the reporting piece
of it, and lots of cool lessons learned there as
well. But on the database side, they use Data Mirror. I
think that company is still around. I'm not sure. But this is like 25 years
ago. And it was it was so cool,
and I was fascinated that somebody had built software to
orchestrate this collection of data. I was like, wow.
That is a good idea. You know, it always makes me feel better, Frank, when
smart people come up with an idea that I've also come up with independently.
It makes me feel like, okay. Maybe I'm onto something. Go through all of
that, data transformation services or DTS, and then finally, SSIS
and this big block. And
what I've noticed and I kinda noticed this trend started
maybe 4 or 5 years ago. I people complained about SSIS before
that. Don't get me wrong. And a lot of it is
because are you sitting down? It's
hard. We're not making it up.
Comparatively, though, like, I I remember when I was at
barnesandnoble.com and which just goes
back a ways. So if you bought a magazine at Barnes and
Noble between 1996 and probably about
2012, 13, you
you interacted with the system I wrote, nice in the late
nineties or at least part of it anyway. So, you
know, that's how I learned EDI, right? Because we get these feeds
from publishers, literally a mainframe would dial up another
mainframe, download the file over a modem.
And, and this is how it worked. And what we did was we pulled down
the raw EDI files and I parsed it
and I had to do that and drop it into an informix database. So it
was a cool writing for GL scripts to to to take
that data in text format and then dump it
into an actual. You were doing data engineering. I was
doing data engineering, which is kind of funny. But like, you know, data engineering as
a discipline is not easy. Right? So SSIS being hard.
I mean, you know, brain surgery brain surgery is hard too. Right?
You you make a good point about it. And it, you know, it took me
a while, especially teaching it. And I would do
4 or 5 day course, originally with solid q and then
eventually on my own. I I wrote my own course. I
found myself adding to Eric's content when I would deliver the
material here. And don't get me wrong. Eric
is still a genius. He was then and he still
is. I just I I had a way of approaching
some, demos and examples that I felt kinda added
to the clarity of the information we were sharing. I
kind of expanded that out and wrote all my own material, my own I
use my own data, that I collect as part of my, weather station
here. And to this day, there are students that
are going through, recordings of that class.
The last recordings I made were back in December 2020,
and I recorded 3 courses on SSIS. The 4
day from 0 to SSIS course was, you know, will take you
from if you can spell SSIS to being a
functional, advanced beginner, low
end intermediate developer. And it was built for
that. It's got labs 13 12, 13 labs
that you do in 2 days, of that course. And then it talks
it kinda changes gears and goes to the care and feeding of SSIS
and ancillary topics. So
I learned a ton about the concepts
of data engineering on while as
while doing SSIS training and consulting and
development. So when I teach it, Frank,
I share these concepts that
I learned. Because you gotta keep in mind, this all came out around the same
time as a data warehouse toolkit book, by,
Kimball and his crew. And the in
fact, I don't know what the relationship was between Microsoft
and Kimbell, but I do know from the horse's mouth
that the, data flow task in SSIS
was modeled to load, Kimball data
warehouses. There's just a lot of functionality baked right in
that, you know, targets those star schemas, and, you know,
it's it's built to do that. There's so, you know, there
was that aspect of it. So at the same time, I'm reading
and learning and, you know, and then going out and teaching
and, you know, and and consulting. There's
this nice amalgam going on. I'm getting information from books.
I'm applying that information on consulting gigs. I'm
figuring out new ways to solve, you know, problems I hadn't
seen before, And then I'm training. So I'm just
rolling all that together. When I do the training, I'm sharing with people, hey. Here's
some first principles, if you will Right. Of data engineering.
And we call it data integration and BI back then.
And star schemas and why you use them and how they work and, you
know, kind of the trade offs that you get. Data explodes a little
bit. Talking about concepts like staging, data,
the benefits of it, why you like, how
you would wanna build your staging tables.
If you're reading from a flat file, everything in
that file is text. Now the text may be
numbers. It may be dates, but it's really just text.
So you built the stage tables with and bar charts.
So at, you know, stuff like that because you wanna get in and get out
just quickly. Memory than the way to do it would be in memory and then,
like, do validation as you do the insert and things like that. There's a there's
a 100 different ways to slice that. Yeah. There really are.
But, you know, when you did, that was that was just pieces and parts of
saying, okay. You know, Tim, I'm teaching you how to use this mechanism,
if you will. Right. SSIS. But I'm also sharing with
you how you would use it and then why you would
use it that way. And, you know, so there's more to
it than just the data engineering. And the point I wanted to make thanks,
Hector. Merry Christmas to you too, Hector. The data
engineering all by itself, just that world,
that's hard all by itself. Yeah. Absolutely. And then the tool
itself was extremely
flexible. And, you know, from the years that you and I have been sharing
about stuff, anytime you say it's it's flexible, you're
also saying, the the it's a sonic way of
saying it's complex. Right. And if
it wasn't flexible, people would say that it's too simple. And, like, it's just
one of those things where now that I'm in a job where I am in
a on the product group, what Microsoft would call PG, a product group or or
team. Yeah. We call it a BU. I I understand. Like, there's
only so many hours in a day that you have engineers and
there's time to market. You have to kind of make these trade offs.
And, you know. That's it. I mean, that's that that I mean,
I had this real eye opening moment with with I think suspect was the guy
who introduced us, who was an evangelist at Microsoft back in the day.
And, you know, I wanted some new shiny feature in Visual
Studio 2005. And, you know, I was complaining about it.
And he kind of pointed out like, look, even Microsoft has limited
resources in terms of people, time and testing and material and
things like that. And I was like, you know, I mean, my god, if Microsoft
has that problem, then I guess everyone has that problem. You know? It turns out
they're just a bunch of software developers just like the rest of us. Turns out
they're all humans. Although maybe now it's mostly AI. Who knows? But,
it's getting there. So so so, you know, I think we both kind
of set the stage for the controversy here. Right. SSI has
been around for at least 20 years,
maybe 25 and SQL Server itself. Let's let's remind folks the
history of SQL Server. It was originally who was it a
partnership with? Sybase? Yes. I believe it was a
Sybase product, completely. And I don't know if it was like And it was like
version 6. Got into the mix, and there was a collaboration
or something, and then they ended up with it,
owning it. That's my best guess on it. I actually
I I know I haven't spoken to her in a while, but I was I'm
friends with and and have co worked with, with
Caitlin Delaney. And she was
with Sybase. Oh, okay. Yep.
So, you know and did we have her as a guest on the show? I
know we wanted to. We we totally need to because that would be Yeah. Interesting
story because I first heard of SQL Server when I was at Barnes and Noble
because at the time we were ready to launch in 19 this is why I
left Barnes and Noble. We're ready to launch by Christmas of 96 with a
Yeah. Linux or Unix based based system based on Spark, Oracle,
and a few other things. No. I'm sorry. 4 g l. It was Ultimate
Formics. And, you know, we had the hardware. We had
everything set up. And then as the story goes, Bill
Gates and, one of the Riggio brothers who was the CEOs
of kind of co CEOs of Barnes and Noble at the time.
Bill Gates had kind of I don't know what he'd done, Jedi Mind Trick.
In August, September of, like, 96,
basically said, no, we're ripping everything we've built so far and we're moving
it over to Microsoft tooling, which at the time was not really mature. I
mean, it was this is like inter dev. I think we had a beta version
of visual inter dev. Yeah. Yeah. Which
was not the best product at the time. Right? It was, you know You know,
I used it At the time. At the time.
Yeah. I I used it, and if you came
from, like, cold fusion or some other development platform.
Yes. Was also awful. Yes. But yeah. So
So I started on inter dev. In fact, that was the first tool
that I I remember downloading for, Visual
Studio. I don't think I downloaded it. I think I went somewhere and bought a
CD or something. Yeah. Yeah. I think I found it in her dev 97 CD,
which was the the second or third version. But, I mean, I we we had
everything written in per on CGI Pearl scripts. Like, we had everything,
and it was just a very different era. But my
take was and this was my I was at the meeting with the CEO and
everyone else. Like, if we don't launch by this Christmas,
people are not going to use us as a habit. Amazon will
take the mindshare and this and that. And then then the
CEO said, sit down, s t f u. Basically, you don't know
how to sell books. You may know technology, but you don't know how
to sell books. Now we can look back at Jeff
Bezos' super yacht and his, you know, moon
missions and all that. These guys have super yachts and moon missions.
Right? They do not, actually. Oh. And my well, I
mean, I'm pretty sure they live in an oceanfront thing in Long Island. But,
he didn't know anything about selling books online either. So I can kinda I
can sit back here, you know, some, you know, good God almost 30 years
later and kind of be smug about it. Right? Right. But
it's just it's just funny. Right? Like, so so what's interesting is and I think
this really cuts to the bone of what this controversy is. And I
have the thing queued up. I can kind of show the screen where you posted
it, where
the fundamentals haven't really changed. Not at all.
Right. Yeah. Binary is still binary.
The debates about schema optimization and things like that are still
very much the same today as they were
20 years now. The numbers are bigger. The stakes are arguably bigger.
But for the most part, the fundamentals haven't changed. And and
I would say this is really where it kind of boiled down to. And this
is this is where the controversy starts. So buckle up, kids.
Let's see. I will share the screen. There's actually
2. I think you talked about one of them. The choices? I'm only
aware of 1. This is the this is the one post,
and I'll drop you the link, to to one of
the others. Right? Yeah. I'll put it in a chat.
I'll I'll send that to you here. Just a second. Along can can understand. So
this is what I saw. And it was basically Kendra
Little, who was a I would say legendary. Scary
smart. She's legendary in in in in the sequel
kind of family, right? Hashtag sequel family. Is that still a thing? I
think so. She's legendary. She used to work at Redgate. I think she worked at
Microsoft, too, at a time.
I think so. But I'm not positive. Well, we can look at LinkedIn. If only
we had that information. But anyway,
so you basically so if you read this and she says
so it says strong disagree. Don't run after every shiny
thing. Again, that is good advice. But, Lord, I would assume
that is her saying. But Lord, don't learn SQL Server and
SSIS if you want to be a data engineer. That's 2 decades too
out of date. Sincerely, a SQL Server expert. I think that's
a bit harsh. She's right about this part. Don't try to change chase out
there if you show anything. So apparently, I can't, and I
can't select a thing. So I read that,
and and I know there's more controversies that are in there as I as
I look at the thing. And you said I humbly submit
data engineering may be accomplished even in the year of our Lord, 2024,
using T SQL, this foul year of our
Lord, 2024. To borrow a phrase from Hunter Thompson,
T SQL, SIS, ADF, Fabric Data Factory
and other technologies supported by Microsoft, which I thought
clearly Microsoft's not going anywhere. Right? Yeah.
And so I basically said
fundamentals never grow out of style. Then I think I wrote again somewhere
like when I looked at the context of it because that's
not what you're supposed to do apparently in social media. You're supposed to react right
away. I did that, by the way, Frank.
I'm guilty. I did not go look at the context. So this is the
original context. I well and, you know,
you pointed that out and I'll I'll be honest, I I'm still
running on second hand information. I have not yet clicked it and gone back
to, to our guest post. Now I can see
it. Now you can see. So so this is what struck me is, well. This
is what struck me as odd. And I know we had talked about it and
I had talked about it. You talked about it. We talked to each other about
it. You know, we talked to our dogs about it. I don't know. Like, but
like, it was kind of like so so when I read the thing, it gets
even stranger. Right? So Yeah. He was talking to
someone, and I guess strictly speaking, even this is secondhand knowledge. Right?
But, so that's the data
scientist in there. Like, well, strictly speaking, this data is also all right. So
so look looking to someone to get a job as a data engineer. Okay?
Right. Unfortunately, he was learning about LLMs and other ML stuff.
I'm like, that's not data engineering. That's a
AI engineering or data science type work. That's more like I think he's he's trying
to set him straight from that. He's like, you're learning the wrong things. That's how
I read those two sentences. I mean, I would say you're learning the right things
if you wanna be an AI practitioner.
Yeah. But I wouldn't call I wouldn't, you know, read up on Langchain,
you know, Ollama and anything LLM and all that stuff
and then call myself a data engineer. I mean, that's Yeah.
That's like a cardiologist cutting up you know, doing your taxes. You
know what I mean? Like Sure. Or or cutting open your brain. Like,
I mean, I suppose there's some similarities, but it's not
the same. Well, I I do like bullet number 1.
Yeah. You know, let's see that. Yeah. This is something I think
that you point out quite a bit. So when you give your talks, either on,
SSIS or ADF, you ask
people, like, how many people here have workloads running in the
in the cloud or right? And then only a quarter of the hands
go up. Well, it's it grew to about
40% the last time I did it, but it's been over a year
since I since I spoke live and asked that question, ran
that little survey. There's a slide usually hidden in,
all of my presentations that has survey up near the very
top. Right. You know, it just and that's that's what the survey is about.
And often, especially
say the last, I said it's been over a year. So let's say from a
year ago and then back maybe 4 years of asking that
question. Almost every time I did that and people
didn't see everyone else's hand go up with theirs,
the those people would come up to me at the end. And usually, their
first comment was, I didn't know
that it was most of the people here were not doing
production jobs in the cloud at this point point with data. I thought we
were way behind and we're the only ones. And my response would be
2 fold. The first would be, that's because Microsoft
marketing is doing an astounding job. That is not a swipe
at Microsoft Marketing. If anything, they deserve a raise
because they were so effective at communicating
how cool this is Right. And how these larger
companies are doing it. You all of the big shows, keynotes,
There's some list of big companies, and they're almost all of them or
companies that you'd wanna work for because it's prestigious.
That's so I don't know if you want it on my personal market. Seem like
everybody's doing it. And I I know I know for a fact it's not always
true because when I worked in the sales for Microsoft, we
would encounter them and there was a pejorative term used internally called server
huggers. Okay. Right. Because like, oh, they're
server huggers. They'll never go to Azure. Right.
So so now, you know, I used to see that it's server hugger as a
pejorative. Now in light of kind of maturity and
working, with more customers and being
more aligned in the open source kind of realm and dealing with
international customers who have very real regulatory concerns.
You're right. Call them smart. Right. It's not, you know,
I didn't so much drink the Kool Aid is I became one with
the Kool Aid. You couldn't tell where I ended and where it began, where I
kind of had this deep programing experience
of. Yeah. That's not always the answer. Right. And I
think that dealing with LLMs and AI and things like
that, I think really makes that more obvious.
Right? Yeah. I totally agree with that. And, you know,
to be fair, and I wanna start with, you know, with being as
positive as I can about this. If I was It's not a negative on any
from scratch. Wasn't. No. I I'm just saying. But if I'm starting
today, day 1, and I wanna go, be a
starter company and and work with data, I it
would be foolish. Foolish to start
today and not go to the cloud. Absolutely.
So and and the reasons are numerous. Yeah. Here's the
thing. The companies there are a handful of
companies, really large companies, mind you,
that have started sent in the cloud age. Let's just call it
that, or the Internet age. There's a small number of them
that have gone on to be huge, but they are really huge.
They're overpowering, oversized. They're larger than the
companies that are previous to the Internet age companies
that have made their way into the Internet. And that's that's
not an accident. However, those
companies, the brick and mortar companies, are the companies calling consultants like
me and asking me to help them either
transition from a purely on premises
environment, managing their data into a cloud environment
or the and back before that, in 20 years ago, when I was first
getting called to do this kind of work, they were just trying to figure out
how to collect their data and then analyze it. And
so, you know, SSIS was a great way to do that. T
SQL was everywhere. Azure Data Factory didn't exist. Yes.
Much less Fabric Data Factory. And so we were just trying to solve
this business problem. And I was trying to couch couch my responses,
especially there was a thread that that got combative, I
would say. And, you know, as we went went down
through that, and I kept trying to say, and
I did. I said over and over again that, you know,
my job is to go help solve these business problems.
And what I meant by that opening line,
that T SQL, SSIS, Azure Data Factory, Fabric Data
Factory, even in 2024 of viable ways to accomplish
data engineering. I I meant that, and I'm not back backing
off that for one minute. I I misunderstood the context of the question,
and I didn't really understand until I listened to your stream
last night where you had gone back and done what I should have done and
read the original post. And you said, yeah. It's kind of a mixed mesh
post. The guy's talking about data engineer, but he's also talking about LLMs
and machine learning. And in the middle of that, he
throws out, you know, this comment about SSIS,
how 90 99.5% of the companies are still using. I
think that estimate is high. I I think it was more of
a, let's make this point that there's still a lot of companies out there
using, T SQL and SSIS to
accomplish this. And this is something that I can't find the comment that I put
in there. I'm looking for it now, but. Yeah. Some of the
comments I can't get to anymore. I don't know why. Maybe they were
reported or maybe they're. Who knows? Right. I mean, social media
does weird things to people psychology. But the
point that I think that I wanna say
that Kendra overlooks. I think everyone overlooks it.
Data and back end systems have a
longer shelf life. And I say
this as someone who was, what, 10, 15 years ago,
strongly ensconced in client development. Right? Whether it was your
Windows, Windows Phone, or other types of Windows
based devices. Right. Or web development. Right.
Those technologies turn over pretty quickly.
Right. You know, you're likely to get
multiple updates per year on a device phone, like an app on
a device, but you're likely to never see,
a radical change or redesign. You'll you'll see a
radical change or web redesign of a website or portions of a website
couple times a year maybe. Right? But you're never gonna see
a radical redesign of a data back end system,
but once or twice a decade. And It's true.
Yeah. And mostly what drives that is scale,
not features. Right. Not features. It's just date or just tend yeah.
Exactly. Right? So if you a 100 x and who could who could accounted for
that, you know, going to the project started. It's a problem. Still a
wonderful problem to have, but but a problem nonetheless. Well, and there's
also the fact that, you know, it's 2,000 whatever now, and
there's still mainframes running. Right? There are still not not not to to
knock on IBM too hard because they are the company of Red Hat. But,
d b 2 is still around, still getting updates. Still backbone of
many Fortune 100 companies that also share the stage with Satya
at these big Microsoft events too. Right? Like Which was mind blowing
for people from the old days of Microsoft. Right? Well,
that's a whole other thing. But, like, you know but, I
mean, it it really boils down to, like, these technologies have a longer shelf
life. So if something is 20 I think we get
hung up. 1 of the threads sub threads in here gets hung up on, you
know, 30, 20 year old technology. We're thinking that, well, you know,
there's a meme of the the little monkey puppet, like, you know,
giving a side eye and then goes like a cringe face, like, and a side
eye where it's like, oh, Windows is, you know, I don't know, 40 year old
technology. And I'm thinking, like, some, you know, Unix people or Linux slash
Linux people are, like, 40 years old is old. You
know? I mean, this stuff goes back much further. So it's but
it's still like and that's not a knock. It's just
No. It's just now that we're in this
industry now for as long as we've been in it and the
industry's been around longer this long, there's just
stuff that is gonna just start aging out, but it doesn't age out as
quickly as we think it does. It's not like it's not like the iPhone. Sure.
Right? Where you the iPhone I don't know what number they up
to. 16, 17. Right? Oh, well, suddenly my iPhone 15 looks bad, and that
happens every year or 2. You this you don't see that in
database systems. Right? The only impetus to really move, say, from, like, SQL Server
2,005 to 2019 is updates stop going. Right?
And that's a whole big project. Yeah. The maintenance cycle. So it goes out
of maintenance. And then you worry that if something crazy happens,
you can't get support for it. And that's
kinda like, you know, it's it it's sort of it I'll say this.
It's analogous to your phone starting to run slow for some unknown
reason. That's funny. Something something on SQL.
Yeah. Something something. Sybase something. Well, and you think about all the
I mean, I mean, and contrary to this, contrary to that statement of these things
have long life shelf lives. Yeah. Is the fact that I mentioned
Informix earlier. Raise your hand if you heard of Informix. Right?
So I've heard of Informix. You've heard of Informix? I mean, we don't count.
But but, like no. But, like, I remember my first
experience with Informix was because some alum of Fordham had
because it was a big shot at Informix. And, and I think we had somebody
who was also a big shot at Silicon Graphics. So we had SGI machines
running at Formix. Right. So I remember my first UNIX I used was an
IRIX system. Right. Which most people today
wouldn't even know what what that means. Right. And, you know, but Informix is
out of business. Sybase is gone.
I can't even think of other names. I know there's more.
Right. But really, the only things that it those have probably been
migrated to SQL Server or Oracle. Well,
or some form of Postgres or something like that. And I
I hear you. You know, there's there's an argument to be made
for, you know, the the cost of maintaining
old software. Right. There there definitely is.
I'll say this about SSIS. I if you learned
SSIS in probably in 2,005 era, between
2,005 and 2,008, that engine
I I don't know how many lines of code were
changed before it was upgraded to 2008
or r two, but it changed. There were some performance tweaks in there. It was
obviously, faster. And then again, that happened in
the 2012, error when we saw
I love that SQL tab.
You know, it's dead. Long live Crystal. So was Crystal ever database or was it
just Crystal Reports? Reports is all I knew. I didn't know about it as a
database. That's all I I I use Crystal Reports and
my favorite thing was it filling up the drive because
it kept caching things. But I
remember the whole idea of just because you place it somewhere, it doesn't
mean it's actually gonna end up there. Like, the whole thing is
but sorry to cut off. That's okay. But SSIS in general,
if you learned it even in, you know, 2006, came out
in November, I think, of 2,005. Even if
you learned it then, it at at a fundamental level, it
hasn't changed that much. And whereas you'll see other software
you Visual Studio is, you know, a software development platform
that allows you to do c sharp and v v and, you know, all of
the stuff. And it allows it still supports for us. I know. I haven't tried
v v in 9 years now. So
it's been a while. But if you look at
how much most software changes from a developer
perspective, and SSIS is software development. So as your
data factory, and any data engineering, that software development,
SSIS is probably in the 95%
of what it was. If if you knew the fundamentals
in 2006, you know those fundamentals in
2024. And Right. Part of the decision
to go make the upgrade, we talked about, you know, maintenance wonders and stuff, and
I I get it. And it's not the same as your phone slowing down. I
said that, but that's a bad analogy. But Well, it is also
it's also, I think, also very relevant to Windows 10. Right? If you're
on Windows 10, your updates are gonna stop in October. That's
true. I don't wanna get on that soapbox and rant. Sure. No. But I
I mean, there's I get reasons for that as well. I don't
like that it's gonna change because I like Windows 10. But,
but yeah. Well, there's there's I'm gonna join you in not
going down that road. But I'll say this. Hey,
Maddie. How are you? The, the
just the fundamentals of data engineering haven't changed. And the
tool itself, you know, if you knew it back then.
And it's, you know, you know it now. And so if you
learned it now, you could go back then and still work
in the previous versions of it with, very little headache. And
that speaks a lot speaks volumes to the,
the team that designed and built that. And
so in addition to the technical reasons for doing this, the business
reasons, kind of revolve around one of my favorite
phrases. I mentioned this in today's newsletters. It's a compelling
reason. Do you have a compelling reason to make this change?
And business people think about this all day every day because
the amount of money that they make, the profit is based directly
on the amount of money that they, you know, they spend and are they getting
this value for it. So if they can improve the performance of
something, say 10 times, and a result of
that is they get, 5 times as many customers,
then that's not a bad investment. That'll just work. But
if you're coming to me and I'm a business that existed
before the Internet, if you're coming to me and saying,
I want you to change to this completely different model,
where, you know, and and the way it's presented
often is you can save money. And that's
true because if I start a new business today, I'm I
couldn't even compete. I'm not gonna be able to stand up the
servers, you know, take that time and buy that hardware and float
that that inventory that I need to manage all that. Whereas,
I can I can pay rent essentially every month
on that service? Right? Right. I I always like to say I
always like to say if I wanted to start a bookstore today,
right, versus 1996,
right, or 1995, depending on when you wanna say when they started.
I mean, Barnes and Noble spent a ton of money. I don't have the exact
number, but I can there's probably tens of 1,000,000 of dollars, probably closer to a
$100,000,000 to just before they had their first customer.
Right? Wow. And that but that was the heyday of the
dotcom. Right? Because they were you know? But then
but if you wanted to start a bookstore today, whether or not it's a good
idea, let's let's just suspend our disbelief for a
second. You can probably do it on a on an average credit
card limit. Because
because your IT is enabled. Right. And and you pay like I
said, you pay fractions of what you would have to do in the brick and
mortar. And most of the initial spend isn't gonna be your servers or hardware.
It's gonna be in development and marketing. Right? Getting the word out
because it's such a noisy market. It did the the market has radically changed.
And I also think imagine go ahead. I'm sorry.
What? Imagine? I was gonna say imagine that you've built
Right. This infrastructure on premises already. You've got all of this done.
It's a sunk cost. We can debate about how to feel about sunk cost.
Right. But it's there. You spent the money and it's there. And
you're not gonna get that 5 x income boost when you move to
the cloud. In fact, in some cases, not
all by by any stretch, but in enough cases, you
move to the cloud and it costs you money. Because when
you're getting the presentation about starting using the metrics of this
new company being started today Right. You're, you know, you're told the
truth. You're not being lied to at all in in any of this.
But often, systems that were designed software
and front end back end systems that were designed, you know, from
the nineties through the mid early 2000.
Those systems were architected in a whole different mindset
of what's the prevalent mindset for today. And as a result of
that Yeah. Yeah. As a result of that, one of the things missing from
the spreadsheet calculation that you're gonna get the ROI
from moving off your on premises servers to the cloud is
that couple of $1,000,000 and about 18
months, of the hit that you're gonna have to
spend rearchitecting Yep. All of your systems so that they
now fit today's paradigm. And frankly,
if you are interested in doing that, you you could go do that
at any you could have done that at any time in the last 10 years
and made that shift. But people didn't do
it because the business people didn't do it because the ROI was not dead. There
was not enough return on that investment. If they wanted to, they
would have spent that money then, but it wasn't gonna improve the bottom line.
In fact, it was gonna hurt the bottom line. And so you see
companies now make this move into the cloud and
then, yeah. Yes. That is a
that that's an astute question to ask. So for those who may be
listening and not viewing this, it says is SQL SQL dev
d b a says, I use Brent's and I'm assuming Brent Ozarks. Brent
Ozars. Problem are you trying to solve by changing this for justifying
upgrades? Brilliant. That's that is brilliant. And he's
right. And the you know, but it's compelling to hear and
read the case studies of of companies that, you
know, were able to do use to to access
$10,000,000 worth of hardware, like you said, on a credit card. And think,
wow, what would that do for us? And the answer is sometimes, yeah, it'll
revolutionize your business. You'll 10 x coming out of this. But other
times, it's like, no. You'll point 8 x.
You know, this isn't as compelling. So it's interesting because, like, I think
there's a number of and I found the article. I'll pull it up. But but
one of the examples, it was either Dropbox or Box. I forget which company it
was. But but they had basically started off, I think, in
AWS. Mhmm. And then they got to a certain size. They actually
figured out it's cheaper for them to design their own servers that are optimized for
mass storage Mhmm. Than doing it. So they started building their own hardware and their
own stuff. But I could tell you, if they were a startup and they went
to a VC saying, we want to start with this on prem, they would have
been laughed out of the building. Yep. Today. Yes. Today they would
have. They mean, you know, and it's
it just shows that the the shifting economics of cloud versus on
prem and and other types of things that I don't think people really have figured
out yet. So this was a really interesting I'm gonna share this tab if
I can show it on the screen. Sources. But that that
use case is you can't, you know, having the compelling
reason to migrate to the cloud, and you can do that upfront.
It's harder. But exactly what you're showing there, you're
sharing that that idea of leaving the cloud, that's
growing. And it it's growing across the board. And I
one of the, metrics for that that's
directly related to what we're talking about here today with
with data engineering, is that there
there's been an increase in 2024 in the
number of, people that reach out to me to talk
about, SSIS help with their systems.
And, I mean, I do consulting in, you know, ADF and fabric, and
most of my consulting has been in ADF. When SSIS was involved,
it was in lifting and shifting SSIS into an Azure SSIS,
integration runtime. But all of a sudden, after
2, 3, 4 years of that, that shifted this
year. And people started reaching out to me with SSIS on
premises consulting things, and I kept up with it. So I was
able to do it. And but there's other
evidence that I will not share. I probably I may be able to, but I'm
just not going to. But it's even better evidence than my
anecdotes about people more people reaching out to me. Right. That
the amount of SSIS being executed in
the world has increased, and it's a double digit percentage
increase just in the past few months.
And I I think I now this is where I start speculating, and I
don't know the answer to that. But we have a our mutual
friend that we, another mutual friend you and I connected with
in November of 2025. Sorry. 2005. Like in the
future. Recently recently worked for a
year and a half, 2 years for this large agency that's
not part of the government, but does money supply stuff.
Oh, okay. I know. I know. Yeah. After getting his MBA from Sloan, you
know, which no. Sloan. No. Sloan is
important. The, you know MIT. School with MIT.
Right. He's a graduate with that. Super smart.
He shares with me when I'm telling him this story, I give him that stat,
and he says, here's what's going on. Economically,
money is more expensive today than it was. And
so he said he said that as he's telling me this as a
cautionary tale because he says it's gonna change. It's good. Money's gonna get
cheap again, and people are gonna flock back to the cloud. That's his
theory that it's all being driven by money, and I don't think he's wrong,
especially I think that's one level. I think that's one lever. I think there's
more than one lever. That is certainly a big one. But I you
know, as someone who I you know, my previous role at Red
Hat and my current role at Red Hat, I have to think globally. Right? And
we don't again, not a commercial for Red Hat even though the
fedora is there. You know, one of the things we do
is we basically provide a data platform end to end that
can run-in any cloud on
prem or, you know, one of the hyperscale. Or hybrid. Yeah. Yeah. Or
hybrid. Right? Where and there was one customer that I spoke with
before I won opportunity to leave. They were, big government agency.
And this big government agency, you know, they have
their own data centers, even though there was a push to
get rid of them all. But they also have because of way contract
government contracts work in the US, they had, foot
you know, money to spend in AWS, money to spend on Azure, and I think
even money to spend on Google Cloud. So the one
advantage that we had that the other ones couldn't is that the he called them
the soft costs of training people how to do he'd do the same
thing to do linear regression in SageMaker and
push them out of production in SageMaker and Azure and in Google.
Right? Yeah. And this was one tool. You learn it
once. You administer it once. The same glass.
It was the same thing. I think those environments are very real. Now those are
probably limited to large customers or kind of the government
agencies that have these kind of contracts and things like that.
Yeah. But also Mhmm. You have a number of
countries that it's just not a good look to move
your data out of country. Right now, in the
US and Canada, we don't have this issue because there's plenty of all the hyperscales
have footprints in Canada and the US. But if you're in
Latin America, which is this where this example comes from, right, there's only
at least as Red Hat defines Latin America, includes Mexico, and basically all the
way down to Antarctica. Mhmm. And only just,
like, 30 some odd countries. Right? Someone's gonna write me hate mail saying that this
is the exact number, but let's just keep the math simple. It's 30. We
love those mail. We do. We love we love the mail. We learn things every
time. Right. I talk about them personally when I get corrected at
at the dinner table because I wanna share that with No. I mean, it's it's
good. I'm not saying don't do it. I'm just trying to keep the math simple
because it's Friday before, you know, basically, we're sure holidays. Sorry, Frank.
I I derailed you. No. That's fine. Only 3 countries
from Mexico down to Antarctica have hyperscaler presences.
Now, the 4th one in the but out of 30.
Right. So it's actually 10% or less realistically. I think it's like
37 countries. I asked Wikipedia and stuff like that.
So less than 10%. Right? Right. If you're in a
country that doesn't have a footprint. If you're in that 90%.
You have to ship it out as a country. You have to be okay with
that or do roll your own solution on a thing. So there was a
government we we won a big contract because they
wanted to do advanced AI and they wanted
to keep it in country. Right? Doesn't necessarily have to be on prem. Could just
be, like, you know, an Equinox data center down the street or something like that.
Right. But within their thing. And it was a government agency, so it wasn't
computer science or even data science. It was political science that really kinda was the
driver there. Right? Because if I'm in country x and I have to move
my and I'm a government agency in country x, I have to move my data
to a sovereign country y. Not a good look.
Yeah. Right? And,
you know, would it really matter? I don't think so. Like, in a but
from a legal point of view, it kinda does. Like, where the data resides in
the residency. And I think if you go to the Azure website
now, they'll actually tell you where the data resides. And they actually interestingly
enough, they get down to granular, at least on the US side to the
state. Right? So, like, it'll say, you know, Virginia and
stuff like that. We have a comment from, or not. I'm gonna hide my
screen so I can look up the Azure map and kind of
demonstrate that. And then I have to figure out where the sources are. There we
go. So you wanna read the comment while I do that while I'm distracted? Sure.
So so many comments, but there are clients who are
considering the technology used in the software package, and they
may escape when they see the old school,
stuffs. I'm so maybe.
And and that may be, you know, I hate to be that guy that says,
oh, that use case is invalid. I don't think so. I'm not aware of it.
That doesn't mean it's invalid. I'm not aware of a lot of things.
But but maybe. And, you know, definitely
have mixed emotions, about that. If you're buying
because the the if you're a client and you're
buying from, some company and you
decide to go to a different company because of the
technology stack that's being used behind there, I
don't know. I think that says more about you as a client than it does
about the company. If they're delivering the service and it's, you know,
the the the rules of data engineering are you get accurate
data as fast as possible, and those priorities
are in that order. Yeah. I don't think the old school stuff second.
The only risk of the old school stuff is it's still maintained or there's still
security packages. That would I mean, if I were in that
position, I would be like, oh, I mean I mean, if you're still using, say,
Sybase 6. Right? You know, like Yeah.
You know, got a little there's definitely a line there, and it's drawn
based on it's drawn different places first. And some of the
reasons, that it is drawn in different places is
security is huge these days. I mean, that's gotta be your number one
concern. And and, you know, it it
goes from there. But if you're delivering the service securely,
I'll just pick that one, then I would say
that, you know, that if if you
lose a client because you're not using the new shiny,
I don't know what you can do about that. I'm trying to think I'm not
gonna say the client's wrong for feeling that way. They probably have valid reasons
for feeling that way. But if, you know, if they wanna if they
wanna make that decision based on that. And I'm looking at Frank's graphic here
of the is that the data centers? This is the one I was telling you
about. Right? So this is this is just Azure, but I would say it's a
pretty good proxy for the other hyperscalers. Right? Mhmm. I would
say Azure at one point had more. I I haven't kept up.
Well, that was one of our talking points when I worked at Microsoft. Right? We
have more than to be honest. Right? But I would say if it's not exactly
more, it's close enough. Right? So there's Mexico. You see the United
States is pretty well covered. So is Canada. Right? So if you were a Canadian
company, you had to keep it in Canada. You had an option. Right?
Yeah. If you're an American company, you have pretty good choices.
If you're Mexico, yep. But if you're in any of these countries in Latin America,
down through South Wales. We only had okay. So now there's Chile.
Okay. Gotcha. Right? So I'm sorry. Now there's
4.
So my math is going to get more complicated right away. Right. So, there's
Brazil. Actually, no, there's still 3.
So there is no footprint for Azure in Argentina.
These little blue things you see, that is Colombia. Those are networking
pops. So basically, from a
networking point of view, if computer science were the only thing that would matter,
then that be that would be acceptable. But data residency is
the issue. So if I go here, the US East 1.
Right? It'll tell you that its
location is Virginia and it's stored at rest in the United States.
Like like here.
Is there any more details? There isn't. US.
Yeah. They used to. Yeah. They don't talk much about that,
about where they are. But one of the
US East Georgia. 1 of the Yep. The US
East 2 is down in Danville, isn't it?
They are or Mecklenburg. I forget which. Well, essentially, they chose a
picture of Richmond. Right.
I mean, these are, you know, with
we kinda touched briefly on on politics in a geopolitical
slash, sovereignty strategic way.
Right. These are huge. And I I know I'm not the one, thinking
about it. But Look at this. Yeah. There's one in Israel. Those satellites, Frank,
are getting it, by the way. Those those
satellites on the graphic, those things are moving at way faster than
normal satellites. Oh, yeah. Yeah. I mean, satellites are going to
change things, but, like, in terms of where the data sits at rest is really
where because ultimately, I think it really boils down to when are the
local police going to barge down a door with a with a court hopefully
with a court order and basically copy everything. Right. That's really what
matters. Right. That turns out that's really what ended up mattering. But
if you look at the Middle East, right, like 1, 2,
3 countries have it.
Right. And that entire region, you
know, obviously with geopolitical tensions being what they have been for a number of
years. Yeah. Moving your data center to any one of these countries may be an
issue for you for your organization or your
regulatory. Right? Europe is kind of the same thing, right, where, you
know, there's Switzerland, there's Italy. And I
know that there's different kind of things in terms of Germany.
It was actually I don't know if it still is now, but it
might have been a, I used to live in
Frankfurt, actually. Yeah. There was actually what they call a
sovereign cloud because there was concern that if it was a US company owning
a data center, that US courts would have jurisdiction there, which is a
brilliant move by a a past administration. I say
brilliant sarcastically in case you're didn't get pick up on that.
Where they thought that they could basically issue a a court order to
demand something from here in Ireland. And
Microsoft fought that because they realized, like, wait a minute. That would mess up our
entire that would cause a lot of problems. Yeah.
And, ultimately, they dropped the case before it was finally
decided. But in order that they could, thing, at one point,
anyway, this is actually owned by a German
company, managed by a German company, and it's leased to Microsoft to to
to have that concern. I think China also operates the same way.
Frank, I was commenting on the, satellites on the graphic
there. Oh, that they were moving around. Yeah. Yeah. They are moving very, very
fast. And it keeps they're moving with us.
But, I mean, keep in mind, though, like, keep in mind, though, that
we're just talking about data residency. There's other things that if you're building a real
solutions, other things to consider. Yeah. Right? Like And there's
a whole lot to that. And Yeah. Yeah. Oh, absolutely. And,
you know, part of the part of it is,
part of what what happens when you start kinda going back to
the data engineering, platforms and stuff that you use.
There are sound business reasons for not making a change,
and there are some unsound business reasons that will
confine you to not making a change. And I I think about this. I'll put
it in context of, of SQL Server.
Companies will come up and they this has happened, and I still have clients
running applications on old servers
because the company that so they
they serve, their clients include
enterprises that care an awful lot about
checking boxes and auditability and all of that stuff. Regulatory
type things, which is not bad. It's just the
way that it is. That's their their business demands something
like this. These companies were formed. They were stood up, and they've got SQL
Server 2,005 running or 2,008 or stuff that's been
out of the maintenance cycle at Microsoft for a long,
long time. And unknown it's also not a well known
fact that if you don't wanna upgrade to version x or
y, you can pay extra money, and Microsoft will maintain
and provide you patches. Right? There there's rumors that
there's at least as of a few years ago, there were still Windows 95 systems
that were, you know and that sounds absurd.
Yeah. But Well, you walked down the, entry to,
Delta flight, and there was one that's what was it? One is 97, I
think, sitting there. 98. Yeah. 98, was it? Yeah. One is 98.
Sorry. Yeah. One is 98, boxes sitting there for the longest time. They're still
I believe they're 1 to 7 now. Still. I I
saw XP. XP. You're right. It is XP. Yeah.
So, you know, you just I kinda noticed this, like, wow. I hadn't seen
that in a while. But it's it's not about
will the new technology run that
SQL Server 2,005 database. The answer is clearly yes.
Well, you can always virtualize something. Right? Like, that's something that, like I
mean, there's that compatibility levels help. Right. There's a number of things that
do it. But here's the kicker. If the application is
not certified to run on that
and you're for you can change it, and you be maybe you have changed
it and tested it and go, yeah. It works. We'll just move it to, you
know, SQL Server 2019 or 2022. We
know it works. But the people you're serving,
people who care way more about checking all the boxes and the regulations
being a 100% and auditable, they won't
allow you to. And it gets even more complex when that company that
originally sold you that software 20 years ago is no longer in
business. Right. So you have no path forward.
I mean, the only Or they get bought by another company that you
don't really like. Exactly. That's happened too.
Exactly. Then A lot of mainframe companies were brought up by I don't wanna
name names, but, like, were brought up, and they it really was, like,
ironically, because they what they do, they they knew they had them. And
ironically, a lot of mainframe migrations happened because of
that. Like, it was And so you've got, you know, you've got that
angle where people are sticking with older systems for whatever reason.
And it's, you know, it goes like I'm saying, my point is that this goes
beyond just the data engineering realm. There are
there are compelling reasons to use,
older software. It may not be anybody's, you
know, satisfactory answer, but it is, you know, those
reasons exist. And if you're the, you know, if you're a
developer who likes using the new shiny and learning the new
stuff, I'm one of those. That's why I'm teaching courses on fabric data
factory right now and watching as it kind of some
days it works and some days it doesn't. We've had that happen on a
number of deliveries this year, with that.
So if you read what I wrote about this and
you come away with Andy's against the new stuff, well, you're just
as wrong as wrong can be. That's not the case at all.
You know, it's an
interesting I mean, so back to the lecture at hand, what kind of kicked this
all off and inspired the stream was
this post where I think the short
answer is everybody's a little right. Everybody's a little
wrong. And as a consultant, you can appreciate these two
words. It depends.
Right? Because, like, you may want to upgrade to the new shiny. I know every
developer wants to do that. And I think the comment for some reason I can't
say is like basically hiring managers will put in a job description. All
there's also the other matter of job descriptions and, you know, job requirements
are. They're always a 100% accurate. Disconnecting
from consensus reality. Yes. I like to say.
But they may want someone with, like, say, ADF
and and and this, but then actually have them working on systems and SSIS
because the hiring manager knows that he or she may not have that open req
for a while and has a in the back of
the mind the idea of moving to that someday.
Sure. But realistically, for the next 2 years, you're gonna work in this site.
Yeah. I mean And there are still large
consulting companies out there that develop brand new
applications in SSIS, brand brand new data
engineering data warehouse. One worse than that or one better depending on your
point of view. A few years ago, I think it was on dotnetrocks. They
were talking about telemetry from Visual Studio. And this back
when I cared about Windows client development.
Basically, WPAF came out in 2006,
2007. Right? XAML.
No. Not XAML. Metro or modern
applications, UWP came out in 2011,
2012. Right? So there's been multiple
frameworks to write when and it's been a number since. But, again, don't really care
about those client development anymore.
Windows Forms is still the number one of
all those, like, ways to develop Windows applications that run on Windows
Mac. Windows Form is still accounts for 80% of development.
Wow. Something someone's going to like, please email me in
hate with hate mail, not hate mail, but like tell me the exact number. But
it was still. He's off. Well, and they kept saying in Visual
Studio 2005 came out. They said, this is it. This is the end of the
line for Windows Forms. We're not updating. We're not adding anything.
And the future is from now on. Right?
Until the future became something else. And then when
I last installed, I think it was Visual
Studio 2016, 2019.
There was improved. They added stuff to Windows Forms, which is kind of funny because
they said they never would. Yeah. But it just because they.
Demand. Right? Customer demand. Ultimately, customer demand
is what pays the bills. So you've got to be very mindful of that.
And, you know, if you if it's 2024 and you're writing
a Windows Forms app, I have questions.
You know, I'm not saying I disagree, but I have many questions.
Arguably, you could say the same thing for UWP or
WPF. Right? You know,
But, again, it really depends. Like, in the last time I had worked with
WPF professionally was it was for when I
was at a, between my stints
at Microsoft, and there was a, you know, there was
a customer who was a mortgage company, and they basically had their mortgage
intake form written in WPF. Right?
And they were having performance issues with it. And I I looked at the code,
and I was like, this is a good lesson, I think, is
that, you know, they loaded up, like, some 6, 700 controls
all at once. Right? Wow. Because there were a lot of
fields and but they were all collapsed and things like that. And I was like,
well, I'm looking at this, and I'm, like, testing it. And I'm like, oh,
dear god. This is gonna be a nightmare because it's 600 controls. You could do
lazy loading and things like that, but then Sure. There could be unintended
consequences there. And then then I happen to notice when I load the
app, the CPU spikes, but the GPU was
hardly touched, which the whole promise of WPF was
that it would offload as much of the rendering
Yeah. Over to the GPU as possible because it was basically built on
XNA, which was a gaming framework. But that, again,
different different sidetrack, and different lifetime ago.
So I'm like, what the heck is going on? So then I'm like, I looked
at the some of the machines. I'm like, they had the generic
GPU driver. So I'm like, just
for grins, let me see if I
can get the proper driver for this device.
All of a sudden the 30, 40 seconds it took to load that initial
screen went down to 5 seconds.
Wow. And that's a big jump. That was a
big jump. And that was like, and I said to the guy,
it's like, look, just installing this driver, you get
a, you know, massive increase in it. Right? Do you
really wanna architect it or are you trying to just you want this to be
faster? Like, what's considered acceptable? And he said, well, under 10 seconds would be
acceptable. And I was like, I could do you better. How about 5 and a
half? Right. And I showed him
and he goes, well, not everybody has a GPU in their device. And I was
like, well, like, you know, this GPU costs
about, I think, 189. For some reason, 189 is stuck
under $200. Yeah. And I'm like,
so you'd have to install it. You have to think about the labor of installing,
like, this cheap GPU. Right? And this, he goes, that's
fine. He goes, look. The cost of redeveloping and
rearchitecting and retesting this versus $200 per
box, plus whatever it takes for somebody to go in with a screwdriver
and update the drivers. Right. It was so like we
ended up not having to touch the code at all. Right? Yeah. It was just
a matter of a driver update, which nice. Because it was like a fixed
price kind of support contract. I was the hero because I
solved the problem with about 3 out 3 to
5 hours of work. Nice. And the customer was happy because
they didn't have to re architect anything. Everybody wins.
Everybody wins. I love it. Happens. But but it it's just it
just goes to show you, like, sometimes the most cost
effective approach isn't is to not
touch anything. You know, and it's although it was a
relatively small amount of money, and and,
manageable amount of time for the client,
sometimes in if you look at the, you know, kind of the the
big, performance tuning
picture. And I I ran into some of this, in the past where
Well, it's not always a happy ending. As as an engineer,
I want to, you know, to fix the the thing that I'm engineering. And if
it's software, then I wanna make the software perform better. And
if it's, you know so I went through one of those experiences and
then, number of circumstances where
but the end result was we we
threw money at it and
bought, better disks.
And I remember telling me about this. I remember you telling me this. Oh, it's
a specific long story, but, yeah, it's back from about 12 years ago. Yeah.
You were, like, just a long SSDs, and it got fast
enough. And so when I did the math on that was the
enterprise level project, there were dozens of people
tuning on a tiger team, and we did make it
go faster. And as a test, what I did was I
rolled the code back to where it was when we
started. I will say now though, tiger the term tiger
team after my Nobody knows what I mean. Sorry. I know what you mean. I
know what you mean. Of individuals, dozens of individuals focused on solving
a particular problem. It's it's like, Although in some cases
slurring. Focused on not solving a particular problem, but that's a story
for another day. Different different story. You're right. But, yeah, we
we did and and I rolled everything back and ran the old
code that we had optimized. You know, it ran super
fast on the SSDs, and it was running okay. It was acceptable,
barely, on, the spindles. But when we rolled it back, it
was, it was the same difference. We actually got a touch more
performance just off the SSDs. And when it you know, you do the math on
that. At that time, SSDs were rather new, and the amount
that I wanted was, not trivial.
I asked for it on a lark, and I was surprised when it showed
up. So she did work for Microsoft. Okay. I'm not
surprised. Kendra is, scary as much. She did consulting. We
Brent's name came up a a few times. She worked with Brent for a while.
She worked at Redgate. Not everyone knows what Redgate is, but they're kind of a
big deal in this in this situation. Yeah. Yeah. They're a big deal. So, like,
clearly, like, I I just find I'd love to get her, like, initial
opinion after factoring in all kind of all of this is that,
you know, ironically, Sync don't run after every shiny thing.
But the the thing that that guy that was originally learning was is
the new shiny thing, ironically. Like, so there's a lot of layers to this.
There's even if you take this kind of at face value of not knowing the
context, there's a lot of layers. But, like, beyond that, there's even more layers. Like,
it becomes this multidimensional problem. It's true. It's like an
ogre. Many layers. Many I was wondering when we're
gonna have a movie reference because it's been a while. There we go. Boom. Shrek's
a classic. Shrek is a classic. The
sequel is not so much, but that's a common case.
Sure. Very, very few Matrix movies looking at here. It was
you know, I I saw the, activity on this and it's best because
since I posted this, I can see how many people looked at that. The the
link I sent you for the other one is kinda the one that started it.
I don't know if you have that link in chat. I think If you wanted
to click on that. I have That was that was a
few days, maybe a week before this. That was the first one I
commented. It was similar sentiment and I shared, you know, again, I
joined the conversation and then, I can't believe is
SSIS dead? Yeah. That's the one. So that
particular one has gotten, like, probably 12
to 15 times as many views. It's happened to You know, you mentioned And
it was the first time I had piped up about anything like this. I'm I
commented on the original one. LinkedIn will pop this up if you write a
lengthy comment like like that one. And
I said, that window exist. Looks like Windows XP
era. Well, that's SSIS. That is
a funny. Like, it's just funny. Yeah. But, And I
commented when this LinkedIn popped up and said, hey. Do you wanna repost this
since you wrote this comment? And I said, sure. And it just pastes the
comment up at the top of the repost. But I can see, like, the number
of people that and that one drew a lot more,
a lot more comments. And like I said, 12 to 15 times
the views. And, and and I communicated
with the original author just to touch about that at all. It it
went nowhere near as I'm
trying to think of the right word. No nowhere. It it
didn't get nearly as heated. I'll say it that way. And it
may just be that, you know, that I'm saying heated, but there was
there was one individual in particular who just very passionate about the tools that they
that they used for. I I I wanted to ask,
that individual, you know, how
many SSIS packages they developed and put into production.
Because I I I I would again, I think I know the answer
to that. This in the this individual conversations like that.
Say that again? Which individual? Not in that one.
Oh. Not it's in the other range. The one we've been looking at. This angle
here, though. Right? Nothing against Kiwi
ETL pools. What do you Yeah. Appreciate so this is this
is an argument of GUI versus Yeah. Straight
code. I think that's also an interesting concept. Low code, no
code versus you know? I like them both, but,
you know I do too. It's for me, it's which solution
matches best and as a consultant who goes to work with other companies a
lot Right. A big factor for me is What they have. You know, what
happens when I'm gone? Can you support it? What what are you most
familiar with? Even if I'm working in a tech a technology
technician, a technology like SSIS, there's really a
couple of ways you can, float it. It. That's good, John.
2 was good. I'm referring to a 3, and I think there was a 4th
one. Oh, really? I didn't even know there was a 4th one. So there you
go. But in SSIS, you can take a more And
the matrix 2 was good as well. Matrix 2 was good as well. I was
That's true. Yeah. You you could take, like, a more
DBA approach, a more T SQL driven approach, or you
can take a more dot net driven approach because you have both
execute SQL tasks and SQL sources and destinations and transformations,
and you have a script task and a script component.
And, you know, which way do you go? Which is the right way to go?
Well, it depends. If you've got a bunch
of dot net developers being tasked with maintaining
the, ETL after I leave, no way. 5.
Good god. Shrek 5 or matrix 5.
Either one sounds like a terrible idea. You know,
whoever's gonna be left behind support net, you gotta make sure that Right. You've done
a a fair enough job of representing their preferences,
on that. And, you know, I I know
I know consultants who come in and say, you know, we gotta do it
this way and you're bad and wrong and you're, you know, you're gonna go out
of business in 18 months if you don't listen to me. Right. That
if I also think too, like, when I first read it I'm sorry. I don't
say hire somebody else. Right. I mean, I when I first read
it, I first read it as should you learn SQL?
Okay. Right? And that's a big debate in the data science community is should should
data scientists learn SQL because, you know, Python can do everything. Just your no. No
knock on my Python can do everything. Should you use Python for everyone everything, I
think, is another question. Right? And I think the answer is no. Yeah. I think
if you're a data scientist if you're a
data scientist or even an AI kinda engineer, whatever the word is
this week, you should still learn SQL
because this one, it's way less complicated than anything else you're doing. And
2, it's kind of the the the the lingua
franca of anything data or data interaction. Right? And
again, Frank, the context comes into play here. Right. There's the there's the
mechanical tool. It's describing a software, you
know, a language is mechanical is one one way of looking at it. But Right.
There's also the problem you're trying to solve. And if if you're gonna do
data exploration or managing or whatever you wanna call it,
then I don't really care which mechanism you use. But don't
tell me one of these mechanisms is better than the other. It
Without a qualifier. For this particular task.
Yeah. So it could be that, you know, one
of them is is good at this one particular thing, and I I would argue
this. I did in my newsletter. Oh, strike 5. Okay. In my
newsletter, I argued, every single tool has something
that they're stronger and there's there's some
feature or some aspect of it that's better than all of
the others. And they also have some weakness
that's worse than all of the others. It's true. And and so
you gotta, you know, you gotta strike that balance. It's gonna depend
on your use case, the parameters, the things that are important to you.
Sometimes it's cash on hand.
Sometimes it's the servers, hardware that you're forced to work
with because you're owned by somebody and they they're not upgrading. They're not
giving you that staffing concerns. The people, their
experience, their languages, that their Well, how easy is it to find someone that
necessarily knows SQL versus knows Python?
Or, the bill rate for someone that knows SQL is probably gonna be different than
the person who knows Pandas. I think so.
I I think there are differences there, but, again, it's gonna depend on the company.
That's true. You're worth our job market. Right. You're worth
more to, you know, this company as a SQL developer than, you
know, maybe than a Pandas developer is to that that company. That's a
possibility. Go go where you get paid the
most, but realize especially if you're new, and I think that
person that in Kendra's original, post that
she was, she was jamming on, that
that was a different scenario. You're right. There's a number of squirrelly things about that
original post, but if you're
a young developer and just getting started, you're in college. I've got a
daughter studying computer science in college right now. And my
recommendation to her is first, learn everything that you
can. Do as much as absolutely as you can.
Pick up that knowledge. But,
be aware that there's more to it than just the, you
know, the the the brain exercise you get and the
thrill you get from seeing the code execute. Keep that thrill. Keep that spark
alive. Right. Right. Right. Right. But, you know, real
realize there's often more to it. And some of those factors you have no control
over, and the person you're working for has no control over.
And, Ed, you know, there's just a number of things totally external to the experience
of writing code that often
impact the experience of writing code. That's very true.
That's very true. K. Actually, going on for,
like, 90 minutes. So Wow.
Goodness. It doesn't seem like it. I've had this much coffee, and I still don't
have to go to the bathroom. That's amazing. Christmas miracle.
Of you. It's a festive miracle or a Christmas miracle. I don't
know. Awesome. But, so
this is this has been great. I think we kinda got to the bottom
of this is that basically, it's a nuanced conversation.
There's no simple answer. I have many
questions though. Why if you're learning LLMs and now, why are you
calling yourself a data engineer? But Yeah. That's a different thing.
Could just be semantics at that point. Could be.
But thanks, John. Thanks, Merdad.
Thanks, Hector and SQL Dev
DBA. I'm sure that's not the
name on on the driver's license, but
you never know. You never know. I wanna get a license plate holder
that has, like a, like, drop table.
Like, so that way when they
take a picture, they do the OCR. Boom. I've seen
the bumper stickers with longer Right. Right. Right. Right. Right. Secret
hacks on them, you know, secret injection attacks. No. Hey, man. Thanks
for tuning in. And, he legally changed his name. That's
cool. Apparently, there's a number of people
who have, like, a last name, and their
last name is Null. Oh, wow. And I
was looking up on YouTube. Like, there's, like, a number of, like, problems that people
have. And my first thought was, wait. Wouldn't that only be the case if
they encased it in quotes or single quotes? Like, wouldn't it?
Apparently, no. Some of these systems Depends. I mean, you
wanna talk old systems. I'm sure DMVs have some pretty
ancient technologies that are still running.
I mean, I can only imagine. Robert
Tables. Yes. Little Bobby Tables. Little Bobby Tables. That's the
one. I tried to name my kids something like that. But
You got overruled if I remember correctly. I did in so many ways. I'm
glad I didn't go with the, initials of, x
a m l. Yeah. That
that You could have. Xavier Anthony
Marcus was, on the table. See and
Lavinia? That would just Yep. That would just flow. That would be
your role. Though, since XAML kinda died,
it's probably a good thing. That's true.
So XML is even not as in vogue as it once
was. Well, you know our mutual friend, mister Kevin
Hazard, aka the Duke of Hazard. The
Duke. He says he says JSON is just hipster
XML. He's right, though. And there's another format called
JSONL. What? Yeah. JSONLines.
Never even heard of that. It I it's only the
I didn't hear about it until the product I work on, Rel AI, actually
uses it to store our data. And, basically, it's a different
between JSON and JSONL? Basically, it's a long line
where each line is a record of or a chunk of
JSON. That's how I interpret it. I'm sure I'll get
corrected, but please correct me on that one. Interesting. Yeah. I'm like, Jason
now? Like, what the heck?
Jason is an acceptable one. That's true. Jason
would've been a good name. But I didn't want him to get
teased on Friday 13th for the rest of his life.
So so,
thanks everyone for tuning in, man. This was awesome. We should do it more often.
Thumbs up on, or, you know, be sure to like, share, subscribe. I gotta do
this to all the things. Like, share, and subscribe. I can show off,
this little graphic.
You you know, Frank, we could do this, we could do this
a lot more often if there anytime I stir up some trouble, we do a
a live stream Oh, god. We do it every day. I Yeah. That's what I'm
saying. You know? No. I think it's interesting. I think it's good because, like, you
know, it the controversy
I think people are starting I just look last time I looked at the thread
thoroughly, people were talking past each other. Yeah. Which I
guess defines all of Reddit, mostly Internet.
You know, it is what it is. Yeah. But
cool, man. I gotta actually, now that I mentioned it, I do have to go
to the restroom. See. You did it to yourself. I did do it to myself.
And thank you, Miranda, for turning in. And
I will play the outro graphic. Excellent.