1 00:00:00,240 --> 00:00:06,680 I'm Miko Pawlikowski, and this is HockeyStick. 2 00:00:06,699 --> 00:00:09,510 Today, we're talking about generative AI for code. 3 00:00:09,790 --> 00:00:12,409 you know, whether software engineers should start worrying 4 00:00:12,409 --> 00:00:13,650 about their job security. 5 00:00:14,090 --> 00:00:17,649 We're talking about chatting to LLMs to help you design, build, 6 00:00:17,800 --> 00:00:21,720 test, and understand software, both online and offline, as 7 00:00:21,720 --> 00:00:22,980 well as tools like the Copilot. 8 00:00:23,884 --> 00:00:25,085 I'm joined by Nathan B. 9 00:00:25,085 --> 00:00:28,765 Crocker, the author of AI-Powered Developer published by Manning 10 00:00:29,125 --> 00:00:33,345 and the CTO and co founder at Checkr, a tokenization startup. 11 00:00:33,745 --> 00:00:36,825 He just finished his book and we're covering his experience with 12 00:00:36,825 --> 00:00:38,875 using AI as a junior developer. 13 00:00:39,324 --> 00:00:43,824 Welcome to this episode and thank you for flying HockeyStick. 14 00:00:44,316 --> 00:00:47,376 So I, had the pleasure to read your book. 15 00:00:47,626 --> 00:00:51,646 I would say that it's a real practitioners guide to using 16 00:00:51,646 --> 00:00:54,346 AI-power, to, work with code. 17 00:00:54,816 --> 00:00:57,966 It's, fairly light on details, so nothing to scare people off. 18 00:00:57,996 --> 00:01:01,876 It basically jumps right into how to get value out of, artificial 19 00:01:01,916 --> 00:01:04,226 intelligence, if you're working with code. 20 00:01:04,696 --> 00:01:08,006 Would you like to tell us a little bit how you ended up writing this book? 21 00:01:08,081 --> 00:01:13,271 I had actually had a number of, co-workers and other developers who are telling me, 22 00:01:13,271 --> 00:01:14,601 "Hey, you got to check out this stuff". 23 00:01:14,621 --> 00:01:15,531 It's really something. 24 00:01:15,861 --> 00:01:17,911 So this was November of 2022. 25 00:01:20,031 --> 00:01:22,351 and, I had no idea what it was. 26 00:01:22,351 --> 00:01:23,961 I, I started looking into it. 27 00:01:23,961 --> 00:01:28,831 It piqued my interest, but, I needed a really, a deep motivation to actually 28 00:01:28,831 --> 00:01:30,681 dive into it and really incorporate it. 29 00:01:30,691 --> 00:01:34,101 Cause you just use something, periodically, lightly, You're 30 00:01:34,101 --> 00:01:35,181 not really engaged with it. 31 00:01:35,711 --> 00:01:40,311 so I, I pitched Manning the book, they liked the idea and it's 32 00:01:40,311 --> 00:01:44,581 really is my journey through, learning how to use these tools. 33 00:01:45,091 --> 00:01:48,601 my journey is really going to mirror the practitioners that are 34 00:01:48,601 --> 00:01:49,931 reading it as they work through it. 35 00:01:49,946 --> 00:01:51,236 So who is it for? 36 00:01:52,076 --> 00:01:55,506 What's the requirement to get value out of this book? 37 00:01:55,556 --> 00:01:57,366 should have some familiarity with Python. 38 00:01:58,916 --> 00:02:02,036 if I just take a step back, it's really for anyone. 39 00:02:02,086 --> 00:02:06,096 early journey, mid journey, as a software developer, as a software architect. 40 00:02:06,516 --> 00:02:09,576 I suppose as a business analyst, you could derive some value. 41 00:02:10,186 --> 00:02:15,386 All the examples are in Python, There's a couple of microservice, chapters, so you 42 00:02:15,386 --> 00:02:19,866 should have some familiarity with that, but it really is about, taking you through 43 00:02:20,806 --> 00:02:25,966 things that you may or may not be familiar with and working with gen AI to really 44 00:02:25,996 --> 00:02:27,886 teach you some of these concepts as well. 45 00:02:28,446 --> 00:02:34,606 Yeah, maybe you graduated from computer science program and you're like fresh out 46 00:02:34,606 --> 00:02:38,606 of college and you want to know how to take your development to the next level. 47 00:02:38,896 --> 00:02:40,776 That would be really be the target demo. 48 00:02:41,776 --> 00:02:47,016 I think the next level part of it is actually the keyword here. 49 00:02:47,516 --> 00:02:50,806 One of the first things that people see when they open your book, I 50 00:02:50,806 --> 00:02:55,176 think it's literally like page one, is the silent promotion. 51 00:02:55,246 --> 00:02:59,766 Everybody all of a sudden overnight became an engineering manager. 52 00:03:00,251 --> 00:03:06,301 Who basically can have a pretty good, junior working for them for free with 53 00:03:06,301 --> 00:03:08,101 no labor laws or anything like that. 54 00:03:09,291 --> 00:03:10,601 Why is that such a big deal? 55 00:03:10,814 --> 00:03:14,314 it's such a big deal because it used to be the rubber duck. 56 00:03:14,324 --> 00:03:18,274 Like you'd have a partner that you can work with, that you can tell to 57 00:03:18,274 --> 00:03:23,044 do things that you can bounce ideas off of, you can look to for some 58 00:03:23,044 --> 00:03:26,044 answers, because they're thinking it's going to be different than yours. 59 00:03:26,484 --> 00:03:29,034 so they might have a different tack and a different approach. 60 00:03:29,434 --> 00:03:31,644 you could farm out a lot of the work that you don't want to do. 61 00:03:31,664 --> 00:03:35,224 The repetitive boilerplate, CRUD operations. 62 00:03:35,644 --> 00:03:40,924 it's all there, but like any junior developer, super smart junior developer, 63 00:03:40,954 --> 00:03:46,894 that is, they can perform some bafflingly poor, thinking and wind up with just some 64 00:03:46,904 --> 00:03:49,154 nonsense that isn't necessarily usable. 65 00:03:49,794 --> 00:03:50,644 so you gotta watch them. 66 00:03:51,644 --> 00:03:59,324 I suspect that if we were to partition the body of listeners to this, or maybe even 67 00:03:59,324 --> 00:04:05,579 developers in general, There'll be almost none in the category of I haven't heard 68 00:04:05,579 --> 00:04:07,469 of it or I haven't even played with that. 69 00:04:08,399 --> 00:04:13,139 Other than people who might be on a very long vacation, "Cast Away" style. 70 00:04:13,169 --> 00:04:15,519 I don't see how you can really escape that. 71 00:04:15,939 --> 00:04:17,649 Then you probably have a category of people. 72 00:04:17,659 --> 00:04:18,689 'Okay, I played with it. 73 00:04:18,709 --> 00:04:21,889 I went to talk to ChatGPT, spit out some code. 74 00:04:21,939 --> 00:04:26,609 I saw roughly what you can do, but I never really got much value out of that'. 75 00:04:27,309 --> 00:04:31,429 And then the category of people who actually go and use it day-to-day. 76 00:04:31,899 --> 00:04:33,849 Because it is helping their job. 77 00:04:33,869 --> 00:04:35,729 There might be some caveats to that. 78 00:04:35,759 --> 00:04:40,879 Obviously, data privacy and not knowing where the code actually goes and not 79 00:04:40,879 --> 00:04:43,869 knowing whether it's going to be trained on and that kind of stuff that can, 80 00:04:43,969 --> 00:04:46,339 throw a wrench in some people's work. 81 00:04:46,369 --> 00:04:51,114 So should we start with the category of people who might have played with 82 00:04:51,114 --> 00:04:56,144 it a little bit, and, they went to ChatGPT, asked the same questions, 83 00:04:56,154 --> 00:04:57,814 "how tall is this building?" 84 00:04:57,824 --> 00:04:59,614 And, "can you search that for me?" 85 00:05:00,084 --> 00:05:04,404 And they stalled there, from a basic development point of 86 00:05:04,414 --> 00:05:09,274 view, what kind of value, can we extract just chatting to ChatGPT. 87 00:05:09,384 --> 00:05:10,054 What can it do? 88 00:05:10,296 --> 00:05:14,856 I had a good, an interesting, experience, very early on when I was doing research 89 00:05:14,856 --> 00:05:17,876 for the book, I had it on my phone, I would carry it around and just 90 00:05:18,056 --> 00:05:22,666 periodically I would hand my phone to someone just to give them a taste. 91 00:05:23,756 --> 00:05:27,956 I remember I was at a party and a woman, she was, an expert, it was 92 00:05:27,956 --> 00:05:30,136 archeology or maybe it was art history. 93 00:05:30,656 --> 00:05:33,676 And she really, started asking me some questions. 94 00:05:33,686 --> 00:05:37,656 Some of them were factually incorrect, but over the course of her conversation 95 00:05:37,656 --> 00:05:43,866 with ChatGPT, she became really impressed with the accuracy and justification for 96 00:05:43,866 --> 00:05:45,216 some of the answers it was providing. 97 00:05:45,506 --> 00:05:50,926 She would say like, 'why did you refer to this as the oldest, example of this 98 00:05:51,526 --> 00:05:55,826 architecture or painting style?' And it gave her a fairly convincing reason. 99 00:05:56,846 --> 00:06:01,309 I think there's value in, exploring this technology, even if you're not 100 00:06:01,309 --> 00:06:05,999 going to use it in your everyday development effort, just because it 101 00:06:05,999 --> 00:06:07,609 gives you a sense for of the future. 102 00:06:07,669 --> 00:06:11,959 things are going to dramatically change, there are implications that are 103 00:06:12,579 --> 00:06:15,009 rippling all throughout academia now. 104 00:06:15,639 --> 00:06:21,929 I feel that you just should keep yourself informed, about what's coming and where, 105 00:06:21,979 --> 00:06:26,459 these changes are going to be made and how it could potentially affect you. 106 00:06:27,459 --> 00:06:32,379 So from the point of actually going and using that, the way that, you described 107 00:06:32,379 --> 00:06:37,469 in your book, what's the experience like at this very lowest level of just 108 00:06:38,349 --> 00:06:40,739 launching ChatGPT, asking some questions. 109 00:06:40,739 --> 00:06:45,669 Because I remember doing a, a while back, it would spit out some code and I had to 110 00:06:45,669 --> 00:06:48,159 copy, paste it, and I had to add imports. 111 00:06:48,209 --> 00:06:49,739 how good is it right now? 112 00:06:49,789 --> 00:06:52,659 you're fresh of writing chapters about that. 113 00:06:53,194 --> 00:06:54,304 How useful is it? 114 00:06:54,654 --> 00:06:58,794 it's going to, it largely depends on if you're going, 3, 3.5 or 4. 115 00:06:59,134 --> 00:07:07,339 4 is much better, exponentially better than 3.5, but, it's solid, I would say. 116 00:07:07,339 --> 00:07:08,369 there's a lot of caveats. 117 00:07:08,659 --> 00:07:14,249 Most likely going to be looking at code that is one to two years old. 118 00:07:14,609 --> 00:07:18,609 so it was trained on data that was, potentially the old version of a library. 119 00:07:18,979 --> 00:07:20,549 they may have had breaking changes. 120 00:07:20,659 --> 00:07:25,569 especially if there's a fast moving language, like I, I was trying 121 00:07:25,569 --> 00:07:29,509 to build something at Rust and I just, I asked ChatGPT to generate 122 00:07:29,509 --> 00:07:32,389 some code and it wouldn't even compile with the newest compiler. 123 00:07:33,699 --> 00:07:36,759 It's good, I would say, but it's not perfect. 124 00:07:36,809 --> 00:07:38,109 It's a long way from perfect. 125 00:07:39,474 --> 00:07:42,974 What are some remarkable limitations that you bumped into? 126 00:07:42,974 --> 00:07:46,074 You must have seen some interesting, funny stuff. 127 00:07:46,254 --> 00:07:47,364 Can you share some of that? 128 00:07:48,166 --> 00:07:51,516 I haven't seen really outrageous things where it was just making up, 129 00:07:52,046 --> 00:07:53,906 libraries or frameworks or anything. 130 00:07:53,976 --> 00:07:57,146 it's, I would say it's unremarkable in its banality. 131 00:07:57,706 --> 00:08:01,956 the Rust example was probably the best one, but it wasn't great. 132 00:08:01,957 --> 00:08:04,917 I wish I had something funny, just jump ahead a little bit. 133 00:08:04,917 --> 00:08:09,197 I had a really hard time in the testing chapter trying to get it to 134 00:08:09,197 --> 00:08:14,537 write good tests and maybe You know, I don't even remember if I mentioned it 135 00:08:14,537 --> 00:08:17,447 in the book, but at one point I just gave up and wrote the test myself. 136 00:08:18,067 --> 00:08:22,577 because it was very hard to get it to understand what was the unit under test. 137 00:08:22,637 --> 00:08:25,017 what was I actually trying to accomplish with my test. 138 00:08:25,357 --> 00:08:29,037 No matter how much context I added, it was always trying to do 139 00:08:29,037 --> 00:08:31,037 something just completely different. 140 00:08:31,737 --> 00:08:34,227 when I was reading that chapter, I was also thinking at the 141 00:08:34,237 --> 00:08:37,997 back of my head, "What does it say about the training data?" 142 00:08:38,647 --> 00:08:39,797 The tests are so poor. 143 00:08:39,797 --> 00:08:39,997 yeah. 144 00:08:39,997 --> 00:08:40,276 Yeah. 145 00:08:40,317 --> 00:08:43,427 Are all those tests just so poorly written that, that's where I end up? 146 00:08:43,437 --> 00:08:45,757 But yeah, let's touch on that in a sec 147 00:08:45,897 --> 00:08:46,607 who needs tests? 148 00:08:47,067 --> 00:08:48,047 We'll test it in production. 149 00:08:48,047 --> 00:08:48,707 It'll be fine. 150 00:08:48,892 --> 00:08:51,592 There you go, testing in production, everybody. 151 00:08:51,812 --> 00:08:52,522 yeah, don't worry. 152 00:08:52,522 --> 00:08:53,442 We can cut that out. 153 00:08:54,317 --> 00:08:55,027 that was a joke. 154 00:08:55,027 --> 00:08:57,647 the most annoying bit was just that you have to chat. 155 00:08:57,757 --> 00:09:02,317 So then obviously you've got things like Copilot that you also cover in your book 156 00:09:02,337 --> 00:09:04,947 or just plugs in your VS code or whatever. 157 00:09:05,747 --> 00:09:06,047 What's. 158 00:09:06,807 --> 00:09:08,467 The added value of that. 159 00:09:08,527 --> 00:09:12,857 Is it just that you can work as auto completion and it's more syntaxic 160 00:09:12,877 --> 00:09:14,157 and you don't have to copy the code. 161 00:09:14,317 --> 00:09:16,307 Do you get any other bonuses out of that? 162 00:09:17,322 --> 00:09:24,292 the real value to a tool like, Copilot, again, versus ChatGPT is it 163 00:09:24,292 --> 00:09:28,362 does keep you in the IDE and it can keep you in that flow state, where 164 00:09:28,362 --> 00:09:29,682 it's only you and the code, right? 165 00:09:29,942 --> 00:09:33,422 Whereas you're not having to pull yourself out of the context, 166 00:09:33,452 --> 00:09:34,712 move to a different window. 167 00:09:35,192 --> 00:09:35,602 and. 168 00:09:36,077 --> 00:09:40,727 for certain projects, the actual code quality for Copilot was better. 169 00:09:40,727 --> 00:09:44,287 just the, on a line by line basis or, class by class basis. 170 00:09:44,927 --> 00:09:47,927 that's almost certainly due to the fact that it was fine 171 00:09:47,927 --> 00:09:49,547 tuned specifically for code. 172 00:09:49,967 --> 00:09:53,927 that's the main benefit I've found It's always adding helpful suggestions, 173 00:09:54,187 --> 00:09:56,347 sometimes not so helpful too suggestions. 174 00:09:56,377 --> 00:09:59,657 Like I don't need it to add a comment about the name of the file that I'm 175 00:09:59,657 --> 00:10:04,337 working on, but, if I can start to define a method and then it gives me a 176 00:10:04,367 --> 00:10:08,907 possible implementation, even if I don't accept it, it's at least, showing me one 177 00:10:08,907 --> 00:10:11,197 possible implementation that I could use. 178 00:10:11,297 --> 00:10:15,177 maybe it's not the exact one, the one I wanted, but having that suggestion 179 00:10:15,207 --> 00:10:19,507 can be very valuable to clarify my thinking or to even, change it. 180 00:10:19,567 --> 00:10:21,777 Maybe it's a better implementation than I was thinking of. 181 00:10:23,117 --> 00:10:25,127 So that's, those are the major advantages I found. 182 00:10:25,597 --> 00:10:31,487 It always works very well in demos where you've got, the usual suspect 183 00:10:31,527 --> 00:10:36,017 and HTTP server in a popular framework in a popular language and 184 00:10:36,767 --> 00:10:40,877 you do something that has been done to death million times on Github. 185 00:10:40,987 --> 00:10:43,777 how well does it work with custom code base? 186 00:10:43,907 --> 00:10:48,837 Oftentimes you find yourself in a situation when your company has a 187 00:10:48,837 --> 00:10:54,187 decent or a large amount of code libraries, stuff that obviously 188 00:10:54,187 --> 00:10:58,527 wasn't trained on, because it's not in public domain, it's not on GitHub. 189 00:10:59,477 --> 00:11:01,877 how well does it work with this kind of situations? 190 00:11:01,927 --> 00:11:04,462 We'll face some challenges there If you're working in a very niche 191 00:11:04,482 --> 00:11:07,752 problem, for example, you're trying to write, an API gateway or something. 192 00:11:07,972 --> 00:11:11,252 I suppose there's probably a good open source examples out there, but if you're 193 00:11:11,252 --> 00:11:15,352 working in a, a fairly niche industry, everything is going to be closed source. 194 00:11:15,602 --> 00:11:18,862 you'll probably struggle, with, it's suggestions. 195 00:11:18,932 --> 00:11:21,012 I don't think they're going to be particularly helpful. 196 00:11:21,582 --> 00:11:28,252 although it's going to try and, in that trying, maybe it does inspire you. 197 00:11:28,642 --> 00:11:31,562 maybe it does give you one possible implementation. 198 00:11:31,732 --> 00:11:34,312 it's really good at just generating something. 199 00:11:34,842 --> 00:11:38,187 and if nothing else, It can help you plan your approach. 200 00:11:38,237 --> 00:11:42,627 and you could ask it questions in line and have it answer, one of the more 201 00:11:42,627 --> 00:11:46,387 interesting things that I found as I was working with the Copilot specifically, 202 00:11:46,427 --> 00:11:49,807 one of the almost magical things is you type in a question in a comment 203 00:11:49,807 --> 00:11:52,697 and then suddenly you prompt it for an answer and it'll give you one. 204 00:11:53,517 --> 00:11:54,947 You're like, that's fairly interesting. 205 00:11:54,947 --> 00:12:02,001 I wouldn't have thought, to take that tack, but then I used it over and over 206 00:12:02,060 --> 00:12:02,260 yeah. 207 00:12:02,260 --> 00:12:07,920 there are this moments of magic and a famous quote about 208 00:12:08,470 --> 00:12:13,570 sufficiently advanced technology being indistinguishable from magic. 209 00:12:14,390 --> 00:12:14,640 Yeah. 210 00:12:14,640 --> 00:12:15,600 People get that a lot. 211 00:12:15,600 --> 00:12:17,260 And I agree with that. 212 00:12:18,270 --> 00:12:19,400 but how does it actually work? 213 00:12:19,400 --> 00:12:24,070 So let's say I've got my VS code open and I've got some code and it's got 214 00:12:24,070 --> 00:12:26,450 some imports and existing code base. 215 00:12:27,730 --> 00:12:33,100 does it upload the whole thing to OpenAI, to be able to generate the useful things? 216 00:12:33,180 --> 00:12:37,630 What's the context that OpenAI ends up having somewhere in the training data? 217 00:12:38,183 --> 00:12:43,513 yeah, it is going to encode, the context, which will be result in a good 218 00:12:43,513 --> 00:12:45,183 portion of that code being uploaded. 219 00:12:45,233 --> 00:12:49,933 OpenAI promises the pinky swear that, it's not being saved anywhere. 220 00:12:50,393 --> 00:12:53,773 I don't think we have any way to assess whether that's true or 221 00:12:53,793 --> 00:12:58,763 not to delve into conspiratorial thinking, but, you should be careful. 222 00:12:58,873 --> 00:13:03,083 certainly if you're working on proprietary software but that's why there are 223 00:13:03,083 --> 00:13:07,243 other alternatives that are entirely offline, that you can delve into if 224 00:13:07,243 --> 00:13:09,173 you're really very privacy concerned. 225 00:13:09,913 --> 00:13:11,963 cause yeah, frankly, we don't know how it's being used 226 00:13:12,423 --> 00:13:13,733 once it leaves our machine. 227 00:13:14,785 --> 00:13:18,645 we'll definitely touch base, on LLAMA and, other alternatives that 228 00:13:18,645 --> 00:13:19,875 you discuss in your book, but. 229 00:13:20,540 --> 00:13:25,880 is there a way to control or at least tell it, 'okay, only upload this, 230 00:13:25,930 --> 00:13:28,880 folder', or is it just fully automatic? 231 00:13:28,880 --> 00:13:30,970 It just decides by itself what it sends. 232 00:13:31,075 --> 00:13:34,120 you can tell it, but is it going to honor that? 233 00:13:34,340 --> 00:13:35,880 I don't really have a good answer. 234 00:13:35,928 --> 00:13:36,258 Okay. 235 00:13:36,328 --> 00:13:36,958 So something to check 236 00:13:37,073 --> 00:13:37,383 yeah. 237 00:13:38,193 --> 00:13:38,943 Something to check. 238 00:13:39,955 --> 00:13:45,245 for anybody who's now browsing manning.com, there is a live version 239 00:13:45,255 --> 00:13:47,345 where you can see elements of the book. 240 00:13:48,015 --> 00:13:51,105 Figure 2.18 is a nice summary. 241 00:13:51,145 --> 00:13:56,575 There is a bunch of figures, there is circle for unsupported, triangle for 242 00:13:56,575 --> 00:14:02,655 supported, and a square for exclusively is comparing ChatGPT, just being used 243 00:14:02,655 --> 00:14:04,925 by itself to Copilot and CodeWhisperer. 244 00:14:05,445 --> 00:14:08,645 and it's summarizing whether it can generate methods, classes, 245 00:14:09,290 --> 00:14:13,210 projects, generate documentation, switch languages and stuff like that. 246 00:14:13,210 --> 00:14:16,310 So for anybody who wants to delve a little bit more into the 247 00:14:16,310 --> 00:14:18,460 details, I think that's very handy. 248 00:14:20,350 --> 00:14:25,860 For anybody who might be beyond that, so they went to ChatGPT, they 249 00:14:25,900 --> 00:14:30,370 spoke to it, and they got some code, and it was a generally pleasant 250 00:14:30,370 --> 00:14:32,050 experience, and they want more. 251 00:14:33,200 --> 00:14:34,070 The use Copilot. 252 00:14:34,440 --> 00:14:36,530 what's the next kind of checkpoint? 253 00:14:36,850 --> 00:14:38,700 Where do they go from there? 254 00:14:38,910 --> 00:14:42,640 How do they start designing software a bit, higher level 255 00:14:42,640 --> 00:14:43,990 than just snippets of code? 256 00:14:44,343 --> 00:14:46,293 How useful is the AI in here? 257 00:14:46,508 --> 00:14:51,778 for example, I was designing something yesterday and I was working through 258 00:14:51,958 --> 00:14:58,718 a conversation with ChatGPT that is one of the key things that the ChatGPT 259 00:14:58,738 --> 00:15:03,088 excels at is helping you to design the software to really underscore that. 260 00:15:03,188 --> 00:15:05,708 not just to design your application. 261 00:15:05,728 --> 00:15:09,518 not just, lines of code, but it's perfectly capable of that. 262 00:15:09,558 --> 00:15:13,418 not even just the classes, but like here's the patterns that you want to apply. 263 00:15:13,428 --> 00:15:14,598 Here's the architecture. 264 00:15:15,028 --> 00:15:18,618 have it generate some of the documents in a text format. 265 00:15:18,618 --> 00:15:21,998 So plant UML or mermaid, like those are really. 266 00:15:22,968 --> 00:15:27,088 What's those are really good, useful things, because then you can always 267 00:15:27,088 --> 00:15:32,808 take those, save those and pass them back to ChatGPT, to refresh the context. 268 00:15:33,348 --> 00:15:37,868 so yeah, As a co founder of, and the CTO of a startup, I found it 269 00:15:37,868 --> 00:15:41,708 really invaluable, as a partner to help me design that software. 270 00:15:41,908 --> 00:15:45,953 I think one of the things that really opened my eyes was that I never thought 271 00:15:45,999 --> 00:15:51,262 to talk to ChatGPT about open source alternatives, and maybe trying to 272 00:15:51,262 --> 00:15:54,571 select a database and talking about the different properties, like it 273 00:15:54,571 --> 00:15:59,999 was just second nature for me to open the different docs and just start 274 00:16:00,026 --> 00:16:02,044 comparing features and stuff like that. 275 00:16:02,053 --> 00:16:07,113 And it never occurred to me that I can just go and ask ChatGPT because it's 276 00:16:07,113 --> 00:16:09,013 got quite a lot of knowledge about that. 277 00:16:09,518 --> 00:16:09,918 Yeah. 278 00:16:10,188 --> 00:16:12,928 think in the book you're talking about, open source alternatives 279 00:16:12,928 --> 00:16:14,428 to what you're writing, which is 280 00:16:14,498 --> 00:16:20,028 an IT asset management, system, Actually, I don't know if this part's going to work. 281 00:16:20,128 --> 00:16:22,348 just, so just be aware or just be advised. 282 00:16:22,798 --> 00:16:24,958 I got a lot of feedback that it was really boring, right? 283 00:16:24,958 --> 00:16:27,598 That people didn't like the actual project that you have 284 00:16:27,598 --> 00:16:29,328 to work on throughout the book. 285 00:16:29,588 --> 00:16:34,458 But I wanted it to be like a boring book on a boring topic, boring, 286 00:16:34,928 --> 00:16:39,048 application, because most of what we write is not interesting. 287 00:16:39,058 --> 00:16:42,748 It's we pick up data and we shuffle it and we move it around, right? 288 00:16:43,108 --> 00:16:44,668 A lot of what we do is not exciting. 289 00:16:45,038 --> 00:16:46,318 it was definitely intentional. 290 00:16:46,518 --> 00:16:50,078 but, again, maybe something to fix in this in a second edition, if it's coming. 291 00:16:50,578 --> 00:16:56,013 but one of the more interesting things about my engagement model with these 292 00:16:56,013 --> 00:17:00,683 tools as I worked with them, to pick up on what you were saying about learning 293 00:17:00,683 --> 00:17:05,343 more about a database or having it, help it select database or selecting 294 00:17:05,343 --> 00:17:08,913 open source projects, Is very early on. 295 00:17:08,943 --> 00:17:11,243 I was being extremely prescriptive. 296 00:17:11,283 --> 00:17:16,933 I would say, create, software that's using this library in this framework 297 00:17:17,293 --> 00:17:19,873 and, this language and all of that. 298 00:17:20,183 --> 00:17:25,823 but later on, and even to this day, when I have a problem, I 299 00:17:25,823 --> 00:17:27,593 feed in the business requirements. 300 00:17:27,593 --> 00:17:30,133 And then I ask it to make recommendations for me. 301 00:17:30,673 --> 00:17:31,873 and then I can assess those. 302 00:17:32,123 --> 00:17:34,253 but at the very least it starts the process. 303 00:17:34,253 --> 00:17:35,571 it gets it going. 304 00:17:35,991 --> 00:17:39,751 so hopefully, that answered your question or was at least in the neighborhood. 305 00:17:40,681 --> 00:17:43,581 Yeah, definitely in the neighborhood, same district. 306 00:17:44,181 --> 00:17:44,851 Same zip code. 307 00:17:44,851 --> 00:17:45,231 Yeah. 308 00:17:45,966 --> 00:17:54,020 the way my mind works is that when I hear the idea of a free, junior available 24/7 309 00:17:54,575 --> 00:18:00,629 my mind wanders to things like already mentioned docs, We hinted at tests coming 310 00:18:00,629 --> 00:18:04,675 a bit later, but I think one of the things that are painful in more than one 311 00:18:04,675 --> 00:18:10,602 way and people never want to do them is refactoring and upgrading to a new version 312 00:18:10,602 --> 00:18:15,184 of something or maybe changing language, which is surprisingly labor-intensive. 313 00:18:15,248 --> 00:18:18,511 It always ends up being more work than it looked initially. 314 00:18:19,311 --> 00:18:22,720 How good, is the AI at the moment in this kind of things? 315 00:18:22,738 --> 00:18:26,629 Refactor, rewriting in a different language, upgrade a library. 316 00:18:26,629 --> 00:18:27,238 Can you just say: 317 00:18:27,248 --> 00:18:30,068 ' Hey, this is a library with a breaking change. 318 00:18:30,588 --> 00:18:33,968 Give me the new version of the library and updated tests and everything'? 319 00:18:35,033 --> 00:18:40,053 if it wasn't, it's post breaking change was in the training 320 00:18:40,053 --> 00:18:41,403 data, you should be fine. 321 00:18:41,893 --> 00:18:45,243 if not, you're going to have a more involved conversation. 322 00:18:46,133 --> 00:18:52,833 but more generally, it does really well in translating from one language to another, 323 00:18:53,323 --> 00:18:54,733 specifically programming languages. 324 00:18:55,053 --> 00:18:55,503 I couldn't. 325 00:18:55,923 --> 00:18:59,603 assess its, quality of English to French or something like that. 326 00:18:59,603 --> 00:19:04,773 But I can tell you, there was a few examples where I was working in Python 327 00:19:05,073 --> 00:19:07,873 and then I said, 'Oh, what would this look like in Go?' And it gave me 328 00:19:08,243 --> 00:19:11,069 just a literal translation into Go. 329 00:19:11,069 --> 00:19:15,543 And I was like, this doesn't feel very idiomatic, make it idiomatic 330 00:19:15,673 --> 00:19:19,123 And it would be, as good or better than I would have written it myself. 331 00:19:19,713 --> 00:19:23,843 so it does surprisingly well in going from one language to another. 332 00:19:24,253 --> 00:19:29,683 it can, and then on to refactoring you can ask it for certain patterns that 333 00:19:29,683 --> 00:19:34,173 you may want to apply as you refactor, different design schemes, like Maybe 334 00:19:34,173 --> 00:19:35,843 I need to pull out an interface. 335 00:19:35,873 --> 00:19:38,683 Maybe I need to, some kind of like parent class. 336 00:19:39,333 --> 00:19:44,413 maybe this needs to be an adapter or you take your pick from the gang, the gang 337 00:19:44,413 --> 00:19:50,483 of four, and it's, it knows them and they can provide examples in any language, that 338 00:19:50,483 --> 00:19:52,063 you can think of that it was trained on. 339 00:19:52,063 --> 00:19:56,163 so it can take a lot of way, a lot of that drudgery away and 340 00:19:56,163 --> 00:19:57,493 a lot of that anxiety away. 341 00:19:58,643 --> 00:20:02,943 One of the most important benefits that we can derive from at least their 342 00:20:02,943 --> 00:20:08,803 current implementation of these tools and of genAI is to just keep us going, 343 00:20:08,893 --> 00:20:13,723 to keep us motivated, to keep us engaged, to keep us building software. 344 00:20:14,073 --> 00:20:19,403 it, it can be really mentally taxing and this can help ease some of that, 345 00:20:19,433 --> 00:20:23,463 intellectual heavy lifting, not that we should just suborn our thinking 346 00:20:23,463 --> 00:20:26,353 to it entirely, but it can help. 347 00:20:27,408 --> 00:20:32,358 Did you notice any discrepancies between quality, in different languages? 348 00:20:32,838 --> 00:20:36,898 Because what I'm picturing is that the body of training data 349 00:20:37,578 --> 00:20:39,368 came from somewhere like GitHub, 350 00:20:39,428 --> 00:20:39,968 probably. 351 00:20:40,608 --> 00:20:43,618 And if you look at GitHub, there's going to be a disproportionate 352 00:20:43,688 --> 00:20:48,283 amount of JavaScript of questionable quality too, but, you're going to 353 00:20:48,293 --> 00:20:53,493 have probably increasing and quite significant amount of Go as well. 354 00:20:53,543 --> 00:20:55,323 but you might not have too much, 355 00:20:55,893 --> 00:20:56,163 I don't 356 00:20:56,163 --> 00:21:00,258 Haskell Yeah, Haskell, 357 00:21:00,408 --> 00:21:02,058 I don't know, SQL, whatever it is. 358 00:21:02,158 --> 00:21:04,558 did you notice anything funny about that? 359 00:21:05,188 --> 00:21:06,198 I would say that is. 360 00:21:06,198 --> 00:21:09,988 roughly in line with what I observed, and I didn't, I wouldn't necessarily have deep 361 00:21:10,318 --> 00:21:13,888 dug, too deep into, very niche languages. 362 00:21:13,988 --> 00:21:18,028 but, definitely the examples that you're gonna find in, if you're working in 363 00:21:18,028 --> 00:21:21,398 Python, Go, JavaScript, or TypeScript. 364 00:21:21,728 --> 00:21:26,368 like those are going to be more voluminous and likely higher quality. 365 00:21:26,808 --> 00:21:31,868 the one time I tried to use it to write, Rust, it failed spectacularly. 366 00:21:31,908 --> 00:21:32,538 It was beautiful. 367 00:21:32,538 --> 00:21:33,148 It was glorious. 368 00:21:33,508 --> 00:21:36,803 I was trying to, throw together an API gateway. 369 00:21:36,963 --> 00:21:41,603 Just see how, just how difficult was this going to be. 370 00:21:42,033 --> 00:21:44,683 and in Rust, I wanted something high-performance. 371 00:21:45,833 --> 00:21:49,853 And I just, I asked it to start writing some code and it created a number of 372 00:21:49,863 --> 00:21:55,563 files and it just, none of it works well together and it wouldn't compile. 373 00:21:55,633 --> 00:21:59,683 Although it's Rust, so it would take a while to convince the 374 00:21:59,683 --> 00:22:01,463 compiler that it's good enough. 375 00:22:02,013 --> 00:22:04,513 but yeah, it was, not the most pleasant experience. 376 00:22:06,208 --> 00:22:11,148 But also, to be fair, at the time, I only spent a few hours learning 377 00:22:11,148 --> 00:22:14,718 the basic syntax of Rust, so I don't know really what I was expecting. 378 00:22:14,878 --> 00:22:17,628 So was it ChatGPT, or was it me, or was it a mixture of both? 379 00:22:17,628 --> 00:22:19,248 probably the latter. 380 00:22:19,248 --> 00:22:22,728 Yeah, I think we all occasionally bump into those weird restrictions 381 00:22:23,233 --> 00:22:25,003 based on the training data. 382 00:22:25,003 --> 00:22:29,503 One that I keep remembering was when I wanted Midjourney to generate 383 00:22:29,513 --> 00:22:31,933 for me a picture of Triceratops. 384 00:22:32,743 --> 00:22:35,563 And it would give me any other dinosaur when I was asking 385 00:22:35,563 --> 00:22:36,813 for it, but not this one. 386 00:22:37,173 --> 00:22:38,693 It was all T-Rex and T-Rex. 387 00:22:38,863 --> 00:22:43,973 then I started throwing random names, give me Brontosaurus, and 388 00:22:43,993 --> 00:22:46,113 it just gave me a Brontosaurus. 389 00:22:46,143 --> 00:22:49,133 So I was very upset at the time, I made peace with that. 390 00:22:49,193 --> 00:22:52,513 And there are some things that just weren't in the training set and 391 00:22:53,173 --> 00:22:54,743 they didn't emerge from training. 392 00:22:56,243 --> 00:22:57,403 come on, a triceratops? 393 00:22:58,228 --> 00:22:58,938 They're like the best. 394 00:22:59,163 --> 00:22:59,333 Yeah. 395 00:22:59,333 --> 00:23:00,313 You would think so, right? 396 00:23:00,345 --> 00:23:00,965 Very weird. 397 00:23:01,025 --> 00:23:01,385 Yeah. 398 00:23:01,475 --> 00:23:05,205 And about listening about this from midjourney, this still needs fixing. 399 00:23:05,645 --> 00:23:08,975 This is months later and you still can't get a decent triceratops. 400 00:23:09,393 --> 00:23:13,253 I was using DALL-E and I asked her for a pug, a Pegacorn. 401 00:23:13,253 --> 00:23:15,443 So that's a pug, a unicorn and a Pegasus. 402 00:23:15,873 --> 00:23:19,113 And I got a pretty good one, pretty good representation. 403 00:23:19,533 --> 00:23:20,803 And then I said, make it cute. 404 00:23:20,843 --> 00:23:23,093 And it was the most adorable thing I've ever seen. 405 00:23:23,243 --> 00:23:23,593 Wow. 406 00:23:23,888 --> 00:23:24,568 but I. 407 00:23:25,028 --> 00:23:26,868 Did not try a triceratops 408 00:23:26,898 --> 00:23:28,248 I know what I'm going to do after this. 409 00:23:28,553 --> 00:23:32,483 Yes, I encourage everyone to go create their own Pug-a-peg-a-corn 410 00:23:32,503 --> 00:23:33,343 exactly. 411 00:23:34,683 --> 00:23:36,483 Let's move to testing software. 412 00:23:37,043 --> 00:23:41,173 so you already said a little bit about how difficult it actually is. 413 00:23:42,603 --> 00:23:44,963 you give a more concrete example? 414 00:23:45,033 --> 00:23:48,863 what is wrong with the test is generating some of the time? 415 00:23:49,178 --> 00:23:53,378 in this case it was really struggling to Figure out what I was actually 416 00:23:53,378 --> 00:23:54,628 trying to do with the test. 417 00:23:54,918 --> 00:23:56,438 specifically, it was an integration test. 418 00:23:56,438 --> 00:24:01,608 And so I was trying to go mostly end to end in terms of, serving data over Rust. 419 00:24:01,878 --> 00:24:04,988 it was missing the point largely of the actual test. 420 00:24:05,118 --> 00:24:06,478 which was very strange. 421 00:24:06,578 --> 00:24:08,708 it was in Python, so there should've been a number of 422 00:24:08,708 --> 00:24:09,998 instances in the training data. 423 00:24:10,418 --> 00:24:11,258 to cover this. 424 00:24:11,278 --> 00:24:16,188 Did you just say, I want an integration tests, test everything, or did you 425 00:24:16,188 --> 00:24:22,398 describe more, end to end I would like the data to flow through the whole thing? 426 00:24:22,613 --> 00:24:25,613 yeah, I felt it was fairly, comprehensive. 427 00:24:25,673 --> 00:24:30,273 I think at that point it's, it was, specifically the test was, Copilot or 428 00:24:30,303 --> 00:24:35,313 I was having Copilot write the test and I believe I even went to ChatGPT and 429 00:24:35,333 --> 00:24:41,053 asked it, 'how would I write a prompt to get Copilot to do an integration 430 00:24:41,053 --> 00:24:43,963 test, end-to-end test for fast API. 431 00:24:44,523 --> 00:24:46,573 and the payload would look like this. 432 00:24:47,963 --> 00:24:51,583 I eventually started having ChatGPT write my prompts for me, 433 00:24:52,153 --> 00:24:53,493 which it did surprisingly well. 434 00:24:54,943 --> 00:24:55,583 And it's meta. 435 00:24:55,763 --> 00:24:56,633 that's very meta. 436 00:24:57,633 --> 00:24:58,173 Okay. 437 00:24:58,843 --> 00:25:02,083 Was there a really good use case in terms of testing? 438 00:25:02,143 --> 00:25:06,433 Unit test it was perfectly fine at, even some cases where I felt 439 00:25:06,483 --> 00:25:08,373 it should have gotten stuck. 440 00:25:08,653 --> 00:25:13,563 so I had a number of, again, not to get too specific, about the actual, the ITAM, 441 00:25:13,573 --> 00:25:17,753 the IT asset management project that is all throughout the corpus of the book, 442 00:25:17,853 --> 00:25:24,083 there's a number of, in accounting, assets depreciated at a certain rate and general 443 00:25:24,113 --> 00:25:27,413 accepted accounting practices outlines a few different ways that you can do it. 444 00:25:27,933 --> 00:25:31,893 and so I, I used a strategy pattern and I had a number of different 445 00:25:31,893 --> 00:25:36,553 ways, that each of the two, to calculate that depreciation. 446 00:25:36,633 --> 00:25:38,633 So the depreciation of the asset, maybe it's straight line. 447 00:25:38,633 --> 00:25:40,343 So it's like over five years. 448 00:25:40,363 --> 00:25:44,323 So one fifth of the value is lost every year and you can write part of that off, 449 00:25:44,333 --> 00:25:49,103 but again, I'm not an accountant, so this does not count as, financial advice, but, 450 00:25:49,678 --> 00:25:49,858 or 451 00:25:50,263 --> 00:25:56,313 or medical, yeah, I'm not a doctor, but it works surprisingly 452 00:25:56,313 --> 00:25:58,273 well, I was pleasantly surprised. 453 00:25:58,773 --> 00:25:59,173 Fair enough. 454 00:25:59,182 --> 00:26:01,809 So we've written some code, we've designed some software. 455 00:26:02,391 --> 00:26:04,691 Let's say that we tested it for the most part. 456 00:26:05,427 --> 00:26:08,246 but the reality of it is that we're probably going to spend more time 457 00:26:08,246 --> 00:26:10,346 reading code and understanding code. 458 00:26:10,437 --> 00:26:13,073 perhaps the code that we wrote a couple of years back. 459 00:26:13,925 --> 00:26:14,215 Yeah. 460 00:26:14,596 --> 00:26:20,404 how well does the part of describing existing code actually work at the moment? 461 00:26:20,635 --> 00:26:25,505 Yeah, it works surprisingly well into, translating the code that you wrote 462 00:26:25,525 --> 00:26:30,895 into giving it a very, simplified answers, descriptions of here's 463 00:26:30,895 --> 00:26:32,435 how it, here's how it's functioned. 464 00:26:32,485 --> 00:26:33,465 Here's how it's working. 465 00:26:33,955 --> 00:26:35,205 here's what it expects. 466 00:26:35,275 --> 00:26:38,555 You can even have it describe an entire system to you. 467 00:26:38,955 --> 00:26:44,155 I have not, I did not though attempt to do, what is probably one of the 468 00:26:44,155 --> 00:26:46,465 hardest things, within that space. 469 00:26:46,755 --> 00:26:50,885 And that is, I didn't feed it a Perl program and ask it what it actually did. 470 00:26:52,805 --> 00:26:56,525 I have this feeling it probably would have broken a ChatGPT. 471 00:26:56,545 --> 00:26:58,030 Sorry. 472 00:26:59,200 --> 00:27:00,480 Takin pot shots at Perl. 473 00:27:01,034 --> 00:27:04,124 you should have given it a regular expression in 474 00:27:04,479 --> 00:27:05,009 Oof. 475 00:27:05,344 --> 00:27:06,707 and try to see what happens. 476 00:27:06,707 --> 00:27:10,316 And then next thing you know, OpenAI's knocking at your door, kicking you out. 477 00:27:10,914 --> 00:27:11,634 That sounds about right. 478 00:27:11,634 --> 00:27:13,944 Or, T100 is just kickin in the door. 479 00:27:14,722 --> 00:27:19,577 I guess in my mind there's this limitation of the amount of context, 480 00:27:19,622 --> 00:27:21,177 length that you can feed it, right? 481 00:27:21,181 --> 00:27:25,322 So if your code base becomes significantly large. 482 00:27:25,986 --> 00:27:29,740 Is that not going to be a problem by getting, to get it to even describe it. 483 00:27:30,211 --> 00:27:32,521 to describe your entire codebase, yes. 484 00:27:32,961 --> 00:27:34,731 but you can start to chunk it up. 485 00:27:34,801 --> 00:27:39,211 you can work around that limitation by sending it only pieces. 486 00:27:39,541 --> 00:27:43,091 And you're probably not going to get the full context there but 487 00:27:43,091 --> 00:27:44,321 it can help guide your intuition. 488 00:27:44,661 --> 00:27:50,161 that's why if you have some kind of class diagram or, some architectural 489 00:27:50,161 --> 00:27:51,991 diagram that's text-based. 490 00:27:52,261 --> 00:27:57,421 so like plain UML, then you can distill your entire, code 491 00:27:57,421 --> 00:27:58,881 base into a single document. 492 00:27:59,911 --> 00:28:05,331 Now it's still, again, might be, if it's a code base of thousands of classes, 493 00:28:05,771 --> 00:28:09,101 you could still hit those limitations, but it's going to be your best bet 494 00:28:09,301 --> 00:28:15,386 to get a distillation in natural language, what your classes or what 495 00:28:15,386 --> 00:28:17,796 your code is trying to attempt to do. 496 00:28:17,796 --> 00:28:23,186 it really does excel at, method by method descriptions of what this does. 497 00:28:23,736 --> 00:28:28,976 between manual browsing through the code and trying to understand the intent and 498 00:28:29,066 --> 00:28:31,496 Jarvis, it's halfway through, right? 499 00:28:31,496 --> 00:28:36,276 It's not quite, here's the intent and here's what it imported and 500 00:28:36,326 --> 00:28:37,876 here's my recommendations, Mr. 501 00:28:37,876 --> 00:28:38,366 Stark. 502 00:28:38,386 --> 00:28:40,996 It's more here's, I can ask about this method. 503 00:28:41,466 --> 00:28:42,546 Can't be bothered to read it. 504 00:28:42,546 --> 00:28:45,346 It's 2000 lines and it can give me the gist of 505 00:28:45,401 --> 00:28:45,951 Exactly. 506 00:28:46,241 --> 00:28:46,821 Exactly. 507 00:28:49,091 --> 00:28:53,591 there's also the security aspect that, you're discussing in one of the chapters. 508 00:28:53,591 --> 00:28:55,341 Can you talk about that a little bit? 509 00:28:55,381 --> 00:28:59,711 Yeah, and actually there's, a funny, story that's embedded in that too. 510 00:29:00,111 --> 00:29:02,781 it's good at, picking up on what we were just talking about it, the non 511 00:29:02,781 --> 00:29:05,551 exclusive path, it can explain ways that your code might be exploited. 512 00:29:06,041 --> 00:29:09,861 it's not the same as having a, security expert on your team. 513 00:29:10,261 --> 00:29:11,201 it will miss things. 514 00:29:11,681 --> 00:29:14,581 but it's definitely better than nothing. 515 00:29:14,791 --> 00:29:17,971 and it can make some pretty great, recommendations in terms of, 516 00:29:18,601 --> 00:29:19,871 how you can structure your code. 517 00:29:20,751 --> 00:29:24,541 one of the funny things, I really wanted an example of, 518 00:29:24,591 --> 00:29:26,861 a SQL injection in the book. 519 00:29:27,181 --> 00:29:33,201 So I actually asked, ChatGPT to give me an example of a SQL injection. 520 00:29:34,031 --> 00:29:34,611 but it wouldn't. 521 00:29:34,611 --> 00:29:42,331 No matter how I tried to coerce it, no matter how I, No, I swear 522 00:29:42,331 --> 00:29:43,311 I'm not doing this for evil. 523 00:29:43,321 --> 00:29:45,041 This is just for illustrative purposes only. 524 00:29:45,451 --> 00:29:50,621 And, it just would not give me a valid, SQL injection exploit 525 00:29:50,711 --> 00:29:51,901 that I could include in the book. 526 00:29:51,901 --> 00:29:54,131 so do with that as you will. 527 00:29:54,181 --> 00:29:54,431 Yeah. 528 00:29:55,168 --> 00:29:58,035 that's a very interesting ethical, discussion about that. 529 00:29:58,035 --> 00:30:01,814 There's probably gonna be some way you can, I don't know if you heard about 530 00:30:01,823 --> 00:30:06,614 that exploit where, if you asked it to do something nefarious, it would say no, but 531 00:30:06,623 --> 00:30:10,906 if you asked it to do something and that something equals ASCII art of something 532 00:30:10,906 --> 00:30:13,164 nefarious, there was no problem at all. 533 00:30:13,739 --> 00:30:19,352 So I suspect, like it's very hard to Limit a model like that because there's endless 534 00:30:19,427 --> 00:30:24,227 opportunities to express it differently and you only need one of them to work. 535 00:30:24,227 --> 00:30:25,610 So an interesting one. 536 00:30:25,652 --> 00:30:28,552 So it will let you do an SQL injection even when your pinkies 537 00:30:28,552 --> 00:30:30,310 were, it was for the good, 538 00:30:30,590 --> 00:30:33,830 And I tried to do that, it was an exploit a little bit earlier on 539 00:30:33,830 --> 00:30:37,950 where you could give it a person, a persona of Dan and Dan is allowed 540 00:30:37,950 --> 00:30:39,780 to do things that ChatGPT isn't. 541 00:30:40,160 --> 00:30:42,460 And it's still, it wouldn't let me do it as, I think it 542 00:30:42,460 --> 00:30:44,590 was Dan and it was an acronym. 543 00:30:45,250 --> 00:30:49,330 but yeah, similar thing, but maybe I should have tried out ASCII art next time. 544 00:30:50,707 --> 00:30:54,967 but listeners do not intentionally put, SQL injection exploits in your code. 545 00:30:55,967 --> 00:30:57,437 oh yeah, that needed to be said. 546 00:30:58,152 --> 00:31:02,102 What are some of the examples of what he was able to, figure 547 00:31:02,102 --> 00:31:05,252 out from your code, in terms of security holes and stuff like that? 548 00:31:06,272 --> 00:31:09,262 Do you have any interesting examples of success? 549 00:31:09,812 --> 00:31:11,302 What did it actually find? 550 00:31:11,702 --> 00:31:14,052 I would have to go back and consult, the book. 551 00:31:14,532 --> 00:31:18,062 my code is just so good that there was no exploits to be made. 552 00:31:18,552 --> 00:31:19,612 No, that's not 553 00:31:19,622 --> 00:31:20,292 And there you go. 554 00:31:20,292 --> 00:31:22,592 So Nathan doesn't want to share too much 555 00:31:22,592 --> 00:31:23,242 about the book. 556 00:31:23,242 --> 00:31:27,162 You're going to have to go and buy it. 557 00:31:27,237 --> 00:31:30,367 of the, yeah, one of the things I wanted, I should have mentioned up front is, 558 00:31:30,577 --> 00:31:38,337 this was the first time, that I had ever built like a true application in Python. 559 00:31:38,397 --> 00:31:42,827 I had used it for scripts previously, just, to do something, but, text 560 00:31:42,847 --> 00:31:44,207 modification, things like that. 561 00:31:44,207 --> 00:31:46,357 But I never built an actual application. 562 00:31:46,627 --> 00:31:49,707 it actually helped me learn how to build applications while using it.. 563 00:31:49,707 --> 00:31:50,747 There's a book called. 564 00:31:50,747 --> 00:31:56,027 Octopus, my teacher, I guess for you, it's more like ChatGPT, my teacher. 565 00:31:56,027 --> 00:31:58,467 All right. 566 00:31:58,737 --> 00:31:59,247 that's good. 567 00:31:59,247 --> 00:32:03,684 we've covered, I think most of the big chunks other than 568 00:32:04,238 --> 00:32:05,402 actually running the software. 569 00:32:05,447 --> 00:32:07,984 let's say that, it runs, we package that. 570 00:32:08,584 --> 00:32:12,829 And then we've got things like Docker, Terraform, the YAML hell that comes 571 00:32:12,829 --> 00:32:18,038 with Kubernetes on one hand, I would expect that this is fairly repetitive. 572 00:32:18,075 --> 00:32:19,711 so ChatGPT would excel. 573 00:32:19,784 --> 00:32:21,447 It's not a very tricky language. 574 00:32:21,447 --> 00:32:25,993 It's just very verbose, and the white spaces make your life miserable. 575 00:32:25,993 --> 00:32:27,593 How good is it with that kind of stuff? 576 00:32:27,679 --> 00:32:32,459 it was actually really good with working out YAML and making, just different 577 00:32:32,459 --> 00:32:37,729 scripts, helping build out, dev pipelines through GitHub actions, things like that. 578 00:32:37,779 --> 00:32:39,909 it did really well. 579 00:32:40,479 --> 00:32:43,239 one of the very interesting things that I discovered though, 580 00:32:43,269 --> 00:32:45,249 was CodeWhisperer, the AWS. 581 00:32:45,249 --> 00:32:49,019 generative AI, large language model actually doesn't support 582 00:32:49,839 --> 00:32:52,289 anything but programming languages. 583 00:32:52,309 --> 00:32:56,519 So it didn't even understand, how to do like Terraform 584 00:32:57,029 --> 00:32:59,969 infrastructure as code, which you'd think it would be very good at. 585 00:33:00,399 --> 00:33:01,449 that was a bit surprising. 586 00:33:01,579 --> 00:33:03,289 but it's by design, it's intentional. 587 00:33:03,589 --> 00:33:05,089 it's hard to see it as a limitation 588 00:33:05,104 --> 00:33:05,534 Curious. 589 00:33:05,544 --> 00:33:09,254 So do they have another tool for the YAMLs of the world? 590 00:33:09,264 --> 00:33:11,784 Or they just out-of-scope'd it. 591 00:33:11,879 --> 00:33:15,139 Yeah, just outta scoped, I didn't try the, what is it, cloud, not 592 00:33:15,139 --> 00:33:20,659 CloudFront, but whatever their, their deployment based, their, code as, or 593 00:33:20,664 --> 00:33:23,059 infrastructure as code, specific thing. 594 00:33:23,064 --> 00:33:23,814 I didn't try that. 595 00:33:23,864 --> 00:33:24,644 maybe I should have. 596 00:33:24,824 --> 00:33:25,384 but, I was. 597 00:33:25,384 --> 00:33:25,884 Shocked. 598 00:33:26,664 --> 00:33:29,834 I think I even mentioned that in the book, that I had originally 599 00:33:29,834 --> 00:33:33,994 intended that chapter to be written in using CodeWhisperer. 600 00:33:34,040 --> 00:33:38,549 let's say, for example, you want a quick Docker file, you can write it. 601 00:33:38,549 --> 00:33:41,740 It's not too hard, but why do it if you can't get it for free? 602 00:33:41,740 --> 00:33:47,685 So what you open a Docker file and you write in a comment what you want it to 603 00:33:47,685 --> 00:33:51,749 do and you hit tab and the magic happens. 604 00:33:51,994 --> 00:33:53,603 That's roughly what you need to do. 605 00:33:53,823 --> 00:33:54,393 Yeah, roughly. 606 00:33:54,393 --> 00:33:57,193 add a prompt as it were in a comment. 607 00:33:57,983 --> 00:33:58,933 it doesn't have to be a comment. 608 00:33:58,963 --> 00:34:01,463 you can just add the prompt and then later delete it. 609 00:34:01,843 --> 00:34:05,263 have it generate the Docker file for you or the, the Kubernetes file. 610 00:34:05,313 --> 00:34:10,843 I don't know if I tried using, patterns in a Terraform, just 611 00:34:10,893 --> 00:34:12,463 to ease some of the repetition. 612 00:34:12,513 --> 00:34:16,963 not necessarily have, the sprawling mess that Terraform can become. 613 00:34:17,513 --> 00:34:20,233 but, I'm sure it could accommodate that as well. 614 00:34:20,263 --> 00:34:26,763 it's both, Copilot and ChatGPT did seem to have extensive knowledge 615 00:34:26,793 --> 00:34:29,213 of Terraform syntax and features. 616 00:34:30,213 --> 00:34:36,013 So that all together adds up to a pretty competent, junior developer, like you 617 00:34:36,013 --> 00:34:39,743 described it at the beginning that you need to supervise, but it can do a lot of 618 00:34:39,743 --> 00:34:43,363 the legwork for you and much faster too. 619 00:34:44,463 --> 00:34:51,243 did you follow Devin, the supposedly first AI-driven coworker, 620 00:34:51,243 --> 00:34:52,583 no, that's interesting. 621 00:34:52,583 --> 00:34:53,263 Tell me more. 622 00:34:53,886 --> 00:34:56,876 it was a few weeks ago, they made this big announcement. 623 00:34:56,896 --> 00:35:00,666 There was a video with a demo showing basically doing the 624 00:35:00,786 --> 00:35:01,976 whole thing from scratch. 625 00:35:02,016 --> 00:35:07,166 So not only did it do the Copilot stuff, but it bootstrapped the whole project, 626 00:35:07,166 --> 00:35:12,336 generated all the files and had, like a browser window that had access to as well. 627 00:35:12,646 --> 00:35:14,626 to actually go and, verify that it works. 628 00:35:14,626 --> 00:35:17,906 Obviously, it was doing something that it always does in these demos, which 629 00:35:17,916 --> 00:35:21,166 is, an HTTP server with a REST API. 630 00:35:21,956 --> 00:35:23,226 Which is cheating if you ask me. 631 00:35:23,406 --> 00:35:28,686 a lot of people were very impressed and there was a lot of, angst among people 632 00:35:28,916 --> 00:35:33,526 on the internet arguing over whether this is the end of software engineering 633 00:35:33,536 --> 00:35:35,896 as we know it, or whether it's a scam. 634 00:35:35,896 --> 00:35:41,416 And then, a few days ago, there was a critique that resurfaced about Devin and 635 00:35:41,416 --> 00:35:46,246 that entire project and whether he was, A little bit polished up in the demo, 636 00:35:46,258 --> 00:35:48,798 In other words, just a typical software demo. 637 00:35:49,458 --> 00:35:51,408 Yeah, that's just typical software demo. 638 00:35:52,118 --> 00:35:56,608 So I think we have a similar problem to maybe different stakes, but to 639 00:35:56,618 --> 00:36:02,378 self driving cars that it can't be like 95% good without supervision has 640 00:36:02,378 --> 00:36:04,998 to be like, I don't know, 99% good. 641 00:36:05,098 --> 00:36:10,328 before we can live it with our supervision and sure, a badly written API. 642 00:36:10,603 --> 00:36:16,203 Probably most of the time it's not gonna get anybody killed, fingers crossed, but, 643 00:36:16,383 --> 00:36:18,633 you still need that supervision, right? 644 00:36:18,663 --> 00:36:20,923 and Devin was supposed to do away with that. 645 00:36:20,923 --> 00:36:24,853 So I'm looking forward to seeing how that story develops and 646 00:36:24,853 --> 00:36:26,303 how they answer the critique. 647 00:36:26,303 --> 00:36:29,013 And I guess when people can actually go and play it, we'll 648 00:36:29,013 --> 00:36:30,881 find out whether, was all fluff. 649 00:36:31,055 --> 00:36:31,515 it's another 650 00:36:31,666 --> 00:36:33,526 Yeah, no, that's interesting. 651 00:36:33,586 --> 00:36:38,356 I have been following, there was a story not long ago, that, because of 652 00:36:38,356 --> 00:36:43,356 the proliferation of, ChatGPT and, and Copilot and alike that the software 653 00:36:43,356 --> 00:36:44,806 has been getting less and less secure. 654 00:36:45,286 --> 00:36:51,895 and, Because, it is easy for, bugs to introduce themselves if you're 655 00:36:51,905 --> 00:36:53,325 really just copying and pasting. 656 00:36:53,605 --> 00:36:56,785 So that's why, we're not at that stage yet. 657 00:36:56,785 --> 00:36:57,645 We may never be. 658 00:36:57,865 --> 00:37:00,625 Where it's just, there's no human in the loop, right? 659 00:37:01,555 --> 00:37:03,935 For these things that it can just generate code on its own 660 00:37:03,935 --> 00:37:05,705 and, push it to production. 661 00:37:05,965 --> 00:37:11,605 the role of a professional developer is here to stay for the foreseeable future. 662 00:37:11,825 --> 00:37:13,485 We're just going to be better at what we do. 663 00:37:13,875 --> 00:37:17,555 again, if we're mindful and not allowing these bugs to just creep in. 664 00:37:17,555 --> 00:37:22,155 I like the optimistic point of view here, but yeah, a lot of people 665 00:37:22,195 --> 00:37:25,245 I think would agree with you that this is like it was going to happen. 666 00:37:25,245 --> 00:37:30,055 Although when you look at some of the software, you do wonder how much of 667 00:37:30,065 --> 00:37:34,565 that was actually supervised and how much was just dumped automatically. 668 00:37:34,955 --> 00:37:36,765 But that's for another story altogether. 669 00:37:36,905 --> 00:37:42,915 Let's talk about the local LLMs, and how good they are by comparison, because 670 00:37:43,355 --> 00:37:48,645 I've poked both, but I've never really done like a side by side comparison 671 00:37:48,705 --> 00:37:50,485 to really tell how good they are. 672 00:37:51,015 --> 00:37:58,685 You used Llama 2, I think, and OpenOrca, and done some side-by-side comparison. 673 00:37:58,685 --> 00:38:02,545 So how good are they compared to what you get with Copilot? 674 00:38:02,915 --> 00:38:06,315 Actually, this was probably my favorite chapter, to write 675 00:38:06,685 --> 00:38:08,575 and to do the research on. 676 00:38:08,935 --> 00:38:09,925 It was just super fun. 677 00:38:10,465 --> 00:38:12,415 it was an old Lema 2 model. 678 00:38:12,465 --> 00:38:14,695 it was generations old at this point. 679 00:38:15,145 --> 00:38:16,975 so I've been meaning to revisit it. 680 00:38:17,065 --> 00:38:19,285 it produced competent, text. 681 00:38:19,625 --> 00:38:22,725 natural language processing, give me a description of this. 682 00:38:22,745 --> 00:38:25,255 the code quality was not great. 683 00:38:25,795 --> 00:38:31,075 but again, I'm sure this was, six months ago or plus. 684 00:38:31,175 --> 00:38:34,805 So I'm sure that the model is 10 times better now. 685 00:38:35,155 --> 00:38:36,645 So definitely worth revisiting. 686 00:38:36,795 --> 00:38:40,605 yeah, I would say on balance, most of the models that I was running, that I was 687 00:38:40,605 --> 00:38:42,945 running locally did not perform as well. 688 00:38:42,995 --> 00:38:46,065 but they performed competently. 689 00:38:46,755 --> 00:38:52,275 so if you were in a pinch and you didn't have access to the internet and you'd had 690 00:38:52,325 --> 00:38:56,835 some foresight and downloaded these models prior, you could still get the job done. 691 00:38:57,975 --> 00:38:59,755 but they wouldn't necessarily be my go to. 692 00:38:59,755 --> 00:39:02,155 although I did, yeah, I did, just recently. 693 00:39:02,206 --> 00:39:05,716 Redownload a new model and it does seem to be much better at this point. 694 00:39:05,856 --> 00:39:07,466 I think it was Mistral. 695 00:39:08,256 --> 00:39:13,066 that's one of the more interesting areas, in my mind, because that helps 696 00:39:13,066 --> 00:39:15,006 get around some of the unknowns. 697 00:39:15,156 --> 00:39:18,336 because it, I did, I turned off my wifi. 698 00:39:18,826 --> 00:39:20,006 I pulled the network cable. 699 00:39:20,006 --> 00:39:24,486 I made sure that I was entirely off the network, prior to using them 700 00:39:24,486 --> 00:39:27,906 because I wanted to make sure no context was leaving my computer. 701 00:39:28,466 --> 00:39:33,226 Privacy, of your code of, personal data is a primary concern. 702 00:39:33,276 --> 00:39:35,556 it's probably the best option out there today. 703 00:39:36,630 --> 00:39:39,090 I think this is a really good argument for that. 704 00:39:39,500 --> 00:39:43,310 A lot of people will be in a situation where their employers are just not 705 00:39:43,360 --> 00:39:44,790 comfortable with just going somewhere. 706 00:39:45,285 --> 00:39:47,835 no matter what pinky swears, you got. 707 00:39:48,405 --> 00:39:51,515 And I think that this opens, like the remainder of the 708 00:39:51,515 --> 00:39:53,035 market that really matters. 709 00:39:53,075 --> 00:39:57,835 And I think we're all waiting for Llama 3 to drop, any week now. 710 00:39:58,265 --> 00:40:02,515 I'm just worried that it might be too big to run comfortably on your M3. 711 00:40:03,125 --> 00:40:06,525 even with quantization, but, let's see, it might actually not increase 712 00:40:06,525 --> 00:40:08,805 in size yet become more competent. 713 00:40:09,315 --> 00:40:12,515 There are other models too, like Santa Coder, I think at some 714 00:40:12,515 --> 00:40:13,985 point was very popular as well. 715 00:40:14,985 --> 00:40:18,775 Did, you manage to get a workflow that's more like Copilot and 716 00:40:18,775 --> 00:40:20,575 less like chatting to ChatGPT? 717 00:40:20,735 --> 00:40:22,335 that would be a really good challenge. 718 00:40:23,735 --> 00:40:28,065 To try to turn one of these into a more of a Copilot model. 719 00:40:28,385 --> 00:40:31,695 it was ultimately one that I just, couldn't get done in time. 720 00:40:32,135 --> 00:40:33,755 um, so no, 721 00:40:33,815 --> 00:40:36,185 hopefully you knew what you were getting yourself into, but you 722 00:40:36,185 --> 00:40:39,885 wrote an AI book, so you have to update it every three months now. 723 00:40:40,276 --> 00:40:41,336 that is true. 724 00:40:41,406 --> 00:40:45,859 one of my favorite titles, of late, was, another Manning book, but it 725 00:40:45,859 --> 00:40:50,516 was the completely, out of date or the complete, yeah, obsolete. 726 00:40:50,536 --> 00:40:51,146 Exactly. 727 00:40:51,316 --> 00:40:52,615 A book on, generative AI. 728 00:40:53,125 --> 00:40:54,365 I thought that was really clever. 729 00:40:54,385 --> 00:40:55,045 lean into it, 730 00:40:56,045 --> 00:40:58,115 yeah, that's a book by David Clinton. 731 00:40:58,255 --> 00:41:02,315 we had him on the podcast, a couple of weeks ago and, I think it's 732 00:41:02,315 --> 00:41:06,840 called the Complete Obsolete Guide to generative AI, it's hilarious. 733 00:41:06,890 --> 00:41:09,600 I had so much fun actually just reading that. 734 00:41:10,160 --> 00:41:12,260 the humor is, is nice level there. 735 00:41:12,660 --> 00:41:15,910 definitely highest possible recommendation buy that book over mine. 736 00:41:17,048 --> 00:41:20,158 probably worth mentioning when Devin thing was going on, there was 737 00:41:20,168 --> 00:41:22,528 some kind of open source response. 738 00:41:22,528 --> 00:41:26,238 I think it was SWE, OpenSWE, something like that. 739 00:41:26,588 --> 00:41:29,398 it's probably easy google'able and I haven't gotten to actually 740 00:41:29,398 --> 00:41:31,013 testing out, but I was supposed to. 741 00:41:32,013 --> 00:41:35,693 I think what we really need to get to is to get one of those open models 742 00:41:35,693 --> 00:41:39,033 to behave, 85% as well as Copilot. 743 00:41:39,053 --> 00:41:40,373 And then it's basically game over. 744 00:41:40,413 --> 00:41:44,323 If I can run it on your laptop, there's no subscription, there's no data leaving. 745 00:41:44,413 --> 00:41:45,823 it's a no brainer at that stage. 746 00:41:46,033 --> 00:41:49,213 what are we doing with the CPU cycles and the GPU cycles on the 747 00:41:49,213 --> 00:41:50,613 MacBook when we're developing? 748 00:41:50,808 --> 00:41:51,258 Anyway. 749 00:41:51,268 --> 00:41:51,548 Yeah. 750 00:41:51,713 --> 00:41:51,893 there 751 00:41:52,038 --> 00:41:52,248 Yeah. 752 00:41:53,193 --> 00:41:55,663 And it just makes, it makes sense like from a corporate, 753 00:41:55,713 --> 00:41:59,283 decision making you could host it, not necessarily, centralized 754 00:41:59,283 --> 00:42:01,093 host training off your own data. 755 00:42:01,283 --> 00:42:02,853 like that's, yeah, that's really game over. 756 00:42:03,543 --> 00:42:07,623 although, yeah, 757 00:42:08,403 --> 00:42:10,303 Another alternative way of saying that. 758 00:42:10,873 --> 00:42:12,153 It's just the beginning of the fun. 759 00:42:12,533 --> 00:42:12,753 yeah. 760 00:42:12,803 --> 00:42:14,043 Beginning of the arms race. 761 00:42:14,343 --> 00:42:14,363 Yeah. 762 00:42:15,583 --> 00:42:17,653 What's the next thing here? 763 00:42:18,013 --> 00:42:22,093 What do you expect to come in the coming months and years, Obviously 764 00:42:22,093 --> 00:42:26,373 wild predictions, all the usual disclaimers, but what's your take? 765 00:42:26,473 --> 00:42:29,543 What's next in coding and AI? 766 00:42:30,558 --> 00:42:31,638 I'm gonna give a boring answer. 767 00:42:31,638 --> 00:42:33,828 I think it's just gonna be incremental improvement. 768 00:42:33,828 --> 00:42:36,688 AGI if it's possible is a long No. 769 00:42:37,538 --> 00:42:38,648 what is, what are they calling it? 770 00:42:38,818 --> 00:42:40,758 Artificial General Intelligence, yeah, AGI. 771 00:42:41,048 --> 00:42:41,198 Yeah. 772 00:42:41,198 --> 00:42:42,608 AGI, it. 773 00:42:43,223 --> 00:42:48,393 I think that's a ways off if at all, if it's even feasible, we'll see incremental 774 00:42:48,393 --> 00:42:52,093 improvements, where the models, I don't want to say hallucinate less because 775 00:42:52,093 --> 00:42:56,033 that's that's a feature, not a bug, but, where they get, more and more 776 00:42:56,033 --> 00:43:02,603 refined, the output, It becomes, more, more timely, we're starting to see where 777 00:43:02,603 --> 00:43:06,173 it can actually connect live to the internet so I just think there's going 778 00:43:06,173 --> 00:43:07,663 to be incremental advances like that. 779 00:43:07,783 --> 00:43:11,023 until there's a real breakthrough and that will, change the game, like in 780 00:43:11,023 --> 00:43:15,023 the same way that, the transformer changed the way that we did the natural 781 00:43:15,023 --> 00:43:16,913 language processing and text generation. 782 00:43:16,933 --> 00:43:19,783 And, until there's something like that, it's just going to 783 00:43:19,783 --> 00:43:21,653 be just incremental improvement. 784 00:43:22,720 --> 00:43:29,266 So life goes on, NVIDIA makes more chips, they make faster chips, they become even 785 00:43:29,266 --> 00:43:34,684 bigger and they leave the gamers behind even further and we make bigger models, we 786 00:43:34,684 --> 00:43:40,329 train them better and we get a little bit closer to a superstar junior developer. 787 00:43:41,002 --> 00:43:43,120 Is that roughly what we're talking about here? 788 00:43:43,259 --> 00:43:44,409 that's what I would predict. 789 00:43:44,499 --> 00:43:48,789 I'm, happy to be wrong Unless it's completely detrimental to all of 790 00:43:48,789 --> 00:43:53,469 our fine men and women out there, giving their blood, sweat and tears 791 00:43:53,469 --> 00:43:55,159 every day in developing software. 792 00:43:55,219 --> 00:43:56,999 Not investment advice, by 793 00:43:57,034 --> 00:43:57,914 Yes, exactly. 794 00:43:58,057 --> 00:44:01,447 another disclaimer for our us-based, clientele. 795 00:44:01,447 --> 00:44:04,587 And what's next for you, Nathan, do you have an eye for the next book? 796 00:44:05,802 --> 00:44:08,342 I have a couple of ideas brewing. 797 00:44:08,352 --> 00:44:12,472 I really am going to avoid doing just a second edition of this. 798 00:44:13,182 --> 00:44:17,352 even though I've alluded to it several times, but I have a couple of ideas 799 00:44:17,352 --> 00:44:22,352 that are really brewing and in terms of okay, so now we know how to use them. 800 00:44:22,642 --> 00:44:26,042 we've had some practice like now let's apply it to very 801 00:44:26,042 --> 00:44:27,582 specific, very niche problems. 802 00:44:27,632 --> 00:44:29,532 and are there ways that we can extend it? 803 00:44:30,092 --> 00:44:33,842 Are there ways that we can, train it on our own data, things like that. 804 00:44:34,572 --> 00:44:39,582 that's where I would see the next logical, area for me to move into. 805 00:44:40,742 --> 00:44:42,992 but I would still want something very practical. 806 00:44:43,392 --> 00:44:45,672 so yeah, I'll probably just wind up doing a second edition. 807 00:44:46,172 --> 00:44:50,392 the book once again is called "AI-Powered Developer". 808 00:44:50,502 --> 00:44:54,512 It's published by Manning, which means that it has been available in 809 00:44:54,512 --> 00:44:56,092 the early access for a while now. 810 00:44:56,712 --> 00:45:02,156 So if you go to manning.com, you can get immediate access and start reading that. 811 00:45:02,467 --> 00:45:06,517 It's currently in production, which means that it's going to take a little bit of 812 00:45:06,517 --> 00:45:11,537 time before it actually hits places like Amazon in a physical copy and before 813 00:45:11,537 --> 00:45:14,467 Nathan can have a party to celebrate that. 814 00:45:15,427 --> 00:45:17,967 my guest was Nathan B. 815 00:45:17,967 --> 00:45:21,217 Crocker, the co founder and CTO at Checkr. 816 00:45:21,547 --> 00:45:22,717 Thank you very much, Nathan. 817 00:45:22,897 --> 00:45:23,117 I'll see you 818 00:45:23,312 --> 00:45:23,772 Thank you. 819 00:45:23,842 --> 00:45:24,552 It was a pleasure. 820 00:45:25,202 --> 00:45:25,622 Be well.