1 00:00:05,840 --> 00:00:10,424 On this episode of Translating Proteomics, co hosts Parag Malik 2 00:00:10,472 --> 00:00:15,576 and Andreas Humer of Nautilus Biotechnology discuss the reproducibility 3 00:00:15,688 --> 00:00:19,672 crisis in biology and specifically focus on how we can 4 00:00:19,696 --> 00:00:24,824 enhance reproducibility in computational proteomics. Key topics 5 00:00:24,872 --> 00:00:30,124 they cover include what the reproducibility crisis is, factors 6 00:00:30,172 --> 00:00:34,348 that make it difficult to replicate multiomics research, and 7 00:00:34,484 --> 00:00:38,640 steps we can take to make biology research more reproducible. 8 00:00:38,980 --> 00:00:43,440 Now here are your hosts, Parag Malik and Andreas Humer. 9 00:00:43,940 --> 00:00:48,188 Welcome back to Translating Proteomics. In today's episode, we'll 10 00:00:48,204 --> 00:00:52,582 cover reproducibility in proteomics research. Reproducibility 11 00:00:52,646 --> 00:00:56,774 is an important topic in all research, but particularly for those 12 00:00:56,862 --> 00:01:01,062 researchers that work with big data or with multi omics data. To 13 00:01:01,086 --> 00:01:05,990 help you wrap your head around the issue today, we will cover why 14 00:01:06,030 --> 00:01:09,570 it is important to have reproducibility in multi omics work, 15 00:01:10,270 --> 00:01:15,142 steps we can take to improve reproducibility, and maybe the role 16 00:01:15,166 --> 00:01:21,004 of AI in helping to create more reproducible results. With that, 17 00:01:21,172 --> 00:01:24,812 I want to congratulate to your recent Gilbert Ohman Award which 18 00:01:24,836 --> 00:01:30,320 is given to you for progress in computational bioinformatics. 19 00:01:30,980 --> 00:01:34,812 During your acceptance lecture, you chose to talk about 20 00:01:34,916 --> 00:01:39,244 reproducibility, so maybe you can give us a little bit of an overview 21 00:01:39,292 --> 00:01:41,372 of the topic you touched there. 22 00:01:41,556 --> 00:01:45,498 I'd be delighted. Reproducibility has been something 23 00:01:45,554 --> 00:01:50,910 that we take for granted as a core aspect of the scientific method. 24 00:01:51,410 --> 00:01:55,850 If we think about observation, we first we make an observation, 25 00:01:56,010 --> 00:02:00,682 we then do some analysis and interpretation, we form a hypothesis 26 00:02:00,826 --> 00:02:03,322 and then we test that hypothesis and we go back and make 27 00:02:03,346 --> 00:02:07,642 more observations. And that cycle from observation to analysis 28 00:02:07,706 --> 00:02:12,082 to testing and back again, is the root of, of what it means to 29 00:02:12,106 --> 00:02:17,474 do science. Now the implicit assumption there is that if somebody 30 00:02:17,522 --> 00:02:23,042 else were to do that same process, they would be able to learn 31 00:02:23,106 --> 00:02:25,842 the same things, they'd be able to come up with the same findings. 32 00:02:25,906 --> 00:02:30,722 They would ideally be able to exactly reproduce what you did at 33 00:02:30,746 --> 00:02:35,090 another time. And unfortunately this is something that 34 00:02:35,130 --> 00:02:40,636 is actually incredibly hard and we've seen so many examples. 35 00:02:40,828 --> 00:02:44,620 There was a, I think the most provocative one was in 2005 from 36 00:02:44,660 --> 00:02:48,412 John Ioannidis and there also have been other ones in the field 37 00:02:48,436 --> 00:02:51,852 of proteomics, specifically looking back at data sets and trying 38 00:02:51,876 --> 00:02:56,480 to analyze them and being unable to reproduce the findings. 39 00:02:57,380 --> 00:03:01,468 There also was an entire special issue, special topic area 40 00:03:01,524 --> 00:03:06,506 in nature to talking about reproducibility and the rationale 41 00:03:06,538 --> 00:03:10,090 for how this happened. Drift of cell lines, drift of materials. 42 00:03:10,250 --> 00:03:15,402 But what we focused on in our lab and in my Lecture was just computational 43 00:03:15,466 --> 00:03:18,906 analysis. So no data collection at all. Just starting 44 00:03:18,938 --> 00:03:22,138 with an existing data set, starting with a friendly collaborator 45 00:03:22,234 --> 00:03:29,224 and saying, could we actually reproduce this work? And it was incredibly 46 00:03:29,272 --> 00:03:34,184 challenging. It literally took years of computational forensics 47 00:03:34,312 --> 00:03:39,080 in order to get as close as possible to reproducing the work 48 00:03:39,120 --> 00:03:40,888 of a friend and colleague of mine. 49 00:03:40,984 --> 00:03:46,216 So just to be clear, so you started with the data that was already 50 00:03:46,288 --> 00:03:49,672 published and you just took the data and tried to reproduce the 51 00:03:49,696 --> 00:03:51,640 data analysis portion of the experiment? 52 00:03:51,720 --> 00:03:54,408 That's right. That's right. So no experimental variation at all? 53 00:03:54,464 --> 00:03:55,144 Oh, wow. 54 00:03:55,272 --> 00:03:59,752 Just computational analysis. And our goal was to do a figure for 55 00:03:59,776 --> 00:04:04,344 figure reproduction of their paper. And we thought this would 56 00:04:04,352 --> 00:04:09,100 be pretty straightforward, and we learned very much otherwise. 57 00:04:09,520 --> 00:04:15,560 Interesting. So I guess it's to be expected that you have variations 58 00:04:15,720 --> 00:04:20,456 in the laboratory, particularly if you work with complex 59 00:04:20,568 --> 00:04:23,750 biological systems. But. But I'm sort of surprised to hear that 60 00:04:23,790 --> 00:04:29,334 even a computational data pipeline, you see so many variables 61 00:04:29,382 --> 00:04:29,830 there. 62 00:04:29,950 --> 00:04:30,390 Yeah. 63 00:04:30,470 --> 00:04:32,486 What was the underlying reason for that? 64 00:04:32,638 --> 00:04:36,982 Well, there were a number. So I think what we particularly observed 65 00:04:37,046 --> 00:04:43,462 is that reproducibility is hard. Again, it's often stated as, 66 00:04:43,486 --> 00:04:46,390 oh, science should be reproducible. But the reality is 67 00:04:46,430 --> 00:04:50,184 that in any of these complex studies, particularly multiomics 68 00:04:50,232 --> 00:04:54,600 ones, you're bringing together many different types of data in order 69 00:04:54,640 --> 00:04:59,176 to analyze it. They're going through potentially dozens of different 70 00:04:59,248 --> 00:05:03,432 tools. Those tools, different software tools, different servers, 71 00:05:03,496 --> 00:05:08,648 websites, et cetera, based upon databases that are used along 72 00:05:08,704 --> 00:05:12,664 the way as part of those tools. So input files, those tools 73 00:05:12,712 --> 00:05:16,818 may have hundreds of parameters. And while we do a really 74 00:05:16,874 --> 00:05:20,754 good job as a field, as a biological community, of depositing 75 00:05:20,802 --> 00:05:25,970 our data, what we don't deposit are the analyses that we 76 00:05:26,010 --> 00:05:30,354 did. So a software version from five years ago may not exist 77 00:05:30,402 --> 00:05:30,706 anymore. 78 00:05:30,738 --> 00:05:31,426 Oh, yeah. 79 00:05:31,538 --> 00:05:35,298 Or worse, it may not even run anymore because the operating system 80 00:05:35,354 --> 00:05:40,338 has changed. A database that was originally available may have 81 00:05:40,394 --> 00:05:45,470 matured or evolved. So a lot of the variation that comes in really 82 00:05:45,590 --> 00:05:51,374 comes down to three particular areas. The first is did we sufficiently 83 00:05:51,422 --> 00:05:55,726 document what was done in the first place? Did we know what the 84 00:05:55,798 --> 00:05:58,606 analysis tools that were used? Were? Do we know the parameters? 85 00:05:58,638 --> 00:06:03,330 Do we knew the input files alongside that? Did we actually save 86 00:06:03,750 --> 00:06:06,382 all of those assets? And what we found is there are a number of 87 00:06:06,406 --> 00:06:09,916 places where there were various very seemingly small details 88 00:06:10,028 --> 00:06:12,760 in the documentation that were left out. 89 00:06:13,220 --> 00:06:16,364 So you just talked about documentation how better documentation 90 00:06:16,412 --> 00:06:21,164 may help researcher later on to sort of reproduce what you did. 91 00:06:21,332 --> 00:06:24,000 What about the tools themselves? 92 00:06:24,340 --> 00:06:26,844 Absolutely. That's really critical. So when we think about 93 00:06:26,932 --> 00:06:31,868 documentation, our primary documentation is a manuscript, it's 94 00:06:31,884 --> 00:06:35,148 a method section in a paper. And that doesn't necessarily always 95 00:06:35,204 --> 00:06:40,062 capture every single facet of a tool. How it was run, what operating 96 00:06:40,126 --> 00:06:44,574 system, what additional plugins might exist. The second piece 97 00:06:44,622 --> 00:06:48,410 of ensuring the reproducibility in this domain is 98 00:06:49,030 --> 00:06:54,290 making sure that all of that chain of what was used in the analysis 99 00:06:55,110 --> 00:07:02,670 remains available and remains executable and that it's just electronically 100 00:07:02,750 --> 00:07:06,166 possible to perform the steps. And what makes that particularly 101 00:07:06,198 --> 00:07:09,734 challenging is that oftentimes complex analyses are done by multiple 102 00:07:09,782 --> 00:07:13,606 individuals as part of a paper. They're done across multiple 103 00:07:13,638 --> 00:07:16,710 different types of software. So something will be done as an R 104 00:07:16,750 --> 00:07:19,574 script, or something will be done in Excel, or something will 105 00:07:19,582 --> 00:07:22,966 be done using a software package. And so bringing together 106 00:07:23,038 --> 00:07:28,342 all of those pieces of the actual workflow is a really non trivial 107 00:07:28,406 --> 00:07:29,130 challenge. 108 00:07:29,470 --> 00:07:35,328 Yeah. So just observing or reading the method section myself 109 00:07:35,424 --> 00:07:39,904 in a paper, often realize that there's many tools and many approaches 110 00:07:39,952 --> 00:07:44,352 and some of them have very fancy names. I can imagine it being 111 00:07:44,376 --> 00:07:50,464 incredibly difficult to train a scientist later on how to reproduce 112 00:07:50,512 --> 00:07:53,580 that. Is there a solution to that? 113 00:07:53,960 --> 00:07:57,892 I think what you've just highlighted is, is tremendous because 114 00:07:58,076 --> 00:08:01,796 it's actually an even larger problem because for any given task, 115 00:08:01,988 --> 00:08:05,844 so within one paper they might say I use this tool. But for instance, 116 00:08:05,892 --> 00:08:09,956 for the task of how do I take a spectrum and identify a protein 117 00:08:09,988 --> 00:08:13,876 from that spectrum, there might be dozens or maybe even hundreds 118 00:08:13,908 --> 00:08:17,348 of different tools that are each capable of doing that task, 119 00:08:17,444 --> 00:08:21,300 but they're not interchangeable. And so one may have 120 00:08:21,340 --> 00:08:25,260 particular biases, another a different set of biases. And so we 121 00:08:25,300 --> 00:08:30,880 really need to first educate people that there is such a diversity, 122 00:08:31,300 --> 00:08:35,360 2 educate them that the different tools may behave differently. 123 00:08:35,700 --> 00:08:40,892 And then I think the third part is that oftentimes the details 124 00:08:40,956 --> 00:08:46,092 of how a given tool works can be very sophisticated. So if we don't 125 00:08:46,156 --> 00:08:49,132 expect people to understand all of those details, we need to 126 00:08:49,156 --> 00:08:52,840 provide some guidance on under what conditions does this work, how 127 00:08:52,880 --> 00:08:57,560 well does this work. And so the training aspect oftentimes when 128 00:08:57,600 --> 00:09:01,144 you're a tool developer, you're focused on validating that 129 00:09:01,152 --> 00:09:04,536 your tool is better than some other tool. That educational component 130 00:09:04,568 --> 00:09:08,420 of how do I train users, how do I share this with the community, 131 00:09:09,840 --> 00:09:13,736 I think is underappreciated just in general in science. And it's 132 00:09:13,768 --> 00:09:19,042 actually quite difficult to get funding for, for education or 133 00:09:19,146 --> 00:09:22,530 even software dissemination or software maintenance. 134 00:09:22,690 --> 00:09:26,274 I think you make a good point. This is where probably a commercial 135 00:09:26,322 --> 00:09:30,770 product is superior in terms of documentation or availability 136 00:09:30,850 --> 00:09:35,106 for training. The question is what prompted you to actually Investigate 137 00:09:35,138 --> 00:09:38,594 that in the first place, one could argue, okay, so there was a 138 00:09:38,602 --> 00:09:42,410 study that was done five years ago. Who cares who? 139 00:09:42,820 --> 00:09:46,444 I think this actually came about in part because I was working 140 00:09:46,612 --> 00:09:50,572 with an amazing investigator at usc, Yolanda Gill. And we were 141 00:09:50,596 --> 00:09:54,444 part of a DARPA project called Simplex. And it was specifically 142 00:09:54,492 --> 00:10:00,924 about the question of how can we develop systems to learn? And 143 00:10:01,092 --> 00:10:05,404 as part of that process, we started building these sophisticated 144 00:10:05,452 --> 00:10:09,388 workflows and building AI layers on top of them to help those 145 00:10:09,444 --> 00:10:12,172 systems determine whether to use software one or software two 146 00:10:12,196 --> 00:10:15,642 or software three. And, and as we started doing that, we came to 147 00:10:15,666 --> 00:10:20,330 recognize just how brittle these workflows were. And so there 148 00:10:20,370 --> 00:10:23,754 was a bit of a convergence at the time of my own lab had put out 149 00:10:23,762 --> 00:10:27,418 the Proteo wizard software, which was used by tens of thousands 150 00:10:27,434 --> 00:10:31,418 of people in the community. And seeing how impactful that was, 151 00:10:31,554 --> 00:10:37,050 and then saying, oh, golly, these chains of tools, this is the 152 00:10:37,090 --> 00:10:42,316 next hard problem is how can we actually capture and make it easier 153 00:10:42,348 --> 00:10:48,492 for people to do complete end to end analyses and then make it 154 00:10:48,516 --> 00:10:53,020 easy for people to record and save those analyses? And so it really 155 00:10:53,060 --> 00:10:58,796 grew out of recognizing just what a big problem doing an analysis 156 00:10:58,828 --> 00:11:02,540 is. And then from there you're like, wow, it's hard enough to do 157 00:11:02,580 --> 00:11:06,476 one. What if I wanted to run it again? Golly, that's really hard. 158 00:11:06,548 --> 00:11:08,732 And then just seeing an opportunity to have some impact. 159 00:11:08,836 --> 00:11:12,220 So clearly a very big problem. But I can imagine this problem getting 160 00:11:12,340 --> 00:11:16,668 even bigger as we go to bigger data sets, technology evolving even 161 00:11:16,724 --> 00:11:22,652 faster. On the other hand, the storage is becoming really cheap. 162 00:11:22,716 --> 00:11:26,604 So you can maintain all of these data sets, but you ultimately 163 00:11:26,652 --> 00:11:30,440 may not be able to really take advantage of them. What do you see? 164 00:11:30,900 --> 00:11:33,284 How do we solve this? How do we address this? 165 00:11:33,412 --> 00:11:37,092 Well, I think what's really interesting to recognize right now, 166 00:11:37,196 --> 00:11:41,508 we're seeing this birth of AI in different domains. And the chatgpts 167 00:11:41,524 --> 00:11:45,684 and clods of the world are trained off of Internet scale data. 168 00:11:45,852 --> 00:11:51,460 So literally every webpage in the world and everything on Wikipedia. 169 00:11:51,620 --> 00:11:57,400 And it takes that scale of data to train some of these models. 170 00:11:57,740 --> 00:12:01,946 Similar things in pathology, they're repositories of billions 171 00:12:01,978 --> 00:12:07,882 of pathology images that have trained transformer models. So I 172 00:12:07,906 --> 00:12:13,070 think that question of how do we get multi ohmic data to that scale, 173 00:12:13,570 --> 00:12:17,402 part of it comes from what is our primary data type, what is our 174 00:12:17,426 --> 00:12:21,402 primary data modality. So with pathology data, it's image data we 175 00:12:21,426 --> 00:12:24,346 understand, same as cats and dogs, we understand how to store 176 00:12:24,378 --> 00:12:29,544 it, we understand how to read it. And so with multiomic data on 177 00:12:29,552 --> 00:12:33,112 the other hand, yes, we have standards, but we don't necessarily 178 00:12:33,176 --> 00:12:36,888 know what to look at in that. We don't know how to run it. There's 179 00:12:36,904 --> 00:12:42,776 not an established workflow. And so I think the challenge is exponentiated. 180 00:12:42,888 --> 00:12:48,312 The data's a much higher dimension. It's not of higher numeric 181 00:12:48,376 --> 00:12:52,380 number. For some cancers, we may have just a handful of patients, 182 00:12:52,800 --> 00:12:58,282 so we have large dimension, low N. And you're adding layers of 183 00:12:58,306 --> 00:13:02,122 just how in the world do I analyze it? So I think the way that 184 00:13:02,146 --> 00:13:06,426 we start to resolve that is in part by creating these repositories, 185 00:13:06,538 --> 00:13:09,498 not just of data, but repositories of analytical tools, 186 00:13:09,594 --> 00:13:13,402 workflows, saving and capturing what is the workflow. And 187 00:13:13,426 --> 00:13:17,642 then we need a layer on top of that of learnings. So our current 188 00:13:17,746 --> 00:13:21,524 vehicle for transmitting knowledge is not a, a scientific 189 00:13:21,572 --> 00:13:25,508 model. It's not a graph that a computer can ingest and say, oh, 190 00:13:25,524 --> 00:13:27,764 hey, this is what the model says, that there's a relationship 191 00:13:27,812 --> 00:13:31,972 between this protein and that protein. Our unit of learning is 192 00:13:31,996 --> 00:13:38,804 a paper written in somewhat obfuscated language. And so one of 193 00:13:38,812 --> 00:13:42,500 the other things we need to do is, as we're making advances in science, 194 00:13:42,660 --> 00:13:47,458 codifying them in a structured manner, is the next layer of, of 195 00:13:47,514 --> 00:13:52,722 how do we actually connect the dots between experiment that was 196 00:13:52,746 --> 00:13:57,250 done, data that was collected, analysis that was done, and learning, 197 00:13:57,410 --> 00:14:00,226 and so building that whole chain all the way through. 198 00:14:00,298 --> 00:14:02,722 But when I listen to you, I think what I'm also hearing is that 199 00:14:02,746 --> 00:14:08,630 there is an argument for actually thinking about data curators, 200 00:14:09,130 --> 00:14:13,490 which have nothing else to do but take key data sets, make sure 201 00:14:13,530 --> 00:14:16,818 that they are correct, reproduce the data analysis portion 202 00:14:16,834 --> 00:14:20,848 of it, and then make sure that we can continually use those data 203 00:14:20,904 --> 00:14:24,816 sets. And if there's an issue, there's someone who actually is responsible 204 00:14:24,848 --> 00:14:28,384 for. I mean, at least that's how Wikipedia is being maintained. 205 00:14:28,432 --> 00:14:31,472 They actually have dedicated curator. So I think that's an interesting 206 00:14:31,536 --> 00:14:34,992 idea to sort of, if we want to build on data sets that we produce 207 00:14:35,056 --> 00:14:41,152 today and have higher level data analysis on those data, that 208 00:14:41,176 --> 00:14:44,800 we actually need specific people that actually worry about 209 00:14:44,840 --> 00:14:45,952 this quality problem. 210 00:14:46,056 --> 00:14:50,880 Well, and I think we do for large consortia, for things like 211 00:14:50,920 --> 00:14:55,024 Protein Atlas is a really great example where you have a huge 212 00:14:55,072 --> 00:14:58,176 amount of infrastructure for making sure that data is collected 213 00:14:58,208 --> 00:15:02,704 in a standardized manner, it's deposited and organized. And so I 214 00:15:02,712 --> 00:15:05,088 think what's amazing about your Wikipedia example is that's 215 00:15:05,104 --> 00:15:09,074 crowdsourced. So yes, you have people working there who are curators, 216 00:15:09,192 --> 00:15:13,406 but you also have the entire community contributing, and that's 217 00:15:13,438 --> 00:15:17,410 Amazing and remarkable. And so how can we get to that level of quality? 218 00:15:18,550 --> 00:15:23,470 With crowdsourcing of data sets and analyses and findings. 219 00:15:23,630 --> 00:15:27,662 So I think, I would argue that in the end, the endpoint should not 220 00:15:27,686 --> 00:15:31,550 be just the paper, but the endpoint should be the data itself 221 00:15:31,710 --> 00:15:35,882 so it can be utilized in next level research. And I think that 222 00:15:35,906 --> 00:15:38,762 would probably require also a very different approach from the 223 00:15:38,786 --> 00:15:41,002 individual investigators at that point. 224 00:15:41,106 --> 00:15:44,618 So I actually, I dislike the concept of an endpoint. 225 00:15:44,714 --> 00:15:45,178 Yeah. 226 00:15:45,274 --> 00:15:52,490 Because I feel like if we're actually, if we're doing the best 227 00:15:52,530 --> 00:15:59,066 we can, we're not stopping a study when the grad student graduates. 228 00:15:59,178 --> 00:15:59,930 Yeah. 229 00:16:00,090 --> 00:16:03,370 Instead, what we're doing is that we're recognizing that the data 230 00:16:03,410 --> 00:16:07,916 we're collecting is part of a larger whole and it's part of a thousand 231 00:16:07,988 --> 00:16:12,028 studies that have yet to be done. And so there really is no end 232 00:16:12,084 --> 00:16:17,084 point. Instead, perhaps there is. Hey, I'm studying this set of 233 00:16:17,092 --> 00:16:20,716 hypotheses. Let me register my hypothesis. Okay. Let me register 234 00:16:20,748 --> 00:16:24,028 the experiments that I did and my findings, both positive and negative. 235 00:16:24,124 --> 00:16:27,116 So not just reporting only the positive results, but the null results 236 00:16:27,188 --> 00:16:32,372 as well. But then how can going further and saying, all right, 237 00:16:32,396 --> 00:16:35,860 well, this data was now deposited, this analysis workflow, 238 00:16:35,940 --> 00:16:39,748 my hypotheses and learning. But can we then go further and say, 239 00:16:39,804 --> 00:16:42,132 all right, well, somebody else is going to pick up that data and 240 00:16:42,156 --> 00:16:44,708 they're going to use it in their study. We did this recently 241 00:16:44,764 --> 00:16:48,436 in a study from my lab where we were looking at nasopharyngeal 242 00:16:48,468 --> 00:16:54,084 carcinoma. We grabbed some data from Pride that was a set of 243 00:16:54,092 --> 00:16:56,932 healthy controls on nasopharyngeal carcinoma. We had 244 00:16:56,956 --> 00:17:01,574 a set of cancer patients. And that allowed us to, to go beyond 245 00:17:01,622 --> 00:17:04,470 the study we'd planned. The study we planned was to investigate 246 00:17:04,550 --> 00:17:08,470 just nasopharyngeal carcinoma patients, identify subgroups and 247 00:17:08,590 --> 00:17:12,678 factors. But the ability to then compare to healthy controls 248 00:17:12,854 --> 00:17:16,902 allowed us to then ask a different set of questions. Unfortunately, 249 00:17:16,966 --> 00:17:20,646 the authors of that first paper who generated this amazing 250 00:17:20,678 --> 00:17:25,062 data, which we leveraged, they got a citation. But it would be great 251 00:17:25,086 --> 00:17:28,734 if we had a way to attribute and say, wow, we took advantage of 252 00:17:28,742 --> 00:17:33,102 this data that was collected. It helped us in this study. And so 253 00:17:33,206 --> 00:17:37,422 that aspect of giving people credit for their contributions beyond 254 00:17:37,486 --> 00:17:39,054 the termination of their paper. 255 00:17:39,182 --> 00:17:41,630 Yeah, And I see what you're arguing. You're not arguing just 256 00:17:41,670 --> 00:17:45,534 the data, but you're arguing the knowledge that is built on a 257 00:17:45,542 --> 00:17:48,510 data set and then another block can build on top of it. 258 00:17:48,550 --> 00:17:53,134 Right. As well as the sort of ongoing continuous testing of hypotheses. 259 00:17:53,262 --> 00:17:56,254 So I started my study with a hypothesis and I said, okay, this 260 00:17:56,262 --> 00:17:58,510 is my hypothesis. I believe there's some relationship between 261 00:17:58,550 --> 00:18:02,090 colon cancer and this. And, and I collected some amount of data. 262 00:18:02,550 --> 00:18:06,606 Five years later, somebody might collect a relevant data set. 263 00:18:06,758 --> 00:18:11,054 It would be amazing if that hypothesis had been registered. And 264 00:18:11,142 --> 00:18:14,174 we were able to then go back and retest that hypothesis in the 265 00:18:14,182 --> 00:18:20,090 context of this new data. So that again, our studies don't end 266 00:18:20,870 --> 00:18:24,158 at the publication of that paper or the graduation of the graduate 267 00:18:24,174 --> 00:18:27,422 student. Instead, they're hypotheses that the community is 268 00:18:27,446 --> 00:18:30,900 owning and is contributing to and chipping away at over time. 269 00:18:31,200 --> 00:18:35,176 So you described a process that from my perspective, would lend 270 00:18:35,208 --> 00:18:38,060 itself to an application of AI. 271 00:18:39,520 --> 00:18:43,080 Absolutely. So maybe just to add a little bit of color there, 272 00:18:43,120 --> 00:18:48,776 when we think about AI, right now, AI is a huge topic and we're 273 00:18:48,808 --> 00:18:53,064 going to have a deep dive talking about AI and biosciences. 274 00:18:53,112 --> 00:18:57,602 Stay tuned, really exciting guests for that. But AI means a lot 275 00:18:57,626 --> 00:19:01,538 of different things. On the simple end, AI means things like 276 00:19:01,674 --> 00:19:05,074 linear regression. It also means things like convolutional neural 277 00:19:05,122 --> 00:19:10,850 networks or GPTs. And so across that. But there are also other 278 00:19:10,890 --> 00:19:16,066 types of AIs. There are AIs that are decision making AIs or that 279 00:19:16,138 --> 00:19:21,836 use logic to make decisions. And so when we think about how AI 280 00:19:21,868 --> 00:19:26,812 can play a role in this, part of it is in decision making, in saying, 281 00:19:26,996 --> 00:19:32,300 hey, look, I'm about to run this workflow. I'm trying to ask 282 00:19:32,340 --> 00:19:39,212 this question, what set of tools should I use for this? And 283 00:19:39,316 --> 00:19:44,332 the AI sitting over that layer to help us craft the best analysis 284 00:19:44,396 --> 00:19:48,412 plan. Another place where the AI, if we think about the sort of 285 00:19:48,436 --> 00:19:52,642 chat universe coming in, if we register our hypotheses and say, 286 00:19:52,666 --> 00:19:55,842 hey, I'm really interested in the relationship between X and Y, 287 00:19:56,026 --> 00:19:58,514 going out to the transformer and saying, hey, can you build me 288 00:19:58,522 --> 00:20:03,314 a workflow? Great. Can you suggest an experiment or two or maybe 289 00:20:03,362 --> 00:20:06,450 suggest some data that's out there that I should incorporate? 290 00:20:06,610 --> 00:20:11,298 And having the AI as part of that, having essentially an AI buddy 291 00:20:11,394 --> 00:20:14,482 helping us throughout the whole experimental process. And then 292 00:20:14,506 --> 00:20:18,458 the last piece really is five years after you finished your study, 293 00:20:18,514 --> 00:20:21,082 the AI is just sitting there in the background paying attention 294 00:20:21,146 --> 00:20:24,426 to what the scientific community is coming up with. And 295 00:20:24,498 --> 00:20:28,586 it can notice by doing machine reading over articles that are published 296 00:20:28,618 --> 00:20:31,802 and say, oh, hey, there's a new data set that was published. 297 00:20:31,946 --> 00:20:34,762 I did machine reading. I saw that they had this conclusion that 298 00:20:34,786 --> 00:20:40,042 sort of either refutes yours or supports yours. Let me go rerun 299 00:20:40,106 --> 00:20:46,894 My analysis incorporating that data and either add support to the 300 00:20:46,902 --> 00:20:50,398 hypothesis or refute it and then send you an email and say, hey, 301 00:20:50,494 --> 00:20:53,646 my little Friday message to you as your AI is that somebody else 302 00:20:53,678 --> 00:20:56,382 published a thing. I did this analysis and here's what came out 303 00:20:56,406 --> 00:21:01,246 of it. So I think the AI tools for that are really rapidly coming 304 00:21:01,318 --> 00:21:06,030 into the world. But it's a non trivial challenge to set up that 305 00:21:06,070 --> 00:21:09,912 kind of constantly surveilling architecture. But I think, I think 306 00:21:09,936 --> 00:21:12,552 we're going to see it. I think there's absolutely a world in the 307 00:21:12,576 --> 00:21:18,280 future where AI is our companion along the way and AI is 308 00:21:18,320 --> 00:21:21,912 helping us with deposition because the whole system was done 309 00:21:22,016 --> 00:21:27,432 in partnership with AI and AI is helping extend our hypotheses 310 00:21:27,496 --> 00:21:31,480 well beyond the study that we initially might have anticipated. 311 00:21:31,560 --> 00:21:34,488 So you really sparked my interest by mentioning that we could 312 00:21:34,544 --> 00:21:38,010 use and AI to sort of help us design experiments. And in fact, 313 00:21:38,050 --> 00:21:41,946 a lot of viewers may actually be agreeing on this because as we 314 00:21:42,018 --> 00:21:47,386 talked about, all of us have limited knowledge in what we do. 315 00:21:47,458 --> 00:21:50,762 And particularly when it comes to experimental design, we usually 316 00:21:50,826 --> 00:21:54,122 refer to bioinformaticians or other colleagues. And wouldn't be 317 00:21:54,146 --> 00:21:58,042 great if you state your hypothesis and an eye guides you 318 00:21:58,066 --> 00:22:00,694 through this. Like, by the way, you need thousands of data points. 319 00:22:00,802 --> 00:22:03,790 Your budget doesn't even support that. So maybe we do a different 320 00:22:03,830 --> 00:22:07,262 type of experiment. I think this would be great first step to 321 00:22:07,286 --> 00:22:10,958 sort of guarantee whenever we spend resources and time in science, 322 00:22:11,054 --> 00:22:14,782 we actually create the quality of data that ultimately lead to the 323 00:22:14,806 --> 00:22:17,502 third step you described, which is an AI in the background 324 00:22:17,566 --> 00:22:21,758 keeps now adding to the knowledge that was created because 325 00:22:21,814 --> 00:22:26,398 we had a guided experiment that yielded automatically very, 326 00:22:26,454 --> 00:22:29,016 very high data quality. 327 00:22:29,208 --> 00:22:32,184 Yeah, I think that's a really interesting concept. There are so 328 00:22:32,192 --> 00:22:34,152 many times as a bioinformaticist where somebody comes 329 00:22:34,176 --> 00:22:37,016 to me and they say, oh, I collected this data, can you find 330 00:22:37,088 --> 00:22:38,180 something in it? 331 00:22:38,960 --> 00:22:40,216 Oh my God, yeah. 332 00:22:40,328 --> 00:22:43,752 And I sit there and I'm like, well, maybe I don't know, what was 333 00:22:43,776 --> 00:22:46,184 your hypothesis? It's like, well, you know that there'd be some 334 00:22:46,192 --> 00:22:50,152 differences here. Like, okay, what are your controls? Like, oh, 335 00:22:50,216 --> 00:22:55,528 well, we don't have any. I'm like, okay, all right. And so having 336 00:22:55,584 --> 00:22:59,272 those conversations up front and at the experimental design phase, 337 00:22:59,416 --> 00:23:02,520 having the AI chatting with you and being like, okay, well to 338 00:23:02,560 --> 00:23:05,480 ask this question, you're underpowered, but you can ask this 339 00:23:05,520 --> 00:23:09,992 question. That would be just so exciting to have that as part 340 00:23:10,016 --> 00:23:13,880 of the process. And I do think this builds upon work that has been 341 00:23:13,920 --> 00:23:18,168 ongoing for a while. Things like Ms. Stats from Olga Witek's 342 00:23:18,184 --> 00:23:23,096 group, which helps you do these power analyses. Workflow engines 343 00:23:23,128 --> 00:23:27,704 like Galaxy that allow just chaining together of tools more easily. 344 00:23:27,752 --> 00:23:31,732 And there are a number of pieces of that ecosystem that are 345 00:23:31,916 --> 00:23:33,320 beginning to exist. 346 00:23:33,980 --> 00:23:38,852 Fascinating. So what are we going to do now? And particularly 347 00:23:38,916 --> 00:23:42,292 some users may be thinking, I totally agree with the vision, but 348 00:23:42,316 --> 00:23:46,212 what can we do today to sort of increase the quality and reproducibility 349 00:23:46,276 --> 00:23:48,068 of experiments? What's your thought? 350 00:23:48,204 --> 00:23:55,894 Yeah, so my top three for the moment, number one, come comes back 351 00:23:55,902 --> 00:23:59,238 to what we just talked about, experimental design, and really being 352 00:23:59,294 --> 00:24:03,302 clear about stating what is the question that I'm asking. And 353 00:24:03,486 --> 00:24:08,134 in our documentation of that right now, we do lack documentation 354 00:24:08,182 --> 00:24:11,590 around what is the stated hypothesis. There are registries 355 00:24:11,670 --> 00:24:17,254 of hypotheses. You can lock in a hypothesis in advance. Some journals 356 00:24:17,382 --> 00:24:23,132 actually allow you to deposit a hypothesis and determine good or 357 00:24:23,156 --> 00:24:27,292 bad, whether it's successful results or not. Do they agree to 358 00:24:27,316 --> 00:24:30,812 publish the result, regardless of what the outcome is, assuming 359 00:24:30,876 --> 00:24:33,644 it was experimentally done properly. So I think step one we 360 00:24:33,652 --> 00:24:36,972 can do is we can be really rigorous about our hypothesis and 361 00:24:36,996 --> 00:24:41,692 experimental planning. Then step two is the documentation of 362 00:24:41,716 --> 00:24:46,444 that. Workflow engines are prevalent now, and so there exist 363 00:24:46,492 --> 00:24:50,106 things like common workflow language which we can use for documenting 364 00:24:50,138 --> 00:24:53,658 the processes that we took. So rather than being limited to things 365 00:24:53,714 --> 00:24:58,362 like a method section in a paper where we do our best to say, 366 00:24:58,386 --> 00:25:01,962 I did this and I did this and I did this by developing what are 367 00:25:01,986 --> 00:25:05,450 called containerized workflows. These are literally little 368 00:25:05,490 --> 00:25:10,602 software computers that record every step that you took and software 369 00:25:10,666 --> 00:25:14,058 versions and everything. So when we do an analysis, doing it 370 00:25:14,114 --> 00:25:18,284 in a workflow engine in a containerized manner will immediately 371 00:25:18,332 --> 00:25:22,780 allow the reproducibility of things. And I think the third is 372 00:25:22,820 --> 00:25:28,012 just continuing to argue for support for funding for these areas 373 00:25:28,116 --> 00:25:32,988 of reproducibility and tools supporting it, because I think oftentimes 374 00:25:33,004 --> 00:25:36,812 we're really excited about the shiny new toy of, hey, this is an 375 00:25:36,836 --> 00:25:40,540 amazing biological insight. But the infrastructure underlying 376 00:25:40,620 --> 00:25:44,296 all of our scientific endeavors can often be quite challenging 377 00:25:44,328 --> 00:25:44,920 to fund. 378 00:25:45,040 --> 00:25:48,392 Yeah, with that I would say welcome to the new lab, welcome to 379 00:25:48,416 --> 00:25:56,120 the future in the lab with AI. So what organizations should our 380 00:25:56,160 --> 00:25:59,832 users be thinking of if they want to make contribution in this 381 00:25:59,856 --> 00:26:00,900 particular field? 382 00:26:01,840 --> 00:26:06,136 I think chatting with your congressman is incredibly important 383 00:26:06,208 --> 00:26:12,470 here. This area of science, this foundational science that is 384 00:26:12,590 --> 00:26:17,734 really unappreciated. And I think the other aspect there is that 385 00:26:17,902 --> 00:26:21,942 the reproducibility crisis in science hasn't fully made it all 386 00:26:21,966 --> 00:26:26,998 the way to Capitol Hill. So advocating to say, hey, this is something 387 00:26:27,054 --> 00:26:30,454 that's going to exponentiate the value and impact of scientific 388 00:26:30,502 --> 00:26:34,982 dollars spent. And so let's make sure that we are spending dollars 389 00:26:35,046 --> 00:26:39,850 to ensure a reproducible and extensible future. I think number 390 00:26:39,890 --> 00:26:42,762 two is there are specific organizations within the government, 391 00:26:42,866 --> 00:26:46,710 things like the National Institute for Standards and Technology, 392 00:26:47,090 --> 00:26:51,002 which helps define how should we store a data set, how should we 393 00:26:51,026 --> 00:26:57,818 run an experiment. I think also the National Institute of General 394 00:26:57,874 --> 00:27:03,258 Medical Sciences has a huge group focused on workflows and development 395 00:27:03,314 --> 00:27:06,442 of software and tools. So those are two really good sources. 396 00:27:06,506 --> 00:27:10,346 And then from on the foundation side, folks like the Chan 397 00:27:10,378 --> 00:27:14,762 Zuckerberg Institute have had specific calls for funding for foundational 398 00:27:14,826 --> 00:27:19,482 tools and so encouraging those efforts to continue. Those would 399 00:27:19,506 --> 00:27:24,570 be really powerful places for folks to drive attention. 400 00:27:24,730 --> 00:27:28,522 Today we discussed a small but incredibly important side of the 401 00:27:28,546 --> 00:27:33,390 reproducibility crisis. Some of the topics we covered includes 402 00:27:33,910 --> 00:27:38,494 biology is very complex and data analysis methods, therefore 403 00:27:38,542 --> 00:27:43,182 are also very complex. Even if we have full access to data sets, 404 00:27:43,246 --> 00:27:46,718 we might not be able to reproduce the data analysis portion 405 00:27:46,734 --> 00:27:50,878 of our experiments. As we discussed, a portion of the problem 406 00:27:50,934 --> 00:27:54,078 is that we don't have sufficient documentation. We often 407 00:27:54,134 --> 00:27:57,262 do not have access to those tools that were used. And so there 408 00:27:57,286 --> 00:28:02,298 is a crisis looming that we should address today. And those includes 409 00:28:02,474 --> 00:28:07,402 arguing for more funding, that our tools mature with science that 410 00:28:07,426 --> 00:28:12,522 we produce today, and that we move forward with a set of tools 411 00:28:12,586 --> 00:28:17,270 that allow us to learn from data sets we produced in the past. 412 00:28:17,890 --> 00:28:22,826 A call to action for all of our listeners. If you have a tool 413 00:28:22,898 --> 00:28:28,612 or a method that allows you to improve data analysis reproducibility, 414 00:28:28,756 --> 00:28:30,960 please reach out and let us know about it. 415 00:28:34,860 --> 00:28:38,596 We hope you enjoyed the Translating Proteomics podcast brought 416 00:28:38,628 --> 00:28:42,612 to you by Nautilus Biotechnology. To contact us or for 417 00:28:42,636 --> 00:28:47,028 further information, please email translatingproteomicsautilus 418 00:28:47,124 --> 00:28:47,300 Bio.