Transcript of the talk, from a Whisper transcription of the audio. Auto-generated: speaker names in particular are unreliable.
========================================================================

# Does AI Accelerate Drug Development: A Global Perspective

Authors: Panle Jia Barwick, Hongyuan Xia, Tianli Xia, Jing Yi  
Discussant: Benjamin Jones  
Video: https://www.youtube.com/watch?v=pwODfPHl7WY&t=948s


## Talk (00:15:48 – 00:42:36)

[00:15:48] Pamela, you ready? Okay, Pamela, go ahead. Thank you. We are very grateful for the  
[00:16:20] conference organizers to put this paper on the program. And this project is about the effect  
[00:16:26] of AI on global drug development and joined with amazing set of colleagues, sorry, co-authors who  
[00:16:33] did all the hard work. And they are here, Jing Yi from China Agriculture University,  
[00:16:40] Hongyuan Xia from Cornell, and Tian Li Xia from Rochester. Okay. So I think it's fair to say  
[00:16:48] that few forms of innovation can rival new medicines in terms of economic magnitude,  
[00:16:54] economic scale, and contribution to society. Worldwide, the annual spending on healthcare  
[00:17:02] exceeded, you know, it's about 10% of GDP and it's currently at around $10 trillion a year.  
[00:17:08] The spending on medicine is about $2 trillion and is projected to exceed $2.4 trillion in just  
[00:17:16] three years. The social returns on those annual spending are equally enormous. It's  
[00:17:22] estimated that three quarters of improvement in life expectancy can be attributed to new medicines,  
[00:17:29] you know, in 30 developing developed countries according to some studies.  
[00:17:33] However, drug development process is expensive, lengthy, and risky. So to give you a concrete  
[00:17:41] example, this is the discovery process of Ozympic, you know, a popular drug for type 2 diabetes.  
[00:17:49] In the late 80s, scientists have discovered that GLP-1, you know, gut hormone, is useful for  
[00:17:58] release insulin and could be useful target for diabetes. However, the hormone cannot survive  
[00:18:06] the bloodstream for more than a few minutes. And it wasn't until 2005 when novel Nordisk  
[00:18:16] discovered a way to re-engineer the molecules so that can stay in a human body for a week.  
[00:18:22] And this discovery quickly, as you can see, led to pre-clinical trials on animals, which turned  
[00:18:28] out to be successful, and then followed by clinical trials, the phase one starting in  
[00:18:33] 08 and concluded by phase three trials in almost 10 years later covering more than 8,000 patients.  
[00:18:41] And by the end of 2017, the drug was approved by US FDA. The entire process lasted 30 plus years  
[00:18:49] from beginning to end. There are two, you know, big buckets of the drug discovery.  
[00:18:55] The first is the discovery of a useful target and molecule and proving, you know, that can be  
[00:19:02] turned into a drug that can be manufactured and has commercial viability is difficult.  
[00:19:07] The second process of proving efficacy and low toxicity among human is equally difficult and  
[00:19:16] challenging. Nowadays, we have a large number of AI tools that can help with this process,  
[00:19:25] among each of the states I list here. You'll notice that all of those, you know, most of those  
[00:19:32] are developed after 2018, which essentially the period that we view as modern AI.  
[00:19:38] For the first part, you know, 2024, the Nobel Prize for Chemistry, one to three scientists who  
[00:19:44] discovered or helped contribute to Alpha-4-2, which can predict with reasonable accuracy the  
[00:19:51] 3D structure of the protein. And that allows scientists to see the 3D structure and  
[00:19:57] essentially the pockets to develop the molecule. And as you can see, the generative AI models now  
[00:20:03] can help scientists to come up with molecules with desired properties. It used to be that the drug  
[00:20:10] companies would spend essentially 10,000, thousands of synthetic experiments to come up with a useful  
[00:20:16] compound. And that process can be helped by the AI, you know, generative models.  
[00:20:22] And the third part, which is, you know, once the compound has the desirable property,  
[00:20:28] you also need to know the toxicity, whether it's safe, you know, for human to use. And that's what  
[00:20:34] the predictive AI models are intending to do. They're essentially predicting what would be  
[00:20:39] the toxicity for the drugs developed. And finally, the large language models are useful for  
[00:20:44] recruiting patients with desirable profiles and also to help with paperwork. And so,  
[00:20:51] because of this, perhaps not surprisingly, there are many enthusiastic industry supporters. And  
[00:20:57] as you can see here, they have identified three key margins that AI could help. This is the quote  
[00:21:04] from Incelacote, the pioneer of AI drugs that claims to AI could help with quantity, you know,  
[00:21:11] for this company in particular. It because it can reduce a number of synthetic experiments that  
[00:21:18] scientists have to do. You can now do more drug projects. And similarly, Pfizer claims that those  
[00:21:26] AI can help with speed instead of going through thousands and tens of thousands of synthetic  
[00:21:33] experiments. Now we can perhaps use way fewer. So in this particular example, the  
[00:21:39] developed process, the designing process, dropped from four years to four months.  
[00:21:44] And there are other examples arguing that not only can AI help with speed, it can also help with  
[00:21:50] success, that the AI design molecules might have higher success rates of proving safety  
[00:21:58] and pass through to human clinical trials. On the other hand, there are also serious  
[00:22:03] counterarguments. In particular, people point to the fact that AI are still not as good or  
[00:22:10] not useful for two key processes that kill most of the drug projects. First is target selection.  
[00:22:17] And second, we will actually talk a lot more about it. Second is human toxicity. AI models can  
[00:22:22] predict reliably whether a drug is toxic. And so it's a question whether AI can change drug  
[00:22:31] development at scale, at the industry scale. That's what we're trying to do here. We're  
[00:22:36] going to first compile a large dataset that covers the entire global drug development,  
[00:22:45] all drug projects launched globally from 2010 to 2025. And then we're going to develop three measures  
[00:22:53] of AI exposure or AI adoption and using three large datasets that cover firm level AI human  
[00:23:02] capital, the disease level AI scientific studies, and the target level existing evidence.  
[00:23:08] The third one is not specific to AI, but it's a very useful input for AI models. And in order to  
[00:23:16] provide some causal estimates rather than simple correlations, we proceed with several  
[00:23:21] approach. One is we're going to use extensive controls. Second, we're going to show you  
[00:23:26] the results from 2010 to 2025 using 2010 to 2017 as an early AI period. And in contrast,  
[00:23:35] that contrasts the results in 18 to 25 with the earlier results. And third, we're going to provide  
[00:23:41] a placebo. And lastly, which is ongoing experiment, we're trying adopting a shape-share  
[00:23:48] instrument to help with causality. The preview of the findings is that  
[00:23:54] doubling the number of AI employees in a firm can lead to 12% actually 13% more drug projects.  
[00:24:02] And we are going to look at how this affects firms. AI does help with the success, meaning the  
[00:24:17] drug projects can successfully progress to human trials. And according to our estimates, doubling  
[00:24:23] AI employees will increase a success rate by 4 percentage points and 6 percentage points by  
[00:24:29] 2025. And the baseline, as you can see, is 1 out of 4. We don't find significant evidence on speed,  
[00:24:36] duration, as well as later stage human trials, although we're still trying to refine our analysis.  
[00:24:44] To put all the numbers in your some kind perspective, if you have turned down, I mean,  
[00:24:50] shutting down the AI employees, changing them to zero, then our analysis suggests that since 2018  
[00:25:01] AI adoption measured by firm employees with AI skills have actually moved to phase one.  
[00:25:15] And if the success rates at the human trials are the same as before, which since we don't  
[00:25:20] find any negative evidence, this would translate to about 230 new drugs in 10 to 15 years.  
[00:25:26] And three quarters of the benefits arise from the improvement in success. And that's why this is,  
[00:25:32] you know, a very important finding. And so many of the people in this room contribute to the literature,  
[00:25:38] so in terms of time, I won't go through that. So the first part, let me just tell you a  
[00:25:44] little bit about the datasets we use. And, you know, all the people, my co-authors did  
[00:25:50] hard work. It wouldn't be possible without them. But the first one is from Reveilleo Labs. You know,  
[00:25:56] many people here used it. It's basically a resume data bank. And we're going to focus on four million  
[00:26:02] workers in the farmer industry and publications in PubMed and to identify AI papers, and then use  
[00:26:19] embedding to translate them into disease country year, you know, category. And we also use  
[00:26:25] open targets to essentially look at the existing scientific evidence for each target  
[00:26:31] at a target disease level. And we lay over the three data sources that allow us to construct AI  
[00:26:38] measures on top of the datasets from sidelines that tracks every drug project worldwide, as well  
[00:26:45] as all trials conducted after. Just to give you, I won't spend much time on the data  
[00:26:50] construction, but just to give you a rough idea of the challenges we encounter, it turned out that  
[00:26:55] existing methods, either from US Patent Office or the papers that we have access to, the existing  
[00:27:03] methods generally a lot of false positive when we think about AI papers or AI patents in medicine,  
[00:27:10] probably because NURS, your network, you know, this is, it can be culture neurons,  
[00:27:15] rather than neurons, rather than the training models. And the DL could be a chemical prefix  
[00:27:22] instead of deep learning. So what we do is de facto developing an oncology, we call that  
[00:27:29] farm AI oncology, which is a dictionary that has both the keywords and more importantly,  
[00:27:35] the context guards, that it has to be new network and training models for this to be an AI  
[00:27:41] paper. And this is developed through extensive, you know, research of the existing literature  
[00:27:47] and helped with human hand reading 2000 documents. And this is constantly refined. If you have any  
[00:27:54] good ideas, let us know, you know, this is a database that we're building. And so in terms  
[00:28:01] of the drug projects, just to give you a rough idea of how many that are worldwide in the  
[00:28:06] number, so we focus on us, Europe and China, which essentially by by and large the most covers  
[00:28:13] most of the drug development worldwide, Europe has about six, a 700 to 800 of drug projects  
[00:28:19] launched each year. And this has been quite stable. In the US in 2010, about 1000 drug projects  
[00:28:25] were launched and then increased in almost doubled by the COVID period and then recently  
[00:28:30] dropped to about 1500. You can see the astounding progress of China, which is, you know, we don't  
[00:28:36] have a lot of stuff about China in this yet. But this is, you know, one area that you can see  
[00:28:42] the progress China has made the number of drug projects started from around around 50 40 in 2010,  
[00:28:49] and now has exceeded 1400, 1300 by the end of our sample period. And because of the  
[00:28:55] success results are higher in China, China now accounts for, you know, more than the  
[00:29:01] face one clinical trials worldwide, more than more than us. In terms of AI skills, AI adoption,  
[00:29:11] or exposure, the progress is equally, you know, impressive. So this first chart graph here  
[00:29:17] shows you the red line is the so we essentially normalize everything at 2010 to 100. The blue is AI  
[00:29:27] employees and the red ones is employees in the farmer sector over a 15 year period, the  
[00:29:34] employees in the farmer sector increase about 85%. But AI employees tripled. And this is an  
[00:29:41] underestimate because we use your resume to classify AI skills retroactively. So, you know,  
[00:29:48] some of the people who later acquired AI skills will say you have your AI skills early on during  
[00:29:54] working experience. So by according to that, the AI in number of employees must have increased  
[00:30:00] larger than this. I think some of the second paper, we actually talk about it, the evidence  
[00:30:04] is pretty similar. We have three tiers of AI papers. So first one, simply statistical  
[00:30:10] learning AI papers. Then we focus on second and third, second would be the deep learning,  
[00:30:15] the more advanced machine learning tools. The third would be we call them name the systems  
[00:30:20] like Alpha 4 would be a name system. And according to our definition, there are very few AI papers  
[00:30:27] prior to 2016, which is probably, you know, as expected. And then from that time on, the  
[00:30:33] numbers just shoot up and then by 2025, China had twice as many AI papers than US.  
[00:30:42] So we're going to look at three outcomes, you know, the three key margins that industry  
[00:30:48] experts identified and then examining how the exposure affect the three outcomes. So all of  
[00:30:56] them have a similar regression design, except the outcomes are slightly different and then  
[00:31:01] the level of observation is slightly different. So the for the first one, we're going to look at  
[00:31:07] the effect of AI exposure on the number of drug projects launched by firm here. Okay,  
[00:31:15] so this is just a quantity measure. And on the right hand side, we're going to primarily focus on  
[00:31:21] the AI number of employees as the measure for now. We have two other measures in  
[00:31:27] hydrogenated test. And so we're in this case here, we control for firm fixed effects and year fixed  
[00:31:33] effects. Okay. And, and then the all as or plus on, you know, they look similar. I'm going to  
[00:31:39] present the you the coefficients event studies from plus some, but the results are quite similar  
[00:31:43] if you do other specification. Oops, sorry. Okay, goes randomly. Okay, so let me just show you  
[00:31:53] some very raw correlation. In 2010. So now we're being firms according to the number of AI employees.  
[00:32:01] Okay, that's x axis. The y axis is a number of new drug projects launched for that being. And in 2010,  
[00:32:09] there's no evidence. There's no correlation between firm AI talent and the number of drug  
[00:32:15] projects here. We this is just raw data. But there's a pretty pronounced and, you know,  
[00:32:22] positive correlation by the end of us our sample period. Farms in the highest being are going to  
[00:32:28] conduct twice as many drug projects than firms with the lowest being. And so this is the event study  
[00:32:36] where we control for firm size, you know, employee as well as firm experience and the  
[00:32:42] number of completed trials, drug projects in past. And you can see that we the green area is  
[00:32:49] what we call early AI period. That's from 2010 to 2017. And then for the second is the modern AI period.  
[00:32:58] The pre AI, the early AI, the coefficient is, you know, small noisy. And there's not so  
[00:33:04] significant pretty flat. And then you can see that effect now increases pretty monotonically  
[00:33:11] and become significant as years go by. Okay, I think this is our first evidence that  
[00:33:17] it's not just pure correlation that good firms hire people AI people and do more trials  
[00:33:23] because we don't see that evidence prior to 2017. And second, we looked for some event in the  
[00:33:31] placebo test. So firms conduct innovative trials for new drugs. They also conduct trials for  
[00:33:37] generic drugs in order to satisfy regulatory requirements, for example, proving the bio  
[00:33:42] equivalency. And here, you know, and then there's a significant overlap between firms who do  
[00:33:47] innovative clinical trials and firms who conduct generic trials. There's no evidence  
[00:33:52] that AI affects the number of generic trials. It shouldn't be. I think we take this as evidence  
[00:33:59] that we will we're picking up can just be some you know, some measure of form quality,  
[00:34:05] because there's no evidence that AI there's there's no effect here.  
[00:34:08] In terms of heterogeneity, this is where I think the you know, pretty, pretty big difference that the  
[00:34:15] effect of AI, you know, if you increase the number of employees, the number of drug increases, the  
[00:34:21] effect of AI is four times as large in China as you know, the other regions. So doubling the  
[00:34:28] number of employees are going to generate 43% more drug trials, sorry, drug projects. And  
[00:34:35] this is where we're currently doing a lot of, you know, research. It could be investigation. It could  
[00:34:40] be that China ahead of US and other regions started an industrial policy on AI as early as 2017.  
[00:34:48] And could be that in China, the, you know, the cost is lower cost conduct trials. And as  
[00:34:53] result, you know, AI becomes more effective. We find that the benefit of AI concentrates  
[00:34:59] existing firms, firms established prior to 2010, and also older and have more experience.  
[00:35:07] In terms of the scope of exploration, we find that AI does help. So the coefficients here are  
[00:35:15] indicate that doubling the number of AI employees generate 15% more, you know, the drug projects  
[00:35:23] in 15% more disease areas, specific diseases, as well as modality. Now, you're not only  
[00:35:30] conduct research in small molecule projects, you are also approaching to other modality,  
[00:35:36] large biologic projects, et cetera. We don't find evidence that AI allows firms to experiment  
[00:35:43] with more targets. I think probably because as you know, one of the critics argued that AI  
[00:35:49] can't yet help us identify targets since the existing knowledge is pretty, pretty thing.  
[00:35:57] And then so the right one is the Heavendale concentration measure, which is exactly because  
[00:36:03] you now explore or exploring more disease categories, a concentration naturally drops.  
[00:36:09] In terms of the direction, we find, so here we group the disease categories into five buckets,  
[00:36:17] where we have a lot of existing AI papers, AI studies, and buckets where they are not or very  
[00:36:23] limited. As you can see, the effect of AI concentrates, there's actually no effect  
[00:36:28] in disease areas where there's very limited existing knowledge. And in fact, concentrates  
[00:36:35] increases monotonically and concentrates in areas with a lot of existing knowledge.  
[00:36:41] And this true externally and internally. So if the firm has conducted a trial or  
[00:36:48] drug projects in a disease category, the effect AI is more pronounced in areas where the drug has  
[00:36:54] expertise. The second, as I have already alluded to, is an impact on success that  
[00:37:01] eventually those drug projects can progress to human trials. If you look at the raw data,  
[00:37:09] again, very similar to what I showed you for quantity, in 2010 there's no evidence that AI is  
[00:37:16] associated with a higher success rate. By the end of our sample period, firms in the highest  
[00:37:22] bank of AI employees are three times as more likely as to enter the success rate, three times as high  
[00:37:30] as firms with the lowest bank of AI employees. And this is an event study where we look at  
[00:37:36] every drug project launched globally over the past 15 years. So the unit analysis is drug project.  
[00:37:44] And we control the most demanding specification of firm disease pair fix effects, disease year  
[00:37:51] fix effects and country year fix effects. Defecto, we're looking at within the firm  
[00:37:57] disease pair as a firm accumulates more AI talent, whether or not the drug project success rate  
[00:38:05] improves. And the evidence is yes. Prior to 2017, there's no evidence of coefficient small  
[00:38:12] and insignificant and increases monotonically over sample period. By the end, doubling the number  
[00:38:18] of AI employees will increase the success rate by six percentage points over the base of 24  
[00:38:25] percentage points. So that's a very significant improvement about 24%, 25%. Similar to what we  
[00:38:31] find before, the effect AI is more pronounced in areas where we have more knowledge in particular,  
[00:38:37] the success, the effect on success rate is higher in disease areas with more AI papers,  
[00:38:44] where the, and also more internal expertise where, you know, essentially if the firm has  
[00:38:50] conducted long drugs in a disease area, then the effect of AI on success is more pronounced.  
[00:38:57] In terms of heterogeneity, we don't find that China has any edge over other regions. The success  
[00:39:03] rate is very similar, but the other impact of AI on success. We also find that AI seems to help  
[00:39:18] the, you know, more complex drugs, the non small molecule, although the effect is not  
[00:39:24] still different from the impact on small molecule drug projects. We look at other  
[00:39:33] outcomes. As I mentioned, we don't find that AI affects the duration. So there's not a big impact  
[00:39:42] duration or speed. We also don't find that AI affected the later phase one, phase two,  
[00:39:49] phase three trial success rate. But we're still, you know, I think arguably the data  
[00:39:53] would be a lot thinner since it takes years for the trials to complete, but we're still trying  
[00:39:59] to refine our analysis there. And, you know, the results, at least the ones that I showed you  
[00:40:05] are pretty robust to one minute remaining time. We want to put all the estimates together  
[00:40:18] into a measurable outcome, which is how has, have AI tools helped us with generating, you know,  
[00:40:25] useful drugs. Okay. So the first one is we're going to look at the, the number of drug projects that  
[00:40:34] successfully progress into human trials between 2018 and 2025. If we have, you know, with the  
[00:40:42] current number of AI employees versus when all the employees are shutting down, shutting down,  
[00:40:47] you know, turn to zero. And there are two margins. As I mentioned, first, we are now  
[00:40:52] having more drug projects. And second, the drug projects are more successful. Putting all together,  
[00:40:58] AI has helped contributed to roughly 1400 more drug, drug projects and into phase one.  
[00:41:14] And the success rates from phase one to phase three of those projects remain the same as what  
[00:41:18] we have seen in the past, then we can expect about to see 233 new drugs in the next 10 to  
[00:41:24] 15 years. To conclude, we find that at the industry level, we find evidence that AI boosts drug  
[00:41:34] development. In particular, we find that firms with more AI talent, much more new drug projects  
[00:41:41] and conduct more exploration in different disease areas and with different modality  
[00:41:47] and also achieve higher success rates. The effect was insignificant prior to 2017 and increases  
[00:41:56] monotonically and becomes much more sizable afterwards. Also, we find that AI works better  
[00:42:04] when there's existing knowledge. The reason that the AI hasn't helped us venture into new  
[00:42:10] targets is because there's just not a lot of biological evidence in new targets. And that's  
[00:42:15] where AI has very limited impact. And adopting AI essentially accounted for 1400 more drug entries  
[00:42:23] into phase one and then ongoing work, working with instruments to help with our cause inference  
[00:42:32] and working with more detailed welfare analysis. Thank you.  

## Discussion (00:42:36 – 00:56:25)

[00:42:36] It's great to be here. Thanks to the organizers for including me and to Penley and your co-authors  
[00:43:09] for this very interesting paper, which I will start discussing shortly. Good. Okay. Here we go.  
[00:43:18] Penley Hong-Wen, Tenley, and Jing. This is a very, very nice paper. It's still early stages.  
[00:43:24] So maybe the results will change. But let me talk quickly about what happens in this paper and  
[00:43:29] then make some broader comments. So the question is really broad. And it does AI or the  
[00:43:35] conclusion of AI in some sense as an input, raised pharmaceutical R&D output. The context is global.  
[00:43:42] So it's drugs everywhere over a fairly long period, 2010 to 25, which gives you a pre-modern AI and  
[00:43:49] post-modern AI window, which is nice. And importantly, this is all types of AI. So we'll  
[00:43:55] talk more about that in a second. It's not a particular tool. It's kind of anything goes here.  
[00:44:00] The main outcomes are drug starts at the firm by year level and also drug progression. That's if you  
[00:44:05] get into pre-clinical as a pre-clinical candidate, do you at least migrate into phase one? And there  
[00:44:11] are important results on both. The R&D inputs are AI employees. That's kind of the main one,  
[00:44:17] which is a count at the firm year level. So instead of looking at, say, R&D spend on AI or  
[00:44:22] something at a firm level, what they can do from a Velo is just look at kind of coding  
[00:44:26] the number of employees, which of course is a pretty good thing to do. That is pretty  
[00:44:30] close to expenditure in some sense and a meaningful input. They also have other evidence about AI  
[00:44:35] methods in biomedical papers by disease year. So you can see where AI seems like it's  
[00:44:42] being more used and relate that to where the firms go, who have AI employees, kind of  
[00:44:47] complementarity there. And they also have evidence on how well known, I guess,  
[00:44:53] the molecular target disease link is. I wasn't quite sure how they measured that from reading  
[00:44:56] the paper at this point, but that's also a piece. The main findings, there's a lot of findings already,  
[00:45:06] but the main findings that she talked about is that drug starts, kind of the sensitivity of drug  
[00:45:10] starts to how many AI employees you have is sort of like zero, and then it starts going up. In  
[00:45:16] other words, and this is kind of confirmatory, not just that it's going up, but that wasn't  
[00:45:20] going up before. So I think this kind of an interesting feature of this paper is that  
[00:45:24] early period you don't see anything and then you do within the same set of firms, etc.  
[00:45:29] That's kind of how many drugs you try kind of top of the funnel. So in some sense, these AI  
[00:45:34] oriented firms are putting more drugs in the top of the funnel, and then do you get from  
[00:45:38] preclinical into a phase one and the successor, I thought this was really quite striking.  
[00:45:42] That was also flat in the kind of pre period, but today having more employees and AI means  
[00:45:47] you're much more likely to migrate a preclinical candidate into an actual phase one trial. So  
[00:45:54] that's kind of interesting. So it's not just you putting more in the top of the funnel, you're  
[00:45:57] kind of getting through with higher quantity into this phase one stage results like a still to come.  
[00:46:03] No signs yet that you're getting more to phase two or anything, but we'll do TBD,  
[00:46:08] I think with some more time. More diseases but same targets, that's important. This is  
[00:46:14] not new molecular targets. So it's kind of I'll talk about that. There are complementarities,  
[00:46:19] you get more of an AI effect in areas where there's more disease related AI biomed research,  
[00:46:23] which is sort of an interesting complementarity. And if a firm has more experience with that  
[00:46:27] drug area, you get stronger effect as well. But also all of this, you get more in the  
[00:46:32] funnel more through the funnel, but nothing's falling faster through the funnel. So you're  
[00:46:35] not like accelerating for any given drug, but you're getting more stuff going and going and  
[00:46:40] getting through. So think about it that way. And you can make a calculation as they did at the end  
[00:46:44] about kind of the marginal gain potentially from from AI, which is a lot of other drugs and  
[00:46:50] not only just starts of drugs more get through the first part of the funnel. So you're going to  
[00:46:53] end up with a lot more drugs kind of on the backside of this, if even at the sort of normal,  
[00:46:59] you know, nine out of 10 drugs fail or something, whatever happens after phase one,  
[00:47:02] you're still going to have a lot more throughput from this. Okay. So the emerging picture  
[00:47:06] is you get kind of a wide instead of pipelines against known molecular targets.  
[00:47:11] There's an increased success rate, at least at early stages. You do not seem to accelerate speed.  
[00:47:17] We don't know about phase two and phase three yet, but you know, TBD,  
[00:47:21] altogether a large expansion and kind of drug throughput on the quantity dimension. Okay,  
[00:47:26] which is pretty exciting. Now, strengths of the paper, I think, you know, the first thing  
[00:47:31] when I first think about this problem, and I thought about it a little bit is like, boy,  
[00:47:34] this is complicated because there's AI as you showed is in so many parts of the drug discovery  
[00:47:39] process, which use different AI is which AI, what kind of AI summer specialized summer large language  
[00:47:44] model summer something else, there's Alpha fold, you know, these are very distinctive tools.  
[00:47:48] So how do we think about even AI in this process? And so this is kind of like a good  
[00:47:54] macro oriented approach. You see like, there's so much going on kind of in this process,  
[00:47:58] we're going to look at kind of front end input, which is like R and D effort,  
[00:48:01] and we're going to look at what comes out the back end in terms of like phase one trials.  
[00:48:04] And I think that's actually a really good way to go. We also want to studies on every individual  
[00:48:09] tool in a sense, but this kind of allows us to kind of cut through that, and say something kind  
[00:48:14] of really big about what's happening in drug discovery overall. CF compare with Alpha fold  
[00:48:20] studies like some done by Josh and others, Jerry's here, right? You know, his paper. So we've  
[00:48:26] seen really nice papers looking at Alpha fold like what that does, but of course,  
[00:48:29] there's only one very, very early upstage part of potential drug pipeline. And so this is kind of,  
[00:48:35] you know, in some sense, the strength of that is you can really learn something in real great detail  
[00:48:38] about that. The weakness is it doesn't really talk about AI and drug discovery kind of overall.  
[00:48:43] And so then these are very complimentary approaches. The results are rich. I love the  
[00:48:47] set of outcomes like the number, the direction, the type of drug progression rate, speed,  
[00:48:52] and then multiple dimensions of inputs, mainly R and D effort, but also kind of  
[00:48:56] intersections with opportunity based on basic science papers on AI, et cetera. And overall,  
[00:49:02] I mean, I look at the where things are now and I think this kind of is coherent and it just feels  
[00:49:06] coherent. The pre-2018 and now the acceleration generics as a placebo look like a placebo.  
[00:49:14] The results are sort of natural. There's nothing that I didn't see any kind of tension  
[00:49:18] when I looked at it like this doesn't make sense because it's sort of it doesn't work  
[00:49:22] with the other figure or something. I felt like, okay, this seems like a pretty coherent story.  
[00:49:27] I think the main challenge, but I mean, you're working on this, is that, you know, firms that  
[00:49:31] are hiring a lot of AI R and D workers are doing that for some, they're making a choice.  
[00:49:37] Why are they making a different choice than other firms? Do they see, you know,  
[00:49:40] the most obvious endogeneity concern would be sort of common joint AI opportunity shocks  
[00:49:45] at the firm level. Some firms may be more able to jump on those or more see them better,  
[00:49:50] they may have better managers. They may have also, why are these firms doing this? They're also really  
[00:49:53] accelerating in drug development. It was a little hard to pin causation on AI here,  
[00:49:57] but they're going to do a shift share IV, which I think would be a reasonable  
[00:50:00] thing to do and then maybe push a bit on the causal, the causal interpretation.  
[00:50:05] So right now I think of it more as a descriptive result. Okay, so all right, so now some  
[00:50:09] comments, broader comments. So, you know, this is broadly about using AI as an research  
[00:50:18] and discovery tool, which, you know, in the cognitive realm, I mean, in R and D, there's sort  
[00:50:24] of more cognition and less physical tasks compared to the real economy. One might think that AI will  
[00:50:28] have its effects first or strongest in the R and D process for that reason. One might also think  
[00:50:35] that if you can, you know, increase the rate of throughput for a dollar, spend on R and D,  
[00:50:39] and you get more delta, delta gain out of that, and just because how we think growth goes in an  
[00:50:44] advanced economy, that is first order. It's more than first order. If we can accelerate productivity  
[00:50:50] growth, we're going to accelerate health gains. This may be in terms of social welfare implications.  
[00:50:54] It's, it's biggest and the most immediate. And so the question is like, you know, why do we think  
[00:50:59] AI can help us do that? And so the natural perspective that we see in a lot of modeling  
[00:51:04] of AI, including my own modeling, is we think of a bunch of tasks, right? And then you have  
[00:51:09] AI sort of taking over some tasks, but not others. And then, you know, it can be much  
[00:51:12] better or much cheaper at a given task. And so we as R and D workers in this context can reallocate  
[00:51:17] our effort to the remaining bottleneck tasks that AI can't do. But then we, you know, with a given  
[00:51:21] amount of wage wages, we can sort of do more of those tasks. And so you could have, it's very  
[00:51:25] natural to think you would get what more drugs in the pipeline, more going through. This  
[00:51:30] would be exactly the kind of thing you might expect if we're automating things. But on  
[00:51:35] the other hand, you know, maybe the big question is, and an open question is how creative  
[00:51:40] can AI be? So one of the main tasks, right? In some sense, you're going to go out somewhere  
[00:51:44] on a map and pick a point to look at, and you're going to do a bunch of execution around that  
[00:51:47] point and see how good it is, right? And clearly, whatever happens at that point with all the  
[00:51:52] execution is depends where on the map you happen to be. It's the creative part that's  
[00:51:56] going to define what you can discover where you choose to go, what the ultimate limit is  
[00:52:01] on the quality of a drug or whatever else in terms of anything. And so I think, you know,  
[00:52:05] among all those tasks, when we think about R&D, kind of the creative direction task where you  
[00:52:10] look, it seems preeminent, or at least it's very strongly complementary and maybe a serious bottleneck.  
[00:52:16] So my metaphor there, I can't go back. I like to think of R&Ds, we're like on a hallway with  
[00:52:23] an infinite set of doors that we could try, and we have to choose which doors to try.  
[00:52:27] And sort of which door you try is clearly determinative of the outcome,  
[00:52:30] of what will it be a broom closet, will it be a hallway to a palace.  
[00:52:34] We have to choose, right? And exploit versus explorer. So as exploit says, you know, look where  
[00:52:38] you are on the hallway, where you have some wisdom, explorers to run down the hallway to a  
[00:52:42] completely new area, which may afford great opportunities, but also might be much more  
[00:52:47] uncertain what you will find. And so when you think about this as sort of the creative classic  
[00:52:50] canonical creative search problem, the question is specifically what would AI do?  
[00:52:55] Now, if you think about creative search, the kind of combinatorial possibilities are infinite,  
[00:53:01] essentially, very fast. It is not, I could prove this to you later if you want, but no computer  
[00:53:06] is going to brute force compute even the tiniest zero percent of molecular possibilities you could  
[00:53:12] consider for anything. Just the number of combinations in a small molecule will quickly  
[00:53:15] overwhelm any computer you could possibly imagine. So you have to have some wisdom  
[00:53:19] about creators judgment. There's something we could call it a world model. I like to call  
[00:53:23] it creative judgment. There's some function in your head, which is like an imperfect map  
[00:53:27] of the landscape, places you have not traveled. But you're thinking, well, maybe there's a big  
[00:53:31] mountain over there. And if I go down there, I might find something really important.  
[00:53:35] Now, if you think about this paper, I think we're seeing a much higher quantity, but also a  
[00:53:43] much higher progression rate. It's like AI is guessing well. That's also consistent with very  
[00:53:47] narrow exploitation of where you already are. It's like you're doing the low risk  
[00:53:51] things around what you already know it's existing targets and making small edits.  
[00:53:55] Yeah, they're likely to work, but they're probably not major breakthroughs.  
[00:53:59] Does that make sense? Whereas explorers to go much further down the hallway,  
[00:54:02] but that requires you to probably pick a new target, have a different understanding of what  
[00:54:05] is going to be called, but they might be much more marginal quality drug improvements.  
[00:54:17] And AI is not solving this. As you said yourself, maybe it's kind of the deeper  
[00:54:20] exploratory problem of say new molecules. And so I think that may be the really the  
[00:54:24] bottleneck that's going to hold. It holds us as humans back, it holds AI back. So I think  
[00:54:29] these world models are very, very important because they tell us about the landscape.  
[00:54:33] Some areas of research like physics, you know where you have the so-called  
[00:54:37] unreasonable effectiveness of mathematics, very simple equations or at least single equations  
[00:54:42] can tell you about the universe far away. Quantum, the quantum model explains everything  
[00:54:49] out of sample. We keep building bigger accelerators and we find exactly the  
[00:54:53] particles that it said we would and they have the same properties the model says.  
[00:54:57] Cosmology, force equals mass times acceleration, you go a long way on building anything you want  
[00:55:01] to build on that on Newton laws of motion, right? They're incredibly powerful world models.  
[00:55:13] And then we go build it. Biology looks to me more like the thing on the right,  
[00:55:18] which is we don't have a world model. There aren't, there are heuristics,  
[00:55:21] there's some funny ones, but we don't really know. And it's partly because I think  
[00:55:25] genetics are very sensitive and fragile. That's why I have toxicity, small edits in a protein  
[00:55:29] and suddenly it's doing really a lot of damage to you. You have off target hits in your body.  
[00:55:33] We don't have a world, we don't even know what most proteins do, complicated interactions in your  
[00:55:37] cell. This is a really choppy landscape. And so I think we've often been stuck  
[00:55:42] with a lot of local exploitation and I think it's quite hard to find exploratory things.  
[00:55:46] So partly it's that the AI is to some extent can, it may be really smart,  
[00:55:51] but it's contained within the world models that we know and it can train on.  
[00:55:55] And we don't really have them in biology. So it's going to have to discover them  
[00:55:58] if they exist, but it made a deeper level be that nature doesn't provide really good world models  
[00:56:02] in some areas of search. And I think that's probably also true in biology. So it's  
[00:56:13] diseases, I'm not saying it's not, but I think this exploratory problem remains really hard,  
[00:56:18] even if you're really, really smart. And so I think that's probably, and that's also  
[00:56:21] consistent with what I'm seeing at least in the first sets of results that you have.  

## Q&A (00:56:25 – 01:10:22)

[00:56:25] Okay, thanks. Oh, we'll take questions. I'm gonna, I'm gonna take the privilege of the first  
[00:56:39] question. And then we'll turn it over to Lee and Dan. Okay, so I noticed three facts in the talk  
[00:56:46] that I think together are really interesting. The first one is that China is now about a third  
[00:56:52] of I think it was phase one development. And even 10 years ago is essentially zero. Okay, so  
[00:56:58] the second fact was that at the margin, AI employees at Chinese companies are no different  
[00:57:07] at converting research into productive drugs than AI employees in Europe and the United States,  
[00:57:16] no better, no worse. The third was I didn't see the fact on what fraction of AI employees are in  
[00:57:24] Chinese companies versus elsewhere, but I did see that in terms of publications, China is now  
[00:57:32] something like half of all output. And so that means that clearly China is more of global drug  
[00:57:41] development. And it seems like, you know, if we take that last fact that AI is going to be a catalyst  
[00:57:48] for that to potentially even increase, because China seems to be betting on AI and drug development  
[00:57:54] at least as much as everybody else. Given that there's a question on what does that mean for  
[00:58:01] global welfare? And also, how do we think about welfare in, let's say in the United States,  
[00:58:09] in the context of Chinese innovation in drug development getting so much faster?  
[00:58:16] Lee, go ahead.  
[00:58:17] Yeah, great paper. Another great paper from, you know, Ponla, such a great economist.  
[00:58:23] So my concern is that in the United States, at least AI will both give it and take it away,  
[00:58:28] right? So it gives in the sense that it seems like AI might be modestly increasing  
[00:58:33] research productivity, but it take it away in that in US venture capital markets,  
[00:58:38] people can only be so excited about anything at one time. And at the moment, I feel like  
[00:58:44] generative AI is this like black hole that's sucking in a huge amount of human resources  
[00:58:49] and potentially limiting new entry into biotech and new investment into biotech.  
[00:58:56] And even if AI makes a given amount of investment modestly more productive,  
[00:59:01] is it in fact enough to, you know, account for that? And I don't know if that's just a question  
[00:59:06] for Ponla and her team or, you know, a broader question for people who know more about biotech  
[00:59:11] trends than I do. Okay. Dan, then Ajay.  
[00:59:16] I wanted to ask an organizational question. So I also started working in this area. And  
[00:59:20] one thing that was very clear early on is a lot of AI was being done by startups  
[00:59:24] at the beginning. And they were always complaining, we don't have access to the data that the big  
[00:59:29] big biopharma has. And biopharma is self contracting out to contracting out to us.  
[00:59:35] What do they subcontract out to us? They got the stuff that's impossible for them to do. They  
[00:59:39] can't figure it out. So the projects we're doing, we're the first in we're doing the hardest  
[00:59:44] projects and organizationally we're challenged. And now you're seeing this rise, you know,  
[00:59:50] let's ignore the pre period. I think that's you're right. Let's think post, you know,  
[00:59:54] post DL deployment 2016. And you've seen this incredible rise. So I guess my question is,  
[01:00:01] is that really organizational change that we're starting to see more in house AI is  
[01:00:06] being combined with domestic resources like the data? So it'd be interesting to unpack that.  
[01:00:14] Okay. I guess just because the timing of all this, your paper on the dependent variable side is focused  
[01:00:28] on new candidates and number of new candidates and making it to phase one. And you know, and you  
[01:00:40] showed a lot of all the different ways that AI is in every you know stage or having an impact.  
[01:00:47] And then amongst all that you had the quotes and the one that stood out to me was the Novartis CEO  
[01:00:53] who said, this is all much to do about nothing. It's not going to have any impact on overall  
[01:00:59] output. So that begs the question of, you know, what does he know that we don't know  
[01:01:04] from in your data? And from the little bit I know of this that the I thought the primary  
[01:01:14] bottleneck in this process was phase two and phase three clinical trials. And in particular phase three  
[01:01:20] and within phase three, particularly finding patients to do to like that fit the criteria  
[01:01:27] to do the clinical trials like finding patients as the bottlenecks. No matter what you do upstream,  
[01:01:32] everything comes to a screeching halt because you got to find the humans. And so my suggestion  
[01:01:40] to you is I've started seeing applications of machine intelligence for accelerating the  
[01:01:49] identification of patients. The one that the specific one I've seen is in gastroenterology  
[01:01:54] and so it's AIs that watch colonoscopy videos and find polyps. So it's a polyp detector  
[01:02:05] and and then identify the people that have the conditions that match what the what the  
[01:02:12] and that can significantly accelerate the finding of patients. And so it may be that like something  
[01:02:18] like that is the unlock that then you know accelerates the whole the whole pipeline.  
[01:02:27] Okay, over here and then Kevin and then Wei Young.  
[01:02:32] So I so from a venture I'm a venture capital investor and so from a venture capital point of  
[01:02:39] view I think you know so one thing for us you know so we talked to a lot of biotech startups  
[01:02:47] and a lot of biotech venture investors and so the the the issue here is that discovery is not  
[01:02:55] has never been the FAMAS core issue. So I think on the margin I think AI can help a lot on  
[01:03:06] for example writing clinical protocols, drafting NDAs, faster, screening,  
[01:03:11] adverse events. But the core problem is that you know someone just mentioned that  
[01:03:17] phase two, phase three actually phase two if you look at data actually phase two  
[01:03:22] um 70% failure rate. So once you know a drug first enters a disease enters a human body  
[01:03:30] and in a disease test you find that in phase two 70% failure rate and um and right now AI cannot  
[01:03:38] predict how individual patients would respond to a drug. That's like a world model problem like  
[01:03:44] so I think you know for AI so especially you know to a to a FAMAS core issue is that  
[01:03:49] whether I think AI can help developing the world model I think you know that's a key issue because  
[01:03:55] when we first invested in Anthropic you know we saw the contract Anthropic Sign with  
[01:04:00] with Novadisk and it's 50 million contract. So we were interested in you know how does the  
[01:04:08] collaboration between Anthropic and Novadisk work and so clearly you know what the response  
[01:04:14] from you know the big pharma is that okay so just like you know what I said before so they are  
[01:04:19] they are actually using you know AI to help with their um you know all these clinical protocols  
[01:04:24] and everything to basically make the discovery process faster but to the point of you know to  
[01:04:31] the point of um phase two phase three clinical trial right now it's not uh you know it's it's  
[01:04:38] still I would say mystery. So that's why like for Anthropic Open AI they want to enter you know  
[01:04:44] biotech industry they want to get revenue from let's say a share of revenues for a drop pipeline but  
[01:04:52] all these FAMAS they decline because they say hey I don't think you have got enough contribution to  
[01:04:57] our you know drug development so yes we would sign a contract with you pay you know 20 million  
[01:05:02] 50 million dollars for for our usage of your your cloud tokens that's perfect but in terms of  
[01:05:08] you know drug discovery eventually like development you know that it's something that you cannot  
[01:05:13] help with. Okay thank you Kevin. Um yes so super interesting panel uh and team um I think maybe the  
[01:05:21] tenor of these questions too suggests something to look at going forward which is like if you talk to  
[01:05:26] the people in the labs and they're thinking about 2030 uh taking a jace book here um like they're  
[01:05:34] basically like how do we solve the phase two phase three thing is like we just build a  
[01:05:38] silicone model of human protein interactions or or even in the short run we can focus on things like  
[01:05:46] when you when you go from ozempic to modern modern versions of GLPs we're basically doing  
[01:05:50] like biomedical engineering on proteins not breaking down inside the bodies not even new  
[01:05:54] molecular targets in the first place. So like it's not obvious that humans have much of an  
[01:05:59] advantage over AI on either of those tasks and like if I'm looking for like where do I expect  
[01:06:06] AI to be able to overcome bottlenecks and simply a growth model I would expect it on  
[01:06:10] things that look more like that rather than like an AI researcher inside Novartis trying to  
[01:06:17] trying to predict targets. Um and I think that your data can help with that because  
[01:06:22] you should be able to like if I like in in your original slide where you're breaking down  
[01:06:26] the process into where we expect every bottlenecks and not like if you can say like it seems like  
[01:06:31] as of 2026 these two areas are completely bottlenecked and something like Ben's models  
[01:06:37] you just don't get any additional drugs in the end anyway and so like who cares about everything  
[01:06:42] until that point the question is like when are we seeing any green shoots of AI contributing on  
[01:06:47] those remaining bottlenecks and and how can that happen um and just the final thing is you might  
[01:06:52] even think that requires changes on the non-AI side like the regulatory side so one story you often  
[01:06:58] hear in China is that um like trials are just much quicker and so a lot of even western firms  
[01:07:05] will do some trials in China for this reason and if I'm gonna have a thousand new candidates come  
[01:07:10] through the pipeline like it's literally impossible right okay so we've got about negative one minute  
[01:07:15] we've got two questions lined up and uh panel wants to go so really quickly Wei Young uh and Jerry  
[01:07:20] if you've like 30 seconds or less and then we'll give you a chance to respond uh yeah uh  
[01:07:27] something that would be interesting to see is a disease burdened by ethnicity and whether  
[01:07:31] drugs coming out of China or are they targeted differently or yeah yeah a quick thing for the  
[01:07:38] human part even though phase two is too early to see you can still see the disease groups that  
[01:07:43] can be mapped to human versus mice like Alzheimer's cannot test on mice but liver  
[01:07:48] can can non-human and for the AI employees I'm not quite sure whether you're marrying here  
[01:07:54] is more about the bioinformatics or those people who are more good at wet labs because if AI  
[01:08:00] complements or substitutes these two skills differently they may have different implications  
[01:08:05] as well okay um I'll give you a minute yes um those are amazing questions let me just address the  
[01:08:16] I think the recurring theme what we're showing is more projects entering into phase one and we're  
[01:08:24] saying in 10 to 15 years you're going to see new drugs but not before precisely because it takes time  
[01:08:31] for the projects go through phase one two three trials and this aligns with you know the CEO's  
[01:08:38] prediction that in 10 to 15 years that you won't see anything until after I think the the time  
[01:08:43] frame is actually consistent second um the in terms of whether it's lower quality the fact  
[01:08:49] that AI projects have a higher success rate indicate that's unlikely to be true if they are  
[01:08:59] marginal then you know I would have and we would have anticipated perhaps trial phase one phase one  
[01:09:13] two three trials meaning so that also indicates maybe quality we can't be on quality measure we  
[01:09:19] can do a little more you're using some you know more innovative mechanism the first we find  
[01:09:26] the first combination in class which also indicates you know some quality so I think on quality we can  
[01:09:31] look more and lastly why China um uh you know I think we're trying so the effect on the world we think  
[01:09:39] this is actually positive more drugs more acute both diseases and better for everybody um and it  
[01:09:45] is true that China's you know in 10 years have made astounding progress but there are several  
[01:09:51] reasons their healthcare program just had a enormous you know policy change starting in 2016  
[01:09:57] and the the government has poured a lot of subsidies for innovative drugs and so I think all of them  
[01:10:11] the you know skyrocketed and talent going back to China and all of them played a role  
[01:10:18] I want to add anything we're we're at time so thank you thanks so much okay uh next paper