Transcript of the talk, from the video's captions. Auto-generated: speaker names in particular are unreliable. ======================================================================== # Do Online Job Postings Capture Job Vacancies? Authors: Michael Dalton, Lisa B. Kahn, Andreas I. Mueller Discussant: Gad Levanon Video: https://www.youtube.com/watch?v=HSCV6-Vfc4U&t=12500s ## Talk (03:28:20 – 03:50:17) [03:28:35] Jesus. >> Okay. Hi everybody. Welcome back to our last and third session of the day on job openings, occupations, and technological [03:28:49] dynamism. I'm Dialin Rier. I work at Emery, the economic measurement research institute as well. Uh, I'll be moderating. So, first up is going to be Lisa Khan on do online job postings [03:29:02] capture job vacancies. >> Yes. [03:29:06] >> Good afternoon everybody. Um, I'm Lisa Khan. I'm from University of Rochester and we're really excited to be uh on this program uh and hearing from all of you uh who uh are the experts. Um this [03:29:20] is uh a joint work with Mike Dalton from the BLS who's here somewhere and uh Andy Mueller who is from Zurich. And um we're interested in measuring vacancies. [03:29:31] Vacancies obviously uh take a central role in the in the measurement of the health of the uh labor market. Um we can use vacancies to try to understand employer behavior in response to incentives. Vacancies help us think [03:29:44] about job finding rates and our critical moments in search and matching models for instance. And vacancies have increasingly been posted online which raises the possibility that we can harness this public information [03:29:59] at more detail. Um and so there have been uh a number of aggregate data products using job vacancies that are posted online to measure vacancy series as well as an explosion of uh researchers using the underlying micro [03:30:14] data for a wide range of super interesting questions. Um and for good reason because the data afford a number of advantages. Uh they can be uh incredibly detailed if you harness all of the information in a job ad. you know [03:30:27] who's hiring, who the employer is, what the job is, what the skills and tasks required are, and a range of other uh important and useful information. Um, you also tend to have a great deal of breadth for studying questions at highly [03:30:41] disagregated detail. For instance, analyses within firm or within local labor markets. Um, and the data are uh timely. Usually, they're available in roughly real time, which has been very helpful uh when the economy is changing [03:30:55] rapidly. for instance, during COVID is one example. Um, and so I may be biased, but the data set that I'm going to talk about, the Burning Glass data, it's just my favorite. Uh, no offense to anyone in the room, it's a favorite data set. But [03:31:09] that's not to say it's without potential shortcomings. Um, we actually know very little about the representativeness of jobs that are posted online. For instance, we know nothing about the employers that post them. And then all [03:31:22] we can do uh until this project it was compare at aggregated levels like industries, occupation levels, geographic levels. Um we also don't know how to convert a posting online to our [03:31:36] um concept of a vacancy that we're more familiar with. So for instance, how many jobs does one online posting actually correspond to? Um and so in this paper, we want to answer these questions and we're going to ask how well do online [03:31:48] postings capture vacancies? So what we've done is we've merged um burning glass job vacancy data now collected by LCAST. This is one of the leading um data providers that scrape uh [03:32:01] jobs that are posted online and code up a rich degree of detail. They argue that they um have uh the near universe of postings uh for the US and a number of other countries. We've linked these to business registry data in the United [03:32:15] States uh based on employer name and location. And this is going to give us uh a range of employer characteristics that are not available in the burning glass data alone. But then it's also going to give us access to Bureau of Labor Statistics surveys such as the job [03:32:30] openings and labor turnover survey which has representative information on hires and openings and unfilled vacancies. Um, so we conceptualize a multiplicative adjustment factor that relates a burning [03:32:44] glass posting to a jolt's opening at the firm by location level. And that is going to allow us to both describe variation in the relationship between the two data sets both over time and [03:32:57] across employer characteristics. Um, and it's also going to allow us to offer a reweing of the burning glass data if you would like your analyses to be representative of the types of firms that have openings as captured in the [03:33:11] jolts. And so we're going to use these weights to assess the impact of making burning glass more representative for previous researchers uses of the data. [03:33:25] kind of um the the big picture punchline is going to be a bit of a white pages yellow pages story. Um large firms are very well represented in online postings and most workers are at large firms and so actually it's going to turn out empirically that the distribution of [03:33:39] burning glass postings is well representative of the distribution of openings in jolts. [03:33:46] Um but on the flip side small firms are not well represented in burning glass and most firms are small. So if you want to say something about the distribution of firms that are hiring, um the burning glass data is going to is going to [03:34:00] misestimate that. And where that really comes in is in measures of um labor market concentration that is really sensitive to sort of a missing tale of small and mid-size firms that aren't really captured. [03:34:11] Okay. Um so uh the first step is merging the burning glass data to the quarterly census of employment and wages. Uh Mike Dalton likes to say that this is two years of his life that is now relegated [03:34:26] to a footnote. Um and indeed I am not going to talk about it here. Um I think we do a pretty good job with our match. [03:34:33] We are matching based on employer name and location. So one limitation is we do have to restrict to job postings that have that information which is about 70% of postings. Some postings don't have an employer name because they're posted by recruiting firms and we can't say [03:34:47] anything about those. Um but among the set that we do try to match, we're able to match uh more than 90% of postings and we've done quite a number of uh validations of this this merging and we [03:35:00] think it does a pretty good job. And that means that the concept that we're thinking of is something that's pretty close to an establishment. It's a firm location pair. Um but we don't have establishments in the job postings data. [03:35:12] And so we are going to uh map this sort of composite establishment to the establishments in QCW sort of just by proportioning by portioning out postings across the various establishments within [03:35:25] a name location pair. And we do a couple of other cleanings um that were based on our our checks of how good a job we were doing on this merge. [03:35:34] Once we have this merge, we have a wide range of we have some firm characteristics, age, size, industry, total wage bill, a few others. Um, and then we also have the job openings and labor turnover survey. Now, in the [03:35:47] jolts, we have representative information on hires and the stock of unfilled vacancies at the end of the month. In the burning glass data, you can think more that as more of the flow of all vacancies um that over the course of the month just restricted to the ones [03:36:02] that burning glass captures. So we need to convert uh the jolt into a concept that's more similar to this flow of all openings. So we're going to use this law of motion that I put up there where n is the flow of new openings in the jolts [03:36:17] and that just equals hires over the month plus any net new unfilled vacancies at the end of the month. J is the unfilled vacancies um at the end of the month. Um and so what that means is when we're relating the two series, [03:36:30] we're going to be relating burning glass openings to the sum of recruiting behavior, which is both hiring and unfilled positions. And so it's that n concept that we're going to be relating to burning glass job openings. [03:36:44] Okay. Um so uh our adjustment factor relates the two uh uh n again is the flow of uh new openings in the jolts and b is the flow of new openings in burning [03:36:57] glass. We're going to relate these two through a multiplicative adjustment factor that's a function of three components which are interesting um uh for researchers. And then the aggregate component is also interesting. Um you [03:37:10] can think about a firm as a type J where a type could be size bins, age bins or some composite of several characteristics. And our adjustment factor is going to be a function of the extensive margin in blue which is uh the [03:37:25] probability that a firm posts online given that they're recruiting. So given that they show up in jolts with openings, do we see any openings posted online? And that's the denominator there. Our intensive margin is [03:37:39] conditional on appearing in both data sets. So conditional on recruit showing up as recruiting in jolts and conditional on on having vacancies. How many openings does one online posting correspond to? That's the intensive [03:37:52] margin which is the n there. And then the final component of the adjustment factor is a nuisance. We think of it as a nuisance parameter that's likely driven by measurement issues. There are some cases where we have um postings in [03:38:06] burning glass that do not correspond to recruiting behavior in the jolt. So some firms just have postings online but they're showing up with neither hires nor unfilled vacancies. Um and this might be because of measurement error or time aggregate in measurement error in [03:38:20] our merge or time aggregation issues. Although we are doing our best to take into account time aggregation issues, there could be some remaining measurement issues. Um and so that component SB is uh taking this into [03:38:33] account. It's the share of job postings that actually corres that actually have openings in the jolts data. So you've got to deflate the um burning glass postings B to take into account that some of them don't correspond to jolts [03:38:48] which is sort of for these purposes considered the ground truth. So the overall adjustment factor A is the product of these three and you can think of these these are moments that you would calculate on a somewhat aggregated [03:39:00] level by uh for instance firm type J and then what you can also clearly see then is or it might be clear is that the the extensive margin is acting like a sampling weight. So you're going to be scaling up uh firms that are not very [03:39:15] well represented online and using the characteristics of the ones that are represented online. And that's exactly how our reweing is going to work. [03:39:24] Um, so we can calculate these moments. There are going to be a series of moments underlying that are producing those three components and then also producing the aggregate adjustment. [03:39:34] We're going to be calculating those for a wide range of of the firm characteristics that we have. And that's going to allow us to describe the variation between the two data sets. Um, but also there's no reason for us to limit ourselves to the jolts because all [03:39:48] of the characteristics in our model are available for the full QCEW. So what we do is we take we we take a model estimated within the jolts and then extrapolate to the full burning glass [03:39:59] sample um to uh basically estimate these weights or adjustment factors on a much larger uh sample where we can um test out questions that researchers have asked uh with the burning glass data. [03:40:13] Um, basically we can run a regression that includes all these QCW characteristics. Um, and but then we also do a second version where we only use characteristics that are available in the in the burning glass data. Um, [03:40:27] uh, because researchers won't have access to this merge in in practice. So, we choose characteristics that are going to help predict the key sources of differences between the two data sets and we're working on being able to release uh those weights. We've done [03:40:41] quite a bit of validation on the model and that's all going to be in the paper. [03:40:44] But now in the interest of time, I'm just going to go ahead and show you some results. [03:40:50] So this is um the average adjustment factor the an establishment level average adjustment factor over time. The burning glass data were available for 2007 and then there was a gap of two years where there are no data and then [03:41:04] we have data to the present. Um and so the declining adjustment factor means that there was quite a bit of convergence between the two data sets over time. So in 2007 this value which [03:41:16] is about 28 means a posting by the average establishment in burning glass needs to be scaled up by about 28 to meet the average o uh opening in um in [03:41:28] the jolts. Um but that falls by about four-fold uh by 2013. And then from there the the series relationship is is fairly stable. Um so there's quite a bit of convergence between the two series [03:41:43] which again that's what's implied by the decline here. Um that becomes fairly stable after 2013. Um we can decompose this into our three components and find that the extensive margin is doing the most work. So there's been the biggest [03:41:57] increase in the likelihood of posting online given that you have an opening. [03:42:02] We can also look at this time series holding constant the composition of firms as we can measure them with all of our firm characteristics and we find still this very similar decline. So this this convergence between the two series [03:42:15] that's indicated here holds within uh firm characteristic but let me show you a little bit about the differences across the characteristics. So one characteristic that generates a pretty strong gradient in the adjustment factor is is these [03:42:29] wage bins. So in the QCW we have the total wage bill and we we uh calculate a wage bill per worker metric and then just divide it into quintiles. And this steep relationship here shows that if you're in the low if you're a firm [03:42:44] that's in the lowest quintile, you have a very high adjustment factor, meaning you're not well represented online, your postings need to get scaled up by a lot to match the distribution of openings. [03:42:55] Um but then this adjustment factor falls by um by more than four-fold for the highest paying firms who are very well represented online where one which is indicated with this line would be perfectly well represented. So we do see this strong gradient with respect to [03:43:10] firm size. We also see a very strong gradient with respect to urbanity where the the central cities and the and the large fringe areas are very well represented but then the non-core uh areas and micro uh statistical areas are [03:43:24] not very well represented. Those gradients uh show up as the as really among the most important in addition to industry which as you could imagine is has a very uh strong gradient. [03:43:35] Um size is also very important for trying to understand the impact on previous researchers questions when we take into account this difference in representation. [03:43:46] Um you can see that there's a reasonably strong gradient with respect to size for these midsize firms. Here I have size bins. So the first bin is 2 through 9, 10 through 49. 50 through 250 is one that is is the bin that has the highest [03:44:01] adjustment factor which means it's not well represented online. and then the adjustment factor falls in half. The largest firms are much better represented online. [03:44:11] But why the the um relationship doesn't continue? The smallest firms are sort of where we're having this issue with our nuisance parameter. So small firms given that they're recruiting in jolts, they actually are very unlikely to have [03:44:24] postings online. But also if they have postings online, they're actually not that likely to be recruiting in jolts. [03:44:30] So we have some kind of mismatch here. And it may be because it's harder for us to merge small firms. It may be because of Jolt sampling and the relationship between size and age. And we have looked into this quite a bit. And for the for the thing that I would like you to take [03:44:45] away now is that any conclusions I'm going to tell you later are not dependent on this this these smallest firm bins. And it turns out that this gradient here with all of these groups being well being less well represented [03:44:59] in jolts compared to the largest firms is the thing that's going to be really important for understanding these previous researchers questions including myself. I don't know why I keep calling them previous researchers. We're going to look at my research questions too. [03:45:10] It's fair game. Um and then um a couple of other characteristics. We don't see too much gradient with respect to firm age bins or firm growth bins. Um, and we've also looked at a wide range of others and all of that is going to be in [03:45:23] the paper. Okay. So, we've learned quite a lot already about uh the differences between the burning glass and the jolts. [03:45:31] Do do these matter? So, um, again, we can use our adjustment as a kind of sampling weight to rewe the data. We don't have to do this in the Jolt. We can do this for the f full burning glass data set as long as we can match it to [03:45:44] these QCW characteristics. And that's what we do. And I'm just going to briefly summarize a couple of exercises for you. Um, one use that I've made of the burning glass data is in studying um, firms demand for skill, especially [03:45:59] skills that what that are uh, likely to increase when a firm is using labor replacing technology. So, abstract tasks, high more educated tasks, computer skills. Um and in a previous paper with Brad Hersby, I showed that [03:46:12] after the Great Recession, labor markets that were harder hit by the Great Recession um saw differential increases in demand for those skills. Um and that in conjunction in conjunction with data on uh capital and and other measures of [03:46:26] firms production functions helped us to argue that actually uh many firms took the opportunity of the Great Recession to accelerate their adoption of labor replacing um technologies. [03:46:36] And that paper relied heavily on this 2007 benchmark which was the only pre- period for burning glass that we had. [03:46:43] And if the composition of firms were changing rapidly over that time period in in ways that could potentially bias us then that could um overturn that conclusion. Um but fortunately you can rest assured that that actually the distribution of skills in burning glass [03:46:58] both in levels and cyclicality are actually very well um uh uh representative of the jolts. Um again because most firms are large, most people work at large firms and it then it turns out that most postings are well [03:47:13] represented if you can look at the postings of basically large firms. [03:47:17] Where this matters more is in measuring local labor market concentration. So many researchers have taken advantage of the the breath of Burning Glass and other online job postings data sets to measure a Herendel index uh of a [03:47:32] concentration of firms that are hiring. So a Herendall index is the uh sum of squared shares across firms that are hiring in a local labor market. Um and so you can measure this with job postings. Um but what we'd really want [03:47:47] to know if you want to know about a Herendel if about the concentration of hiring firms is you'd like to know that for actual openings or actual hires which is here we can use our adjustment to adjust the second moment as well. um and then we can therefore give you an [03:48:02] adjusted uh Herffendall index of postings. So here is one summary of the relationship between the actual Herendall index of postings uh by market versus the adjusted Herendel index. So [03:48:16] underlying this bin scatter are all of the local labor markets in the US. Here we're doing commuting zone by fourdigit industry, but you could do this a number of different ways. And you see the slope of this line is about.3 [03:48:30] um and far below the 45 degree line. And so what this means is the level of labor market concentration needs to be scaled down to about a third of what you would measure in burning glass because of this big tail of small and medium firms that [03:48:44] burning glass doesn't capture. That makes a big difference in the Herendall calculation. [03:48:50] However, there's still information content in these market concentration measures. Here, what we do is we just look at ordinal comparisons. So, you can rank local labor markets based on their [03:49:02] Herendall index of um actual postings and then rank them based on the Herendall index using our adjustment. [03:49:10] and uh you find a much stronger relationship that's much closer to the 45°ree line. Which means if you want to put a Herendall on the right hand side of your regression, you're going to capture meaningful variation in um uh the relation in concentration across [03:49:24] markets. One should just not take seriously the level of concentration which which people have tended to do. Um so that's sort of a brief summary of what we've done. We merged 90% of job [03:49:36] postings in Burning Glass to the QCW and the Jolts and were able to explore systematic variation in Burning Glass postings both over time and across establishment characteristics and we find mostly a positive result that [03:49:50] burning glass well matches the Jolts distribution even without our adjustment weights because of the large firms are very well represented. But for some research questions about the distribution of firms that are hiring, one would really want to use um our [03:50:03] adjustment weights or or something else uh because of the under reppresentation of small firms. Okay, thank you. ## Discussion — Gad Levanon (04:13:04 – 04:28:34) *Shared discussion for Session 3: Job Openings, Occupations, and Technological Dynamism; the discussant covers every paper in the session.* [04:13:04] Okay, thank you Hiju. And our discussant for this session is Gad Lebanon. [04:13:51] Thank you. Hi everyone. H it's a a pleasure to be here. Um I am very happy to uh to been assigned those two papers. I'm I'm a [04:14:03] user more of than a creator of data and I work a lot with job postings and I uh also work a lot on AI and its impact. So [04:14:15] I really learned a lot from those uh two papers. Um I will briefly talk I I will um everything here is brief but I will uh [04:14:27] give some big picture um comments on the the two papers um and I will sprinkle a little uh big and bold ideas here and there uh to to [04:14:40] add to the conversation. Um just one general uh comment also listening to the discussion here uh in in recent hours. I I think [04:14:54] what we've learned or listened to today mostly is how to use alternative data to improve uh official data. But I think the the real opportunity of alternative data is actually to use the alternative [04:15:08] data itself to do analysis. And I think researchers should spend more of their time improving those alternative measures to make them uh better and more [04:15:21] useful. So I'm very happy that Lisa and her colleagues decided to do that. Um, one uh clarifying comment. Lisa kept talking about the burning glass data and [04:15:36] I'm the co-founder of the Burning Glass Institute. So, I just want to tell you that it's completely two different organizations. [04:15:44] It's not our data. The only reason they are called the same is because the CEO of Burning Glass Technologies that later became Litecast is Matt Seagelman and he's also the president of the Burning [04:15:57] Glass Institute. But we are completely different organization. It's not our data. If anything, there is a mutual feeling of hostility and animosity between the organization. [04:16:10] erh they indeed sued us to want us to change our name. So um so you know my bias is going in um so I'll start with [04:16:23] with Lisa's paper which I think is I didn't read all papers but I think it's probably the best paper we have on the biases of online uh job openings. Um [04:16:38] uh and I'm not going to go go over everything they found, but um I think you you can think of it as there is the raw postings. [04:16:51] There are adjusted postings that you can adjust yourself using publicly available data like the jolts breakdowns by industry and and region. And then there is the adjustment that you can make with [04:17:05] Lisa's method. Um and uh I think uh this is a a a better it's clearly better um than raw postings and that's what I I [04:17:19] think the the author showed the how how big the the the difference when you adjust um using their their method. I I think one thing I would recommend for the paper is [04:17:33] also to try since a lot of people are able to adjust themselves using the jolts publicly available data. How much how different is your your results your adjusted [04:17:46] results would be compared to adjusting it with publicly available data. I believe it will be better because you have information [04:17:58] on variation within industry that you can't just do with jobs. [04:18:05] Um yeah, so that that is one uh suggestion to to improve the data or or show that it is more useful. [04:18:15] Um here is what I So I'm very proud of my hair. I in my age I have a full set no hair loss [04:18:29] and very little white hair. But uh the white hair that I do have is trying to because of trying to make sense of time [04:18:40] series of of job postings. And uh I I think um most users of uh of postings are looking at changes over [04:18:54] time as opposed to just using level. So if let's say QCW in terms of quality is at like 97 and CES is at 90 [04:19:07] I think uh levels of posting are at 70 and time series of posting are like 13 14. So that's I think where a lot of the [04:19:20] work I think needs to happen. So I encourage um I encourage you to use it. [04:19:26] H I think this method can help but uh to fully fix it I think uh there's a lot of un um glamorous work in dealing with things like uh uh job posting coming and [04:19:40] going scraping from other job boards all kinds of a lot of non-economic reasons so I think calling for the community h to um [04:19:52] to to do uh that I think that's there is the great potential when it comes to posting is improving the the time series. [04:20:02] Um how can it help the rest of us? I think you are producing a set of weights uh if we want to better improve our uh postings. If you publish those weights [04:20:15] or better if BLS publishes those weights uh and takes over uh this I think it will be a good public service. In fact, I think BLS should take all have their [04:20:28] own postings data set and uh have another program. I'm sure I'm not the first one who thinks about it, but big and bold ideas that is one. Um and I think the the method is bigger [04:20:42] than this paper. I think any public any any private data that has an employer identified you can and I'm curious did like is any other data set ADP or or [04:20:55] anything are they using something like that probably not so I think there is a lot of opportunity here to check the biases of many other databases using your your method including our own burning glass [04:21:09] institute data and be happy to to try it on So that's uh the first paper. I'm already behind. Um so let me get to the other paper. Um and give you some context. [04:21:24] So first of all uh there is a big debate. Has a AI already displaced jobs? I think the answer is yes. The big question is how much? I think it's above 100,000 and below 5 [04:21:38] million. How did I get to those numbers? I I made them up but I think 95% of the people would agree that that's the number of displacement that already [04:21:49] happened but it's a big uh range and uh 100,000 is very different than 5 million. So we need to know better. Um I think no no studies found aggregate [04:22:02] effect. Um some found for young workers h in most exposed jobs. uh but uh this paper um did find aggregate effect. So this in terms of I think [04:22:16] you're holding me back upside down. I have five minutes not two. Um so so um and I think well I am in trouble now. uh so let me think what I'm going to the [04:22:30] point here that there are reasons anything that based on exposure measures I think it's uh the exposure measures that are lundu and felton they have very little to do with job [04:22:44] displacement so any paper that uses those measures is um will it will be a miracle to find a a significant impact and that's where the the paper comes in [04:22:57] I I think they it's a high uphill climb to to get significant results with using this exposure measures. But the the big idea here is that yeah, let's add to the [04:23:10] exposure measure a um an adoption measure because even if the technology there if the employers don't make an effort to adopt the technology, you won't see labor [04:23:23] market impacts. Uh so I think that that is a a good idea. Um as I mentioned before I I question the exposure measure. I've worked it to myself. I think it's [04:23:36] it's such a tiny part of what how AI is impacting um the labor market. So using it as the sole proxy is is is [04:23:48] a non-starter in my opinion. But adding to the to adoption measure I I think is in the step in the right direction. I think the adoption measure essentially as it's built now is more of [04:24:02] a function of the occupation distribution of the of each industry. So I I think you could find better adoption measures. But when I looked at the [04:24:14] results I kind of tend to believe them. erh the the ones at the top um are indeed I I'll finish in in two minutes so forgive me we don't have any other presentation after that right so we can [04:24:28] go for a long time um so uh um the ones at the top I really do believe that are in the top and the one at the bottom are are at the bottom I think for good reasons I [04:24:44] think there are ways to improve the the adoption um adoption measure and I am happy to chat with the authors afterwards on how to do that. Um what do I make of the results? So first just I [04:24:59] think the paper needs a a a clear table that shows all the specifications for all the outcome variables and the results. Otherwise it's hard to to tell um what's really happening here. But I [04:25:13] do believe that adoption helps when you interact it with the exposure measure. [04:25:20] Um I think taking it to CPS is the right move because you can then see flows and not just talks and it seems like the results are stronger for for [04:25:32] um for flows. I I would strongly recommend using this also on the ACS and the OEWS. [04:25:40] They are much larger. The OES has its issue in time series, but once there are ways to get around it and and it's extremely strong and well measured. Um, but at the end of the day, I think the [04:25:54] results are only as good as the exposure and adoption measures behind them. And I think that's not just for this paper, but for the community at large, it is you are bound to uh to fail if that's [04:26:07] like your exposure measure. That's how you measure the impact of AI. Um well, it's up to you, but I have four big and bold ideas. If you want to give me two more minutes, [04:26:21] one minute. Okay. So, um I thought like because I was asked by the the organizer to also think go beyond those two papers and think about other things. So I just [04:26:32] want to give you a quick look at um how profiles which are LinkedIn profiles and postings can do for studying AI things that you cannot do in your uh with [04:26:47] government official data. So for example use cases in postings uh there are a lot of information in job postings not just about the candidate but about what the project that they are going to work on. [04:27:02] We we created an AI use cases um from postings and career histories. Uh hundreds of thousands of uh of AI use cases like build a chatbot for customer [04:27:14] service um build an AI tutor things like that. Now you could uh once you have this use case you can connect the use case uh to the impacted occupation like [04:27:28] in the example of customer service chatbot the impacted occupation is a a customer service worker. So you can use that to create a an kind of an an a new [04:27:41] kind of adoption or deployment index that is based not on theory but on actual actions of that employers are are taking. Uh you can also uh build like if [04:27:53] you now I asked the BLS map me the all the jobs that were created because of AI. There is no way that that you can do that. But with for example postings and [04:28:06] career and LinkedIn you can do that. You can identify all AI companies. you can identify AR roles in nonAI companies and I we started working on that and I I [04:28:20] think you'll be able to have like a a number that is quite believable of the number of jobs that would not have existed if AI didn't exist. So that's another example that uh where you can ## Q&A (04:28:34 – 04:37:59) *Shared floor discussion for Session 3: Job Openings, Occupations, and Technological Dynamism.* [04:28:34] use it. Um and let me pause here. I very gen thank you for your generous time counting and uh thank you >> and thank you Gad for that nice uh [04:28:54] discussion. So we did uh and do have some time for for questions. We're running a bit behind, but uh we've had some great uh presentations and just wanted to leave it open if anybody has [04:29:07] any questions or comments. And I see there's some microphones floating. There you go. [04:29:14] >> Yeah. Uh I guess I introduced myself on recovery. Um so just following up on discussion at the end. Um the adoption measure. Um so so with the exposure [04:29:27] measure I can see clearly you know this is it's a notion of sub substitution right with the adoption measure a company posts uh you know requirements for any ice I'm not so sure this is about substitution [04:29:42] and not compliment right and and so I was even more surprised to see then that if I read this correctly we saw I guess high high was the biggest effect I could have easily seen it's high low that that [04:29:57] um this is the biggest effect. Anyway, I'm not so sure that just from the naive sort of outside view that your adoption measure is really about substitution which I suppose you wanted to uh measure [04:30:10] but anyway that's my question. Thanks Mike institute. Um so having done [04:30:32] some work in AI and looking at onet data and I know we have a job posting data. [04:30:38] Um, I think there's a limit in terms of what we can learn just from existing databases. And one of the things I would encourage us to think about is is there a need to follow up Gad's comment on use cases to do a lot more qualitative research um sort of in real time to [04:30:53] figure out what's going on in industry right now. Um, so that I think the databases themselves are slow to catch up like ONET is slow. Um and I think that uh some sort of vehicle by which we can get data perhaps through the job [04:31:06] post and talking to incumbents and occupations or industry experts in terms of what they're doing could be a very useful way of sort of approaching this. [04:31:28] Anybody else? Okay, looks like not >> Oh, sorry. [04:31:39] >> Thank you. John Paul for city. Just a question for Lisa about your nuisance parameter. You're talking in part about firms who appear in the Burning Glass, but then you don't see them actually having vacancies. So, there are firms [04:31:53] who will just go fishing in the labor market and they're always posting just to see what comes up. So I guess one way you observe that in the data is you're looking for firms who are always [04:32:05] posting and and never hiring very rarely hiring. So have you looked at that kind of pattern? Can that help you understand your nuisance parameter and maybe do something uh additionally to maybe take [04:32:18] those firms out of the the burning glass data set? [04:32:34] Yeah, thank you. I think that's a great question. Um, yes. So, these are firms who um they have postings. They didn't have hires and they also don't have unfilled vacancies according to the Joel's definition and Jolt's definition has these requirements like active [04:32:48] recruiting. Um, and so it's I don't think it's that likely that they're the the phantom vacancies, but I think your other question is about the firms that just always have a posting all the time. [04:33:00] And so that's really what's going and so those are what what the white cast and glass people do is just every now and then they call it a new posting. Um, and so so like after 60 days they the same [04:33:13] thing's been up. So like fast food they just always have these openings up, right? Um, and uh so the they recycle it. We did try our best to look at the rates of that the rates of that across industries for instance and it doesn't [04:33:28] really match the patterns. Um but I don't actually uh for instance like food and accommodation services it doesn't have a particularly high nuisance factor. Um but I haven't actually crossed that with size to see if it's small accommodation look at that. Thank [04:33:43] you. PG, did you? [04:33:57] >> Um, thank you so much for the great questions and the um support of the discussion. Um really appreciate you know all of um your comments and also questions um raised by the audience. So [04:34:11] um let me address you know a few um issues that the discussion raised. the first of all you know whether this AI exposure adoption that we currently have is the best thing certainly I think it's an open research you know area and you [04:34:24] know we are actually interested in you know exploring the novel cultures so that you know so in that sense your comment about maybe perhaps we can get comparisons about you know what what AI is really doing for the labor market outcomes based on use cases and also you [04:34:38] know as Michael said you know the qualitative research on the measuring um you know the AI is large policy effects you know based on the you know the real time data well you know there certainly you know open um you know research [04:34:52] question and you know we're very interested in you know pursuing um that route as well um but about the issue that whether this AI adoption is you know the substitute or compliment and [04:35:04] why does it really have destruction effect it's actually open question it's also an empirical question while we claim in our papers that AI adoption represents you know demand for AI you [04:35:18] know using role but we didn't you know make any assumption about whether it's a compliment or substitute you know so but empirically we found out that workers who are highly a exposed AI and adoption and also workers you know who you know [04:35:32] whose job you know has is exposed to high AI adoption their unemployment goes more and they have experienced a lower and you know job rate and also they experienced you know uh larger decline in the job buying rate. So it's an [04:35:46] empirical question what's really doing and certainly it's also a you know like research question as well and they comment about you know what AI adoption is really measuring um so it's actually a pretty good question they will get the [04:36:01] same adoption you know like orders or similar coariation or the magnitude if we aggregate the exposure measures using the occupation and employment weight we'll get the same thing or not that's [04:36:14] question and um but you know but you know in this paper we actually show that there's an interesting margin that has not been considered in the literature. [04:36:24] So for example you know there are managers you know who who are in the industry with the higher or lower adoption. So for example, there are managers in the construction industries and managers in the computer services industry and they seem to have you know [04:36:38] different types of the labor demand jobs and same for teachers but teachers are low exposure group but there are teachers who are in K12 schools and teachers in college and universities and they seem to have different labor market outcomes. So in that sense our claim [04:36:52] again is that the what is captured the interaction between exposure and adoption can be you know something important predictor of the you know labor market uh outcomes. Um so Nick do you have something [04:37:06] okay thank you so much you learned a lot. [04:37:09] >> Okay let's thank our authors uh presenters and discuss them. [04:37:17] Okay. Uh, so if you signed up for dinner, we have an announcement. So the the location is the Sapphire Room and it's on the penthouse level. So if you go out these doors and to the right, take the elevator to the penthouse level. Um, and it's at the end of the [04:37:32] hall is where you'll find the restaurant. Um, and that starts about 6:00 where the reception time. [04:37:56] 12 seconds.